Stuff, that helped me to add IP-subnet to KVM host

Posted on

I had a server, that run Centos8 and had several KVM-virtualized machnies. Most of virtual servers had dedicated IPs and when one customer wanted to get several IPs, our ISP instead of individual IPs gave us whole subnetwork (/28). For some reason I really struggled several days to make it working.

Just for the history and “just in case someone has the same situation” or “i will need it again”.

As I mentioned, before I started – I had several machines, that were using individual IPs and I had one virtual network switch for local/private servers, which was operating in NAT mode.

I was not expecting anything complicated – created second network switch for granted IP range and that’s where I stopped.

Problem one. I was not able to activate new network inside cockpit web interface due to some mystical PTR error. To fix this I had to edit network configuration XML and remote “ptr” option.

# list all network switches/networks
virsh net-list --all
# edit specific network
virsh net-edit <network_name>
# start network
virsh net-start <network_name>
# set network to auto-start
virsh net-autostart <network_name>

Good sources for reading and commands:
https://linuxconfig.org/how-to-use-bridged-networking-with-libvirt-and-kvm
https://computingforgeeks.com/how-to-create-and-configure-bridge-networking-for-kvm-in-linux/
https://libvirt.org/sources/virshcmdref/html/sect-net-undefine.html

 

All right – this got me second virtual network up and running, I was so happy! Until I realized there was a really strange behavior or we call it problem two – for some reason traffic was flowing from the external sites to new IP-range, but not the other way. I was able to ping newly created machine which was assigned to new network. I was able to ping host-ip from that new machine, but I was not able to ping any external IP.

I don’t know, what was wrong. I could not find out. But here is a solution, that worked for me.

  • deleted newly created virtual network with public IP range (since it was new, it was not yet in any use)
  • stopped servers, which were using local IPs. These were some technical machines and they could stay offline without causing problems.
  • deleted default (LAN) virtual network
  • created virtual network for public IPs (range, that I have just got from ISP). I still have same issue with PTR error, so but I already knew how to fix it (problem one)
  • created virtual network for local servers. I did not use name default.

It all just started to work as it should!!! I was so happy again!

Finally I had to solve the smallest issue or the third problem. This problem was caused by non-trivial situation – I had to add this new IP subnet to one of the servers, which was already using IP from another network. When I did that – of course – server was sending replies through it’s main network interface, which was wrong – basically you never get ping reply this way. Server has to reply from the same IP/network, as the one where it received request on.

It was easy to fix. Here are 3 commands, that I run on a VPS (running Centos7) with two different networks:

echo 200 ips2 >> /etc/iproute2/rt_tables
ip rule add from <new_network_ip>/28 table ips2
ip route add default via <new_network_router_ip> dev eth1 table ips2

Small explanation

  • ips2 – just a tag/name – you can choose something else;
  • <new_network_ip>/28 – this can be either single IP or network/network mask – this is what we got from our ISP. Using network range would make routing easier for several IPs.
  • eth1 – this is secondary network adapter, that I had to add to server and which is assigned to new virtual network (the one with <new_network_ip>)

These commands will work instantly. Remember to make them permanent by adding the following info to config files. Pay attention to ethernet device, routing tag and IP.

echo "from <new_network_ip>/28 table isp2" > /etc/sysconfig/network-scripts/rule-eth1
echo "default via <new_network_router_ip> dev eth1 table isp2" > /etc/sysconfig/network-scripts/route-eth1

Source for this solution:
https://unix.stackexchange.com/questions/4420/reply-on-same-interface-as-incoming

 

Finally customer was happy too! 🙂

 

Linux, Security , , , ,