Stuff, that helped me to add IP-subnet to KVM host
I had a server, that run Centos8 and had several KVM-virtualized machnies. Most of virtual servers had dedicated IPs and when one customer wanted to get several IPs, our ISP instead of individual IPs gave us whole subnetwork (/28). For some reason I really struggled several days to make it working.
Just for the history and “just in case someone has the same situation” or “i will need it again”.
As I mentioned, before I started – I had several machines, that were using individual IPs and I had one virtual network switch for local/private servers, which was operating in NAT mode.
I was not expecting anything complicated – created second network switch for granted IP range and that’s where I stopped.
Problem one. I was not able to activate new network inside cockpit web interface due to some mystical PTR error. To fix this I had to edit network configuration XML and remote “ptr” option.
# list all network switches/networks virsh net-list --all # edit specific network virsh net-edit <network_name> # start network virsh net-start <network_name> # set network to auto-start virsh net-autostart <network_name>
Good sources for reading and commands:
All right – this got me second virtual network up and running, I was so happy! Until I realized there was a really strange behavior or we call it problem two – for some reason traffic was flowing from the external sites to new IP-range, but not the other way. I was able to ping newly created machine which was assigned to new network. I was able to ping host-ip from that new machine, but I was not able to ping any external IP.
I don’t know, what was wrong. I could not find out. But here is a solution, that worked for me.
- deleted newly created virtual network with public IP range (since it was new, it was not yet in any use)
- stopped servers, which were using local IPs. These were some technical machines and they could stay offline without causing problems.
- deleted default (LAN) virtual network
- created virtual network for public IPs (range, that I have just got from ISP). I still have same issue with PTR error, so but I already knew how to fix it (problem one)
- created virtual network for local servers. I did not use name default.
It all just started to work as it should!!! I was so happy again!
Finally I had to solve the smallest issue or the third problem. This problem was caused by non-trivial situation – I had to add this new IP subnet to one of the servers, which was already using IP from another network. When I did that – of course – server was sending replies through it’s main network interface, which was wrong – basically you never get ping reply this way. Server has to reply from the same IP/network, as the one where it received request on.
It was easy to fix. Here are 3 commands, that I run on a VPS (running Centos7) with two different networks:
echo 200 ips2 >> /etc/iproute2/rt_tables ip rule add from <new_network_ip>/28 table ips2 ip route add default via <new_network_router_ip> dev eth1 table ips2
- ips2 – just a tag/name – you can choose something else;
- <new_network_ip>/28 – this can be either single IP or network/network mask – this is what we got from our ISP. Using network range would make routing easier for several IPs.
- eth1 – this is secondary network adapter, that I had to add to server and which is assigned to new virtual network (the one with <new_network_ip>)
These commands will work instantly. Remember to make them permanent by adding the following info to config files. Pay attention to ethernet device, routing tag and IP.
echo "from <new_network_ip>/28 table isp2" > /etc/sysconfig/network-scripts/rule-eth1 echo "default via <new_network_router_ip> dev eth1 table isp2" > /etc/sysconfig/network-scripts/route-eth1
Source for this solution:
Finally customer was happy too! 🙂