虚拟机通过L3agent访问外网的流程测试

by

2016-11-13

23:30:52

Leave a comment on 虚拟机通过L3agent访问外网的流程测试

Networking

,

已经初始化过的租户，创建一台虚拟机，只带内网（私有网）作为源HOST，目标HOST找一个外网IP即可，最好能够自行抓包

SRC：192.168.65.125
DST：2.106.123.234

Step1：

SRC端发送ICMP包，DST进行抓包

~$ sudo tcpdump -i eth1 icmp and dst host 2.106.123.234
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
22:23:49.856024 IP 236.115.121.34 > 2.106.123.234: ICMP echo request, id 21474, seq 1, length 64
22:23:50.856840 IP 236.115.121.34 > 2.106.123.234: ICMP echo request, id 21474, seq 2, length 64
22:23:51.858001 IP 236.115.121.34 > 2.106.123.234: ICMP echo request, id 21474, seq 3, length 64
22:23:52.858985 IP 236.115.121.34 > 2.106.123.234: ICMP echo request, id 21474, seq 4, length 64

可以看到SRC访问外网，的确访问成功了    ，但是比较奇怪的是SRC IPaddr变了，感觉像是做了转换

Step2：

分而治之，进入租户Router network namespace，首先查看设备

~$ sudo ip netns exec qrouter-0aeca8b7-8a38-41b1-ace7-ee9918e2b229 ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
4192: tun0:  mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100
    link/none
    inet 192.168.66.1/23 scope global tun0
       valid_lft forever preferred_lft forever
2182112: ha-2b8cf5e0-b1:  mtu 1400 qdisc htb state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:8e:51:d7 brd ff:ff:ff:ff:ff:ff
    inet 192.168.65.124/23 brd 192.168.65.255 scope global ha-2b8cf5e0-b1
       valid_lft forever preferred_lft forever
    inet 192.168.64.1/23 scope global secondary ha-2b8cf5e0-b1
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe8e:51d7/64 scope link
       valid_lft forever preferred_lft forever
2182122: qg-0aeca8b7-8a:  mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 00:16:3e:fe:2b:11 brd ff:ff:ff:ff:ff:ff
    inet 169.254.43.17/18 brd 169.254.63.255 scope global qg-0aeca8b7-8a
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fefe:2b11/64 scope link
       valid_lft forever preferred_lft forever

重点关注HA和QG两个设备，接着查看一下namespace里的防火墙规则

~$ sudo ip netns exec qrouter-0aeca8b7-8a38-41b1-ace7-ee9918e2b229 iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N neutron-postrouting-bottom
-N neutron-vpn-agen-OUTPUT
-N neutron-vpn-agen-POSTROUTING
-N neutron-vpn-agen-PREROUTING
-N neutron-vpn-agen-float-snat
-N neutron-vpn-agen-snat
-A PREROUTING -j neutron-vpn-agen-PREROUTING
-A OUTPUT -j neutron-vpn-agen-OUTPUT
-A POSTROUTING -j neutron-vpn-agen-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-postrouting-bottom -j neutron-vpn-agen-snat
-A neutron-vpn-agen-POSTROUTING ! -i qg-0aeca8b7-8a ! -o qg-0aeca8b7-8a -m conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-vpn-agen-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-vpn-agen-snat -j neutron-vpn-agen-float-snat
-A neutron-vpn-agen-snat -s 192.168.64.0/23 -j SNAT --to-source 169.254.43.17

看下SNAT规则，将所有SRC地址在192.168.64.0/23这个CIDR段中的包的SRC IPaddr转换成169.254.43.17，从这里应该可以得到结论是包从虚拟机里到网关，路由之后，被iptables给进行了SNAT处理进行源IP地址转换，通过抓包验证

抓HA设备，此时源IP地址并未发生改变

~$ sudo ip netns exec qrouter-0aeca8b7-8a38-41b1-ace7-ee9918e2b229 tcpdump -i ha-2b8cf5e0-b1 icmp and dst host 2.106.123.234
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ha-2b8cf5e0-b1, link-type EN10MB (Ethernet), capture size 262144 bytes
22:34:07.158564 IP 192.168.65.125 > 2.106.123.234: ICMP echo request, id 21972, seq 1, length 64
22:34:08.160388 IP 192.168.65.125 > 2.106.123.234: ICMP echo request, id 21972, seq 2, length 64
22:34:09.161634 IP 192.168.65.125 > 2.106.123.234: ICMP echo request, id 21972, seq 3, length 64
22:34:10.163036 IP 192.168.65.125 > 2.106.123.234: ICMP echo request, id 21972, seq 4, length 64

抓QG设备，可以看到都做了源地址转换

~$ sudo ip netns exec qrouter-0aeca8b7-8a38-41b1-ace7-ee9918e2b229 tcpdump -i qg-0aeca8b7-8a icmp and dst host 2.106.123.234
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on qg-0aeca8b7-8a, link-type EN10MB (Ethernet), capture size 262144 bytes
22:37:25.055601 IP 169.254.43.17 > 2.106.123.234: ICMP echo request, id 22115, seq 1, length 64
22:37:26.057475 IP 169.254.43.17 > 2.106.123.234: ICMP echo request, id 22115, seq 2, length 64
22:37:27.058381 IP 169.254.43.17 > 2.106.123.234: ICMP echo request, id 22115, seq 3, length 64
22:37:28.059661 IP 169.254.43.17 > 2.106.123.234: ICMP echo request, id 22115, seq 4, length 64

Step3：

进入l3agent这个namespace，查看下设备

~$ sudo ip netns exec l3agent ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
10: eth0.102@if6:  mtu 1500 qdisc noqueue state UP group default
    link/ether a0:36:9f:51:55:34 brd ff:ff:ff:ff:ff:ff
    inet 192.168.8.34/22 brd 192.168.11.255 scope global eth0.102
       valid_lft forever preferred_lft forever
    inet6 fe80::a236:9fff:fe51:5534/64 scope link
       valid_lft forever preferred_lft forever
20: eth2.106@if2:  mtu 1500 qdisc noqueue state UP group default
    link/ether ec:f4:bb:c8:e1:0c brd ff:ff:ff:ff:ff:ff
    inet 236.115.121.34/25 brd 236.115.121.127 scope global eth2.106
       valid_lft forever preferred_lft forever
    inet6 fe80::eef4:bbff:fec8:e10c/64 scope link
       valid_lft forever preferred_lft forever
31: ex-l3dngw:  mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 00:16:3e:fe:00:01 brd ff:ff:ff:ff:ff:ff
    inet 169.254.0.1/18 brd 169.254.63.255 scope global ex-l3dngw
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fefe:1/64 scope link
       valid_lft forever preferred_lft forever

同样，先看下防火墙规则

~$ sudo ip netns exec l3agent iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N neutron-postrouting-bottom
-N neutron-vpn-agen-OUTPUT
-N neutron-vpn-agen-POSTROUTING
-N neutron-vpn-agen-PREROUTING
-N neutron-vpn-agen-float-snat
-N neutron-vpn-agen-snat
-A PREROUTING -j neutron-vpn-agen-PREROUTING
-A OUTPUT -j neutron-vpn-agen-OUTPUT
-A POSTROUTING -j neutron-vpn-agen-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-postrouting-bottom -j neutron-vpn-agen-snat
-A neutron-vpn-agen-POSTROUTING -s 169.254.0.0/18 ! -o ex-l3dngw -j MASQUERADE
-A neutron-vpn-agen-snat -j neutron-vpn-agen-float-snat

看倒数第二条，169.254为源IP的包，不能再从ox-l3dngw这个设备走，而包本来就是从这个设备来的，不能再从这个设备回去，因此包接下来会被转发到剩下的两个设备继续走下去

简单看下路由走向

~$ sudo ip netns exec l3agent ip r get 2.106.123.234
2.106.123.234 via 236.115.121.1 dev eth2.106  src 236.115.121.34
    cache

可见接着包会从eth2.106这个设备出去

先抓包试试，设备ex-l3dngw

~$ sudo ip netns exec l3agent tcpdump -i ex-l3dngw icmp and dst host 2.106.123.234
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ex-l3dngw, link-type EN10MB (Ethernet), capture size 262144 bytes
23:16:55.604382 IP 169.254.43.17 > 2.106.123.234: ICMP echo request, id 23916, seq 1, length 64
23:16:56.606226 IP 169.254.43.17 > 2.106.123.234: ICMP echo request, id 23916, seq 2, length 64
23:16:57.607478 IP 169.254.43.17 > 2.106.123.234: ICMP echo request, id 23916, seq 3, length 64
23:16:58.608955 IP 169.254.43.17 > 2.106.123.234: ICMP echo request, id 23916, seq 4, length 64

设备eth2.106的IP地址十分熟悉，一开始在DST HOST抓包，显示的SRC IPaddr就是236.115.121.34，因此八九不离十源IP会被转换

~$ sudo ip netns exec l3agent tcpdump -i eth2.106 icmp and dst host 2.106.123.234
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth2.106, link-type EN10MB (Ethernet), capture size 262144 bytes
23:18:49.841721 IP 236.115.121.34 > 2.106.123.234: ICMP echo request, id 24014, seq 1, length 64
23:18:50.850721 IP 236.115.121.34 > 2.106.123.234: ICMP echo request, id 24014, seq 2, length 64
23:18:51.839906 IP 236.115.121.34 > 2.106.123.234: ICMP echo request, id 24014, seq 3, length 64
23:18:52.840890 IP 236.115.121.34 > 2.106.123.234: ICMP echo request, id 24014, seq 4, length 64

这样包就最终达到了目的端公网IP那里

 

发表回复