devstack默认还是集中式Router，虽然neutron能够创建分布式Router，但是没法绑到L3-Agent，这里可以修改一个地方的设置
# Distributed Virtual Router (DVR) configuration
# Can be:
# - ``legacy`` - No DVR functionality
# - ``dvr_snat`` - Controller or single node DVR
# - ``dvr`` - Compute node in multi-node DVR
#
Q_DVR_MODE=${Q_DVR_MODE:-legacy}
if [[ "$Q_DVR_MODE" != "legacy" ]]; then
Q_ML2_PLUGIN_MECHANISM_DRIVERS=openvswitch,linuxbridge,l2population
fi
修改的文件是devstack/lib/neutron-legacy，我这里是单节点devstack，因此改成Q_DVR_MODE=${Q_DVR_MODE:-dvr_snat}即可
修改完之后，重新执行一次stack.sh，完成安装检查一下namespace，可以发现多了个snat，但是qrouter没看明白，dvr难道不应该是创建了虚拟机之后才创建qrouter namespace么？
[lihui@openstack ~]$ ip netns list snat-49166726-b647-4793-a440-f55c3badecf9 qrouter-49166726-b647-4793-a440-f55c3badecf9 qdhcp-c237b6bc-d956-4a8c-9c29-23cea957e87b
qrouter的namespace里就一个qr，默认安装好之后创建的一个private network的网关
[lihui@openstack ~]$ sudo ip netns exec qrouter-49166726-b647-4793-a440-f55c3badecf9 ip a
1: lo: mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
16: qr-50a2fc04-62: mtu 1450 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:80:a0:5b brd ff:ff:ff:ff:ff:ff
inet 10.0.0.1/26 brd 10.0.0.63 scope global qr-50a2fc04-62
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe80:a05b/64 scope link
valid_lft forever preferred_lft forever
看下这个snat，一个sq，一个qg
[lihui@openstack ~]$ sudo ip netns exec snat-49166726-b647-4793-a440-f55c3badecf9 ip a
1: lo: mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
17: sg-cfadb20f-1f: mtu 1450 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:2e:5a:ab brd ff:ff:ff:ff:ff:ff
inet 10.0.0.10/26 brd 10.0.0.63 scope global sg-cfadb20f-1f
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe2e:5aab/64 scope link
valid_lft forever preferred_lft forever
18: qg-e1881820-06: mtu 1500 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:dd:73:13 brd ff:ff:ff:ff:ff:ff
inet 7.7.7.10/24 brd 7.7.7.255 scope global qg-e1881820-06
valid_lft forever preferred_lft forever
inet6 2001:db8::1/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fedd:7313/64 scope link
valid_lft forever preferred_lft forever
创建虚拟机，namespace没啥变化，创建一个fip
[lihui@openstack ~]$ neutron floatingip-create --subnet ac5049ba-406a-4813-81a6-7935f917e562 a6e80ed0-4523-41ec-9c57-2010e3aa7a09 neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. Created a new floatingip: +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | created_at | 2017-08-27T14:33:15Z | | description | | | fixed_ip_address | | | floating_ip_address | 7.7.7.11 | | floating_network_id | a6e80ed0-4523-41ec-9c57-2010e3aa7a09 | | id | 89695c31-9672-4e59-a708-6fe17ddb3a3c | | port_id | | | project_id | da83407d8fbd488b8ce7eccb70eedc97 | | revision_number | 1 | | router_id | | | status | DOWN | | tenant_id | da83407d8fbd488b8ce7eccb70eedc97 | | updated_at | 2017-08-27T14:33:15Z | +---------------------+--------------------------------------+
绑上private port
[lihui@openstack ~]$ neutron floatingip-associate 89695c31-9672-4e59-a708-6fe17ddb3a3c b78a49b8-85bd-49f6-9387-d31c11f31a58 neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. Associated floating IP 89695c31-9672-4e59-a708-6fe17ddb3a3c 您在 /var/spool/mail/root 中有邮件 [lihui@openstack ~]$ [lihui@openstack ~]$ neutron floatingip-list | grep 7.7.7.11 neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. | 89695c31-9672-4e59-a708-6fe17ddb3a3c | da83407d8fbd488b8ce7eccb70eedc97 | 10.0.0.6 | 7.7.7.11 | b78a49b8-85bd-49f6-9387-d31c11f31a58 |
这个时候，就有变化了，多冒出来了一个fip的namespace
[lihui@openstack ~]$ ip netns list fip-a6e80ed0-4523-41ec-9c57-2010e3aa7a09 snat-49166726-b647-4793-a440-f55c3badecf9 qrouter-49166726-b647-4793-a440-f55c3badecf9 qdhcp-c237b6bc-d956-4a8c-9c29-23cea957e87b
看看fip的名空间有啥
[lihui@openstack ~]$ sudo ip netns exec fip-a6e80ed0-4523-41ec-9c57-2010e3aa7a09 ip a
1: lo: mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: fpr-49166726-b: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 62:5a:10:56:9a:e1 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 169.254.109.47/31 scope global fpr-49166726-b
valid_lft forever preferred_lft forever
inet6 fe80::605a:10ff:fe56:9ae1/64 scope link
valid_lft forever preferred_lft forever
25: fg-8ec3a5c0-ff: mtu 1500 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:94:a9:5b brd ff:ff:ff:ff:ff:ff
inet 7.7.7.13/24 brd 7.7.7.255 scope global fg-8ec3a5c0-ff
valid_lft forever preferred_lft forever
inet6 2001:db8::8/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe94:a95b/64 scope link
valid_lft forever preferred_lft forever
一个fg，看样子可以类比qg口，应该是最终出去的口，fpr暂时一脸懵逼，169.254，肯定做中转的
看看qrouter名空间
[lihui@openstack ~]$ sudo ip netns exec qrouter-49166726-b647-4793-a440-f55c3badecf9 ip a
1: lo: mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: rfp-49166726-b: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether ae:2d:c7:37:3c:f0 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 169.254.109.46/31 scope global rfp-49166726-b
valid_lft forever preferred_lft forever
inet6 fe80::ac2d:c7ff:fe37:3cf0/64 scope link
valid_lft forever preferred_lft forever
16: qr-50a2fc04-62: mtu 1450 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:80:a0:5b brd ff:ff:ff:ff:ff:ff
inet 10.0.0.1/26 brd 10.0.0.63 scope global qr-50a2fc04-62
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe80:a05b/64 scope link
valid_lft forever preferred_lft forever
多出来了一个rfp口，也是169.254，而且和上面fpr是同一段子网里，名字也这么相似，其实它们是一对veth pair
ping一下fip，不通
[lihui@openstack ~]$ ping 7.7.7.11 PING 7.7.7.11 (7.7.7.11) 56(84) bytes of data. ^C --- 7.7.7.11 ping statistics --- 8 packets transmitted, 0 received, 100% packet loss, time 6999ms
放通安全组
[lihui@openstack ~]$ neutron security-group-rule-create --remote-ip-prefix 0.0.0.0/0 --direction ingress 2d6cce81-3d62-418a-87ae-f8f3d4515ebb
通了
[lihui@openstack ~]$ ping 7.7.7.11 PING 7.7.7.11 (7.7.7.11) 56(84) bytes of data. 64 bytes from 7.7.7.11: icmp_seq=1 ttl=62 time=6.21 ms 64 bytes from 7.7.7.11: icmp_seq=2 ttl=62 time=1.12 ms ^C --- 7.7.7.11 ping statistics ---
登陆虚拟机
[lihui@openstack ~]$ ssh cirros@7.7.7.11
The authenticity of host '7.7.7.11 (7.7.7.11)' can't be established.
RSA key fingerprint is 19:8e:a4:27:e2:54:56:2b:00:c3:68:e0:b7:4a:3b:18.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '7.7.7.11' (RSA) to the list of known hosts.
cirros@7.7.7.11's password:
$ ip a
1: lo: mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: mtu 1450 qdisc pfifo_fast qlen 1000
link/ether fa:16:3e:da:ee:eb brd ff:ff:ff:ff:ff:ff
inet 10.0.0.6/26 brd 10.0.0.63 scope global eth0
inet6 fe80::f816:3eff:feda:eeeb/64 scope link
valid_lft forever preferred_lft forever
$
下面就直接虚拟机里连通外网，查看一下流量走向，正常应该是从br-ex出去到物理网卡，但由于我这里整个openstack搭建在一个虚拟机里，有一个floating ip，直接通过虚拟机里ping这个即可
首先是qrouter里，虚拟机过来的包
[lihui@openstack ~]$ sudo ip netns exec qrouter-49166726-b647-4793-a440-f55c3badecf9 tcpdump -i qr-50a2fc04-62 icmp and src host 10.0.0.6 -en tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on qr-50a2fc04-62, link-type EN10MB (Ethernet), capture size 65535 bytes 00:30:21.974767 fa:16:3e:da:ee:eb > fa:16:3e:80:a0:5b, ethertype IPv4 (0x0800), length 842: 10.0.0.6 > 59.111.127.107: ICMP echo request, id 33537, seq 1580, length 808 00:30:22.975393 fa:16:3e:da:ee:eb > fa:16:3e:80:a0:5b, ethertype IPv4 (0x0800), length 842: 10.0.0.6 > 59.111.127.107: ICMP echo request, id 33537, seq 1581, length 808
这里的MAC地址是：vm_mac => qr_mac
只有回来的DNAT
[lihui@openstack ~]$ sudo ip netns exec qrouter-49166726-b647-4793-a440-f55c3badecf9 iptables -t nat -S | grep DNAT -A neutron-l3-agent-POSTROUTING ! -i rfp-49166726-b ! -o rfp-49166726-b -m conntrack ! --ctstate DNAT -j ACCEPT -A neutron-l3-agent-PREROUTING -d 7.7.7.11/32 -i rfp-49166726-b -j DNAT --to-destination 10.0.0.6
策略路由
[lihui@openstack ~]$ sudo ip netns exec qrouter-49166726-b647-4793-a440-f55c3badecf9 ip rule 0: from all lookup local 32766: from all lookup main 32767: from all lookup default 57480: from 10.0.0.6 lookup 16 167772161: from 10.0.0.1/26 lookup 167772161
查看表16
[lihui@openstack ~]$ sudo ip netns exec qrouter-49166726-b647-4793-a440-f55c3badecf9 ip r list table 16 default via 169.254.109.47 dev rfp-49166726-b
下一跳，从rfp口走到了fip名空间里的
2: fpr-49166726-b: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 62:5a:10:56:9a:e1 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 169.254.109.47/31 scope global fpr-49166726-b
valid_lft forever preferred_lft forever
inet6 fe80::605a:10ff:fe56:9ae1/64 scope link
valid_lft forever preferred_lft forever
看下nat表
[lihui@openstack ~]$ sudo ip netns exec qrouter-49166726-b647-4793-a440-f55c3badecf9 iptables -t nat -S -P PREROUTING ACCEPT -P INPUT ACCEPT -P OUTPUT ACCEPT -P POSTROUTING ACCEPT -N neutron-l3-agent-OUTPUT -N neutron-l3-agent-POSTROUTING -N neutron-l3-agent-PREROUTING -N neutron-l3-agent-float-snat -N neutron-l3-agent-snat -N neutron-postrouting-bottom -A PREROUTING -j neutron-l3-agent-PREROUTING -A OUTPUT -j neutron-l3-agent-OUTPUT -A POSTROUTING -j neutron-l3-agent-POSTROUTING -A POSTROUTING -j neutron-postrouting-bottom -A neutron-l3-agent-POSTROUTING ! -i rfp-49166726-b ! -o rfp-49166726-b -m conntrack ! --ctstate DNAT -j ACCEPT -A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697 -A neutron-l3-agent-PREROUTING -d 7.7.7.11/32 -i rfp-49166726-b -j DNAT --to-destination 10.0.0.6 -A neutron-l3-agent-float-snat -s 10.0.0.6/32 -j SNAT --to-source 7.7.7.11 -A neutron-l3-agent-snat -j neutron-l3-agent-float-snat -A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat
倒数第三条，做了一次SNAT，源地址转换，可以抓包确认一下
[lihui@openstack ~]$ sudo ip netns exec qrouter-49166726-b647-4793-a440-f55c3badecf9 tcpdump -i rfp-49166726-b icmp and dst host 59.111.127.107 -en tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on rfp-49166726-b, link-type EN10MB (Ethernet), capture size 65535 bytes 00:41:30.024913 ae:2d:c7:37:3c:f0 > 62:5a:10:56:9a:e1, ethertype IPv4 (0x0800), length 842: 7.7.7.11 > 59.111.127.107: ICMP echo request, id 35073, seq 432, length 808 00:41:31.025607 ae:2d:c7:37:3c:f0 > 62:5a:10:56:9a:e1, ethertype IPv4 (0x0800), length 842: 7.7.7.11 > 59.111.127.107: ICMP echo request, id 35073, seq 433, length 808 00:41:32.026232 ae:2d:c7:37:3c:f0 > 62:5a:10:56:9a:e1, ethertype IPv4 (0x0800), length 842: 7.7.7.11 > 59.111.127.107: ICMP echo request, id 35073, seq 434, length 808
这里的MAC地址：rfp_mac => fpr_mac
