DEVSTACK里DVR和FIP

devstack默认还是集中式Router,虽然neutron能够创建分布式Router,但是没法绑到L3-Agent,这里可以修改一个地方的设置

# Distributed Virtual Router (DVR) configuration
# Can be:
# - ``legacy``   - No DVR functionality
# - ``dvr_snat`` - Controller or single node DVR
# - ``dvr``      - Compute node in multi-node DVR
#
Q_DVR_MODE=${Q_DVR_MODE:-legacy}
if [[ "$Q_DVR_MODE" != "legacy" ]]; then
    Q_ML2_PLUGIN_MECHANISM_DRIVERS=openvswitch,linuxbridge,l2population
fi

修改的文件是devstack/lib/neutron-legacy,我这里是单节点devstack,因此改成Q_DVR_MODE=${Q_DVR_MODE:-dvr_snat}即可

修改完之后,重新执行一次stack.sh,完成安装检查一下namespace,可以发现多了个snat,但是qrouter没看明白,dvr难道不应该是创建了虚拟机之后才创建qrouter namespace么?

[lihui@openstack ~]$ ip netns list
snat-49166726-b647-4793-a440-f55c3badecf9
qrouter-49166726-b647-4793-a440-f55c3badecf9
qdhcp-c237b6bc-d956-4a8c-9c29-23cea957e87b

qrouter的namespace里就一个qr,默认安装好之后创建的一个private network的网关

[lihui@openstack ~]$ sudo ip netns exec qrouter-49166726-b647-4793-a440-f55c3badecf9 ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
16: qr-50a2fc04-62:  mtu 1450 qdisc noqueue state UNKNOWN
    link/ether fa:16:3e:80:a0:5b brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/26 brd 10.0.0.63 scope global qr-50a2fc04-62
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe80:a05b/64 scope link
       valid_lft forever preferred_lft forever

看下这个snat,一个sq,一个qg

[lihui@openstack ~]$ sudo ip netns exec snat-49166726-b647-4793-a440-f55c3badecf9 ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
17: sg-cfadb20f-1f:  mtu 1450 qdisc noqueue state UNKNOWN
    link/ether fa:16:3e:2e:5a:ab brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.10/26 brd 10.0.0.63 scope global sg-cfadb20f-1f
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe2e:5aab/64 scope link
       valid_lft forever preferred_lft forever
18: qg-e1881820-06:  mtu 1500 qdisc noqueue state UNKNOWN
    link/ether fa:16:3e:dd:73:13 brd ff:ff:ff:ff:ff:ff
    inet 7.7.7.10/24 brd 7.7.7.255 scope global qg-e1881820-06
       valid_lft forever preferred_lft forever
    inet6 2001:db8::1/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fedd:7313/64 scope link
       valid_lft forever preferred_lft forever

创建虚拟机,namespace没啥变化,创建一个fip

[lihui@openstack ~]$ neutron floatingip-create --subnet ac5049ba-406a-4813-81a6-7935f917e562 a6e80ed0-4523-41ec-9c57-2010e3aa7a09
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Created a new floatingip:
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| created_at          | 2017-08-27T14:33:15Z                 |
| description         |                                      |
| fixed_ip_address    |                                      |
| floating_ip_address | 7.7.7.11                             |
| floating_network_id | a6e80ed0-4523-41ec-9c57-2010e3aa7a09 |
| id                  | 89695c31-9672-4e59-a708-6fe17ddb3a3c |
| port_id             |                                      |
| project_id          | da83407d8fbd488b8ce7eccb70eedc97     |
| revision_number     | 1                                    |
| router_id           |                                      |
| status              | DOWN                                 |
| tenant_id           | da83407d8fbd488b8ce7eccb70eedc97     |
| updated_at          | 2017-08-27T14:33:15Z                 |
+---------------------+--------------------------------------+

绑上private port

[lihui@openstack ~]$ neutron floatingip-associate 89695c31-9672-4e59-a708-6fe17ddb3a3c b78a49b8-85bd-49f6-9387-d31c11f31a58
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Associated floating IP 89695c31-9672-4e59-a708-6fe17ddb3a3c
您在 /var/spool/mail/root 中有邮件
[lihui@openstack ~]$
[lihui@openstack ~]$ neutron floatingip-list | grep 7.7.7.11
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
| 89695c31-9672-4e59-a708-6fe17ddb3a3c | da83407d8fbd488b8ce7eccb70eedc97 | 10.0.0.6         | 7.7.7.11            | b78a49b8-85bd-49f6-9387-d31c11f31a58 |

这个时候,就有变化了,多冒出来了一个fip的namespace

[lihui@openstack ~]$ ip netns list
fip-a6e80ed0-4523-41ec-9c57-2010e3aa7a09
snat-49166726-b647-4793-a440-f55c3badecf9
qrouter-49166726-b647-4793-a440-f55c3badecf9
qdhcp-c237b6bc-d956-4a8c-9c29-23cea957e87b

看看fip的名空间有啥

[lihui@openstack ~]$ sudo ip netns exec fip-a6e80ed0-4523-41ec-9c57-2010e3aa7a09 ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: fpr-49166726-b:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 62:5a:10:56:9a:e1 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.109.47/31 scope global fpr-49166726-b
       valid_lft forever preferred_lft forever
    inet6 fe80::605a:10ff:fe56:9ae1/64 scope link
       valid_lft forever preferred_lft forever
25: fg-8ec3a5c0-ff:  mtu 1500 qdisc noqueue state UNKNOWN
    link/ether fa:16:3e:94:a9:5b brd ff:ff:ff:ff:ff:ff
    inet 7.7.7.13/24 brd 7.7.7.255 scope global fg-8ec3a5c0-ff
       valid_lft forever preferred_lft forever
    inet6 2001:db8::8/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe94:a95b/64 scope link
       valid_lft forever preferred_lft forever

一个fg,看样子可以类比qg口,应该是最终出去的口,fpr暂时一脸懵逼,169.254,肯定做中转的

看看qrouter名空间

[lihui@openstack ~]$ sudo ip netns exec qrouter-49166726-b647-4793-a440-f55c3badecf9 ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: rfp-49166726-b:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether ae:2d:c7:37:3c:f0 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.109.46/31 scope global rfp-49166726-b
       valid_lft forever preferred_lft forever
    inet6 fe80::ac2d:c7ff:fe37:3cf0/64 scope link
       valid_lft forever preferred_lft forever
16: qr-50a2fc04-62:  mtu 1450 qdisc noqueue state UNKNOWN
    link/ether fa:16:3e:80:a0:5b brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/26 brd 10.0.0.63 scope global qr-50a2fc04-62
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe80:a05b/64 scope link
       valid_lft forever preferred_lft forever

多出来了一个rfp口,也是169.254,而且和上面fpr是同一段子网里,名字也这么相似,其实它们是一对veth pair

ping一下fip,不通

[lihui@openstack ~]$ ping 7.7.7.11
PING 7.7.7.11 (7.7.7.11) 56(84) bytes of data.
^C
--- 7.7.7.11 ping statistics ---
8 packets transmitted, 0 received, 100% packet loss, time 6999ms

放通安全组

[lihui@openstack ~]$ neutron security-group-rule-create --remote-ip-prefix 0.0.0.0/0 --direction ingress  2d6cce81-3d62-418a-87ae-f8f3d4515ebb

通了

[lihui@openstack ~]$ ping 7.7.7.11
PING 7.7.7.11 (7.7.7.11) 56(84) bytes of data.
64 bytes from 7.7.7.11: icmp_seq=1 ttl=62 time=6.21 ms
64 bytes from 7.7.7.11: icmp_seq=2 ttl=62 time=1.12 ms
^C
--- 7.7.7.11 ping statistics ---

登陆虚拟机

[lihui@openstack ~]$ ssh cirros@7.7.7.11
The authenticity of host '7.7.7.11 (7.7.7.11)' can't be established.
RSA key fingerprint is 19:8e:a4:27:e2:54:56:2b:00:c3:68:e0:b7:4a:3b:18.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '7.7.7.11' (RSA) to the list of known hosts.
cirros@7.7.7.11's password:
$ ip a
1: lo:  mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0:  mtu 1450 qdisc pfifo_fast qlen 1000
    link/ether fa:16:3e:da:ee:eb brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.6/26 brd 10.0.0.63 scope global eth0
    inet6 fe80::f816:3eff:feda:eeeb/64 scope link
       valid_lft forever preferred_lft forever
$

下面就直接虚拟机里连通外网,查看一下流量走向,正常应该是从br-ex出去到物理网卡,但由于我这里整个openstack搭建在一个虚拟机里,有一个floating ip,直接通过虚拟机里ping这个即可

首先是qrouter里,虚拟机过来的包

[lihui@openstack ~]$ sudo ip netns exec qrouter-49166726-b647-4793-a440-f55c3badecf9 tcpdump -i qr-50a2fc04-62 icmp and src host 10.0.0.6 -en
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on qr-50a2fc04-62, link-type EN10MB (Ethernet), capture size 65535 bytes
00:30:21.974767 fa:16:3e:da:ee:eb > fa:16:3e:80:a0:5b, ethertype IPv4 (0x0800), length 842: 10.0.0.6 > 59.111.127.107: ICMP echo request, id 33537, seq 1580, length 808
00:30:22.975393 fa:16:3e:da:ee:eb > fa:16:3e:80:a0:5b, ethertype IPv4 (0x0800), length 842: 10.0.0.6 > 59.111.127.107: ICMP echo request, id 33537, seq 1581, length 808

这里的MAC地址是:vm_mac => qr_mac

只有回来的DNAT

[lihui@openstack ~]$ sudo ip netns exec qrouter-49166726-b647-4793-a440-f55c3badecf9 iptables -t nat -S | grep DNAT
-A neutron-l3-agent-POSTROUTING ! -i rfp-49166726-b ! -o rfp-49166726-b -m conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 7.7.7.11/32 -i rfp-49166726-b -j DNAT --to-destination 10.0.0.6

策略路由

[lihui@openstack ~]$ sudo ip netns exec qrouter-49166726-b647-4793-a440-f55c3badecf9 ip rule
0:	from all lookup local
32766:	from all lookup main
32767:	from all lookup default
57480:	from 10.0.0.6 lookup 16
167772161:	from 10.0.0.1/26 lookup 167772161

查看表16

[lihui@openstack ~]$ sudo ip netns exec qrouter-49166726-b647-4793-a440-f55c3badecf9 ip r list table 16
default via 169.254.109.47 dev rfp-49166726-b

下一跳,从rfp口走到了fip名空间里的

2: fpr-49166726-b:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 62:5a:10:56:9a:e1 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.109.47/31 scope global fpr-49166726-b
       valid_lft forever preferred_lft forever
    inet6 fe80::605a:10ff:fe56:9ae1/64 scope link
       valid_lft forever preferred_lft forever

看下nat表

[lihui@openstack ~]$ sudo ip netns exec qrouter-49166726-b647-4793-a440-f55c3badecf9 iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N neutron-l3-agent-OUTPUT
-N neutron-l3-agent-POSTROUTING
-N neutron-l3-agent-PREROUTING
-N neutron-l3-agent-float-snat
-N neutron-l3-agent-snat
-N neutron-postrouting-bottom
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-POSTROUTING ! -i rfp-49166726-b ! -o rfp-49166726-b -m conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-PREROUTING -d 7.7.7.11/32 -i rfp-49166726-b -j DNAT --to-destination 10.0.0.6
-A neutron-l3-agent-float-snat -s 10.0.0.6/32 -j SNAT --to-source 7.7.7.11
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat

倒数第三条,做了一次SNAT,源地址转换,可以抓包确认一下

[lihui@openstack ~]$ sudo ip netns exec qrouter-49166726-b647-4793-a440-f55c3badecf9 tcpdump -i rfp-49166726-b icmp and dst host 59.111.127.107 -en
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on rfp-49166726-b, link-type EN10MB (Ethernet), capture size 65535 bytes
00:41:30.024913 ae:2d:c7:37:3c:f0 > 62:5a:10:56:9a:e1, ethertype IPv4 (0x0800), length 842: 7.7.7.11 > 59.111.127.107: ICMP echo request, id 35073, seq 432, length 808
00:41:31.025607 ae:2d:c7:37:3c:f0 > 62:5a:10:56:9a:e1, ethertype IPv4 (0x0800), length 842: 7.7.7.11 > 59.111.127.107: ICMP echo request, id 35073, seq 433, length 808
00:41:32.026232 ae:2d:c7:37:3c:f0 > 62:5a:10:56:9a:e1, ethertype IPv4 (0x0800), length 842: 7.7.7.11 > 59.111.127.107: ICMP echo request, id 35073, seq 434, length 808

这里的MAC地址:rfp_mac => fpr_mac

发表评论