先说一下测试思路，环境还是ALL-IN-ONE单节点openstack，两个不同的VPC，各自创建VPN服务，只需要各自VPC对应的VPN-Service里的出口网关IP地址能通即可；可vpn-service里出口IP默认是Router Gateway IP，而只需要将两个VPC的external网络都设置为同一网段，这样就能达到目标，详细步骤就是：
1：创建vpc-1和vpc-2
2：vpc-1和vpc-2都设置同一段external网络作为router-gateway
3：vpc-1和vpc-2各自创建vpn-service，对应出口网关在同一网段
4：各自创建ikepolicy，ipsecpolicy，ipsec-vpn
5：两个vpc分别创建vm，测试两者连通性
大概如下
测试步骤如下：
1：VPC1以及对应的ipsec vpn
neutron net-create vpn-network-1 neutron subnet-create --name vpn-subnet-1 vpn-network-1 2.3.4.0/24 neutron router-create vpn-router-1 neutron router-interface-add vpn-router-1 vpn-subnet-1 neutron router-gateway-set vpn-router-1 public neutron vpn-ikepolicy-create vpn-ikepolicy-1 neutron vpn-ipsecpolicy-create vpn-ipsecpolicy-1 neutron vpn-service-create vpn-router-1 vpn-subnet-1 neutron ipsec-site-connection-create --vpnservice-id 2a728147-436b-4fbf-a065-92e2d36bd9b2 --ikepolicy-id f3ff940c-cf1b-4777-9da6-f9641e653b41 --ipsecpolicy-id 8f4e0157-cd48-4342-8a60-89d6ef122668 --peer-address 172.24.4.4 --peer-id 172.24.4.4 --peer-cidr 4.3.2.0/24 --psk lihui_key
2：VPC2以及对应的ipsec vpn
neutron net-create vpn-network-2 neutron subnet-create --name vpn-subnet-2 vpn-network-2 4.3.2.0/24 neutron router-create vpn-router-2 neutron router-interface-add vpn-router-2 vpn-subnet-2 neutron router-gateway-set vpn-router-2 public neutron vpn-ikepolicy-create vpn-ikepolicy-2 neutron vpn-ipsecpolicy-create vpn-ipsecpolicy-2 neutron vpn-service-create vpn-router-2 vpn-subnet-2 neutron ipsec-site-connection-create --vpnservice-id 5ed48e6e-ec99-4149-808c-0a42e45c5dc2 --ikepolicy-id 7ffc882a-9dc4-4ec2-8cc0-ef3330ad3371 --ipsecpolicy-id b4228923-c4b2-47a9-ae6b-2b7b2ffe6aa8 --peer-address 172.24.4.3 --peer-id 172.24.4.3 --peer-cidr 2.3.4.0/24 --psk lihui_key
3：分别创建虚拟机
lihui@l-openstack:~$ nova list +--------------------------------------+------+--------+------------+-------------+-----------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------+--------+------------+-------------+-----------------------+ | cd84c43a-b05c-44a3-9c12-6174d0931a69 | vm-1 | ACTIVE | - | Running | vpn-network-1=2.3.4.4 | | 2f27000a-c732-4976-a9fe-829394b28302 | vm-2 | ACTIVE | - | Running | vpn-network-2=4.3.2.3 | +--------------------------------------+------+--------+------------+-------------+-----------------------+
4：开放ingress安全组
lihui@l-openstack:~$ neutron security-group-rule-create --direction ingress --remote-ip-prefix 0.0.0.0/0 bd9ececf-1849-4182-8579-dad13e6fa520 Created a new security_group_rule: +-------------------+--------------------------------------+ | Field | Value | +-------------------+--------------------------------------+ | direction | ingress | | ethertype | IPv4 | | id | 9b9b6ede-393c-42ae-84e9-2605d009d8a3 | | port_range_max | | | port_range_min | | | protocol | | | remote_group_id | | | remote_ip_prefix | 0.0.0.0/0 | | security_group_id | bd9ececf-1849-4182-8579-dad13e6fa520 | | tenant_id | ba744b4c95da4c5b8bedf4b6c08dccb3 | +-------------------+--------------------------------------+
5：分别从qrouter namespace里ssh登陆虚拟机
lihui@l-openstack:~$ sudo ip netns exec qrouter-1315b29a-9b47-47a9-847f-3636d0ebc89a ssh cirros@2.3.4.4
The authenticity of host '2.3.4.4 (2.3.4.4)' can't be established.
RSA key fingerprint is 0b:f3:ae:b0:62:d5:f3:e8:40:e6:f2:e5:68:db:71:94.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '2.3.4.4' (RSA) to the list of known hosts.
cirros@2.3.4.4's password:
$ ip a
1: lo: mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: mtu 1450 qdisc pfifo_fast qlen 1000
link/ether fa:16:3e:86:fc:82 brd ff:ff:ff:ff:ff:ff
inet 2.3.4.4/24 brd 2.3.4.255 scope global eth0
inet6 fe80::f816:3eff:fe86:fc82/64 scope link
valid_lft forever preferred_lft forever
6：vpc类子网连通性验证，vm1->vm2；一开始出现了首包ARP失败的问题，相互ping一下就ok了
$ ping 4.3.2.3 PING 4.3.2.3 (4.3.2.3): 56 data bytes 64 bytes from 4.3.2.3: seq=0 ttl=62 time=3.838 ms 64 bytes from 4.3.2.3: seq=1 ttl=62 time=1.678 ms 64 bytes from 4.3.2.3: seq=2 ttl=62 time=1.446 ms 64 bytes from 4.3.2.3: seq=3 ttl=62 time=1.574 ms 64 bytes from 4.3.2.3: seq=4 ttl=62 time=1.478 ms 64 bytes from 4.3.2.3: seq=5 ttl=62 time=1.349 ms 64 bytes from 4.3.2.3: seq=6 ttl=62 time=1.533 ms
7：如何确认ipsec流量呢，直接qr和qg口抓包
qr1口，上行还未封装，下行已经解封
lihui@l-openstack:~$ sudo ip netns exec qrouter-1315b29a-9b47-47a9-847f-3636d0ebc89a tshark -i qr-4a84b9a6-ca -f "icmp" tshark: Lua: Error during loading: [string "/usr/share/wireshark/init.lua"]:46: dofile has been disabled due to running Wireshark as superuser. See http://wiki.wireshark.org/CaptureSetup/CapturePrivileges for help in running Wireshark as an unprivileged user. Running as user "root" and group "root". This could be dangerous. Capturing on 'qr-4a84b9a6-ca' 1 0.000000 2.3.4.4 -> 4.3.2.3 ICMP 98 Echo (ping) request id=0x7f01, seq=104/26624, ttl=64 2 0.000664 4.3.2.3 -> 2.3.4.4 ICMP 98 Echo (ping) reply id=0x7f01, seq=104/26624, ttl=62 (request in 1) 3 1.001958 2.3.4.4 -> 4.3.2.3 ICMP 98 Echo (ping) request id=0x7f01, seq=105/26880, ttl=64 4 1.002515 4.3.2.3 -> 2.3.4.4 ICMP 98 Echo (ping) reply id=0x7f01, seq=105/26880, ttl=62 (request in 3) 5 2.004055 2.3.4.4 -> 4.3.2.3 ICMP 98 Echo (ping) request id=0x7f01, seq=106/27136, ttl=64 6 2.004716 4.3.2.3 -> 2.3.4.4 ICMP 98 Echo (ping) reply id=0x7f01, seq=106/27136, ttl=62 (request in 5) 7 3.006048 2.3.4.4 -> 4.3.2.3 ICMP 98 Echo (ping) request id=0x7f01, seq=107/27392, ttl=64 8 3.006639 4.3.2.3 -> 2.3.4.4 ICMP 98 Echo (ping) reply id=0x7f01, seq=107/27392, ttl=62 (request in 7) 9 4.008060 2.3.4.4 -> 4.3.2.3 ICMP 98 Echo (ping) request id=0x7f01, seq=108/27648, ttl=64 10 4.008706 4.3.2.3 -> 2.3.4.4 ICMP 98 Echo (ping) reply id=0x7f01, seq=108/27648, ttl=62 (request in 9)
qg1口，ESP协议，加密相关，注意看第二个ESP和ICMP的时间戳，是一模一样的，不太清楚是否就和vxlan报文添加了vxlan header和udp头一个意思，是否ICMP就是ESP加密的payload，我的理解就是上行已封装成对外出口IP，下行两个包是一个解封装过程
lihui@l-openstack:~$ sudo ip netns exec qrouter-1315b29a-9b47-47a9-847f-3636d0ebc89a tshark -i qg-6deb7bb7-c8 tshark: Lua: Error during loading: [string "/usr/share/wireshark/init.lua"]:46: dofile has been disabled due to running Wireshark as superuser. See http://wiki.wireshark.org/CaptureSetup/CapturePrivileges for help in running Wireshark as an unprivileged user. Running as user "root" and group "root". This could be dangerous. Capturing on 'qg-6deb7bb7-c8' 1 0.000000 172.24.4.3 -> 172.24.4.4 ESP 166 ESP (SPI=0x4ef282c7) 2 0.000503 172.24.4.4 -> 172.24.4.3 ESP 166 ESP (SPI=0xc0029bdd) 3 0.000503 4.3.2.3 -> 2.3.4.4 ICMP 98 Echo (ping) reply id=0x7f01, seq=141/36096, ttl=63 4 1.002210 172.24.4.3 -> 172.24.4.4 ESP 166 ESP (SPI=0x4ef282c7) 5 1.002700 172.24.4.4 -> 172.24.4.3 ESP 166 ESP (SPI=0xc0029bdd) 6 1.002700 4.3.2.3 -> 2.3.4.4 ICMP 98 Echo (ping) reply id=0x7f01, seq=142/36352, ttl=63 7 2.004328 172.24.4.3 -> 172.24.4.4 ESP 166 ESP (SPI=0x4ef282c7) 8 2.004813 172.24.4.4 -> 172.24.4.3 ESP 166 ESP (SPI=0xc0029bdd) 9 2.004813 4.3.2.3 -> 2.3.4.4 ICMP 98 Echo (ping) reply id=0x7f01, seq=143/36608, ttl=63 10 3.006449 172.24.4.3 -> 172.24.4.4 ESP 166 ESP (SPI=0x4ef282c7) 11 3.006978 172.24.4.4 -> 172.24.4.3 ESP 166 ESP (SPI=0xc0029bdd) 12 3.006978 4.3.2.3 -> 2.3.4.4 ICMP 98 Echo (ping) reply id=0x7f01, seq=144/36864, ttl=63
qg2口，解封装，以及上行封装
lihui@l-openstack:~$ sudo ip netns exec qrouter-46aeb937-98b1-4b8c-a119-f8d29ba108e8 tshark -i qg-9f5f9982-99 tshark: Lua: Error during loading: [string "/usr/share/wireshark/init.lua"]:46: dofile has been disabled due to running Wireshark as superuser. See http://wiki.wireshark.org/CaptureSetup/CapturePrivileges for help in running Wireshark as an unprivileged user. Running as user "root" and group "root". This could be dangerous. Capturing on 'qg-9f5f9982-99' 1 0.000000 172.24.4.3 -> 172.24.4.4 ESP 166 ESP (SPI=0x4ef282c7) 2 0.000000 2.3.4.4 -> 4.3.2.3 ICMP 98 Echo (ping) request id=0x7f01, seq=880/28675, ttl=63 3 0.000539 172.24.4.4 -> 172.24.4.3 ESP 166 ESP (SPI=0xc0029bdd) 4 1.001345 172.24.4.3 -> 172.24.4.4 ESP 166 ESP (SPI=0x4ef282c7) 5 1.001345 2.3.4.4 -> 4.3.2.3 ICMP 98 Echo (ping) request id=0x7f01, seq=881/28931, ttl=63 6 1.001930 172.24.4.4 -> 172.24.4.3 ESP 166 ESP (SPI=0xc0029bdd) 7 2.003556 172.24.4.3 -> 172.24.4.4 ESP 166 ESP (SPI=0x4ef282c7) 8 2.003556 2.3.4.4 -> 4.3.2.3 ICMP 98 Echo (ping) request id=0x7f01, seq=882/29187, ttl=63 9 2.004166 172.24.4.4 -> 172.24.4.3 ESP 166 ESP (SPI=0xc0029bdd) 10 3.005913 172.24.4.3 -> 172.24.4.4 ESP 166 ESP (SPI=0x4ef282c7) 11 3.005913 2.3.4.4 -> 4.3.2.3 ICMP 98 Echo (ping) request id=0x7f01, seq=883/29443, ttl=63 12 3.006493 172.24.4.4 -> 172.24.4.3 ESP 166 ESP (SPI=0xc0029bdd)
qr2口，来和回
lihui@l-openstack:~$ sudo ip netns exec qrouter-46aeb937-98b1-4b8c-a119-f8d29ba108e8 tshark -i qr-5d202b73-21 tshark: Lua: Error during loading: [string "/usr/share/wireshark/init.lua"]:46: dofile has been disabled due to running Wireshark as superuser. See http://wiki.wireshark.org/CaptureSetup/CapturePrivileges for help in running Wireshark as an unprivileged user. Running as user "root" and group "root". This could be dangerous. Capturing on 'qr-5d202b73-21' 1 0.000000 2.3.4.4 -> 4.3.2.3 ICMP 98 Echo (ping) request id=0x7f01, seq=1034/2564, ttl=62 2 0.000374 4.3.2.3 -> 2.3.4.4 ICMP 98 Echo (ping) reply id=0x7f01, seq=1034/2564, ttl=64 (request in 1) 3 1.002398 2.3.4.4 -> 4.3.2.3 ICMP 98 Echo (ping) request id=0x7f01, seq=1035/2820, ttl=62 4 1.002764 4.3.2.3 -> 2.3.4.4 ICMP 98 Echo (ping) reply id=0x7f01, seq=1035/2820, ttl=64 (request in 3) 5 2.004286 2.3.4.4 -> 4.3.2.3 ICMP 98 Echo (ping) request id=0x7f01, seq=1036/3076, ttl=62 6 2.004656 4.3.2.3 -> 2.3.4.4 ICMP 98 Echo (ping) reply id=0x7f01, seq=1036/3076, ttl=64 (request in 5) 7 3.006374 2.3.4.4 -> 4.3.2.3 ICMP 98 Echo (ping) request id=0x7f01, seq=1037/3332, ttl=62 8 3.006715 4.3.2.3 -> 2.3.4.4 ICMP 98 Echo (ping) reply id=0x7f01, seq=1037/3332, ttl=64 (request in 7) 9 4.007966 2.3.4.4 -> 4.3.2.3 ICMP 98 Echo (ping) request id=0x7f01, seq=1038/3588, ttl=62 10 4.008583 4.3.2.3 -> 2.3.4.4 ICMP 98 Echo (ping) reply id=0x7f01, seq=1038/3588, ttl=64 (request in 9) 11 5.009055 2.3.4.4 -> 4.3.2.3 ICMP 98 Echo (ping) request id=0x7f01, seq=1039/3844, ttl=62 12 5.009407 4.3.2.3 -> 2.3.4.4 ICMP 98 Echo (ping) reply id=0x7f01, seq=1039/3844, ttl=64 (request in 11)
8：看看qrouter里，就添加了一条到对端vpc的路由，下一跳为本端vpn的出口网关ip
lihui@l-openstack:~$ sudo ip netns exec qrouter-1315b29a-9b47-47a9-847f-3636d0ebc89a ip r default via 172.24.4.1 dev qg-6deb7bb7-c8 2.3.4.0/24 dev qr-4a84b9a6-ca proto kernel scope link src 2.3.4.1 4.3.2.0/24 dev qg-6deb7bb7-c8 scope link mtu 1500 172.24.4.0/24 dev qg-6deb7bb7-c8 proto kernel scope link src 172.24.4.3 lihui@l-openstack:~$ lihui@l-openstack:~$ lihui@l-openstack:~$ sudo ip netns exec qrouter-46aeb937-98b1-4b8c-a119-f8d29ba108e8 ip r default via 172.24.4.1 dev qg-9f5f9982-99 2.3.4.0/24 dev qg-9f5f9982-99 scope link mtu 1500 4.3.2.0/24 dev qr-5d202b73-21 proto kernel scope link src 4.3.2.1 172.24.4.0/24 dev qg-9f5f9982-99 proto kernel scope link src 172.24.4.4
将vpn删除，路由规则也删除了
lihui@l-openstack:~$ sudo ip netns exec qrouter-1315b29a-9b47-47a9-847f-3636d0ebc89a ip r default via 172.24.4.1 dev qg-6deb7bb7-c8 2.3.4.0/24 dev qr-4a84b9a6-ca proto kernel scope link src 2.3.4.1 172.24.4.0/24 dev qg-6deb7bb7-c8 proto kernel scope link src 172.24.4.3 lihui@l-openstack:~$ lihui@l-openstack:~$ sudo ip netns exec qrouter-46aeb937-98b1-4b8c-a119-f8d29ba108e8 ip r default via 172.24.4.1 dev qg-9f5f9982-99 4.3.2.0/24 dev qr-5d202b73-21 proto kernel scope link src 4.3.2.1 172.24.4.0/24 dev qg-9f5f9982-99 proto kernel scope link src 172.24.4.4
看到这里也就明白了，vpn的作用，除了加密之外，还有就是将vpc与对端通信的流量路由到本端vpn的出口网关，达到能和对端vpn出口网关进行ipsec协商的目的
