社区Devstack打通网络折腾

作者: LiHui 分类: Networking 发布时间: 2017-06-30 23:50

虚拟机VMware Fusion,操作系统CentOS7.2,网络配置桥接

devstack搭建好单节点openstack之后,直接创建虚拟机

[lihui@openstack devstack]$ nova boot --flavor 1 --image 5291f14d-a20d-41c7-a809-87b0a02a2e40 --nic net-id=8f91ac0f-54e4-4bbb-834b-551301aaa0d0 --nic net-id=11d20d58-99e8-4688-b9ab-a35d7149bdd8 --key-name dev --admin-pass 000000 new-vm
+--------------------------------------+-----------------------------------------------------------------+
| Property                             | Value                                                           |
+--------------------------------------+-----------------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                          |
| OS-EXT-AZ:availability_zone          |                                                                 |
| OS-EXT-SRV-ATTR:host                 | -                                                               |
| OS-EXT-SRV-ATTR:hostname             | new-vm                                                          |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                                               |
| OS-EXT-SRV-ATTR:instance_name        |                                                                 |
| OS-EXT-SRV-ATTR:kernel_id            |                                                                 |
| OS-EXT-SRV-ATTR:launch_index         | 0                                                               |
| OS-EXT-SRV-ATTR:ramdisk_id           |                                                                 |
| OS-EXT-SRV-ATTR:reservation_id       | r-bki2qg27                                                      |
| OS-EXT-SRV-ATTR:root_device_name     | -                                                               |
| OS-EXT-SRV-ATTR:user_data            | -                                                               |
| OS-EXT-STS:power_state               | 0                                                               |
| OS-EXT-STS:task_state                | scheduling                                                      |
| OS-EXT-STS:vm_state                  | building                                                        |
| OS-SRV-USG:launched_at               | -                                                               |
| OS-SRV-USG:terminated_at             | -                                                               |
| accessIPv4                           |                                                                 |
| accessIPv6                           |                                                                 |
| adminPass                            | 000000                                                          |
| config_drive                         |                                                                 |
| created                              | 2017-07-16T15:18:00Z                                            |
| description                          | -                                                               |
| flavor                               | m1.tiny (1)                                                     |
| hostId                               |                                                                 |
| host_status                          |                                                                 |
| id                                   | 5941b42b-1b26-4cce-be1c-ec2e5632bbcd                            |
| image                                | cirros-0.3.5-x86_64-disk (5291f14d-a20d-41c7-a809-87b0a02a2e40) |
| key_name                             | dev                                                             |
| locked                               | False                                                           |
| metadata                             | {}                                                              |
| name                                 | new-vm                                                          |
| os-extended-volumes:volumes_attached | []                                                              |
| progress                             | 0                                                               |
| security_groups                      | default                                                         |
| status                               | BUILD                                                           |
| tags                                 | []                                                              |
| tenant_id                            | 32e905a2f692421da826ce67e11a3fec                                |
| updated                              | 2017-07-16T15:18:00Z                                            |
| user_id                              | 8f40ab52e7174816a502128aa0c10991                                |
+--------------------------------------+-----------------------------------------------------------------+

namespace里连通虚拟机,不通

[lihui@openstack devstack]$ sudo ip netns exec qrouter-70ad5580-33a1-4f77-b242-1c3d2b71af8c ping 10.0.0.9
PING 10.0.0.9 (10.0.0.9) 56(84) bytes of data.

 看下安全组

[lihui@openstack devstack]$ nova interface-list 5941b42b-1b26-4cce-be1c-ec2e5632bbcd
+------------+--------------------------------------+--------------------------------------+----------------------------------------------+-------------------+
| Port State | Port ID                              | Net ID                               | IP addresses                                 | MAC Addr          |
+------------+--------------------------------------+--------------------------------------+----------------------------------------------+-------------------+
| ACTIVE     | a08c3748-2069-43ae-ae86-e58264f540ff | 11d20d58-99e8-4688-b9ab-a35d7149bdd8 | 172.24.4.9,2001:db8::a                       | fa:16:3e:3c:61:a9 |
| ACTIVE     | cc545042-dd6e-40c2-8783-af7591ad440d | 8f91ac0f-54e4-4bbb-834b-551301aaa0d0 | 10.0.0.9,fd8d:4a3:aeb0:0:f816:3eff:fef1:25e3 | fa:16:3e:f1:25:e3 |
+------------+--------------------------------------+--------------------------------------+----------------------------------------------+-------------------+

查找PORT对应安全组

[lihui@openstack ~]$ neutron port-show a08c3748-2069-43ae-ae86-e58264f540ff | grep security
| port_security_enabled | True                                                                               |
| security_groups       | a35a41e6-77c8-45a8-b9ba-0c16a24bc910                                               |

添加ingress放通规则

[lihui@openstack devstack]$ neutron security-group-rule-create --direction ingress --remote-ip-prefix 0.0.0.0/0 a35a41e6-77c8-45a8-b9ba-0c16a24bc910
Created a new security_group_rule:
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| created_at        | 2017-07-16T15:21:11Z                 |
| description       |                                      |
| direction         | ingress                              |
| ethertype         | IPv4                                 |
| id                | 169ac2f6-d9ef-4c59-a003-63980ecf39f0 |
| port_range_max    |                                      |
| port_range_min    |                                      |
| project_id        | 32e905a2f692421da826ce67e11a3fec     |
| protocol          |                                      |
| remote_group_id   |                                      |
| remote_ip_prefix  | 0.0.0.0/0                            |
| revision_number   | 1                                    |
| security_group_id | a35a41e6-77c8-45a8-b9ba-0c16a24bc910 |
| tenant_id         | 32e905a2f692421da826ce67e11a3fec     |
| updated_at        | 2017-07-16T15:21:11Z                 |
+-------------------+--------------------------------------+

通了

[lihui@openstack devstack]$ sudo ip netns exec qrouter-70ad5580-33a1-4f77-b242-1c3d2b71af8c ping 10.0.0.9
PING 10.0.0.9 (10.0.0.9) 56(84) bytes of data.

▽
64 bytes from 10.0.0.9: icmp_seq=1 ttl=64 time=1.72 ms
64 bytes from 10.0.0.9: icmp_seq=2 ttl=64 time=0.507 ms
^C
--- 10.0.0.9 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms

看下到public IP

[lihui@openstack devstack]$ ping 172.24.4.9
PING 172.24.4.9 (172.24.4.9) 56(84) bytes of data.
From 172.24.4.1 icmp_seq=1 Destination Host Unreachable
From 172.24.4.1 icmp_seq=2 Destination Host Unreachable
From 172.24.4.1 icmp_seq=3 Destination Host Unreachable
From 172.24.4.1 icmp_seq=4 Destination Host Unreachable
^C
--- 172.24.4.9 ping statistics ---
5 packets transmitted, 0 received, +4 errors, 100% packet loss, time 4005ms
pipe 4

看下qg

[lihui@openstack devstack]$ sudo ip netns exec qrouter-70ad5580-33a1-4f77-b242-1c3d2b71af8c ip a show qg-9452f9cc-51
15: qg-9452f9cc-51:  mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
    link/ether fa:16:3e:56:98:50 brd ff:ff:ff:ff:ff:ff
    inet 172.24.4.2/24 brd 172.24.4.255 scope global qg-9452f9cc-51
       valid_lft forever preferred_lft forever
    inet6 2001:db8::9/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe56:9850/64 scope link
       valid_lft forever preferred_lft forever

连通性无误

[lihui@openstack devstack]$ ping 172.24.4.2
PING 172.24.4.2 (172.24.4.2) 56(84) bytes of data.
64 bytes from 172.24.4.2: icmp_seq=1 ttl=64 time=0.292 ms
64 bytes from 172.24.4.2: icmp_seq=2 ttl=64 time=0.076 ms
64 bytes from 172.24.4.2: icmp_seq=3 ttl=64 time=0.054 ms
64 bytes from 172.24.4.2: icmp_seq=4 ttl=64 time=0.055 ms

确认qg在br-int上

[lihui@openstack ~]$ sudo ovs-ofctl show br-int | grep qg-9452f9cc-51
 17(qg-9452f9cc-51): addr:00:00:00:00:b0:37

也就是vm-tap-qbr-namespace这个链路有问题

获取VNC地址登陆

[lihui@openstack ~]$ nova get-vnc-console 5941b42b-1b26-4cce-be1c-ec2e5632bbcd novnc
+-------+-------------------------------------------------------------------------------------+
| Type  | Url                                                                                 |
+-------+-------------------------------------------------------------------------------------+
| novnc | http://192.168.100.31:6080/vnc_auto.html?token=4927a9c5-62ac-4ec4-a6c6-fbf53b25e546 |
+-------+-------------------------------------------------------------------------------------+

宿主机无法打开,查看端口,未放开

[lihui@openstack ~]$ telnet 192.168.10.31 6080
Trying 192.168.10.31...
^C

放通6080端口

[lihui@openstack ~]$ sudo iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 6080 -j ACCEPT
[lihui@openstack ~]$ sudo lsof -i:6080
COMMAND     PID  USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
nova-novn 69421 lihui    3u  IPv4 135635      0t0  TCP *:6080 (LISTEN)
[lihui@openstack ~]$ ip a | grep 100.31
    inet 192.168.100.31/24 brd 192.168.100.255 scope global dynamic ens33
[lihui@openstack ~]$ sudo iptables -vnL --line-numbers | grep 6080
13       0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:6080

终于通了

[lihui@openstack ~]$ telnet 192.168.100.31 6080
Trying 192.168.100.31...
Connected to 192.168.100.31.
Escape character is '^]'.
^C
Connection closed by foreign host.

可虚拟机里是MINI版,没有desktop,而宿主机mac上依旧telnet不通,抓包,有去无回

[lihui@openstack ~]$ sudo tcpdump -i ens33 port 6080 -en
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens33, link-type EN10MB (Ethernet), capture size 65535 bytes
00:21:42.328150 a4:5e:60:d6:4f:df > 00:0c:29:8c:4e:c7, ethertype IPv4 (0x0800), length 78: 192.168.100.11.55146 > 192.168.100.31.6080: Flags [S], seq 391903154, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 715508390 ecr 0,sackOK,eol], length 0
^C
1 packet captured
3 packets received by filter
0 packets dropped by kernel

不清楚是不是vnc有问题,先放弃

VM到qg不通,查看br-int流表

[lihui@openstack ~]$ sudo ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
 cookie=0xd957818e88e64f11, duration=60.796s, table=0, n_packets=0, n_bytes=0, idle_age=60, priority=10,icmp6,in_port=19,icmp_type=136 actions=resubmit(,24)
 cookie=0xd957818e88e64f11, duration=60.738s, table=0, n_packets=0, n_bytes=0, idle_age=60, priority=10,icmp6,in_port=20,icmp_type=136 actions=resubmit(,24)
 cookie=0xd957818e88e64f11, duration=60.789s, table=0, n_packets=2, n_bytes=84, idle_age=5, priority=10,arp,in_port=19 actions=resubmit(,24)
 cookie=0xd957818e88e64f11, duration=60.733s, table=0, n_packets=0, n_bytes=0, idle_age=60, priority=10,arp,in_port=20 actions=resubmit(,24)
 cookie=0xd957818e88e64f11, duration=64.506s, table=0, n_packets=0, n_bytes=0, idle_age=64, priority=2,in_port=1 actions=drop
 cookie=0xd957818e88e64f11, duration=60.805s, table=0, n_packets=7, n_bytes=686, idle_age=5, priority=9,in_port=19 actions=resubmit(,25)
 cookie=0xd957818e88e64f11, duration=60.750s, table=0, n_packets=0, n_bytes=0, idle_age=60, priority=9,in_port=20 actions=resubmit(,25)
 cookie=0xd957818e88e64f11, duration=61.609s, table=0, n_packets=0, n_bytes=0, idle_age=61, priority=3,in_port=1,vlan_tci=0x0000/0x1fff actions=mod_vlan_vid:2,NORMAL
 cookie=0xd957818e88e64f11, duration=64.671s, table=0, n_packets=31, n_bytes=1770, idle_age=5, priority=0 actions=NORMAL
 cookie=0xd957818e88e64f11, duration=64.675s, table=23, n_packets=0, n_bytes=0, idle_age=64, priority=0 actions=drop
 cookie=0xd957818e88e64f11, duration=60.802s, table=24, n_packets=0, n_bytes=0, idle_age=60, priority=2,icmp6,in_port=19,icmp_type=136,nd_target=fe80::f816:3eff:fef1:25e3 actions=NORMAL
 cookie=0xd957818e88e64f11, duration=60.799s, table=24, n_packets=0, n_bytes=0, idle_age=60, priority=2,icmp6,in_port=19,icmp_type=136,nd_target=fd8d:4a3:aeb0:0:f816:3eff:fef1:25e3 actions=NORMAL
 cookie=0xd957818e88e64f11, duration=60.746s, table=24, n_packets=0, n_bytes=0, idle_age=60, priority=2,icmp6,in_port=20,icmp_type=136,nd_target=fe80::f816:3eff:fe3c:61a9 actions=NORMAL
 cookie=0xd957818e88e64f11, duration=60.741s, table=24, n_packets=0, n_bytes=0, idle_age=60, priority=2,icmp6,in_port=20,icmp_type=136,nd_target=2001:db8::a actions=NORMAL
 cookie=0xd957818e88e64f11, duration=60.792s, table=24, n_packets=2, n_bytes=84, idle_age=5, priority=2,arp,in_port=19,arp_spa=10.0.0.9 actions=resubmit(,25)
 cookie=0xd957818e88e64f11, duration=60.735s, table=24, n_packets=0, n_bytes=0, idle_age=60, priority=2,arp,in_port=20,arp_spa=172.24.4.9 actions=resubmit(,25)
 cookie=0xd957818e88e64f11, duration=64.668s, table=24, n_packets=0, n_bytes=0, idle_age=64, priority=0 actions=drop
 cookie=0xd957818e88e64f11, duration=60.821s, table=25, n_packets=9, n_bytes=770, idle_age=5, priority=2,in_port=19,dl_src=fa:16:3e:f1:25:e3 actions=NORMAL
 cookie=0xd957818e88e64f11, duration=60.755s, table=25, n_packets=0, n_bytes=0, idle_age=60, priority=2,in_port=20,dl_src=fa:16:3e:3c:61:a9 actions=NORMAL
[lihui@openstack ~]$

查看port 20是什么口

[lihui@openstack ~]$ sudo ovs-ofctl show br-int | grep "20("
 20(qvoa08c3748-20): addr:56:0f:95:11:2f:d6

table 24和25,看样子ARP都没成功

[lihui@openstack ~]$ sudo ip netns exec qrouter-70ad5580-33a1-4f77-b242-1c3d2b71af8c arp -a
? (172.24.4.9) at  on qg-9452f9cc-51
? (10.0.0.2) at fa:16:3e:62:45:8a [ether] on qr-ae962c02-ea
? (10.0.0.9) at fa:16:3e:f1:25:e3 [ether] on qr-ae962c02-ea
gateway (172.24.4.1) at 12:c0:86:a9:b0:49 [ether] on qg-9452f9cc-51

测试下

[lihui@openstack ~]$ sudo ip netns exec qrouter-70ad5580-33a1-4f77-b242-1c3d2b71af8c arping -I qg-9452f9cc-51 172.24.4.9
ARPING 172.24.4.9 from 172.24.4.2 qg-9452f9cc-51



^CSent 20 probes (20 broadcast(s))
Received 0 response(s)

 在qvo口上抓ARP,有去无回

[lihui@openstack ~]$ sudo tcpdump -i qvoa08c3748-20 arp -en
tcpdump: WARNING: qvoa08c3748-20: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on qvoa08c3748-20, link-type EN10MB (Ethernet), capture size 65535 bytes
01:33:55.544333 fa:16:3e:56:98:50 > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.24.4.9 (Broadcast) tell 172.24.4.2, length 28
01:33:56.545358 fa:16:3e:56:98:50 > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.24.4.9 (Broadcast) tell 172.24.4.2, length 28
01:33:57.548094 fa:16:3e:56:98:50 > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.24.4.9 (Broadcast) tell 172.24.4.2, length 28
01:33:58.551932 fa:16:3e:56:98:50 > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.24.4.9 (Broadcast) tell 172.24.4.2, length 28
01:33:59.552289 fa:16:3e:56:98:50 > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.24.4.9 (Broadcast) tell 172.24.4.2, length 28
01:34:00.552910 fa:16:3e:56:98:50 > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.24.4.9 (Broadcast) tell 172.24.4.2, length 28
01:34:01.553441 fa:16:3e:56:98:50 > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.24.4.9 (Broadcast) tell 172.24.4.2, length 28
01:34:02.554331 fa:16:3e:56:98:50 > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.24.4.9 (Broadcast) tell 172.24.4.2, length 28
01:34:03.555434 fa:16:3e:56:98:50 > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.24.4.9 (Broadcast) tell 172.24.4.2, length 28
01:34:04.555929 fa:16:3e:56:98:50 > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.24.4.9 (Broadcast) tell 172.24.4.2, length 28
01:34:05.556140 fa:16:3e:56:98:50 > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.24.4.9 (Broadcast) tell 172.24.4.2, length 28
01:34:06.556483 fa:16:3e:56:98:50 > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.24.4.9 (Broadcast) tell 172.24.4.2, length 28

这就坑了,又没法VNC登进虚拟机,突然想到,namespace里不是可以连通内网IP么,直接登陆

[lihui@openstack devstack]$ sudo ip netns exec qrouter-70ad5580-33a1-4f77-b242-1c3d2b71af8c ssh -i dev.private cirros@10.0.0.9
$
$
$ ip a
1: lo:  mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0:  mtu 1450 qdisc pfifo_fast qlen 1000
    link/ether fa:16:3e:f1:25:e3 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.9/26 brd 10.0.0.63 scope global eth0
    inet6 fd8d:4a3:aeb0:0:f816:3eff:fef1:25e3/64 scope global dynamic
       valid_lft 86389sec preferred_lft 14389sec
    inet6 fe80::f816:3eff:fef1:25e3/64 scope link
       valid_lft forever preferred_lft forever
3: eth1:  mtu 1500 qdisc noop qlen 1000
    link/ether fa:16:3e:3c:61:a9 brd ff:ff:ff:ff:ff:ff
$ arp -a
host-10-0-0-1.openstacklocal (10.0.0.1) at fa:16:3e:97:da:84 [ether]  on eth0
host-10-0-0-2.openstacklocal (10.0.0.2) at fa:16:3e:62:45:8a [ether]  on eth0

好事,登陆进来了,对比eth1的MAC地址和上面流表in_port=20的MAC地址,发现是一致的,也就是说这个eth1应该就是public IP地址,至于没有UP以及分配IP地址的原因应该是cirros镜像的问题,手动UP并指定IP地址

$ sudo ifconfig eth1 172.24.4.9/24 up
$ ip a show eth1
3: eth1:  mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether fa:16:3e:3c:61:a9 brd ff:ff:ff:ff:ff:ff
    inet 172.24.4.9/16 brd 172.24.255.255 scope global eth1
    inet6 fe80::f816:3eff:fe3c:61a9/64 scope link
       valid_lft forever preferred_lft forever
$

namespace外面,终于也能够通虚拟机了

[lihui@openstack ~]$ ping 172.24.4.9
PING 172.24.4.9 (172.24.4.9) 56(84) bytes of data.
64 bytes from 172.24.4.9: icmp_seq=1 ttl=64 time=1.08 ms
64 bytes from 172.24.4.9: icmp_seq=2 ttl=64 time=0.579 ms
64 bytes from 172.24.4.9: icmp_seq=3 ttl=64 time=1.02 ms
64 bytes from 172.24.4.9: icmp_seq=4 ttl=64 time=0.686 ms
64 bytes from 172.24.4.9: icmp_seq=5 ttl=64 time=0.936 ms
64 bytes from 172.24.4.9: icmp_seq=6 ttl=64 time=1.29 ms
^C
--- 172.24.4.9 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5007ms

密钥登陆

[lihui@openstack devstack]$ ssh -i dev.private cirros@172.24.4.9
The authenticity of host '172.24.4.9 (172.24.4.9)' can't be established.
RSA key fingerprint is 04:ed:ac:47:1c:f0:1b:df:8f:7e:22:e1:3a:a1:e5:ca.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.24.4.9' (RSA) to the list of known hosts.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0664 for 'dev.private' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
bad permissions: ignore key: dev.private
cirros@172.24.4.9's password:

设置600,再次登陆,搞定

[lihui@openstack devstack]$ chmod 600 dev.private
[lihui@openstack devstack]$ ssh -i dev.private cirros@172.24.4.9
$ ip a
1: lo:  mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0:  mtu 1450 qdisc pfifo_fast qlen 1000
    link/ether fa:16:3e:f1:25:e3 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.9/26 brd 10.0.0.63 scope global eth0
    inet6 fd8d:4a3:aeb0:0:f816:3eff:fef1:25e3/64 scope global dynamic
       valid_lft 86399sec preferred_lft 14399sec
    inet6 fe80::f816:3eff:fef1:25e3/64 scope link
       valid_lft forever preferred_lft forever
3: eth1:  mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether fa:16:3e:3c:61:a9 brd ff:ff:ff:ff:ff:ff
    inet 172.24.4.9/16 brd 172.24.255.255 scope global eth1
    inet6 fe80::f816:3eff:fe3c:61a9/64 scope link
       valid_lft forever preferred_lft forever
$

如此一来就可以直接登陆虚拟机了,至于VNC的问题有空再看 

纯属恶搞

浙ICP备16024533号

浙公网安备 33010802007459号