最近做了一个十分奇怪的测试,虽然还没get到这么做的好处
图看上去很简单,两个namespace,可以看成共享内核的容器,分别通过veth peer设备连到了Linux Bridge上,通过OVS创建的TAP设备也绑到了Linux Bridge上,中间这部分也放在了一个network namespace里,是为了防止如果是私有网络,那么就不应该暴露出来,其实还有一个前提,这一切都是在物理机上来进行
正常情况下可能TAP设备对应着一个虚拟机,而且不同租户不同的网络,对应的vlan tag都是不一样的,如果按照这种场景,这里物理机只能提供一个租户来使用,而且上面两个namespace就像是把TAP设备当做网关来使用一样,这里就有两步,首先所有的网络设备都可以直接来创建,当然有的必须在命名空间中创建,然后网络信息方面可以调底层创建neutron port,要用的就是相应的网络信息
首先是最上层两个namespace和两对veth peer设备
~$ sudo ip netns exec ns1 ip a 1: lo: mtu 65536 qdisc noop state DOWN group default link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2214: veth1: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether b2:f5:71:7f:90:40 brd ff:ff:ff:ff:ff:ff inet 10.177.81.180/23 scope global veth1 valid_lft forever preferred_lft forever inet6 fe80::b0f5:71ff:fe7f:9040/64 scope link valid_lft forever preferred_lft forever ~$ sudo ip netns exec ns2 ip a 1: lo: mtu 65536 qdisc noop state DOWN group default link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2218: veth2: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 76:07:17:be:72:e5 brd ff:ff:ff:ff:ff:ff inet 10.177.81.182/23 scope global veth2 valid_lft forever preferred_lft forever inet6 fe80::7407:17ff:febe:72e5/64 scope link valid_lft forever preferred_lft forever
其次中间Bridge呆的地方也创建namespace,将veth peer的另一端两个设备放进去,里面创建Linux Bridge,将两个veth peer设备绑在Bridge上,最后调OVS接口将neutron port映射成TAP设备挂载到物理机上,也绑在Linux Bridge上
~$ sudo ip netns exec br_ns ip a 1: lo: mtu 65536 qdisc noop state DOWN group default link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: br-1: mtu 1500 qdisc noqueue state UP group default link/ether 96:f4:88:de:5b:ec brd ff:ff:ff:ff:ff:ff inet 10.177.81.181/23 scope global br-1 valid_lft forever preferred_lft forever inet6 fe80::44d4:e2ff:fe1d:2eca/64 scope link valid_lft forever preferred_lft forever 2213: vnet1: mtu 1500 qdisc pfifo_fast master br-1 state UP group default qlen 1000 link/ether ee:05:cb:23:62:b7 brd ff:ff:ff:ff:ff:ff inet6 fe80::ec05:cbff:fe23:62b7/64 scope link valid_lft forever preferred_lft forever 2216: tap4cf6fdaa-f9: mtu 1500 qdisc noqueue master br-1 state UNKNOWN group default link/ether 96:f4:88:de:5b:ec brd ff:ff:ff:ff:ff:ff inet6 fe80::94f4:88ff:fede:5bec/64 scope link valid_lft forever preferred_lft forever 2217: vnet2: mtu 1500 qdisc pfifo_fast master br-1 state UP group default qlen 1000 link/ether ae:30:f7:9a:4e:35 brd ff:ff:ff:ff:ff:ff inet6 fe80::ac30:f7ff:fe9a:4e35/64 scope link valid_lft forever preferred_lft forever
这里调用ovs接口来创建并挂载TAP设备的接口稍微比较长,提供一个简单的shell脚本
#!/bin/bash port_uuid=$1 tap_uuid="tap"${port_uuid:0:11} mac_uuid=$2 ovs-vsctl -- --if-exists del-port $tap_uuid ovs-vsctl -- add-port br-int $tap_uuid -- set Interface $tap_uuid type=internal -- set Interface $tap_uuid external-ids:iface-id=$port_uuid -- set Interface $tap_uuid external-ids:iface-status=active -- set Interface $tap_uuid external-ids:attached-mac=$mac_uuid
比较复杂,创建neutron port之后,将UUID和MAC地址当参数传进去,就会返回一个挂载在物理机上的TAP设备,而这个TAP设备放到Bridge所在的namespace里最终并绑上Linux Bridge
查看Linux Bridge信息
~$ sudo brctl show bridge name bridge id STP enabled interfaces br-1 8000.96f488de5bec no tap4cf6fdaa-f9 vnet1 vnet2
要注意的是,bridge最终还是要up起来,甚至新创neutron port的IP信息也分配给Linux Bridge才能通,这里简单抓下包确认连通性无误
~$ sudo ip netns exec ns1 ping 10.177.81.182 PING 10.177.81.182 (10.177.81.182) 56(84) bytes of data. 64 bytes from 10.177.81.182: icmp_req=1 ttl=64 time=0.070 ms 64 bytes from 10.177.81.182: icmp_req=2 ttl=64 time=0.045 ms 64 bytes from 10.177.81.182: icmp_req=3 ttl=64 time=0.032 ms ^C --- 10.177.81.182 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 1999ms rtt min/avg/max/mdev = 0.032/0.049/0.070/0.015 ms hzlihui15@10-180-0-39:~/map$ ~$ sudo ip netns exec ns2 tcpdump -i veth2 icmp -en tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on veth2, link-type EN10MB (Ethernet), capture size 65535 bytes 17:24:35.462462 b2:f5:71:7f:90:40 > 76:07:17:be:72:e5, ethertype IPv4 (0x0800), length 98: 10.177.81.180 > 10.177.81.182: ICMP echo request, id 10822, seq 1, length 64 17:24:35.462503 76:07:17:be:72:e5 > b2:f5:71:7f:90:40, ethertype IPv4 (0x0800), length 98: 10.177.81.182 > 10.177.81.180: ICMP echo reply, id 10822, seq 1, length 64 17:24:36.461515 b2:f5:71:7f:90:40 > 76:07:17:be:72:e5, ethertype IPv4 (0x0800), length 98: 10.177.81.180 > 10.177.81.182: ICMP echo request, id 10822, seq 2, length 64 17:24:36.461531 76:07:17:be:72:e5 > b2:f5:71:7f:90:40, ethertype IPv4 (0x0800), length 98: 10.177.81.182 > 10.177.81.180: ICMP echo reply, id 10822, seq 2, length 64 17:24:37.461516 b2:f5:71:7f:90:40 > 76:07:17:be:72:e5, ethertype IPv4 (0x0800), length 98: 10.177.81.180 > 10.177.81.182: ICMP echo request, id 10822, seq 3, length 64 17:24:37.461532 76:07:17:be:72:e5 > b2:f5:71:7f:90:40, ethertype IPv4 (0x0800), length 98: 10.177.81.182 > 10.177.81.180: ICMP echo reply, id 10822, seq 3, length 64 ^C 6 packets captured 6 packets received by filter 0 packets dropped by kernel ~$ sudo ip netns exec br_ns tcpdump -i br-1 icmp -en tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on br-1, link-type EN10MB (Ethernet), capture size 65535 bytes 17:24:44.662475 b2:f5:71:7f:90:40 > 76:07:17:be:72:e5, ethertype IPv4 (0x0800), length 98: 10.177.81.180 > 10.177.81.182: ICMP echo request, id 10849, seq 1, length 64 17:24:44.662519 76:07:17:be:72:e5 > b2:f5:71:7f:90:40, ethertype IPv4 (0x0800), length 98: 10.177.81.182 > 10.177.81.180: ICMP echo reply, id 10849, seq 1, length 64 17:24:45.661503 b2:f5:71:7f:90:40 > 76:07:17:be:72:e5, ethertype IPv4 (0x0800), length 98: 10.177.81.180 > 10.177.81.182: ICMP echo request, id 10849, seq 2, length 64 17:24:45.661530 76:07:17:be:72:e5 > b2:f5:71:7f:90:40, ethertype IPv4 (0x0800), length 98: 10.177.81.182 > 10.177.81.180: ICMP echo reply, id 10849, seq 2, length 64 17:24:46.661500 b2:f5:71:7f:90:40 > 76:07:17:be:72:e5, ethertype IPv4 (0x0800), length 98: 10.177.81.180 > 10.177.81.182: ICMP echo request, id 10849, seq 3, length 64 17:24:46.661519 76:07:17:be:72:e5 > b2:f5:71:7f:90:40, ethertype IPv4 (0x0800), length 98: 10.177.81.182 > 10.177.81.180: ICMP echo reply, id 10849, seq 3, length 64 ^C 6 packets captured 6 packets received by filter 0 packets dropped by kernel