Neutron里OpenvSwitch Security Group的神奇BUG

作者: LiHui 分类: Networking 发布时间: 2017-01-10 23:45

在测试云网络OVS安全组的过程中,遇到了一些奇怪的问题,其中最神奇的当属下面一个

简单说下基于OpenvSwitch的安全组,实质上就是一个firewall,只不过是基于ovs的,也就是针对PORT的,但是也分ingress和egress,这里可以对应iptables里的INPUT和OUTPUT链,需要切记的是ingress的限制是白名单,也就是添加一条规则,表明了允许该规则的包能够进来,而egress的限制是黑名单,也就是默认出方向畅通无阻,但该方向安全组规则是不让满足规则的包出去,与此同时对应的四元组也有所不同,对于IP Addr来说,ingress针对的是src ipaddr,egress针对的是dst ipaddr,而对于PORT来说,ingress和egress针对的永远是dst port,因此可以看到,假如在测试当中,源和目的两方,假如security group不做任何修改,入方向包就全部丢掉了,所以ingress方向需要首先全部放行,再来逐一验证安全组规则

大致原理内容简述完了,下面是一个神奇的BUG

测试的具体用例就是egress方向端口的屏蔽,上面已经说过了,egress方向默认全开放,指定的端口就是要被屏蔽的,因此这里通过iperf工具来进行流量测试,这里只需要知道一点,iperf的server端默认监听端口是5001

A:安全组规则egress方向指定tcp协议,端口5001,经过测试流量的确被过滤掉了

B:安全组规则egress方向指定tcp协议,端口5002,经过测试流量正常发送和接收,没被过滤掉,大概长这样

$ neutron security-group-show 61e43f12-12ec-48a1-b2d5-c7392290978c
+----------------------+--------------------------------------------------------------------+
| Field                | Value                                                              |
+----------------------+--------------------------------------------------------------------+
| description          |                                                                    |
| id                   | 61e43f12-12ec-48a1-b2d5-c7392290978c                               |
| name                 | group-egress-tcp-115.236.127.223-5002                              |
| security_group_rules | {                                                                  |
|                      |      "remote_group_id": null,                                      |
|                      |      "direction": "egress",                                        |
|                      |      "remote_ip_prefix": "115.236.127.223/32",                     |
|                      |      "protocol": "tcp",                                            |
|                      |      "tenant_id": "10e5051a1cee4f8ebb0e8b5d877de581",              |
|                      |      "port_range_max": 5002,                                       |
|                      |      "security_group_id": "61e43f12-12ec-48a1-b2d5-c7392290978c",  |
|                      |      "port_range_min": 5002,                                       |
|                      |      "ethertype": "IPv4",                                          |
|                      |      "id": "f2994944-958b-4f06-aed4-4f3dccf903fc"                  |
|                      | }                                                                  |
| tenant_id            | 10e5051a1cee4f8ebb0e8b5d877de581                                   |
+----------------------+--------------------------------------------------------------------+

~# iperf -c 115.236.127.223 -t 10000 -i 1
------------------------------------------------------------
Client connecting to 115.236.127.223, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 115.236.127.222 port 56948 connected with 115.236.127.223 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec  57.4 MBytes   481 Mbits/sec
[  3]  1.0- 2.0 sec  57.2 MBytes   480 Mbits/sec
[  3]  2.0- 3.0 sec  56.9 MBytes   477 Mbits/sec
[  3]  3.0- 4.0 sec  57.1 MBytes   479 Mbits/sec
[  3]  4.0- 5.0 sec  57.2 MBytes   480 Mbits/sec
[  3]  5.0- 6.0 sec  56.9 MBytes   477 Mbits/sec
[  3]  6.0- 7.0 sec  57.4 MBytes   481 Mbits/sec
[  3]  7.0- 8.0 sec  57.6 MBytes   483 Mbits/sec

~# iperf -s -i 1
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 115.236.127.223 port 5001 connected with 115.236.127.222 port 56948
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0- 1.0 sec  57.0 MBytes   478 Mbits/sec
[  4]  1.0- 2.0 sec  57.0 MBytes   478 Mbits/sec
[  4]  2.0- 3.0 sec  57.0 MBytes   478 Mbits/sec
[  4]  3.0- 4.0 sec  57.0 MBytes   478 Mbits/sec
[  4]  4.0- 5.0 sec  57.0 MBytes   479 Mbits/sec
[  4]  5.0- 6.0 sec  57.0 MBytes   478 Mbits/sec
[  4]  6.0- 7.0 sec  57.0 MBytes   478 Mbits/sec
[  4]  7.0- 8.0 sec  57.0 MBytes   479 Mbits/sec

C:安全组规则egress方向指定tcp协议,端口5000,经过测试流量被过滤掉了,居然被丢掉了!

$ neutron security-group-show a762e5d7-d3c1-490b-92e2-ffe844a80650
+----------------------+--------------------------------------------------------------------+
| Field                | Value                                                              |
+----------------------+--------------------------------------------------------------------+
| description          |                                                                    |
| id                   | a762e5d7-d3c1-490b-92e2-ffe844a80650                               |
| name                 | group-egress-tcp-115.236.127.223-5000                              |
| security_group_rules | {                                                                  |
|                      |      "remote_group_id": null,                                      |
|                      |      "direction": "egress",                                        |
|                      |      "remote_ip_prefix": "115.236.127.223/32",                     |
|                      |      "protocol": "tcp",                                            |
|                      |      "tenant_id": "10e5051a1cee4f8ebb0e8b5d877de581",              |
|                      |      "port_range_max": 5000,                                       |
|                      |      "security_group_id": "a762e5d7-d3c1-490b-92e2-ffe844a80650",  |
|                      |      "port_range_min": 5000,                                       |
|                      |      "ethertype": "IPv4",                                          |
|                      |      "id": "5edefe65-48da-4b52-b257-dd5b147ef1a6"                  |
|                      | }                                                                  |
| tenant_id            | 10e5051a1cee4f8ebb0e8b5d877de581                                   |
+----------------------+--------------------------------------------------------------------+


~# iperf -c 115.236.127.223 -t 10000 -i 1


^C

上面的操作,iperf的client和server端都没有做任何变动,修改的只是neutron port的security group规则,理论上指定了5001端口,才会被过滤,5000和5002理应都不会丢掉流量,可见5000也丢掉是一个奇怪的问题,以为有其他规则影响,查找原因足足测试了3次!!终于确认这是一个BUG

至于测试过程中为什么偏偏选个5000和5002,就近原则吧,当然random一个也可以,不过作为一般边界值测试容易出问题来说,靠近5001两边的值出问题的可能性也不小

至于这个问题,在社区也能够找到这个BUG,可惜这功能咱们上得太晚了,不然给社区提这个BUG的说不定就是我了

https://bugs.launchpad.net/neutron/+bug/1611991

这哥们测试用例应该和我一致,只不过端口号他选的是22和23,没有写ingress还是egress应该是默认的ingress,只不过只设置了22端口,结果23端口也生效了

Seen on master devstack, ubuntu xenial.

Steps to reproduce:

1. Enable ovs firewall in /etc/neutron/plugins/ml2/ml2.conf

[securitygroup]
firewall_driver = openvswitch

2. Create a security group with icmp, tcp to 22.

3. Boot a VM, assign a floating ip.

4. Check that port 23 can be accessed via tcp (telnet, nc, etc).

可见这个问题影响远远不止一两个端口,而是某种算法或者什么原因导致规则出现BUG

下面有一个哥们说

The bug is in port masking, 22 is masked by tp_src=0x16/0xfffe which matches number 23 as well. Good catch!

Changed in neutron:
importance:	Undecided → High

这里还是说的一样的,接着往下看,一堆人轮流轰炸了一堆之后,来了个fix

Reviewed: https://review.openstack.org/353782
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0494f212aa625a03587af3d75e823008f1198012
Submitter: Jenkins
Branch: master

commit 0494f212aa625a03587af3d75e823008f1198012
Author: Inessa Vasilevskaya
Date: Thu Aug 11 02:21:29 2016 +0300

    ovsfw: fix troublesome port_rule_masking

    In several cases port masking algorithm borrowed
    from networking_ovs_dpdk didn't behave correctly.
    This caused non-restricted ports to be open due to
    wrong tp_src field value in resulting ovs rules.

    This was fixed by alternative port masking
    implementation.

    Functional and unit tests to cover the bug added as well.

    Co-Authored-By: Jakub Libosvar 
    Co-Authored-By: IWAMOTO Toshihiro 

这里说明了,ovs Firewall,修复了port_rule_masking的问题,在某些情况下端口屏蔽的算法不对;结果就是由于ovs规则里错误的tp_src字段的值导致非限制的端口也屏蔽了

修改了一大波,可以看看commit

https://git.openstack.org/cgit/openstack/neutron/commit/?id=dd75f7e96afc713b57ad4ab21f01175be7b571fe

这规则屏蔽算法是在看着心碎,本来还想来个BUG分享,看这问题原因还是算了吧,总结就四个字:算法有误

解决办法就是,将上面的Reviewd版本合进来,这已经Commit到Master里了

Reviewed: https://review.openstack.org/353782
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0494f212aa625a03587af3d75e823008f1198012

更新完之后恢复正常

$ neutron security-group-show a762e5d7-d3c1-490b-92e2-ffe844a80650
+----------------------+--------------------------------------------------------------------+
| Field                | Value                                                              |
+----------------------+--------------------------------------------------------------------+
| description          |                                                                    |
| id                   | a762e5d7-d3c1-490b-92e2-ffe844a80650                               |
| name                 | group-egress-tcp-115.236.127.223-5000                              |
| security_group_rules | {                                                                  |
|                      |      "remote_group_id": null,                                      |
|                      |      "direction": "egress",                                        |
|                      |      "remote_ip_prefix": "115.236.127.223/32",                     |
|                      |      "protocol": "tcp",                                            |
|                      |      "tenant_id": "10e5051a1cee4f8ebb0e8b5d877de581",              |
|                      |      "port_range_max": 5000,                                       |
|                      |      "security_group_id": "a762e5d7-d3c1-490b-92e2-ffe844a80650",  |
|                      |      "port_range_min": 5000,                                       |
|                      |      "ethertype": "IPv4",                                          |
|                      |      "id": "5edefe65-48da-4b52-b257-dd5b147ef1a6"                  |
|                      | }                                                                  |
| tenant_id            | 10e5051a1cee4f8ebb0e8b5d877de581                                   |
+----------------------+--------------------------------------------------------------------+

~# iperf -s -i 1
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 115.236.127.223 port 5001 connected with 115.236.127.222 port 39420
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0- 1.0 sec  57.0 MBytes   478 Mbits/sec
[  4]  1.0- 2.0 sec  57.0 MBytes   479 Mbits/sec
[  4]  2.0- 3.0 sec  57.0 MBytes   478 Mbits/sec
[  4]  3.0- 4.0 sec  57.0 MBytes   478 Mbits/sec
[  4]  4.0- 5.0 sec  57.0 MBytes   478 Mbits/sec
[  4]  5.0- 6.0 sec  57.0 MBytes   478 Mbits/sec
[  4]  6.0- 7.0 sec  57.0 MBytes   478 Mbits/sec
[  4]  7.0- 8.0 sec  57.0 MBytes   478 Mbits/sec
[  4]  8.0- 9.0 sec  57.0 MBytes   478 Mbits/sec
[  4]  9.0-10.0 sec  57.0 MBytes   478 Mbits/sec
[  4] 10.0-11.0 sec  57.0 MBytes   478 Mbits/sec
[  4] 11.0-12.0 sec  57.0 MBytes   478 Mbits/sec

 

纯属恶搞

浙ICP备16024533号

浙公网安备 33010802007459号