测试框架Tempest进阶:第一坑一头雾水

对于OpenStack里API接口的测试,由于按需一直在变,云网络接口相对较少,一直放在持续集成测试中,而云主机部分接口实在太多,涉及到的模块也多,而主机部分大多生命周期操作以及验证都还涉及到多个物理节点,因此目前仅仅在一整个OpenStack环境通过整体功能回归验证其正确性,而对于接口层验证目前还是比较老,供提交代码之后来触发的接口测试,而对于新增功能接口,都没有实时添加进去,更甚之由于更新频率过快过多,tempest里很多老接口都无法正常通过,因此修复目前所有接口的问题,以及添加新的接口任重而道远,随便看一个简单的试试水就坑了

$ nosetests -sv tempest/api/compute/flavors/test_flavors.py
tempest.api.compute.flavors.test_flavors.FlavorsTestJSON.create_test_server ... FAIL
tempest.api.compute.flavors.test_flavors.FlavorsTestJSON.test_get_flavor ... ok
tempest.api.compute.flavors.test_flavors.FlavorsTestJSON.test_list_flavors ... ok
tempest.api.compute.flavors.test_flavors.FlavorsTestJSON.test_list_flavors_detailed_filter_by_min_disk ... ok
tempest.api.compute.flavors.test_flavors.FlavorsTestJSON.test_list_flavors_detailed_filter_by_min_ram ... ok
tempest.api.compute.flavors.test_flavors.FlavorsTestJSON.test_list_flavors_detailed_limit_results ... ok
tempest.api.compute.flavors.test_flavors.FlavorsTestJSON.test_list_flavors_detailed_using_marker ... ok
tempest.api.compute.flavors.test_flavors.FlavorsTestJSON.test_list_flavors_filter_by_min_disk ... ok
tempest.api.compute.flavors.test_flavors.FlavorsTestJSON.test_list_flavors_filter_by_min_ram ... ok
tempest.api.compute.flavors.test_flavors.FlavorsTestJSON.test_list_flavors_limit_results ... ok
tempest.api.compute.flavors.test_flavors.FlavorsTestJSON.test_list_flavors_using_marker ... ok
tempest.api.compute.flavors.test_flavors.FlavorsTestJSON.test_list_flavors_with_detail ... ok

======================================================================
FAIL: tempest.api.compute.flavors.test_flavors.FlavorsTestJSON.create_test_server

莫名其妙,怎么会有创建虚拟机,错误信息网络不对

2016-12-18 16:26:21,373 Response Status: 400
2016-12-18 16:26:21,374 Nova request id: req-0f0cc17b-47e3-4f97-a883-c2fb849e33f5
2016-12-18 16:26:21,374 Response Headers: {'content-length': '115', 'date': 'Sun, 18 Dec 2016 08:26:21 GMT', 'content-type': 'application/json; charset=UTF-8', 'connection': 'close'}
2016-12-18 16:26:21,374 Response Body: {"badRequest": {"message": "Multiple possible networks found, use a Network ID to be more specific.", "code": 400}}
}}}

根据requestID,查看对应API的request里BODY有关网络的详细信息

2016-12-18 16:26:21.288 23084 DEBUG neutronclient.client [-] RESP:{'date': 'Sun, 18 Dec 2016 08:26:21 GMT', 'status': '200', 'content-length': '2030', 'cont
ent-type': 'application/json', 'content-location': 'http://xx.xx.xx.xx:9696/v2.0/networks.json?shared=True'} {"networks": [{"status": "ACTIVE", "subnets": [
"347492b6-00b5-4749-a293-3668cf291b31"], "support_azs": ["xx"], "extra_network_opts": [{"opt_value": "0", "opt_name": "network_max_port"}, {"opt_value": "[\
"xx\"]", "opt_name": "support_azs"}], "name": "public_admin_xx", "router:external": false, "tenant_id": "a33d46db489949939537c94eeb089ae6", "segments": [{"n
etwork_id": "78763287-e904-426a-b2b3-9b180830c77e", "mtu": 1500, "id": "c03c3d5b-9482-4570-ae66-153bc1455b74", "provider:physical_network": "public-xx", "ne
twork_type": "flat"}], "admin_state_up": true, "provider:network_type": "flat", "shared": true, "port_security_enabled": true, "mtu": 1500, "id": "78763287-
e904-426a-b2b3-9b180830c77e"}, {"status": "ACTIVE", "subnets": ["838ea352-3622-4cdc-97ab-21dfab80fd5d"], "support_azs": ["xx"], "extra_network_opts": [{"opt
_value": "[\"xx\"]", "opt_name": "support_azs"}], "name": "public_admin_xx_1", "router:external": false, "tenant_id": "a33d46db489949939537c94eeb089ae6", "s
egments": [{"network_id": "961fcd33-34b6-440e-b1f8-1c431f0e0691", "mtu": 1500, "id": "df69e7ce-cbfb-4302-b6d4-30af6e213526", "provider:physical_network": "p
ublic-xx-1", "network_type": "flat"}], "admin_state_up": true, "provider:network_type": "flat", "shared": true, "port_security_enabled": true, "mtu": 1500,
"id": "961fcd33-34b6-440e-b1f8-1c431f0e0691"}, {"status": "ACTIVE", "subnets": ["3f4a0aef-73bb-4240-a4d8-187dc0d9208f"], "support_azs": ["xx"], "extra_netwo
rk_opts": [{"opt_value": "0", "opt_name": "network_max_port"}, {"opt_value": "[\"xx\"]", "opt_name": "support_azs"}], "name": "public_admin_xx", "router:ext
ernal": false, "tenant_id": "a33d46db489949939537c94eeb089ae6", "segments": [{"network_id": "dae73933-9e60-4ad7-b48c-046aca96343f", "mtu": 1500, "id": "9ae7
e3a9-bb70-43e4-acf6-45c4474deec7", "provider:physical_network": "public-xx", "network_type": "flat"}], "admin_state_up": true, "provider:network_type": "fla
t", "shared": true, "port_security_enabled": true, "mtu": 1500, "id": "dae73933-9e60-4ad7-b48c-046aca96343f"}]}
http_log_resp /srv/stack/nova/lib/python2.7/site-packages/neutronclient/common/utils.py:218

里面貌似传入了public网络,猜测tempest默认public来创建server,至于是之前就是这样还是后来有人改过就不从得知了,接着就只有看源码了,看创建虚拟机的传入的参数即可

test_flavor.py里,都是flavor接口调用python框架进行assert验证的,根本就没看到RPC操作,就只有顺着看构造函数有没有了

class FlavorsTestJSON(base.BaseV2ComputeTest):
_interface = 'json'

@classmethod
def setUpClass(cls):
super(FlavorsTestJSON, cls).setUpClass()
cls.client = cls.flavors_client

简单看下调用关系

class BaseV2ComputeTest(BaseComputeTest):

@classmethod
def setUpClass(cls):
super(BaseV2ComputeTest, cls).setUpClass()
cls.servers_client = cls.os.servers_client
cls.flavors_client = cls.os.flavors_client
cls.images_client = cls.os.images_client
cls.extensions_client = cls.os.extensions_client
cls.floating_ips_client = cls.os.floating_ips_client
cls.keypairs_client = cls.os.keypairs_client
cls.security_groups_client = cls.os.security_groups_client
cls.quotas_client = cls.os.quotas_client
cls.limits_client = cls.os.limits_client
cls.volumes_extensions_client = cls.os.volumes_extensions_client
cls.volumes_client = cls.os.volumes_client
cls.interfaces_client = cls.os.interfaces_client
cls.fixed_ips_client = cls.os.fixed_ips_client
cls.availability_zone_client = cls.os.availability_zone_client
cls.aggregates_client = cls.os.aggregates_client
cls.services_client = cls.os.services_client
cls.instance_usages_audit_log_client = \
cls.os.instance_usages_audit_log_client
cls.hypervisor_client = cls.os.hypervisor_client
cls.servers_client_v3_auth = cls.os.servers_client_v3_auth
cls.certificates_client = cls.os.certificates_client

就对照flavor_client来看

class BaseComputeTest(tempest.test.BaseTestCase):
"""Base test case class for all Compute API tests."""

force_tenant_isolation = False

@classmethod
def setUpClass(cls):
super(BaseComputeTest, cls).setUpClass()
if not cls.config.service_available.nova:
skip_msg = ("%s skipped as nova is not available" % cls.__name__)
raise cls.skipException(skip_msg)

os = cls.get_client_manager()

cls.os = os
cls.build_interval = cls.config.compute.build_interval
cls.build_timeout = cls.config.compute.build_timeout
cls.ssh_user = cls.config.compute.ssh_user
cls.image_ssh_user = cls.config.compute.image_ssh_user
cls.image_ssh_password = cls.config.compute.image_ssh_password
cls.image_alt_ssh_user = cls.config.compute.image_alt_ssh_user
cls.image_alt_ssh_password = cls.config.compute.image_alt_ssh_password
cls.image_ref = cls.config.compute.image_ref
cls.image_ref_alt = cls.config.compute.image_ref_alt
cls.flavor_ref = cls.config.compute.flavor_ref
cls.flavor_ref_alt = cls.config.compute.flavor_ref_alt
cls.servers = []
cls.images = []
cls.multi_user = cls.get_multi_user()

继续看基类到case.py里了,至今没有找到创建server的相关内容,因此这里放弃了

可后面的更不太可能,全都是通过调用RESTFul API来验证接口正确性

看看中途调用的函数

@classmethod
def get_client_manager(cls):
"""
Returns an Openstack client manager
"""
cls.isolated_creds = isolated_creds.IsolatedCreds(cls.__name__)

force_tenant_isolation = getattr(cls, 'force_tenant_isolation', None)
if (cls.config.compute.allow_tenant_isolation or
force_tenant_isolation):
creds = cls.isolated_creds.get_primary_creds()
username, tenant_name, password = creds
os = clients.Manager(username=username,
password=password,
tenant_name=tenant_name,
interface=cls._interface)
else:
os = clients.Manager(interface=cls._interface)
return os

Manager函数里继续跟踪flavor_client

elif interface == 'json':
self.certificates_client = CertificatesClientJSON(*client_args)
self.certificates_v3_client = CertificatesV3ClientJSON(
*client_args)
self.baremetal_client = BaremetalClientJSON(*client_args)
self.servers_client = ServersClientJSON(*client_args)
self.servers_v3_client = ServersV3ClientJSON(*client_args)
self.limits_client = LimitsClientJSON(*client_args)
self.images_client = ImagesClientJSON(*client_args)
self.keypairs_v3_client = KeyPairsV3ClientJSON(*client_args)
self.keypairs_client = KeyPairsClientJSON(*client_args)
self.keypairs_v3_client = KeyPairsV3ClientJSON(*client_args)
self.quotas_client = QuotasClientJSON(*client_args)
self.quotas_v3_client = QuotasV3ClientJSON(*client_args)
self.flavors_client = FlavorsClientJSON(*client_args)
self.flavors_v3_client = FlavorsV3ClientJSON(*client_args)
self.extensions_v3_client = ExtensionsV3ClientJSON(*client_args)
self.extensions_client = ExtensionsClientJSON(*client_args)

最后就能找到借口测试RestFul API具体URL,BODY的地方

class FlavorsClientJSON(RestClient):

def __init__(self, config, username, password, auth_url, tenant_name=None):
super(FlavorsClientJSON, self).__init__(config, username, password,
auth_url, tenant_name)
self.service = self.config.compute.catalog_type

def list_flavors(self, params=None):
url = 'flavors'
if params:
url += '?%s' % urllib.urlencode(params)

resp, body = self.get(url)
body = json.loads(body)
return resp, body['flavors']

API接口流程都走了一遍了,但是依旧没有找到调用创建server的地方

先找到函数,打个断电,单步调试一下

@classmethod
def create_test_server(cls, **kwargs):
print(“\n###### start #######")
import pdb;pdb.set_trace()
print("###### stop #######")
"""Wrapper utility that returns a test server."""

执行结果,确认的确是执行了这里

$ nosetests -sv flavors/test_flavors.py
tempest.api.compute.flavors.test_flavors.FlavorsTestJSON.create_test_server ...
###### start #######
> /Users/lihui/work/openstack/tempest-ci/tempest/api/compute/base.py(120)create_test_server()
-> print("###### stop #######")
(Pdb) n
###### stop #######

那就接下来看调用的函数,create_server

def create_server(self, name, image_ref, flavor_ref, **kwargs):
"""
Creates an instance of a server.
name (Required): The name of the server.
image_ref (Required): Reference to the image used to build the server.
flavor_ref (Required): The flavor used to build the server.
Following optional keyword arguments are accepted:
adminPass: Sets the initial root password.
key_name: Key name of keypair that was created earlier.
meta: A dictionary of values to be used as metadata.
personality: A list of dictionaries for files to be injected into
the server.
security_groups: A list of security group dicts.
networks: A list of network dicts with UUID and fixed_ip.
user_data: User data for instance.
availability_zone: Availability zone in which to launch instance.
accessIPv4: The IPv4 access address for the server.
accessIPv6: The IPv6 access address for the server.
min_count: Count of minimum number of instances to launch.
max_count: Count of maximum number of instances to launch.
disk_config: Determines if user or admin controls disk configuration.
return_reservation_id: Enable/Disable the return of reservation id

根据注释,我们关心的是networks这个参数,先确认参数

if isinstance(option, tuple):
post_param = option[0]
key = option[1]
else:
post_param = option
key = option
value = kwargs.get(key)

从这里可以看到,body的value都是从kwargs传进来的,倒着找还是来自create_test_server

捋一下,现在的问题还是,找不到调用创建server的地方,莫名其妙

换一个testcase还是如此

$ nosetests -sv keypairs/test_keypairs.py
tempest.api.compute.keypairs.test_keypairs.KeyPairsTestJSON.create_test_server ... FAIL

继续换一个网络的就不调用

$ nosetests -sv api/network/test_extensions.py
tempest.api.network.test_extensions.ExtensionsTestJSON.test_list_show_extensions ... ok

----------------------------------------------------------------------
Ran 1 test in 3.695s

OK

 往窗外瞄了两眼,突然心里一震,调用的那函数名是不是带了test字样???

OMG,create_test_server,哎,自己傻了,肯定是我执行testcase某一个基类里的函数

于是,再来了一遍,继承关系

class FlavorsTestJSON(base.BaseV2ComputeTest):
_interface = 'json'

@classmethod
def setUpClass(cls):
super(FlavorsTestJSON, cls).setUpClass()
cls.client = cls.flavors_client

基类里没有

class BaseV2ComputeTest(BaseComputeTest):

@classmethod
def setUpClass(cls):
super(BaseV2ComputeTest, cls).setUpClass()

继续基类,就有了

class BaseComputeTest(tempest.test.BaseTestCase):
"""Base test case class for all Compute API tests."""

force_tenant_isolation = False

@classmethod
def setUpClass(cls):

.......中间省略........

@classmethod
def create_test_server(cls, **kwargs):
"""Wrapper utility that returns a test server."""

这才是真正的原因,在执行flavor这个testcase的时候,会遍历继承的类以及其函数,而这个函数里带了test字样,因此再用nose执行的时候,也默认会执行,而且因为是在构造函数里,所以会最先执行,因此一开始思路不对,不应该直接寻找该函数的调用关系,而应该同时注意nose的执行原理

做一下简单测试,将create_test_server重命名一下,反正此时不创建虚拟机也不会出错

改成这样

@classmethod
def create_server(cls, **kwargs):
"""Wrapper utility that returns a test server."""
name = data_utils.rand_name(cls.__name__ + "-instance")
if 'name' in kwargs:
name = kwargs.pop('name')
flavor = kwargs.get('flavor', cls.flavor_ref)
image_id = kwargs.get('image_id', c

执行一下,果然就没有执行

$ nosetests -sv flavors/test_flavors.py
tempest.api.compute.flavors.test_flavors.FlavorsTestJSON.test_get_flavor ... ok
tempest.api.compute.flavors.test_flavors.FlavorsTestJSON.test_list_flavors ... ok
tempest.api.compute.flavors.test_flavors.FlavorsTestJSON.test_list_flavors_detailed_filter_by_min_disk ... ok
tempest.api.compute.flavors.test_flavors.FlavorsTestJSON.test_list_flavors_detailed_filter_by_min_ram ... ok
tempest.api.compute.flavors.test_flavors.FlavorsTestJSON.test_list_flavors_detailed_limit_results ... ok
tempest.api.compute.flavors.test_flavors.FlavorsTestJSON.test_list_flavors_detailed_using_marker ... ok
tempest.api.compute.flavors.test_flavors.FlavorsTestJSON.test_list_flavors_filter_by_min_disk ... ok
tempest.api.compute.flavors.test_flavors.FlavorsTestJSON.test_list_flavors_filter_by_min_ram ... ok
tempest.api.compute.flavors.test_flavors.FlavorsTestJSON.test_list_flavors_limit_results ... ok
tempest.api.compute.flavors.test_flavors.FlavorsTestJSON.test_list_flavors_using_marker ... ok
tempest.api.compute.flavors.test_flavors.FlavorsTestJSON.test_list_flavors_with_detail ... ok

----------------------------------------------------------------------
Ran 11 tests in 4.947s

OK

就是这个原因,执行testcase得细致

接下来再次回到上面的问题,执行create_test_server出错,根据API返回可以看到是400,看下完整的函数

@classmethod
def create_test_server(cls, **kwargs):
"""Wrapper utility that returns a test server."""
name = data_utils.rand_name(cls.__name__ + "-instance")
if 'name' in kwargs:
name = kwargs.pop('name')
flavor = kwargs.get('flavor', cls.flavor_ref)
image_id = kwargs.get('image_id', cls.image_ref)

resp, body = cls.servers_client.create_server(
name, image_id, flavor, **kwargs)

# handle the case of multiple servers
servers = [body]
if 'min_count' in kwargs or 'max_count' in kwargs:
# Get servers created which name match with name param.
r, b = cls.servers_client.list_servers()
servers = [s for s in b['servers'] if s['name'].startswith(name)]

if 'wait_until' in kwargs:
for server in servers:
try:
cls.servers_client.wait_for_server_status(
server['id'], kwargs['wait_until'])
except Exception as ex:
if ('preserve_server_on_error' not in kwargs
or kwargs['preserve_server_on_error'] is False):
for server in servers:
try:
cls.servers_client.delete_server(server['id'])
except Exception:
pass
raise ex

cls.servers.extend(servers)

return resp, body

由于创建虚拟机所有的参数都在参数**kwargs里,可惜此时是nose触发的函数执行,并没有其它函数的调用,因此就没有其它的参数传进来,也就是为空,那么继续看下调用真正的创建虚拟机函数create_server

def create_server(self, name, image_ref, flavor_ref, **kwargs):
"""
Creates an instance of a server.
name (Required): The name of the server.
image_ref (Required): Reference to the image used to build the server.
flavor_ref (Required): The flavor used to build the server.
Following optional keyword arguments are accepted:
adminPass: Sets the initial root password.
key_name: Key name of keypair that was created earlier.
meta: A dictionary of values to be used as metadata.
personality: A list of dictionaries for files to be injected into
the server.
security_groups: A list of security group dicts.
networks: A list of network dicts with UUID and fixed_ip.
user_data: User data for instance.
availability_zone: Availability zone in which to launch instance.
accessIPv4: The IPv4 access address for the server.
accessIPv6: The IPv6 access address for the server.
min_count: Count of minimum number of instances to launch.
max_count: Count of maximum number of instances to launch.
disk_config: Determines if user or admin controls disk configuration.
return_reservation_id: Enable/Disable the return of reservation id
"""
post_body = {
'name': name,
'imageRef': image_ref,
'flavorRef': flavor_ref
}

for option in ['personality', 'adminPass', 'key_name',
#'security_groups',
'networks', 'user_data',
'availability_zone', 'accessIPv4', 'accessIPv6',
'min_count', 'max_count', ('metadata', 'meta'),
('OS-DCF:diskConfig', 'disk_config'),
'return_reservation_id']:
if isinstance(option, tuple):
post_param = option[0]
key = option[1]
else:
post_param = option
key = option
value = kwargs.get(key)
if value is not None:
post_body[post_param] = value
if 'security_groups' in kwargs:
s_groups = [{'name': s_group}
for s_group in kwargs['security_groups']]
post_body['security_groups'] = s_groups
if kwargs.get('query', None):
s_hints = {'query': kwargs['query']}
post_body = json.dumps({'server': post_body,
'os:scheduler_hints': s_hints})
else:
post_body = json.dumps({'server': post_body})
resp, body = self.post('servers', post_body, self.headers)

body = json.loads(body)
# NOTE(maurosr): this deals with the case of multiple server create
# with return reservation id set True
if 'reservation_id' in body:
return resp, body
return resp, body['server']

在执行Restful API的POST里,创建虚拟机传入的参数在post_body里,默认有name,image,flavor三个参数,其它如果有从kwargs里传进来的,都append到post_body这个dict里

for option in ['personality', 'adminPass', 'key_name',
#'security_groups',
'networks', 'user_data',
'availability_zone', 'accessIPv4', 'accessIPv6',
'min_count', 'max_count', ('metadata', 'meta'),
('OS-DCF:diskConfig', 'disk_config'),
'return_reservation_id']:
if isinstance(option, tuple):
post_param = option[0]
key = option[1]
else:
post_param = option
key = option
value = kwargs.get(key)
if value is not None:
post_body[post_param] = value

一个都没有传入,所以只通过那三个参数来创建云主机,而缺少network导致出错,可以手动client执行一下,确认下结果

$ nova boot --flavor=1 --image=8812db39-e111-40f7-a57a-dd389a8f5cdc  test-network
ERROR: Multiple possible networks found, use a Network ID to be more specific. (HTTP 400)
(Request-ID: req-2846c588-5481-4120-a381-be630567e2f7)

错误信息一样,就印证了上面的错误结果,莫名其妙先执行一个错误的case,暂时未明原因

收工,两个疑点搞定,问题解决~!

 

发表回复