吐血定位端口映射影响JMeter分布式压测的异常问题

线下压测平台联调,测试都挺顺利的,因为平台部署,以及JMeter的master和slave节点都在同一个环境,不存在任何网络问题,可今天折腾发布环境,的确是和我想得不太一样

要求是平台的一些数据,入库都在环境A,而环境B是是一个压测环境,专门可以部署被压测服务的,而一开始没想压测服务以及压力机还能跨环境的,整体开发考虑到master节点只是起一个控制调度的作用,负载要求还好,因此将压测平台和master节点放在同一个节点来考虑的,那么剩下的网络需要互通的就只有master和slave之间互通就行了

UntitledImage

正常如果是同一个环境,直接master和所有slave互通,就不需要中间一堆的端口映射,而如今本身环境A和B是不通的,因此他们之间的交互假如无法直接路由网络打通,那就只有端口映射

其实如果考虑将平台服务部署在A,将master的相关jmeter东西放在环境B和slave一起,就不需要考虑他们之间交互,只需要平台服务远程调用master节点压测命令即可,当然这是事后诸葛

言归正传,现在主要是为了说明问题,定位下原因;上图的大致流程就是将不通的网络通过端口映射打通,就不多说了

还好以前做过网络,找各个环境运维按我要求打通之后,特意抓包测试了一下像tcp连接,报文等等,确认无误

但是,现在的问题是,我已经确认网络连接无误,但是在分布式压测的时候,master指定了映射的IP1:PORT1,却无法打到slave节点,错误原因也很奇葩,说连不上IP_SLAVE1:PORT_SLAVE1,这就奇怪了,因为本来master就连不上slave节点,因此才做的端口映射,只需要将包打到IP1:PORT1,就会自动转发到IP_SLAVE1:PORT_SLAVE1,况且我master启动都没指定IP1:PORT1,哪里来的这种报错信息,难不成还动态查找哪些启动了

文字描述太难看了,直接来个例子说明,端口映射11.22.33.44:55555 => 1.2.3.4.1099

slave节点

$ ./jmeter-server -Djava.rmi.server.hostname=1.2.3.4
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/docker/logs/nps/tuya-jmeter/lib/log4j-slf4j-impl-2.13.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/docker/logs/nps/tuya-jmeter/lib/ext/jmeter-plugins-dubbo-2.7.8-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Using local port: 1099
Created remote object: UnicastServerRef2 [liveRef: [endpoint:[1.2.3.4:1099](local),objID:[-1a3fbb9b:1771fc07d74:-7fff, 134469626239589308]]]

 

master节点

$ ./jmeter -n -t baidu.jmx -R 11.22.33.44:55555 -l my.jtl
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/mnt/logs/apps/nbest-perf-service/nps/tuya-jmeter/lib/log4j-slf4j-impl-2.13.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/mnt/logs/apps/nbest-perf-service/nps/tuya-jmeter/lib/ext/jmeter-plugins-dubbo-2.7.8-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Creating summariser
Created the tree successfully using baidu.jmx
Configuring remote engine: 11.22.33.44:55555
Starting distributed test with remote engines: [11.22.33.44:55555] @ Wed Jan 20 20:56:37 CST 2021 (1611147397067)



Error in rconfigure() method java.rmi.ConnectException: Connection refused to host: 1.2.3.4; nested exception is:
	java.net.ConnectException: Connection refused (Connection refused)
Remote engines have been started:[]
The following remote engines have not started:[11.22.33.44:55555]
Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445

最奇怪的就是Error那行,Connection refused to host: 1.2.3.4,我都没指定过1.2.3.4,唯一的来源就是master探测到了slave节点jmeter-server进程起来了,并且指定了remote ip是1.2.3.4,但是因为这个ip是无法访问成功的,说明指定的-R 11.22.33.44:55555映射生效了,可是端口映射生效了应该就直接转发到1.2.3.4:1099才对,为什么会报连不上呢,各种猜测,网上也没答案,只有看下JMeter源码,具体的报错信息了,这也是第一次看源代码

首先是rconfigure()这个方法报错,调用了一个rmi超时了,理解一下应该是master节点远程调用slave节点进行数据交互,不通超时;先就不用管了,正好下面有一行,执行的engine为空:Remote engines have been started:[]

/**
* Starts a remote testing engines
*
* @param addresses list of the DNS names or IP addresses of the remote testing engines
*/
@SuppressWarnings("JdkObsolete")
public void start(List<String> addresses) {
long now = System.currentTimeMillis();
println("Starting distributed test with remote engines: " + addresses + " @ " + new Date(now) + " (" + now + ")");
List<String> startedEngines = new ArrayList<>(addresses.size());
List<String> failedEngines = new ArrayList<>(addresses.size());
for (String address : addresses) {
JMeterEngine engine = engines.get(address);
try {
if (engine != null) {
engine.runTest();
startedEngines.add(address);
} else {
log.warn(HOST_NOT_FOUND_MESSAGE, address);
failedEngines.add(address);
}
} catch (IllegalStateException | JMeterEngineException e) { // NOSONAR already reported to user
failedEngines.add(address);
JMeterUtils.reportErrorToUser(e.getMessage(), JMeterUtils.getResString("remote_error_starting")); // $NON-NLS-1$
}
}
println("Remote engines have been started:" + startedEngines);
if (!failedEngines.isEmpty()) {
errln("The following remote engines have not started:" + failedEngines);
}
}

接着看,这里的startedEngines为空,说明engine为空,而engines是一个map

private final Map<String, JMeterEngine> engines = new HashMap<>();

看下他在哪里有更新

public void init(List<String> addresses, HashTree tree) {
// converting list into mutable version
List<String> addrs = new ArrayList<>(addresses);

for (int tryNo = 0; tryNo < retriesNumber; tryNo++) {
if (tryNo > 0) {
println("Following remote engines will retry configuring: " + addrs+", pausing before retry for " + retriesDelay + "ms");
try {
Thread.sleep(retriesDelay);
} catch (InterruptedException e) { // NOSONAR
throw new IllegalStateException("Interrupted while initializing remote engines:"+addrs, e);
}
}

int idx = 0;
while (idx < addrs.size()) {
String address = addrs.get(idx);
println("Configuring remote engine: " + address);
JMeterEngine engine = getClientEngine(address.trim(), tree);
if (engine != null) {
engines.put(address, engine);
addrs.remove(address);
} else {
println("Failed to configure " + address);
idx++;
}
}

if (addrs.isEmpty()) {
break;
}
}

if (!addrs.isEmpty()) {
String msg = "Following remote engines could not be configured:" + addrs;
if (!continueOnFail || engines.isEmpty()) {
stop();
throw new RuntimeException(msg); // NOSONAR
} else {
println(msg);
println("Continuing without failed engines...");
}
}
}

其中下面这行,在报错的地方有打印出address就是我们配置的ip和port

println("Configuring remote engine: " + address);

而下面这行,有打印这个ip和port没有configured

String msg = "Following remote engines could not be configured:" + addrs;

说明问题在addrs不为空,那么engines这里为空,说明engine为空,因此急需看getClientEngine

private JMeterEngine getClientEngine(String address, HashTree testTree) {
JMeterEngine engine;
try {
engine = createEngine(address);
engine.configure(testTree);
if (!remoteProps.isEmpty()) {
engine.setProperties(remoteProps);
}
return engine;
} catch (Exception ex) {
log.error("Failed to create engine at {}", address, ex);
JMeterUtils.reportErrorToUser(ex.getMessage(),
JMeterUtils.getResString("remote_error_init") + ": " + address); // $NON-NLS-1$ $NON-NLS-2$
return null;
}
}

既然是返回null,那应该是抛异常了,因此继续看createEngine

/**
* A factory method that might be overridden for unit testing
*
* @param address address for engine
* @return engine instance
* @throws RemoteException if registry can't be contacted
* @throws NotBoundException when name for address can't be found
*/
protected JMeterEngine createEngine(String address) throws RemoteException, NotBoundException {
return new ClientJMeterEngine(address);
}

继续

public ClientJMeterEngine(String hostAndPort) throws NotBoundException, RemoteException {
this.remote = getEngine(hostAndPort);
this.hostAndPort = hostAndPort;
}

hostAndPort已经确认有值了,那就看getEngine

private static RemoteJMeterEngine getEngine(String hostAndPort)
throws RemoteException, NotBoundException {
final String name = RemoteJMeterEngineImpl.JMETER_ENGINE_RMI_NAME; // $NON-NLS-1$ $NON-NLS-2$
String host = hostAndPort;
int port = RmiUtils.DEFAULT_RMI_PORT;
int indexOfSeparator = hostAndPort.indexOf(':');
if (indexOfSeparator >= 0) {
host = hostAndPort.substring(0, indexOfSeparator);
String portAsString = hostAndPort.substring(indexOfSeparator+1);
port = Integer.parseInt(portAsString);
}
Registry registry = LocateRegistry.getRegistry(
host,
port,
RmiUtils.createClientSocketFactory());
Remote remobj = registry.lookup(name);
if (remobj instanceof RemoteJMeterEngine){
final RemoteJMeterEngine rje = (RemoteJMeterEngine) remobj;
if (remobj instanceof RemoteObject){
RemoteObject robj = (RemoteObject) remobj;
System.out.println("Using remote object: "+robj.getRef().remoteToString()); // NOSONAR
}
return rje;
}
throw new RemoteException("Could not find "+name);
}

可以看到这里是远程调用,最终master就是这样来控制slave节点进行一个分布式压测,lookup查找获取slave远程注册表对象,因此问题可能就是没有获取到,或者是由于网络问题没有查找到

那么问题其实就在slave节点,其实我启动jmeter-server的时候已经制定了-Djava.rmi.server.hostname=,但是由于这里是端口映射,因此这个IP制定的是中转IP和端口,那么可能导致jmeter这种分布式压测方式在某个地方没有完全打通,试过将所有隐射端口全部开放,还是报这个错误,而且master和slave在同一个环境网络下,是没有这个问题的,因此应该还是这种网络打通方式的问题,暂时没有进一步研究实质原因是啥,有空再继续研究下

发表评论