Skip to content

<fix>[host]: adaptive EMA-based ping timeout for large-scale clusters [ZSTAC-67534] #3391

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
MatheMatrix wants to merge 7 commits into
base: 5.5.12
Choose a base branch
from
+319 −14
Open

<fix>[host]: adaptive EMA-based ping timeout for large-scale clusters [ZSTAC-67534] #3391

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
20160c7
<fix>[host]: adaptive EMA-based ping timeout for large-scale clusters
AlanJager Feb 26, 2026
c8365ef
<fix>[host]: add GlobalConfig switch for adaptive ping timeout
AlanJager Feb 26, 2026
e55f1da
<fix>[kvm]: isolate P0 ping traffic to dedicated connection pool
AlanJager Feb 27, 2026
c9f2f22
<test>[core]: add unit tests for connection pool config and ping isol…
AlanJager Feb 27, 2026
5dd8f9c
<fix>[core]: add connection pool observability via DumpConnectionPool…
AlanJager Feb 27, 2026
b785677
<fix>[core]: add proactive pool health alarm (L1 built-in immune)
AlanJager Feb 27, 2026
91c1e02
<fix>[rest]: fix code-review issues: SSL CM registry, EMA leak, timeo…
AlanJager Feb 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions compute/src/main/java/org/zstack/compute/host/HostGlobalConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@ public class HostGlobalConfig {
public static GlobalConfig PING_HOST_INTERVAL = new GlobalConfig(CATEGORY, "ping.interval");
@GlobalConfigValidation(numberGreaterThan = 1)
public static GlobalConfig PING_HOST_TIMEOUT = new GlobalConfig(CATEGORY, "ping.timeout");
@GlobalConfigValidation
@GlobalConfigDef(defaultValue = "false", type = Boolean.class, description = "enable adaptive EMA-based ping timeout per host")
public static GlobalConfig PING_ADAPTIVE_TIMEOUT_ENABLED = new GlobalConfig(CATEGORY, "ping.adaptiveTimeout.enable");
@GlobalConfigValidation(numberGreaterThan = 0)
public static GlobalConfig MAXIMUM_PING_FAILURE = new GlobalConfig(CATEGORY, "ping.maxFailure");
@GlobalConfigValidation(numberGreaterThan = -1)
Expand Down
41 changes: 41 additions & 0 deletions compute/src/main/java/org/zstack/compute/host/HostTrackImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,39 @@ public class HostTrackImpl implements HostTracker, ManagementNodeChangeListener,
private static boolean alwaysStartRightNow = false;
private static final Cache<String, Long> skippedPingHostDeadline = CacheBuilder.newBuilder().expireAfterWrite(1, TimeUnit.HOURS).build();

// EMA adaptive timeout: tracks per-host exponential moving average of ping response times (in seconds)
private static final double EMA_ALPHA = 0.2;
private static final double EMA_SAFETY_FACTOR = 3.0;
private static final ConcurrentHashMap<String, Double> pingResponseEma = new ConcurrentHashMap<>();

/**
* Returns the ping timeout for the given host (in seconds).
* When adaptive timeout is enabled (host.ping.adaptiveTimeout.enable=true),
* uses EMA of observed response times * safety factor, floored by the configured value.
* When disabled, returns the configured ping.timeout directly.
*/
public static long getAdaptiveTimeout(String hostUuid) {
long configured = HostGlobalConfig.PING_HOST_TIMEOUT.value(Long.class);
if (!HostGlobalConfig.PING_ADAPTIVE_TIMEOUT_ENABLED.value(Boolean.class)) {
return configured;
}
Double ema = pingResponseEma.get(hostUuid);
if (ema == null) {
return configured;
}
long adaptive = (long) Math.ceil(ema * EMA_SAFETY_FACTOR);
return Math.max(configured, adaptive);
}

/**
* Updates the per-host EMA of ping response times using exponential moving average.
* New EMA = alpha * sample + (1 - alpha) * oldEMA.
*/
private static void updatePingResponseEma(String hostUuid, double responseTimeSec) {
pingResponseEma.merge(hostUuid, responseTimeSec,
(oldEma, sample) -> EMA_ALPHA * sample + (1 - EMA_ALPHA) * oldEma);
}

@Autowired
private DatabaseFacade dbf;
@Autowired
Expand Down Expand Up @@ -136,6 +169,7 @@ private void track() {
PingHostMsg msg = new PingHostMsg();
msg.setHostUuid(uuid);
bus.makeLocalServiceId(msg, HostConstant.SERVICE_ID);
final long pingStartMs = System.currentTimeMillis();
bus.send(msg, new CloudBusCallBack(null) {
Comment on lines +172 to 173
Copy link

@coderabbitai coderabbitai bot Feb 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

fd -t f -name "HostTrackImpl.java"

Repository: MatheMatrix/zstack

Length of output: 292

🏁 Script executed:

cat -n compute/src/main/java/org/zstack/compute/host/HostTrackImpl.java | sed -n '160,200p'

Repository: MatheMatrix/zstack

Length of output: 2717

🌐 Web query:

Java System.nanoTime() vs System.currentTimeMillis() elapsed time measurement best practice

💡 Result:

Best practice (elapsed time)

Use System.nanoTime() to measure elapsed time (durations, timeouts, benchmarking inside one JVM). It’s specifically intended for elapsed-time measurement and uses a fixed-but-arbitrary origin, so only differences are meaningful. It also has at least the resolution of currentTimeMillis() and is not tied to wall-clock time. [1]

long t0 = System.nanoTime();
// work
long elapsedNanos = System.nanoTime() - t0;

When to use System.currentTimeMillis()

Use System.currentTimeMillis() for wall-clock timestamps (e.g., logging, “what time is it?”, storing an epoch-millis timestamp). It returns milliseconds since 1970-01-01T00:00Z (UTC), but its granularity may be tens of milliseconds depending on OS, and it can jump due to clock adjustments. [1]

Prefer java.time for “time-of-day” APIs

For most application code, prefer Instant.now() (optionally via an injected Clock for testability) rather than calling currentTimeMillis() directly; Clock.millis() is defined as equivalent to System.currentTimeMillis(). [2]

Rule of thumb:

  • Elapsed time / timeouts: nanoTime() [1]
  • Current date/time timestamp: Instant.now() / Clock (or currentTimeMillis() if you truly need a primitive epoch-millis) [1][2]

Sources:
[1] Oracle JDK java.lang.System Javadoc (nanoTime/currentTimeMillis) (docs.oracle.com)
[2] Oracle JDK java.time.Clock Javadoc (docs.oracle.com)

Citations:

请改用单调时钟统计 RTT，避免 EMA 被系统时钟跳变污染

第 172 行与第 186 行使用 System.currentTimeMillis() 计算响应时间。当系统时钟调整（NTP 校时、时钟回拨）时，会产生负值或异常大值，直接污染 EMA 计算，导致自适应超时抖动或长期偏大。应改用 System.nanoTime()（单调时钟），它不受系统时钟调整影响。

建议修复 
-            final long pingStartMs = System.currentTimeMillis();
+            final long pingStartNs = System.nanoTime();
@@
-                    double responseTimeSec = (System.currentTimeMillis() - pingStartMs) / 1000.0;
+                    double responseTimeSec = Math.max(0D,
+                            (System.nanoTime() - pingStartNs) / 1_000_000_000.0);
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@compute/src/main/java/org/zstack/compute/host/HostTrackImpl.java` around
lines 172 - 173, The RTT measurement in HostTrackImpl currently uses
System.currentTimeMillis() (pingStartMs) before sending the bus message and
again when computing RTT in the CloudBusCallBack; switch to a monotonic clock by
using System.nanoTime() for start and end deltas (e.g., store pingStartNs from
System.nanoTime(), compute deltaNs = System.nanoTime() - pingStartNs) and
convert to milliseconds for the EMA/timeout logic (deltaMs = deltaNs /
1_000_000L); update both the send-site (where pingStartMs is set) and the
callback-site (where RTT is calculated and fed into the EMA/timeout) to use the
new variables so EMA and adaptive timeout code in HostTrackImpl remain monotonic
and immune to system clock adjustments.

@Override
public void run(MessageReply reply) {
Expand All @@ -148,6 +182,12 @@ private ReconnectDecision makeReconnectDecision(MessageReply reply) {
return ReconnectDecision.DoNothing;
}

// update EMA with observed response time on successful ping
double responseTimeSec = (System.currentTimeMillis() - pingStartMs) / 1000.0;
updatePingResponseEma(uuid, responseTimeSec);
logger.debug(String.format("[Host Tracker]: host[uuid:%s] ping response time %.2fs, adaptive timeout %ds",
uuid, responseTimeSec, getAdaptiveTimeout(uuid)));

PingHostReply r = reply.castReply();
if (r.isNoReconnect()) {
return ReconnectDecision.DoNothing;
Expand Down Expand Up @@ -280,6 +320,7 @@ public void untrackHost(String huuid) {
t.cancel();
}
trackers.remove(huuid);
pingResponseEma.remove(huuid);
logger.debug(String.format("stop tracking host[uuid:%s]", huuid));
}

Expand Down
7 changes: 7 additions & 0 deletions conf/globalConfig/host.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,13 @@
<defaultValue>10</defaultValue>
<type>java.lang.Integer</type>
</config>
<config>
<category>host</category>
<name>ping.adaptiveTimeout.enable</name>
<description>Enable adaptive EMA-based ping timeout that adjusts per host based on observed response times. When disabled, the fixed ping.timeout value is used.</description>
<defaultValue>false</defaultValue>
<type>java.lang.Boolean</type>
</config>
<config>
<category>host</category>
<name>maintenanceMode.ignoreError</name>
Expand Down
6 changes: 4 additions & 2 deletions core/src/main/java/org/zstack/core/CoreGlobalProperty.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,10 +44,12 @@ public class CoreGlobalProperty {
public static int REST_FACADE_CONNECT_TIMEOUT;
@GlobalProperty(name = "RESTFacade.echoTimeout", defaultValue = "60")
public static int REST_FACADE_ECHO_TIMEOUT;
@GlobalProperty(name = "RESTFacade.maxPerRoute", defaultValue = "2")
@GlobalProperty(name = "RESTFacade.maxPerRoute", defaultValue = "50")
public static int REST_FACADE_MAX_PER_ROUTE;
@GlobalProperty(name = "RESTFacade.maxTotal", defaultValue = "128")
@GlobalProperty(name = "RESTFacade.maxTotal", defaultValue = "2048")
public static int REST_FACADE_MAX_TOTAL;
@GlobalProperty(name = "RESTFacade.connectionRequestTimeout", defaultValue = "8000")
public static int REST_FACADE_CONNECTION_REQUEST_TIMEOUT;
/**
* When set RestServer.maskSensitiveInfo to true, sensitive info will be
* masked see @NoLogging.
Expand Down
7 changes: 6 additions & 1 deletion core/src/main/java/org/zstack/core/cloudbus/CloudBusImpl3.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,12 @@ public class CloudBusImpl3 implements CloudBus, CloudBusIN {
private final Map<String, EndPoint> endPoints = new HashMap<>();
private final Map<String, Envelope> envelopes = new ConcurrentHashMap<>();
private final Map<String, java.util.function.Consumer> messageConsumers = new ConcurrentHashMap<>();
private final static TimeoutRestTemplate http = RESTFacade.createRestTemplate(CoreGlobalProperty.REST_FACADE_READ_TIMEOUT, CoreGlobalProperty.REST_FACADE_CONNECT_TIMEOUT);
// R1+R3: explicit pool params matching chain queue capacity (CloudBusGlobalProperty.HTTP_MAX_CONN)
private final static TimeoutRestTemplate http = RESTFacade.createRestTemplate(
CoreGlobalProperty.REST_FACADE_READ_TIMEOUT,
CoreGlobalProperty.REST_FACADE_CONNECT_TIMEOUT,
CloudBusGlobalProperty.HTTP_MAX_CONN,
10);

public static final String HTTP_BASE_URL = "/cloudbus";
public static final FutureCompletion SEND_CONFIRMED = new FutureCompletion(null);
Expand Down
114 changes: 109 additions & 5 deletions core/src/main/java/org/zstack/core/rest/RESTFacadeImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
import org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor;
import org.apache.http.impl.nio.reactor.IOReactorConfig;
import org.apache.http.nio.reactor.IOReactorException;
import org.apache.http.pool.PoolStats;
import org.apache.logging.log4j.ThreadContext;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.*;
Expand All @@ -30,6 +31,7 @@
import org.zstack.core.telemetry.TelemetryGlobalProperty;
import org.zstack.core.thread.AsyncThread;
import org.zstack.core.thread.CancelablePeriodicTask;
import org.zstack.core.thread.PeriodicTask;
import org.zstack.core.thread.ThreadFacade;
import org.zstack.core.thread.ThreadFacadeImpl.TimeoutTaskReceipt;
import org.zstack.core.timeout.ApiTimeoutManager;
Expand Down Expand Up @@ -79,6 +81,15 @@ public class RESTFacadeImpl implements RESTFacade {
private String callbackUrl;
private TimeoutRestTemplate template;
private AsyncRestTemplate asyncRestTemplate;
// P0: dedicated ping pool — isolated from business traffic (R2)
private AsyncRestTemplate pingAsyncRestTemplate;
// R5: store connection managers for DumpConnectionPoolStatus observability
private PoolingNHttpClientConnectionManager asyncConnManager;
private PoolingNHttpClientConnectionManager pingConnManager;
// ThreadLocal allows asyncJsonPostForPing() to inject the ping template without changing asyncJson() signature
private static final ThreadLocal<AsyncRestTemplate> ASYNC_TEMPLATE_OVERRIDE = new ThreadLocal<>();
// R2: P0 ping pool capacity — covers largest expected cluster (3000 nodes)
private static final int PING_POOL_MAX_TOTAL = 3000;
private String baseUrl;
private String sendCommandUrl;
private String callbackHostName;
Expand Down Expand Up @@ -182,6 +193,28 @@ void init() {
logger.debug(sb.toString());
});

// R5: pool observability — trigger via: kill -USR2 <pid> or zstack-ctl dump_connection_pool
DebugManager.registerDebugSignalHandler("DumpConnectionPoolStatus", () -> {
StringBuilder sb = new StringBuilder();
sb.append("\n============= BEGIN: Connection Pool Status (R5) =================\n");
if (asyncConnManager != null) {
PoolStats s = asyncConnManager.getTotalStats();
sb.append(String.format("async-pool : maxTotal=%-5d leased=%-5d available=%-5d pending=%d%n",
CoreGlobalProperty.REST_FACADE_MAX_TOTAL, s.getLeased(), s.getAvailable(), s.getPending()));
} else {
sb.append("async-pool : not initialized\n");
}
if (pingConnManager != null) {
PoolStats s = pingConnManager.getTotalStats();
sb.append(String.format("ping-pool(P0): maxTotal=%-5d leased=%-5d available=%-5d pending=%d%n",
pingConnManager.getMaxTotal(), s.getLeased(), s.getAvailable(), s.getPending()));
} else {
sb.append("ping-pool(P0): not initialized\n");
}
sb.append("============= END: Connection Pool Status ========================\n");
logger.debug(sb.toString());
});

port = Platform.getManagementNodeServicePort();

IptablesUtils.insertRuleToFilterTable(String.format("-A INPUT -p tcp -m state --state NEW -m tcp --dport %s -j ACCEPT", port));
Expand Down Expand Up @@ -211,15 +244,60 @@ void init() {

logger.debug(String.format("RESTFacade built callback url: %s", callbackUrl));
template = RESTFacade.createRestTemplate(CoreGlobalProperty.REST_FACADE_READ_TIMEOUT, CoreGlobalProperty.REST_FACADE_CONNECT_TIMEOUT);
// R5: capture connection managers for DumpConnectionPoolStatus observability
PoolingNHttpClientConnectionManager[] cmRef = new PoolingNHttpClientConnectionManager[1];
asyncRestTemplate = createAsyncRestTemplate(
CoreGlobalProperty.REST_FACADE_READ_TIMEOUT,
CoreGlobalProperty.REST_FACADE_CONNECT_TIMEOUT,
CoreGlobalProperty.REST_FACADE_MAX_TOTAL,
CoreGlobalProperty.REST_FACADE_MAX_PER_ROUTE,
CoreGlobalProperty.REST_FACADE_MAX_TOTAL);
cmRef);
asyncConnManager = cmRef[0];
// P0 ping pool: maxPerRoute=2 (1 in-flight + 1 queued per host), maxTotal covers full cluster
pingAsyncRestTemplate = createAsyncRestTemplate(
10000, // readTimeout = ping timeout (10s)
3000, // connectTimeout = fast fail (3s)
PING_POOL_MAX_TOTAL, // maxTotal: full cluster capacity
2, // maxPerRoute: 2 allows 1 in-flight + 1 queued, avoids head-of-line block
cmRef);
pingConnManager = cmRef[0];

// L1 built-in immune: proactive pool utilization alarm every 60s (R5)
thdf.submitPeriodicTask(new PeriodicTask() {
@Override public TimeUnit getTimeUnit() { return TimeUnit.SECONDS; }
@Override public long getInterval() { return 60; }
@Override public String getName() { return "http-pool-health-check"; }

@Override
public void run() {
if (asyncConnManager != null) {
PoolStats s = asyncConnManager.getTotalStats();
long maxTotal = CoreGlobalProperty.REST_FACADE_MAX_TOTAL;
if (s.getPending() > 0 || s.getLeased() * 5 > maxTotal * 4) {
logger.warn(String.format("[POOL-ALARM] async-pool HIGH: leased=%d available=%d pending=%d maxTotal=%d",
s.getLeased(), s.getAvailable(), s.getPending(), maxTotal));
}
}
if (pingConnManager != null) {
PoolStats s = pingConnManager.getTotalStats();
if (s.getPending() > 0) {
logger.warn(String.format("[POOL-ALARM] ping-pool(P0) CONGESTED: leased=%d available=%d pending=%d — P0 ping may be delayed",
s.getLeased(), s.getAvailable(), s.getPending()));
}
}
}
});
}

// timeout are in milliseconds; delegates to 5-param overload (backward compat)
// Parameter order matches createRestTemplate: (readTimeout, connectTimeout, maxTotal, maxPerRoute)
private static AsyncRestTemplate createAsyncRestTemplate(int readTimeout, int connectTimeout, int maxTotal, int maxPerRoute) {
return createAsyncRestTemplate(readTimeout, connectTimeout, maxTotal, maxPerRoute, null);
}

// timeout are in milliseconds
private static AsyncRestTemplate createAsyncRestTemplate(int readTimeout, int connectTimeout, int maxPerRoute, int maxTotal) {
// R5: outCm captures the connection manager for observability (DumpConnectionPoolStatus)
private static AsyncRestTemplate createAsyncRestTemplate(int readTimeout, int connectTimeout, int maxTotal, int maxPerRoute,
PoolingNHttpClientConnectionManager[] outCm) {
PoolingNHttpClientConnectionManager connectionManager;
try {
connectionManager = new PoolingNHttpClientConnectionManager(new DefaultConnectingIOReactor(IOReactorConfig.DEFAULT));
Expand All @@ -229,6 +307,7 @@ private static AsyncRestTemplate createAsyncRestTemplate(int readTimeout, int co

connectionManager.setDefaultMaxPerRoute(maxPerRoute);
connectionManager.setMaxTotal(maxTotal);
if (outCm != null) outCm[0] = connectionManager;

CloseableHttpAsyncClient httpAsyncClient = HttpAsyncClients.custom()
.setConnectionManager(connectionManager)
Expand All @@ -237,7 +316,8 @@ private static AsyncRestTemplate createAsyncRestTemplate(int readTimeout, int co
HttpComponentsAsyncClientHttpRequestFactory cf = new HttpComponentsAsyncClientHttpRequestFactory(httpAsyncClient);
cf.setConnectTimeout(connectTimeout);
cf.setReadTimeout(readTimeout);
cf.setConnectionRequestTimeout(connectTimeout * 2);
// R4: connectionRequestTimeout must be < operation timeout (e.g. ping timeout=10s)
cf.setConnectionRequestTimeout(Math.min(connectTimeout * 2, CoreGlobalProperty.REST_FACADE_CONNECTION_REQUEST_TIMEOUT));

AsyncRestTemplate asyncRestTemplate = new AsyncRestTemplate(cf);
RESTFacade.setMessageConverter(asyncRestTemplate.getMessageConverters());
Expand Down Expand Up @@ -574,7 +654,9 @@ public void success(HttpEntity<String> responseEntity) {
logger.trace(String.format("json %s [%s], %s", method.toString(), url, req));
}

ListenableFuture<ResponseEntity<String>> f = asyncRestTemplate.exchange(url, method, req, String.class);
AsyncRestTemplate override = ASYNC_TEMPLATE_OVERRIDE.get();
AsyncRestTemplate tmpl = override != null ? override : asyncRestTemplate;
ListenableFuture<ResponseEntity<String>> f = tmpl.exchange(url, method, req, String.class);
f.addCallback(rsp -> {}, e -> wrapper.fail(err(ORG_ZSTACK_CORE_REST_10003, SysErrors.HTTP_ERROR, e.getLocalizedMessage())));
} catch (RestClientException e) {
logger.warn(String.format("Unable to %s to %s: %s", method.toString(), url, e.getMessage()));
Expand All @@ -600,6 +682,28 @@ public void asyncJsonPost(String url, String body, AsyncRESTCallback callback) {
asyncJsonPost(url, body, callback, TimeUnit.MILLISECONDS, timeout);
}

@Override
public void asyncJsonPostForPing(String url, Object body, AsyncRESTCallback callback) {
// R2: inject dedicated ping pool via ThreadLocal; cleared in finally to prevent leaks
ASYNC_TEMPLATE_OVERRIDE.set(pingAsyncRestTemplate);
try {
asyncJsonPost(url, body, callback);
} finally {
ASYNC_TEMPLATE_OVERRIDE.remove();
}
Comment on lines +685 to +693
Copy link

@coderabbitai coderabbitai bot Feb 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

ThreadLocal 覆盖值在嵌套调用下会被误清理。

Line 622 直接 remove() 会在“外层已设置覆盖值、内层再次调用”时把外层状态也清掉。建议保存旧值并在 finally 恢复，避免嵌套调用时模板选择被破坏。

建议修复 
 `@Override`
 public void asyncJsonPostForPing(String url, Object body, AsyncRESTCallback callback) {
-    ASYNC_TEMPLATE_OVERRIDE.set(pingAsyncRestTemplate);
+    AsyncRestTemplate previous = ASYNC_TEMPLATE_OVERRIDE.get();
+    ASYNC_TEMPLATE_OVERRIDE.set(pingAsyncRestTemplate);
     try {
         asyncJsonPost(url, body, callback);
     } finally {
-        ASYNC_TEMPLATE_OVERRIDE.remove();
+        if (previous == null) {
+            ASYNC_TEMPLATE_OVERRIDE.remove();
+        } else {
+            ASYNC_TEMPLATE_OVERRIDE.set(previous);
+        }
     }
 }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
@Override
public void asyncJsonPostForPing(String url, Object body, AsyncRESTCallback callback) {
// R2: inject dedicated ping pool via ThreadLocal; cleared in finally to prevent leaks
ASYNC_TEMPLATE_OVERRIDE.set(pingAsyncRestTemplate);
try {
asyncJsonPost(url, body, callback);
} finally {
ASYNC_TEMPLATE_OVERRIDE.remove();
}
`@Override`
public void asyncJsonPostForPing(String url, Object body, AsyncRESTCallback callback) {
AsyncRestTemplate previous = ASYNC_TEMPLATE_OVERRIDE.get();
ASYNC_TEMPLATE_OVERRIDE.set(pingAsyncRestTemplate);
try {
asyncJsonPost(url, body, callback);
} finally {
if (previous == null) {
ASYNC_TEMPLATE_OVERRIDE.remove();
} else {
ASYNC_TEMPLATE_OVERRIDE.set(previous);
}
}
}
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@core/src/main/java/org/zstack/core/rest/RESTFacadeImpl.java` around lines 615
- 623, The ThreadLocal override in asyncJsonPostForPing currently calls
ASYNC_TEMPLATE_OVERRIDE.remove() which clears any outer nested value; change
asyncJsonPostForPing to save the previous value (e.g. a local variable old =
ASYNC_TEMPLATE_OVERRIDE.get()), set ASYNC_TEMPLATE_OVERRIDE to
pingAsyncRestTemplate, call asyncJsonPost(url, body, callback) in the try, and
in finally restore the previous value (if old is null call remove() else
ASYNC_TEMPLATE_OVERRIDE.set(old)) so nested callers keep their template; update
references to ASYNC_TEMPLATE_OVERRIDE, pingAsyncRestTemplate, and asyncJsonPost
accordingly.

}

@Override
public void asyncJsonPostForPing(String url, Object body, AsyncRESTCallback callback, TimeUnit unit, long timeout) {
// R2: inject dedicated ping pool via ThreadLocal; cleared in finally to prevent leaks
ASYNC_TEMPLATE_OVERRIDE.set(pingAsyncRestTemplate);
try {
asyncJsonPost(url, body, callback, unit, timeout);
} finally {
ASYNC_TEMPLATE_OVERRIDE.remove();
}
}

@Override
public HttpEntity<String> httpServletRequestToHttpEntity(HttpServletRequest req) {
try {
Expand Down
Loading