[WIP] Add JFR scrubbing before profile upload by jbachorik · Pull Request #10577 · DataDog/dd-trace-java

jbachorik · 2026-02-12T13:24:26Z

What Does This Do

Adds configurable scrubbing of sensitive fields in JFR profiling recordings before upload to the backend. Sensitive data (system properties, JVM arguments, environment variables, command lines) is replaced with 'x' characters using a two-pass scrubbing algorithm based on jafar-parser.

Motivation

Prevents accidental exposure of sensitive information (credentials, API keys, internal paths) in profiling data uploaded to Datadog backend. Provides opt-in security enhancement with fail-closed default behavior.

Additional Notes

Implementation

New profiling-scrubber module with JFR scrubbing implementation
Configuration via dd.profiling.scrub.enabled (default: false)
Fail-closed by default; configurable fail-open mode available
File-backed optimization for ddprof recordings (avoids re-materialization)
Scrubs: jdk.InitialSystemProperty, jdk.JVMInformation, jdk.InitialEnvironmentVariable, jdk.SystemProcess
Smoke test validates end-to-end functionality

GraalVM Native-Image Compatibility Fix

Fixed "Substitution target not loaded" errors that occurred when building native images with the agent attached and profiling enabled. The issue was triggered by adding profiling-scrubber to the agent classpath.

Root cause (unclear): When the agent is attached during native-image compilation (-J-javaagent:dd-java-agent.jar), GraalVM's native-image compiler discovers JMXFetch substitutions even for applications that don't use JMXFetch, resulting in errors because the target classes aren't available.

Solution: Made JMXFetch substitutions conditional on JMXFetch presence by checking at runtime whether org.datadog.jmxfetch.App is on the classpath before registering substitutions. This works around GraalVM 20.0's lack of the @TargetClass(onlyWith=...) field (available in GraalVM 21+).

Files changed:

AnnotationSubstitutionProcessorInstrumentation.java - Added runtime check before registering JMXFetch substitutions
GraalNativeImageModule.java - Made helper class names conditional on JMXFetch presence

Test Coverage

82.15% instruction coverage, 72.37% branch coverage
76 comprehensive tests across 10 test suites:
- Varint encoding/decoding with boundary values
- File operations (buffer edge cases: <64KB, >64KB, empty, exact)
- Guard functions (conditional/unconditional scrubbing)
- Error handling (file I/O failures, permissions)
- Complex scenarios (multiple fields, missing metadata, parsability)
- Property-based fuzzing (100+ iterations per test)
- Integration tests with real 3.6MB JFR file
Coverage thresholds enforced (70%/60%) with documented exclusions for untestable code

Contributor Checklist

Format the title according to the contribution guidelines
Assign the `type:` and (`comp:` or `inst:`) labels in addition to any other useful labels
Avoid using `close`, `fix`, or any linking keywords when referencing an issue
Use `solves` instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, migration, or deletion
Update public documentation with any new configuration flags or behaviors

Jira ticket: PROF-13548

pr-commenter · 2026-02-12T14:26:26Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	jb/jfr_redacting
git_commit_date	1770902831	1771103311
git_commit_sha	`2ebc964`	`2dc7b4b`
release_version	1.60.0-SNAPSHOT~2ebc964340	1.60.0-SNAPSHOT~2dc7b4ba83

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1771105312	1771105312
ci_job_id	1427439032	1427439032
ci_pipeline_id	96616345	96616345
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-6o6m71ab 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-6o6m71ab 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 2 performance improvements and 2 performance regressions! Performance is the same for 61 metrics, 6 unstable metrics.

scenario	Δ mean execution_time	candidate mean execution_time	baseline mean execution_time
scenario:startup:insecure-bank:tracing:Debugger	worse [+2.429ms; +8.183ms] or [+3.973%; +13.384%]	66.447ms	61.141ms
scenario:startup:petclinic:appsec:Debugger	better [-10.845ms; -5.956ms] or [-16.117%; -8.852%]	58.888ms	67.288ms
scenario:startup:petclinic:appsec:Telemetry	better [-706.676µs; -327.727µs] or [-7.233%; -3.355%]	9.253ms	9.770ms
scenario:startup:petclinic:tracing:Debugger	worse [+5.151ms; +8.048ms] or [+8.369%; +13.074%]	68.155ms	61.555ms

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.60.0-SNAPSHOT~2dc7b4ba83, baseline=1.60.0-SNAPSHOT~2ebc964340

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.062 s) : 0, 1062473
Total [baseline] (8.753 s) : 0, 8753042
Agent [candidate] (1.072 s) : 0, 1072488
Total [candidate] (8.781 s) : 0, 8780633
section iast
Agent [baseline] (1.23 s) : 0, 1229998
Total [baseline] (9.332 s) : 0, 9331830
Agent [candidate] (1.236 s) : 0, 1236282
Total [candidate] (9.343 s) : 0, 9343461

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.062 s	-
Agent	iast	1.23 s	167.525 ms (15.8%)
Total	tracing	8.753 s	-
Total	iast	9.332 s	578.788 ms (6.6%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.072 s	-
Agent	iast	1.236 s	163.794 ms (15.3%)
Total	tracing	8.781 s	-
Total	iast	9.343 s	562.828 ms (6.4%)

gantt
    title insecure-bank - break down per module: candidate=1.60.0-SNAPSHOT~2dc7b4ba83, baseline=1.60.0-SNAPSHOT~2ebc964340

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.182 ms) : 0, 1182
crashtracking [candidate] (1.224 ms) : 0, 1224
BytebuddyAgent [baseline] (628.143 ms) : 0, 628143
BytebuddyAgent [candidate] (632.122 ms) : 0, 632122
AgentMeter [baseline] (28.843 ms) : 0, 28843
AgentMeter [candidate] (29.193 ms) : 0, 29193
GlobalTracer [baseline] (256.968 ms) : 0, 256968
GlobalTracer [candidate] (259.576 ms) : 0, 259576
AppSec [baseline] (32.699 ms) : 0, 32699
AppSec [candidate] (33.184 ms) : 0, 33184
Debugger [baseline] (61.141 ms) : 0, 61141
Debugger [candidate] (66.447 ms) : 0, 66447
Remote Config [baseline] (625.586 µs) : 0, 626
Remote Config [candidate] (621.296 µs) : 0, 621
Telemetry [baseline] (13.733 ms) : 0, 13733
Telemetry [candidate] (10.804 ms) : 0, 10804
Flare Poller [baseline] (3.833 ms) : 0, 3833
Flare Poller [candidate] (3.868 ms) : 0, 3868
section iast
crashtracking [baseline] (1.168 ms) : 0, 1168
crashtracking [candidate] (1.189 ms) : 0, 1189
BytebuddyAgent [baseline] (794.921 ms) : 0, 794921
BytebuddyAgent [candidate] (800.251 ms) : 0, 800251
AgentMeter [baseline] (11.235 ms) : 0, 11235
AgentMeter [candidate] (11.482 ms) : 0, 11482
GlobalTracer [baseline] (247.612 ms) : 0, 247612
GlobalTracer [candidate] (249.248 ms) : 0, 249248
IAST [baseline] (26.936 ms) : 0, 26936
IAST [candidate] (27.67 ms) : 0, 27670
AppSec [baseline] (33.898 ms) : 0, 33898
AppSec [candidate] (34.558 ms) : 0, 34558
Debugger [baseline] (66.106 ms) : 0, 66106
Debugger [candidate] (63.872 ms) : 0, 63872
Remote Config [baseline] (535.433 µs) : 0, 535
Remote Config [candidate] (539.032 µs) : 0, 539
Telemetry [baseline] (8.743 ms) : 0, 8743
Telemetry [candidate] (8.598 ms) : 0, 8598
Flare Poller [baseline] (3.553 ms) : 0, 3553
Flare Poller [candidate] (3.532 ms) : 0, 3532

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.60.0-SNAPSHOT~2dc7b4ba83, baseline=1.60.0-SNAPSHOT~2ebc964340

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.066 s) : 0, 1065794
Total [baseline] (10.89 s) : 0, 10889970
Agent [candidate] (1.065 s) : 0, 1064877
Total [candidate] (10.792 s) : 0, 10792475
section appsec
Agent [baseline] (1.244 s) : 0, 1244024
Total [baseline] (10.963 s) : 0, 10962936
Agent [candidate] (1.236 s) : 0, 1236028
Total [candidate] (10.819 s) : 0, 10818736
section iast
Agent [baseline] (1.231 s) : 0, 1230671
Total [baseline] (11.165 s) : 0, 11165328
Agent [candidate] (1.234 s) : 0, 1234414
Total [candidate] (11.241 s) : 0, 11241257
section profiling
Agent [baseline] (1.199 s) : 0, 1198732
Total [baseline] (11.03 s) : 0, 11030471
Agent [candidate] (1.192 s) : 0, 1191528
Total [candidate] (10.862 s) : 0, 10862449

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.066 s	-
Agent	appsec	1.244 s	178.23 ms (16.7%)
Agent	iast	1.231 s	164.877 ms (15.5%)
Agent	profiling	1.199 s	132.938 ms (12.5%)
Total	tracing	10.89 s	-
Total	appsec	10.963 s	72.966 ms (0.7%)
Total	iast	11.165 s	275.358 ms (2.5%)
Total	profiling	11.03 s	140.501 ms (1.3%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.065 s	-
Agent	appsec	1.236 s	171.151 ms (16.1%)
Agent	iast	1.234 s	169.537 ms (15.9%)
Agent	profiling	1.192 s	126.651 ms (11.9%)
Total	tracing	10.792 s	-
Total	appsec	10.819 s	26.261 ms (0.2%)
Total	iast	11.241 s	448.783 ms (4.2%)
Total	profiling	10.862 s	69.975 ms (0.6%)

gantt
    title petclinic - break down per module: candidate=1.60.0-SNAPSHOT~2dc7b4ba83, baseline=1.60.0-SNAPSHOT~2ebc964340

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.185 ms) : 0, 1185
crashtracking [candidate] (1.17 ms) : 0, 1170
BytebuddyAgent [baseline] (628.772 ms) : 0, 628772
BytebuddyAgent [candidate] (628.139 ms) : 0, 628139
AgentMeter [baseline] (28.948 ms) : 0, 28948
AgentMeter [candidate] (28.872 ms) : 0, 28872
GlobalTracer [baseline] (257.598 ms) : 0, 257598
GlobalTracer [candidate] (257.091 ms) : 0, 257091
AppSec [baseline] (32.747 ms) : 0, 32747
AppSec [candidate] (32.647 ms) : 0, 32647
Debugger [baseline] (61.555 ms) : 0, 61555
Debugger [candidate] (68.155 ms) : 0, 68155
Remote Config [baseline] (628.337 µs) : 0, 628
Remote Config [candidate] (605.001 µs) : 0, 605
Telemetry [baseline] (14.539 ms) : 0, 14539
Telemetry [candidate] (9.127 ms) : 0, 9127
Flare Poller [baseline] (4.555 ms) : 0, 4555
Flare Poller [candidate] (3.832 ms) : 0, 3832
section appsec
crashtracking [baseline] (1.179 ms) : 0, 1179
crashtracking [candidate] (1.179 ms) : 0, 1179
BytebuddyAgent [baseline] (660.892 ms) : 0, 660892
BytebuddyAgent [candidate] (658.646 ms) : 0, 658646
AgentMeter [baseline] (12.042 ms) : 0, 12042
AgentMeter [candidate] (11.932 ms) : 0, 11932
GlobalTracer [baseline] (259.27 ms) : 0, 259270
GlobalTracer [candidate] (257.888 ms) : 0, 257888
IAST [baseline] (25.354 ms) : 0, 25354
IAST [candidate] (25.295 ms) : 0, 25295
AppSec [baseline] (168.419 ms) : 0, 168419
AppSec [candidate] (173.369 ms) : 0, 173369
Debugger [baseline] (67.288 ms) : 0, 67288
Debugger [candidate] (58.888 ms) : 0, 58888
Remote Config [baseline] (661.769 µs) : 0, 662
Remote Config [candidate] (652.301 µs) : 0, 652
Telemetry [baseline] (9.77 ms) : 0, 9770
Telemetry [candidate] (9.253 ms) : 0, 9253
Flare Poller [baseline] (3.76 ms) : 0, 3760
Flare Poller [candidate] (3.603 ms) : 0, 3603
section iast
crashtracking [baseline] (1.184 ms) : 0, 1184
crashtracking [candidate] (1.184 ms) : 0, 1184
BytebuddyAgent [baseline] (795.059 ms) : 0, 795059
BytebuddyAgent [candidate] (797.444 ms) : 0, 797444
AgentMeter [baseline] (11.23 ms) : 0, 11230
AgentMeter [candidate] (11.277 ms) : 0, 11277
GlobalTracer [baseline] (248.162 ms) : 0, 248162
GlobalTracer [candidate] (249.203 ms) : 0, 249203
IAST [baseline] (26.833 ms) : 0, 26833
IAST [candidate] (26.99 ms) : 0, 26990
AppSec [baseline] (32.076 ms) : 0, 32076
AppSec [candidate] (35.212 ms) : 0, 35212
Debugger [baseline] (68.195 ms) : 0, 68195
Debugger [candidate] (64.965 ms) : 0, 64965
Remote Config [baseline] (533.546 µs) : 0, 534
Remote Config [candidate] (543.396 µs) : 0, 543
Telemetry [baseline] (8.61 ms) : 0, 8610
Telemetry [candidate] (8.69 ms) : 0, 8690
Flare Poller [baseline] (3.474 ms) : 0, 3474
Flare Poller [candidate] (3.498 ms) : 0, 3498
section profiling
crashtracking [baseline] (1.209 ms) : 0, 1209
crashtracking [candidate] (1.21 ms) : 0, 1210
BytebuddyAgent [baseline] (687.178 ms) : 0, 687178
BytebuddyAgent [candidate] (682.979 ms) : 0, 682979
AgentMeter [baseline] (8.725 ms) : 0, 8725
AgentMeter [candidate] (8.893 ms) : 0, 8893
GlobalTracer [baseline] (217.383 ms) : 0, 217383
GlobalTracer [candidate] (215.857 ms) : 0, 215857
AppSec [baseline] (33.054 ms) : 0, 33054
AppSec [candidate] (32.533 ms) : 0, 32533
Debugger [baseline] (67.96 ms) : 0, 67960
Debugger [candidate] (67.062 ms) : 0, 67062
Remote Config [baseline] (627.437 µs) : 0, 627
Remote Config [candidate] (628.367 µs) : 0, 628
Telemetry [baseline] (9.031 ms) : 0, 9031
Telemetry [candidate] (8.932 ms) : 0, 8932
Flare Poller [baseline] (3.827 ms) : 0, 3827
Flare Poller [candidate] (3.646 ms) : 0, 3646
ProfilingAgent [baseline] (99.502 ms) : 0, 99502
ProfilingAgent [candidate] (99.66 ms) : 0, 99660
Profiling [baseline] (100.068 ms) : 0, 100068
Profiling [candidate] (100.239 ms) : 0, 100239

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	jb/jfr_redacting
git_commit_date	1770902831	1771103311
git_commit_sha	`2ebc964`	`2dc7b4b`
release_version	1.60.0-SNAPSHOT~2ebc964340	1.60.0-SNAPSHOT~2dc7b4ba83

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1771105808	1771105808
ci_job_id	1427439033	1427439033
ci_pipeline_id	96616345	96616345
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-0ikegjj2 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-0ikegjj2 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 2 performance regressions! Performance is the same for 19 metrics, 15 unstable metrics.

scenario	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p95	Δ mean throughput	candidate mean agg_http_req_duration_p50	candidate mean agg_http_req_duration_p95	candidate mean throughput	baseline mean agg_http_req_duration_p50	baseline mean agg_http_req_duration_p95	baseline mean throughput
scenario:load:petclinic:no_agent:high_load	worse [+0.573ms; +1.988ms] or [+3.271%; +11.360%]	unsure [+0.349ms; +3.118ms] or [+1.187%; +10.598%]	unstable [-43.030op/s; +9.530op/s] or [-16.602%; +3.677%]	18.783ms	31.155ms	242.438op/s	17.502ms	29.421ms	259.188op/s
scenario:load:petclinic:code_origins:high_load	worse [+0.562ms; +1.454ms] or [+3.267%; +8.453%]	unsure [+34.592µs; +1355.717µs] or [+0.120%; +4.714%]	unstable [-38.021op/s; +13.396op/s] or [-14.399%; +5.073%]	18.208ms	29.453ms	251.750op/s	17.200ms	28.758ms	264.062op/s

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.60.0-SNAPSHOT~2dc7b4ba83, baseline=1.60.0-SNAPSHOT~2ebc964340
    dateFormat X
    axisFormat %s
section baseline
no_agent (18.009 ms) : 17826, 18192
.   : milestone, 18009,
appsec (18.599 ms) : 18413, 18785
.   : milestone, 18599,
code_origins (17.67 ms) : 17492, 17848
.   : milestone, 17670,
iast (17.773 ms) : 17597, 17949
.   : milestone, 17773,
profiling (18.809 ms) : 18615, 19003
.   : milestone, 18809,
tracing (17.655 ms) : 17479, 17830
.   : milestone, 17655,
section candidate
no_agent (19.257 ms) : 19059, 19454
.   : milestone, 19257,
appsec (18.442 ms) : 18257, 18627
.   : milestone, 18442,
code_origins (18.537 ms) : 18352, 18723
.   : milestone, 18537,
iast (17.688 ms) : 17509, 17867
.   : milestone, 17688,
profiling (18.595 ms) : 18407, 18783
.   : milestone, 18595,
tracing (18.004 ms) : 17824, 18185
.   : milestone, 18004,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	18.009 ms [17.826 ms, 18.192 ms]	-
appsec	18.599 ms [18.413 ms, 18.785 ms]	589.573 µs (3.3%)
code_origins	17.67 ms [17.492 ms, 17.848 ms]	-339.518 µs (-1.9%)
iast	17.773 ms [17.597 ms, 17.949 ms]	-235.958 µs (-1.3%)
profiling	18.809 ms [18.615 ms, 19.003 ms]	799.822 µs (4.4%)
tracing	17.655 ms [17.479 ms, 17.83 ms]	-354.616 µs (-2.0%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	19.257 ms [19.059 ms, 19.454 ms]	-
appsec	18.442 ms [18.257 ms, 18.627 ms]	-814.649 µs (-4.2%)
code_origins	18.537 ms [18.352 ms, 18.723 ms]	-719.467 µs (-3.7%)
iast	17.688 ms [17.509 ms, 17.867 ms]	-1.569 ms (-8.1%)
profiling	18.595 ms [18.407 ms, 18.783 ms]	-662.002 µs (-3.4%)
tracing	18.004 ms [17.824 ms, 18.185 ms]	-1.253 ms (-6.5%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.60.0-SNAPSHOT~2dc7b4ba83, baseline=1.60.0-SNAPSHOT~2ebc964340
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.184 ms) : 1173, 1195
.   : milestone, 1184,
iast (3.217 ms) : 3178, 3256
.   : milestone, 3217,
iast_FULL (6.006 ms) : 5945, 6068
.   : milestone, 6006,
iast_GLOBAL (3.627 ms) : 3568, 3687
.   : milestone, 3627,
profiling (2.114 ms) : 2093, 2134
.   : milestone, 2114,
tracing (1.811 ms) : 1795, 1826
.   : milestone, 1811,
section candidate
no_agent (1.194 ms) : 1182, 1206
.   : milestone, 1194,
iast (3.257 ms) : 3212, 3303
.   : milestone, 3257,
iast_FULL (5.787 ms) : 5729, 5844
.   : milestone, 5787,
iast_GLOBAL (3.582 ms) : 3527, 3636
.   : milestone, 3582,
profiling (2.209 ms) : 2189, 2229
.   : milestone, 2209,
tracing (1.793 ms) : 1778, 1808
.   : milestone, 1793,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.184 ms [1.173 ms, 1.195 ms]	-
iast	3.217 ms [3.178 ms, 3.256 ms]	2.033 ms (171.6%)
iast_FULL	6.006 ms [5.945 ms, 6.068 ms]	4.822 ms (407.1%)
iast_GLOBAL	3.627 ms [3.568 ms, 3.687 ms]	2.443 ms (206.3%)
profiling	2.114 ms [2.093 ms, 2.134 ms]	929.135 µs (78.4%)
tracing	1.811 ms [1.795 ms, 1.826 ms]	626.206 µs (52.9%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.194 ms [1.182 ms, 1.206 ms]	-
iast	3.257 ms [3.212 ms, 3.303 ms]	2.063 ms (172.8%)
iast_FULL	5.787 ms [5.729 ms, 5.844 ms]	4.593 ms (384.6%)
iast_GLOBAL	3.582 ms [3.527 ms, 3.636 ms]	2.388 ms (199.9%)
profiling	2.209 ms [2.189 ms, 2.229 ms]	1.015 ms (85.0%)
tracing	1.793 ms [1.778 ms, 1.808 ms]	598.873 µs (50.2%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	jb/jfr_redacting
git_commit_date	1770902831	1771103311
git_commit_sha	`2ebc964`	`2dc7b4b`
release_version	1.60.0-SNAPSHOT~2ebc964340	1.60.0-SNAPSHOT~2dc7b4ba83

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1771105624	1771105624
ci_job_id	1427439034	1427439034
ci_pipeline_id	96616345	96616345
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-19bq5clx 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-19bq5clx 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.60.0-SNAPSHOT~2dc7b4ba83, baseline=1.60.0-SNAPSHOT~2ebc964340
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.997 s) : 14997000, 14997000
.   : milestone, 14997000,
appsec (14.932 s) : 14932000, 14932000
.   : milestone, 14932000,
iast (18.004 s) : 18004000, 18004000
.   : milestone, 18004000,
iast_GLOBAL (17.785 s) : 17785000, 17785000
.   : milestone, 17785000,
profiling (14.915 s) : 14915000, 14915000
.   : milestone, 14915000,
tracing (14.609 s) : 14609000, 14609000
.   : milestone, 14609000,
section candidate
no_agent (14.884 s) : 14884000, 14884000
.   : milestone, 14884000,
appsec (14.959 s) : 14959000, 14959000
.   : milestone, 14959000,
iast (18.3 s) : 18300000, 18300000
.   : milestone, 18300000,
iast_GLOBAL (17.612 s) : 17612000, 17612000
.   : milestone, 17612000,
profiling (14.839 s) : 14839000, 14839000
.   : milestone, 14839000,
tracing (14.707 s) : 14707000, 14707000
.   : milestone, 14707000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.997 s [14.997 s, 14.997 s]	-
appsec	14.932 s [14.932 s, 14.932 s]	-65.0 ms (-0.4%)
iast	18.004 s [18.004 s, 18.004 s]	3.007 s (20.1%)
iast_GLOBAL	17.785 s [17.785 s, 17.785 s]	2.788 s (18.6%)
profiling	14.915 s [14.915 s, 14.915 s]	-82.0 ms (-0.5%)
tracing	14.609 s [14.609 s, 14.609 s]	-388.0 ms (-2.6%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.884 s [14.884 s, 14.884 s]	-
appsec	14.959 s [14.959 s, 14.959 s]	75.0 ms (0.5%)
iast	18.3 s [18.3 s, 18.3 s]	3.416 s (23.0%)
iast_GLOBAL	17.612 s [17.612 s, 17.612 s]	2.728 s (18.3%)
profiling	14.839 s [14.839 s, 14.839 s]	-45.0 ms (-0.3%)
tracing	14.707 s [14.707 s, 14.707 s]	-177.0 ms (-1.2%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.60.0-SNAPSHOT~2dc7b4ba83, baseline=1.60.0-SNAPSHOT~2ebc964340
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.465 ms) : 1453, 1476
.   : milestone, 1465,
appsec (3.765 ms) : 3544, 3986
.   : milestone, 3765,
iast (2.249 ms) : 2180, 2318
.   : milestone, 2249,
iast_GLOBAL (2.287 ms) : 2218, 2356
.   : milestone, 2287,
profiling (2.082 ms) : 2026, 2137
.   : milestone, 2082,
tracing (2.044 ms) : 1991, 2097
.   : milestone, 2044,
section candidate
no_agent (1.467 ms) : 1456, 1479
.   : milestone, 1467,
appsec (3.769 ms) : 3547, 3992
.   : milestone, 3769,
iast (2.238 ms) : 2169, 2306
.   : milestone, 2238,
iast_GLOBAL (2.283 ms) : 2215, 2352
.   : milestone, 2283,
profiling (2.071 ms) : 2017, 2126
.   : milestone, 2071,
tracing (2.044 ms) : 1991, 2098
.   : milestone, 2044,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.465 ms [1.453 ms, 1.476 ms]	-
appsec	3.765 ms [3.544 ms, 3.986 ms]	2.3 ms (157.1%)
iast	2.249 ms [2.18 ms, 2.318 ms]	783.912 µs (53.5%)
iast_GLOBAL	2.287 ms [2.218 ms, 2.356 ms]	822.319 µs (56.1%)
profiling	2.082 ms [2.026 ms, 2.137 ms]	617.162 µs (42.1%)
tracing	2.044 ms [1.991 ms, 2.097 ms]	579.377 µs (39.6%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.467 ms [1.456 ms, 1.479 ms]	-
appsec	3.769 ms [3.547 ms, 3.992 ms]	2.302 ms (156.9%)
iast	2.238 ms [2.169 ms, 2.306 ms]	770.506 µs (52.5%)
iast_GLOBAL	2.283 ms [2.215 ms, 2.352 ms]	816.173 µs (55.6%)
profiling	2.071 ms [2.017 ms, 2.126 ms]	604.071 µs (41.2%)
tracing	2.044 ms [1.991 ms, 2.098 ms]	577.276 µs (39.3%)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Add static imports, fields, initialization, and getters for PROFILING_SCRUB_ENABLED, PROFILING_SCRUB_FAIL_OPEN, and PROFILING_SCRUB_EXCLUDE_EVENTS to complete config implementation. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

jbachorik · 2026-02-13T13:00:29Z

Test Coverage Update

Added comprehensive test coverage for the profiling-scrubber module:

Coverage Achieved

82.15% instruction coverage (695 covered / 151 missed)
72.37% branch coverage (55 covered / 21 missed)
76 tests across 10 test suites (from 3 basic tests)

Test Suite Breakdown

VarintOperations (14 tests): Boundary value testing including Integer.MAX_VALUE
FileOperations (5 tests): Buffer edge cases (<64KB, >64KB, empty, exact size, with offset)
GuardFunctions (3 tests): Conditional and unconditional scrubbing logic
ErrorHandling (3 tests): File I/O failures and permission errors
ComplexScrubbingScenarios (6 tests): Multiple fields, missing metadata, output parsability
TypeScrubbingTests (3 tests): equals/hashCode contract verification
FuzzTests (13 tests): Property-based testing with 100+ iterations per seed
DefaultScrubDefinitionTest (7 tests): Reflection-based configuration testing
Integration Tests (3 tests): End-to-end with real 3.6MB JFR file

Build Configuration

Coverage thresholds enforced: 70% instruction, 60% branch
DefaultScrubDefinition excluded from coverage verification (ConfigProvider is final/untestable)
All checks passing: tests, coverage, spotless, spotbugs ✅

This provides strong confidence in the security-critical JFR scrubbing functionality.

Using .class literals causes eager class loading, making @TargetClass annotations visible to GraalVM's annotation processor even when we conditionally skip adding them to the result list. By loading the classes reflectively with Class.forName(), we prevent them from being discovered when JMXFetch is not present. Also increased native-image heap from 4GB to 8GB to avoid OOM during compilation with profiler enabled. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Tracing instrumentation in ScrubRecordingDataListener causes build-time initialization of tracer classes during GraalVM native-image compilation when the agent is attached via -J-javaagent. This results in "Classes that should be initialized at run time got initialized during image building" errors. Removing the spans/metrics maintains native-image compatibility while preserving the core scrubbing functionality. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Changed profiling-scrubber's internal-api dependency from 'api' to 'implementation'. The 'api' scope created a transitive dependency chain (scrubber→internal-api→dd-trace-api) that exposed tracer classes through the scrubber's API surface, causing them to initialize at build time during GraalVM native-image compilation. Using 'implementation' keeps internal-api as an internal dependency, preventing the initialization cascade. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Disabled exception profiling during native-image build to prevent initialization of 44 config/bootstrap classes. The agent's exception instrumentation was triggering during GraalVM class scanning. Changes: - Remove static imports with System.getProperty() initializers - Disable datadog.ExceptionSample event during native-image build - Remove internal-api dependency from profiling-scrubber - Clean up native-image annotation substitution comments Result: 0 initialization errors (down from 44) Note: Native-image build now crashes with SIGBUS during GC. See NATIVE_IMAGE_FIX_STATUS.md for details. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

jbachorik added type: enhancement Enhancements and improvements comp: profiling Profiling AI AI-assisted changes labels Feb 12, 2026

jbachorik changed the title ~~Add JFR scrubbing before profile upload~~ [WIP] Add JFR scrubbing before profile upload Feb 12, 2026

jbachorik added tag: ai generated Largely based on code generated by an AI or LLM and removed AI AI-assisted changes labels Feb 12, 2026

jbachorik force-pushed the jb/jfr_redacting branch from 322c7a1 to 51136f3 Compare February 12, 2026 13:30

jbachorik and others added 2 commits February 12, 2026 15:32

Add JFR scrubbing before profile upload

6f14caf

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Add scrubbing configs to metadata

79d9e6c

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

jbachorik force-pushed the jb/jfr_redacting branch from 3ded12a to 79d9e6c Compare February 12, 2026 14:32

jbachorik and others added 6 commits February 12, 2026 15:42

Add tracing to JFR scrubbing operation

a0a13b0

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Mark scrubbing span as internal operation

28dd95a

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Use _dd. prefix for internal scrubbing metrics

9d763a0

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Fix compilation: use addThrowable() for errors

51ac782

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Add Config implementation for scrubbing

4f296cf

Add static imports, fields, initialization, and getters for PROFILING_SCRUB_ENABLED, PROFILING_SCRUB_FAIL_OPEN, and PROFILING_SCRUB_EXCLUDE_EVENTS to complete config implementation. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Dummy commit

d1fbbee

jbachorik force-pushed the jb/jfr_redacting branch from d1fbbee to 619a3cd Compare February 13, 2026 12:51

jbachorik force-pushed the jb/jfr_redacting branch from 423e6a9 to d1fbbee Compare February 13, 2026 17:14

jbachorik force-pushed the jb/jfr_redacting branch from 85dbcbf to 6397775 Compare February 13, 2026 19:37

jbachorik and others added 3 commits February 13, 2026 20:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Add JFR scrubbing before profile upload#10577

[WIP] Add JFR scrubbing before profile upload#10577
jbachorik wants to merge 12 commits intomasterfrom
jb/jfr_redacting

jbachorik commented Feb 12, 2026 •

edited

Loading

Uh oh!

pr-commenter bot commented Feb 12, 2026 •

edited

Loading

Uh oh!

jbachorik commented Feb 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

jbachorik commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Implementation

GraalVM Native-Image Compatibility Fix

Test Coverage

Contributor Checklist

Uh oh!

pr-commenter bot commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

jbachorik commented Feb 13, 2026

Test Coverage Update

Coverage Achieved

Test Suite Breakdown

Build Configuration

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jbachorik commented Feb 12, 2026 •

edited

Loading

pr-commenter bot commented Feb 12, 2026 •

edited

Loading