LLM Observability SDK #8781

gary-huang · 2025-05-07T15:09:00Z

What Does This Do

adds the LLM Observability SDK, it can be enabled using the env var DD_LLMOBS_ENABLED or the property dd.llmobs.enabled

full docs can be found here: https://docs.datadoghq.com/llm_observability/sdk/java/

example of traces produced

example span

different evaluation metrics supported

Motivation

JIRA

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

pr-commenter · 2025-05-07T15:36:52Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	gary/llmobs-sdk-merge
git_commit_date	1752058280	1752064429
git_commit_sha	`4acd1c7`	`0cf7162`
release_version	1.51.0-SNAPSHOT~4acd1c74e1	1.51.0-SNAPSHOT~0cf7162d66

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1752066261	1752066261
ci_job_id	1020458854	1020458854
ci_pipeline_id	70036166	70036166
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-2-nk79kof6 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-2-nk79kof6 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 44 metrics, 9 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.51.0-SNAPSHOT~0cf7162d66, baseline=1.51.0-SNAPSHOT~4acd1c74e1

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (996.872 ms) : 0, 996872
Total [baseline] (10.644 s) : 0, 10643664
Agent [candidate] (1.003 s) : 0, 1002865
Total [candidate] (10.667 s) : 0, 10666535
section appsec
Agent [baseline] (1.174 s) : 0, 1173570
Total [baseline] (10.709 s) : 0, 10709326
Agent [candidate] (1.184 s) : 0, 1183876
Total [candidate] (10.762 s) : 0, 10761759
section iast
Agent [baseline] (1.131 s) : 0, 1131277
Total [baseline] (10.783 s) : 0, 10782857
Agent [candidate] (1.141 s) : 0, 1140722
Total [candidate] (10.82 s) : 0, 10819842
section profiling
Agent [baseline] (1.245 s) : 0, 1245142
Total [baseline] (10.939 s) : 0, 10938568
Agent [candidate] (1.251 s) : 0, 1250902
Total [candidate] (10.976 s) : 0, 10976215

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	996.872 ms	-
Agent	appsec	1.174 s	176.698 ms (17.7%)
Agent	iast	1.131 s	134.405 ms (13.5%)
Agent	profiling	1.245 s	248.269 ms (24.9%)
Total	tracing	10.644 s	-
Total	appsec	10.709 s	65.663 ms (0.6%)
Total	iast	10.783 s	139.193 ms (1.3%)
Total	profiling	10.939 s	294.904 ms (2.8%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.003 s	-
Agent	appsec	1.184 s	181.011 ms (18.0%)
Agent	iast	1.141 s	137.857 ms (13.7%)
Agent	profiling	1.251 s	248.037 ms (24.7%)
Total	tracing	10.667 s	-
Total	appsec	10.762 s	95.224 ms (0.9%)
Total	iast	10.82 s	153.307 ms (1.4%)
Total	profiling	10.976 s	309.68 ms (2.9%)

gantt
    title petclinic - break down per module: candidate=1.51.0-SNAPSHOT~0cf7162d66, baseline=1.51.0-SNAPSHOT~4acd1c74e1

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (688.251 ms) : 0, 688251
BytebuddyAgent [candidate] (692.974 ms) : 0, 692974
GlobalTracer [baseline] (242.659 ms) : 0, 242659
GlobalTracer [candidate] (243.514 ms) : 0, 243514
AppSec [baseline] (30.299 ms) : 0, 30299
AppSec [candidate] (30.525 ms) : 0, 30525
Debugger [baseline] (5.975 ms) : 0, 5975
Debugger [candidate] (5.985 ms) : 0, 5985
Remote Config [baseline] (683.618 µs) : 0, 684
Remote Config [candidate] (698.028 µs) : 0, 698
Telemetry [baseline] (8.271 ms) : 0, 8271
Telemetry [candidate] (8.255 ms) : 0, 8255
section appsec
BytebuddyAgent [baseline] (709.653 ms) : 0, 709653
BytebuddyAgent [candidate] (716.662 ms) : 0, 716662
GlobalTracer [baseline] (234.671 ms) : 0, 234671
GlobalTracer [candidate] (236.817 ms) : 0, 236817
AppSec [baseline] (170.825 ms) : 0, 170825
AppSec [candidate] (171.207 ms) : 0, 171207
Debugger [baseline] (5.717 ms) : 0, 5717
Debugger [candidate] (5.751 ms) : 0, 5751
Remote Config [baseline] (602.607 µs) : 0, 603
Remote Config [candidate] (606.804 µs) : 0, 607
Telemetry [baseline] (8.123 ms) : 0, 8123
Telemetry [candidate] (8.167 ms) : 0, 8167
IAST [baseline] (23.113 ms) : 0, 23113
IAST [candidate] (23.656 ms) : 0, 23656
section iast
BytebuddyAgent [baseline] (806.941 ms) : 0, 806941
BytebuddyAgent [candidate] (814.356 ms) : 0, 814356
GlobalTracer [baseline] (232.191 ms) : 0, 232191
GlobalTracer [candidate] (233.503 ms) : 0, 233503
AppSec [baseline] (31.078 ms) : 0, 31078
AppSec [candidate] (30.515 ms) : 0, 30515
Debugger [baseline] (5.744 ms) : 0, 5744
Debugger [candidate] (5.741 ms) : 0, 5741
Remote Config [baseline] (570.003 µs) : 0, 570
Remote Config [candidate] (579.122 µs) : 0, 579
Telemetry [baseline] (7.88 ms) : 0, 7880
Telemetry [candidate] (7.992 ms) : 0, 7992
IAST [baseline] (26.132 ms) : 0, 26132
IAST [candidate] (27.145 ms) : 0, 27145
section profiling
BytebuddyAgent [baseline] (678.43 ms) : 0, 678430
BytebuddyAgent [candidate] (681.383 ms) : 0, 681383
GlobalTracer [baseline] (361.305 ms) : 0, 361305
GlobalTracer [candidate] (363.052 ms) : 0, 363052
AppSec [baseline] (32.927 ms) : 0, 32927
AppSec [candidate] (33.304 ms) : 0, 33304
Debugger [baseline] (9.901 ms) : 0, 9901
Debugger [candidate] (9.967 ms) : 0, 9967
Remote Config [baseline] (665.175 µs) : 0, 665
Remote Config [candidate] (674.244 µs) : 0, 674
Telemetry [baseline] (9.454 ms) : 0, 9454
Telemetry [candidate] (9.449 ms) : 0, 9449
ProfilingAgent [baseline] (103.933 ms) : 0, 103933
ProfilingAgent [candidate] (104.238 ms) : 0, 104238
Profiling [baseline] (103.958 ms) : 0, 103958
Profiling [candidate] (104.261 ms) : 0, 104261

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.51.0-SNAPSHOT~0cf7162d66, baseline=1.51.0-SNAPSHOT~4acd1c74e1

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (996.079 ms) : 0, 996079
Total [baseline] (8.557 s) : 0, 8556553
Agent [candidate] (998.087 ms) : 0, 998087
Total [candidate] (8.576 s) : 0, 8576450
section iast
Agent [baseline] (1.135 s) : 0, 1134751
Total [baseline] (9.308 s) : 0, 9307957
Agent [candidate] (1.143 s) : 0, 1143136
Total [candidate] (9.32 s) : 0, 9319837

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	996.079 ms	-
Agent	iast	1.135 s	138.672 ms (13.9%)
Total	tracing	8.557 s	-
Total	iast	9.308 s	751.404 ms (8.8%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	998.087 ms	-
Agent	iast	1.143 s	145.049 ms (14.5%)
Total	tracing	8.576 s	-
Total	iast	9.32 s	743.387 ms (8.7%)

gantt
    title insecure-bank - break down per module: candidate=1.51.0-SNAPSHOT~0cf7162d66, baseline=1.51.0-SNAPSHOT~4acd1c74e1

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (688.604 ms) : 0, 688604
BytebuddyAgent [candidate] (689.318 ms) : 0, 689318
GlobalTracer [baseline] (241.634 ms) : 0, 241634
GlobalTracer [candidate] (242.674 ms) : 0, 242674
AppSec [baseline] (30.379 ms) : 0, 30379
AppSec [candidate] (30.381 ms) : 0, 30381
Debugger [baseline] (5.969 ms) : 0, 5969
Debugger [candidate] (5.968 ms) : 0, 5968
Remote Config [baseline] (674.762 µs) : 0, 675
Remote Config [candidate] (678.786 µs) : 0, 679
Telemetry [baseline] (8.199 ms) : 0, 8199
Telemetry [candidate] (8.267 ms) : 0, 8267
section iast
BytebuddyAgent [baseline] (809.629 ms) : 0, 809629
BytebuddyAgent [candidate] (815.361 ms) : 0, 815361
GlobalTracer [baseline] (232.521 ms) : 0, 232521
GlobalTracer [candidate] (233.808 ms) : 0, 233808
AppSec [baseline] (28.145 ms) : 0, 28145
AppSec [candidate] (32.174 ms) : 0, 32174
Debugger [baseline] (5.749 ms) : 0, 5749
Debugger [candidate] (6.666 ms) : 0, 6666
Remote Config [baseline] (573.159 µs) : 0, 573
Remote Config [candidate] (591.587 µs) : 0, 592
Telemetry [baseline] (8.005 ms) : 0, 8005
Telemetry [candidate] (8.06 ms) : 0, 8060
IAST [baseline] (29.449 ms) : 0, 29449
IAST [candidate] (25.632 ms) : 0, 25632

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	gary/llmobs-sdk-merge
git_commit_date	1752058280	1752064429
git_commit_sha	`4acd1c7`	`0cf7162`
release_version	1.51.0-SNAPSHOT~4acd1c74e1	1.51.0-SNAPSHOT~0cf7162d66

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1752065942	1752065942
ci_job_id	1020458855	1020458855
ci_pipeline_id	70036166	70036166
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-3-7a3qyhcb 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-3-7a3qyhcb 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 1 performance improvements and 2 performance regressions! Performance is the same for 9 metrics, 12 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:insecure-bank:no_agent:high_load	worse [+86.556µs; +188.925µs] or [+2.036%; +4.444%]	unstable [-142.028op/s; +75.465op/s] or [-13.162%; +6.994%]	4.389ms	1045.781op/s	4.251ms	1079.062op/s
scenario:load:insecure-bank:iast_GLOBAL:high_load	worse [+346.148µs; +765.323µs] or [+3.438%; +7.602%]	unstable [-72.723op/s; +24.911op/s] or [-15.757%; +5.397%]	10.623ms	437.625op/s	10.067ms	461.531op/s
scenario:load:petclinic:appsec:high_load	better [-2.394ms; -1.448ms] or [-4.810%; -2.909%]	unstable [-3.205op/s; +10.830op/s] or [-3.407%; +11.515%]	47.843ms	97.862op/s	49.764ms	94.050op/s

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.51.0-SNAPSHOT~0cf7162d66, baseline=1.51.0-SNAPSHOT~4acd1c74e1
    dateFormat X
    axisFormat %s
section baseline
no_agent (4.251 ms) : 4205, 4298
.   : milestone, 4251,
iast (9.247 ms) : 9095, 9398
.   : milestone, 9247,
iast_FULL (13.732 ms) : 13456, 14007
.   : milestone, 13732,
iast_GLOBAL (10.067 ms) : 9880, 10254
.   : milestone, 10067,
profiling (8.772 ms) : 8635, 8908
.   : milestone, 8772,
tracing (7.602 ms) : 7488, 7717
.   : milestone, 7602,
section candidate
no_agent (4.389 ms) : 4341, 4438
.   : milestone, 4389,
iast (9.277 ms) : 9128, 9427
.   : milestone, 9277,
iast_FULL (13.896 ms) : 13620, 14173
.   : milestone, 13896,
iast_GLOBAL (10.623 ms) : 10421, 10825
.   : milestone, 10623,
profiling (8.521 ms) : 8388, 8654
.   : milestone, 8521,
tracing (7.518 ms) : 7413, 7624
.   : milestone, 7518,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.251 ms [4.205 ms, 4.298 ms]	-
iast	9.247 ms [9.095 ms, 9.398 ms]	4.995 ms (117.5%)
iast_FULL	13.732 ms [13.456 ms, 14.007 ms]	9.48 ms (223.0%)
iast_GLOBAL	10.067 ms [9.88 ms, 10.254 ms]	5.816 ms (136.8%)
profiling	8.772 ms [8.635 ms, 8.908 ms]	4.52 ms (106.3%)
tracing	7.602 ms [7.488 ms, 7.717 ms]	3.351 ms (78.8%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.389 ms [4.341 ms, 4.438 ms]	-
iast	9.277 ms [9.128 ms, 9.427 ms]	4.888 ms (111.4%)
iast_FULL	13.896 ms [13.62 ms, 14.173 ms]	9.507 ms (216.6%)
iast_GLOBAL	10.623 ms [10.421 ms, 10.825 ms]	6.234 ms (142.0%)
profiling	8.521 ms [8.388 ms, 8.654 ms]	4.132 ms (94.1%)
tracing	7.518 ms [7.413 ms, 7.624 ms]	3.129 ms (71.3%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.51.0-SNAPSHOT~0cf7162d66, baseline=1.51.0-SNAPSHOT~4acd1c74e1
    dateFormat X
    axisFormat %s
section baseline
no_agent (37.786 ms) : 37490, 38083
.   : milestone, 37786,
appsec (49.764 ms) : 49304, 50224
.   : milestone, 49764,
code_origins (47.415 ms) : 46983, 47847
.   : milestone, 47415,
iast (44.667 ms) : 44286, 45047
.   : milestone, 44667,
profiling (51.288 ms) : 50829, 51748
.   : milestone, 51288,
tracing (42.613 ms) : 42258, 42967
.   : milestone, 42613,
section candidate
no_agent (37.177 ms) : 36878, 37476
.   : milestone, 37177,
appsec (47.843 ms) : 47425, 48261
.   : milestone, 47843,
code_origins (46.375 ms) : 45968, 46781
.   : milestone, 46375,
iast (43.37 ms) : 42989, 43751
.   : milestone, 43370,
profiling (49.808 ms) : 49359, 50257
.   : milestone, 49808,
tracing (43.013 ms) : 42650, 43376
.   : milestone, 43013,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	37.786 ms [37.49 ms, 38.083 ms]	-
appsec	49.764 ms [49.304 ms, 50.224 ms]	11.978 ms (31.7%)
code_origins	47.415 ms [46.983 ms, 47.847 ms]	9.629 ms (25.5%)
iast	44.667 ms [44.286 ms, 45.047 ms]	6.88 ms (18.2%)
profiling	51.288 ms [50.829 ms, 51.748 ms]	13.502 ms (35.7%)
tracing	42.613 ms [42.258 ms, 42.967 ms]	4.826 ms (12.8%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	37.177 ms [36.878 ms, 37.476 ms]	-
appsec	47.843 ms [47.425 ms, 48.261 ms]	10.666 ms (28.7%)
code_origins	46.375 ms [45.968 ms, 46.781 ms]	9.198 ms (24.7%)
iast	43.37 ms [42.989 ms, 43.751 ms]	6.193 ms (16.7%)
profiling	49.808 ms [49.359 ms, 50.257 ms]	12.631 ms (34.0%)
tracing	43.013 ms [42.65 ms, 43.376 ms]	5.836 ms (15.7%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	gary/llmobs-sdk-merge
git_commit_date	1752058280	1752064429
git_commit_sha	`4acd1c7`	`0cf7162`
release_version	1.51.0-SNAPSHOT~4acd1c74e1	1.51.0-SNAPSHOT~0cf7162d66

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1752066666	1752066666
ci_job_id	1020458856	1020458856
ci_pipeline_id	70036166	70036166
cpu_model	Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-wiggy0aq 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-wiggy0aq 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.51.0-SNAPSHOT~0cf7162d66, baseline=1.51.0-SNAPSHOT~4acd1c74e1
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.488 ms) : 1476, 1500
.   : milestone, 1488,
appsec (2.737 ms) : 2656, 2818
.   : milestone, 2737,
iast (2.551 ms) : 2452, 2649
.   : milestone, 2551,
iast_GLOBAL (2.607 ms) : 2509, 2706
.   : milestone, 2607,
profiling (2.401 ms) : 2316, 2486
.   : milestone, 2401,
tracing (2.32 ms) : 2240, 2399
.   : milestone, 2320,
section candidate
no_agent (1.49 ms) : 1478, 1502
.   : milestone, 1490,
appsec (2.727 ms) : 2647, 2808
.   : milestone, 2727,
iast (2.56 ms) : 2461, 2659
.   : milestone, 2560,
iast_GLOBAL (2.603 ms) : 2504, 2702
.   : milestone, 2603,
profiling (2.388 ms) : 2304, 2472
.   : milestone, 2388,
tracing (2.32 ms) : 2240, 2400
.   : milestone, 2320,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.488 ms [1.476 ms, 1.5 ms]	-
appsec	2.737 ms [2.656 ms, 2.818 ms]	1.249 ms (84.0%)
iast	2.551 ms [2.452 ms, 2.649 ms]	1.063 ms (71.4%)
iast_GLOBAL	2.607 ms [2.509 ms, 2.706 ms]	1.119 ms (75.2%)
profiling	2.401 ms [2.316 ms, 2.486 ms]	913.331 µs (61.4%)
tracing	2.32 ms [2.24 ms, 2.399 ms]	831.846 µs (55.9%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.49 ms [1.478 ms, 1.502 ms]	-
appsec	2.727 ms [2.647 ms, 2.808 ms]	1.238 ms (83.1%)
iast	2.56 ms [2.461 ms, 2.659 ms]	1.07 ms (71.8%)
iast_GLOBAL	2.603 ms [2.504 ms, 2.702 ms]	1.113 ms (74.7%)
profiling	2.388 ms [2.304 ms, 2.472 ms]	898.481 µs (60.3%)
tracing	2.32 ms [2.24 ms, 2.4 ms]	830.119 µs (55.7%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.51.0-SNAPSHOT~0cf7162d66, baseline=1.51.0-SNAPSHOT~4acd1c74e1
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.107 s) : 15107000, 15107000
.   : milestone, 15107000,
appsec (14.317 s) : 14317000, 14317000
.   : milestone, 14317000,
iast (17.869 s) : 17869000, 17869000
.   : milestone, 17869000,
iast_GLOBAL (17.434 s) : 17434000, 17434000
.   : milestone, 17434000,
profiling (14.658 s) : 14658000, 14658000
.   : milestone, 14658000,
tracing (14.372 s) : 14372000, 14372000
.   : milestone, 14372000,
section candidate
no_agent (15.272 s) : 15272000, 15272000
.   : milestone, 15272000,
appsec (14.46 s) : 14460000, 14460000
.   : milestone, 14460000,
iast (18.113 s) : 18113000, 18113000
.   : milestone, 18113000,
iast_GLOBAL (17.224 s) : 17224000, 17224000
.   : milestone, 17224000,
profiling (14.471 s) : 14471000, 14471000
.   : milestone, 14471000,
tracing (14.683 s) : 14683000, 14683000
.   : milestone, 14683000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.107 s [15.107 s, 15.107 s]	-
appsec	14.317 s [14.317 s, 14.317 s]	-790.0 ms (-5.2%)
iast	17.869 s [17.869 s, 17.869 s]	2.762 s (18.3%)
iast_GLOBAL	17.434 s [17.434 s, 17.434 s]	2.327 s (15.4%)
profiling	14.658 s [14.658 s, 14.658 s]	-449.0 ms (-3.0%)
tracing	14.372 s [14.372 s, 14.372 s]	-735.0 ms (-4.9%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.272 s [15.272 s, 15.272 s]	-
appsec	14.46 s [14.46 s, 14.46 s]	-812.0 ms (-5.3%)
iast	18.113 s [18.113 s, 18.113 s]	2.841 s (18.6%)
iast_GLOBAL	17.224 s [17.224 s, 17.224 s]	1.952 s (12.8%)
profiling	14.471 s [14.471 s, 14.471 s]	-801.0 ms (-5.2%)
tracing	14.683 s [14.683 s, 14.683 s]	-589.0 ms (-3.9%)

github-actions · 2025-07-08T15:51:25Z

Hi! 👋 Looks like you updated a Git Submodule.
If this was not intentional please make sure to:

Update the submodule to the latest commit on the master branch using the git submodule update command,
Check you properly set up your environment for contributing.

github-actions · 2025-07-08T16:24:52Z

Hi! 👋 Looks like you updated a Git Submodule.
If this was not intentional please make sure to:

Update the submodule to the latest commit on the master branch using the git submodule update command,
Check you properly set up your environment for contributing.

github-actions · 2025-07-09T12:29:01Z

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

* add APIs for llm obs * add llm message class to support llm spans * follow java convention of naming Id instead of ID * add codeowners

* add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * spotless * add APIs for llm obs sdk (#8135) * add APIs for llm obs * add llm message class to support llm spans * follow java convention of naming Id instead of ID * add codeowners * rename ID to Id according to java naming conventions * Undo change to integrations-core submodule * fix build gradle * rm empty line * fix test

* add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * impl llmobs agent and llmobs apis * impl llmobs agent * working writer * add support for llm message and tool calls * cleaned up whitespace * resolve merge conflicts * remaining merge conflicts * fix bad method call * fixed llmobs intake creation if llmobs not enabled * removed print statements * added tests for llmobsspanmapper * fixed coverage for tags --------- Co-authored-by: Nayeem Kamal <[email protected]>

* add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * impl llmobs agent and llmobs apis * impl llmobs agent * working writer * add support for llm message and tool calls * impl llmobs agent and llmobs apis * use new ctx api to track parent span * cleaned up whitespace * resolve merge conflicts * remaining merge conflicts * fix bad method call * fixed llmobs intake creation if llmobs not enabled * removed print statements * ran spotless * added tests for llmobsspanmapper * fixed coverage for tags --------- Co-authored-by: Nayeem Kamal <[email protected]> Co-authored-by: Nayeem Kamal <[email protected]>

* add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * impl llmobs agent and llmobs apis * impl llmobs agent * working writer * add support for llm message and tool calls * impl llmobs agent and llmobs apis * use new ctx api to track parent span * add api for evals * working impl supporting both agentless and agent * handle null tags and default to default ml app if null or empty string provided in the override * cleaned up whitespace * resolve merge conflicts * remaining merge conflicts * fix bad method call * fixed llmobs intake creation if llmobs not enabled * removed print statements * ran spotless * ran spotless * added tests for llmobsspanmapper * fixed coverage for tags --------- Co-authored-by: Nayeem Kamal <[email protected]> Co-authored-by: Nayeem Kamal <[email protected]>

| Package | Type | Package file | Manager | Update | Change | |---|---|---|---|---|---| | [com.google.errorprone:error_prone_annotations](https://errorprone.info) ([source](https://github.com/google/error-prone)) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `2.39.0` -> `2.40.0` | | [org.apache.commons:commons-lang3](https://commons.apache.org/proper/commons-lang/) ([source](https://gitbox.apache.org/repos/asf/commons-lang.git)) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `3.17.0` -> `3.18.0` | | [org.jetbrains.kotlinx.binary-compatibility-validator](https://github.com/Kotlin/binary-compatibility-validator) | plugin | misk/gradle/libs.versions.toml | gradle | patch | `0.18.0` -> `0.18.1` | | [com.datadoghq:dd-trace-api](https://github.com/datadog/dd-trace-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `1.50.1` -> `1.51.0` | | [software.amazon.awssdk:sdk-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.77` -> `2.31.78` | | [software.amazon.awssdk:sqs](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.77` -> `2.31.78` | | [software.amazon.awssdk:dynamodb-enhanced](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.77` -> `2.31.78` | | [software.amazon.awssdk:dynamodb](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.77` -> `2.31.78` | | [software.amazon.awssdk:aws-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.77` -> `2.31.78` | | [software.amazon.awssdk:bom](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.77` -> `2.31.78` | | [software.amazon.awssdk:auth](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.77` -> `2.31.78` | --- ### Release Notes <details> <summary>google/error-prone (com.google.errorprone:error_prone_annotations)</summary> ### [`v2.40.0`](https://github.com/google/error-prone/releases/tag/v2.40.0): Error Prone 2.40.0 Changes: - Bug fixes and improvements - Releases (including snapshots) have migrated from [OSSRH to the Central Publisher Portal](https://central.sonatype.org/pages/ossrh-eol/#process-to-migrate) Full changelog: google/error-prone@v2.39.0...v2.40.0 </details> <details> <summary>Kotlin/binary-compatibility-validator (org.jetbrains.kotlinx.binary-compatibility-validator)</summary> ### [`v0.18.1`](https://github.com/Kotlin/binary-compatibility-validator/releases/tag/0.18.1) [Compare Source](Kotlin/binary-compatibility-validator@0.18.0...0.18.1) #### What's Changed - Fixed a bug preventing use of cross-compilation support during KLIB dump validation \[[#304](https://github.com/Kotlin/binary-compatibility-validator/issues/304)]\[[#306](https://github.com/Kotlin/binary-compatibility-validator/issues/306)] </details> <details> <summary>datadog/dd-trace-java (com.datadoghq:dd-trace-api)</summary> ### [`v1.51.0`](https://github.com/DataDog/dd-trace-java/releases/tag/v1.51.0): 1.51.0 ### Components #### Application Security Management (IAST) - 🐛 Fix verify error when ctor params are used after a call site ([#9083](DataDog/dd-trace-java#9083) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - 🐛 Limit the maximum size of the location path in IAST vulnerabilities ([#9028](DataDog/dd-trace-java#9028) - [@jandro996](https://github.com/jandro996)) - 🐛 Fix IAST gRPC handler with null superclass ([#8984](DataDog/dd-trace-java#8984) - [@smola](https://github.com/smola)) - ✨ Optimize IAST Vulnerability Detection ([#8885](DataDog/dd-trace-java#8885) - [@jandro996](https://github.com/jandro996)) #### Application Security Management (WAF) - ✨ Upgrade libddwaf-java to 15.0.0 ([#9022](DataDog/dd-trace-java#9022) - [@sezen-datadog](https://github.com/sezen-datadog)) - ✨ Extract RestEasy json body response schemas ([#9015](DataDog/dd-trace-java#9015) - [@jandro996](https://github.com/jandro996)) - ✨ Extract Jersey json body response schemas ([#9014](DataDog/dd-trace-java#9014) - [@jandro996](https://github.com/jandro996)) - ✨ Extract Ratpack json body response schemas ([#9013](DataDog/dd-trace-java#9013) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Enable API Security by default and make it lazy loading ([#9009](DataDog/dd-trace-java#9009) - [@smola](https://github.com/smola)) - ✨ Extract Vert.x json body response schemas ([#9001](DataDog/dd-trace-java#9001) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Extract Play json body response schemas ([#8995](DataDog/dd-trace-java#8995) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - 🐛 Fix Jackson nodes introspection for request/response schema extraction ([#8980](DataDog/dd-trace-java#8980) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Extract Spring json body response schemas ([#8938](DataDog/dd-trace-java#8938) - [@sezen-datadog](https://github.com/sezen-datadog)) - ✨ Default obfuscation regexp update ([#8937](DataDog/dd-trace-java#8937) - [@sezen-datadog](https://github.com/sezen-datadog)) #### Build & Tooling - ✨ Cancel GitLab running pipeline on new PR push ([#9023](DataDog/dd-trace-java#9023) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - ✨ Migrate publishing to Maven Central Portal ([#8807](DataDog/dd-trace-java#8807) - [@sarahchen6](https://github.com/sarahchen6)) #### Continuous Integration Visibility - 🐛 Fix Test Optimization to work with JDK 24 ([#9114](DataDog/dd-trace-java#9114) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Add repo root as safe directory on git client creation ([#9033](DataDog/dd-trace-java#9033) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Add PR number tag and improve PR information building ([#8990](DataDog/dd-trace-java#8990) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Update impacted tests logic ([#8923](DataDog/dd-trace-java#8923) - [@daniel-mohedano](https://github.com/daniel-mohedano)) #### Data Streams Monitoring - 🧹 Clean up DSM context injection ([#8776](DataDog/dd-trace-java#8776) - [@PerfectSlayer](https://github.com/PerfectSlayer)) #### Database Monitoring - 🐛 Set trace\_injected in try block ([#9025](DataDog/dd-trace-java#9025) - [@natashadada](https://github.com/natashadada)) #### Dynamic Instrumentation - 🐛 Add source file tracking enable option ([#9115](DataDog/dd-trace-java#9115) - [@jpbempel](https://github.com/jpbempel)) - ✨ Add java.util.Date support ([#9111](DataDog/dd-trace-java#9111) - [@jpbempel](https://github.com/jpbempel)) - ✨ Update file probe format ([#9047](DataDog/dd-trace-java#9047) - [@jpbempel](https://github.com/jpbempel)) - ✨ add safe local var hoisting ([#9034](DataDog/dd-trace-java#9034) - [@jpbempel](https://github.com/jpbempel)) - 🧹 Add new config for debugger upload interval ([#8959](DataDog/dd-trace-java#8959) - [@jpbempel](https://github.com/jpbempel)) - ✨ Enable Code Origin with Dynamic instrumentation ([#8940](DataDog/dd-trace-java#8940) - [@jpbempel](https://github.com/jpbempel)) #### ML Observability (LLMObs) - 💡 LLM Observability SDK ([#8781](DataDog/dd-trace-java#8781) - [@gary-huang](https://github.com/gary-huang), [@nayeem-kamal](https://github.com/nayeem-kamal)) #### Metrics - 🐛 Ensure client stat reporter is started when the agent is not available at bootstrap ([#9082](DataDog/dd-trace-java#9082) - [@amarziali](https://github.com/amarziali)) - ✨ Create metric: appsec.waf.config\_errors ([#8394](DataDog/dd-trace-java#8394) - [@sezen-datadog](https://github.com/sezen-datadog)) #### Platform components - ✨ Introduce environment component ([#9071](DataDog/dd-trace-java#9071) - [@PerfectSlayer](https://github.com/PerfectSlayer)) #### Profiling - 🐛 Remove annoying warning for smap event parsing ([#9119](DataDog/dd-trace-java#9119) - [@jbachorik](https://github.com/jbachorik)) - 🐛 Fix ByteCountingInputStream when reading past EOF ([#8988](DataDog/dd-trace-java#8988) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) #### Realtime User Monitoring - ✨ Add RUM SDK injection for servlet based web servers ([#9110](DataDog/dd-trace-java#9110) - [@PerfectSlayer](https://github.com/PerfectSlayer) [@amarziali](https://github.com/amarziali)) #### Telemetry - ✨ Update the config origin metric to match what it's mapping ([#9045](DataDog/dd-trace-java#9045) - [@sezen-datadog](https://github.com/sezen-datadog)) #### Testing - ✨ Add testing for latest stable version (JDK 24) ([#8875](DataDog/dd-trace-java#8875) - [@sarahchen6](https://github.com/sarahchen6)) #### Trace context propagation - 🐛 Fix bug with dropping baggage when `TracePropagationBehaviorExtract=IGNORE` ([#9037](DataDog/dd-trace-java#9037) - [@mhlidd](https://github.com/mhlidd)) - 🐛 Fix ArrayIndexOutOfBoundsException in PercentEscaper ([#9032](DataDog/dd-trace-java#9032) - [@mhlidd](https://github.com/mhlidd)) #### Tracer core - 🐛 Fix `Error` handling for trace interceptors ([#9097](DataDog/dd-trace-java#9097) - [@AlexeyKuznetsov-DD](https://github.com/AlexeyKuznetsov-DD)) - 💡 Add wildcard feature for `DD_TRACE_HEADER_TAGS` and enabling for Http Response headers ([#9067](DataDog/dd-trace-java#9067) - [@mhlidd](https://github.com/mhlidd)) #### Tracer public API - 💡 Add LLM Observability SDK ([#8781](DataDog/dd-trace-java#8781) - [@gary-huang](https://github.com/gary-huang)) ### Instrumentations #### Akka instrumentation - 🐛 Fix NPE in akka-http and pekko-http integrations ([#9019](DataDog/dd-trace-java#9019) - [@mcculls](https://github.com/mcculls)) #### Eclipse Vert.x instrumentation - ✨ Extract Vert.x json body response schemas ([#9001](DataDog/dd-trace-java#9001) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Write http.route tag as soon as possible in vert.x ([#8952](DataDog/dd-trace-java#8952) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) #### JAX-WS instrumentation - 💡⚠️ Enable jax-ws integration by default ([#9030](DataDog/dd-trace-java#9030) - [@bm1549](https://github.com/bm1549)) - ✨ Extract Jersey json body response schemas ([#9014](DataDog/dd-trace-java#9014) - [@jandro996](https://github.com/jandro996)) #### Mule instrumentation - 🐛 Propagate grizzly http span in filters if nothing is active ([#9016](DataDog/dd-trace-java#9016) - [@amarziali](https://github.com/amarziali)) #### Play Framework instrumentation - ✨ Extract Play json body response schemas ([#8995](DataDog/dd-trace-java#8995) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) #### Ratpack instrumentation - ✨ Extract Ratpack json body response schemas ([#9013](DataDog/dd-trace-java#9013) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) #### Spring instrumentation - ✨ Extract Spring json body response schemas ([#8938](DataDog/dd-trace-java#8938) - [@sezen-datadog](https://github.com/sezen-datadog)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - "after 6pm every weekday,before 2am every weekday" in timezone Australia/Melbourne, Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Never, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). GitOrigin-RevId: 649b690d4c9d7dcb572c457f0802b42b8e3e682e

gary-huang force-pushed the gary/llmobs-sdk-merge branch from 896f40a to 2be1b73 Compare May 14, 2025 18:23

gary-huang changed the title ~~[wip] LLM Observability SDK merge branch~~ LLM Observability SDK Jul 9, 2025

nayeem-kamal approved these changes Jul 9, 2025

View reviewed changes

gary-huang marked this pull request as ready for review July 9, 2025 12:28

gary-huang requested review from a team as code owners July 9, 2025 12:28

gary-huang requested review from daniel-mohedano, bric3 and ygree and removed request for a team July 9, 2025 12:28

gary-huang and others added 7 commits July 9, 2025 08:33

add APIs for llm obs sdk (#8135)

ed68795

* add APIs for llm obs * add llm message class to support llm spans * follow java convention of naming Id instead of ID * add codeowners

updated to master submodule

e151520

fix CODEOWNERS

0cf7162

gary-huang force-pushed the gary/llmobs-sdk-merge branch from 2befa1c to 0cf7162 Compare July 9, 2025 12:33

gary-huang added the comp: mlobs ML Observability (LLMObs) label Jul 9, 2025

nayeem-kamal added type: feature request comp: api Tracer public API and removed comp: mlobs ML Observability (LLMObs) labels Jul 9, 2025

nayeem-kamal enabled auto-merge (squash) July 9, 2025 12:35

nayeem-kamal added the comp: mlobs ML Observability (LLMObs) label Jul 9, 2025

nayeem-kamal merged commit 1126e5e into master Jul 9, 2025
510 of 512 checks passed

nayeem-kamal deleted the gary/llmobs-sdk-merge branch July 9, 2025 13:33

github-actions bot added this to the 1.51.0 milestone Jul 9, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

LLM Observability SDK #8781

LLM Observability SDK #8781

Uh oh!

gary-huang commented May 7, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented May 7, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

github-actions bot commented Jul 9, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

LLM Observability SDK #8781

LLM Observability SDK #8781

Uh oh!

Conversation

gary-huang commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

pr-commenter bot commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

github-actions bot commented Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gary-huang commented May 7, 2025 •

edited

Loading

pr-commenter bot commented May 7, 2025 •

edited

Loading

github-actions bot commented Jul 9, 2025 •

edited

Loading