implement LLM Obs SDK spans APIs #8390

gary-huang · 2025-02-14T11:41:26Z

What Does This Do

adds the implementation for llm obs spans

see #8135 for list of APIs that is implemented and usage

Motivation

Additional Notes

to be merged to the branch gary/llmobs-sdk-merge after #8135 has been merged

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: MLOB-1462

pr-commenter · 2025-02-19T15:35:03Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	gary/impl-llm-obs-api
git_commit_date	1747229243	1747255060
git_commit_sha	`7c80dbe`	`9ac1ab7`
release_version	1.50.0-SNAPSHOT~7c80dbede3	1.50.0-SNAPSHOT~9ac1ab7bb3

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1747257386	1747257386
ci_job_id	938959489	938959489
ci_pipeline_id	65134690	65134690
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-lrzlpyic-project-304-concurrent-0-a840qpmp 6.8.0-1028-aws #30~22.04.1-Ubuntu SMP Sun Apr 20 06:03:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-lrzlpyic-project-304-concurrent-0-a840qpmp 6.8.0-1028-aws #30~22.04.1-Ubuntu SMP Sun Apr 20 06:03:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 56 metrics, 15 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.50.0-SNAPSHOT~9ac1ab7bb3, baseline=1.50.0-SNAPSHOT~7c80dbede3

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.03 s) : 0, 1030479
Total [baseline] (8.659 s) : 0, 8658593
Agent [candidate] (1.019 s) : 0, 1019059
Total [candidate] (8.694 s) : 0, 8694280
section iast
Agent [baseline] (1.157 s) : 0, 1156518
Total [baseline] (9.221 s) : 0, 9220981
Agent [candidate] (1.15 s) : 0, 1150363
Total [candidate] (9.217 s) : 0, 9217346
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.156 s) : 0, 1156425
Total [baseline] (9.206 s) : 0, 9206299
Agent [candidate] (1.148 s) : 0, 1148482
Total [candidate] (9.203 s) : 0, 9203413
section iast_TELEMETRY_OFF
Agent [baseline] (1.144 s) : 0, 1144475
Total [baseline] (9.209 s) : 0, 9209201
Agent [candidate] (1.153 s) : 0, 1153135
Total [candidate] (9.27 s) : 0, 9269797

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.03 s	-
Agent	iast	1.157 s	126.039 ms (12.2%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.156 s	125.946 ms (12.2%)
Agent	iast_TELEMETRY_OFF	1.144 s	113.996 ms (11.1%)
Total	tracing	8.659 s	-
Total	iast	9.221 s	562.388 ms (6.5%)
Total	iast_HARDCODED_SECRET_DISABLED	9.206 s	547.706 ms (6.3%)
Total	iast_TELEMETRY_OFF	9.209 s	550.608 ms (6.4%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.019 s	-
Agent	iast	1.15 s	131.303 ms (12.9%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.148 s	129.422 ms (12.7%)
Agent	iast_TELEMETRY_OFF	1.153 s	134.076 ms (13.2%)
Total	tracing	8.694 s	-
Total	iast	9.217 s	523.066 ms (6.0%)
Total	iast_HARDCODED_SECRET_DISABLED	9.203 s	509.133 ms (5.9%)
Total	iast_TELEMETRY_OFF	9.27 s	575.517 ms (6.6%)

gantt
    title insecure-bank - break down per module: candidate=1.50.0-SNAPSHOT~9ac1ab7bb3, baseline=1.50.0-SNAPSHOT~7c80dbede3

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (688.439 ms) : 0, 688439
BytebuddyAgent [candidate] (682.084 ms) : 0, 682084
GlobalTracer [baseline] (242.121 ms) : 0, 242121
GlobalTracer [candidate] (240.348 ms) : 0, 240348
AppSec [baseline] (55.552 ms) : 0, 55552
AppSec [candidate] (54.325 ms) : 0, 54325
Debugger [baseline] (7.819 ms) : 0, 7819
Debugger [candidate] (8.307 ms) : 0, 8307
Remote Config [baseline] (707.504 µs) : 0, 708
Remote Config [candidate] (700.531 µs) : 0, 701
Telemetry [baseline] (12.177 ms) : 0, 12177
Telemetry [candidate] (9.766 ms) : 0, 9766
section iast
BytebuddyAgent [baseline] (807.181 ms) : 0, 807181
BytebuddyAgent [candidate] (803.054 ms) : 0, 803054
GlobalTracer [baseline] (231.508 ms) : 0, 231508
GlobalTracer [candidate] (230.959 ms) : 0, 230959
AppSec [baseline] (50.936 ms) : 0, 50936
AppSec [candidate] (49.407 ms) : 0, 49407
Debugger [baseline] (5.941 ms) : 0, 5941
Debugger [candidate] (5.842 ms) : 0, 5842
Remote Config [baseline] (599.613 µs) : 0, 600
Remote Config [candidate] (574.032 µs) : 0, 574
Telemetry [baseline] (7.908 ms) : 0, 7908
Telemetry [candidate] (7.819 ms) : 0, 7819
IAST [baseline] (28.817 ms) : 0, 28817
IAST [candidate] (29.181 ms) : 0, 29181
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (807.02 ms) : 0, 807020
BytebuddyAgent [candidate] (801.163 ms) : 0, 801163
GlobalTracer [baseline] (231.807 ms) : 0, 231807
GlobalTracer [candidate] (230.673 ms) : 0, 230673
AppSec [baseline] (50.587 ms) : 0, 50587
AppSec [candidate] (47.794 ms) : 0, 47794
Debugger [baseline] (6.038 ms) : 0, 6038
Debugger [candidate] (5.92 ms) : 0, 5920
Remote Config [baseline] (596.951 µs) : 0, 597
Remote Config [candidate] (582.285 µs) : 0, 582
Telemetry [baseline] (8.023 ms) : 0, 8023
Telemetry [candidate] (7.843 ms) : 0, 7843
IAST [baseline] (28.709 ms) : 0, 28709
IAST [candidate] (30.98 ms) : 0, 30980
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (797.925 ms) : 0, 797925
BytebuddyAgent [candidate] (804.166 ms) : 0, 804166
GlobalTracer [baseline] (230.531 ms) : 0, 230531
GlobalTracer [candidate] (231.885 ms) : 0, 231885
AppSec [baseline] (53.656 ms) : 0, 53656
AppSec [candidate] (56.307 ms) : 0, 56307
Debugger [baseline] (5.961 ms) : 0, 5961
Debugger [candidate] (6.026 ms) : 0, 6026
Remote Config [baseline] (588.889 µs) : 0, 589
Remote Config [candidate] (610.909 µs) : 0, 611
Telemetry [baseline] (7.853 ms) : 0, 7853
Telemetry [candidate] (7.926 ms) : 0, 7926
IAST [baseline] (24.387 ms) : 0, 24387
IAST [candidate] (22.489 ms) : 0, 22489

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.50.0-SNAPSHOT~9ac1ab7bb3, baseline=1.50.0-SNAPSHOT~7c80dbede3

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.033 s) : 0, 1032621
Total [baseline] (10.533 s) : 0, 10533223
Agent [candidate] (1.02 s) : 0, 1020197
Total [candidate] (10.527 s) : 0, 10526882
section appsec
Agent [baseline] (1.168 s) : 0, 1167565
Total [baseline] (10.691 s) : 0, 10691116
Agent [candidate] (1.16 s) : 0, 1160265
Total [candidate] (10.698 s) : 0, 10697777
section iast
Agent [baseline] (1.158 s) : 0, 1157675
Total [baseline] (10.939 s) : 0, 10938911
Agent [candidate] (1.158 s) : 0, 1157562
Total [candidate] (10.893 s) : 0, 10892942
section profiling
Agent [baseline] (1.277 s) : 0, 1277358
Total [baseline] (10.874 s) : 0, 10873673
Agent [candidate] (1.277 s) : 0, 1276692
Total [candidate] (10.946 s) : 0, 10945691

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.033 s	-
Agent	appsec	1.168 s	134.944 ms (13.1%)
Agent	iast	1.158 s	125.054 ms (12.1%)
Agent	profiling	1.277 s	244.737 ms (23.7%)
Total	tracing	10.533 s	-
Total	appsec	10.691 s	157.894 ms (1.5%)
Total	iast	10.939 s	405.688 ms (3.9%)
Total	profiling	10.874 s	340.45 ms (3.2%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.02 s	-
Agent	appsec	1.16 s	140.067 ms (13.7%)
Agent	iast	1.158 s	137.364 ms (13.5%)
Agent	profiling	1.277 s	256.494 ms (25.1%)
Total	tracing	10.527 s	-
Total	appsec	10.698 s	170.895 ms (1.6%)
Total	iast	10.893 s	366.06 ms (3.5%)
Total	profiling	10.946 s	418.809 ms (4.0%)

gantt
    title petclinic - break down per module: candidate=1.50.0-SNAPSHOT~9ac1ab7bb3, baseline=1.50.0-SNAPSHOT~7c80dbede3

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (690.61 ms) : 0, 690610
BytebuddyAgent [candidate] (683.223 ms) : 0, 683223
GlobalTracer [baseline] (242.589 ms) : 0, 242589
GlobalTracer [candidate] (240.616 ms) : 0, 240616
AppSec [baseline] (54.957 ms) : 0, 54957
AppSec [candidate] (54.584 ms) : 0, 54584
Debugger [baseline] (7.765 ms) : 0, 7765
Debugger [candidate] (6.927 ms) : 0, 6927
Remote Config [baseline] (719.629 µs) : 0, 720
Remote Config [candidate] (705.402 µs) : 0, 705
Telemetry [baseline] (12.183 ms) : 0, 12183
Telemetry [candidate] (10.585 ms) : 0, 10585
section appsec
BytebuddyAgent [baseline] (704.354 ms) : 0, 704354
BytebuddyAgent [candidate] (699.982 ms) : 0, 699982
GlobalTracer [baseline] (238.2 ms) : 0, 238200
GlobalTracer [candidate] (236.782 ms) : 0, 236782
AppSec [baseline] (176.432 ms) : 0, 176432
AppSec [candidate] (174.853 ms) : 0, 174853
Debugger [baseline] (5.975 ms) : 0, 5975
Debugger [candidate] (5.938 ms) : 0, 5938
Remote Config [baseline] (635.779 µs) : 0, 636
Remote Config [candidate] (624.114 µs) : 0, 624
Telemetry [baseline] (7.438 ms) : 0, 7438
Telemetry [candidate] (7.766 ms) : 0, 7766
IAST [baseline] (21.807 ms) : 0, 21807
IAST [candidate] (21.619 ms) : 0, 21619
section iast
BytebuddyAgent [baseline] (807.925 ms) : 0, 807925
BytebuddyAgent [candidate] (808.747 ms) : 0, 808747
GlobalTracer [baseline] (231.933 ms) : 0, 231933
GlobalTracer [candidate] (232.022 ms) : 0, 232022
AppSec [baseline] (52.401 ms) : 0, 52401
AppSec [candidate] (50.306 ms) : 0, 50306
Debugger [baseline] (5.967 ms) : 0, 5967
Debugger [candidate] (5.869 ms) : 0, 5869
Remote Config [baseline] (598.966 µs) : 0, 599
Remote Config [candidate] (572.387 µs) : 0, 572
Telemetry [baseline] (7.911 ms) : 0, 7911
Telemetry [candidate] (7.799 ms) : 0, 7799
IAST [baseline] (27.215 ms) : 0, 27215
IAST [candidate] (28.591 ms) : 0, 28591
section profiling
BytebuddyAgent [baseline] (672.687 ms) : 0, 672687
BytebuddyAgent [candidate] (672.291 ms) : 0, 672291
GlobalTracer [baseline] (373.991 ms) : 0, 373991
GlobalTracer [candidate] (373.671 ms) : 0, 373671
AppSec [baseline] (61.822 ms) : 0, 61822
AppSec [candidate] (61.885 ms) : 0, 61885
Debugger [baseline] (6.26 ms) : 0, 6260
Debugger [candidate] (6.308 ms) : 0, 6308
Remote Config [baseline] (653.126 µs) : 0, 653
Remote Config [candidate] (652.647 µs) : 0, 653
Telemetry [baseline] (8.239 ms) : 0, 8239
Telemetry [candidate] (8.207 ms) : 0, 8207
ProfilingAgent [baseline] (102.858 ms) : 0, 102858
ProfilingAgent [candidate] (103.033 ms) : 0, 103033
Profiling [baseline] (102.882 ms) : 0, 102882
Profiling [candidate] (103.057 ms) : 0, 103057

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-05-14T20:46:45	2025-05-14T20:54:31
git_branch	master	gary/impl-llm-obs-api
git_commit_date	1747229243	1747255060
git_commit_sha	`7c80dbe`	`9ac1ab7`
release_version	1.50.0-SNAPSHOT~7c80dbede3	1.50.0-SNAPSHOT~9ac1ab7bb3
start_time	2025-05-14T20:46:30	2025-05-14T20:54:16

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1747256472	1747256472
ci_job_id	938959490	938959490
ci_pipeline_id	65134690	65134690
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-lrzlpyic-project-304-concurrent-1-yl118kr7 6.8.0-1028-aws #30~22.04.1-Ubuntu SMP Sun Apr 20 06:03:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-lrzlpyic-project-304-concurrent-1-yl118kr7 6.8.0-1028-aws #30~22.04.1-Ubuntu SMP Sun Apr 20 06:03:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 13 metrics, 17 unstable metrics.

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.50.0-SNAPSHOT~9ac1ab7bb3, baseline=1.50.0-SNAPSHOT~7c80dbede3
    dateFormat X
    axisFormat %s
section baseline
no_agent (386.557 µs) : 367, 406
.   : milestone, 387,
iast (518.368 µs) : 497, 540
.   : milestone, 518,
iast_FULL (737.969 µs) : 716, 760
.   : milestone, 738,
iast_GLOBAL (568.239 µs) : 546, 590
.   : milestone, 568,
iast_HARDCODED_SECRET_DISABLED (524.218 µs) : 501, 547
.   : milestone, 524,
iast_INACTIVE (469.005 µs) : 447, 491
.   : milestone, 469,
iast_TELEMETRY_OFF (511.429 µs) : 488, 535
.   : milestone, 511,
tracing (475.972 µs) : 453, 499
.   : milestone, 476,
section candidate
no_agent (383.191 µs) : 364, 403
.   : milestone, 383,
iast (525.068 µs) : 503, 547
.   : milestone, 525,
iast_FULL (741.278 µs) : 719, 763
.   : milestone, 741,
iast_GLOBAL (558.341 µs) : 537, 580
.   : milestone, 558,
iast_HARDCODED_SECRET_DISABLED (540.176 µs) : 516, 564
.   : milestone, 540,
iast_INACTIVE (479.88 µs) : 459, 501
.   : milestone, 480,
iast_TELEMETRY_OFF (524.384 µs) : 501, 548
.   : milestone, 524,
tracing (465.813 µs) : 443, 488
.   : milestone, 466,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	386.557 µs [367.1 µs, 406.014 µs]	-
iast	518.368 µs [496.552 µs, 540.185 µs]	131.811 µs (34.1%)
iast_FULL	737.969 µs [716.121 µs, 759.816 µs]	351.411 µs (90.9%)
iast_GLOBAL	568.239 µs [546.336 µs, 590.141 µs]	181.682 µs (47.0%)
iast_HARDCODED_SECRET_DISABLED	524.218 µs [501.367 µs, 547.07 µs]	137.661 µs (35.6%)
iast_INACTIVE	469.005 µs [446.782 µs, 491.229 µs]	82.448 µs (21.3%)
iast_TELEMETRY_OFF	511.429 µs [487.919 µs, 534.94 µs]	124.872 µs (32.3%)
tracing	475.972 µs [453.325 µs, 498.619 µs]	89.415 µs (23.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	383.191 µs [363.543 µs, 402.838 µs]	-
iast	525.068 µs [502.912 µs, 547.225 µs]	141.878 µs (37.0%)
iast_FULL	741.278 µs [719.188 µs, 763.367 µs]	358.087 µs (93.4%)
iast_GLOBAL	558.341 µs [536.74 µs, 579.941 µs]	175.15 µs (45.7%)
iast_HARDCODED_SECRET_DISABLED	540.176 µs [516.3 µs, 564.052 µs]	156.985 µs (41.0%)
iast_INACTIVE	479.88 µs [458.573 µs, 501.187 µs]	96.689 µs (25.2%)
iast_TELEMETRY_OFF	524.384 µs [501.171 µs, 547.596 µs]	141.193 µs (36.8%)
tracing	465.813 µs [443.334 µs, 488.292 µs]	82.622 µs (21.6%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.50.0-SNAPSHOT~9ac1ab7bb3, baseline=1.50.0-SNAPSHOT~7c80dbede3
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.364 ms) : 1344, 1385
.   : milestone, 1364,
appsec (1.737 ms) : 1713, 1761
.   : milestone, 1737,
appsec_no_iast (1.732 ms) : 1709, 1755
.   : milestone, 1732,
code_origins (1.674 ms) : 1648, 1700
.   : milestone, 1674,
iast (1.529 ms) : 1504, 1553
.   : milestone, 1529,
profiling (1.561 ms) : 1536, 1586
.   : milestone, 1561,
tracing (1.509 ms) : 1485, 1534
.   : milestone, 1509,
section candidate
no_agent (1.374 ms) : 1355, 1394
.   : milestone, 1374,
appsec (1.734 ms) : 1711, 1758
.   : milestone, 1734,
appsec_no_iast (1.744 ms) : 1721, 1768
.   : milestone, 1744,
code_origins (1.693 ms) : 1666, 1720
.   : milestone, 1693,
iast (1.53 ms) : 1506, 1554
.   : milestone, 1530,
profiling (1.514 ms) : 1490, 1538
.   : milestone, 1514,
tracing (1.502 ms) : 1477, 1527
.   : milestone, 1502,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.364 ms [1.344 ms, 1.385 ms]	-
appsec	1.737 ms [1.713 ms, 1.761 ms]	373.072 µs (27.3%)
appsec_no_iast	1.732 ms [1.709 ms, 1.755 ms]	367.782 µs (27.0%)
code_origins	1.674 ms [1.648 ms, 1.7 ms]	309.79 µs (22.7%)
iast	1.529 ms [1.504 ms, 1.553 ms]	164.423 µs (12.1%)
profiling	1.561 ms [1.536 ms, 1.586 ms]	196.628 µs (14.4%)
tracing	1.509 ms [1.485 ms, 1.534 ms]	145.198 µs (10.6%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.374 ms [1.355 ms, 1.394 ms]	-
appsec	1.734 ms [1.711 ms, 1.758 ms]	359.989 µs (26.2%)
appsec_no_iast	1.744 ms [1.721 ms, 1.768 ms]	370.038 µs (26.9%)
code_origins	1.693 ms [1.666 ms, 1.72 ms]	318.619 µs (23.2%)
iast	1.53 ms [1.506 ms, 1.554 ms]	155.595 µs (11.3%)
profiling	1.514 ms [1.49 ms, 1.538 ms]	139.514 µs (10.2%)
tracing	1.502 ms [1.477 ms, 1.527 ms]	127.776 µs (9.3%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	gary/impl-llm-obs-api
git_commit_date	1747229243	1747255060
git_commit_sha	`7c80dbe`	`9ac1ab7`
release_version	1.50.0-SNAPSHOT~7c80dbede3	1.50.0-SNAPSHOT~9ac1ab7bb3

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1747256966	1747256966
ci_job_id	938959491	938959491
ci_pipeline_id	65134690	65134690
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-6ddtk55q-project-304-concurrent-0-4118mn8t 6.8.0-1028-aws #30~22.04.1-Ubuntu SMP Sun Apr 20 06:03:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-6ddtk55q-project-304-concurrent-0-4118mn8t 6.8.0-1028-aws #30~22.04.1-Ubuntu SMP Sun Apr 20 06:03:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~9ac1ab7bb3, baseline=1.50.0-SNAPSHOT~7c80dbede3
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.475 ms) : 1464, 1487
.   : milestone, 1475,
appsec (2.424 ms) : 2374, 2475
.   : milestone, 2424,
iast (2.199 ms) : 2136, 2262
.   : milestone, 2199,
iast_GLOBAL (2.236 ms) : 2173, 2299
.   : milestone, 2236,
profiling (2.054 ms) : 2003, 2105
.   : milestone, 2054,
tracing (2.03 ms) : 1981, 2080
.   : milestone, 2030,
section candidate
no_agent (1.479 ms) : 1468, 1491
.   : milestone, 1479,
appsec (2.417 ms) : 2367, 2467
.   : milestone, 2417,
iast (2.192 ms) : 2129, 2255
.   : milestone, 2192,
iast_GLOBAL (2.254 ms) : 2190, 2318
.   : milestone, 2254,
profiling (2.037 ms) : 1987, 2087
.   : milestone, 2037,
tracing (2.022 ms) : 1973, 2070
.   : milestone, 2022,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.475 ms [1.464 ms, 1.487 ms]	-
appsec	2.424 ms [2.374 ms, 2.475 ms]	949.044 µs (64.3%)
iast	2.199 ms [2.136 ms, 2.262 ms]	723.824 µs (49.1%)
iast_GLOBAL	2.236 ms [2.173 ms, 2.299 ms]	760.544 µs (51.6%)
profiling	2.054 ms [2.003 ms, 2.105 ms]	578.258 µs (39.2%)
tracing	2.03 ms [1.981 ms, 2.08 ms]	554.849 µs (37.6%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.479 ms [1.468 ms, 1.491 ms]	-
appsec	2.417 ms [2.367 ms, 2.467 ms]	937.589 µs (63.4%)
iast	2.192 ms [2.129 ms, 2.255 ms]	712.932 µs (48.2%)
iast_GLOBAL	2.254 ms [2.19 ms, 2.318 ms]	774.764 µs (52.4%)
profiling	2.037 ms [1.987 ms, 2.087 ms]	558.092 µs (37.7%)
tracing	2.022 ms [1.973 ms, 2.07 ms]	542.58 µs (36.7%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~9ac1ab7bb3, baseline=1.50.0-SNAPSHOT~7c80dbede3
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.734 s) : 14734000, 14734000
.   : milestone, 14734000,
appsec (14.678 s) : 14678000, 14678000
.   : milestone, 14678000,
iast (18.444 s) : 18444000, 18444000
.   : milestone, 18444000,
iast_GLOBAL (18.092 s) : 18092000, 18092000
.   : milestone, 18092000,
profiling (15.043 s) : 15043000, 15043000
.   : milestone, 15043000,
tracing (14.982 s) : 14982000, 14982000
.   : milestone, 14982000,
section candidate
no_agent (15.358 s) : 15358000, 15358000
.   : milestone, 15358000,
appsec (14.79 s) : 14790000, 14790000
.   : milestone, 14790000,
iast (18.839 s) : 18839000, 18839000
.   : milestone, 18839000,
iast_GLOBAL (17.602 s) : 17602000, 17602000
.   : milestone, 17602000,
profiling (15.089 s) : 15089000, 15089000
.   : milestone, 15089000,
tracing (15.169 s) : 15169000, 15169000
.   : milestone, 15169000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.734 s [14.734 s, 14.734 s]	-
appsec	14.678 s [14.678 s, 14.678 s]	-56.0 ms (-0.4%)
iast	18.444 s [18.444 s, 18.444 s]	3.71 s (25.2%)
iast_GLOBAL	18.092 s [18.092 s, 18.092 s]	3.358 s (22.8%)
profiling	15.043 s [15.043 s, 15.043 s]	309.0 ms (2.1%)
tracing	14.982 s [14.982 s, 14.982 s]	248.0 ms (1.7%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.358 s [15.358 s, 15.358 s]	-
appsec	14.79 s [14.79 s, 14.79 s]	-568.0 ms (-3.7%)
iast	18.839 s [18.839 s, 18.839 s]	3.481 s (22.7%)
iast_GLOBAL	17.602 s [17.602 s, 17.602 s]	2.244 s (14.6%)
profiling	15.089 s [15.089 s, 15.089 s]	-269.0 ms (-1.8%)
tracing	15.169 s [15.169 s, 15.169 s]	-189.0 ms (-1.2%)

github-actions · 2025-02-20T21:18:04Z

Hi! 👋 Looks like you updated a Git Submodule.
If this was not intentional please make sure to:

Update the submodule to the latest commit on the master branch using the git submodule update command,
Check you properly set up your environment for contributing.

gary-huang · 2025-06-04T19:35:44Z

The startup benchmarks show a lot of regressions - has anyone looked into why that might be?

merging master fixed this

* add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * spotless * add APIs for llm obs sdk (#8135) * add APIs for llm obs * add llm message class to support llm spans * follow java convention of naming Id instead of ID * add codeowners * rename ID to Id according to java naming conventions * Undo change to integrations-core submodule * fix build gradle * rm empty line * fix test

* add APIs for llm obs sdk (#8135) * add APIs for llm obs * add llm message class to support llm spans * follow java convention of naming Id instead of ID * add codeowners * implement LLM Obs SDK spans APIs (#8390) * add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * spotless * add APIs for llm obs sdk (#8135) * add APIs for llm obs * add llm message class to support llm spans * follow java convention of naming Id instead of ID * add codeowners * rename ID to Id according to java naming conventions * Undo change to integrations-core submodule * fix build gradle * rm empty line * fix test * LLM Obs SDK Mapper (#8372) * add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * impl llmobs agent and llmobs apis * impl llmobs agent * working writer * add support for llm message and tool calls * cleaned up whitespace * resolve merge conflicts * remaining merge conflicts * fix bad method call * fixed llmobs intake creation if llmobs not enabled * removed print statements * added tests for llmobsspanmapper * fixed coverage for tags --------- Co-authored-by: Nayeem Kamal <[email protected]> * updated to master submodule * LLM Obs SDK use context API for parent children span linkage (#8711) * add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * impl llmobs agent and llmobs apis * impl llmobs agent * working writer * add support for llm message and tool calls * impl llmobs agent and llmobs apis * use new ctx api to track parent span * cleaned up whitespace * resolve merge conflicts * remaining merge conflicts * fix bad method call * fixed llmobs intake creation if llmobs not enabled * removed print statements * ran spotless * added tests for llmobsspanmapper * fixed coverage for tags --------- Co-authored-by: Nayeem Kamal <[email protected]> Co-authored-by: Nayeem Kamal <[email protected]> * LLM Obs SDK evaluation metrics submission (#8688) * add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * impl llmobs agent and llmobs apis * impl llmobs agent * working writer * add support for llm message and tool calls * impl llmobs agent and llmobs apis * use new ctx api to track parent span * add api for evals * working impl supporting both agentless and agent * handle null tags and default to default ml app if null or empty string provided in the override * cleaned up whitespace * resolve merge conflicts * remaining merge conflicts * fix bad method call * fixed llmobs intake creation if llmobs not enabled * removed print statements * ran spotless * ran spotless * added tests for llmobsspanmapper * fixed coverage for tags --------- Co-authored-by: Nayeem Kamal <[email protected]> Co-authored-by: Nayeem Kamal <[email protected]> * fix CODEOWNERS --------- Co-authored-by: Nayeem Kamal <[email protected]> Co-authored-by: Nayeem Kamal <[email protected]>

gary-huang force-pushed the gary/impl-llm-obs-api branch from 3bcbe81 to c437b9c Compare February 19, 2025 14:47

gary-huang force-pushed the gary/impl-llm-obs-api branch 3 times, most recently from 871d358 to 180ccf5 Compare February 20, 2025 21:17

gary-huang changed the base branch from master to gary/add-llm-obs-api February 20, 2025 21:18

gary-huang force-pushed the gary/impl-llm-obs-api branch 5 times, most recently from 94136c5 to 7064f57 Compare February 20, 2025 23:13

gary-huang force-pushed the gary/impl-llm-obs-api branch 2 times, most recently from 9cb3e0a to 76a4b2f Compare March 5, 2025 19:00

gary-huang force-pushed the gary/add-llm-obs-api branch from 77ddec8 to cb95fbb Compare March 5, 2025 19:04

gary-huang force-pushed the gary/impl-llm-obs-api branch 5 times, most recently from a55367c to 40dcdfe Compare March 6, 2025 20:51

gary-huang force-pushed the gary/add-llm-obs-api branch from 7f41706 to a9ec5dd Compare March 6, 2025 21:05

gary-huang force-pushed the gary/impl-llm-obs-api branch from 40dcdfe to acde475 Compare March 6, 2025 21:07

gary-huang force-pushed the gary/add-llm-obs-api branch from a9ec5dd to c75df4b Compare March 6, 2025 23:01

gary-huang force-pushed the gary/impl-llm-obs-api branch from acde475 to 829ffb1 Compare March 7, 2025 00:00

gary-huang force-pushed the gary/add-llm-obs-api branch 2 times, most recently from c1f63dc to 6e250a2 Compare March 7, 2025 13:34

gary-huang force-pushed the gary/impl-llm-obs-api branch 4 times, most recently from d548421 to d9a57b3 Compare March 7, 2025 14:06

gary-huang requested review from a team as code owners May 14, 2025 18:18

gary-huang requested review from smola, jandro996, jpbempel and tonyredondo and removed request for a team May 14, 2025 18:18

Undo change to integrations-core submodule

c1c38e4

gary-huang force-pushed the gary/llmobs-sdk-merge branch from 896f40a to 2be1b73 Compare May 14, 2025 18:23

gary-huang removed request for a team, smola, jandro996, jpbempel and tonyredondo May 14, 2025 18:23

gary-huang added 4 commits May 14, 2025 15:54

fix build gradle

d01034a

Merge branch 'gary/llmobs-sdk-merge' into gary/impl-llm-obs-api

7a74cf6

rm empty line

23d6923

fix test

9ac1ab7

gary-huang merged commit 77291d8 into gary/llmobs-sdk-merge Jun 4, 2025
516 of 520 checks passed

gary-huang deleted the gary/impl-llm-obs-api branch June 4, 2025 19:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

implement LLM Obs SDK spans APIs #8390

implement LLM Obs SDK spans APIs #8390

Uh oh!

gary-huang commented Feb 14, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented Feb 19, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Feb 20, 2025

Uh oh!

gary-huang commented Jun 4, 2025

Uh oh!

Uh oh!

Uh oh!

implement LLM Obs SDK spans APIs #8390

implement LLM Obs SDK spans APIs #8390

Uh oh!

Conversation

gary-huang commented Feb 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

pr-commenter bot commented Feb 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

github-actions bot commented Feb 20, 2025

Uh oh!

gary-huang commented Jun 4, 2025

Uh oh!

Uh oh!

Uh oh!

gary-huang commented Feb 14, 2025 •

edited

Loading

pr-commenter bot commented Feb 19, 2025 •

edited

Loading