V3.29.0 proposal #3427

szegedi · 2023-07-20T14:46:13Z

Features

profiling: Associate tracing span IDs with samples in the wall profiler ([profiling] Associate tracing span IDs with samples in the wall profiler #3371)
iast: Taint request URI (Taint request URI #3302)
iast: Detect missing header vulnerabilities (Detect missing header vulnerabilities #3269)
graphql: implement naming schema for graphql plugin (implement naming schema for graphql plugin #3279)

Enhancements

openai: be defensive about response object shape (openai: be defensive about response object shape #3408)
tracing: Ensure OTel span name is copied to resource.name (Ensure OTel span name is copied to resource.name #3412)
config: Fix service representation config (Fix service representation config #3419)
couchbase: Re-enable Couchbase testing (Re-enable Couchbase testing #3265)
ci: Add BITBUCKET_GIT_HTTP_ORIGIN env variable (Add BITBUCKET_GIT_HTTP_ORIGIN env variable #3401)

Bug fixes

serverless: Handle case where Lambda handler is using callbacks instead of promises (fix: Handle case where Lambda handler is using callbacks instead of p… #3404)
ci-visibility: Do not modify how jest reports timeouts ([ci-visibility] Do not modify how jest reports timeouts #3399)
profiling: Allow https datadog agent url in continuous profiling (Continuous Profiler doesn't work with a datadog agent serving on HTTPS #3190)
tracing: Fix distributed tracing issue caused by single span ingestion changes (Fix distributed tracing issue caused by single span ingestion changes #3413)
tracing: Make datadog distributed tracing header take priority over tracecontext (Make datadog distributed tracing header take priority over tracecontext #3414)
tracing: Exclude empty telemetry metrics (Exclude empty telemetry metrics #3421)
openai: make services entirely optional when initializing (openai: make services entirely optional when initializing #3420)

Minor stuff (probably not notable for release notes)

profiling: Fix erroneous resetting of profiler state to undefined (Fix erroneous resetting of profiler state to undefined #3394)
Update yarn.lock with current pprof-nodejs version ([profiling] Update yarn.lock with current pprof-nodejs version #3398)
delete old build scripts (delete old build scripts #3396)
Add eslint rule for object-curly-newline (Add eslint rule for object-curly-newline #3406)
Fix a phrase in README (Fix a phrase in README #3405)
Fix same dependency detection (Fix same dependency detection #3386)
Ignore esbuild output from integration tests in esbuild (Ignore esbuild output from integration tests in esbuild #3410)
Fix flaky serverless integration test (Fix flaky serverless integration test #3418)
Fix native-iast-taint-tracking version number (Fix native-iast-taint-tracking version number #3433)
openai: remove String#replaceAll for Node.js <=14 (openai: remove String#replaceAll for Node.js <=14 #3432)
Make integration test for jest compatible with node 12 ([ci-visibility] Make integration test for jest compatible with node 12 #3440)

…ler (#3371) * Initial code for code hotspots and endpoint aggregation in wall profiler Hide functionality behind DD_PROFILING_EXPERIMENTAL_CODEHOTSPOTS_ENABLED and DD_PROFILING_EXPERIMENTAL_ENDPOINT_COLLECTION_ENABLED flags. Profiler tracks changes of active span by subscribing to async hooks before channel and to AsyncResourceStorage enter channel that notifies when current store is changed with enterWith/run. Profiler remembers previous span and previous started spans, upon change of active span, it checks if a sample has been taken by native profiler addon, then if that's the case, it updates the context of the sample with span id and root span id computed from previous span / started spans. It also add the tags from the last web span to the context to allow computation of endpoint later on. Minor changes to other parts of the code to make it all work: === * Report error if start fails * Avoid clearing span context tags upon export Profiler may need the tags to determine span resource name for endpoint aggregation: profiler keeps a reference on web span that was active when each profiling sample is taken. Then during profile export, span tags are used to determine endpoint. Endpoint cannot be determined right away because sometimes tags necessary to determine endpoint are only set just before span is finished. * Add request tags just after span creation For web spans, add request tags just after span creation instead of when span finishes. This is needed for profiler code hotspots implementation to determine which span are web spans and should be used to find current endpoint.

* Taint request URI * Add check for safe tainted origins on unvalidated redirect analyzer * Change assertion construction for unvalidated redirect analyzer test * Add metric for uri sourcing * Fix PR comments

* Detect X-Content-Type-Options missing header * HSTS Header missing analyzer and refactor of xcontenttype-header-missing-aanalyzer * Move function to class method * use startswith instead of index===0 * Do not send evidence if value is undefined * Fix comment in PR and add test * Changes to support telemetry * Rename method name * Rename analyzer object key

Forgot to update yarn.lock when bumping pprof-nodejs version in a previous PR.

* Fix same dependency detection * Do not send same dependency:version twice * Clear map on stop * Update savedDependencies to savedDependenciesToSend to improve understanding

…romises (#3404)

Co-authored-by: Attila Szegedi <[email protected]>

…#3413)

…xt (#3414)

* address and fix edge cases * move to better tracer option name for service name flattening

* implement naming schema for graphql plugin * implement schema v1 and add tests --------- Co-authored-by: Thomas Hunter II <[email protected]> Co-authored-by: Jordi Bertran de Balanda <[email protected]>

github-actions · 2023-07-20T14:47:08Z

Overall package size

Self size: 4.91 MB
Deduped: 58.01 MB
No deduping: 58.1 MB

Dependency sizes

name	version	self size	total size
@datadog/native-iast-taint-tracking	1.5.0	14.86 MB	14.86 MB
@datadog/native-appsec	3.2.0	13.38 MB	13.39 MB
@datadog/pprof	3.1.0	10.66 MB	11.5 MB
protobufjs	7.2.4	2.74 MB	6.52 MB
@datadog/native-iast-rewriter	2.0.1	2.09 MB	2.1 MB
@opentelemetry/core	1.14.0	872.87 kB	1.47 MB
@datadog/native-metrics	2.0.0	898.77 kB	1.3 MB
@opentelemetry/api	1.4.1	780.32 kB	780.32 kB
msgpack-lite	0.1.26	201.16 kB	281.59 kB
opentracing	0.14.7	194.81 kB	194.81 kB
semver	7.5.3	93.39 kB	123.79 kB
@datadog/sketches-js	2.1.0	109.9 kB	109.9 kB
lodash.sortby	4.7.0	75.76 kB	75.76 kB
lru-cache	7.14.0	74.95 kB	74.95 kB
ipaddr.js	2.0.1	59.52 kB	59.52 kB
int64-buffer	0.1.10	49.18 kB	49.18 kB
ignore	5.2.0	48.87 kB	48.87 kB
import-in-the-middle	1.3.5	34.34 kB	38.81 kB
istanbul-lib-coverage	3.2.0	29.34 kB	29.34 kB
retry	0.10.1	27.44 kB	27.44 kB
lodash.uniq	4.5.0	25.01 kB	25.01 kB
limiter	1.1.5	23.17 kB	23.17 kB
lodash.kebabcase	4.1.1	17.75 kB	17.75 kB
lodash.pick	4.4.0	16.33 kB	16.33 kB
node-abort-controller	3.0.1	14.33 kB	14.33 kB
crypto-randomuuid	1.0.0	11.18 kB	11.18 kB
diagnostics_channel	1.1.0	7.07 kB	7.07 kB
path-to-regexp	0.1.7	6.78 kB	6.78 kB
koalas	1.0.2	6.47 kB	6.47 kB
methods	1.1.2	5.29 kB	5.29 kB
module-details-from-path	1.0.3	4.47 kB	4.47 kB

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

codecov · 2023-07-20T14:49:04Z

Codecov Report

❗ No coverage uploaded for pull request base (v3.x@51eabba). Click here to learn what that means.
The diff coverage is n/a.

❗ Current head ec246cb differs from pull request most recent head d71337c. Consider uploading reports for the commit d71337c to get more accurate results

@@           Coverage Diff           @@
##             v3.x    #3427   +/-   ##
=======================================
  Coverage        ?   84.50%           
=======================================
  Files           ?      218           
  Lines           ?     8572           
  Branches        ?       33           
=======================================
  Hits            ?     7244           
  Misses          ?     1328           
  Partials        ?        0

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

pr-commenter · 2023-07-21T07:44:46Z

Benchmarks

Benchmark execution time: 2023-07-21 13:09:00

Comparing candidate commit d71337c in PR branch v3.29.0-proposal with baseline commit 51eabba in branch v3.x.

Found 1 performance improvements and 33 performance regressions! Performance is the same for 675 metrics, 29 unstable metrics.

scenario:appsec-iast-no-vulnerability-control-14

🟥 cpu_usage_percentage [+4.627%; +8.296%]

scenario:appsec-iast-no-vulnerability-control-16

🟥 cpu_usage_percentage [+5.224%; +8.123%]

scenario:exporting-pipeline-0.4-14

🟥 cpu_user_time [+53.619ms; +59.568ms] or [+21.433%; +23.811%]
🟥 execution_time [+57.744ms; +62.154ms] or [+20.180%; +21.721%]
🟥 instructions [+122.3M instructions; +123.3M instructions] or [+16.388%; +16.511%]
🟥 max_rss_usage [+3.088KB; +3.391KB] or [+5.756%; +6.320%]

scenario:exporting-pipeline-0.4-16

🟥 cpu_user_time [+52.153ms; +56.814ms] or [+21.300%; +23.203%]
🟥 execution_time [+53.604ms; +58.684ms] or [+19.303%; +21.132%]
🟥 instructions [+121.0M instructions; +121.5M instructions] or [+17.167%; +17.238%]
🟥 max_rss_usage [+3.632KB; +3.737KB] or [+6.509%; +6.697%]

scenario:exporting-pipeline-0.4-18

🟥 cpu_user_time [+53.661ms; +59.601ms] or [+20.904%; +23.218%]
🟥 execution_time [+59.073ms; +62.019ms] or [+20.345%; +21.360%]
🟥 instructions [+126.5M instructions; +127.0M instructions] or [+16.701%; +16.767%]

scenario:exporting-pipeline-0.4_with_stats-14

🟥 cpu_user_time [+55.115ms; +60.452ms] or [+21.826%; +23.940%]
🟥 execution_time [+58.547ms; +62.921ms] or [+20.391%; +21.915%]
🟥 instructions [+122.2M instructions; +123.1M instructions] or [+16.295%; +16.419%]
🟥 max_rss_usage [+3.402KB; +3.713KB] or [+6.362%; +6.945%]

scenario:exporting-pipeline-0.4_with_stats-16

🟥 cpu_user_time [+51.437ms; +56.531ms] or [+20.908%; +22.978%]
🟥 execution_time [+55.029ms; +59.475ms] or [+19.689%; +21.280%]
🟥 instructions [+123.8M instructions; +124.3M instructions] or [+17.502%; +17.577%]
🟥 max_rss_usage [+3.972KB; +4.050KB] or [+7.112%; +7.251%]

scenario:exporting-pipeline-0.4_with_stats-18

🟥 cpu_user_time [+53.500ms; +59.802ms] or [+20.673%; +23.108%]
🟥 execution_time [+58.960ms; +62.543ms] or [+20.115%; +21.338%]
🟥 instructions [+126.3M instructions; +126.8M instructions] or [+16.624%; +16.699%]

scenario:exporting-pipeline-0.5-14

🟥 instructions [+46.7M instructions; +47.1M instructions] or [+7.725%; +7.789%]

scenario:exporting-pipeline-0.5-16

🟥 instructions [+54.3M instructions; +54.6M instructions] or [+9.404%; +9.454%]

scenario:exporting-pipeline-0.5-18

🟥 instructions [+59.5M instructions; +59.9M instructions] or [+10.687%; +10.757%]

scenario:exporting-pipeline-0.5_with_stats-14

🟥 instructions [+47.0M instructions; +47.4M instructions] or [+7.729%; +7.800%]

scenario:exporting-pipeline-0.5_with_stats-16

🟥 instructions [+54.0M instructions; +54.3M instructions] or [+9.317%; +9.359%]

scenario:exporting-pipeline-0.5_with_stats-18

🟥 instructions [+58.4M instructions; +59.0M instructions] or [+10.438%; +10.545%]

scenario:plugin-graphql-with-depth-and-collapse-on-18

🟥 max_rss_usage [+120.285KB; +162.931KB] or [+14.642%; +19.833%]

scenario:plugin-graphql-with-depth-off-18

🟥 max_rss_usage [+111.814KB; +161.590KB] or [+13.565%; +19.604%]

scenario:plugin-graphql-with-depth-on-max-18

🟥 max_rss_usage [+119.288KB; +148.464KB] or [+14.409%; +17.933%]

scenario:spans-finish-later-14

🟩 max_rss_usage [-11.087KB; -10.876KB] or [-5.568%; -5.462%]

* Couchbase: Update Docker Container repo This repo uses a slightly modified version of the Couchbase docker image for testing. This modified version is required to do some base configuration of the Couchbase container (such as password setup, memory settings, and more). Prior to this change, we used a container image/repo owned by Sam Brenner (sabrenner). This change forks the sabrenner repository and adds the container to DataDog's github container repository. * Couchbase: update tested versions This change updates several of the versions of Couchbase which we test against. In particular: * 2.6.12 replaces 2.6.5 due to an inability to compile 2.6.5 in the latest Node releases * 3.0.7 replaces 3.0.0 due to test timeout issues causing flakiness * 3.1.2 replaces 3.1.0 due to test timeout issues causing flakiness These changes put the couchbase dependency in a state where it can be re-introduced to the CI tests. * Couchbase: Re-enable testing Prior to this change, Couchbase tests were disabled due to constant failures and flaky tests. Recent changes have fixed the testing issues by: * Upgrading Couchbase server from 6.0.1 to 6.6.5 * Updating the versions of Couchbase client we use to test This change re-enables testing the Couchbase integration now that tests are working and stable.

* try to fix flaky test * Update test-gcloud-function.sh * Update test-gcloud-function.sh * Update test-gcloud-function.sh * Update test-gcloud-function.sh

* Add BITBUCKET_GIT_HTTP_ORIGIN env variable * Update CI spec JSONs

#3440)

juan-fernandez

looks good from CI Visibility's perspective

nsavoire and others added 21 commits July 20, 2023 16:41

Taint request URI (#3302)

fa35b91

* Taint request URI * Add check for safe tainted origins on unvalidated redirect analyzer * Change assertion construction for unvalidated redirect analyzer test * Add metric for uri sourcing * Fix PR comments

Fix erroneous resetting of profiler state to undefined (#3394)

08de2cc

Update yarn.lock with current pprof-nodejs version (#3398)

02f81ab

Forgot to update yarn.lock when bumping pprof-nodejs version in a previous PR.

openai: be defensive about response object shape (#3408)

95cb464

delete old build scripts (#3396)

ca809c3

Add eslint rule for object-curly-newline (#3406)

f7b67c5

Fix a phrase in README (#3405)

86cf7e7

Fix same dependency detection (#3386)

c89039d

* Fix same dependency detection * Do not send same dependency:version twice * Clear map on stop * Update savedDependencies to savedDependenciesToSend to improve understanding

fix: Handle case where Lambda handler is using callbacks instead of p…

f8dee9f

…romises (#3404)

Ignore esbuild output from integration tests in esbuild (#3410)

a0c3914

[ci-visibility] Do not modify how jest reports timeouts (#3399)

03402ea

Allow https datadog agent url in continuous profiling (#3190) (#3194)

952002c

Co-authored-by: Attila Szegedi <[email protected]>

Fix distributed tracing issue caused by single span ingestion changes (…

37e24c1

…#3413)

Ensure OTel span name is copied to resource.name (#3412)

85652ea

Make datadog distributed tracing header take priority over traceconte…

890c950

…xt (#3414)

Exclude empty telemetry metrics (#3421)

13c9b6d

openai: make services entirely optional when initializing (#3420)

f1c20cb

Fix service representation config (#3419)

1c5ecc9

* address and fix edge cases * move to better tracer option name for service name flattening

implement naming schema for graphql plugin (#3279)

e181f57

* implement naming schema for graphql plugin * implement schema v1 and add tests --------- Co-authored-by: Thomas Hunter II <[email protected]> Co-authored-by: Jordi Bertran de Balanda <[email protected]>

szegedi requested review from a team as code owners July 20, 2023 14:46

szegedi added the semver-minor label Jul 20, 2023

tabgok and others added 3 commits July 21, 2023 10:20

Fix flaky serverless integration test (#3418)

b169d3e

* try to fix flaky test * Update test-gcloud-function.sh * Update test-gcloud-function.sh * Update test-gcloud-function.sh * Update test-gcloud-function.sh

Fix native-iast-taint-tracking version number (#3433)

9ecc004

openai: remove String#replaceAll for Node.js <=14 (#3432)

d175f66

szegedi force-pushed the v3.29.0-proposal branch from ec246cb to 66454e5 Compare July 21, 2023 08:20

ericlaz and others added 3 commits July 21, 2023 14:57

Add BITBUCKET_GIT_HTTP_ORIGIN env variable (#3401)

5f8ba68

* Add BITBUCKET_GIT_HTTP_ORIGIN env variable * Update CI spec JSONs

[ci-visibility] Make integration test for jest compatible with node 12 (

50066ae

#3440)

v3.29.0

d71337c

szegedi force-pushed the v3.29.0-proposal branch from 66454e5 to d71337c Compare July 21, 2023 12:57

juan-fernandez approved these changes Jul 21, 2023

View reviewed changes

tlhunter approved these changes Jul 21, 2023

View reviewed changes

simon-id approved these changes Jul 21, 2023

View reviewed changes

tlhunter merged commit e214bcf into v3.x Jul 21, 2023

szegedi deleted the v3.29.0-proposal branch December 19, 2023 09:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

V3.29.0 proposal #3427

V3.29.0 proposal #3427

Uh oh!

szegedi commented Jul 20, 2023 •

edited

Loading

Uh oh!

github-actions bot commented Jul 20, 2023 •

edited

Loading

Uh oh!

codecov bot commented Jul 20, 2023 •

edited

Loading

Uh oh!

pr-commenter bot commented Jul 21, 2023 •

edited

Loading

Uh oh!

juan-fernandez left a comment

Uh oh!

Uh oh!

V3.29.0 proposal #3427

V3.29.0 proposal #3427

Uh oh!

Conversation

szegedi commented Jul 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Features

Enhancements

Bug fixes

Minor stuff (probably not notable for release notes)

Uh oh!

github-actions bot commented Jul 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overall package size

Dependency sizes

Uh oh!

codecov bot commented Jul 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pr-commenter bot commented Jul 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

scenario:appsec-iast-no-vulnerability-control-14

scenario:appsec-iast-no-vulnerability-control-16

scenario:exporting-pipeline-0.4-14

scenario:exporting-pipeline-0.4-16

scenario:exporting-pipeline-0.4-18

scenario:exporting-pipeline-0.4_with_stats-14

scenario:exporting-pipeline-0.4_with_stats-16

scenario:exporting-pipeline-0.4_with_stats-18

scenario:exporting-pipeline-0.5-14

scenario:exporting-pipeline-0.5-16

scenario:exporting-pipeline-0.5-18

scenario:exporting-pipeline-0.5_with_stats-14

scenario:exporting-pipeline-0.5_with_stats-16

scenario:exporting-pipeline-0.5_with_stats-18

scenario:plugin-graphql-with-depth-and-collapse-on-18

scenario:plugin-graphql-with-depth-off-18

scenario:plugin-graphql-with-depth-on-max-18

scenario:spans-finish-later-14

Uh oh!

juan-fernandez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

szegedi commented Jul 20, 2023 •

edited

Loading

github-actions bot commented Jul 20, 2023 •

edited

Loading

codecov bot commented Jul 20, 2023 •

edited

Loading

pr-commenter bot commented Jul 21, 2023 •

edited

Loading