snyk: fix parameter handling and allow issue update ingestion

[efd6]

Proposed commit message

snyk: fix parameter handling and allow issue update ingestion

The previous implementation of the issues data stream did not correctly
handle request parameters; type and scan_item.id are not multiple value
parameters[1], and effective_severity_level and status use a comma
separated element syntax[2]. scan_item.id and scan_item.type are
corequisites[2] so this is documented in the configuration UI to avoid
confusion.

The previous design of the data stream intentionally only collected the
first seen occurrence of an issue, this behaviour has been changed so
that all updates are collected.

The stack version is updated to allow for deletion of old request trace logs.

[1]https://docs.snyk.io/snyk-api/reference/issues#get-orgs-org_id-issues
[2]experimentally determined

Note

Tested against a real endpoint.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Closes [snyk]: Fingerprint should include the status #14356
• Fixes [snyk]: Invalid format for parameter scan_item.type: multiple values for single value parameter #14342
• Closes snyk: add attributes.{ignored,status,updated_at} field values to fingerprint #14989

Screenshots

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

Package snyk 👍(1) 💚(0) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
issues	4329	3636.36	-692.64 (-16%)	💔

To see the full report comment with /test benchmark fullreport

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[clement-fouque]

I confirm that it's working as expected with the default configuration. There are no error generated.

[clement-fouque]

The new fingerprint is working: we have now 2 docs when an issue is updated.

But the timestamp is incorrect. I believe it should be set on field snyk.issues.attributes.updated_at and not snyk.issues.attributes.created_at

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 1fe1bdf

History

• 💚 Build #31053 succeeded ce200ff
• 💚 Build #31044 succeeded 5f3d6a9
• 💚 Build #30982 succeeded 6691f3f

cc @efd6

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[clement-fouque]

I tested this branch again and I can attest the following:

• The default Snyk configuration is working ([snyk]: Invalid format for parameter scan_item.type: multiple values for single value parameter #14342)
• Fingerprint is taking into account the updated_at field ([snyk]: Fingerprint should include the status #14356)

Both issues can be closed when this PR is merged.

cc @mitchell-rutigliano @smiller-elastic

[efd6]

@clement-fouque If this is satisfactory, please either dismiss your review or approve the change.

[clement-fouque]

LGTM

[elastic-vault-github-plugin-prod]

Package snyk - 3.0.0 containing this change is available at https://epr.elastic.co/package/snyk/3.0.0/