Skip to content

[vsphere] add extra grok pattern to cover more log formats #15274

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
+117 −7
Open

[vsphere] add extra grok pattern to cover more log formats #15274

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
088a6c7
[vsphere] add extra grok pattern to cover more log formats
stefans-elastic Sep 10, 2025
51867ef
Merge branch 'main' into vsphere-grok
stefans-elastic Sep 10, 2025
5dc8583
pattern cleanup
stefans-elastic Sep 10, 2025
3335501
Update packages/vsphere/changelog.yml
mykola-elastic Sep 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/vsphere/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.22.0"
changes:
- description: Add extra grok pattern to cover more log formats
type: enhancement
link: https://github.com/elastic/integrations/pull/15274
- version: "1.21.0"
changes:
- description: Improve documentation
Expand Down
5 changes: 4 additions & 1 deletion packages/vsphere/data_stream/log/_dev/test/pipeline/test-format-common.log
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,4 +151,7 @@
<14>1 2022-12-29T01:33:33.284655+00:00 vspherehost01 vpxd 6225 - - Event [162431924] [1-1] [2022-12-29T01:33:33.284478Z] [vim.event.EventEx] [info] [philipp] [] [162431924] [Failed login philipp from 192.168.11.1 at 12/29/2022 01:33:33 GMT in SSO]
<14>1 2021-09-06T14:40:13.289354+00:00 vcenter vpxd 58650 - - Event [575793] [1-1] [2021-09-06T14:40:13.288346Z] [vim.event.UserLogoutSessionEvent] [info] [VSPHERE.LOCAL\Administrator] [] [575793] [User VSPHERE.LOCAL\[email protected] logged out (login time: Monday, 06 September, 2021 02:40:13 PM, number of API invocations: 75,133, user agent: Go-http-client/1.1)]
<166>1 2024-09-18T21:30:05.155Z esxihost01 Hostd: info vsansystem[21254123] [vSAN@1234 sub=AccessChecker opId=011a11e2-7123] Shared secret from 192.168.0.1 logged in as VMware-client/6.5.0
<166>1 2024-09-18T21:30:05.155Z esxihost01 Hostd: info vsansystem[21254123] [vSAN@1234 sub=AccessChecker opId=011a11e2-7123] SSL thumbprint logged in as VMware-client/6.5.0
<166>1 2024-09-18T21:30:05.155Z esxihost01 Hostd: info vsansystem[21254123] [vSAN@1234 sub=AccessChecker opId=011a11e2-7123] SSL thumbprint logged in as VMware-client/6.5.0
<110>1 2025-09-10T01:01:25.113Z PC-ESXI-VSAN-P01 envoy 21004234 - [proxy.disconnect@2345 key2=\"\\\"CP\\\"\" subject=\"\" ip=\"127.0.0.1\" priority=\"info\" vmw_vcenter=\"prod-vc02.sphere.com\" vmw_vcenter_id=\"550e8400-e29b-41d4-a716-446655440000\" vmw_vr_ops_id=\"b5cb8b7e-f1d8-4191-a5fe-391af4d592c3\" result=\"success\" vmw_cluster=\"PROD-VM01\" vmw_datacenter=\"TestCenter\" vmw_object_id=\"host-112233\" port=\"45296\" facility=\"13\" object=\"proxy\"]
<134>1 2025-09-10T15:43:11.026Z prod-vc01 vpxd-main - - [Originator@6884 key2=\"\\\"CP\\\"\" vmw_cluster=\"PROD-P01\" vmw_datacenter=\"TestCenter\" vmw_object_id=\"vm-112233\" vmw_host=\"esxi-p01.sphere.com\" priority=\"info\" vmw_vcenter=\"prod-vc01.sphere.com\" vmw_vcenter_id=\"550e8400-e29b-41d4-a716-446655440000\" facility=\"local1\" vmw_vr_ops_id=\"b5cb8b7e-f1d8-4191-a5fe-391af4d592c3\"] 2025-09-10T03:34:13.023+08:00 info vpxd[07317] [Originator@6875 sub=vpxLri opID=6598c432] [VpxLRO] -- FINISH lri-111222333
<166>1 2025-09-10T12:15:33.834Z PC-ESXI-HCI-P01.sphere.com envoy-access 2188123 [Originator@6534 key2=\"\\\"CP\\\"\" vmw_cluster=\"PROD-VM01\" vmw_datacenter=\"TestCenter\" vmw_object_id=\"host-007\" priority=\"info\" vmw_vcenter=\"prod-vc02.sphere.com\" vmw_vcenter_id=\"550e8400-e29b-41d4-a716-446655440000\" facility=\"local1\" vmw_vr_ops_id=\"b5cb8b7e-f1d8-4191-a5fe-391af4d592c3\"] POST /sdk 200 via_upstream - 452 256 gzip 0 0 0 127.0.0.1:60792 HTTP/1.1 TLSv1.2 127.0.0.1:443 127.0.0.1:57833 HTTP/1.1 - 127.0.0.1:8307 - \"QueryNetworkHint\"
109 changes: 105 additions & 4 deletions packages/vsphere/data_stream/log/_dev/test/pipeline/test-format-common.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5932,7 +5932,7 @@
"authentication"
],
"kind": "event",
"original": "<166>1 2024-09-18T21:30:05.155Z esxihost01 Hostd: info vsansystem[21254123] [vSAN@1234 sub=AccessChecker opId=011a11e2-7123] SSL thumbprint logged in as VMware-client/6.5.0 ",
"original": "<166>1 2024-09-18T21:30:05.155Z esxihost01 Hostd: info vsansystem[21254123] [vSAN@1234 sub=AccessChecker opId=011a11e2-7123] SSL thumbprint logged in as VMware-client/6.5.0",
"outcome": "success",
"type": [
"info"
Expand All @@ -5954,7 +5954,7 @@
}
}
},
"message": "info vsansystem[21254123] [vSAN@1234 sub=AccessChecker opId=011a11e2-7123] SSL thumbprint logged in as VMware-client/6.5.0 ",
"message": "info vsansystem[21254123] [vSAN@1234 sub=AccessChecker opId=011a11e2-7123] SSL thumbprint logged in as VMware-client/6.5.0",
"process": {
"name": "Hostd"
},
Expand All @@ -5969,8 +5969,109 @@
"name": "Other"
},
"name": "Other",
"original": "VMware-client/6.5.0 "
"original": "VMware-client/6.5.0"
}
}
},
{
"@timestamp": "2025-09-10T01:01:25.113Z",
"ecs": {
"version": "8.11.0"
},
"event": {
"kind": "event",
"original": "<110>1 2025-09-10T01:01:25.113Z PC-ESXI-VSAN-P01 envoy 21004234 - [proxy.disconnect@2345 key2=\\\"\\\\\\\"CP\\\\\\\"\\\" subject=\\\"\\\" ip=\\\"127.0.0.1\\\" priority=\\\"info\\\" vmw_vcenter=\\\"prod-vc02.sphere.com\\\" vmw_vcenter_id=\\\"550e8400-e29b-41d4-a716-446655440000\\\" vmw_vr_ops_id=\\\"b5cb8b7e-f1d8-4191-a5fe-391af4d592c3\\\" result=\\\"success\\\" vmw_cluster=\\\"PROD-VM01\\\" vmw_datacenter=\\\"TestCenter\\\" vmw_object_id=\\\"host-112233\\\" port=\\\"45296\\\" facility=\\\"13\\\" object=\\\"proxy\\\"]"
},
"host": {
"name": "PC-ESXI-VSAN-P01"
},
"log": {
"syslog": {
"facility": {
"code": 13,
"name": "Log audit"
},
"priority": 110,
"severity": {
"code": 6,
"name": "Informational"
}
}
},
"message": "[proxy.disconnect@2345 key2=\\\"\\\\\\\"CP\\\\\\\"\\\" subject=\\\"\\\" ip=\\\"127.0.0.1\\\" priority=\\\"info\\\" vmw_vcenter=\\\"prod-vc02.sphere.com\\\" vmw_vcenter_id=\\\"550e8400-e29b-41d4-a716-446655440000\\\" vmw_vr_ops_id=\\\"b5cb8b7e-f1d8-4191-a5fe-391af4d592c3\\\" result=\\\"success\\\" vmw_cluster=\\\"PROD-VM01\\\" vmw_datacenter=\\\"TestCenter\\\" vmw_object_id=\\\"host-112233\\\" port=\\\"45296\\\" facility=\\\"13\\\" object=\\\"proxy\\\"]",
"process": {
"name": "envoy",
"pid": 21004234
},
"tags": [
"preserve_original_event"
]
},
{
"@timestamp": "2025-09-10T15:43:11.026Z",
"ecs": {
"version": "8.11.0"
},
"event": {
"kind": "event",
"original": "<134>1 2025-09-10T15:43:11.026Z prod-vc01 vpxd-main - - [Originator@6884 key2=\\\"\\\\\\\"CP\\\\\\\"\\\" vmw_cluster=\\\"PROD-P01\\\" vmw_datacenter=\\\"TestCenter\\\" vmw_object_id=\\\"vm-112233\\\" vmw_host=\\\"esxi-p01.sphere.com\\\" priority=\\\"info\\\" vmw_vcenter=\\\"prod-vc01.sphere.com\\\" vmw_vcenter_id=\\\"550e8400-e29b-41d4-a716-446655440000\\\" facility=\\\"local1\\\" vmw_vr_ops_id=\\\"b5cb8b7e-f1d8-4191-a5fe-391af4d592c3\\\"] 2025-09-10T03:34:13.023+08:00 info vpxd[07317] [Originator@6875 sub=vpxLri opID=6598c432] [VpxLRO] -- FINISH lri-111222333"
},
"host": {
"name": "prod-vc01"
},
"log": {
"syslog": {
"facility": {
"code": 16,
"name": "Local 0"
},
"priority": 134,
"severity": {
"code": 6,
"name": "Informational"
}
}
},
"message": "[Originator@6884 key2=\\\"\\\\\\\"CP\\\\\\\"\\\" vmw_cluster=\\\"PROD-P01\\\" vmw_datacenter=\\\"TestCenter\\\" vmw_object_id=\\\"vm-112233\\\" vmw_host=\\\"esxi-p01.sphere.com\\\" priority=\\\"info\\\" vmw_vcenter=\\\"prod-vc01.sphere.com\\\" vmw_vcenter_id=\\\"550e8400-e29b-41d4-a716-446655440000\\\" facility=\\\"local1\\\" vmw_vr_ops_id=\\\"b5cb8b7e-f1d8-4191-a5fe-391af4d592c3\\\"] 2025-09-10T03:34:13.023+08:00 info vpxd[07317] [Originator@6875 sub=vpxLri opID=6598c432] [VpxLRO] -- FINISH lri-111222333",
"process": {
"name": "vpxd-main"
},
"tags": [
"preserve_original_event"
]
},
{
"@timestamp": "2025-09-10T12:15:33.834Z",
"ecs": {
"version": "8.11.0"
},
"event": {
"kind": "event",
"original": "<166>1 2025-09-10T12:15:33.834Z PC-ESXI-HCI-P01.sphere.com envoy-access 2188123 [Originator@6534 key2=\\\"\\\\\\\"CP\\\\\\\"\\\" vmw_cluster=\\\"PROD-VM01\\\" vmw_datacenter=\\\"TestCenter\\\" vmw_object_id=\\\"host-007\\\" priority=\\\"info\\\" vmw_vcenter=\\\"prod-vc02.sphere.com\\\" vmw_vcenter_id=\\\"550e8400-e29b-41d4-a716-446655440000\\\" facility=\\\"local1\\\" vmw_vr_ops_id=\\\"b5cb8b7e-f1d8-4191-a5fe-391af4d592c3\\\"] POST /sdk 200 via_upstream - 452 256 gzip 0 0 0 127.0.0.1:60792 HTTP/1.1 TLSv1.2 127.0.0.1:443 127.0.0.1:57833 HTTP/1.1 - 127.0.0.1:8307 - \\\"QueryNetworkHint\\\""
},
"host": {
"name": "PC-ESXI-HCI-P01.sphere.com"
},
"log": {
"syslog": {
"facility": {
"code": 20,
"name": "Local 4"
},
"priority": 166,
"severity": {
"code": 6,
"name": "Informational"
}
}
},
"message": "[Originator@6534 key2=\\\"\\\\\\\"CP\\\\\\\"\\\" vmw_cluster=\\\"PROD-VM01\\\" vmw_datacenter=\\\"TestCenter\\\" vmw_object_id=\\\"host-007\\\" priority=\\\"info\\\" vmw_vcenter=\\\"prod-vc02.sphere.com\\\" vmw_vcenter_id=\\\"550e8400-e29b-41d4-a716-446655440000\\\" facility=\\\"local1\\\" vmw_vr_ops_id=\\\"b5cb8b7e-f1d8-4191-a5fe-391af4d592c3\\\"] POST /sdk 200 via_upstream - 452 256 gzip 0 0 0 127.0.0.1:60792 HTTP/1.1 TLSv1.2 127.0.0.1:443 127.0.0.1:57833 HTTP/1.1 - 127.0.0.1:8307 - \\\"QueryNetworkHint\\\"",
"process": {
"name": "envoy-access",
"pid": 2188123
},
"tags": [
"preserve_original_event"
]
}
]
}
3 changes: 2 additions & 1 deletion packages/vsphere/data_stream/log/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ processors:
(%{POSINT:process.pid:long}|-) - -%{SPACE}%{GREEDYDATA:message}"
- "^(%{ECS_SYSLOG_PRIORITY})?%{TIMESTAMP_ISO8601:_tmp.timestamp}%{SPACE}%{HOST}%{SPACE}%{DATA:process.name}(?:\\[%{POSINT:process.pid:long}\\])?\\:
%{GREEDYDATA:message}"
- "^(%{ECS_SYSLOG_PRIORITY})?%{TIMESTAMP_ISO8601:_tmp.timestamp}%{SPACE}%{HOST}%{SPACE}%{NOTSPACE:process.name}%{SPACE}(%{POSINT:process.pid:long}|-)( -)?%{SPACE}%{GREEDYDATA:message}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Any specififc reason for using {NOTSPACE:process.name} instead of {DATA:process.name} ?
  2. Have we officially verified that vSphere logs can have process id without [ ] ?

- "^ \\(%{TIMESTAMP_ISO8601:_tmp.timestamp} %{GREEDYDATA:message}\\)%{GREEDYDATA:_tmp.drop}"
pattern_definitions:
ECS_SYSLOG_PRIORITY: "<%{NONNEGINT:log.syslog.priority:long}>(\\d )?"
Expand Down Expand Up @@ -134,4 +135,4 @@ on_failure:
ignore_failure: true
- append:
field: error.message
value: "{{ _ingest.on_failure_message }}"
value: "{{ _ingest.on_failure_message }}"
2 changes: 1 addition & 1 deletion packages/vsphere/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
title: VMware vSphere
format_version: "3.0.2"
name: vsphere
version: "1.21.0"
version: "1.22.0"
description: This Elastic integration collects metrics and logs from vSphere/vCenter servers
type: integration
categories:
Expand Down