Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion packages/airflow/_dev/build/build.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
dependencies:
ecs:
reference: "git@v8.11.0"
reference: "git@v8.17.0"
14 changes: 6 additions & 8 deletions packages/airflow/_dev/build/docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,11 @@ Use the Airflow integration to:
- Create informative visualizations to track usage trends, measure key metrics, and derive actionable business insights.
- Monitor your workflows' performance and status in real-time.

## Data streams
## Compatibility

The Airflow package is tested with Airflow `2.4.0` and `3.0.6`. It should work with any `2.*` and `3.*` versions.

## What data does this integration collect?

The Airflow integration gathers metric data.

Expand All @@ -23,17 +27,11 @@ Data stream:
Note:
- Users can monitor and view metrics within the ingested documents for Airflow in the `metrics-*` index pattern from `Discover`.

## Compatibility

The Airflow module is tested with Airflow `2.4.0`. It should work with versions `2.0.0` and later.

## What do I need to use this integration?

Users require Elasticsearch to store and search user data, and Kibana to visualize and manage it. They can utilize the hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on their own hardware.

To ingest data from Airflow, users must have [StatsD](https://github.com/statsd/statsd) to receive the same.

## Setup
## How do I deploy this integration?

For step-by-step instructions on how to set up an integration, see the [Getting started](https://www.elastic.co/guide/en/starting-with-the-elasticsearch-platform-and-its-solutions/current/getting-started-observability.html) guide.

Expand Down
5 changes: 5 additions & 0 deletions packages/airflow/changelog.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.0.0"
changes:
- description: Make Airflow package GA.
type: enhancement
link: https://github.com/elastic/integrations/pull/15287
- version: "0.11.0"
changes:
- description: Improve documentation
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
service_notify_signal: SIGHUP
vars:
host: 0.0.0.0
port: 8125
vars: ~
data_stream:
vars:
host: 0.0.0.0
port: 8125
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ description: Pipeline for processing airflow data
processors:
- set:
field: ecs.version
value: "8.11.0"
value: "8.17.0"
- rename:
field: statsd
target_field: airflow
Expand Down
16 changes: 15 additions & 1 deletion packages/airflow/data_stream/statsd/manifest.yml
Original file line number Diff line number Diff line change
@@ -1,10 +1,24 @@
type: metrics
title: Airflow metrics
release: experimental
streams:
- input: statsd/metrics
enabled: true
title: Airflow metrics
description: Collect Airflow metrics
vars:
- name: host
type: text
title: Host
multi: false
required: true
show_user: true
default: localhost
- name: port
type: text
title: Port
multi: false
required: true
show_user: true
default: "8125"
elasticsearch:
index_mode: "time_series"
2 changes: 1 addition & 1 deletion packages/airflow/data_stream/statsd/sample_event.json
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
"type": "metrics"
},
"ecs": {
"version": "8.11.0"
"version": "8.17.0"
},
"elastic_agent": {
"id": "97400795-188c-4140-a1ee-0002078c785d",
Expand Down
16 changes: 7 additions & 9 deletions packages/airflow/docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,11 @@ Use the Airflow integration to:
- Create informative visualizations to track usage trends, measure key metrics, and derive actionable business insights.
- Monitor your workflows' performance and status in real-time.

## Data streams
## Compatibility

The Airflow package is tested with Airflow `2.4.0` and `3.0.6`. It should work with any `2.*` and `3.*` versions.

## What data does this integration collect?

The Airflow integration gathers metric data.

Expand All @@ -23,17 +27,11 @@ Data stream:
Note:
- Users can monitor and view metrics within the ingested documents for Airflow in the `metrics-*` index pattern from `Discover`.

## Compatibility

The Airflow module is tested with Airflow `2.4.0`. It should work with versions `2.0.0` and later.

## What do I need to use this integration?

Users require Elasticsearch to store and search user data, and Kibana to visualize and manage it. They can utilize the hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on their own hardware.

To ingest data from Airflow, users must have [StatsD](https://github.com/statsd/statsd) to receive the same.

## Setup
## How do I deploy this integration?

For step-by-step instructions on how to set up an integration, see the [Getting started](https://www.elastic.co/guide/en/starting-with-the-elasticsearch-platform-and-its-solutions/current/getting-started-observability.html) guide.

Expand Down Expand Up @@ -95,7 +93,7 @@ An example event for `statsd` looks as following:
"type": "metrics"
},
"ecs": {
"version": "8.11.0"
"version": "8.17.0"
},
"elastic_agent": {
"id": "97400795-188c-4140-a1ee-0002078c785d",
Expand Down
20 changes: 1 addition & 19 deletions packages/airflow/manifest.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
name: airflow
title: Airflow
version: "0.11.0"
version: "1.0.0"
description: Airflow Integration.
type: integration
format_version: "3.0.0"
Expand All @@ -16,32 +16,14 @@ icons:
title: Airflow logo
size: 32x32
type: image/svg+xml
vars:
- name: host
type: text
title: Host
multi: false
required: true
show_user: true
default: localhost
- name: port
type: text
title: Port
multi: false
required: true
show_user: true
default: "8125"
policy_templates:
- name: server
title: Airflow metrics
description: Collect metrics from Airflow instances
data_streams:
- statsd
inputs:
- type: "statsd/metrics"
title: "Collect Airflow metrics"
description: "Collecting Airflow metrics"
input_group: metrics
owner:
github: elastic/obs-infraobs-integrations
type: elastic