Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/styles/base/Dictionary.txt
Original file line number Diff line number Diff line change
Expand Up @@ -829,6 +829,7 @@ uid
UIs
UIs
ulimit
ulimits
unary
unassign
uncheck
Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
When running multiple Data Plane nodes, there is no thread-safe behavior between nodes. In high-load scenarios, you may observe the same message being delivered multiple times across different Data Plane nodes
When running multiple data plane nodes, there is no thread-safe behavior between nodes. In high-load scenarios, you may observe the same message being delivered multiple times across different data plane nodes.

To minimize duplicate message delivery in a multi-node setup, consider:
* Using a single Data Plane node for consuming messages from specific topics
* Using a single data plane node for consuming messages from specific topics
* Implementing idempotency handling in your consuming application
* Monitoring Consumer Group offsets across your Data Plane nodes
* Monitoring Consumer Group offsets across your data plane nodes
37 changes: 37 additions & 0 deletions app/_includes/plugins/confluent-kafka-consume/websocket.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
In `websocket` mode, the plugin maintains a bi-directional WebSocket connection with the client,
allowing for continuous delivery of {{ include.broker }} messages to the client.

Here's how it works:
1. Establish a WebSocket connection to the Route where the {{ include.name }} plugin is enabled and mode is set to `websocket`.
1. {{site.base_gateway}} continuously streams messages as JSON text frames.
1. Optionally, client sends acknowledgments (`client-acks`) for each message or batch to enable `at-least-once` delivery semantics.

This approach provides real-time message flow without the limitations of HTTP polling.

{% mermaid %}
sequenceDiagram
participant Client
participant Kong as {{site.base_gateway}}
participant Broker as Message Broker

Client->>Kong: Establish WebSocket connection
Kong->>Broker: Connect to broker

loop Continuous message delivery
Broker->>Kong: Broker message
Kong->>Client: Stream JSON text frame

opt client-acks
Client->>Kong: Acknowledge message/batch
end
end

{% endmermaid %}

> _Figure 1: The diagram shows the bi-directional WebSocket flow where the {{include.name}} plugin is running in `websocket` mode, and messages are streamed as JSON text frames._

This mode provides parity with HTTP-based consumption, including support for:
* Message keys
* Topic filtering
* {{ include.broker }} authentication and TLS
* Auto or manual offset commits
1 change: 1 addition & 0 deletions app/_indices/event-gateway.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ sections:
- path: /plugins/kafka-upstream/
- path: /plugins/confluent/
- path: /plugins/confluent-consume/
- path: /plugins/solace-consume/
- path: /plugins/solace-upstream/
auto_exclude: true
- title: "Kafka Native Event Proxy (KNEP)"
Expand Down
4 changes: 3 additions & 1 deletion app/_kong_plugins/confluent-consume/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,9 @@ Kong also provides a [plugin for publishing messages to Confluent Cloud](/plugin

## Implementation details

{% include /plugins/confluent-kafka-consume/implementation-details.md %}
The plugin supports the following modes of operation:
* `http-get`: Consume messages via HTTP GET requests (default)
* `server-sent-events`: Stream messages using [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events)

## Message delivery guarantees

Expand Down
24 changes: 6 additions & 18 deletions app/_kong_plugins/kafka-consume/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,26 +64,14 @@ Kong also provides Kafka plugins for publishing messages:

## Implementation details

{% include /plugins/confluent-kafka-consume/implementation-details.md slug=page.slug %}
The plugin supports the following modes of operation:
* `http-get`: Consume messages via HTTP GET requests (default)
* `server-sent-events`: Stream messages using [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events)
* `websocket` {% new_in 3.11 %}: Stream messages over a [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) connection

### WebSocket mode {% new_in 3.11 %}

## WebSocket mode {% new_in 3.11 %}

In `websocket` mode, the plugin maintains a bi-directional WebSocket connection with the client. This allows:
* Continuous delivery of Kafka messages to the client
* Optional client acknowledgments (`client-acks`) for each message or batch, enabling `at-least-once` delivery semantics
* Real-time message flow without the limitations of HTTP polling

To consume messages via WebSocket:
1. Establish a WebSocket connection to the route where the plugin is enabled and `mode` is set to `websocket`
1. Optionally, send acknowledgment messages to indicate successful processing
1. Messages will be streamed as text frames in JSON format

This mode provides parity with HTTP-based consumption, including support for:
* Message keys
* Topic filtering
* Kafka authentication and TLS
* Auto or manual offset commits
{% include /plugins/confluent-kafka-consume/websocket.md slug=page.slug broker='Kafka' name=page.name %}

## Message delivery guarantees

Expand Down
9 changes: 9 additions & 0 deletions app/_kong_plugins/solace-consume/changelog.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"3.12.0.0": [
{
"message": "Added Solace Consume plugin.",
"type": "feature",
"scope": "Plugin"
}
]
}
36 changes: 36 additions & 0 deletions app/_kong_plugins/solace-consume/examples/auto.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
description: Configure the Solace Consume plugin to consume messages via automatic format detection.

title: Auto mode

weight: 999

requirements:
- You have a [Solace PubSub+ Event Broker](https://docs.solace.com/Get-Started/tutorial/event-broker-set-up.htm)
- You have a [Solace message queue](https://docs.solace.com/Messaging/Guaranteed-Msg/Queues.htm)

variables:
host:
description: "The Solace broker host URL. For example: `tcp://my.broker:55555` (plain) or `tcps://my.broker:55443` (secure)."
value: $SOLACE_BROKER_HOST
queue_name:
description: The name of the message queue to consume from.
value: $QUEUE_NAME

config:
mode: AUTO
session:
host: ${host}
authentication:
scheme: BASIC
username: admin
password: admin
flow:
binds:
- name: ${queue_name}

tools:
- deck
- admin-api
- konnect-api
- kic
- terraform
35 changes: 35 additions & 0 deletions app/_kong_plugins/solace-consume/examples/polling.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
description: Configure the Solace Consume plugin in polling mode.

title: Polling mode
weight: 998

requirements:
- You have a [Solace PubSub+ Event Broker](https://docs.solace.com/Get-Started/tutorial/event-broker-set-up.htm)
- You have a [Solace message queue](https://docs.solace.com/Messaging/Guaranteed-Msg/Queues.htm)

variables:
host:
description: "The Solace broker host URL. For example: `tcp://my.broker:55555` (plain) or `tcps://my.broker:55443` (secure)."
value: $SOLACE_BROKER_HOST
queue_name:
description: The name of the message queue to consume from.
value: $QUEUE_NAME

config:
mode: POLLING
session:
host: ${host}
authentication:
scheme: BASIC
username: admin
password: admin
flow:
binds:
- name: ${queue_name}

tools:
- deck
- admin-api
- konnect-api
- kic
- terraform
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
description: Configure the Solace Consume plugin to stream messages using server-sent events.

title: Server-sent events mode

weight: 997

requirements:
- You have a [Solace PubSub+ Event Broker](https://docs.solace.com/Get-Started/tutorial/event-broker-set-up.htm)
- You have a [Solace message queue](https://docs.solace.com/Messaging/Guaranteed-Msg/Queues.htm)

variables:
host:
description: "The Solace broker host URL. For example: `tcp://my.broker:55555` (plain) or `tcps://my.broker:55443` (secure)."
value: $SOLACE_BROKER_HOST
queue_name:
description: The name of the message queue to consume from.
value: $QUEUE_NAME

config:
mode: SERVER-SENT-EVENTS
session:
host: ${host}
authentication:
scheme: BASIC
username: admin
password: admin
flow:
binds:
- name: ${queue_name}

tools:
- deck
- admin-api
- konnect-api
- kic
- terraform
58 changes: 58 additions & 0 deletions app/_kong_plugins/solace-consume/examples/ssl.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
description: Configure the Solace Consume plugin to consume messages over SSL.

weight: 850

extended_description: |
Configure the Solace Consume plugin to consume messages over SSL.

By default, SSL certification validation is disabled. In a production environment, we recommend enabing SSL validation.

# ----- Commented out as the following is not supported for now; uncomment when support is added -----
# In your Solace configuration, set up the following:
# * `SESSION_SSL_TRUST_STORE_DIR`: Path to the trust store directory. For example, `/usr/local/share/lua/5.1/kong/plugins/solace/cert`.
# * `SESSION_SSL_VALIDATE_CERTIFICATE_DATE`: Indicates if the session connection should fail when receiving a certificate with an invalid date.
# * `SESSION_SSL_VALIDATE_CERTIFICATE_HOST`: Indicates if the session connection should fail when receiving a certificate with an invalid host.
# * `SESSION_SSL_CIPHER_SUITES`: Comma-separated list of the cipher suites.

# See [Configuring Secure Sessions](https://docs.solace.com/API/API-Developer-Guide/Creating-Secure-Sessions.htm) in the Solace documentation for details.
# ------

title: Consume messages over SSL
weight: 900

requirements:
- You have a [Solace PubSub+ Event Broker](https://docs.solace.com/Get-Started/tutorial/event-broker-set-up.htm)
- You have a [Solace message queue](https://docs.solace.com/Messaging/Guaranteed-Msg/Queues.htm)

variables:
host:
description: "The Solace broker host URL. For example, `tcps://my.broker:55443`"
value: $SOLACE_BROKER_HOST
queue_name:
description: The name of the message queue to consume from.
value: $QUEUE_NAME

config:
session:
host: ${host}
ssl_validate_certificate: true
authentication:
scheme: BASIC
username: admin
password: admin
properties:
SESSION_SSL_TRUST_STORE_DIR: /usr/local/share/lua/5.1/kong/plugins/solace/cert
SESSION_SSL_VALIDATE_CERTIFICATE_DATE: "true"
SESSION_SSL_VALIDATE_CERTIFICATE_HOST: "true"
SESSION_SSL_CIPHER_SUITES: "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"

flow:
binds:
- name: ${queue_name}

tools:
- deck
- admin-api
- konnect-api
- kic
- terraform
51 changes: 51 additions & 0 deletions app/_kong_plugins/solace-consume/examples/uri-capture.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
title: Capture request URI
description: Capture the URI of the request and use it as the message destination name.

weight: 800
extended_description: |
You can use `uri_captures` to capture the URI of the request and use it as the message destination name.

For example, if you have a Route that looks like this:

```sh
~/demo/(?topic_id_1\w+)/hello/(?topic_id_2\w+)`
```

You can use the following configuration:
```yaml
name: /topic/$(uri_captures['topic_id_1'])/test/$(uri_captures['topic_id_2'])
```
If the request doesn't match, the plugin will remove the placeholder and the `/`, so the final destination will be `/topic/test/`.

{:.info}
> **Note:** Use single quotes for the topic ID, such as `$(uri_captures['topic_id_1']`. Double quotes will cause errors.

requirements:
- You have a [Solace PubSub+ Event Broker](https://docs.solace.com/Get-Started/tutorial/event-broker-set-up.htm)
- You have a [Solace message queue](https://docs.solace.com/Messaging/Guaranteed-Msg/Queues.htm)


variables:
host:
description: "The Solace broker host URL. For example: `tcp://my.broker:55555` (plain) or `tcps://my.broker:55443` (secure)."
value: $SOLACE_BROKER_HOST

config:
session:
host: ${host}
ssl_validate_certificate: true
authentication:
scheme: BASIC
username: admin
password: admin

flow:
binds:
- name: /topic/$(uri_captures['topic_id_1'])/test/$(uri_captures['topic_id_2'])

tools:
- deck
- admin-api
- konnect-api
- kic
- terraform
36 changes: 36 additions & 0 deletions app/_kong_plugins/solace-consume/examples/websocket.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
description: Configure the Solace Consume plugin to stream messages over WebSockets.

title: WebSocket mode

weight: 996

requirements:
- You have a [Solace PubSub+ Event Broker](https://docs.solace.com/Get-Started/tutorial/event-broker-set-up.htm)
- You have a [Solace message queue](https://docs.solace.com/Messaging/Guaranteed-Msg/Queues.htm)

variables:
host:
description: "The Solace broker host URL. For example: `ws://my.broker:80` (plain) or `wss://my.broker:443` (secure)"
value: $SOLACE_BROKER_HOST
queue_name:
description: The name of the message queue to consume from.
value: $QUEUE_NAME

config:
mode: WEBSOCKET
session:
host: ${host}
authentication:
scheme: BASIC
username: admin
password: admin
flow:
binds:
- name: ${queue_name}

tools:
- deck
- admin-api
- konnect-api
- kic
- terraform
Loading
Loading