polish bridge config

[svix-onelson]

Alright y'all. After a bunch of conversations about the overall config and mental model to apply to the project (from the user perspective), this PR ended up huge. It'll be a tough review. I'm sorry 😭

This is a pretty large shift.

Here's a breakdown of the broad strokes:

• The Plugin trait was removed from -types.
• New traits added: SenderInput and ReceiverOutput to represent the
  non-svix portions of the dataflow.
• Removed the -plugin-webhook-receiver plugin crate.
  • moved the http server from the -plugin-webhook-receiver crate into svix-bridge.
  • "queue forwarders" into the -plugin-queue-consumer crate.
• Renamed the -plugin-queue-consumer crate to just -plugin-queue (it
  now provides both SenderInput and ReceiverOutput implementation for the various queue backends.
• all crates etc have been renamed: s/svix-webhook-bridge/svix-bridge/

What else, what else, what else...

The example config has been split into 2: one showing senders and another for receivers.
The readme has been gutted (but probably needs to be fleshed out a bit more... again). It felt silly to effectively duplicate the example configs in code fence blocks a 2nd time.

To sweeten the deal, new test coverage has been added for select portions of the config parsing as well as for the HTTP receiver stuff.

Both the sender and receiver example configs are now checked in the test suite to make sure they at least parse without failure. These tests should be expanded to inspect the actual data, but this diff is so big... 😭

We didn't have any tests for the HTTP handler execution before this, so that's very nice to have.

[svix-james]

LGTM

[tasn]

@svix-onelson, can you do a "diff" (maybe in a new PR) that's just the result files? As in, just have diffs where new files are added so that we can comment on the syntax easily (like a PR review) with the annoying diff lines. Does this make sense?

[svix-onelson]

@svix-onelson, can you do a "diff" (maybe in a new PR) that's just the result files? As in, just have diffs where new files are added so that we can comment on the syntax easily (like a PR review) with the annoying diff lines. Does this make sense?

Yeah, I think that's what I did to kick off the conversation. I'll copy the example file to a new file.

[svix-onelson]

@tasn 5937888 adds a full copy of the example config (svix-webhook-bridge.example2.yaml). Drop your notes on that one and I can try to incorporate the updates to make it that real.

[svix-onelson]

Discussed with @tasn - there are a couple of action items to clear before this moves forward.
The plan is to first do some exploratory "what if" example configs to examine:

• what if the jargon was more specific, less generic?
  • Ex: splitting the config into "producers" and consumers instead of just "plugins."
• what if we rebrand in terms of [webhook] "senders" and "receivers" instead of queue-consumer vs webhook-receiver
• what if the HTTP interface was a shared resource (like the JS runtime), "owned by bridge" and plugins optionally register routes mapping back to functions?
• (building on prev) what if (input, transformation, output) was the shape of all config items?
  • make it so HTTP or webhook is an input and the outputs are the various queue backends.

So, I guess the idea will be to sketch out these alternate realities and see which feels the most accessible, then after, look at the code changes needed to make them real.

I'll post a series of new yaml files to explore these ideas as subsequent revs, then we'll discuss.

[svix-onelson]

Omitted the requeue_on_nack -- let's assume we can pick a default either way and document what it is, though this feels like a "stopped clock is correct twice a day" situation to me.

[svix-onelson]

I prefer "webhook" to "http" since we're biased towards POST requests with JSON objects for bodies. I'd expect an "HTTP" input to have a lot more config options to let me go wild, but "webhook" sort of curbs that feeling for me.

[svix-onelson]

So, I'm going to dig in on the supporting refactors to make .example5.{sender,receiver}.yaml a reality and assume that the larger effort changes required will apply for wherever we finally land.

The biggest change required seems to be

• moving the HTTP server code into the bridge binary
• (probably) splitting Plugin trait into 2 separate types to represent sending vs receiving.

There might be some other things in the HTTP code that needs to change but I'm not sure what that might include right now.

The rewrangling of configs to match the examples is probably the smallest lift overall.

---

If there are extra changes in direction for the config part, I'm betting the refactoring of the HTTP stuff will still make sense. It feels right that the binary owns the HTTP server just like it owns the JS Runtime.

The doubt I have is that we might want to listen on other protocols in the future, and making the listener "belong to the plugin" would make the most sense for that, but as they say: "you ain't gonna need it" (YAGNI). That's tomorrow's problem.

[tasn]

Everything looks excellent in this and the senders examples!

One thing I'd make sure that we support in this YAML is loading from env var. Does YAML by any chance have an extension that supports just loading from env vars so I can do? key: $MY_KEY in the config and then we'll load the key from env var or env var file?

[svix-onelson]

Nothing is provided for this in serde_yaml from what I saw. We could pre-process the config to expand vars before parsing the YAML using something like shellexpand.

In the past I'd used envsubst, in a shell command but that's a bit clumsy for launching services. My use case was for deployments, etc where you're usually running a series of commands anyway.

[tasn]

No worries, this can be second step. I think shellexpand sounds scary/too much.
Anyhow, if there's something that can do it well we can consider it, but as long as it's very limited with what it does.