Distinguish all vs local events being persisted in the "Event Send Time Quantiles" graph #18510
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…me Quantiles" graph
MadLittleMods
commented
Jun 3, 2025
| { | ||
| "$$hashKey": "object:329", | ||
| "color": "#B877D9", | ||
| "hideTooltip": true, |
There was a problem hiding this comment.
I've added the event persist rate back to the tooltip so it's possible to get an accurate reading.
MadLittleMods
commented
Jun 3, 2025
| "yBucketBound": "auto" | ||
| }, | ||
| { | ||
| "aliasColors": {}, |
There was a problem hiding this comment.
The random changes are just whatever Grafana 10.4.3 is deciding to do (https://grafana.vpn.infra.element.dev/api/health)
For reference, we're using Grafana 10.4.1 on https://grafana.matrix.org/api/health
MadLittleMods
deleted the
madlittlemods/local-vs-all-events-being-persisted-in-event-send-time-graph-in-grafana
branch
|
Thanks for the review @reivilibre 🐌 |
Michael-Ixo
pushed a commit
to ixoworld/synapse
that referenced
this pull request
Jun 25, 2025
- Improvements to generate config documentation from JSON Schema file. ([\element-hq#18522](element-hq#18522)) - Add support for [MSC4155](matrix-org/matrix-spec-proposals#4155) Invite Filtering. ([\element-hq#18288](element-hq#18288)) - Add experimental `user_may_send_state_event` module API callback. ([\element-hq#18455](element-hq#18455)) - Add experimental `get_media_config_for_user` and `is_user_allowed_to_upload_media_of_size` module API callbacks that allow overriding of media repository maximum upload size. ([\element-hq#18457](element-hq#18457)) - Add experimental `get_ratelimit_override_for_user` module API callback that allows overriding of per-user ratelimits. ([\element-hq#18458](element-hq#18458)) - Pass `room_config` argument to `user_may_create_room` spam checker module callback. ([\element-hq#18486](element-hq#18486)) - Support configuration of default and extra user types. ([\element-hq#18456](element-hq#18456)) - Successful requests to `/_matrix/app/v1/ping` will now force Synapse to reattempt delivering transactions to appservices. ([\element-hq#18521](element-hq#18521)) - Support the import of the `RatelimitOverride` type from `synapse.module_api` in modules and rename `messages_per_second` to `per_second`. ([\element-hq#18513](element-hq#18513)) - Remove destinations from sending if not whitelisted. ([\element-hq#18484](element-hq#18484)) - Fixed room summary API incorrectly returning that a room is private in the room summary response when the join rule is omitted by the remote server. Contributed by @nexy7574. ([\element-hq#18493](element-hq#18493)) - Prevent users from adding themselves to their own user ignore list. ([\element-hq#18508](element-hq#18508)) - Generate config documentation from JSON Schema file. ([\element-hq#17892](element-hq#17892)) - Mention `CAP_NET_BIND_SERVICE` as an alternative to running Synapse as root in order to bind to a privileged port. ([\element-hq#18408](element-hq#18408)) - Surface hidden Admin API documentation regarding fetching of scheduled tasks. ([\element-hq#18516](element-hq#18516)) - Mark the new module APIs in this release as experimental. ([\element-hq#18536](element-hq#18536)) - Mark dehydrated devices in the [List All User Devices Admin API](https://element-hq.github.io/synapse/latest/admin_api/user_admin_api.html#list-all-devices). ([\element-hq#18252](element-hq#18252)) - Reduce disk wastage by cleaning up `received_transactions` older than 1 day, rather than 30 days. ([\element-hq#18310](element-hq#18310)) - Distinguish all vs local events being persisted in the "Event Send Time Quantiles" graph (Grafana). ([\element-hq#18510](element-hq#18510))
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Distinguish all vs local events being persisted in the "Event Send Time Quantiles" graph in Grafana
As discovered by @devonh, we use
synapse_storage_events_persisted_events_total(which tracks all persisted events) for the "Events" rate in the "Event Send Time Quantiles" graph. This is pretty misleading as I would expect it to be the rate of events being sent given the graph title, "Event Send Time Quantiles".Since the event persistence queues are shared for local and remote events from federation and will block local events being sent, I think it does still make sense to have the event persist rate. I've updated the graph to include the rate of "Local events being persisted" and the rate of "All events being persisted". I think this properly disambiguates and clarifies what the graph is trying to show.
Pull Request Checklist
EventStoretoEventWorkerStore.".code blocks.