Skip to content

Conversation

OrlovEvgeny
Copy link
Contributor

@OrlovEvgeny OrlovEvgeny commented Jul 20, 2025

This PR implements the uc logs command for viewing aggregated logs from all containers of a service across the Uncloud cluster

Part 0

  • New uc logs <service> command that aggregates logs from all service containers
  • Real-time streaming via gRPC from each machine in the cluster
  • Support for standard Docker log options: --follow, --tail, --timestamps, --since, --until

Two Merge Models:

  • Fast mode (default): Logs are printed immediately as received
    • Minimal latency and resource usage
    • Best-effort ordering between machines
    • Suitable for common debugging
  • Strict ordering mode (--strict-order flag):
    • Guaranteed chronological order across all machines
    • Uses min-heap based k-way merge algorithm

Commands

uncloud git:(feat/uncloud-logs) ./uc log --help

View logs from all replicas of a service.

The logs command retrieves and displays logs from all running replicas
of the specified service across all machines in the cluster.

Usage:
  uncloud logs SERVICE [flags]

Aliases:
  logs, log

Flags:
  -c, --context string   Name of the cluster context. (default is the current context)
  -f, --follow           Follow log output
  -h, --help             help for logs
      --since string     Show logs since timestamp or relative (42m for 42 minutes)
      --strict-order     Merge logs in strict chronological order (slower but accurate)
      --tail int         Number of lines to show from the end of the logs (default -1)
  -t, --timestamps       Show timestamps
      --until string     Show logs before a timestamp or relative (42m for 42 minutes)

Global Flags:
      --connect string          Connect to a remote cluster machine without using the Uncloud configuration file.
                                Format: [ssh://]user@host[:port] or tcp://host:port
      --uncloud-config string   Path to the Uncloud configuration file. (default "~/.config/uncloud/config.yaml")

Examples

Show details... - Show the last 5 lines of each replica
./uc log excalidraw --tail 5
image
  • Show logs for the last 6 minutes
./uc log excalidraw --since 6m
image
  • Show the last 10 lines of each replica, and merge logs in strict chronological order (slower but accurate)
./uc log excalidraw -t --tail 10 --strict-order
image

Part 1 WIP

Implement a persistent log storage that saves container logs to disk on each machine, enabling access to historical logs even after containers are removed or restarted

related issue #12

@OrlovEvgeny
Copy link
Contributor Author

Hey @tonyo @psviderski !
As this PR is still wip, feel free to drop any suggestions for improving the logger
happy to hear your thoughts

@tonyo
Copy link
Collaborator

tonyo commented Jul 20, 2025

Great stuff 💪

Haven't looked too deep yet, but big +1 for following the docker / docker-compose semantics.

Implement a persistent log storage that saves container logs to disk on each machine, enabling access to historical logs even after containers are removed or restarted

That might blow up the PR size, I'd suggest to leave it for a follow-up PR and focus on basic functionality and user interface here.

Copy link
Owner

@psviderski psviderski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the direction!

To give you some food for thoughts, here is a talk about grpc-proxy from one of the Sidero (Talos linux) developers who forked grpc-proxy and add support for 1-Many requests, including for streaming methods. We use this project.

I wonder, if we can encapsulate some of the complexity currently implemented on the client in a server side gRPC call ServiceLogs. Note that this call could be broadcasted to N machines at the same time and it will return a stream that would combine all the streams from the machines. So basically all the multiplexing will be done by grpc-proxy.

What we'd need to do on the client is to reorder log entries if strict order is required. I believe this could reliably be done by using some buffer and periodic checkpoint entries when containers don't send logs. This way we can flush the next log entry once we received at least one entry from each machine.

This idea might not work though but feel free to investigate if you like.

// Read initial entries from each channel
for _, ch := range machineChannels {
select {
case entry, ok := <-ch:
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a risk that a particular machine won't have any logs for a while so we will block here?

})
}
default:
// Channel is truly empty, don't re-add
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could it become non empty later if we're using the --follow mode?
This one could be tricky. I have a feeling that we might need to send empty "checkpoint" entries on a regular interval to streams when there are no container logs. They may help distinguish the cases when there are issues with communicating with the machine and there are no logs from this machine.


message ContainerLogsResponse {
// Stream type: 1 = stdout, 2 = stderr
int32 stream_type = 1;
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI protobuf support enums:

enum RecordType {
UNSPECIFIED = 0;
A = 1;
AAAA = 2;
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants