Skip to content

Conversation

lepetitops
Copy link
Contributor

Pull Request Description

Add operationId query parameter support for /backup/status endpoint

Overview

This PR adds support for the operationid query parameter to the /backup/status endpoint, allowing users to retrieve the status of a specific operation by its ID.

Problem

Currently, the /backup/status endpoint only returns the status of the most recent operation. When multiple backup operations are running or have been executed, there's no way to check the status of a specific operation using its operation_id that was returned during operation creation.

Solution

  • Extended ActionRowStatus struct with OperationId field to store operation IDs in status tracking
  • Added new methods in AsyncStatus:
    • StartWithOperationId() - starts commands with explicit operationId
    • GetStatusByOperationId() - retrieves status by operationId
  • Updated all API handlers (create, restore, upload, download) to use StartWithOperationId
  • Enhanced /backup/status endpoint to support optional operationid query parameter
  • Maintained full backward compatibility - existing behavior unchanged when no operationid parameter provided

API Usage

Get status of all operations (existing behavior):

curl -s localhost:7171/backup/status | jq .

Get status of specific operation by operationId:

curl -s "localhost:7171/backup/status?operationid=<operation_id>" | jq .

Example Workflow

# 1. Create backup and get operation_id
response=$(curl -s -X POST localhost:7171/backup/create?name=my-backup)
operation_id=$(echo $response | jq -r '.operation_id')

# 2. Check specific operation status
curl -s "localhost:7171/backup/status?operationid=$operation_id" | jq .

Backward Compatibility

Fully backward compatible

  • Existing clients continue to work without changes
  • operationid parameter is optional
  • When operationid is not provided, behavior is identical to current implementation

Limitations

  • operationId is stored in memory only (not persistent across server restarts)
  • operationId is auto-generated as UUID for each operation
  • Invalid operationId returns empty array []

Testing

  • ✅ Code compiles without errors
  • ✅ Backward compatibility verified
  • ✅ New functionality tested with unit tests
  • ✅ Edge cases handled (non-existent operationId)

Files Changed

  • pkg/status/status.go - Extended structures and added new methods
  • pkg/server/server.go - Updated handlers and httpStatusHandler

This enhancement improves the API usability for monitoring specific operations while maintaining full backward compatibility.

Copy link
Collaborator

@Slach Slach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

look
test/intergration_test.go

test failed

2025-07-23T11:06:26.9842739Z     integration_test.go:4071: 
2025-07-23T11:06:26.9844263Z         	Error Trace:	/home/runner/work/clickhouse-backup/clickhouse-backup/test/integration/integration_test.go:4071
2025-07-23T11:06:26.9847086Z         	            				/home/runner/work/clickhouse-backup/clickhouse-backup/test/integration/integration_test.go:1228
2025-07-23T11:06:26.9849770Z         	            				/home/runner/work/clickhouse-backup/clickhouse-backup/test/integration/integration_test.go:1171
2025-07-23T11:06:26.9850726Z         	Error:      	Received unexpected error:
2025-07-23T11:06:26.9852169Z         	            	code: 117, message: Unknown field found while parsing JSONEachRow format: operation_id
2025-07-23T11:06:26.9852909Z         	Test:       	TestServerAPI

- Updated ClickHouse table schema for system.backup_actions to include operation_id field
- This fixes JSONEachRow parsing error when the API returns ActionRowStatus with operation_id
- Updated both the actual table creation query and documentation comment
@lepetitops lepetitops requested a review from Slach July 23, 2025 14:13
@lepetitops
Copy link
Contributor Author

Could you let me know if I should update the README as part of this PR, or will one of the maintainers take care of it after approval?

@coveralls
Copy link

coveralls commented Jul 23, 2025

Pull Request Test Coverage Report for Build 16474847974

Details

  • 21 of 40 (52.5%) changed or added relevant lines in 2 files are covered.
  • 272 unchanged lines in 11 files lost coverage.
  • Overall coverage decreased (-1.9%) to 66.349%

Changes Missing Coverage Covered Lines Changed/Added Lines %
pkg/server/server.go 8 11 72.73%
pkg/status/status.go 13 29 44.83%
Files with Coverage Reduction New Missed Lines %
pkg/backup/restore.go 1 69.17%
cmd/clickhouse-backup/main.go 2 77.27%
pkg/backup/create.go 2 74.44%
pkg/storage/object_disk/object_disk.go 5 66.48%
pkg/config/config.go 8 71.98%
pkg/status/status.go 9 66.86%
pkg/backup/backuper.go 17 73.66%
pkg/server/server.go 18 58.91%
pkg/storage/general.go 20 59.93%
pkg/storage/s3.go 25 46.24%
Totals Coverage Status
Change from base Build 16322345567: -1.9%
Covered Lines: 9754
Relevant Lines: 14701

💛 - Coveralls

Copy link
Collaborator

@Slach Slach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, please change ReadMe.md with related changes.

@lepetitops
Copy link
Contributor Author

I’ve updated the ReadMe.md and Examples.md

@lepetitops lepetitops requested a review from Slach July 23, 2025 15:23
Copy link
Collaborator

@Slach Slach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, actually you already have commandId
Why just not return it in all cases when status.Start
and poll status with CommandId instead of new operation_id?

@lepetitops
Copy link
Contributor Author

lepetitops commented Jul 24, 2025

commandId is just an array of integer indices used for internal management - would it be good API design to expose that to clients?
since operationId is already used as the external command identifier (for callbacks), it makes sense to use it for external status tracking too (imho)

or maybe it makes sense to move away from using two different IDs, stick with only one, and implement its generation using UUID?

@lepetitops lepetitops requested a review from Slach July 24, 2025 11:22
@Slach
Copy link
Collaborator

Slach commented Jul 24, 2025

ok. got it, let's merge

@Slach Slach added this to the 2.6.30 milestone Jul 24, 2025
@Slach Slach merged commit 16fa5da into Altinity:master Jul 24, 2025
45 of 46 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants