Migration of workloads

[sabrinasadik]

Is your feature request related to a problem? Please describe

Scenario 1: My seed key was compromised. I want to change the wallet address that is linked to my workloads.
Scenario 2: As a company, we have some deployments on the TF Grid that are managed by a team. One of the team members has left the company and should not have access to those deployments anymore.

Describe the solution you'd like

It should be possible to change the ownership (i.e., wallet address) of the workloads.

Describe alternatives you've considered

Sameh checked, but the 'bonded' account can only be used to determine the discount tier (using two wallet balances), not to alter who pays for the workload.

Additional context

We need to think more organisational POV as well, instead of just individual people. Workloads should be migratable or at least the wallet address should be changeable. On DO or OVH for example, I can change my payment method. I believe this should be possible with TF as well.

[sameh-farouk]

Below is a concise design that fits our constraints:
Goals:

• Migrate all workloads from old account A to new account B.
• Do NOT pause/destroy deployments on nodes.
• Old account stops paying and controlling; new account pays/controls.
• Backwards-compatible: existing flows work unchanged when not used.

Facts:

• Contracts store only twin_id and not the account.
• Billing dynamically resolves the payer from the twin’s current account each cycle, and Stash is resolved per twin.
• Authorization for contract updates/cancel checks the twin’s current account at runtime.

Contracts and billing depend on the twin’s current account_id. If you change the twin’s account_id, all existing contracts of that twin will immediately bill the new account and accept control from the new account without touching each contract.

Proposal: Twin ownership transfer (covers “all workloads of account A”)

Instead of iterating over contracts or re-creating them, implement a two-phase transfer of the twin’s ownership (account) in pallet-tfgrid:

• New extrinsics (two-phase, additive):
  • request_twin_transfer(twin_id) signed by new_account (Should have accepted T&C, and have no twin yet).
    • Stores TwinTransferRequest { twin_id, old_account, new_account, expiry, status=Pending }.
    • Emits TwinTransferRequested(twin_id, old_account, new_account).
  • accept_twin_transfer(request_id) signed by the current twin owner (old_account).
    • Validates pending request.
    • Atomically:
      • Update Twins[twin_id].account_id = new_account.
      • Update TwinIdByAccountID: remove old mapping; insert new mapping.
    • Emits:
      • TwinOwnershipTransferred(twin_id, old_account, new_account).
      • plus the existing TwinUpdated(twin) to preserve existing listener behavior.
• Optional: cancel_twin_transfer(request_id) and expiry handling.

What this achieves:

• All workloads/contracts for that twin move to the new payer/controller instantly (no per-contract updates).
• Billing loop continues unchanged; it will start reserving from new_account on next bill since it calls Twins[twin_id].account_id dynamically.
• Authorization on all contract extrinsics moves to new_account immediately (since checks read the twin live).
• No zos changes or workload pauses; no contract ID churn.

Pros:

• O(1) state change: no iteration over potentially thousands of contracts.
• No redeploy or zos-side rewrite.
• Backward compatible; internal pallets already resolve payer via twin.
• No weight spikes; no batching needed.

Remaining challenge:

• distribution like billing always reads the current twin owner dynamically
• Reserved funds are held on the account that performed the last reserve. After a twin transfer, src_twin.account_id points to the new account, but previously reserved funds still sit on the old account. Distribution then sources from the new account and can error.

Options:

• Option A: move everything reserved right now without per-contract checks (I'm leaning toward this one).

  • Trade-off: While no need to pre-settle or heavy iteration at transfer time, if any other module uses reserve on the old account, you’ll move their funds too (bad). However, the reserved balance is used today exclusively by TF smart‑contract billing on our chain (and I will double-check).

• Option B: Track reserve source per contract

  • Trade-off: While no need to pre-settle or heavy iteration at transfer time, it requires a runtime storage migration to initialize a new field to track the reserve source.

[sameh-farouk]

@sabrinasadik, regarding the remaining balance in the old account, should it be handled as part of the ownership transfer, or should the user be responsible for transferring it separately as a standard transfer transaction?

The reserved portion of the balance (the due amount, which has been partially processed and is still pending transfer to Binficatures) is already being moved to the new account, which means a partial fund transfer is already taking place. The question is whether to fully automate the entire process (ownership transfer + fund transfer) or only do the ownership transfer and leave the non-reserved portion of the funds in the old account for the user to manage it himself and potentially initiate a standard transfer.

[sabrinasadik]

I think the user should be responsible for that. If they want to transfer, they can do that themselves.

[sameh-farouk]

I think the user should be responsible for that. If they want to transfer, they can do that themselves.

Agree

[sameh-farouk]

Update:

• A simple, robust two-step twin ownership transfer flow has been implemented: request → accept, with cancel by current owner.
• The design emphasizes clarity, safety, and easy client/indexer consumption.

What changed from the initial design

• Initiator

  • The current owner initiates the transfer to a specified new account.
  • The new account must explicitly accept; the current owner can cancel at any time before acceptance.

• State tracking

  • No explicit “status” field is stored.
  • A transfer is considered “pending” if a request exists; absence means no pending transfer.
  • Exactly one pending request per twin is enforced.

• Expiry

  • No expiry logic is enforced in v1.
  • A creation block number (created_at) is recorded on each request for future optional cleanup/housekeeping.

• Naming semantics

  • Renamed “old/new account” to “from/to” for transfers.
  • All events and docs now use from/to consistently.

Event model

• Requested: includes request_id, twin_id, from, to
• Transferred: includes request_id, twin_id, from, to
• Canceled: includes request_id, twin_id, from, to
• Rationale:
  • request_id enables correlation and makes event streams indexer-friendly.
  • Including twin_id and from/to in every event reduces client lookups and simplifies analytics.

Validation and safety

• Preconditions:
  • Twin must exist, and the origin must be its current owner to request/cancel.
  • The “to” account must have accepted Terms & Conditions and must not already own a twin.
  • Only the “to” account can accept.
• Accounting:
  • Reserved balance is repatriated from “from” to “to” upon acceptance, before ownership change.

Client and ecosystem impact

• Clients and indexers can track transfers exclusively via events, keyed by request_id.

Testing and status

• Unit tests cover request, accept, cancel, error paths, and event emission.

Rationale behind final decisions

• Switching initiator to current owner:
  • Aligns with ownership authority; avoids push-transfers; makes accountability explicit.
• Dropping explicit status tracking:
  • Simplifies the lifecycle; storage presence is the source of truth; eliminates redundant state.
• Dropping expiry logic (adding created_at only):
  • Avoids premature complexity; enables optional cleanup later without constraining UX now.
• Renaming to from/to:
  • Matches common “transfer” semantics; clearer and more concise for event streams.

Next steps:

• If desired, implement an expiry logic, and/or optional cleanup process (off-chain or periodic) that uses created_at to prune very old requests (optional).
• Update our GraphQL processor to store the new events.

[sameh-farouk]

Update:

• Benchmarks in /.github/workflows/060_generate_benchmark_weights.yml now take >2x longer and produce >2x higher weights than before.
• We rely on a consistent self-hosted runner to keep fees predictable. This looks like a hardware/environment change.

Impact

• Weights/fees appear inflated due to runner variance, not code changes.
• Release blocker until runner consistency is restored.

Hypothesis

• Self-hosted runner was replaced or reconfigured (lower-spec CPU, throttling, cgroup limits, etc).

Request to Ops

• Verify which host backs the runner label for the workflow and whether it changed.
• Confirm CPU model, cores/threads, frequencies, RAM, limits, and recent changes.
• Check Docker/base image, kernel, Rust toolchain, and background load.

Artifacts

• Last “good” vs first “regressed” runs (durations).
• Diff of generated weights (before vs after).
• Current runner metadata:

Current runner version: '2.328.0'
Runner name: 'threefoldtech-self-hosted'
Runner group name: 'digitaltwin'
Machine name: 'flistrunner'

• Old runner metadata:
  Old runner logs are no longer available on GitHub