Feat: Implement clean, graceful shutdown for RMB Peer

[sameh-farouk]

Is your feature request related to a problem? Please describe

Implement a graceful shutdown for rmb-sdk-go/peer so all relay connection goroutines and the processing loop exit cleanly without races, leaks, or panics.

• Avoid closing shared channels (which may lead to race/panic).
• Proactively send WebSocket close handshake to reduce protocol errors (observed in the relay-side logs).
• Close Substrate connection

Motivation

• goleak report goroutines like:
  - net.(*sysDialer).dialParallel (from websocket dial)
  • gethrpc.(*Client).dispatch (substrate client reader)
• Some practice in the code noticed that it risked race conditions due to unsafe channel closes and could leave goroutines running
• Websocket “connection reset without closing handshake” errors on the relay-side logs on shutdown of the peer.

Describe the solution you'd like

NA

[sameh-farouk]

Update:
Added tests to show the goroutines leak before and after the fixes.

Those goroutines are expected “before” and are eliminated “after” by threading ctx through connect/retry and closing substrate on shutdown.

Before:

=== RUN   TestPeerShutdownNoLeak
{"level":"debug","url":"wss://tfchain.dev.grid.tf/ws","time":"2025-08-13T02:08:13+03:00","message":"connecting"}
2025/08/13 02:08:13 Connecting to wss://tfchain.dev.grid.tf/ws...
{"level":"info","twin":5993,"session":"","time":"2025-08-13T02:09:12+03:00","message":"starting peer"}
--- PASS: TestPeerShutdownNoLeak (62.10s)
PASS
{"level":"debug","url":"wss://loyal-equal-bison.ngrok-free.app","time":"2025-08-13T02:09:15+03:00","message":"connecting"}
goleak: Errors on successful test run: found unexpected goroutines:
[Goroutine 19 in state select, with github.com/centrifuge/go-substrate-rpc-client/v4/gethrpc.(*Client).dispatch on top of the stack:
github.com/centrifuge/go-substrate-rpc-client/v4/gethrpc.(*Client).dispatch(0xc0002dc000, {0xecbc00, 0xc0003dc550})
        /home/sameh/go/pkg/mod/github.com/centrifuge/go-substrate-rpc-client/v4@v4.0.12/gethrpc/client.go:560 +0x4e9
created by github.com/centrifuge/go-substrate-rpc-client/v4/gethrpc.initClient in goroutine 36
        /home/sameh/go/pkg/mod/github.com/centrifuge/go-substrate-rpc-client/v4@v4.0.12/gethrpc/client.go:223 +0x585
 Goroutine 6 in state select, with net.(*sysDialer).dialParallel on top of the stack:
net.(*sysDialer).dialParallel(0xc0001ca000, {0xecac78, 0xc0001d4150}, {0xc000298380, 0x6, 0x8}, {0xc000298400, 0x6, 0x8})
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/net/dial.go:628 +0x78c
net.(*Dialer).DialContext(0xc0002ee240, {0xecac78, 0xc0001d4150}, {0xcea437, 0x3}, {0xc000034240, 0x24})
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/net/dial.go:578 +0xc52
github.com/gorilla/websocket.(*Dialer).DialContext.func4({0xcea437, 0x3}, {0xc000034240, 0x24})
        /home/sameh/go/pkg/mod/github.com/gorilla/websocket@v1.5.3/client.go:279 +0x8b
github.com/gorilla/websocket.(*Dialer).DialContext.func5({0xcea437, 0x3}, {0xc000034240, 0x24})
        /home/sameh/go/pkg/mod/github.com/gorilla/websocket@v1.5.3/client.go:287 +0x82
github.com/gorilla/websocket.(*Dialer).DialContext(0x129a3c0, {0xecab60, 0x12c5ec0}, {0xc0002160f0, 0xe2}, 0x0)
        /home/sameh/go/pkg/mod/github.com/gorilla/websocket@v1.5.3/client.go:321 +0x1cf0
github.com/gorilla/websocket.(*Dialer).Dial(...)
        /home/sameh/go/pkg/mod/github.com/gorilla/websocket@v1.5.3/client.go:117
github.com/threefoldtech/tfgrid-sdk-go/rmb-sdk-go/peer.(*InnerConnection).connect(0xc0003dc230)
        /home/sameh/Projects/tfgrid-sdk-go/rmb-sdk-go/peer/connection.go:224 +0x251
github.com/threefoldtech/tfgrid-sdk-go/rmb-sdk-go/peer.(*InnerConnection).listenAndServe(0xc0003dc230, {0xecac78, 0xc0002e9500}, 0xc00002c1c0)
        /home/sameh/Projects/tfgrid-sdk-go/rmb-sdk-go/peer/connection.go:190 +0x3a
github.com/threefoldtech/tfgrid-sdk-go/rmb-sdk-go/peer.(*InnerConnection).Start.func1()
        /home/sameh/Projects/tfgrid-sdk-go/rmb-sdk-go/peer/connection.go:176 +0x1b3
created by github.com/threefoldtech/tfgrid-sdk-go/rmb-sdk-go/peer.(*InnerConnection).Start in goroutine 36
        /home/sameh/Projects/tfgrid-sdk-go/rmb-sdk-go/peer/connection.go:172 +0x112
 Goroutine 20 in state IO wait, with internal/poll.runtime_pollWait on top of the stack:
internal/poll.runtime_pollWait(0x7322ec932ea8, 0x72)
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc0002dc6a0, 0x72, 0x0)
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/internal/poll/fd_poll_runtime.go:84 +0xb1
internal/poll.(*pollDesc).waitRead(...)
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0002dc680, {0xc00058c000, 0xa000, 0xa000})
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/internal/poll/fd_unix.go:165 +0x453
net.(*netFD).Read(0xc0002dc680, {0xc00058c000, 0xa000, 0xa000})
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/net/fd_posix.go:55 +0x4b
net.(*conn).Read(0xc000196208, {0xc00058c000, 0xa000, 0xa000})
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/net/net.go:194 +0xad
crypto/tls.(*atLeastReader).Read(0xc0003c6390, {0xc00058c000, 0xa000, 0xa000})
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/crypto/tls/conn.go:809 +0x84
bytes.(*Buffer).ReadFrom(0xc00031c2b8, {0xec76a0, 0xc0003c6390})
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/bytes/buffer.go:211 +0x10f
crypto/tls.(*Conn).readFromUntil(0xc00031c008, {0xec7840, 0xc000196208}, 0x5)
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/crypto/tls/conn.go:831 +0x1d6
crypto/tls.(*Conn).readRecordOrCCS(0xc00031c008, 0x0)
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/crypto/tls/conn.go:629 +0x359
crypto/tls.(*Conn).readRecord(...)
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/crypto/tls/conn.go:591
crypto/tls.(*Conn).Read(0xc00031c008, {0xc000479800, 0x400, 0x2?})
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/crypto/tls/conn.go:1385 +0x2d0
bufio.(*Reader).fill(0xc0003d91a0)
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/bufio/bufio.go:113 +0x2a4
bufio.(*Reader).Peek(0xc0003d91a0, 0x2)
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/bufio/bufio.go:152 +0xc5
github.com/gorilla/websocket.(*Conn).read(0xc0001fa840, 0x2)
        /home/sameh/go/pkg/mod/github.com/gorilla/websocket@v1.5.3/conn.go:371 +0x53
github.com/gorilla/websocket.(*Conn).advanceFrame(0xc0001fa840)
        /home/sameh/go/pkg/mod/github.com/gorilla/websocket@v1.5.3/conn.go:809 +0x11b
github.com/gorilla/websocket.(*Conn).NextReader(0xc0001fa840)
        /home/sameh/go/pkg/mod/github.com/gorilla/websocket@v1.5.3/conn.go:1009 +0x134
github.com/gorilla/websocket.(*Conn).ReadJSON(0xc0001fa840, {0xc33700, 0xc0003c6378})
        /home/sameh/go/pkg/mod/github.com/gorilla/websocket@v1.5.3/json.go:50 +0x35
github.com/centrifuge/go-substrate-rpc-client/v4/gethrpc.(*jsonCodec).Read(0xc0003dc550)
        /home/sameh/go/pkg/mod/github.com/centrifuge/go-substrate-rpc-client/v4@v4.0.12/gethrpc/json.go:196 +0x8a
github.com/centrifuge/go-substrate-rpc-client/v4/gethrpc.(*Client).read(0xc0002dc000, {0xecbc00, 0xc0003dc550})
        /home/sameh/go/pkg/mod/github.com/centrifuge/go-substrate-rpc-client/v4@v4.0.12/gethrpc/client.go:633 +0xa4
created by github.com/centrifuge/go-substrate-rpc-client/v4/gethrpc.(*Client).dispatch in goroutine 19
        /home/sameh/go/pkg/mod/github.com/centrifuge/go-substrate-rpc-client/v4@v4.0.12/gethrpc/client.go:557 +0x217
 Goroutine 21 in state IO wait, with internal/poll.runtime_pollWait on top of the stack:
internal/poll.runtime_pollWait(0x7322ec932d90, 0x77)
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc00021e320, 0x77, 0x0)
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/internal/poll/fd_poll_runtime.go:84 +0xb1
internal/poll.(*pollDesc).waitWrite(...)
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/internal/poll/fd_poll_runtime.go:93
internal/poll.(*FD).WaitWrite(...)
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/internal/poll/fd_unix.go:692
net.(*netFD).connect(0xc00021e300, {0xecac78, 0xc0002e8000}, {0xc00006f888?, 0x4d84d5?}, {0xec6780, 0xc000530000})
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/net/fd_unix.go:141 +0xb65
net.(*netFD).dial(0xc00021e300, {0xecac78, 0xc0002e8000}, {0xecc4b8, 0x0}, {0xecc4b8, 0xc00019dc20}, 0x0)
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/net/sock_posix.go:124 +0x479
net.socket({0xecac78, 0xc0002e8000}, {0xcea437, 0x3}, 0x2, 0x1, 0x0, 0x0, {0xecc4b8, 0x0}, ...)
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/net/sock_posix.go:70 +0x3ec
net.internetSocket({0xecac78, 0xc0002e8000}, {0xcea437, 0x3}, {0xecc4b8, 0x0}, {0xecc4b8, 0xc00019dc20}, 0x1, 0x0, ...)
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/net/ipsock_posix.go:167 +0x127
net.(*sysDialer).doDialTCPProto(0xc0001ca000, {0xecac78, 0xc0002e8000}, 0x0, 0xc00019dc20, 0x0)
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/net/tcpsock_posix.go:85 +0x189
net.(*sysDialer).doDialTCP(...)
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/net/tcpsock_posix.go:75
net.(*sysDialer).dialTCP(0xc0001ca000, {0xecac78, 0xc0002e8000}, 0x0, 0xc00019dc20)
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/net/tcpsock_posix.go:71 +0x14a
net.(*sysDialer).dialSingle(0xc0001ca000, {0xecac78, 0xc0002e8000}, {0xec9518, 0xc00019dc20})
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/net/dial.go:721 +0x4e5
net.(*sysDialer).dialSerial(0xc0001ca000, {0xecac08, 0xc0003dc0a0}, {0xc000298380, 0x6, 0x8?})
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/net/dial.go:686 +0x292
net.(*sysDialer).dialParallel.func1({0xecac08, 0xc0003dc0a0}, 0x1)
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/net/dial.go:606 +0xf0
created by net.(*sysDialer).dialParallel in goroutine 6
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/net/dial.go:621 +0x425
 Goroutine 22 in state select, with net.(*netFD).connect.func2 on top of the stack:
net.(*netFD).connect.func2()
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/net/fd_unix.go:118 +0xba
created by net.(*netFD).connect in goroutine 21
        /home/sameh/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/net/fd_unix.go:117 +0x576
]
FAIL    github.com/threefoldtech/tfgrid-sdk-go/rmb-sdk-go/peer  62.567s
FAIL

After:

=== RUN   TestPeerShutdownNoLeak
{"level":"debug","url":"wss://tfchain.dev.grid.tf/ws","time":"2025-08-13T02:16:03+03:00","message":"connecting"}
2025/08/13 02:16:03 Connecting to wss://tfchain.dev.grid.tf/ws...
{"level":"info","twin":5993,"session":"","time":"2025-08-13T02:16:22+03:00","message":"starting peer"}
--- PASS: TestPeerShutdownNoLeak (20.05s)
{"level":"debug","url":"wss://loyal-equal-bison.ngrok-free.app","error":"context deadline exceeded","time":"2025-08-13T02:16:23+03:00","message":"relay worker stopping: context canceled"}
{"level":"debug","error":"context deadline exceeded","time":"2025-08-13T02:16:23+03:00","message":"peer parent context canceled; closing peer"}
{"level":"debug","reason":"peer context canceled","error":"context deadline exceeded","time":"2025-08-13T02:16:23+03:00","message":"peer.process exited"}
{"level":"debug","time":"2025-08-13T02:16:23+03:00","message":"closing substrate connection"}
PASS
ok      github.com/threefoldtech/tfgrid-sdk-go/rmb-sdk-go/peer  21.070s