grpcproxy: use metadata instead of context withvalue in with client auth token #19033
Conversation
|
Hi @krijohs. Thanks for your PR. I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Added test case which reproduces the issue, with included change it passes but without fails |
krijohs
force-pushed
the
grpcproxy-client-auth-token
branch
from
3d3b143 to
8d9e89f
Compare
|
Hi @krijohs, thanks for your pull request. Ideally, we would want to discuss the issue and possible solutions before a pull request. Could you please open an issue so other members with more expertise in this area can jump in? Thanks again. |
|
Hello @ivanvc ok, got it will open an issue so possible solutions can be discussed, thanks. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions. |
|
/reopen |
|
/ok-to-test |
krijohs
force-pushed
the
grpcproxy-client-auth-token
branch
from
8d9e89f to
78a893d
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
... and 77 files with indirect coverage changes @@ Coverage Diff @@
## main #19033 +/- ##
==========================================
- Coverage 69.21% 64.96% -4.26%
==========================================
Files 419 419
Lines 34745 34770 +25
==========================================
- Hits 24049 22588 -1461
- Misses 9300 10815 +1515
+ Partials 1396 1367 -29 Continue to review full report in Codecov by Sentry.
🚀 New features to boost your workflow:
|
|
/retest |
krijohs
force-pushed
the
grpcproxy-client-auth-token
branch
from
100f331 to
c6735a5
Compare
|
@krijohs, could you please rebase your branch with the latest upstream |
…uth token Signed-off-by: Kristoffer Johansson <[email protected]>
krijohs
force-pushed
the
grpcproxy-client-auth-token
branch
from
c6735a5 to
873b3bc
Compare
|
@ivanvc sure no problem, just rebased and pushed |
|
/retest |
There was a problem hiding this comment.
Although not an expert on this part of the code, I ran the tests without the fix to util.go, and they failed.
/cc @ahrtr @serathius
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: ivanvc, krijohs The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
| require.NoError(t, clusterCli.Put(ctx, "/test/1", "test", config.PutOptions{})) | ||
| require.NoError(t, err) | ||
|
|
||
| time.Sleep(time.Second * 2) |
There was a problem hiding this comment.
This seems unnecessary.
| time.Sleep(time.Second * 2) |
There was a problem hiding this comment.
Updated, was long time ago i submitted PR, but this sleep is to ensure the token has expired when the subsequent watchers connect, to be able to reproduce issue in the e2e test
| md := metadata.Pairs(rpctypes.TokenFieldNameGRPC, token) | ||
| return metadata.NewOutgoingContext(ctx, md) |
There was a problem hiding this comment.
The change looks good.
It look like the previous implementation is wrong. All the usage on rpctypes.TokenFieldNameGRPC should be via the package google.golang.org/grpc/metadata (e.g,metadata.FromIncomingContext, metadata.NewOutgoingContext). However, grpcproxy gets the client auto token using grpc/metadata package, but re-add it using raw context WithValue method.
It seems that the grpcprocy couldn't pass the client auth token to the backend etcdserver at all. Could anyone double confirm this? thx
There was a problem hiding this comment.
From testing i noticed that the proxy did pass auth tokens to the etcd server successfully for the initial client that connected. That is, the first client connecting to the proxy worked fine and could authenticate and establish the watch. But after the initial client's token expired subsequent clients that tried to connect could not.
This can be reproduced with the e2e test
krijohs
force-pushed
the
grpcproxy-client-auth-token
branch
from
873b3bc to
b1fc707
Compare
|
/retest |
Signed-off-by: Kristoffer Johansson <[email protected]>
Signed-off-by: Kristoffer Johansson <[email protected]>
krijohs
force-pushed
the
grpcproxy-client-auth-token
branch
from
b1fc707 to
5edb2be
Compare
|
/retest |
Change to use metadata instead of
context.WithValueto ensure each proxy watcher client has a new stream created with its token.Previously context.WithValue resulted in
streamKeyFromCtxreturning an empty string in the clientv3 watcher, causing stream reuse.When new clients connected to proxy after the token expired (token for the initial client which connected) the reused stream's context would still contain the expired token. This caused auth failures when
isWatchPermittedon cluster checked the stream's context resulting in hanging proxy watcher clients.Issue can be reproduced by setting a low
--auth-token-ttlon cluster and connect 1 client first to proxy and then connect a second one after token expired.