Disable `secrets` validation in jobs when using dynamic `environment`

[SpencerKaiser]

Describe the bug
Referencing a secret that is stored at the environment level instead of a repository secret throws an warning

To Reproduce
Steps to reproduce the behavior:

1. Make a reference to a secret that is NOT at the repo level and is instead an environment secret
2. Observe warning inline and in the Problems tab

Expected behavior
Warnings will only be shown for secrets that are not found at the environment level OR the repo level

Screenshots

Extension Version
v0.25.2

[cschleiden]

@SpencerKaiser are you able to share more of the workflow? Are those variables defined at the job or step level? Does the job contain environment: prod or is it using a dynamic expression to determine the environment to use?

[SpencerKaiser]

I shared snippets below but I'd rather not share the entirety unless I can do it via email or Slack or something... want me to email it to the email on your profile? Looks like you're staff at GitHub, so feel free to grab my email or whatever if you have access to it 👀

To answer your questions, the variables are just passed directly into steps and the environment has a default but I use an input via a workflow_dispatch to override it (see the bottom for a very important related note):

Trigger (push was removed for brevity):

on:
  workflow_dispatch:
    inputs:
      environment:
        description: 'Deployment Environment'
        type: environment
        required: true

Parent job initialization with relevant step (other steps removed):

jobs:
  release:
    name: Build and Push
    environment:
      name: ${{ github.event.inputs.environment || 'stage' }}
    runs-on: ubuntu-latest
    steps:
      - name: Deploy stack
        run: yarn [REDACTED]
        env:
          DEPLOYMENT_ENVIRONMENT: ${{ env.ENVIRONMENT_NAME }}
          SENTRY_DSN: ${{ secrets.SENTRY_DSN }}
          SENTRY_ENVIRONMENT: ${{ secrets.SENTRY_ENVIRONMENT }}
          ACM_CERTIFICATE_ID: ${{ secrets.ACM_CERTIFICATE_ID }}

We're setting about 20 environment variables in that step by passing them to a yarn script; every single repo secret does not have the warning and every single environment secret shows the warning.

💡 Lastly, you're definitely on to something with the dynamic expression... I replaced the environment to make it static and the warnings cleared:

jobs:
  release:
    name: Build and Push
    environment:
      name: stage

---

More snippets if needed

Two other steps with different uses are also throwing the warning:

      - uses: 8BitJonny/gh-get-current-pr@1.4.0
        id: PR
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          sha: ${{ github.event.pull_request.head.sha }} # Context access might be invalid: pull_request

In the above the thing throwing the warning isn't a secret; happy to open an issue for that too if you think it's unrelated

Another job with the same error:

  notify-on-failure:
    name: Send Status to Slack
    needs: [release]
    if: failure()
    runs-on: ubuntu-latest
    steps:
      - name: Failure Notification
        id: slack
        uses: slackapi/slack-github-action@v1.16.0
        with:
          channel-id: [REDACTED TEXT]
          payload: [REDACTED JSON]
        env:
          SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }} # Context access might be invalid: SLACK_BOT_TOKEN

[cschleiden]

@SpencerKaiser no that helps, thanks for the details!

Yes, so we detect what environment is being used in a job, and only then add the environment secrets to the secrets context for that job.

Obviously this doesn't work when environment: uses a dynamic expression that we cannot evaluate at edit time. The fix for now would be to disable secrets validation for such a job.

I've updated the issue title to reflect this a bit better.