From 0a1a94a4a5103541d450909deb22c9d6c20a1d04 Mon Sep 17 00:00:00 2001 From: Sam Bolian Date: Wed, 21 Dec 2022 15:58:02 -0600 Subject: [PATCH 1/5] restructured debugging files --- pages/debugging/application-issues.md | 45 --------- pages/debugging/index.md | 4 - pages/reference/troubleshooting.md | 134 -------------------------- src/NavData.tsx | 26 ++--- 4 files changed, 6 insertions(+), 203 deletions(-) delete mode 100644 pages/debugging/application-issues.md delete mode 100644 pages/debugging/index.md delete mode 100644 pages/reference/troubleshooting.md diff --git a/pages/debugging/application-issues.md b/pages/debugging/application-issues.md deleted file mode 100644 index 5d00cc23..00000000 --- a/pages/debugging/application-issues.md +++ /dev/null @@ -1,45 +0,0 @@ ---- -title: Application Issues -description: Figuring out what is going wrong with your deployed applications. ---- - -Issues with applications are often due to issues with the underlying Pods. You can use `kubectl` commands to get, find logs for, and delete pods as necessary. Note that Plural automatically configures `kubectl` for use with your Plural cluster. - -## With Plural CLI - -To find Pods related to an application with the CLI, you can run: - -``` -kubectl get pods -n -``` - -If you see failed Pods, you can get the logs for the Pods by running: - -``` -kubectl logs -n -``` - -We also curate a list of helpful logging shortcuts for each application, which you can use the `plural logs` subcommand for, eg with - -``` -plural logs list # shows all log tails available -plural logs tail # tails that specific log -``` - -To delete problematic Pods, run: - -``` -kubectl delete pod -``` - -In most cases, kubernetes will restart the pod for you. You can always also run `plural bounce` to regenerate your deleted Pods. - -## With Plural Console - -If you have the Plural Console installed, you can debug your Kubernetes Pods with the following steps: - -1. Navigate to the `Components` tab and click on the failing component - -2. Scroll to the `Pods` section at the bottom of the screen and hit the red trash can button located on the right of the screen to delete the failing Pod. - -3. Head back to the `Builds` tab in the sidebar and click the `Bounce` button in the top right to redeploy your deleted Pods. \ No newline at end of file diff --git a/pages/debugging/index.md b/pages/debugging/index.md deleted file mode 100644 index 066239e1..00000000 --- a/pages/debugging/index.md +++ /dev/null @@ -1,4 +0,0 @@ ---- -title: Debugging -description: Some tips to help debug your Plural installation. ---- \ No newline at end of file diff --git a/pages/reference/troubleshooting.md b/pages/reference/troubleshooting.md deleted file mode 100644 index b70a6572..00000000 --- a/pages/reference/troubleshooting.md +++ /dev/null @@ -1,134 +0,0 @@ ---- -title: Troubleshooting -description: >- - Here is a non-exhaustive list of common issues or errors during the install or - operation of Plural. If you see an error not listed here or get stuck in - general, just hop into our Discord for help. ---- - -## Git Errors -### Could not compare workspace to origin -`Failed to get git information: Could not compare current workspace to origin, do you have an 'origin' remote configured, or does your repo not have an inital commit?` - -This error either means you cannot push information to Git, or you do not have a remote branch set up to track your local one. Make sure you've added your SSH keys to Github and verify that your origin is set by running `git remote -v`. - -### Out of date -`Your local workspace is not in sync with remote, either 'git pull' recent changes or 'git push' any missed changes.` - -This error can happen if you're ahead of or behind your remote by a few commits, so try `git push` if `git pull` does not resolve the issue. - - -## Cloud Errors -### Google Credentials -`You don't have necessary services enabled. Please run: 'gcloud services enable serviceusage.googleapis.com cloudresourcemanager.googleapis.com container.googleapis.com' with an owner of the project to enable or enable them in the GCP console.` - -Ensure that you've run the gcloud command in the correct project, and make sure you have owner rights. - -### AWS Credentials -`Failed to get aws account (is your aws cli configured?)` - -Ensure your AWS CLI is set up, that you have the correct profile chosen, and that you're authenticated in to AWS. If necessary, run `export AWS_PROFILE=` in the terminal you are running Plural in and auth in with `aws sso login`. - - -### IAM policy update 403 permission denied - -You will see permission errors if your Cloud account does not have permissions to create the IAM roles needed by Plural. Apply the correct permissions to the user account that Plural is deploying as. - -### Requested project not found - -`Error 404: The requested project was not found., notFound` - -If you are running `plural build` and encounter a `project requested not found` error it's possible that your application default credential is set to the wrong gcp project. Run: - -``` -gcloud auth application-default login -``` - -to reset the credential and reauthorize the browser for the correct project. - -## Initialization Errors -### Workspace Initialization -`Your workspace hasn't been configured, try running 'plural init'` -`Could not find workspace.yaml file, you might need to run 'plural init'` - -Base cloud provider setup for a Plural repository is stored in your workspace.yaml file. This should be created when running `plural init`. - -### Kubconfig missing -`2022/12/19 16:56:24 stat /home/plural/.kube/config: no such file or directory` - -This will happen because for whatever reason your kubeconfig is not available locally. This will occasionally happen if your cloud shell pod was recently recreated, if you're using a new laptop, or if the file was deleted/expired accidentally. To fix, run: - -``` -plural wkspace kube-init -``` - -in your repo and we'll generate a new one for you. - -## Deployment Errors -### Invalid apiVersion for K8s - -`error: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1" error: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1" exit status 1` - -You might see this when attempting to sync crds or run helm commands in a run of `plural deploy`. It's due to legacy versions of the aws cli generating deprecated kubeconfigs, if you upgrade your cli and rerun `plural deploy` it should be able to proceed successfully. - - -### Failed deploy model - -`Failed deploy model due to Internal error occurred: failed calling webhook "mtargetgroupbinding.elbv2.k8s.aws": Post "https://aws-load-balancer-webhook-service.bootstrap.svc:443/mutate-elbv2-k8s-aws-v1beta1-targetgroupbinding?timeout=10s"` - -The aws load balancer controller webhook is not fully reliable; if you see this error it's just a matter of recreating the webhook which you can do with: - -``` -kubectl delete validatingwebhookconfiguration aws-load-balancer-webhook -n bootstrap -plural bounce bootstrap -``` - -### Error acquiring state lock -`Terraform acquires a state lock to protect the state from being written by multiple users at the same time. Please resolve the issue above and try again.` - -If your deploy is interrupted, it's possible terraform state gets confused. To fix this, you'll need to: - -``` -cd /terraform -terraform force-unlock -``` - -You should be able to find the state lock id from the error message. Terraform stacks for each app are located in a standard terraform folder each time. - -### May not specify more than one provider type - -If you are running `plural deploy` and encounter the error below there may be stale state from the previous install. - -``` -Error: UPGRADE FAILED: cannot patch "letsencrypt-prod" with kind ClusterIssuer: admission webhook "webhook.cert-manager.io" denied the request: spec.acme.solvers[0].dns01.route53: Forbidden: may not specify more than one provider type -``` - -Run: - -``` -helm del bootstrap -n bootstrap -plural deploy -``` - -### Error reading route table association - -`error reading Route Table Association () : Empty result` - -This is a possible terraform race condition where the route tables think they're being read before they've been created, but in fact, they have already been created. As a temporary fix, just wait some time and rerun the deploy: - -``` -plural deploy -``` - -### Cloud resource limits exceeded - -It's possible plural will deploy resources that exceed your cloud limits. In general the most pessimistic of these are load balancer limits and VPC limits. Be sure you have headroom in both those dimensions. This can lead to terraform state corruption, in which case you'll need to do something along the lines of for all resources missing from your terraform state: - -``` -cd application-name/terraform -terraform import path_to_terraform_resource resource_id -``` - -If you devise a better way to recover crashed terraform state, please give us a shout in our [Discord](https://discord.gg/pluralsh). We'd love to automate this better. - - diff --git a/src/NavData.tsx b/src/NavData.tsx index 14bb003b..57b9f455 100644 --- a/src/NavData.tsx +++ b/src/NavData.tsx @@ -209,30 +209,16 @@ const rootNavData: NavMenu = deepFreeze([ title: 'Debugging', sections: [ { - href: '/debugging', - title: 'Debugging', - sections: [ - { - href: '/debugging/health-checks', - title: ' Health Checks', - }, - { - href: '/debugging/proxies', - title: 'Proxies', - }, - { - href: '/debugging/logs', - title: 'Logs', - }, - ], + title: 'Debugging Applications', + href: '/debugging/debugging-applications', }, { - title: 'Troubleshooting', - href: '/reference/troubleshooting', + href: '/debugging/debugging-tools', + title: 'Tools for Debugging', }, { - title: 'Application Issues', - href: '/debugging/application-issues', + title: 'Common Errors', + href: '/debugging/common-errors', }, ], }, From 2287f277c93cd2bc86f69b1987a9c377f99d6aa0 Mon Sep 17 00:00:00 2001 From: Sam Bolian Date: Wed, 21 Dec 2022 15:58:55 -0600 Subject: [PATCH 2/5] added new debugging pages --- pages/debugging/debugging-applications.md | 45 +++++++++++++ pages/debugging/debugging-tools.md | 81 +++++++++++++++++++++++ 2 files changed, 126 insertions(+) create mode 100644 pages/debugging/debugging-applications.md create mode 100644 pages/debugging/debugging-tools.md diff --git a/pages/debugging/debugging-applications.md b/pages/debugging/debugging-applications.md new file mode 100644 index 00000000..6e197267 --- /dev/null +++ b/pages/debugging/debugging-applications.md @@ -0,0 +1,45 @@ +--- +title: Debugging Applications +description: Figuring out what is going wrong with your deployed applications. +--- + +Issues with applications are often due to issues with the underlying Pods. You can use `kubectl` commands to get, find logs for, and delete pods as necessary. Note that Plural automatically configures `kubectl` for use with your Plural cluster. + +## With Plural CLI + +To find Pods related to an application with the CLI, you can run: + +``` +kubectl get pods -n +``` + +If you see failed Pods, you can get the logs for the Pods by running: + +``` +kubectl logs -n +``` + +We also curate a list of helpful logging shortcuts for each application, which you can use the `plural logs` subcommand for, eg with + +``` +plural logs list # shows all log tails available +plural logs tail # tails that specific log +``` + +To delete problematic Pods, run: + +``` +kubectl delete pod +``` + +In most cases, kubernetes will restart the pod for you. You can always also run `plural bounce` to regenerate your deleted Pods. + +## With Plural Console + +If you have the Plural Console installed, you can debug your Kubernetes Pods with the following steps: + +1. Navigate to the `Components` tab and click on the failing component + +2. Scroll to the `Pods` section at the bottom of the screen and hit the red trash can button located on the right of the screen to delete the failing Pod. + +3. Head back to the `Builds` tab in the sidebar and click the `Bounce` button in the top right to redeploy your deleted Pods. \ No newline at end of file diff --git a/pages/debugging/debugging-tools.md b/pages/debugging/debugging-tools.md new file mode 100644 index 00000000..ded40a46 --- /dev/null +++ b/pages/debugging/debugging-tools.md @@ -0,0 +1,81 @@ +--- +title: Tools for Debugging +description: Additional resources to help you monitor and debug applications. +--- +## Health Checks + +Every application has a general application-level health check, which aggregates the statuses of all core kubernetes components and generates a digestable summary for human consumption. These are viewable in the application switcher in the admin console, or by running: + +``` +plural watch +``` + +This will generate output like: + +``` +Application: console (0.5.35) READY +plural admin console + +Components Ready: 15/15 + +Ready Components: +- service/console :: Ready +- service/console-headless :: Ready +- service/console-master :: Ready +- service/console-replica :: Ready +- service/plural-console :: Ready +- service/plural-console-repl :: Ready +... +``` + +If a component is not ready, it will also generate hints to kubectl commands which might help debug further. + +## Proxies + +Plural also helps with discovering useful proxy commands for inspecting core resources. This is done via the `Proxy` crd. A decent example is the proxy to the admin consoles underlying Postgres database: + +```yaml +apiVersion: platform.plural.sh/v1alpha1 +kind: Proxy +metadata: + name: db + labels: +spec: + type: db + target: statefulset/console-postgresql + credentials: + secret: console-postgresql + key: postgresql-password + user: console + dbConfig: + name: console + engine: postgres + port: 5432 +``` + +The interface is quite powerful, and supports things like fetching db credentials and initiating a sql shell, or spawning a web ui via `kubectl port-forward` and printing the credentials to stdout to allow a user easily log in. + +## Logs + +Application-aware log tails can be baked into any Plural application to prevent a user from having to dig through the Kubernetes API to find the specific Pod they should tail. They can be discovered and watched using: + +```shell {% showHeader=false %} +plural logs list +plural logs tail +``` + +Now the relevant logs will be streamed to stdout. + +The specification for these is baked into the `LogTail` crd, for example: + +```yaml +apiVersion: platform.plural.sh/v1alpha1 +kind: LogTail +metadata: + name: aws-load-balancer + labels: +spec: + limit: 50 + target: deployment/bootstrap-aws-load-balancer-controller + follow: true +``` \ No newline at end of file From 702c40ed199309ab28feb977b1585eae1d83f8f6 Mon Sep 17 00:00:00 2001 From: Klink <85062+dogmar@users.noreply.github.com> Date: Thu, 15 Jun 2023 10:16:49 -0700 Subject: [PATCH 3/5] Update pages/debugging/debugging-tools.md Co-authored-by: Abhi Vaidyanatha --- pages/debugging/debugging-tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/debugging/debugging-tools.md b/pages/debugging/debugging-tools.md index ded40a46..3919431c 100644 --- a/pages/debugging/debugging-tools.md +++ b/pages/debugging/debugging-tools.md @@ -4,7 +4,7 @@ description: Additional resources to help you monitor and debug applications. --- ## Health Checks -Every application has a general application-level health check, which aggregates the statuses of all core kubernetes components and generates a digestable summary for human consumption. These are viewable in the application switcher in the admin console, or by running: +Every application has a general application-level health check, which aggregates the statuses of all core Kubernetes components and generates a digestible summary for human viewing. These are viewable in the application switcher in the admin console, or by running: ``` plural watch From df3fdd7681cff4d66db7a90ce73fcb5ef94476a5 Mon Sep 17 00:00:00 2001 From: Klink <85062+dogmar@users.noreply.github.com> Date: Thu, 15 Jun 2023 10:17:03 -0700 Subject: [PATCH 4/5] Update pages/debugging/debugging-tools.md Co-authored-by: Abhi Vaidyanatha --- pages/debugging/debugging-tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/debugging/debugging-tools.md b/pages/debugging/debugging-tools.md index 3919431c..57fedaa4 100644 --- a/pages/debugging/debugging-tools.md +++ b/pages/debugging/debugging-tools.md @@ -28,7 +28,7 @@ Ready Components: ... ``` -If a component is not ready, it will also generate hints to kubectl commands which might help debug further. +If a component is not ready, it will also generate hints to kubectl commands which might help debug them further. ## Proxies From 1b01ea5e6b1c4c2165fa2d7b5f8816bff3a52694 Mon Sep 17 00:00:00 2001 From: Klink <85062+dogmar@users.noreply.github.com> Date: Thu, 15 Jun 2023 10:18:13 -0700 Subject: [PATCH 5/5] Update pages/debugging/debugging-tools.md Co-authored-by: Abhi Vaidyanatha --- pages/debugging/debugging-tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/debugging/debugging-tools.md b/pages/debugging/debugging-tools.md index 57fedaa4..58458145 100644 --- a/pages/debugging/debugging-tools.md +++ b/pages/debugging/debugging-tools.md @@ -53,7 +53,7 @@ spec: port: 5432 ``` -The interface is quite powerful, and supports things like fetching db credentials and initiating a sql shell, or spawning a web ui via `kubectl port-forward` and printing the credentials to stdout to allow a user easily log in. +The interface is quite powerful, and supports things like fetching database credentials, initiating a SQL shell, spawning a web UI via `kubectl port-forward`, or printing the credentials to stdout to allow a user to easily log in. ## Logs