-
Notifications
You must be signed in to change notification settings - Fork 45
Docs/hashicorp key management.md #114
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
fa19ad7
docs: Fixed example to point to the correct secret.
mariadb-stefangenov 7f38a1c
docs: New Section For Hashicorp Plugin
mariadb-stefangenov 744d9cc
docs: Modified docker images to point to the hashicorp ones
mariadb-stefangenov f3a18b4
docs: Add details for tls
mariadb-stefangenov 139c870
docs: finalized docs for hashicorp-key-management, also updated other…
mariadb-stefangenov a0d818d
Update tools/mariadb-enterprise-operator/plugins/hashicorp-key-manage…
mariadb-stefangenov 4cb6f04
Update tools/mariadb-enterprise-operator/plugins/hashicorp-key-manage…
mariadb-stefangenov bce98c7
Update tools/mariadb-enterprise-operator/plugins/hashicorp-key-manage…
mariadb-stefangenov c05d622
Update tools/mariadb-enterprise-operator/plugins/hashicorp-key-manage…
mariadb-stefangenov 6e0eb0a
Update tools/mariadb-enterprise-operator/plugins/hashicorp-key-manage…
mariadb-stefangenov 697077b
Update tools/mariadb-enterprise-operator/plugins/hashicorp-key-manage…
mariadb-stefangenov 1ce8e7b
Update tools/mariadb-enterprise-operator/plugins/hashicorp-key-manage…
mariadb-stefangenov 4886709
Update tools/mariadb-enterprise-operator/plugins/hashicorp-key-manage…
mariadb-stefangenov 3f7fdb7
Merge branch 'main' into docs/hashicorp-key-management.md
mariadb-stefangenov f32aa14
docs: removed confusing example
mariadb-stefangenov File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
300 changes: 300 additions & 0 deletions
300
tools/mariadb-enterprise-operator/plugins/hashicorp-key-management.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,300 @@ | ||
| # Hashicorp Key Management Plugin | ||
|
|
||
| The [Hashicorp Key Management Pugin](../../../server/security/securing-mariadb/encryption/data-at-rest-encryption/key-management-and-encryption-plugins/hashicorp-key-management-plugin.md) is used to implement encryption using keys stored in the Hashicorp Vault KMS. | ||
|
|
||
| {% hint style="info" %} | ||
| For more information about configuring the plugin as well as different capabilities, please check the [documentation](../../../server/security/securing-mariadb/encryption/data-at-rest-encryption/key-management-and-encryption-plugins/hashicorp-key-management-plugin.md). This guide will cover a minimal example for configuring the plugin with the operator. | ||
| {% endhint %} | ||
|
|
||
| ## Configuring TDE in MariaDB Using Hashicorp Key Management Plugin | ||
|
|
||
| Transparent Data Encryption (TDE) can be configured in MariaDB leveraging the Hashicorp Key Management Plugin. | ||
|
|
||
| ### Requirements | ||
|
|
||
| - Running and accessible Vault KMS setup with a valid SSL certificate. | ||
| - Vault is unsealed and you've logged in to it with `vault login $AUTH_TOKEN`, where $AUTH_TOKEN is an authentication token given to you by an administrator | ||
| - `openssl` for generating secrets | ||
|
|
||
| ### Steps | ||
|
|
||
| 1. **Creating A New Key-Value Store In Vault.** | ||
| Create a new key-value store and take note of the `path`. In our example we will use `mariadb`. | ||
| ```sh | ||
| vault secrets enable -path /mariadb -version=2 kv | ||
| ``` | ||
|
|
||
| 2. **Adding necessary secrets.** | ||
| We will put 2 secrets with ids `1` and `2`. `2` will be used for temporary files, while `1` will be used for everything else. It is not | ||
| neccessary to create 2 of them and in that case, temporary files will use `1`. | ||
|
|
||
| Note: Here you should use the `path` we chose in the previous step. | ||
| ```sh | ||
| vault kv put /mariadb/1 data="$(openssl rand -hex 32)" | ||
| vault kv put /mariadb/2 data="$(openssl rand -hex 32)" | ||
| ``` | ||
|
|
||
| 3. **(Optional) Create An Authentication Token With Policy.** | ||
| This step can be skipped if you want to use your own token. Consult with a Vault administrator regarding this. | ||
| Policies are Vault's way to restrict access to what you are allowed to do. The following is a policy that should be used by the token following the least permission principle. | ||
| ```sh | ||
| cat <<'EOF' | vault policy write -non-interactive mariadb - | ||
| # Allow access to MariaDB secrets | ||
| path "mariadb/*" { | ||
| capabilities = ["create", "read", "update", "delete", "list"] | ||
| } | ||
|
|
||
| # Allow reading the mount configuration | ||
| path "sys/mounts/mariadb/tune" { | ||
| capabilities = ["read"] | ||
| } | ||
| EOF | ||
| ``` | ||
| After which, we can create a new token with the given policy. | ||
| ```sh | ||
| vault token create -policy mariadb | ||
| ``` | ||
| You will see output similar to: | ||
| ``` | ||
| Key Value | ||
| --- ----- | ||
| token EXAMPLE_TOKEN | ||
| token_accessor utFtmh98YAAJyYdxEVN3SFQA | ||
| token_duration 768h | ||
| token_renewable true | ||
| token_policies ["default" "mariadb"] | ||
| identity_policies [] | ||
| policies ["default" "mariadb"] | ||
| ``` | ||
| Your new token is: `EXAMPLE_TOKEN`. | ||
|
|
||
| 4. **Create A `Secret` For the vault token.** | ||
| Now that you've either created a new token, or are using an existing one, we need to create a secret with it. | ||
| ```sh | ||
| export TOKEN="EXAMPLE_TOKEN" | ||
| kubeclt create secret generic mariadb-vault-token --from-literal=token="$TOKEN" | ||
| ``` | ||
|
|
||
| 5. **Create a Secret for the Certificate Authority (CA) used to issue the Vault certificate.** | ||
| For further information, consult [the docs](../../../server/security/securing-mariadb/encryption/data-at-rest-encryption/key-management-and-encryption-plugins/hashicorp-key-management-plugin.md#hashicorp-key-management-vault-ca) | ||
| If you have the certificate locally in a file called `ca.crt` you can run: | ||
|
|
||
| ```sh | ||
| kubectl create secret generic vault-tls --from-file=./ca.crt | ||
| ``` | ||
|
|
||
| 5. **Create A MariaDB Custom Resource.** | ||
| The final step is creating a new MariaDB instance. | ||
|
|
||
| **mariadb-vault.yaml** | ||
| ```yaml | ||
| --- | ||
| apiVersion: v1 | ||
| kind: Secret | ||
| metadata: | ||
| name: mariadb # Used to hold the mariadb and root user passwords | ||
| labels: | ||
| enterprise.mariadb.com/watch: "" | ||
| stringData: | ||
| password: MariaDB11! | ||
| root-password: MariaDB11! | ||
| --- | ||
| apiVersion: enterprise.mariadb.com/v1alpha1 | ||
| kind: MariaDB | ||
| metadata: | ||
| name: mariadb | ||
| spec: | ||
| image: docker.mariadb.com/enterprise-server:11.4.7-4.3 | ||
| rootPasswordSecretKeyRef: | ||
| name: mariadb | ||
| key: password | ||
|
|
||
| username: mariadb | ||
| passwordSecretKeyRef: | ||
| name: mariadb-password | ||
| key: password | ||
| generate: true | ||
| database: mariadb | ||
|
|
||
| port: 3306 | ||
|
|
||
| storage: | ||
| size: 1Gi | ||
| # storageClassName: csi-hostpath-sc | ||
|
|
||
| myCnf: | | ||
| [mariadb] | ||
| bind-address=* | ||
| default_storage_engine=InnoDB | ||
| binlog_format=row | ||
| innodb_autoinc_lock_mode=2 | ||
| innodb_buffer_pool_size=800M | ||
| max_allowed_packet=256M | ||
|
|
||
| plugin_load_add = hashicorp_key_management | ||
| hashicorp-key-management-vault-url=https://vault-0.vault-internal.default.svc.cluster.local:8200/v1/mariadb | ||
| hashicorp-key-management-caching-enabled=ON | ||
| hashicorp-key-management-vault-ca=/etc/vault/certs/ca.crt | ||
|
|
||
| innodb_encrypt_tables = FORCE | ||
| innodb_encrypt_log = ON | ||
| innodb_encrypt_temporary_tables = ON | ||
| encrypt_tmp_disk_tables = ON | ||
| encrypt_tmp_files = ON | ||
| encrypt_binlog = ON | ||
| aria_encrypt_tables = ON | ||
|
|
||
| innodb_encryption_threads = 4 | ||
| innodb_encryption_rotation_iops = 2000 | ||
|
|
||
| env: | ||
| - name: VAULT_TOKEN # This is where our token is defined! | ||
mariadb-mmontes marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| valueFrom: | ||
| secretKeyRef: | ||
| name: mariadb-vault-token | ||
| key: token | ||
|
|
||
| resources: | ||
| requests: | ||
| cpu: 100m | ||
| memory: 128Mi | ||
| limits: | ||
| memory: 1Gi | ||
|
|
||
| metrics: | ||
| enabled: true | ||
|
|
||
| volumes: | ||
| - name: vault-certificates | ||
| secret: | ||
| secretName: vault-tls | ||
| defaultMode: 0600 | ||
| volumeMounts: | ||
| - name: vault-certificates | ||
| mountPath: /etc/vault/certs/ | ||
| ``` | ||
| `kubectl apply -f mariadb-vault.yaml` | ||
|
|
||
| 6. **Verify Encryption Works.** | ||
| ```sh | ||
| kubectl run mariadb-connect --rm -it --image=mariadb:11.4 -- bash -c "mariadb -u root -p'MariaDB11!' --ssl=false -h mariadb" | ||
| ``` | ||
|
|
||
| You should see something along the lines of: | ||
|
|
||
| ``` | ||
| If you don't see a command prompt, try pressing enter. | ||
| Welcome to the MariaDB monitor. Commands end with ; or \g. | ||
| Your MariaDB connection id is 95 | ||
| Server version: 11.4.7-4-MariaDB-enterprise MariaDB Enterprise Server | ||
|
|
||
| Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. | ||
|
|
||
| Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. | ||
|
|
||
| MariaDB [(none)]> | ||
| ``` | ||
|
|
||
| At this point, you can check the encryption status: | ||
| ```sql | ||
| SELECT * from information_schema.INNODB_TABLESPACES_ENCRYPTION; | ||
| ``` | ||
|
|
||
| If you create a new database and then table, the above query should return additional information about them. Something like: | ||
| ```sql | ||
| MariaDB [my_db]> SELECT * from information_schema.INNODB_TABLESPACES_ENCRYPTION; | ||
| +-----------------+-------------------+-----------------+---------------------+----------------+----------------------+ | ||
| | NAME | ENCRYPTION_SCHEME | MIN_KEY_VERSION | CURRENT_KEY_VERSION | CURRENT_KEY_ID | ROTATING_OR_FLUSHING | | ||
| +-----------------+-------------------+-----------------+---------------------+----------------+----------------------+ | ||
| | innodb_system | 1 | 1 | 1 | 1 | 0 | | ||
| | innodb_undo001 | 1 | 1 | 1 | 1 | 0 | | ||
| | innodb_undo002 | 1 | 1 | 1 | 1 | 0 | | ||
| | innodb_undo003 | 1 | 1 | 1 | 1 | 0 | | ||
| | mysql/innodb_ta | 1 | 1 | 1 | 1 | 0 | | ||
| | mysql/innodb_in | 1 | 1 | 1 | 1 | 0 | | ||
| | mysql/gtid_slav | 1 | 1 | 1 | 1 | 0 | | ||
| | mysql/transacti | 1 | 1 | 1 | 1 | 0 | | ||
| | my_db/people | 1 | 1 | 1 | 1 | 0 | | ||
| +-----------------+-------------------+-----------------+---------------------+----------------+----------------------+ | ||
| ``` | ||
| Note: The above query is truncated. In reality, you will see a few more columns. | ||
|
|
||
| ## Day-2 Operations | ||
|
|
||
| ### Rotating Secrets | ||
|
|
||
| 1. **Put A New Secret In Vault.** | ||
| After logging in to vault, you can run again: | ||
| ```sh | ||
| vault kv put /mariadb/1 data="$(openssl rand -hex 32)" | ||
| vault kv put /mariadb/2 data="$(openssl rand -hex 32)" | ||
| ``` | ||
| This will start re-encrypting data. | ||
|
|
||
| 2. **Monitor Re-Encryption.** | ||
| ```sh | ||
| kubectl run mariadb-connect --rm -it --image=mariadb:11.4 -- bash -c "mariadb -u root -p'MariaDB11!' --ssl=false -h mariadb" | ||
| ``` | ||
| If you check the encrpytion status again: | ||
| ```sql | ||
| SELECT * from information_schema.INNODB_TABLESPACES_ENCRYPTION; | ||
mariadb-mmontes marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| ``` | ||
| You should see `CURRENT_KEY_VERSION` column start getting updated to point to the new key version. | ||
mariadb-stefangenov marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| ```sql | ||
| MariaDB [my_db]> SELECT * from information_schema.INNODB_TABLESPACES_ENCRYPTION; | ||
| +-----------------+-------------------+-----------------+---------------------+----------------+----------------------+ | ||
| | NAME | ENCRYPTION_SCHEME | MIN_KEY_VERSION | CURRENT_KEY_VERSION | CURRENT_KEY_ID | ROTATING_OR_FLUSHING | | ||
| +-----------------+-------------------+-----------------+---------------------+----------------+----------------------+ | ||
| | innodb_system | 1 | 1 | 2 | 1 | 0 | | ||
| | innodb_undo001 | 1 | 1 | 2 | 1 | 0 | | ||
| | innodb_undo002 | 1 | 1 | 2 | 1 | 0 | | ||
| | innodb_undo003 | 1 | 1 | 2 | 1 | 0 | | ||
| | mysql/innodb_ta | 1 | 1 | 2 | 1 | 0 | | ||
| | mysql/innodb_in | 1 | 1 | 2 | 1 | 0 | | ||
| | mysql/gtid_slav | 1 | 1 | 2 | 1 | 0 | | ||
| | mysql/transacti | 1 | 1 | 2 | 1 | 0 | | ||
| | my_db/people | 1 | 1 | 2 | 1 | 0 | | ||
| +-----------------+-------------------+-----------------+---------------------+----------------+----------------------+ | ||
| ``` | ||
|
|
||
| ### Rotating Token | ||
| Make sure when rotating the token, to do so in advance of the token expiring. | ||
|
|
||
| 1. **Acquire a new token and update the secret.** | ||
|
|
||
| ```sh | ||
| export TOKEN="EXAMPLE_TOKEN" | ||
| kubeclt create secret generic mariadb-vault-token --from-literal=token="$TOKEN" | ||
| ``` | ||
|
|
||
| 2. **Restart The MariaDB StatefulSet.** | ||
| MariaDB will continue using the old token until the StatefulSet is restarted. If the name of the `MariaDB` CR is `mariadb` (as per the example above), then: | ||
| ```sh | ||
| kubectl rollout restart statefulset mariadb | ||
| ``` | ||
|
|
||
| ## Known Issues/Limitations | ||
|
|
||
| ### **Vault Not Being Accessible Will Result In MariaDB Not Working** | ||
|
|
||
| As MariaDB uses Vault to fetch it's decryption key, in case that Vault becomes unavailable, it will result in MariaDB not being able to fetch the decryption key and hence stop working. While the Hashicorp plugin has a configurable cache, that should be set and will result in MariaDB still working for a few seconds to minutes, depending on configuration, the cache is not reliable as it's ephemeral and short lived. | ||
|
|
||
| ### **Deleting The Decryption Key Will Make Your Data Inaccessible.** | ||
|
|
||
| It is recommended to back up the decryption key so accidental deletions will not result in issues. | ||
|
|
||
| ### **Decryption Key Must Be Hexadecimal** | ||
|
|
||
| Use the following to generate correct decryption keys. | ||
| ```sh | ||
| openssl rand -hex 32 | ||
| ``` | ||
|
|
||
| ### **Rotating The Decryption Key Before A Previous Re-Encryption Has Finished, Will Result In Data Corruption.** | ||
| To check the re-encryption progress, you can run: | ||
|
|
||
| ```sql | ||
| SELECT * from information_schema.INNODB_TABLESPACES_ENCRYPTION; | ||
| ``` | ||
|
|
||
| Look for the `CURRENT_KEY_VERSION` and make sure they are in sync with the latest version you have in Vault. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.