© Carbon60 Operating Co LTD
© Carbon60 Operating Co LTD
Etcd is a distributed, consistent and highly-available key value store used as the Kubernetes backing store for all cluster data, making it a core component of every K8s deployment.
Due to its central role etcd may contain sensitive information related to access of the deployed services and their associated components, such as database credentials, CA keys, LDAP logins credentials it is a premium target for malicious attacks.
Historically, in traditional, non-containerised environments, this data was NOT stored in such a centralised manner as credentials were usually under an ownership of a specific team that was responsible for maintaining a certain component of the stack: the DB access credentials, for example, were known only to the DBA team, CA keys have been in the hands of few selected System Administrators etc.
With K8s, the required approach is notably different as credentials are now kept within a single central place (etcd), which, if not properly hardened, can lead to serious security breaches as the attacker may now create fake certificates, access databases and applications.
Managing and hardening your secrets becomes even more critical with tools such as Helm and Tiller; these tools allow you to install (or redeploy) an entire K8s based datacenter within minutes and they constantly interact with etcd.
The Center for Internet Security (CIS) came up with this publicly available document providing guidance on how to properly harden and secure your Kubernetes cluster.
The only single recommendation CIS provides regarding hardening etcd is using TLS:
ETCD is a highly available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. Its access should be restricted to specifically designated clients and peers only. Authentication to ETCD is based on whether the certificate presented was issued by a trusted certificate authority. There is no checking of certificate attributes such as common name or subject alternative name. As such, if any attackers were able to gain access to any certificate issued by the trusted certificate authority, they would be able to gain full access to the ETCD database. Use a different certificate authority for ETCD from the one used for Kubernetes.
However, using TLS on its own is not sufficient as a solution. Every certificate created and signed with the same CA has the potential to access every service inside the cluster. The problem is further exacerbated if a single CA is used for all k8s clusters. Even when each kubernetes cluster has a dedicated CA, new client keys can be easily created but as easily revoked. Once again, any new keys created automatically have access to every service in the targeted k8s cluster.
Because of the severity of the security risks associated with etcd, we will look into 2 additional methods that can be implemented to further secure your etcd data:
Encrypting secrets (and/or other resources) in etcdUsing certificates to stop clients from accessing the etcd server
To follow the steps illustrated in the following sections, it is necessary to start up a Kubernetes cluster. This can be done using any of the methods immediately below.
Vanilla K8s:
install script for latest kubernetes 1.10. This is the first version that installs etcd with tls install script for older kubernetes versions, when etcd was not installed with tls by default
This also works on openshift platform. You can see the install script here. All relative commands for openshift are in this script.
The install scripts have been tested on AWS Centos ami. It should work for you too if you use the same image.
Starting with K8s 1.7 (and etcd v3) you can encrypt resources inside etcd using several different algorithms. At the very least, you should encrypt all your secrets. It is especially true if you are using Helm as a lot of Helm charts require LDAP or DB credentials to be directly made available in the ConfigMaps.
The encryption follows a very simple rule:
encrypt using the first provider defineddecrypt after locating a functional provider at checking each provider in the order the providers are defined
To implement the full workflow, it is necessary to add the experimental-encryption-provider-config flag to the apiserver
Define the EncryptionConfig config file (place the content in /etc/kubernetes/pki/encryption-config.yaml)
kind: EncryptionConfig apiVersion: v1 resources: - resources: - secrets providers: - identity: {}
Within the file, the resources.resources field is an array of Kubernetes resource names that should be encrypted. The providers array is an ordered list of the possible encryption providers.
Enable experimental-encryption-provider-config in the kube-apiserver. Edit /etc/kubernetes/manifests/kube-apiserver.yaml and add:
spec: containers: - command: - kube-apiserver - --experimental-encryption-provider-config=/etc/kubernetes/pki/encryption-config.yaml
Restart the apiserver. Because the API server is being run as a static pod, kubelet will restart it when the configuration change is detected. Otherwise, you will need to restart the service yourself.
We also install the etcd package in order to print the data from inside the etcd server:
# yum install etcd -y Resolving Dependencies --> Running transaction check ---> Package etcd.x86_64 0:3.2.18-1.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ================================================================================================================================================================================================================================== Package Arch Version Repository Size ================================================================================================================================================================================================================================== Installing: etcd x86_64 3.2.18-1.el7 optymyze_external_rpms 9.3 M Transaction Summary ================================================================================================================================================================================================================================== Install 1 Package Total download size: 9.3 M Installed size: 42 M Downloading packages: etcd-3.2.18-1.el7.x86_64.rpm | 9.3 MB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : etcd-3.2.18-1.el7.x86_64 1/1 Uploading Package Profile Verifying : etcd-3.2.18-1.el7.x86_64 1/1 Installed: etcd.x86_64 0:3.2.18-1.el7 Complete! Uploading Enabled Repositories Report Loaded plugins: fastestmirror, priorities, product-id
Also, to make the commands shorter, set an alias for etcdctl command with TLS parameters. Here we will use the certificates paths created by kubeadm-1.10. You should update them for your specific cluster if needed (check by runninggrep — ‘–etcd’ /etc/kubernetes/manifests/kube-apiserver.yaml).
Meaning of variables:
DIR — path where the k8s certificates are createdSSL_OPS — etcdctl parameters to enable TLS connectivitySECRETS_PATH — path in etcd where kubernetes keeps secrets
DIR=/etc/kubernetes/pki/ SSL_OPTS="--cacert=${DIR}/etcd/ca.crt --cert=${DIR}/apiserver-etcd-client.crt --key=${DIR}/apiserver-etcd-client.key --endpoints=localhost:2379" SECRETS_PATH=/registry/secrets
Test that we can list stuff in etcd:
# ETCDCTL_API=3 etcdctl $SSL_OPTS get --keys-only=true --prefix $SECRETS_PATH /registry/secrets/default/default-token-rhwwn /registry/secrets/kube-public/default-token-9qfc8 /registry/secrets/kube-system/attachdetach-controller-token-clvsn .............
To demonstrate the difference of our solution, we begin with no encryption. This provider doesn’t do any encryption. It can be used in case you want to decrypt everything or just to test.
Let’s create a secret and read it directly from etcd. You should be able to clearly see the key name and key value:
# kubectl create secret generic secret1 --from-literal=XX_mykey_XX=ZZ_mydata_ZZ secret "secret1" created # kubectl get secret secret1 -o yaml apiVersion: v1 data: XX_mykey_XX: WlpfbXlkYXRhX1pa kind: Secret metadata: creationTimestamp: 2018-06-18T13:11:54Z name: secret1 namespace: default resourceVersion: "20410585" selfLink: /api/v1/namespaces/default/secrets/secret1 uid: 2bb3b7df-72f9-11e8-ad5f-005056b1028d type: Opaque # ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret1 -w fields | grep Value "Value" : "k8sx00nfnx02v1x12x06Secretx12snLnasecret1x12x00x1aadefault"x00*$2bb3b7df-72f9-11e8-ad5f-005056b1028d2x008x00Bbbx9aߞxd9x05x10x00zx00x12x1bnvXX_mykey_XXx12fZZ_mydata_ZZx1ax06Opaquex1ax00"x00"
Let’s add an encryption algorithm to see what happens. We choose aescbcbecause this is the recommended choice for encryption at rest.
Update encryption-config.yaml:
kind: EncryptionConfig apiVersion: v1 resources: - resources: - secrets providers: - aescbc: keys: - name: key1 secret: c2VjcmV0IGlzIHNlY3VyZQ== - name: key2 secret: dGhpcyBpcyBwYXNzd29yZA== - identity: {}
Since kubelet only monitors pods defined in /etc/kubernetes/manifests, this change will not be caught, so we need to restart the apiserver manually:
docker stop $(docker ps | grep k8s_kube-apiserver | gawk '{print $1}')
Test:
# kubectl create secret generic secret2 --from-literal=XX_mykey_XX=ZZ_mydata_ZZ secret "secret2" created # kubectl get secret secret2 -o yaml apiVersion: v1 data: XX_mykey_XX: WlpfbXlkYXRhX1pa kind: Secret metadata: creationTimestamp: 2018-06-18T14:23:06Z name: secret2 namespace: default resourceVersion: "20418382" selfLink: /api/v1/namespaces/default/secrets/secret2 uid: 1e4f5d2f-7303-11e8-8c2c-005056b1028d type: Opaque # ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret2 -w fields | grep Value "Value" : "k8s:enc:aescbc:v1:key1:7^İxe9xc8x1exa7̔=D+x9e%x1axf4x10o@xecxc14&<Zxd1xdexfaxca-'#xa2Kx1cxffx101ax86xb0xd7.xa9x19x04x93mxa1xeexacDex95/xd8xe7xaehp~xc9x0exe9x8f}x9ax8axb0fxf9xebxb7u007f@x87xa0xa6x98xe78xd0+xd45"Sx17x8cx84xa6ㅽbxdaxe6xfcxa1xd9[[~x82xfbKSx82xf0>oxc1 x8b&{xa1rx14Unx03xf7x1f=xe5x1b xa7txed[x8axecxb8xf1xe4xe2xc1x81xb00=cbl·ɬx12`xf2|x1btxe4#xcd"
The new secret was encrypted now with “k8s:enc:aescbc:v1:key1”.
Let’s encrypt all the other secrets as well:
# kubectl get secrets --all-namespaces -o json | kubectl replace -f - secret "default-token-rhwwn" replaced secret "secret1" replaced secret "secret2" replaced secret "default-token-9qfc8" replaced secret "attachdetach-controller-token-clvsn" replaced secret "bootstrap-signer-token-xgnfg" replaced ....
Check that the old secret is now encrypted:
# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret1 -w fields | grep Value "Value" : "k8s:enc:aescbc:v1:key1:xdanW0~x83xe4x80Ճ$Jx1exa2x02zxc9vxd1xd0$)xb2Kx9fxc2xffxcdJ5xfa"x13xc4fx86xc0{PxceWx9exd1z;b$x97xe8xb4lxd0xfaxd8 xe2Vcx8cxa2xcdxe5xb0x04(lx18x13xbfxe2xb7|xf1mxef)xfdx97xcbk-"xbax819xcf,_xf6fxPxf2x13x94x9bxcaxf4xde{dxcbxceqx84qxaexaax06x14xb7qx1d|Lx8eSx8cxc9$x8ex80Dxf0xdaxe2sixb6,@xa2xf9xaexf2~xe3wx8e4fr{ex0f'xccxf6xe7xaddx83^xdbx03xf1jTx13>"
Each resource is encrypted with a specific key. If you change the value of that key, kubernetes will not be able to decode it anymore. You can test this by swapping the name of the keys and try to retrieve it:
# cat /etc/kubernetes/pki/encryption-config.yaml kind: EncryptionConfig apiVersion: v1 resources: - resources: - secrets providers: - aescbc: keys: - name: key2 secret: c2VjcmV0IGlzIHNlY3VyZQ== - name: key1 secret: dGhpcyBpcyBwYXNzd29yZA== - identity: {} # docker stop $(docker ps | grep k8s_kube-apiserver | gawk '{print $1}') 000e03b50c0f # kubectl get secret secret2 -o yaml Error from server (InternalError): Internal error occurred: invalid PKCS7 data (empty or not padded)
In this example we encrypt a secret with a new algorithm and check that different secrets are encrypted with different providers. After that encrypt everything with the new provider. This will change the encryption algorithm to all previous keys to the new one.
Change encryption-config.yaml, so that the first provider to be the secretbox provider:
kind: EncryptionConfig apiVersion: v1 resources: - resources: - secrets providers: - secretbox: keys: - name: key1 secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY= - aescbc: keys: - name: key1 secret: c2VjcmV0IGlzIHNlY3VyZQ== - name: key2 secret: dGhpcyBpcyBwYXNzd29yZA== - identity: {}
Restart the apiserver:
# docker stop $(docker ps | grep k8s_kube-apiserver | gawk '{print $1}')
Verify that secrets are encrypted correctly: old secret is using aescbc, new one will use secretbox.
# kubectl create secret generic secret3 --from-literal=XX_mykey_XX=ZZ_mydata_ZZ secret "secret3" created # ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret1 -w fields | grep Value "Value" : "k8s:enc:aescbc:v1:key1:xdanW0~x83xe4x80Ճ$Jx1exa2x02zxc9vxd1xd0$)xb2Kx9fxc2xffxcdJ5xfa"x13xc4fx86xc0{PxceWx9exd1z;b$x97xe8xb4lxd0xfaxd8 xe2Vcx8cxa2xcdxe5xb0x04(lx18x13xbfxe2xb7|xf1mxef)xfdx97xcbk-"xbax819xcf,_xf6fxPxf2x13x94x9bxcaxf4xde{dxcbxceqx84qxaexaax06x14xb7qx1d|Lx8eSx8cxc9$x8ex80Dxf0xdaxe2sixb6,@xa2xf9xaexf2~xe3wx8e4fr{ex0f'xccxf6xe7xaddx83^xdbx03xf1jTx13>" # ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret3 -w fields | grep Value "Value" : "k8s:enc:secretbox:v1:key1:xbaxf8,Q@xb9xb6q3$kx04xeeVx99|Z'xdeE<xa5xa9nx91uxb9]RYxcccxe3x13x8bu07b4Qx91x9cR2xccxc5xd9x0ex19?xcax1chxdex1d%xa3Nx85Hxb0xf6֢xe6xabx06xf6x960{xdbxd8^eQxb3x05x03x06)x05JHx16x18fpx9eu<txeax06x12xf1۹yu007fx15xe5x1dxefx8a2Gx85'x94nx1dx99x85ku3xa2~x12x04xe5x84~xaaGxd3nx98x95xa0xc8_1Bxcbx0fxb7;x80xe1xRx86ijfxefxd7SAx950MQfz~)x13xc5xf1xf8x91x14x9d_xbax82[=Mx81Ox1dFNjxc1x98xe4"
Migrate all secrets to the new provider:
# kubectl get secrets --all-namespaces -o json | kubectl replace -f -
Here we use the same provider for encryption, but we add a new key. Everything from this moment will be encrypted with the new key. Old values are encrypted with the previous key. At the end we migrate everything to be encrypted with the new key.
Add a new key to be the first for the secretbox provider (which is still the first provider)
kind: EncryptionConfig apiVersion: v1 resources: - resources: - secrets providers: - secretbox: keys: - name: key2 secret: sAkccgM28JdPNCX9FfTcloYet1zp4OEAtHyViT038zM= - name: key1 secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY= - aescbc: keys: - name: key1 secret: c2VjcmV0IGlzIHNlY3VyZQ== - name: key2 secret: dGhpcyBpcyBwYXNzd29yZA== - identity: {}
Restart and verify that the new values are encrypted with the new key:
# docker stop $(docker ps | grep k8s_kube-apiserver | gawk '{print $1}') 4bdac1937570 # kubectl create secret generic secret4 --from-literal=XX_mykey_XX=ZZ_mydata_ZZ secret "secret4" created # ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret4 -w fields | grep Value "Value" : "k8s:enc:secretbox:v1:key2:x92xeeyjx96xfc쵪-8x0exa7x9axb0x16xe2xb8Jf_x81xecxf65xa9x1axe5\xۛ%Ҝxbbaxxbfx00KzxabaDx1cx94x87xaervsPxf3qxf3xaeHxb8x95-xefr*[ylxf3/xc4x0fx00ax132fxe1x17xbfxffxb4;<xecxc2x01xa8xc8fxffxcdxf3ʦx83Px01xcdux16x16xfaxbax8fxe6xe5x05x96xf7k,xaaxeax0fx99x8fxb3xc7xe6xa4=x93x8axf3Sx17xc6Srxeexea㟷x00x945oxe8x8e:Wxacotxeaj,Px14xbexd0x13xf91Yxf0xf0x93fWxcczD3xb9xa0xb4x9exefx1aEx16xc8j_TXxae"
Etcd can use 2 methods to authorize users:
With username and passwordWith certificates if started with ”–client-cert-auth=true”. It will use the CN from the certificate as the username.
Unfortunately there are a couple of issues with this:
https://github.com/coreos/etcd/issues/9816: auth doesn’t work at all with the default etcd version. You need to update your etcd to version 3.2.18https://github.com/coreos/etcd/issues/9691: this affects you if you try to create the openshift user inside etcd. But for this one there is also an workaround
We use cfssl to create the certificates we need to connect to etcd. Lets download the binaries and put them in our path:
# curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O && /bin/mv ./cfssl_linux-amd64 /bin/cfssl && chmod +x /bin/cfssl # curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -O && /bin/mv ./cfssljson_linux-amd64 /bin/cfssljson && chmod +x /bin/cfssljson
In order to enable etcd ACL, first create the root user:
# ETCDCTL_API=3 etcdctl $SSL_OPTS user add root:secretpass User root created
Create some roles to test our certificates.
Users with this role should have read access in the entire cluster:
# ETCDCTL_API=3 etcdctl $SSL_OPTS role add readonly_all Role readonly_all created # ETCDCTL_API=3 etcdctl $SSL_OPTS role grant-permission readonly_all --prefix=true read / Role readonly_all updated
Allows users to read and write everywhere:
# ETCDCTL_API=3 etcdctl $SSL_OPTS role add readwrite_all Role readwrite_all created # ETCDCTL_API=3 etcdctl $SSL_OPTS role grant-permission readwrite_all --prefix=true readwrite / Role readwrite_all updated
User with this role should only be allowed to access part of etcd tree:
# ETCDCTL_API=3 etcdctl $SSL_OPTS role add readonly_secrets Role readonly_secrets created # ETCDCTL_API=3 etcdctl $SSL_OPTS role grant-permission readonly_secrets --prefix=true read $SECRETS_PATH Role readonly_secrets updated
Role that allows a user to read/write a specific key only:
# ETCDCTL_API=3 etcdctl $SSL_OPTS role add readwrite_secret4 Role readwrite_secret4 created # ETCDCTL_API=3 etcdctl $SSL_OPTS role grant-permission readwrite_secret4 readwrite $SECRETS_PATH/default/secret4 Role readwrite_secret4 updated
Create users and add assign specific roles to them. Generate random passwords because we don’t expect to use them:
# ETCDCTL_API=3 etcdctl $SSL_OPTS user add reader:$(head -c 32 /dev/urandom | base64) User reader created # ETCDCTL_API=3 etcdctl $SSL_OPTS user add viewsecrets:$(head -c 32 /dev/urandom | base64) User viewsecrets created # ETCDCTL_API=3 etcdctl $SSL_OPTS user add admin:$(head -c 32 /dev/urandom | base64) User admin created # ETCDCTL_API=3 etcdctl $SSL_OPTS user add usersecret4:$(head -c 32 /dev/urandom | base64) User usersecret4 created # ETCDCTL_API=3 etcdctl $SSL_OPTS user grant-role reader readonly_all Role readonly_all is granted to user reader # ETCDCTL_API=3 etcdctl $SSL_OPTS user grant-role viewsecrets readonly_secrets Role readonly_secrets is granted to user viewsecrets # ETCDCTL_API=3 etcdctl $SSL_OPTS user grant-role admin readwrite_all Role readwrite_all is granted to user admin # ETCDCTL_API=3 etcdctl $SSL_OPTS user grant-role usersecret4 readwrite_secret4 Role readwrite_secret4 is granted to user usersecret4
Since we are enabling user authorization, we need to have special permissions for the user used by the apiserver to connect to the etcd cluster: we will give it the root role. Kubernetes installation has 2 connections to the etcd server: apiserver and the livenessProbe.
openssl x509 -noout -text -in /etc/kubernetes/pki/apiserver-etcd-client.crt | grep "Subject:" openssl x509 -noout -text -in /etc/kubernetes/pki/etcd/healthcheck-client.crt | grep "Subject:"
Create an user with the name from the CN field of the certificate:
# ETCDCTL_API=3 etcdctl $SSL_OPTS user add kube-apiserver-etcd-client:$(head -c 32 /dev/urandom | base64) User kube-apiserver-etcd-client created # ETCDCTL_API=3 etcdctl $SSL_OPTS user add kube-etcd-healthcheck-client:$(head -c 32 /dev/urandom | base64) User kube-etcd-healthcheck-client created # ETCDCTL_API=3 etcdctl $SSL_OPTS user grant-role kube-apiserver-etcd-client root Role root is granted to user kube-apiserver-etcd-client # ETCDCTL_API=3 etcdctl $SSL_OPTS user grant-role kube-etcd-healthcheck-client root Role root is granted to user kube-etcd-healthcheck-client
Enable authentication:
# ETCDCTL_API=3 etcdctl $SSL_OPTS auth enable Authentication Enabled
From this moment, nothing can connect to the etcd cluster without proper certificates. Let’s create a certificate with a user that it’s not define in etcd and check that it doesn’t have access at all:
# function create_certificates { NAME=$1 cat <<EOF | cfssl gencert -config=ca-config.json -profile=client -ca $CA_PATH/ca.crt -ca-key $CA_PATH/ca.key - | cfssljson -bare $NAME {"CN": "$NAME","key": {"algo": "rsa","size": 2048}} EOF SSL_OPTS="--cacert=$CA_PATH/ca.crt --cert=$PWD/$NAME.pem --key=$PWD/$NAME-key.pem --endpoints=$HOSTNAME:2379" } # CA_PATH=/etc/kubernetes/pki/etcd # cfssl print-defaults config > ca-config.json # create_certificates tester 2018/06/18 18:52:54 [INFO] generate received request 2018/06/18 18:52:54 [INFO] received CSR 2018/06/18 18:52:54 [INFO] generating key: rsa-2048 2018/06/18 18:52:54 [INFO] encoded CSR 2018/06/18 18:52:54 [INFO] signed certificate with serial number 722235405009026418318053946143102861163105227800 2018/06/18 18:52:54 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for websites. For more information see the Baseline Requirements for the Issuance and Management of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org); specifically, section 10.2.3 ("Information Requirements"). # ETCDCTL_API=3 etcdctl $SSL_OPTS get / --keys-only --prefix=true Error: etcdserver: permission denied
Allow an admin user to access the cluster for 2 hours only:
# cfssl print-defaults config | sed s/8760/2/ > ca-config.json # create_certificates admin
Admin user can do list:
# ETCDCTL_API=3 etcdctl $SSL_OPTS get / --keys-only --prefix=true /registry/apiregistration.k8s.io/apiservices/v1. /registry/apiregistration.k8s.io/apiservices/v1.apps /registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io /registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io
Get a key:
# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret1 -w fields | grep Value "Value" : "k8s:enc:aescbc:v1:key1:xdanW0~x83xe4x80Ճ$Jx1exa2x02zxc9vxd1xd0$)xb2Kx9fxc2xffxcdJ5xfa"x13xc4fx86xc0{PxceWx9exd1z;b$x97xe8xb4lxd0xfaxd8 xe2Vcx8cxa2xcdxe5xb0x04(lx18x13xbfxe2xb7|xf1mxef)xfdx97xcbk-"xbax819xcf,_xf6fxPxf2x13x94x9bxcaxf4xde{dxcbxceqx84qxaexaax06x14xb7qx1d|Lx8eSx8cxc9$x8ex80Dxf0xdaxe2sixb6,@xa2xf9xaexf2~xe3wx8e4fr{ex0f'xccxf6xe7xaddx83^xdbx03xf1jTx13>"
Delete the key:
# ETCDCTL_API=3 etcdctl $SSL_OPTS del $SECRETS_PATH/default/secret1 1 # ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/secret1 -w fields | grep Value
Increase the date and try to list:
# date $(date +%m%d%H%M%Y.%S -d '+1 hour') # ETCDCTL_API=3 etcdctl $SSL_OPTS get --keys-only --prefix=true / /registry/apiregistration.k8s.io/apiservices/v1. /registry/apiregistration.k8s.io/apiservices/v1.apps /registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io /registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io /registry/apiregistration.k8s.io/apiservices/v1.autoscaling /registry/apiregistration.k8s.io/apiservices/v1.batch # date $(date +%m%d%H%M%Y.%S -d '+1 hour') # ETCDCTL_API=3 etcdctl $SSL_OPTS get --keys-only --prefix=true / Error: context deadline exceeded # date $(date +%m%d%H%M%Y.%S -d '-2 hour')
Check that user ‘reader’ has access everywhere and can’t delete anything:
# cfssl print-defaults config > ca-config.json # create_certificates reader # ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret2 -w fields | grep Value "Value" : "k8s:enc:aescbc:v1:key1:7^İxe9xc8x1exa7̔=D+x9e%x1axf4x10o@xecxc14&<Zxd1xdexfaxca-'#xa2Kx1cxffx101ax86xb0xd7.xa9x19x04x93mxa1xeexacDex95/xd8xe7xaehp~xc9x0exe9x8f}x9ax8axb0fxf9xebxb7u007f@x87xa0xa6x98xe78xd0+xd45"Sx17x8cx84xa6ㅽbxdaxe6xfcxa1xd9[[~x82xfbKSx82xf0>oxc1 x8b&{xa1rx14Unx03xf7x1f=xe5x1b xa7txed[x8axecxb8xf1xe4xe2xc1x81xb00=cbl·ɬx12`xf2|x1btxe4#xcd" # ETCDCTL_API=3 etcdctl $SSL_OPTS del $SECRETS_PATH/default/secret2 Error: etcdserver: permission denied
Check that user ‘viewsecrets’ has access only to read secrets:
# create_certificates viewsecrets # ETCDCTL_API=3 etcdctl $SSL_OPTS get --keys-only --prefix=true / Error: etcdserver: permission denied # ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret3 -w fields | grep Value "Value" : "k8s:enc:secretbox:v1:key1:xbaxf8,Q@xb9xb6q3$kx04xeeVx99|Z'xdeE<xa5xa9nx91uxb9]RYxcccxe3x13x8bu07b4Qx91x9cR2xccxc5xd9x0ex19?xcax1chxdex1d%xa3Nx85Hxb0xf6֢xe6xabx06xf6x960{xdbxd8^eQxb3x05x03x06)x05JHx16x18fpx9eu<txeax06x12xf1۹yu007fx15xe5x1dxefx8a2Gx85'x94nx1dx99x85ku3xa2~x12x04xe5x84~xaaGxd3nx98x95xa0xc8_1Bxcbx0fxb7;x80xe1xRx86ijfxefxd7SAx950MQfz~)x13xc5xf1xf8x91x14x9d_xbax82[=Mx81Ox1dFNjxc1x98xe4" # ETCDCTL_API=3 etcdctl $SSL_OPTS del $SECRETS_PATH/default/secret3 Error: etcdserver: permission denied
Check that ‘usersecret4’ can access only a specific key:
# create_certificates usersecret4 # ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret3 -w fields | grep Value Error: etcdserver: permission denied # ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret4 -w fields | grep Value "Value" : "k8s:enc:secretbox:v1:key2:x92xeeyjx96xfc쵪-8x0exa7x9axb0x16xe2xb8Jf_x81xecxf65xa9x1axe5\xۛ%Ҝxbbaxxbfx00KzxabaDx1cx94x87xaervsPxf3qxf3xaeHxb8x95-xefr*[ylxf3/xc4x0fx00ax132fxe1x17xbfxffxb4;<xecxc2x01xa8xc8fxffxcdxf3ʦx83Px01xcdux16x16xfaxbax8fxe6xe5x05x96xf7k,xaaxeax0fx99x8fxb3xc7xe6xa4=x93x8axf3Sx17xc6Srxeexea㟷x00x945oxe8x8e:Wxacotxeaj,Px14xbexd0x13xf91Yxf0xf0x93fWxcczD3xb9xa0xb4x9exefx1aEx16xc8j_TXxae" $ ETCDCTL_API=3 etcdctl $SSL_OPTS del $SECRETS_PATH/default/secret4 1
In a cluster with multiple masters, where etcd servers listen on all interfaces and not on localhost, limiting the access to etcd is vital. Kubernetes manages this with RBAC, but by default etcd is only protected by requiring the client to have a valid certificate. While employing EncryptionConfig can take care of most of the issues, it is still possible to have data in etcd that is not fully encrypted. Since confidential data can now be required in configmaps, even in statefulsets and deployments as environment variables, using simply EncryptionConfig is not sufficient. If you cut access entirely to etcd by using authentication and only allow the apiserver to connect, you protect yourself from leaking sensitive data to others. You can create short lived certificates for any other uses and no longer need to worry about the longevity of the certificates. The solution is also applicable to the users added previously to etcd: if you don’t feel confident about what happened to certificates created for them before, you can revoke the users access, or delete them.
