Kubectl debug: Time to say goodbye to baking diagnostic tools into your containers.

Released as an alpha feature in Kubernetes 1.18, kubectl debug now gives the ability to load an ephemeral container into your pod. This is an enormous win for developer experience as many of us will have had to include tools for diagnosing pod issues ( conntrack, netstat, vi and dig all spring to mind) into the container image.

This brings a new age of diagnostic democratisation and security for all shapes and sizes of containers. No longer forcing the fetch of tooling into the pod at runtime and making analysis of distroless images more accessible.

I spent a few hours testing it out and thought I’d share my initial impressions.

The setup

For my example, I wanted to put something as simple as possible together as a demonstration. I’ve created a basic Nginx reverse proxy, applying some load and demonstrating how we can debug it easily with kubectl debug.

I’ve created a basic kind cluster to test this locally.

cat <<EOF | kind create cluster --config=- 
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
kubeadmConfigPatches:
# enable EphemeralContainers feature gate
- |
  kind: ClusterConfiguration
  metadata:
    name: config
  apiServer:
    extraArgs:
      "feature-gates": "EphemeralContainers=true"
  scheduler:
    extraArgs:
      "feature-gates": "EphemeralContainers=true"
  controllerManager:
    extraArgs:
      "feature-gates": "EphemeralContainers=true"
- |
  kind: InitConfiguration
  metadata:
    name: config
  nodeRegistration:
    kubeletExtraArgs:
      "feature-gates": "EphemeralContainers=true"
- |
  kind: KubeletConfiguration
  featureGates:
    EphemeralContainers: true
- |
  kind: KubeProxyConfiguration
  featureGates:
    EphemeralContainers: true
EOF

Kind is so fast and awesome!

This will give us a 1 node cluster with feature flags enabled for ephemeral containers.

Install the deployment with helm…

helm install nginx stable/nginx-ingress
kubectl apply -f - << EOF
kind: ConfigMap
apiVersion: v1
metadata:
  name: ingress-configmap
data:
  enable-access-log-for-default-backend: "true"
EOF

Enabling this config is important for ngxtop to detect any server activity on the default backend.

Let’s apply some load.
I am using vegeta to load Nginx with requests.

kubectl port-forward svc/nginx-nginx-ingress-controller 8080:80 &
echo "GET http://localhost:8080" | vegeta attack -rate=10/

Setup done.

We’re going to fire off a bunch of requests at the Nginx default backend.

Sure it’s contrived but it’s enough to demonstrate ephemeral containers.

Debugging

With a large number of incoming requests hitting our reverse proxy, it is a normal activity to want to monitor the performance of the pod.

In this case, I’d like to use ngxtop to investigate the process.

ngxtop is designed to run in a short-period time just like the top command for troubleshooting and monitoring your Nginx server.

POD=$(kubectl get pod -l run=my-nginx -o jsonpath="{.items[0].metadata.name}")
 kubectl exec -it $POD ngxtop
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "e7db35b81cf45fbfc2b8843e95daf7d27ce86701e00d2ac3336256968150d8f3": OCI runtime exec failed: exec failed: container_linux.go:346: starting container process caused "exec: \"ngxtop\": executable file not found in $PATH": unknown

Oh no! ngxtop isn’t installed by default on the nginx pod.

But what if there was a way to get the tooling I need into my pod without having to bake it into the container?

Rise of the ephemeral container

An ephemeral container is a type of container that you can run from inside an existing pod. It is the feature underpinning the new capabilities released with kubectl debug.

Here we set our target $POD and the container that we want to mount our new container against (image). The docker image tibbar/ngxtop-debug being pulled in here is a simple container I made that uses ngxtop with python2.

POD=$(kubectl get pod -l app=nginx-ingress -o jsonpath="{.items[0].metadata.name}")
 kubectl alpha debug -it $POD --image=tibbar/ngxtop-debug:latest                    
Defaulting debug container name to debugger-mk8ck.
If you don't see a command prompt, try pressing enter.
# which ngxtop
/usr/local/bin/ngxtop

Great! So now we have a container within the pod that has our selected tooling ready to go.

This means when we leverage process space sharing we can see what’s going on inside our Nginx container!.

$ ngxtop top remote_addr
running for 20 seconds, 3215 records processed: 159.62 req/sec

top remote_addr
| remote_addr     |   count |
|-----------------+---------|
| 118.173.177.161 |      20 |

We can even reattach after exiting a session…

Session ended, resume using 'kubectl alpha attach my-nginx-75897978cd-5c24r -c debugger-vhc7t -i -t' command when the pod is running

Verdict

The new debug feature that leverages ephemeral containers is absolutely fantastic. This improvement will allow developers to move towards distroless images for services such as proxies and data stores, lowering footprint and vulnerability attack surface ( We’ve all seen someone curl down binaries into a container before).

To summarise:

• Mounting an ephemeral container with a weaponised set of tools to diagnose other containers in your pod.
• Reducing container diagnostic binaries lowers footprint and attack profile.
• Developer productivity win as it allows for the creation of specific containers for debugging e.g kubectl alpha debug -it $POD --image=alex/kafka-tools

I love it!