cert-manager-troubleshooting

name: cert-manager-troubleshooting description: Use when TLS certificates are not being issued or renewed, when Certificate or CertificateRequest resources show errors, when ACME challenges fail, or when Issuer or ClusterIssuer resources are not ready

cert-manager Troubleshooting

Diagnose failures in cert-manager — the controller that automates TLS certificate issuance and renewal from sources like Let's Encrypt (ACME), HashiCorp Vault, Venafi, and self-signed CAs.

Keywords

cert-manager, certificate, tls, ssl, https, letsencrypt, acme, issuer, clusterissuer, certificaterequest, order, challenge, http01, dns01, self-signed, ca, renew, renewal, expired, not-ready, ingress, annotation

When to Use This Skill

• Certificate resources show Ready: False or Issuing for too long
• CertificateRequest is stuck or shows errors
• ACME challenges (HTTP-01 or DNS-01) are failing
• Issuer or ClusterIssuer shows Ready: False
• TLS secrets are not being created or contain expired certificates
• Ingress TLS annotations are not triggering certificate creation
• Certificate renewal is not happening before expiry

When NOT to Use

• DNS records are not being created (even if needed for DNS-01 challenges) → use external-dns-troubleshooting
• Issuer credentials come from an external store and the store is failing → use external-secrets-troubleshooting
• cert-manager resources are delivered by Flux and not appearing → use flux-troubleshooting

Related Skills

• external-dns-troubleshooting - DNS records required for DNS-01 challenges
• external-secrets-troubleshooting - Issuer credentials from external stores
• kyverno-troubleshooting - Policies may block cert-manager resources
• k8s-namespace-troubleshooting - General namespace diagnosis
• k8s-platform-operations - Cluster-wide operations
• flux-troubleshooting - GitOps delivery of cert-manager resources

Quick Reference

Diagnostic Workflow

TLS certificate not working?
├─ Certificate resource exists?
│   ├─ No → Check Ingress annotations or create Certificate (Section 4)
│   └─ Yes → Check Certificate status
│       ├─ Ready: True → Certificate issued, check if correct (Section 6)
│       └─ Ready: False → Check CertificateRequest
│           ├─ No CertificateRequest → Issuer reference broken (Section 2)
│           ├─ CertificateRequest pending → Check Issuer health (Section 2)
│           └─ CertificateRequest denied → Check approval/RBAC
├─ Using ACME (Let's Encrypt)?
│   ├─ Order created? → Check Order status
│   │   ├─ Order pending → Check Challenge status (Section 3)
│   │   ├─ Order invalid → All challenges failed
│   │   └─ Order errored → ACME account issue (Section 2)
│   └─ Challenge status?
│       ├─ HTTP-01 pending → Solver pod/ingress issues (Section 3)
│       └─ DNS-01 pending → DNS provider issues (Section 3)
└─ Certificate expired?
    └─ Renewal not triggered → Check renewal config (Section 5)

Section 1: Controller Health

# Check all cert-manager components
kubectl get pods -n cert-manager -o wide

# Controller logs (main component)
kubectl logs -n cert-manager deploy/cert-manager --tail=200

# Webhook logs (validates/converts resources)
kubectl logs -n cert-manager deploy/cert-manager-webhook --tail=100

# CA injector logs (injects CA bundles)
kubectl logs -n cert-manager deploy/cert-manager-cainjector --tail=100

# Check CRDs
kubectl get crd | grep cert-manager

# Check webhook connectivity
kubectl get validatingwebhookconfigurations | grep cert-manager
kubectl get mutatingwebhookconfigurations | grep cert-manager

Controller Issues

Section 2: Issuer and ClusterIssuer

# Check Issuer status
kubectl get issuer -n ${NS} -o wide
kubectl describe issuer ${ISSUER_NAME} -n ${NS}

# Check ClusterIssuer status
kubectl get clusterissuer -o wide
kubectl describe clusterissuer ${ISSUER_NAME}

# Check the Certificate's issuer reference
kubectl get certificate ${CERT_NAME} -n ${NS} -o jsonpath='{.spec.issuerRef}'

ACME Issuer (Let's Encrypt)

# Check ACME account registration
kubectl describe issuer ${ISSUER_NAME} -n ${NS} | grep -A10 "Status:"

# Check ACME account secret
kubectl get secret ${ISSUER_NAME}-account-key -n ${NS} 2>/dev/null

# Verify ACME server URL
kubectl get issuer ${ISSUER_NAME} -n ${NS} -o jsonpath='{.spec.acme.server}'

CA Issuer

# Check CA secret exists and has required keys
kubectl get secret ${CA_SECRET} -n ${NS}
kubectl get secret ${CA_SECRET} -n ${NS} -o jsonpath='{.data}' | jq 'keys'
# Must contain: tls.crt, tls.key

Vault Issuer

# Check Vault connection
kubectl describe issuer ${ISSUER_NAME} -n ${NS} | grep -A20 "Spec:"

# Logs for Vault errors
kubectl logs -n cert-manager deploy/cert-manager --tail=200 | grep -iE 'vault|pki|sign|auth'

Section 3: ACME Challenges (HTTP-01 and DNS-01)

# List all challenges
kubectl get challenge -A -o wide

# Describe failing challenge
kubectl describe challenge ${CHALLENGE_NAME} -n ${NS}

# Check challenge state
kubectl get challenge -A -o json | jq -r '.items[] | "\(.metadata.namespace)/\(.metadata.name)\t\(.spec.type)\t\(.status.state)\t\(.status.reason // "no reason")"'

HTTP-01 Challenge Troubleshooting

# Check solver pod
kubectl get pods -n ${NS} -l acme.cert-manager.io/http01-solver=true

# Check solver Ingress/Service
kubectl get ingress -n ${NS} -l acme.cert-manager.io/http01-solver=true
kubectl get svc -n ${NS} -l acme.cert-manager.io/http01-solver=true

# Test challenge URL locally
# The URL is: http://<domain>/.well-known/acme-challenge/<token>
kubectl get challenge ${CHALLENGE_NAME} -n ${NS} -o jsonpath='{.spec.token}'

DNS-01 Challenge Troubleshooting

# Check which DNS provider is configured
kubectl get issuer ${ISSUER_NAME} -n ${NS} -o json | jq '.spec.acme.solvers[].dns01'

# Check for DNS provider credentials
kubectl get challenge ${CHALLENGE_NAME} -n ${NS} -o json | jq '.spec.solver.dns01'

# Verify TXT record was created
dig TXT _acme-challenge.${DOMAIN} @8.8.8.8

# Check cert-manager DNS provider logs
kubectl logs -n cert-manager deploy/cert-manager --tail=300 | grep -iE 'dns|challenge|present|cleanup|txt'

Common DNS-01 Providers

Section 4: Certificate and Ingress Configuration

# Check Certificate spec
kubectl get certificate ${CERT_NAME} -n ${NS} -o yaml

# Check CertificateRequest chain
kubectl get certificaterequest -n ${NS} -o wide
kubectl describe certificaterequest ${CR_NAME} -n ${NS}

# Check if Ingress annotations trigger cert creation
kubectl get ingress -n ${NS} -o json | jq '.items[] | {name: .metadata.name, annotations: .metadata.annotations, tls: .spec.tls}'

Ingress Annotation Issues

Certificate Spec Problems

Section 5: Renewal and Expiry

# Check certificate dates
kubectl get certificate -A -o json | jq -r '.items[] | "\(.metadata.namespace)/\(.metadata.name)\t\(.status.notAfter)\t\(.status.renewalTime)\tReady=\(.status.conditions[0].status)"'

# Check from the actual TLS secret
kubectl get secret ${SECRET} -n ${NS} -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates -subject

# To force renewal, recommend to user:
# kubectl cert-manager renew ${CERT_NAME} -n ${NS}

Let's Encrypt Rate Limits

Section 6: Verifying Issued Certificates

# Check certificate details from the K8s Secret
kubectl get secret ${SECRET} -n ${NS} -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -text | head -30

# Verify cert chain
kubectl get secret ${SECRET} -n ${NS} -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl verify -show_chain

# Check cert matches key
CERT_MOD=$(kubectl get secret ${SECRET} -n ${NS} -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -modulus | md5)
KEY_MOD=$(kubectl get secret ${SECRET} -n ${NS} -o jsonpath='{.data.tls\.key}' | base64 -d | openssl rsa -noout -modulus | md5)
echo "Match: $([ "$CERT_MOD" = "$KEY_MOD" ] && echo YES || echo NO)"

# Check if staging cert (not trusted in browsers)
kubectl get secret ${SECRET} -n ${NS} -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -issuer
# Staging issuer contains "(STAGING)" or "Fake LE"

MCP Tools Available

When the appropriate MCP servers are connected, prefer these over raw kubectl where available:

• mcp__flux-operator-mcp__get_kubernetes_resources - Query Certificates, CertificateRequests, Orders, Challenges, Issuers
• mcp__flux-operator-mcp__get_kubernetes_logs - Retrieve cert-manager controller, webhook, and cainjector logs
• mcp__flux-operator-mcp__get_kubernetes_metrics - Check cert-manager resource consumption

Common Mistakes

Behavioural Guidelines

1. Follow the resource chain — Certificate → CertificateRequest → Order → Challenge. Diagnose at the deepest failing layer.
2. Never display raw private key contents — Examining certificate metadata (dates, SANs, issuer) is safe. Modulus comparison (openssl rsa -noout -modulus) for cert/key matching is acceptable, but never print full tls.key data.
3. Check the Issuer first — If the Issuer is Ready: False, no Certificate can be issued.
4. Distinguish staging from production — Staging certs are not browser-trusted but have much higher rate limits for testing.
5. Watch for rate limits — Let's Encrypt production has strict limits. Check kubectl describe order for rate limit messages before retrying.
6. Verify DNS resolution — For both HTTP-01 and DNS-01, the domain must resolve correctly. Use dig to verify.
7. Check all cert-manager components — The controller, webhook, and cainjector all play different roles. A webhook failure blocks resource creation entirely.