Automated Kubernetes Gateway Certificates Management with Cert Manager

8 min readMay 9, 2024

In this article, we will explore how to use cert-manager to automatically manage Kubernetes Gateway certificates, thereby enhancing cluster ingress security and improving management efficiency.

Background

Kubernetes Gateway

Perhaps, it should be precisely called the Kubernetes Gateway API (hereafter referred to as GWAPI). GWAPI is an open-source project managed by the SIG-NETWORK community. It is a standard that does not provide an implementation. The Gateway API is considered a successor to the Kubernetes Ingress API and was first introduced in the 2019 Ingress Revolution. The first version was released in November 2020, and GWAPI has evolved rapidly, with version 1.0.0 already available and 1.1.0 about to be released (with 1.1.0-rc2 already published).

GWAPI offers a method to manage both internal and external traffic in Kubernetes clusters. It provides a more detailed and modular approach to describing routing and gateway configurations, making it more flexible and scalable compared to the traditional Ingress API.

cert-manager

cert-manager is a specialized tool designed for automated certificate management in Kubernetes. It simplifies the management of X.509 certificates within the Kubernetes cluster, enabling easy issuance, renewal, and management of SSL/TLS certificates for various resources such as services, applications, and internal communications. This is crucial for ensuring the security of data transmission.

cert-manager has the following main features:

Automated Certificate Issuance and Renewal: cert-manager can handle the application, renewal, and lifecycle management of certificates automatically.
Support for Multiple Issuance Sources: cert-manager supports a variety of Certificate Authorities (CAs), including self-signed certificates; using public CAs like Let’s Encrypt through ACME; and enterprise-owned CA systems, such as HashiCorp Vault and RustyVault by Tongsuo-Project.
Kubernetes Native Integration: As a native component of Kubernetes, cert-manager uses CRDs to define resources like Issuers and Certificates.

Particularly the last point, cert-manager can seamlessly integrate with other components of the Kubernetes ecosystem, such as the integration with K8s Gateway discussed in this article.

Why cert-manager?

In Kubernetes environments, the Gateway acts as the primary ingress component of the cluster, managing incoming traffic. Among its many features, security is particularly prominent and crucial. Encryption of communications is a primary measure to ensure data security. By using TLS/SSL certificates, the Gateway ensures that all data entering and leaving the cluster is encrypted. This not only protects data from eavesdropping and tampering but also ensures that communication between clients and services is secure. In addition to this one-way SSL encryption, it can also be configured to support two-way SSL encryption for end-to-end security verification.

Using cert-manager to manage Kubernetes Gateway certificates offers several advantages, significantly enhancing operational efficiency, boosting system security, and supporting enterprise-level deployment and management in a scalable and maintainable way:

Automated Certificate Management: Developers and administrators can reduce manual intervention, thus lowering the risk of errors and service interruptions due to manual handling of certificate-related tasks (such as forgetting to renew).
Integration with External CAs: cert-manager supports various certificate authorities, including Let’s Encrypt, HashiCorp Vault, Venafi, etc. This makes it easy to use commercial or self-built CAs in a Kubernetes environment and ensures compliance and certification of certificates.
Security and Compliance: Automated certificate rotation and the use of short-lived certificates enhance security and compliance (auto-updating certificates mean that certificates stolen by attackers are only useful for a short validity period).
Native Integration with Kubernetes: Operating cert-manager’s CRDs is as user-friendly as managing GWAPI’s CRDs, and it integrates seamlessly with GWAPI.

Demonstration

Overview

FSM Gateway, as one of the implementations of the Gateway API, is also one of the many components of the FSM ecosystem, bearing the responsibility of managing ingress traffic for Kubernetes. In this demonstration, we will use the FSM Gateway as the gateway solution, utilizing its TLS Offloading feature to enhance the security of upstream services.
As mentioned earlier, cert-manager supports multiple issuance sources. For simplicity, we will choose the simplest option of self-signed CA certificates.

Prerequisites

K8s cluster >=1.22
kubectl
helm

Installing FSM Gateway

Refer to the FSM Gateway installation documentation. We will use the CLI for installation, with the current version of FSM being 1.2.4.

system=$(uname -s | tr '[:upper:]' '[:lower:]')
arch=$(uname -m | sed -E 's/x86_/amd/' | sed -E 's/aarch/arm/')
release=v1.2.4
curl -L https://github.com/flomesh-io/fsm/releases/download/$release/fsm-$release-$system-$arch.tar.gz | tar -vxzf -
./$system-$arch/fsm version

Execute the following command to install.

fsm install \
    --set=fsm.fsmGateway.enabled=true

After successful installation, you can see that the GatewayClass fsm-gateway-cls has been registered.

kubectl get gatewayclass
NAME              CONTROLLER                      ACCEPTED   AGE
fsm-gateway-cls   flomesh.io/gateway-controller   True       106s

Installing cert-manager

The installation of cert-manager can be done using kubectl manifests or a helm chart. We will choose the helm chart here. Remember to install the CRDs during installation.

helm repo add jetstack https://charts.jetstack.io --force-update
helm repo update

helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.14.5 \
  --set installCRDs=true

Since we are using a self-signed CA certificate, before creating an issuer, first self-sign a CA certificate.

openssl genrsa 2048 > ca-key.pem

openssl req -new -x509 -nodes -days 365000 \
   -key ca-key.pem \
   -out ca-cert.pem \
   -subj '/CN=flomesh.io'

Store the CA certificate in the Secret ca-key-pair, specifying the namespace cert-manager.

kubectl create secret generic -n cert-manager ca-key-pair \
  --from-file=tls.crt=./ca-cert.pem \
  --from-file=tls.key=./ca-key.pem

Here, we directly create a ClusterIssuer of type ca, specifying the secret ca-key-pair we created above.

cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ca-issuer-sample
spec:
  ca:
    secretName: ca-key-pair
EOF

Once everything is ready, let’s deploy a sample application for testing.

Deploying the Sample Application

Create an httpbin service.

kubectl create namespace httpbin
cat <<EOF | kubectl apply -n httpbin -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
  template:
    metadata:
      labels:
        app: httpbin
    spec:
      containers:
      - name: httpbin
        image: kennethreitz/httpbin
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
spec:
  selector:
    app: httpbin
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 80
EOF

Issuing TLS Certificates

To issue TLS certificates, we need to create a CR Certificate. cert-manager will monitor this resource, then issue a certificate through the resource's configuration and store it in the secret specified by the field secretName.

The certificate is stored in the secret simple-gateway-cert.
The certificate’s validity period duration is set to 60m.
The certificate rotation advance time renewBefore is set to 59m, meaning it rotates once every minute.
The common name commonName is set to foo.example.com.
Most importantly, specify the certificate’s issuer as ClusterIssuer ca-issuer-sample.

cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: simple-gateway-cert
  namespace: httpbin
spec:
  secretName: simple-gateway-cert
  duration: 60m
  renewBefore: 59m
  subject:
    organizations:
      - flomesh-io
  commonName: foo.example.com
  issuerRef:
    name: ca-issuer-sample
    kind: ClusterIssuer
    group: cert-manager.io
EOF

After creating the Certificate, you can find the cert-manager-created secret in the namespace httpbin, type kubernetes.io/tls.

kubectl get secret simple-gateway-cert
NAME                  TYPE                DATA   AGE
simple-gateway-cert   kubernetes.io/tls   3      20s

With the certificate in hand, we can now expose an HTTPS access point for the httpbin service.

Creating the Gateway and Routes

To expose the httpbin service to the outside of the cluster, we need to create a gateway and routes:

Listener is listening on port 8000.
Protocol is set to TLS.
Certificate is provided through the secret simple-gateway-cert.
Most importantly, the mode is set to Terminate.

cat <<EOF | kubectl apply -n httpbin -f -
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: simple-fsm-gateway
spec:
  gatewayClassName: fsm-gateway-cls
  listeners:
  - name: https
    port: 8000
    protocol: HTTPS
    tls:
      certificateRefs:
      - kind: Secret
        name: simple-gateway-cert
      mode: Terminate
    allowedRoutes:
      namespaces:
        from: Same
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: http-route-foo
spec:
  parentRefs:
  - name: simple-fsm-gateway
    port: 8000
  hostnames:
    - foo.example.com
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: httpbin
      port: 8080
EOF

Testing

If you access without providing the CA certificate, you might encounter issues like the one below.

curl https://foo.example.com/headers --connect-to foo.example.com:443:$GATEWAY_IP:8000
curl: (60) SSL certificate problem: unable to get local issuer certificate

Specify the CA certificate and try accessing again with the -v parameter to see a successful request.

curl --cacert ca-cert.pem https://foo.example.com/headers  --connect-to foo.example.com:443:$GATEWAY_IP:8000 -v
* Connecting to hostname: 198.19.249.153
* Connecting to port: 8000
*   Trying 198.19.249.153:8000...
* Connected to 198.19.249.153 (198.19.249.153) port 8000
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: ca-cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: O=flomesh-io; CN=foo.example.com
*  start date: May  8 14:59:33 2024 GMT
*  expire date: May  8 15:59:33 2024 GMT
*  common name: foo.example.com (matched)
*  issuer: CN=flomesh.io
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://foo.example.com/headers
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: foo.example.com]
* [HTTP/2] [1] [:path: /headers]
* [HTTP/2] [1] [user-agent: curl/8.4.0]
* [HTTP/2] [1] [accept: */*]
> GET /headers HTTP/2
> Host: foo.example.com
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/2 200
< server: gunicorn/19.9.0
< date: Wed, 08 May 2024 15:00:07 GMT
< content-type: application/json
< content-length: 141
< access-control-allow-origin: *
< access-control-allow-credentials: true
<
{
  "headers": {
    "Accept": "*/*",
    "Connection": "keep-alive",
    "Host": "foo.example.com",
    "User-Agent": "curl/8.4.0"
  }
}
* Connection #0 to host 198.19.249.153 left intact

From the output debug content, we can also find detailed information about the TLS handshake, such as the certificate validity period start date: May 8 14:59:33 2024 GMT and expire date: May 8 15:59:33 2024 GMT.

After waiting for 1 minute and making another request, you will see that the certificate information has been updated. At this point, if you check the Certificate resource, you can confirm that the certificate has indeed been rotated.

kubectl describe certificate simple-gateway-cert

Events:
  Type    Reason     Age               From                                       Message
  ----    ------     ----              ----                                       -------
  Normal  Issuing    61s               cert-manager-certificates-trigger          Renewing certificate as renewal was scheduled at 2024-05-08 15:03:33 +0000 UTC
  Normal  Requested  61s               cert-manager-certificates-request-manager  Created new CertificateRequest resource "simple-gateway-cert-1"
  Normal  Issuing    1s                cert-manager-certificates-trigger          Renewing certificate as renewal was scheduled at 2024-05-08 15:04:33 +0000 UTC
  Normal  Reused     1s (x2 over 61s)  cert-manager-certificates-key-manager      Reusing private key stored in existing Secret resource "simple-gateway-cert"
  Normal  Requested  1s                cert-manager-certificates-request-manager  Created new CertificateRequest resource "simple-gateway-cert-2"
  Normal  Issuing    1s (x2 over 61s)  cert-manager-certificates-issuing          The certificate has been successfully issued

Summary

Using cert-manager to automatically manage Kubernetes Gateway certificates can significantly enhance security and management efficiency. Through automated certificate handling, such as automatic renewals and rotation, cert-manager reduces the administrative burden and minimizes human errors, ensuring continuous system security. The integration and flexibility of this tool make it an ideal choice for maintaining the security of Kubernetes clusters.