Docker Hub Rate Limit Era: How Spegel’s Stateless Cache Enables Offline Image Sharing

TL;DR

Spegel is a very interesting project that can help us achieve image sharing within Kubernetes clusters, improve image pull speeds, and reduce dependency on external image registries. For scenarios such as offline or intranet environments, bandwidth optimization and cost control, disaster recovery, and high availability, especially in bypassing Docker Hub image pull limits, Spegel is an excellent choice.

Background

Docker Hub’s restrictions are becoming increasingly stringent. Starting from April 1st this year, unauthenticated users are limited to pulling images up to 10 times per hour, and this restriction is based on IP addresses or IPv6 subnets. This means that in a local area network where multiple users share a public IP, this limitation will be even more severe. If you receive a 429 Too Many Requests response, it indicates that the limit has been exceeded, and the pull request has been throttled.

Docker Hub has been free for many years, but as the number of users has increased, so have the costs. To ensure service stability, Docker Hub now requires payment, which is an inevitable trend. After all, maintaining a service of such global scale comes with high operational costs. There is no such thing as a free lunch.

However, for individual developers or small teams, such restrictions may impact their daily development.

How to Avoid Rate Limiting

1. Log in to Docker Hub

The simplest method is to log in to Docker Hub, which increases the number of image pulls to 100 per hour. This should be sufficient for individual developers.

For Kubernetes users, this can be achieved by creating a Docker Hub Secret.

kubectl create secret docker-registry dockerhub --docker-server=https://index.docker.io/v1/ --docker-username=USER_NAME --docker-password=PASSWORD

2. Use Other Image Registries

Docker Hub is not the only image registry. Many other registries offer free services, although they may not have as many images as Docker Hub. Examples include:

• Quay.io: Red Hat’s image registry, known for high security, supports private repositories, but has fewer images.
• GHCR: GitHub Container Registry, a service provided by GitHub, allows building and pushing images via GitHub Actions.
• Cloud Provider Products: Such as Alibaba Cloud, Tencent Cloud, Huawei Cloud, Azure, GCP, and AWS, all offer image registry services, suitable for ecosystems on public clouds.

3. Self-Hosted Image Registry

Self-hosting an image registry is the best approach, allowing full control over image pulls and pushes without restrictions. Examples include:

• Harbor: A CNCF project, an enterprise-grade registry open-sourced by VMware, supporting RBAC, image scanning, audit logs, and more.
• Nexus Repository: A powerful repository by Sonatype, supporting Maven, Docker, NPM, and more.
• Docker Registry: The official Docker registry, which can be self-hosted but has limited features, lacking a UI (can be paired with docker-registry-browser for visualization), ideal for personal development and testing environments.

4. Avoid Using the Latest Tag

Using fixed versions instead of the latest tag can prevent frequent pulls due to changes in the latest tag.

5. Avoid Using the Always Pull Policy When Unnecessary

In Kubernetes, imagePullPolicy: Always can be changed to IfNotPresent or even Never to avoid frequent image pulls. This is also a correct approach to image usage.

6. Use Cluster Local Cache

Using a cluster-local image cache is a compromise, staying connected to Docker Hub while avoiding frequent pulls that trigger rate limits. Reusing already pulled images can prevent frequent pulls and improve pull speeds. When Docker Hub is unavailable (due to network or service outages), the local cache can provide images, ensuring service continuity.

This is the solution we are introducing today: using Spegel stateless image cache to bypass Docker Hub’s new rate limits.

Spegel Stateless Image Cache

What is Spegel

Spegel, meaning “mirror” in Swedish, is a stateless local OCI registry mirroring tool designed for Kubernetes clusters, aimed at optimizing image pull efficiency and reducing dependency on external image registries. Using the P2P protocol, it supports an intra-cluster image sharing mechanism, allowing nodes within the cluster to share images, improving pull speeds. It is suitable for offline or intranet environments, bandwidth optimization and cost control, disaster recovery, and high availability scenarios.

Core features of Spegel:

1. Intra-Cluster Image Sharing Mechanism: Spegel allows each node in a Kubernetes cluster to act as a local image registry. When a node pulls an image for the first time, other nodes can directly fetch the image from this node, eliminating the need to repeatedly access external registries. This peer-to-peer (P2P) sharing mechanism significantly reduces cross-network pull overhead, enhancing workload startup speeds.
2. Stateless Design and High Compatibility: As a stateless service, Spegel does not rely on persistent storage, achieving image distribution solely through inter-node communication. It is compatible with mainstream container runtimes (e.g., Containerd).
3. Flexible Image Source Configuration: Supports configuring image sharing from public or private registries.

How Spegel Works

Here, we borrow an official diagram to illustrate how Spegel works.

Spegel has three components: Registry, Routing and Discovery, and Advertisement Mechanism.

1. Spegel does not rely on centralized image storage but uses the local storage of each node in the cluster as temporary cache. When a node pulls an image from an external registry (e.g., Docker Hub) for the first time, the image is cached locally and recorded in the registry.
2. Nodes broadcast the registry via a Distributed Hash Table (DHT, Spegel uses Kademlia DHT’s Go implementation) and record which nodes have cached specific images. When a new node requests an image, it can quickly locate the available node for pulling. It prioritizes pulling from within the cluster; if unavailable, it pulls from external registries.
3. Spegel integrates with container runtimes (currently only Containerd). By configuring Containerd Registry Mirrors, Spegel is registered as a “mirror endpoint” for the registry, intercepting default image pull requests.

Spegel Compatibility

Spegel has been tested for compatibility with the following Kubernetes distributions. Green indicates Spegel is ready to use, yellow indicates additional configuration is needed, and red indicates Spegel cannot be used.

StatusDistribution🟢AKS🟢Minikube🟡EKS🟡K3S and RKE2🟡Kind🟡Talos🟡VKE🔴GKE🔴DigitalOcean

K3s and Spegel

K3s’s built-in registry mirror (Embedded Registry Mirror) has been available as an experimental feature since January 2024: v1.26.13+k3s1, v1.27.10+k3s1, v1.28.6+k3s1, v1.29.1+k3s1. The GA version will be available starting December 2024: v1.29.12+k3s1, v1.30.8+k3s1, v1.31.4+k3s1.

K3s embeds Spegel’s functionality, allowing direct use of Spegel within K3s clusters without additional installation or deployment.

Start the K3s server with the --embedded-registry parameter, or set embedded-registry: true in the configuration file. Once enabled, two ports 6443 and 5001 will be opened on all nodes. 6443 serves as the local OCI repository port, and 5001 is used for peer-to-peer broadcasting of available images among nodes.

The 5001 port can be changed via the K3S_P2P_PORT environment variable. All nodes must have the same port setting; changing it after setup is not supported or recommended.

Next, we will demonstrate deploying Spegel on a K3s cluster to achieve intra-cluster image sharing.

Demonstration

The demonstration requires at least two virtual machines: 1 server node (master) and 1 agent node (member1).

1. Configure Containerd Registry Mirror

Configure the Containerd registry mirror on K3s via the /etc/rancher/k3s/registries.yaml file. This must be configured on all nodes.

Execute the following command on both nodes, ensuring all nodes prioritize pulling images from local and other nodes for docker.io and registry.k8s.io:

sudo mkdir -p /etc/rancher/k3s/
sudo tee /etc/rancher/k3s/registries.yaml <<EOF
mirrors:   
  docker.io:  
  registry.k8s.io:
EOF

2. Create the Cluster

Here, I recommend my previously written 1-Minute Quick Setup Guide, with the latest script updated to k3s-cluster-automation. Download the script locally and execute the following command:

export HOSTS="13.75.122.224 52.175.36.37"

setupk3s \
    --k3s-version v1.29.12+k3s1 \
    --embedded-registry \
    --mini

You may see the following logs on K3s nodes, but don’t worry, this is because Kademlia DHT requires at least 20 nodes to form a stable topology.

Mar 01 12:04:54 master k3s[21794]: 2025–03–01T12:04:54.302Z WARN dht go-libp2p-kad-dht@v0.25.2/lookup.go:43 network size estimator track peers: expected bucket size number of peers

3. Prepare Test Images

When deploying applications, we need an image that does not exist on Docker Hub, pre-saved on the server node. Here, we use the nginx image, tagging it with a different name:

sudo crictl pull docker.io/library/nginx:1.27.4
sudo ctr -n k8s.io images tag docker.io/library/nginx:1.27.4 docker.io/library/nginx-not-exist:1.27.4

Check if the image was successfully tagged:

# server node: master
sudo crictl img list | grep nginx-not-exist
docker.io/library/nginx-not-exist            1.27.4              b52e0b094bc0e       72.2MB

This image exists only on the server node and is not present on the agent node.

4. Deploy the Application

Deploy a simple Deployment using this image:

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx-not-exist:1.27.4
        ports:
        - containerPort: 80 
EOF

Check the Pod’s status and see that it is scheduled to the agent node member1 and runs successfully:

kubectl get po -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP          NODE      NOMINATED NODE   READINESS GATES
nginx-74ddfd8c7d-8cn4v   1/1     Running   0          11s   10.42.1.2   member1   <none>           <none>

Use kubectl describe to view the Pod's details and see that the image was pulled from the server node, resulting in a very fast pull:

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  66s   default-scheduler  Successfully assigned default/nginx-74ddfd8c7d-8cn4v to member1
  Normal  Pulling    66s   kubelet            Pulling image "nginx-not-exist:1.27.4"
  Normal  Pulled     63s   kubelet            Successfully pulled image "nginx-not-exist:1.27.4" in 3.323s (3.323s including waiting)
  Normal  Created    63s   kubelet            Created container nginx
  Normal  Started    63s   kubelet            Started container nginx