Docker Hub Rate Limit Era: How Spegel’s Stateless Cache Enables Offline Image Sharing
TL;DR
Spegel is a very interesting project that can help us achieve image sharing within Kubernetes clusters, improve image pull speeds, and reduce dependency on external image registries. For scenarios such as offline or intranet environments, bandwidth optimization and cost control, disaster recovery, and high availability, especially in bypassing Docker Hub image pull limits, Spegel is an excellent choice.
Background
Docker Hub’s restrictions are becoming increasingly stringent. Starting from April 1st this year, unauthenticated users are limited to pulling images up to 10 times per hour, and this restriction is based on IP addresses or IPv6 subnets. This means that in a local area network where multiple users share a public IP, this limitation will be even more severe. If you receive a 429 Too Many Requests
response, it indicates that the limit has been exceeded, and the pull request has been throttled.
Docker Hub has been free for many years, but as the number of users has increased, so have the costs. To ensure service stability, Docker Hub now requires payment, which is an inevitable trend. After all, maintaining a service of such global scale comes with high operational costs. There is no such thing as a free lunch.
However, for individual developers or small teams, such restrictions may impact their daily development.
How to Avoid Rate Limiting
1. Log in to Docker Hub
The simplest method is to log in to Docker Hub, which increases the number of image pulls to 100 per hour. This should be sufficient for individual developers.
For Kubernetes users, this can be achieved by creating a Docker Hub Secret.
kubectl create secret docker-registry dockerhub --docker-server=https://index.docker.io/v1/ --docker-username=USER_NAME --docker-password=PASSWORD
2. Use Other Image Registries
Docker Hub is not the only image registry. Many other registries offer free services, although they may not have as many images as Docker Hub. Examples include:
- Quay.io: Red Hat’s image registry, known for high security, supports private repositories, but has fewer images.
- GHCR: GitHub Container Registry, a service provided by GitHub, allows building and pushing images via GitHub Actions.
- Cloud Provider Products: Such as Alibaba Cloud, Tencent Cloud, Huawei Cloud, Azure, GCP, and AWS, all offer image registry services, suitable for ecosystems on public clouds.
3. Self-Hosted Image Registry
Self-hosting an image registry is the best approach, allowing full control over image pulls and pushes without restrictions. Examples include:
- Harbor: A CNCF project, an enterprise-grade registry open-sourced by VMware, supporting RBAC, image scanning, audit logs, and more.
- Nexus Repository: A powerful repository by Sonatype, supporting Maven, Docker, NPM, and more.
- Docker Registry: The official Docker registry, which can be self-hosted but has limited features, lacking a UI (can be paired with docker-registry-browser for visualization), ideal for personal development and testing environments.
4. Avoid Using the Latest Tag
Using fixed versions instead of the latest
tag can prevent frequent pulls due to changes in the latest
tag.
5. Avoid Using the Always Pull Policy When Unnecessary
In Kubernetes, imagePullPolicy: Always
can be changed to IfNotPresent
or even Never
to avoid frequent image pulls. This is also a correct approach to image usage.
6. Use Cluster Local Cache
Using a cluster-local image cache is a compromise, staying connected to Docker Hub while avoiding frequent pulls that trigger rate limits. Reusing already pulled images can prevent frequent pulls and improve pull speeds. When Docker Hub is unavailable (due to network or service outages), the local cache can provide images, ensuring service continuity.
This is the solution we are introducing today: using Spegel stateless image cache to bypass Docker Hub’s new rate limits.
Spegel Stateless Image Cache
What is Spegel
Spegel, meaning “mirror” in Swedish, is a stateless local OCI registry mirroring tool designed for Kubernetes clusters, aimed at optimizing image pull efficiency and reducing dependency on external image registries. Using the P2P protocol, it supports an intra-cluster image sharing mechanism, allowing nodes within the cluster to share images, improving pull speeds. It is suitable for offline or intranet environments, bandwidth optimization and cost control, disaster recovery, and high availability scenarios.
Core features of Spegel:
- Intra-Cluster Image Sharing Mechanism: Spegel allows each node in a Kubernetes cluster to act as a local image registry. When a node pulls an image for the first time, other nodes can directly fetch the image from this node, eliminating the need to repeatedly access external registries. This peer-to-peer (P2P) sharing mechanism significantly reduces cross-network pull overhead, enhancing workload startup speeds.
- Stateless Design and High Compatibility: As a stateless service, Spegel does not rely on persistent storage, achieving image distribution solely through inter-node communication. It is compatible with mainstream container runtimes (e.g., Containerd).
- Flexible Image Source Configuration: Supports configuring image sharing from public or private registries.
How Spegel Works
Here, we borrow an official diagram to illustrate how Spegel works.
Spegel has three components: Registry, Routing and Discovery, and Advertisement Mechanism.
- Spegel does not rely on centralized image storage but uses the local storage of each node in the cluster as temporary cache. When a node pulls an image from an external registry (e.g., Docker Hub) for the first time, the image is cached locally and recorded in the registry.
- Nodes broadcast the registry via a Distributed Hash Table (DHT, Spegel uses Kademlia DHT’s Go implementation) and record which nodes have cached specific images. When a new node requests an image, it can quickly locate the available node for pulling. It prioritizes pulling from within the cluster; if unavailable, it pulls from external registries.
- Spegel integrates with container runtimes (currently only Containerd). By configuring Containerd Registry Mirrors, Spegel is registered as a “mirror endpoint” for the registry, intercepting default image pull requests.
Spegel Compatibility
Spegel has been tested for compatibility with the following Kubernetes distributions. Green indicates Spegel is ready to use, yellow indicates additional configuration is needed, and red indicates Spegel cannot be used.
StatusDistribution🟢AKS🟢Minikube🟡EKS🟡K3S and RKE2🟡Kind🟡Talos🟡VKE🔴GKE🔴DigitalOcean
K3s and Spegel
K3s’s built-in registry mirror (Embedded Registry Mirror) has been available as an experimental feature since January 2024: v1.26.13+k3s1, v1.27.10+k3s1, v1.28.6+k3s1, v1.29.1+k3s1. The GA version will be available starting December 2024: v1.29.12+k3s1, v1.30.8+k3s1, v1.31.4+k3s1.
K3s embeds Spegel’s functionality, allowing direct use of Spegel within K3s clusters without additional installation or deployment.
Start the K3s server with the --embedded-registry
parameter, or set embedded-registry: true
in the configuration file. Once enabled, two ports 6443
and 5001
will be opened on all nodes. 6443
serves as the local OCI repository port, and 5001
is used for peer-to-peer broadcasting of available images among nodes.
The
5001
port can be changed via theK3S_P2P_PORT
environment variable. All nodes must have the same port setting; changing it after setup is not supported or recommended.
Next, we will demonstrate deploying Spegel on a K3s cluster to achieve intra-cluster image sharing.
Demonstration
The demonstration requires at least two virtual machines: 1 server node (master) and 1 agent node (member1).
1. Configure Containerd Registry Mirror
Configure the Containerd registry mirror on K3s via the /etc/rancher/k3s/registries.yaml
file. This must be configured on all nodes.
Execute the following command on both nodes, ensuring all nodes prioritize pulling images from local and other nodes for docker.io
and registry.k8s.io
:
sudo mkdir -p /etc/rancher/k3s/
sudo tee /etc/rancher/k3s/registries.yaml <<EOF
mirrors:
docker.io:
registry.k8s.io:
EOF
2. Create the Cluster
Here, I recommend my previously written 1-Minute Quick Setup Guide, with the latest script updated to k3s-cluster-automation. Download the script locally and execute the following command:
export HOSTS="13.75.122.224 52.175.36.37"
setupk3s \
--k3s-version v1.29.12+k3s1 \
--embedded-registry \
--mini
You may see the following logs on K3s nodes, but don’t worry, this is because Kademlia DHT requires at least 20 nodes to form a stable topology.
Mar 01 12:04:54 master k3s[21794]: 2025–03–01T12:04:54.302Z WARN dht go-libp2p-kad-dht@v0.25.2/lookup.go:43 network size estimator track peers: expected bucket size number of peers
3. Prepare Test Images
When deploying applications, we need an image that does not exist on Docker Hub, pre-saved on the server node. Here, we use the nginx image, tagging it with a different name:
sudo crictl pull docker.io/library/nginx:1.27.4
sudo ctr -n k8s.io images tag docker.io/library/nginx:1.27.4 docker.io/library/nginx-not-exist:1.27.4
Check if the image was successfully tagged:
# server node: master
sudo crictl img list | grep nginx-not-exist
docker.io/library/nginx-not-exist 1.27.4 b52e0b094bc0e 72.2MB
This image exists only on the server node and is not present on the agent node.
4. Deploy the Application
Deploy a simple Deployment using this image:
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx-not-exist:1.27.4
ports:
- containerPort: 80
EOF
Check the Pod’s status and see that it is scheduled to the agent node member1 and runs successfully:
kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-74ddfd8c7d-8cn4v 1/1 Running 0 11s 10.42.1.2 member1 <none> <none>
Use kubectl describe
to view the Pod's details and see that the image was pulled from the server node, resulting in a very fast pull:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 66s default-scheduler Successfully assigned default/nginx-74ddfd8c7d-8cn4v to member1
Normal Pulling 66s kubelet Pulling image "nginx-not-exist:1.27.4"
Normal Pulled 63s kubelet Successfully pulled image "nginx-not-exist:1.27.4" in 3.323s (3.323s including waiting)
Normal Created 63s kubelet Created container nginx
Normal Started 63s kubelet Started container nginx