Docker
Help! We’ve ran into a DockerHub rate limit!
About
Yes, it is still happining. In 2025! Here you will find:
- Podman Dockerhub Mirror Configuration
- K8s Quickfix: Rewriting Existing K8s Resources
- Permanent Mirror Configuration for
containerd - K8s Admission Webhook to do the same
Podman Dockerhub Mirror Configuration
~/.config/containers/registries.conf.d/dockerhub-mirror.conf:
[[registry]]
prefix = "docker.io"
insecure = false
blocked = false
location = "public.ecr.aws/docker"
[[registry.mirror]]
location = "mirror.gcr.io"
[[registry.mirror]]
location = "gitlab.com/acme-org/dependency_proxy/containers"
[[registry.mirror]]
location = "registry-1.docker.io"
[[registry.mirror]]
location = "123456789012.dkr.ecr.us-east-1.amazonaws.com/docker-io"
I hope you are using ecr-login for your ECR registries ;)
export REGISTRY_AUTH_FILE=$HOME/.config/containers/auth.json
{
"auths": {
"docker.io": {
"auth": "eGw4ZGVwXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXem40VQ=="
},
"gitlab.com": {
"auth": "cmVXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXSYQ=="
},
"registry.gitlab.com": {
"auth": "cmVXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXSYQ=="
}
},
"credHelpers": {
"*": "",
"123456789012.dkr.ecr.us-east-1.amazonaws.com": "ecr-login",
"345678901234.dkr.ecr.us-east-1.amazonaws.com": "ecr-login"
}
}
K8s Quickfix: Rewriting Existing K8s Resources
$ cd $(mktemp -d)
$ (
kubectl get pods --field-selector=status.phase=Pending -A -ojson | jq -c '.items[]';
kubectl get deployments -ojson -A | jq -c '.items[]';
kubectl get replicasets -ojson -A | jq -c '.items[]';
kubectl get daemonsets -ojson -A | jq -c '.items[]';
) > /tmp/cluster.jsonl
$ cat /tmp/cluster.jsonl \
| jq -r '
def parse_into_parts:
. as $i
|capture(
"^((?<host>[a-zA-Z0-9-]+\\.[a-zA-Z0-9.-]+)/)?"
+ "(:(?<port>[0-9]+))?"
+ "((?<path>[a-zA-Z0-9-._/]+)/)?"
+ "(?<image>[a-zA-Z0-9-._]+)"
+ "((:(?<tag>[a-z0-9_.-]+))|(@(?<digest>sha256:[a-z0-9]+)))?$"
) // error("couldnt parse \($i)");
def qualify_oci_image:
if (.host==null) then .host="docker.io" end
|if (.path==null and .host=="docker.io") then .path="library" end
# |if (.tag==null and .digest==null) then .tag="latest" end
;
def glue_parts:
[
if (.host) then .host else "" end,
if (.port) then ":\(.port)" else "" end,
if (.host) then "/" else "" end,
if (.path) then "\(.path)/" else "" end,
.image,
if (.digest) then "@\(.digest)" elif (.tag) then ":\(.tag)" else "" end
]|join("")
;
def fix_oci_image:
. as $i
|parse_into_parts
|qualify_oci_image
|if (.path=="bitnami") then .path="bitnamilegacy" else . end
|if (.host=="docker.io") then (.host="123456780123.dkr.ecr.us-east-1.amazonaws.com"|.path="docker-io/\(.path)") else . end
|glue_parts;
[
..|objects|(.initContainers[]?,.containers[]?)
|(.image|fix_oci_image) as $newImage
|select(.image!=$newImage)
|"\(.name)=\($newImage)"
] as $p
|select($p|length > 0)
|"kubectl set image \(.kind) -n \(.metadata.namespace) \(.metadata.name) \($p|join(" "))"
Permanent Mirror Configuration for containerd
(
# patch /etc/containerd/config.toml for automatically picking dockerhub mirror
containerd_config_version="$(grep -oP '^\s*version\s*=\s*\K\d+' /etc/containerd/config.toml)"
p=""
case "$containerd_config_version" in
2) p="io.containerd.grpc.v1.cri";;
3) p="io.containerd.cri.v1.images";;
*) echo "unsupported"; return;;
esac
cat <<-EOM >> /etc/containerd/config.d/dockerhub-mirrors.toml
[plugins]
[plugins."$p".registry]
[plugins."$p".registry.mirrors]
[plugins."$p".registry.mirrors."docker.io"]
endpoint = [
"public.ecr.aws/docker",
"mirror.gcr.io",
"gitlab.com/acme-org/dependency_proxy/containers",
"123456789012.dkr.ecr.us-east-1.amazonaws.com/docker-io",
"docker.io",
]
[plugins."$p".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."gitlab.com".auth]
# https://gitlab.com/groups/acme-org/-/settings/access_tokens?page=1
username = "dependency-proxy"
password = "glpat-XXXXXXXXXXXXXXXXXXXX"
[plugins."$p".registry.configs."docker.io".auth]
username = "acme-org"
password = "dckr_pat_3Xi_XXXXXXXXXXXXXXXXXXXXXXX"
auth = "dckr_pat_3Xi_XXXXXXXXXXXXXXXXXXXXXXX"
EOM
fi
)
if ! containerd config dump 1>/dev/null; then
echo "exiting since containerd config is bad" >&2
exit 1
fi
How to get AWS-CLI v2 down from 127M to 67M
Follow these steps:
|
|
Notes
I do think there can be much more improved in botocore (like compressing assets) or just rewritting AWS CLI to Golang.
Techstack n - 1 is dead!
TL;DR TechStack n-1 is dead. It ended with the rise of the clouds and software release cycles going down to weeks due to containerized CIs.
Against ‘it’s stable and mature so let it run’
Beeing OpenSource-based, Ubuntu already had the concept of point releases every 6 months when the Docker and K8s hit the world and gave automated CIs a big boost in making system containers. Some years after Docker itself switched to a 3-month release cycle. So did the Linux Kernel with 2-3 months. Firefox 4-weeks.
AWS CLI V2 is generally available since February 2020. BUT…
The AWS CLI V2 is distributed as a binary package, but this new distribution method comes with a few caveats.
It is distributed as a binary package (built on Python with PyInstaller that has bundled native libs) BUT…
- Problem #1 this requires a decent GLIBC and probably won’t run on older Red Hat distributions.
- Problem #2 as it turns out, many people use the official docker images to build their images and push to AWS ECR. And since these images are based on Alpine and therefore use MUSL and NOT GLIBC, it simply won’t run (but well, people have seen Alpine pipelines in AWS builds now, although they do NOT support any other Linux than their own; probably THEY HAVE TO).
- Problem #3 you now have to download the application to update it (hey, the way to install software on Windows or Mac? But hey, an own updater with
aws/install --update). - Problem #4 forget installing and updating from pip for now; well, you can directly pip from the repo, BUT it’s tricky. Although the version says 2.0.38 (no beta or alpha!) it still requires a
botocore==2.0.0dev42. I guess they should suffix a “ga” for “General Availability”. - Problem #5 If you belong to the majority (or the people with broken python installations) please refrain from saying “I have no problems! It’s better now!”
Anyway, they have some nice features now, and it seems they packed aws-shell right into the CLI:
https://aws.amazon.com/blogs/developer/aws-cli-v2-is-now-generally-available/
https://www.youtube.com/watch?v=U5y7JI_mHk8
Docker on Windows: CIFS v3.02 mounts failing with big file count
Oh, well I love Docker and Windows – NOT! Another issue: https://github.com/docker/for-win/issues/2285
Description
We use containers for our developer environments. The projects are built with tools shipped within the containers. While that works like a charm for OSX and Linux, we face problems with some containers on Windows which have a heavy load of source files. The project directory is mounted to the container by CIFS 3.02.
As workaround we have found out, that builds do work with CIFS 2.0, but not with 2.1 or 3.02 mounts. If the build is failing it complains about “File not found”. Yet the file is there and can be read!
Using confd and consul for config management
This tutorial demonstrates how to use confd and consul to manage application configurations. consul acts as a key/value store for your configuration data, while confd uses templates to generate configuration files based on the values in consul. This approach helps to decouple configuration from your application code.
Consul is used in clouds as a key/value store to hold configs, just like etcd, Zookeeper, AWS SSM, etc. Additionally, consul can provide a DNS server for service discovery.