Devops
Help! We’ve ran into a DockerHub rate limit!
About
Yes, it is still happining. In 2025! Here you will find:
- Podman Dockerhub Mirror Configuration
- K8s Quickfix: Rewriting Existing K8s Resources
- Permanent Mirror Configuration for
containerd - K8s Admission Webhook to do the same
Podman Dockerhub Mirror Configuration
~/.config/containers/registries.conf.d/dockerhub-mirror.conf:
[[registry]]
prefix = "docker.io"
insecure = false
blocked = false
location = "public.ecr.aws/docker"
[[registry.mirror]]
location = "mirror.gcr.io"
[[registry.mirror]]
location = "gitlab.com/acme-org/dependency_proxy/containers"
[[registry.mirror]]
location = "registry-1.docker.io"
[[registry.mirror]]
location = "123456789012.dkr.ecr.us-east-1.amazonaws.com/docker-io"
I hope you are using ecr-login for your ECR registries ;)
export REGISTRY_AUTH_FILE=$HOME/.config/containers/auth.json
{
"auths": {
"docker.io": {
"auth": "eGw4ZGVwXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXem40VQ=="
},
"gitlab.com": {
"auth": "cmVXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXSYQ=="
},
"registry.gitlab.com": {
"auth": "cmVXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXSYQ=="
}
},
"credHelpers": {
"*": "",
"123456789012.dkr.ecr.us-east-1.amazonaws.com": "ecr-login",
"345678901234.dkr.ecr.us-east-1.amazonaws.com": "ecr-login"
}
}
K8s Quickfix: Rewriting Existing K8s Resources
$ cd $(mktemp -d)
$ (
kubectl get pods --field-selector=status.phase=Pending -A -ojson | jq -c '.items[]';
kubectl get deployments -ojson -A | jq -c '.items[]';
kubectl get replicasets -ojson -A | jq -c '.items[]';
kubectl get daemonsets -ojson -A | jq -c '.items[]';
) > /tmp/cluster.jsonl
$ cat /tmp/cluster.jsonl \
| jq -r '
def parse_into_parts:
. as $i
|capture(
"^((?<host>[a-zA-Z0-9-]+\\.[a-zA-Z0-9.-]+)/)?"
+ "(:(?<port>[0-9]+))?"
+ "((?<path>[a-zA-Z0-9-._/]+)/)?"
+ "(?<image>[a-zA-Z0-9-._]+)"
+ "((:(?<tag>[a-z0-9_.-]+))|(@(?<digest>sha256:[a-z0-9]+)))?$"
) // error("couldnt parse \($i)");
def qualify_oci_image:
if (.host==null) then .host="docker.io" end
|if (.path==null and .host=="docker.io") then .path="library" end
# |if (.tag==null and .digest==null) then .tag="latest" end
;
def glue_parts:
[
if (.host) then .host else "" end,
if (.port) then ":\(.port)" else "" end,
if (.host) then "/" else "" end,
if (.path) then "\(.path)/" else "" end,
.image,
if (.digest) then "@\(.digest)" elif (.tag) then ":\(.tag)" else "" end
]|join("")
;
def fix_oci_image:
. as $i
|parse_into_parts
|qualify_oci_image
|if (.path=="bitnami") then .path="bitnamilegacy" else . end
|if (.host=="docker.io") then (.host="123456780123.dkr.ecr.us-east-1.amazonaws.com"|.path="docker-io/\(.path)") else . end
|glue_parts;
[
..|objects|(.initContainers[]?,.containers[]?)
|(.image|fix_oci_image) as $newImage
|select(.image!=$newImage)
|"\(.name)=\($newImage)"
] as $p
|select($p|length > 0)
|"kubectl set image \(.kind) -n \(.metadata.namespace) \(.metadata.name) \($p|join(" "))"
Permanent Mirror Configuration for containerd
(
# patch /etc/containerd/config.toml for automatically picking dockerhub mirror
containerd_config_version="$(grep -oP '^\s*version\s*=\s*\K\d+' /etc/containerd/config.toml)"
p=""
case "$containerd_config_version" in
2) p="io.containerd.grpc.v1.cri";;
3) p="io.containerd.cri.v1.images";;
*) echo "unsupported"; return;;
esac
cat <<-EOM >> /etc/containerd/config.d/dockerhub-mirrors.toml
[plugins]
[plugins."$p".registry]
[plugins."$p".registry.mirrors]
[plugins."$p".registry.mirrors."docker.io"]
endpoint = [
"public.ecr.aws/docker",
"mirror.gcr.io",
"gitlab.com/acme-org/dependency_proxy/containers",
"123456789012.dkr.ecr.us-east-1.amazonaws.com/docker-io",
"docker.io",
]
[plugins."$p".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."gitlab.com".auth]
# https://gitlab.com/groups/acme-org/-/settings/access_tokens?page=1
username = "dependency-proxy"
password = "glpat-XXXXXXXXXXXXXXXXXXXX"
[plugins."$p".registry.configs."docker.io".auth]
username = "acme-org"
password = "dckr_pat_3Xi_XXXXXXXXXXXXXXXXXXXXXXX"
auth = "dckr_pat_3Xi_XXXXXXXXXXXXXXXXXXXXXXX"
EOM
fi
)
if ! containerd config dump 1>/dev/null; then
echo "exiting since containerd config is bad" >&2
exit 1
fi
jq - The Swiss Army knife of JSON (2023)
web | pdf | pptx | google slides
Techstack n - 1 is dead!
TL;DR TechStack n-1 is dead. It ended with the rise of the clouds and software release cycles going down to weeks due to containerized CIs.
Against ‘it’s stable and mature so let it run’
Beeing OpenSource-based, Ubuntu already had the concept of point releases every 6 months when the Docker and K8s hit the world and gave automated CIs a big boost in making system containers. Some years after Docker itself switched to a 3-month release cycle. So did the Linux Kernel with 2-3 months. Firefox 4-weeks.
5¢ on YAML in the DevOps world
YAML Fatigue is Real: Are We Forgetting the “Why” Behind Declarative Config?
Lately, it feels like everywhere you look, there’s a YAML wrapper.
Often, a simple envsubst would suffice. I see many apps and frameworks that – let’s be honest – often just feel like a slightly polished YAML layer over a simple API call. And then there are the countless YAML-based Task Runner projects that are trying so hard to be the next-gen CI configuration (taskctl?… 😬).
Headbanging to Docker (Workshop 2020)
Thought Experiment: Automation in Big Corporate
Thought Experiment: Automation in a Big Corporate Environment
Goal: We want to transition from owning our own fleet of cars to using a taxi service as a utility (service provider).
However, we don’t know enough about the taxi business and feel uncertain. Therefore, our first step is to hire drivers to operate our existing cars as makeshift taxis. These drivers, however, are not trained taxi drivers, they don’t know the routes, and the navigation system is a proprietary product with outdated maps. It comes from a reputable manufacturer, which is useless in practice. To keep them awake and encourage them to learn the routes, we also provide our drivers with coffee from a well-known, premium coffee chain.
Infojunk October 2018
This is a collection of interesting links and resources I came across in October 2018, covering a wide range of topics including browser extensions, collaborative coding, Linux, AWS, and more.
Browser Extensions
- I don’t care about cookies
- Imagus or HoverZoom+ to enlarge images on mouse over (don’t use HoverZoom since it’s a data hog).
- Amazon Infinite Scroll
Collaborative Coding
Focusing on IDEs. Web-based solutions are mostly ignored.
- Floobits - IntelliJ, SublimeText, Atom, vscode-plugin in the works
- CodeStream - the new and fancy one
- tmate - terminal sharing over tmux
- Visual Studio Live Share - Visual Studio Code
- AWS Cloud9 - coding for the cloud
- ScreenHero - h264 video streaming and now bought by Slack.io, lacking Linux support
Linux
- Use Chromium to have Hardware Acceleration in your YouTube Videos - don’t forget to install the h264ify browser extension to force h264 (Chrome chooses VP9 by default which is currently not accelerated).
- Touchpad Gestures for Gnome
- There are no GTK3 themes! Remove theming support?
NodeJS
DevOps
AI/MachineLearning
AWS
JmesPath is not as powerful as jq, but Amazon AWS probably chose it since it might be faster and the query-selectors are a bit more sophisticated (?).
Docker for Developers (2018)
Embracing the Cloud (2018)
Using confd and consul for config management
This tutorial demonstrates how to use confd and consul to manage application configurations. consul acts as a key/value store for your configuration data, while confd uses templates to generate configuration files based on the values in consul. This approach helps to decouple configuration from your application code.
Consul is used in clouds as a key/value store to hold configs, just like etcd, Zookeeper, AWS SSM, etc. Additionally, consul can provide a DNS server for service discovery.