Devops

Help! We’ve ran into a DockerHub rate limit!

About

Yes, it is still happining. In 2025! Here you will find:

Podman Dockerhub Mirror Configuration

~/.config/containers/registries.conf.d/dockerhub-mirror.conf:

[[registry]]                                                                                              
prefix = "docker.io"                                                      
insecure = false                                                                              
blocked = false
location = "public.ecr.aws/docker"  

[[registry.mirror]]
location = "mirror.gcr.io"

[[registry.mirror]]
location = "gitlab.com/acme-org/dependency_proxy/containers"

[[registry.mirror]]
location = "registry-1.docker.io"                                                              

[[registry.mirror]]
location = "123456789012.dkr.ecr.us-east-1.amazonaws.com/docker-io"

I hope you are using ecr-login for your ECR registries ;)

export REGISTRY_AUTH_FILE=$HOME/.config/containers/auth.json
{
  "auths": {
    "docker.io": {
      "auth": "eGw4ZGVwXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXem40VQ=="
    },
    "gitlab.com": {
      "auth": "cmVXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXSYQ=="
    },
    "registry.gitlab.com": {
      "auth": "cmVXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXSYQ=="
    }
  },
  "credHelpers": {
    "*": "",
    "123456789012.dkr.ecr.us-east-1.amazonaws.com": "ecr-login",
    "345678901234.dkr.ecr.us-east-1.amazonaws.com": "ecr-login"
  }
}

K8s Quickfix: Rewriting Existing K8s Resources


$ cd $(mktemp -d)

$ (
  kubectl get pods --field-selector=status.phase=Pending -A -ojson | jq -c '.items[]';
  kubectl get deployments -ojson -A | jq -c '.items[]';
  kubectl get replicasets -ojson -A | jq -c '.items[]';
  kubectl get daemonsets -ojson -A | jq -c '.items[]';
) > /tmp/cluster.jsonl

$ cat /tmp/cluster.jsonl \
  | jq -r '
    def parse_into_parts:
      . as $i
      |capture(
        "^((?<host>[a-zA-Z0-9-]+\\.[a-zA-Z0-9.-]+)/)?"  
        + "(:(?<port>[0-9]+))?"
        + "((?<path>[a-zA-Z0-9-._/]+)/)?"
        + "(?<image>[a-zA-Z0-9-._]+)"
        + "((:(?<tag>[a-z0-9_.-]+))|(@(?<digest>sha256:[a-z0-9]+)))?$"
      ) // error("couldnt parse \($i)");

    def qualify_oci_image:
      if (.host==null) then .host="docker.io" end
      |if (.path==null and .host=="docker.io") then .path="library" end
      # |if (.tag==null and .digest==null) then .tag="latest" end
      ;

    def glue_parts:
      [
        if (.host) then .host else "" end,
        if (.port) then ":\(.port)" else "" end,
        if (.host) then "/" else "" end,
        if (.path) then "\(.path)/" else "" end,
        .image,
        if (.digest) then "@\(.digest)" elif (.tag) then ":\(.tag)" else "" end
      ]|join("")
      ;

    def fix_oci_image:
      . as $i
      |parse_into_parts
      |qualify_oci_image
      |if (.path=="bitnami") then .path="bitnamilegacy" else . end
      |if (.host=="docker.io") then (.host="123456780123.dkr.ecr.us-east-1.amazonaws.com"|.path="docker-io/\(.path)") else . end
      |glue_parts;
    
    [
      ..|objects|(.initContainers[]?,.containers[]?)
      |(.image|fix_oci_image) as $newImage
      |select(.image!=$newImage)
      |"\(.name)=\($newImage)"
    ] as $p
    |select($p|length > 0)
    |"kubectl set image \(.kind) -n \(.metadata.namespace) \(.metadata.name) \($p|join(" "))"
    

Permanent Mirror Configuration for containerd

(
	# patch /etc/containerd/config.toml for automatically picking dockerhub mirror

	containerd_config_version="$(grep -oP '^\s*version\s*=\s*\K\d+' /etc/containerd/config.toml)"
	p=""
	case "$containerd_config_version" in
		2) p="io.containerd.grpc.v1.cri";;
		3) p="io.containerd.cri.v1.images";;
		*) echo "unsupported"; return;;
	esac
	cat <<-EOM >> /etc/containerd/config.d/dockerhub-mirrors.toml
[plugins]

  [plugins."$p".registry]

    [plugins."$p".registry.mirrors]

      [plugins."$p".registry.mirrors."docker.io"]
        endpoint = [
          "public.ecr.aws/docker",
          "mirror.gcr.io",
          "gitlab.com/acme-org/dependency_proxy/containers",
          "123456789012.dkr.ecr.us-east-1.amazonaws.com/docker-io",
          "docker.io",
        ]

    [plugins."$p".registry.configs]
      [plugins."io.containerd.grpc.v1.cri".registry.configs."gitlab.com".auth]
      	# https://gitlab.com/groups/acme-org/-/settings/access_tokens?page=1
        username = "dependency-proxy"
        password = "glpat-XXXXXXXXXXXXXXXXXXXX"

      [plugins."$p".registry.configs."docker.io".auth]
        username = "acme-org"
        password = "dckr_pat_3Xi_XXXXXXXXXXXXXXXXXXXXXXX"
        auth = "dckr_pat_3Xi_XXXXXXXXXXXXXXXXXXXXXXX"
EOM		
	fi
)

if ! containerd config dump 1>/dev/null; then
   echo "exiting since containerd config is bad" >&2
   exit 1
fi

jq - The Swiss Army knife of JSON (2023)

web | pdf | pptx | google slides

Techstack n - 1 is dead!

TL;DR TechStack n-1 is dead. It ended with the rise of the clouds and software release cycles going down to weeks due to containerized CIs.

Against ‘it’s stable and mature so let it run’

Death of Sophocles
The Death of Sophocles (Creative Commons)

Beeing OpenSource-based, Ubuntu already had the concept of point releases every 6 months when the Docker and K8s hit the world and gave automated CIs a big boost in making system containers. Some years after Docker itself switched to a 3-month release cycle. So did the Linux Kernel with 2-3 months. Firefox 4-weeks.

5¢ on YAML in the DevOps world

YAML Fatigue is Real: Are We Forgetting the “Why” Behind Declarative Config?

Lately, it feels like everywhere you look, there’s a YAML wrapper.

Often, a simple envsubst would suffice. I see many apps and frameworks that – let’s be honest – often just feel like a slightly polished YAML layer over a simple API call. And then there are the countless YAML-based Task Runner projects that are trying so hard to be the next-gen CI configuration (taskctl?… 😬).

Headbanging to Docker (Workshop 2020)

Thought Experiment: Automation in Big Corporate

Thought Experiment: Automation in a Big Corporate Environment

Goal: We want to transition from owning our own fleet of cars to using a taxi service as a utility (service provider).

However, we don’t know enough about the taxi business and feel uncertain. Therefore, our first step is to hire drivers to operate our existing cars as makeshift taxis. These drivers, however, are not trained taxi drivers, they don’t know the routes, and the navigation system is a proprietary product with outdated maps. It comes from a reputable manufacturer, which is useless in practice. To keep them awake and encourage them to learn the routes, we also provide our drivers with coffee from a well-known, premium coffee chain.

Infojunk October 2018

This is a collection of interesting links and resources I came across in October 2018, covering a wide range of topics including browser extensions, collaborative coding, Linux, AWS, and more.

Browser Extensions

Collaborative Coding

Focusing on IDEs. Web-based solutions are mostly ignored.

Linux

NodeJS

DevOps

AI/MachineLearning

AWS

JmesPath is not as powerful as jq, but Amazon AWS probably chose it since it might be faster and the query-selectors are a bit more sophisticated (?).

Docker for Developers (2018)

Embracing the Cloud (2018)

Using confd and consul for config management

This tutorial demonstrates how to use confd and consul to manage application configurations. consul acts as a key/value store for your configuration data, while confd uses templates to generate configuration files based on the values in consul. This approach helps to decouple configuration from your application code.

Consul is used in clouds as a key/value store to hold configs, just like etcd, Zookeeper, AWS SSM, etc. Additionally, consul can provide a DNS server for service discovery.