Archives

Multi, Mono, Meta, Manifest – Composite Repository?

There was this discussion about whether to use Mono- or Multi-Repositories? I won’t pick it up again.

Some cool people suggested: why not use the best of both worlds and use meta-repos?!

I Was Interested in “Meta-Repo” Tooling

I asked myself how these tools would solve problems and gave some of them a try:

• There are multi-repository management tools that work by tagging or grouping repos, like gr or mu-repo.
• Tools that just do subdirectory iteration over your repos, like gitbatch.
• Many git-extras repositories with subdirectory iteration scripts that do the same, like git multi (the one I am using ;)).
• Tools which combine multi-repo manifests with different VCS systems, like myrepos (old and mature, often found within your Linux system package management).
• Tools that try to standardize the directory layout for managing your repositories, like ghq (you should definitely use such a layout!).
• Tools which basically reassemble what git submodule does, like mu-repo, git-metarepo, or meta …

How to get AWS-CLI v2 down from 127M to 67M

Follow these steps:

 0
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

FROM alpine:3.12

# 1. Install glibc compatibility for Alpine
RUN apk --no-cache add binutils \
    && echo "Getting libc libraries" \
    && curl -sL https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub -o /etc/apk/keys/sgerrand.rsa.pub \
    && curl -sLO https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VER}/glibc-${GLIBC_VER}.apk \
    && curl -sLO https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VER}/glibc-bin-${GLIBC_VER}.apk \
    && echo "Installing libc libraries" \
    && apk add --no-cache \
        glibc-${GLIBC_VER}.apk \
        glibc-bin-${GLIBC_VER}.apk

# 2. Install rush parallel runner (temporary)
# This is used for optimizing the botocore data later.
RUN echo "Installing rush parallel runner …

Microsoft WSL2 kernel modifications

If you want to dig into: for now it seems all to be HyperV related. I maybe wrong since I haven’t reviewed the code itself.

It’s based on the next 5.4.x kernel - probably since Ubuntu Focal also has 5.4 on LTS.

git clone --depth 1 --branch v5.4.51 https://github.com/gregkh/linux.git upstream & pid1=$!
git clone --depth 1 --branch linux-msft-5.4.51  https://github.com/microsoft/WSL2-Linux-Kernel.git wsl2 & pid2=$!
wait $pid1 $pid2

File differences

diff -qr --exclude=.git upstream wsl2 | tee diff.txt
cat  \
    <(cat diff.txt | grep -oP '(Files upstream/)\K[^ ]+') \
    <(cat diff.txt | grep -oP '(Only in wsl2/)\K.+' | sed 's|: |/|g') \
    | sort -u \
    | (while read f; do if [[ -f "wsl2/$f" ]]; then echo "$f"; fi; done;) \
| tee files.txt

Get all commits on diffed files

Improvement: Diff the original commit list with WSL source

Techstack n - 1 is dead!

TL;DR TechStack n-1 is dead. It ended with the rise of the clouds and software release cycles going down to weeks due to containerized CIs.

Against ‘it’s stable and mature so let it run’

Beeing OpenSource-based, Ubuntu already had the concept of point releases every 6 months when the Docker and K8s hit the world and gave automated CIs a big boost in making system containers. Some years after Docker itself switched to a 3-month release cycle. So did the Linux Kernel with 2-3 months. Firefox 4-weeks.

Download an LFS backed file from GitLab.com without git and git-lfs installed

It is possible to download a Git LFS-backed file from GitLab.com without having git or git-lfs installed by using the GitLab API directly. This article provides two shell scripts that demonstrate how to do this.

Well, the API is there and you can do it already!

Just dig into what git is doing by a test-clone with any LFS repo:

export GIT_CURL_VERBOSE=1
export GIT_TRACE_CURL=1
git clone <my-repo> 2>&1 | tee git-clone.log

From there you are able to figure out what is happening on SSH.

AWS CLI V2 is generally available since February 2020. BUT…

The AWS CLI V2 is distributed as a binary package, but this new distribution method comes with a few caveats.

It is distributed as a binary package (built on Python with PyInstaller that has bundled native libs) BUT…

• Problem #1 this requires a decent GLIBC and probably won’t run on older Red Hat distributions.
• Problem #2 as it turns out, many people use the official docker images to build their images and push to AWS ECR. And since these images are based on Alpine and therefore use MUSL and NOT GLIBC, it simply won’t run (but well, people have seen Alpine pipelines in AWS builds now, although they do NOT support any other Linux than their own; probably THEY HAVE TO).
• Problem #3 you now have to download the application to update it (hey, the way to install software on Windows or Mac? But hey, an own updater with aws/install --update).
• Problem #4 forget installing and updating from pip for now; well, you can directly pip from the repo, BUT it’s tricky. Although the …

#blacklivesmatter vs. a git master

Ich sehe es so: Systemischer Rassismus* führte dazu, dass eine heterogene, aber ethnische Gruppe von Menschen aufgrund ihrer Hautfarbe benachteiligt wurde. Im Laufe der Zeit wurde das Ungleichgewicht durch all diese kleinen Benachteiligungen so stark, dass nun, informell und formell, eine Ausbalancierung stattfindet – quasi als „The White Man’s Legacy“.

Leider wird dabei oft wieder gegen die falschen Ziele geschossen oder über das Ziel hinausgegangen: Gerade in unserer Zeit der Selbstradikalisierung ist es besonders einfach, sich mit dem Ziel, „ein Unrecht zu korrigieren“ (correct a wrong), zu profilieren. Wie oft in der Politik zählt ohnehin nur noch das „moralisch Korrekte“, da sich damit am einfachsten uninformierte Follower rekrutieren lassen. Ethisch gerechte und rationale Entscheidungen treten in den Hintergrund.

5¢ on YAML in the DevOps world

YAML Fatigue is Real: Are We Forgetting the “Why” Behind Declarative Config?

Lately, it feels like everywhere you look, there’s a YAML wrapper.

Often, a simple envsubst would suffice. I see many apps and frameworks that – let’s be honest – often just feel like a slightly polished YAML layer over a simple API call. And then there are the countless YAML-based Task Runner projects that are trying so hard to be the next-gen CI configuration (taskctl?… 😬).

CSV tools

Here are some CSV utilities that may come in handy since most teams deal with this format.

• daff - part of my default toolset ;)
• csvq - part of my default toolset ;)
• tsv-utils - interesting read
• xsv - yeah, the ripgrep guy
• gron - now part of my default toolset (yeah, json but faster than jq with “flatten” filter; line grep is powerful)
• coopy - patch/diff for CSV and spreadsheet data
• rq - in trial

‘Ternary Operator’ with JQ: Checking for Empty Values

As you noticed I use JQ commonly like RegExp. I recently used select(.!=null) to filter for non-null values. Turns out I missed something in docs:

jq -nc '{a:1},[1,23],true,null,42|values'

OUTPUT

{"a":1}
[1,23]
true
42

jq -nc '{a:1},[1,23],true,null,42|iterables'

OUTPUT

{"a":1}
[1,23]

jq -nc '{a:1},[1,23],true,null,42|nulls'

OUTPUT

null

Now that can be used to set defaults:

jq -nc '.notset?|values // "default"'

OUTPUT

"default"

$ jq -nc '.notset|values // "default"'

OUTPUT