Multi, Mono, Meta, Manifest 鈥 Composite Repository?

Mono or Multiple-Repositories? Meta-Repos!

There was this discussion about whether to do Mono or Multiple-Repositories? I won鈥檛 pick it up again.

Some cool guys that suggested: why not use the best of the two world and use meta-repos?!

I was interested in 鈥渕eta-repo鈥 tooling

I asked myself how they would solve problems and gave soem of them a try:

  • there are multi-repository management tools that work with by tagging or grouping repos like gr or mu-repo
  • tools that just do subdirectory iteration over your repos like gitbatch and 鈥
  • many git-extras repisitories with subdirectory iteration scripts that do the same like git multi – the one i am using 馃槈
  • tools which combine multi-repo manifests with different VCS systems like myrepos 鈥 old and mature, within your linux system package management
  • tools that try to standardize the directory layout for managing your reositories like ghq 鈥 definitly you should use such layout!
  • tools which basically reassemble what git submodule does like mu-repo or git-metarepo or meta (see below)
  • tools that just checkout all your org鈥檚 repositories from GitHub, GitLab or BitBucket like ghorg
  • and git itself, who was not lazy and added git for-each-repo which also does simple iteration (experimental status!)
  • and then there was git-slave where it鈥檚 master deceided it was an anachronistic project and decided to give it up

In the end most tools mentioned here do not focus on workflow problems rather than introducing new manifests or config files 馃檨

But basicallygit submodule is a meta-repo since we have .submodules!

Yes, that鈥檚 aboslutly valid. .submodules basically behaves like a multiple repository manifest (and you can set submodules also to active:false)

  • You can use git submodule foreach to create/commit/manage multiple branches
  • You could just setup a workspace or manifest repository that reassembles what other tools do

But isn鈥檛 there more to it?

  • There-repo are tools like repo which is used by Android ecosystem or the Alibaba Golang rewrite git-repo-go

  • They are using a manifest repository as well but you don鈥檛 need you to check out everything and it has an efficient way of managing dependencies
    (Android OS has 2000 repositories with repos different devices and drivers!)

  • They add pre-/post-process like symlinking or comamnd execution over mutliple repisitories (just like myrepos )
  • They introduce the concept of a topic (your current task) and solve workflow problems like having to change several sources in different repositories for one feature, diffing over mutliple repisitories and automatically creating a code review (using gerrit)

Let鈥檚 redefine our problem!

We want git submodules and an umbrella to add some workflow features wouldn鈥檛 be that wrong?

I did a search and came up with git-meta (not to be mistaken by other projects of the same name). They also have a nyce presention that bring up the mist important thoughts (WATCH IT!)

Unfortunatly GitLab docs did not return results on how to better solve where to put merge requests on meta-repos etc.

And then there is the more established meta which has 1.4K stars (only?) . But they try to reinvent .submodules and add a ton of (un)related features with like mono-meta-repo project migration. And they even use .metain their own repository to defined plugin-dependencies #facepalm.

The main issues on meta-repos are still GitLab and how to integrate your workflow (like multiple merge requests; microservices should be decoupled and should not break!).

I don鈥檛 know what tool is better – try it out! I just say i would use

So .submodules alone make a repository a meta-repository?

Yes and no 鈥 when you treat don鈥檛 treat dependencies like 3rd party libraries but an actual composite of multi-repos you need to work on.

Meta-Repos are a workaround for big too-big-to-be-fun mono-repositories and a half-baked solution for composite workspaces (like needed for microservices) since needed umbrella workflows are not integrated into GitHub or GitLab. If the problem doesn鈥檛 hurt don鈥檛 try to solve it since you create new problems. You can still use Virtual Workspaces like linking repo folders in your IDE or via symlink. Just try to stay focused on one task before having multiple workspaces open at the same time.

  • Multi-Pipeline setup can be simplified if using a meta-repo (or submodules)
  • Each affected service (repository) on your working topic will have a merge-request
  • .submodulesshould have CI checks (like is every repository-branch correctly set in .submodules ?)

Is 鈥榤eta-repo鈥 a buzzword?

Yes. Basically it鈥檚 multi-repo and not much new except a new view on the workflow issues of having a composite repos.

Still you can use git subtreein your project but keep in mind when a repository is shared among many projects/workspaces it may diverge since upstreams are treated as second-class citizen and it鈥檚 harder to maintain to branch from dev on new features.

But then again: git-meta say鈥檚 in their project description:

Repository for the git-meta project — build your own monorepo using Git submodules

Multi, Mono, Meta – Composite Repo?

What now?

Further Readings

How to get AWS-CLI v2 down from 127M to 67M

Take these steps:

FROM alpine:3.12

# install glibc compatibility for alpine
RUN apk --no-cache add binutils \
    && echo "getting libc libraries" \
    && curl -sL -o /etc/apk/keys/ \
    && curl -sLO${GLIBC_VER}/glibc-${GLIBC_VER}.apk \
    && curl -sLO${GLIBC_VER}/glibc-bin-${GLIBC_VER}.apk \
    && echo "installing libc libraries" \
    && apk add --no-cache \
        glibc-${GLIBC_VER}.apk \
        glibc-bin-${GLIBC_VER}.apk \
    && echo "installing of rush parallel runner (temporary)" \
    && curl -sSfL  -o - \
        | tar -C /tmp/ -zxf - \
    && echo "installing AWS CLI" \
    && curl -sL -o \
    && unzip \
        -x 'aws/dist/awscli/examples/**' 'aws/dist/docutils/**'\
    && aws/install  \
    && echo "Cleaning and sliming AWS CLI installation" \
    && rm -rvf \ \
        aws \
        /usr/local/aws-cli/v2/*/dist/aws_completer \
        /usr/local/aws-cli/v2/*/dist/awscli/data/ac.index \
        /usr/local/aws-cli/v2/*/dist/awscli/examples \
    && find /usr/local/aws-cli/v2/current/dist/botocore/data/ -name "*.json" -name "examples*" -exec rm {} \; \
    && find /usr/local/aws-cli/v2/current/dist/botocore/data/ -name "*.json" \
        | /tmp/rush 'echo "optimizing {}"; jq -cer "del(.. | .documentation?)" "{}" > "{}.tmp" && mv "{}.tmp" "{}"' \
    && echo "Removing artificats" \
    && rm /tmp/rush \
    && apk --no-cache del \
        binutils \
    && rm glibc-${GLIBC_VER}.apk \
    && rm glibc-bin-${GLIBC_VER}.apk \
    && rm -rf /var/cache/apk/*

I do think there can be much more improved in botocore (like compressing assets) or just rewritting AWS CLI to Golang. I hope if an v3 hits the floor it is in Golang (for the easy of use; go sdk is already very powerful and maybe there is an alternative client – but would be fine if that is 100% compatible to AWS CLI).

“Ternary operator” with JQ: Checking for empty values

As you noticed I use JQ commonly like RegExp. I recently used select(.!=null) to filter for non-null values. Turns out I missed something in docs:

jq -nc '{a:1},[1,23],true,null,42|values'


jq -nc '{a:1},[1,23],true,null,42|iterables'


jq -nc '{a:1},[1,23],true,null,42|nulls'



Now that can be used to set defaults:

jq -nc '.notset?|values // "default"'


$ jq -nc '.notset|values // "default"'



Or just use it to do some little validation on expected input:

jq -nc '.notset|error("has no value")'


jq: error (at <unknown>): has no value
jq -nc '{foo:"bar"}|(.baz|values // error("baz not set")|.'


jq: error (at <unknown>): has no value
jq -nc '{foo:"bar"}|(.baz|values // error("baz not set")) as $e|.'
jq: error (at <unknown>): baz not set
jq -nc '{foo:"bar"}|(.foo|values // error("foo not set")) as $e|.'


jq -nc 'def chkkey(k): .[k]|values // error(k + "not set"), {foo:"bar"}|(.foo|values // error("foo is empty")) as $e|.'


jq -nc 'def notempty(k): .[k]|values // error(k + " is empty or not set"); {foo:"bar"}|notempty("foo")'


jq -nc 'def notempty(k): .[k]|values // error(k + " is empty or not set"); {baz:"bar"}|notempty("foo")'


jq: error (at <unknown>): foo is empty or not set

Let’s turn it into a helper function:

jq -nc 'def notempty(k):
        . as $in
            (k|arrays // [k])[] as $k
            |$in[$k]|values // error($k + " is empty or not set")
        ) as $x


jq -nc 'def notempty(k):
        . as $in
            (k|arrays // [k])[] as $k
            |$in[$k]|values // error($k + " is empty or not set")
        ) as $x
        {foo:"bar", baz: "mar"}|notempty(["meer"])'


jq: error (at <unknown>): meer is empty or not set

Microsoft WSL2 kernel modifications

If you want to dig into: for now it seems all to be HyperV related. I maybe wrong since I haven’t reviewed the code itself.

It’s based on the next 5.4.x kernel – probably since Ubuntu Focal also has 5.4 on LTS.

git clone --depth 1 --branch v5.4.51 upstream & pid1=$!
git clone --depth 1 --branch linux-msft-5.4.51 wsl2 & pid2=$!
wait $pid1 $pid2
File differences
diff -qr --exclude=.git upstream wsl2 | tee diff.txt
cat  \
    <(cat diff.txt | grep -oP '(Files upstream/)\K[^ ]+') \
    <(cat diff.txt | grep -oP '(Only in wsl2/)\K.+' | sed 's|: |/|g') \
    | sort -u \
    | (while read f; do if [[ -f "wsl2/$f" ]]; then echo "$f"; fi; done;) \
| tee files.txt
Get all commits on diffed files

Improvement: Diff the original commit list with WSL source

cd wsl2
git fetch origin
git pull --unshallow 
cat ../files.txt \
    | xargs -I @ git log --follow --pretty="%h - %s" -- @ \
    | tee /dev/stderr | sort -u \
    > ../commits.txt

(improvement: or maybe output author as well)

Get all commits with HyperV (improvement: or sort by author)
sort -u ../commits.txt | rg -i 'hyper(-)?v' | sed -r 's|([a-f0-9]+) - |\1\t|g' | sort -k2 | tee ../hyperv-commits.txt
Modified files in Microsoft Source
Commits from Microsoft Source
e984097b553e    add support for Hyper-V reference time counter
dbcb4a1a3f16    arch/tile: add hypervisor-based character driver for SPI flash ROM
712c6ff4dba4    arm64: add hypervisor stub
b11930e43b57    arm64: hyperv: Add core Hyper-V include files
483435370e2c    arm64: hyperv: Add hypercall and register access functions
97da1296d15a    arm64: hyperv: Add memory alloc/free functions for Hyper-V size pages
67709ad6616d    arm64: hyperv: Allow hv_get_raw_timer() definition to be override
6d499ac018f0    arm64: hyperv: Enable vDSO
cf9fe18b3e20    arm64: vdso: hyperv: Map tsc page into vDSO if enabled
dd2cb348613b    clocksource/drivers: Continue making Hyper-V clocksource ISA agnostic
bd00cd52d5be    clocksource/drivers/hyperv: Add Hyper-V specific sched clock function
adb87ff4f96c    clocksource/drivers/hyperv: Allocate Hyper-V TSC page statically
3e2d94535adb    clocksource/drivers/hyperv: Enable TSC page clocksource on 32bit
20ae0f017682    clocksource/drivers/hyper-v: Reserve PAGE_SIZE space for tsc page
ffe45a955788    clocksource/drivers/hyper-v: Untangle stimers and timesync from clocksources
fd1fea6834d0    clocksource/drivers: Make Hyper-V clocksource ISA agnostic
db985220778e    doc,xen: document hypervisor sysfs nodes for xen
9aa8b50b2b3d    Drivers: hv: Add Hyper-V balloon driver
6c248aad81c8    Drivers: hv: Base autoeoi enablement based on hypervisor hints
59a084a70afa    drivers: hv: Cleanup the kvp related state in hyperv.h
3053c762444a    drivers:hv: Define the channel type for Hyper-V PCI Express pass-through
d27d38353730    Drivers: hv: Enable Hyper-V code to be built on ARM64
619848bd0743    drivers:hv: Export a function that maps Linux CPU num onto Hyper-V proc num
a108393dbf76    drivers:hv: Export the API to invoke a hypercall on Hyper-V
2939437ce8f2    drivers: hv: kvp: Move the contents of hv_kvp.h to hyperv.h
4f39bcfd1c13    drivers/hv: Move HV_SYNIC_STIMER_COUNT into Hyper-V UAPI x86 header
5b423efe11e8    drivers/hv: Move struct hv_message into UAPI Hyper-V x86 header
c71acc4c74dd    drivers/hv: Move struct hv_timer_message_payload into UAPI Hyper-V x86 header
18f098618aa0    drivers/hv: Move VMBus hypercall codes into Hyper-V UAPI header
eafa7072e7cd    Drivers: hv: Move vmbus version definitions to hyperv.h
d6f3609d2b4c    Drivers: hv: restore hypervcall page cleanup before kexec
81b18bce48af    Drivers: HV: Send one page worth of kmsg dump over Hyper-V during panic
917ea427c786    Drivers: hv: Setup a mapping for Hyper-V's notion cpu ID
c75efa974e01    drivers/hv: share Hyper-V SynIC constants with userspace
3647a83d9dcf    Drivers: hv: util: move kvp/vss function declarations to hyperv_vmbus.h
765e33f5211a    Drivers: hv: vmbus: Break out ISA independent parts of mshyperv.h
63ed4e0c67df    Drivers: hv: vmbus: Consolidate all Hyper-V specific clocksource code
7fb96565e3e1    Drivers: hv: vmbus: Consolidate all offer GUID definitions in hyperv.h
e8d6ca023efc    Drivers: hv: vmbus: define the new offer type for Hyper-V socket (hvsock)
2048157ad02e    Drivers: hv: vmbus: fix the building warning with hyperv-keyboard
687f32e6d9bd    Drivers: hv: vmbus: Move some ring buffer functions to hyperv.h
8de8af7e0873    Drivers: hv: vmbus: Move the extracting of Hypervisor version information
68a2d20b79b1    drivers/video: add Hyper-V Synthetic Video Frame Buffer Driver
6db7199407ca    drivers/virt: introduce Freescale hypervisor management driver
5267cf02c779    hv: Add hyperv.h to uapi headers
dee863b571b0    hv: export current Hyper-V clocksource
0592969e73ae    hv: fail the probing immediately when we are not in hyperv platform
2726f95e0b6c    hv: hyperv.h: remove unneeded forward declarations of structures
7a4ba88cc11e    hv: hyperv.h: remove unused module macros
a832a1eba940    hv: remove a bunch of unused debug macros from hyperv.h
9f3e28e375a8    hv: remove free_channel() from hyperv.h
15b80d641793    hv: remove struct hv_device_info from hyperv.h
5557e8a60570    hv: remove unused LOWORD and HIWORD macros from hyperv.h
ae0078fcf0a5    hv_sock: implements Hyper-V transport for Virtual Sockets (AF_VSOCK)
421463b80b40    hyper-v: Add myself as additional MAINTAINER
c3582a2c4d0b    hyperv: Add support for vNIC hot removal
16be29f096c7    Hyper-V: arm64: Support for repetitive hypercall
f45be72c8ec0    hyperv: Fix spelling of HV_UNKOWN
7415aea6072b    hyper-v: Globalize vp_index
6b22e36f3f19    Hyper-V: support for cold memory heat hint hypercall
3a68c34930f2    Hyper-V: support for free page reporting
05183189ee5d    hyper-v: Update MAINTAINERS
057841713cff    hyper-v: Use fast hypercall for HVCALL_SIGNAL_EVENT
597ff72f3de8    hyper-v: use GFP_KERNEL for hv_context.hv_numa_map
a88464a8b0ff    kvm: add hyper-v crash msrs values
1d5103c11e32    KVM: Add HYPER-V header file
6f6a657c9998    KVM/Hyper-V/VMX: Add direct tlb flush support
2a342ed57756    KVM: PPC: Implement hypervisor interface
faeb7833eee0    kvm: x86: hyperv: guest->host event signaling via eventfd
e516cebb4f2b    kvm/x86: Hyper-V HV_X64_MSR_RESET msr
9eec50b8bbe1    kvm/x86: Hyper-V HV_X64_MSR_VP_RUNTIME support
b2d8b167e15b    KVM: x86: hyper-v: set NoNonArchitecturalCoreSharing CPUID bit when SMT is impossible
1f4b34f825e8    kvm/x86: Hyper-V SynIC timers
8ed6d76781dd    kvm/x86: Rename Hyper-V long spin wait hypercall
ceef7d10dfb6    KVM: x86: VMX: hyper-v: Enlightened MSR-Bitmap support
32d5860a9e3c    MAINTAINERS: Add Hyper-V IOMMU driver into Hyper-V CORE AND DRIVERS scope
bd77c321945f    MAINTAINERS - add keyboard driver to Hyper-V file list
f92ca80b28a0    MAINTAINERS: add keyboard driver to Hyper-V file list
db46e14f8a3c    MAINTAINERS: Add missed file for Hyper-V
31f9783f4838    MAINTAINERS: Add myself to the Xen Hypervisor Interface and remove Chris Wright.
919691733197    MAINTAINERS: Change mailing list for Hyper-V CORE AND DRIVERS
bafe1e79e05d    MAINTAINERS: Fix Hyperv vIOMMU driver file name
cb4f131e1f2c    MAINTAINERS: Patch monkey for the Hyper-V code
a4162747b796    MAINTAINERS: update Hyper-V file list
5fe47c158d60    Merged PR 3218930: Hyper-V: Support for free page reporting
cc2dd4027a43    mshyperv: fix recognition of Hyper-V guest crash MSR's
33be96e47cc2    net/hyperv: Add flow control based on hi/low watermark
4d447c9a6ebc    net/hyperv: Add support for jumbo frame up to 64KB
c31c151b1c4a    net/hyperv: Fix the page buffer when an RNDIS message goes beyond page boundary
1194c4133195    nfit: Add Hyper-V NVDIMM DSM command set to white list
348dd93e40c1    PCI: hv: Add a Hyper-V PCI interface driver for software backchannel interface
4daace0d8ce8    PCI: hv: Add paravirtual PCI front-end for Microsoft Hyper-V VMs
0475f9ea8e2c    perf_counters: allow users to count user, kernel and/or hypervisor events
e712209a9e0b    perf: Fix hypervisor branch sampling permission check
3cdc20e51791    [POWERPC] Celleb: hypervisor console driver
3200761a1ad5    Revert "Merged PR 3218930: Hyper-V: Support for free page reporting
91c17449fe04    Revert "x86/hyper-v: include hyperv/ only when CONFIG_HYPERV is set"
44a01d5ba8a4    [S390] s390/hvc_console: z/VM IUCV hypervisor console support
a363bf7bd23f    Staging: hv: Add a subset of definitions from vmbus_api.h to hyperv.h
3e7ee4902fe6    Staging: hv: add the Hyper-V virtual bus
5c47340061ff    Staging: hv: Create a common header for all hyperv drivers to include
7692fd4d441a    Staging: hv: fix smp problems in the hyperv core code
8ff3e6fc58d7    Staging: hv: Include asm/hyperv.h in hyperv.h
3f335ea2131b    Staging: hv: Include the newly created header file in all of the relevant hyperv files
09d50ff8a233    Staging: hv: make the Hyper-V virtual bus code build
7effffb7a01b    Staging: hv: Move a subset of definitions from ring_buffer.h to hyperv.h
46a971913611    Staging: hv: move hyperv code out of staging directory
c35470b2297f    Staging: hv: Move the contents of channel.h to hyperv.h
b56dda06b4dd    Staging: hv: Move the contents of channel_mgmt.h to hyperv.h
f63c9149d052    Staging: hv: Move the contents of logging.h to hyperv.h
f7c6dfda0533    Staging: hv: Move the contents of the file version_info.h to hyperv.h
b189702dd9e1    Staging: hv: Move the contents of utils.h to hyperv.h
35ea09c39037    Staging: hv: Move the contents of vmbus_api.h to hyperv.h
517d8dc686f4    Staging: hv: Move the contents of vmbus_channel_interface.h to hyperv.h
27b5b3ca6903    Staging: hv: Move the contents of vmbus.h to hyperv.h
50ed40e0d06d    Staging: hv: Move the contents of vmbus_packet_format.h to hyperv.h
407dd1644302    Staging: hv: remove unneeded asm include file in hyperv.h
a73e6b7c508f    Staging: hv: Remove xen legacy code and check for Hyper-V
afbdc4a98b6b    Staging: hv: vmbus_drv: Include the contents of hv_api.h in hyperv_vmbus.h
89b2ca478126    Staging: hv: vmbus_drv: Move the contents of vmbus_private.h to vmbus_hyperv.h
773b79f7a7c7    tracing/hyper-v: Trace hyperv_mmu_flush_tlb_others()
a752ee56ad84    tty: Update hypervisor tty drivers to use core stdout parsing code.
af0a5646cb8d    use the new async probing feature for the hyperv drivers
0bde95690d65    virtio_net: Add support for VLAN filtering in the hypervisor
faa9b39f0e9d    virtio_net: propagate linkspeed/duplex settings from the hypervisor
90eedf0cbe4b    vmbus: use resource for hyperv mmio region
db34bbb767bd    X86: Add a check to catch Xen emulation of Hyper-V
e08cae4181af    x86: Clean up the hypervisor layer
9df56f19a500    x86: Correctly detect hypervisor
a2a47c6c3d1a    x86: Detect running on a Microsoft HyperV system
9279aa55061a    x86: Export the symbol ms_hyperv
bc2b0331e077    X86: Handle Hyper-V vmbus interrupts as special hypervisor interrupts
894266466aa7    x86/headers/UAPI: Use __u64 instead of u64 in <uapi/asm/hyperv.h>
e2768eaa1ca4    x86/hyperv: Add a function to read both TSC and TSC page value simulateneously
5d75a747596b    x86: hyperv: add CPUID bit for crash handlers
eb914cfe72f4    X86/Hyper-V: Add flush HvFlushGuestPhysicalAddressSpace hypercall support
8c3e44bde7fd    x86/hyperv: Add functions to allocate/deallocate page for Hyper-V
f726c4620df3    x86/hyperv: Add GUEST_IDLE_MSR support
cc4edae4b924    x86/hyper-v: Add HvFlushGuestAddressList hypercall support
e9a7fda29a56    x86/hyperv: Add interrupt handler annotations
a46d15cc1ae5    x86/hyper-v: allocate and use Virtual Processor Assist Pages
213ff44ae4eb    x86/hyper-V: Allocate the IDT entry early in boot
1e2ae9ec072f    x86/hyperv: Avoid reporting bogus NMI status for Gen2 instances
ca3ba2a2f4a4    x86, hyperv: Bypass the timer_irq_works() check
2cf0284223a4    x86/hyperv: Check frequency MSRs presence according to the specification
1aec169673d7    x86: Hyperv: Cleanup the irq mess
a3b742439292    x86/hyperv: Clear vCPU banks between calls to avoid flushing unneeded vCPUs
9a2d78e291a7    X86/Hyper-V: Consolidate the allocation of the hypercall input page
90ab9d551093    x86, hyperv: Correctly guard the local APIC calibration code
fcd3f6222a4e    x86/hyperv: Create and use Hyper-V page definitions
68d1eb72ee99    x86/hyper-v: define struct hv_enlightened_vmcs and clean field bits
5431390b3039    x86/hyper-v: detect nested features
a4987defc1e6    x86/hyper-v: Do some housekeeping in hyperv-tlfs.h
220d6586ecb4    x86/hyper-v: Drop HV_X64_CONFIGURE_PROFILER definition
68bb7bfb7985    X86/Hyper-V: Enable IPI enlightenments
1de72c706488    x86/hyper-v: Enable PIT shutdown quirk
3a025de64bf8    x86/hyperv: Enable PV qspinlock for Hyper-V
366f03b0cf90    X86/Hyper-V: Enhanced IPI enlightenment
6b48cb5f8347    X86/Hyper-V: Enlighten APIC access
13b5be56d1c5    x86: hyperv: Fix brown paperbag typos reported by Fenguangs build robot
d68ce0177c1e    x86, hyperv: Fix build error due to missing <asm/apic.h> include
9cd05ad2910b    x86/hyper-v: Fix definition of HV_MAX_FLUSH_REP_COUNT
7a83247e010a    x86/Hyper-V: Fix definition of struct hv_vp_assist_page
b42967dcac1d    x86/hyper-v: Fix indentation in hv_do_fast_hypercall16()
1e034743e918    x86/hyperv: Fix the build in the !CONFIG_KEXEC_CORE case
1268ed0c474a    x86/hyper-v: Fix the circular dependency in IPI enlightenment
76d388cd72ab    x86: hyperv: Fixup the (brain) damage caused by the irq cleanup
9fa023174294    x86, HyperV: fix up the license to mshyperv.c
9e7827b5ea4c    x86, hyperv: Get the local APIC timer frequency from the hypervisor
59107e2f4883    x86/hyperv: Handle unknown NMIs on one CPU when unknown_nmi_panic
7eff7ded02d1    x86, hyperv: Handle Xen emulation of Hyper-V more gracefully
41cfe2a2a7f4    x86/hyperv: Hide pv_ops access for CONFIG_PARAVIRT=n
2d2ccf24939c    x86/Hyper-V/hv_apic: Build the Hyper-V APIC conditionally
cb20e5f2c8d6    x86, hyperv: HYPERV depends on X86_LOCAL_APIC
53e52966901a    x86/hyper-v: Implement hv_do_fast_hypercall16
bd2a9adaadb8    x86/hyperv: Implement hv_get_tsc_page()
806c89273bab    x86/hyper-v: Implement rep hypercalls
2e252fbf777d    x86/hyper-v: include hyperv/ only when CONFIG_HYPERV is set
79cadff2d92b    x86/hyper-v: Include hyperv/ only when CONFIG_HYPERV is set
6f4151c89b7d    x86: Hyper-V: Integrate the clocksource with Hyper-V detection code
6a8edbd0c54a    x86/hyper-v: Introduce fast hypercall implementation
3998d095354d    x86, hypervisor: add missing <linux/module.h>
88b094fb8d4f    x86: Hypervisor detection and get tsc_freq from hypervisor
96f6e775b586    x86, hypervisor: Export the x86_hyper* symbols
fc53662f13b8    x86/hyper-v: Make hv_do_hypercall() inline
3c433679ab66    x86: hyperv: Make it build with CONFIG_HYPERV=m again
32c6590d1268    x86, hyperv: Mark the Hyper-V clocksource as being continuous
88c9281a9fba    x86/hyperv: Mark the Hyper-V TSC as unstable
ec084491727b    x86/hyper-v: Mark TLFS structures packed
4c08edd30501    x86, hyperv: Move a variable to avoid an unused variable warning
415bd1cd3a42    x86/hyper-v: move definitions from TLFS to hyperv-tlfs.h
5a4858032217    x86/hyper-v: move hyperv.h out of uapi
c9c92bee5330    x86/hyper-v: move struct hv_flush_pcpu{,ex} definitions to common header
0aa67255f54d    x86/hyper-v: move synic/stimer control structures definitions to hyperv-tlfs.h
0733379b512c    x86/hyperv: Move TSC reading method to asm/mshyperv.h
71c2a2d0a81f    x86/hyperv: Read TSC frequency from a synthetic MSR
93286261de1b    x86/hyperv: Reenlightenment notifications support
32068f6527b8    x86: Hyper-V: register clocksource only if its advertised
1278f58cdee6    x86/hyper-v: Remove duplicated HV_X64_EX_PROCESSOR_MASKS_RECOMMENDED definition
c629421a9900    x86, hyperv: remove PCI dependency
a1efa9b70097    x86/hyper-v: rename ipi_arg_{ex,non_ex} structures
22734de636fc    x86/Hyper-V: Report crash data in die() when panic_on_oops is set
f54812b91794    x86/Hyper-V: Report crash register data or kmsg before running crash kernel
b99d7940a0f2    x86: hyperv: report value of misc_features
f7c0f50f1857    x86/hyperv: Set to "Hyper-V"
84fdfafab849    x86/Hyper-V: Set x2apic destination mode to physical when x2apic is available
dd018597a074    x86/hyper-v: stash the max number of virtual/logical processor
628f54cc6451    x86/hyper-v: Support extended CPU ranges for TLB flush hypercalls
2ffd9e33ce4a    x86/hyper-v: Use hypercall for remote TLB flush
98f65ad45844    x86/kvm/hyper-v: remove stale entries from vec_bitmap/auto_eoi_bitmap on vector change
6a058a1eadc3    x86/kvm/hyper-v: use stimer config definition from hyperv-tlfs.h
26fcd952d5c9    x86/mshyperv: Remove excess #includes from mshyperv.h
e70e5892b28c    x86/retpoline/hyperv: Convert assembler indirect jumps
702644ec1cab    x86/timers: Move simple_udelay_calibration past init_hypervisor_platform
03b2a320b19f    x86/virt: Add enum for hypervisors to replace x86_hyper
f3614646005a    x86/virt, x86/platform: Add ->guest_late_init() callback to hypervisor_x86 structure

Techstack n – 1 is dead!

TL;DR TechStack n-1 is dead. It ended with the rise of the clouds and software release cycles going down to weeks due to containerized CIs.

The Death of Sophocles (Creative Commons)

Beeing OpenSource-based, Ubuntu already had the concept of point releases every 6 months when the Docker and K8s hit the world and gave automated CIs a big boost in making system containers. Some years after Docker itself switched to a 3-month release cycle. So did the Linux Kernel with 2-3 months. Firefox 4-weeks.

And Microsoft? Windows 10 became a platform. Service Releases for the end customers every 6 months. Insider Program became a built-in feature of Windows 10. Now you can call that a rolling release!

Given that and comparing that to the old paradigm of “we don’t run edgy software” it may take 8 weeks or 6 months until we have already reached TechStack n – 2.

Now compare that with the speed of your management decision processes!

Sure, not every feature is important but some of them are business critical just like WSL2. Or sometimes you get a security requirement like please change the base image of the systems. Sometimes you save time and money if you just deploy applications as services by helm chart instead of fixing a custom setup for weeks. Sometimes it’s easier to just rollback a deployment on Kubernetes than replaying a build process in hope to rebuild a lost artifact (build once, deploy many).

And then you have security: there infrastructure is also changing. Snapshot technologies are on the rise (hypervisors/ostree/next gen filesystems like ZFS and BTRFS).

Back to WSL2. Why? Dev/Production parity. And maintaining one tool chain based on an image. Still you could use VMs and some indeed still do you Vagrant with all it’s problems Both Apple and Microsoft strife for best developer experience on their systems, simple because OpenSource and Linux has triumphed and it is run on the future market which is cloud native.

That is why every big company is behind the (corporate) Linux Foundation and (corporate) CNCF (Cloud Native Computing Foundation). And that is probably why their certs are so cheap: because we need skilled developers that know their platform.

So, how do we keep up? Clouds are evolving fast and are ephemeral. So we need to become that too.

Not everyone is living at the edge and that’s also a requirement for a healthy system.

We need early adopters that can surf on the static noise generated by cloud native experiments, that can see where the ocean is moving. These are the OpenSource contributors and the ones birding GitHub repositories on their laptops.

We need those who can build stable container ships that sail the ocean. Those who know what’s happening on the technology radar.

We need skeptical ones that can see beyond fancy applications and hero myths to the bottom of underlying bare metal infrastructure.

We need people with long time experience knowing old and forgotten tricks.

Running that old techstack n-1 paradigm in modern environments has become dangerous and negligent.

We need to close the gaps in the DevSecOps Infinity Loop.

PS.: Sure, for datacenters keep running your RHEL or CentOS with support until 2026!

Download an LFS backed file from without `git` and `git-lfs` installed

Well, the API is there and you can do it already!

Just dig into the what git is doing by a test-clone with any LFS repo:

export GIT_CURL_VERBOSE=1                                                                               
export GIT_TRACE_CURL=1                                 
git clone <my-repo> 2>&1 | tee git-clone.log                                                

From there you are able to figure what is happing on SSH:

Now: download an LFS backed file from without git and git-lfs installed!

#!/usr/bin/env bash

PROJECT="${1}" # your-company/or-repo/whatever
FILEPATH="${2}" # foo/bar.tar.gz
REF="${3:-master}" # branch
SSHID="${4:-$HOME/.ssh/ecdsa_id}" # like $HOME/.ssh/mycompany

OID="$(set -o pipefail; curl -fSL -H "PRIVATE-TOKEN: ${GITLAB_TOKEN}" "${PROJECT//\//%2F}/repository/files/${FILEPATH//\//%2F}/raw?ref=${REF}" -o - | grep '^oid' | cut -d: -f2)"
AUTH="$(set -o pipefail; ssh -o IdentitiesOnly=yes -o IdentityFile="${SSHID}" git-lfs-authenticate "${PROJECT}.git" download | jq -er '.header.Authorization')"
mkdir -p "${FILEPATH%%/*}"
curl -H "Authorization: $AUTH" "${PROJECT}.git/gitlab-lfs/objects/${OID}" -o "${FILEPATH}"
file "${FILEPATH}"

And you can even do it without using SSH!

#!/usr/bin/env bash

PROJECT="${1}" # your-company/or-repo/whatever
FILEPATH="${2}" # foo/bar.tar.gz
BRANCH="${3:-master}" # branch

# get LFS file info
IFS=$'\n' LFS_OBJECT_INFO=($(curl -fsSL -H "PRIVATE-TOKEN: ${GITLAB_TOKEN}" "${PROJECT//\//%2F}/repository/files/${FILEPATH//\//%2F}/raw?ref=${BRANCH}" -o -))
OID="$(echo "${LFS_OBJECT_INFO[@]}" | grep -oP "oid sha256:\K[^\s]+")"
SIZE="$(echo "${LFS_OBJECT_INFO[@]}" | grep -oP "size \K[^\s]+")"
# request actions for given lfs objects
LFS_OBJECT_REQUEST_JSON="$(jq -rnc --arg oid $OID --arg size $SIZE --arg ref "refs/heads/$BRANCH" '{"operation":"download","objects":[{"oid":$oid,"size":$size}],"transfers":["lfs-standalone-file","basic"],"ref":{"name": $ref }}')"
LFS_OBJECT_ACTION_DOWNLOAD="$(set -o pipefail; curl -sSlf -H "Content-Type: application/json" -X POST -d "$JSON" "https://oauth2:${GITLAB_TOKEN}${PROJECT}.git/info/lfs/objects/batch" | jq -er '.objects[0]')"
AUTH="$(echo "$LFS_OBJECT_ACTION_DOWNLOAD" | jq -er '.header.Authorization')"
DOWNLOAD_URL="$(echo "$LFS_OBJECT_ACTION_DOWNLOAD" | jq -er '.href')"
mkdir -p "${FILEPATH%%/*}"
curl -H "Authorization: ${AUTH}" "${DOWNLOAD_URL}" -o "${FILEPATH}"

AWS CLI V2 is general available since Februray 2020. BUT 鈥

It is distributed as binary package (built on python and with pyinstaller that has bundled native libs) BUT鈥

  • Problem #1 this requires a decent GLIBC and probably won’t run on older redhat distributions
  • Problem #2 as it turns out many people use the official docker images to build their images and push to AWS ECR. and since these images are based on alpine and therefore use MUSL and NOT GLIBC it simply won’t run (but well people have seen now alpine pipelines in AWS builds although they do NOT support any other linux than their own; probably THEY HAVE TO)
  • Problem #3 you now have to download the application to update it (hey, the way to install software on windows or mac? but hey an own updater with aws/install –update)
  • Problem #4 forget installing and updating from pip for now; well you can directly pip from repo BUT it’s tricky. although the version says 2.0.38 (no beta or alpha!) it still requires a botocore==2.0.0dev42. i guess they should suffix a “ga” for “General Availability”.
  • Problem #5 If you belong to the majority (or the people with broken python installations) please refrain from saying “I have no problems! It’s better now!”

Anyway, they have some nyce features now and it seems they packed aws-shell right into to CLI:

UPDATE: It is possible to use GLIBC in Alpine and also get the size of WS CLI down:

How to get AWS-CLI v2 down from 127M to 67M

#blacklivesmatter in Software Industry und So…

Ich sehe das so: Systemischer Rassismus* f眉hrte dazu dass eine heterogene aber ethnische Gruppe von Menschen bez眉glich ihrer Hautfarbe benachteiligt wurde. Irgendwann f眉hrte es dazu, dass die Ungleichgewicht durch die ganzen kleinen Benachteiligungen so stark wurde, dass informell und formell jetzt eine Ausbalancierung stattfindet; “The White Man’s Legacy” sozusagen鈥

Leider wird dabei wieder gegen falsche Ziele oder 眉ber das Ziel hinaus gescho脽en: Es ist aber vor allem in unserer Zeit der Selbstradikalisierung besonders einfach, sich mit dem “correct a wrong” zu profilieren. Es z盲hlt wie in der Politik ohnehin oft nur noch das “moralisch korrekte”, da sich damit am einfachsten ill-informed Follower rekrutieren lassen. Ethisch gerechte sowie rationale Entscheidungen treten in den Hintergrund.

Getreu “yesterday’s heroes, tomorrow’s fool” d眉rfen wir wohl damit leben, dass Leits盲tze wie “Wenn nur einer im Team sich damit mehr inkludiert f眉hlt, dann machen wir das” unseren Hedonismus befl眉geln und dadurch auch ein Pressure auf Need-to-Change entsteht. Auch wenn dadurch mal was einfach total deppert benamst wird.

Ich denke das gute Kommunikation bei denen, die sich diskriminiert f眉hlen, dazu f眉hren kann, auch mal den Blickwinkel zu justieren. Nur weil in einem Chat jemand mal das falsche Wort getroffen hat, hei脽t dass nicht, dass es ein b枚se Person ist. And you are blocked! Zu kontrovers! Cancel Culture l盲sst gr眉脽en.

“Big solutions for small problems”.

Also lassen wir doch mal aber einen “Code of Conduct” zum Selbstzweck zu weaponizen – oder gebn wir uns vielelicht mal M眉phe zu erkennen, wenn das geschieht. Denn steht man nicht for Logik und Verstand ein kann das bei vielen Justice Warriorn schnell zu einer toxischen Kommunikationskultur f眉hren (you know “logic is a patriarchal thing”). Trotzem, das Selbstwertgef眉hl dieser Personen sollte man nicht verletzen! Entweder arrangiert man sich, wenn nicht passt man vielleicht doch nicht zu der Gruppe.

Ich bin aber auch ein Vertreter davon, dass Sprache Realit盲t schafft. Aber es bringt nix Denkm盲ler sinnlos zu zerst枚ren, weil damit viele auch Sinnbilder einer Kultur verbinden. Es muss Verst盲ndniss geschaffen werden, welche negativen Bilder, besser Narrative andere damit verbinden. Wenn zu kritisch, dann vielleicht ab damit ins Museum? Oder: “yeah, it’s time to lower the rebel flag!

Vielleicht m眉ssen wir ein St眉ck weit mit dem “Unsinn” mitschwimmen, und es vielleicht auch mal als Anregung sehen unsere eigenen Sichtweisen in Frage zu stellen.

Ist es nicht 盲hnlich wie beim Feminismus? eine Frauenquote betrachte ich strukturell als ungerecht, aber wir wissen mittlerweile auch, dass Frauen sich sehr positiv auf den Climatus auswirken, eben Diversit盲t. Das Problem ist eher der Maskulinismus, das Patriarchat, das nur als Anti-These zum Feminismus steht und es nicht geschafft hat, sich zu emanzipieren. So, ist jetzt besser zu sagen: ich schwimme als Whity mit allem mit, oder bin ich eher ein Mensch mit Intelligenz, Verantwortungsbewu脽tsein der sich erlaubt zu sagen: “Ich verstehe deine Punkte, aber Master kommt aus dem lateinischen!”

Und apropos Patriarchat: es ist tot, das Problem ist wir leben noch in den Ruinen von Rom.

Vielleicht sollte man sich auch mal 眉berlegen welche Pers枚nlichsdefizite oder -defekte man selbst hat und warum man sich so getriggert f眉hlt. Und das f盲ngt bei dem Qu盲lgeist der heutigen Zeit, Narzismus an, der ja bekannterweise durch Sozialisisation entstehen kann. Aber es geht ja weiter鈥 Depression, Borderline, bipolar…

Naja, mehr Gedanken zu dem Thema von Bryan Lunduke: Linux Kernel blacklists “blacklist”

UPDATE: That’s it about: CNN’s Van Jones brought to tears as Joe Biden wins US election

5垄 on YAML in the DevOps world

Envsubst as a GitHub Action, Serverless Frameworks that just look like a YAML wrapper around an API, YAML based TaskRunners that somewhat reensemble the requirements of a CI . Not sure if everyoke understood the convention over configuration thing and why would do descriptive instead of imperative (CloudFormation anyone?). It reminds me more of people now pushing thin layered YAML wrappers since the micro NPM projects now look a bit old-school on the curriculum vitae 馃檮 Well, obviously it’s about getting things to be easier to grasp. But then again sometimes find odd solutions when you hit the limits. The question is rather: who does find the most usable one? Interesting read:

CSV tools

Some CSV utils which may come handy since most teams are dealing with that kind of formats 馃槈
  • – part of my default toolset 馃槈
  • – part of my default toolset 馃槈
  • – interesting read
  • – yeah, the ripgrep guy
  • – now part of my default toolset (yeah, json but faster than jq with “flatten” filter; line grep is powerful)
  • – patch/diff for CSV and spreadsheet data
  • – in trial