Script

Download an LFS backed file from GitLab.com without `git` and `git-lfs` installed

It is possible to download a Git LFS-backed file from GitLab.com without having git or git-lfs installed by using the GitLab API directly. This article provides two shell scripts that demonstrate how to do this.

Well, the API is there and you can do it already!

Just dig into what git is doing by a test-clone with any LFS repo:

export GIT_CURL_VERBOSE=1
export GIT_TRACE_CURL=1
git clone <my-repo> 2>&1 | tee git-clone.log

From there you are able to figure out what is happening on SSH.

Pure BASH interactive CLI/TUI menu (single-/multi-select/checkboxes)

This post presents a first version of a pure bash script for creating interactive command-line menus with single-select, multi-select, and checkbox functionality.

First version. To be refactored.

Asciinema recording

Gist Link

Inspired by

https://serverfault.com/questions/144939/multi-select-menu-in-bash-script
partly based on https://www.bughunter2k.de/blog/cursor-controlled-selectmenu-in-bash by Ingo Hollmann

Notes

This is a hacky first implementation for my shell tools/dotfiles (ZSH).
The intention is to use it for CLI wizards (my aim is NOT a full-blown curses TUI window interface).
I converted TPUT to ANSI-sequences to spare command executions (e.g. tput ed | xxd --plain).

Permission to copy and modify is granted under the Creative Commons Attribution 4.0 license.

PHP: just in case posix_isatty() is missing

This article provides a couple of PHP code snippets from StackOverflow that can be used to determine if a PHP script is running in an interactive terminal (TTY). These are useful workarounds for environments where the posix_isatty() function is not available.

Short function

function is_a_tty()
{
    static $result;
    if (is_null($result)) {
        $fp = fopen('php://stdin', 'r');
        $stat = fstat($fp);
        $mode = $stat['mode'] & 0170000; // S_IFMT
        $result = $mode == 0020000; // S_IFCHR
        fclose($fp);
    }
    return $result;
}

Info Class from StackOverflow by leigh

<?php

class IOMode
{
    public $stdin;
    public $stdout;
    public $stderr;

    private function getMode(&$dev, $fp)
    {
        $stat = fstat($fp);
        $mode = $stat['mode'] & 0170000; // S_IFMT

        $dev = new StdClass;

        $dev->isFifo = $mode == 0010000; // S_IFIFO
        $dev->isChr  = $mode == 0020000; // S_IFCHR
        $dev->isDir  = $mode == 0040000; // S_IFDIR
        $dev->isBlk  = $mode == 0060000; // S_IFBLK
        $dev->isReg  = $mode == 0100000; // S_IFREG
        $dev->isLnk  = $mode == 0120000; // S_IFLNK
        $dev->isSock = $mode == 0140000; // S_IFSOCK
    }

    public function __construct()
    {
        $this->getMode($this->stdin, fopen('php://stdin', 'r'));
        $this->getMode($this->stdout, fopen('php://stdout', 'w'));
        $this->getMode($this->stderr, fopen('php://stderr', 'w'));
    }
}

$io = new IOMode;

Input

$ php io.php
// Character device as input
// $io->stdin->isChr  == true

$ echo | php io.php
// Input piped from another command
// $io->stdin->isFifo == true

$ php io.php < infile
// Input from a regular file (name taken verbatim from C headers)
// $io->stdin->isReg  == true

$ mkdir test
$ php io.php < test
// Directory used as input
// $io->stdin->isDir  == true
Output:

$ php io.php
// $io->stdout->isChr  == true

$ php io.php | cat
// $io->stdout->isFifo == true

$ php io.php > outfile
// $io->stdout->isReg  == true
Error:

$ php io.php
// $io->stderr->isChr  == true

$ php io.php 2>&1 | cat
// stderr redirected to stdout AND piped to another command
// $io->stderr->isFifo == true

$ php io.php 2>error
// $io->stderr->isReg  == true

I hope this PHP stuff finally dies.

bash: shell table output to json

This post presents a Python script that converts tabular command-line output into a more versatile JSON format. This allows for easier data manipulation using tools like jq, as an alternative to complex text-processing pipelines in bash.

You know that sometimes it would be really great to format a shell output to a more versatile format like JSON or YAML that you can process with jq instead of writing long pipes with text-processing.

GitLab: checkout all available repositories

This guide provides a set of shell commands to automate the process of checking out all available repositories from one or more GitLab instances. It leverages the GitLab API, jq, and parallel to efficiently clone projects.

Generate a private token

https://<GITLAB-SERVER1>/profile/personal_access_tokens
https://<GITLAB-SERVER2>/profile/personal_access_tokens

Checkout a list of all available repositories

QUERY=".[] | .path_with_namespace + "\t" + .ssh_url_to_repo" # JQ Query
curl --request GET --header "PRIVATE-TOKEN: <PRIVATE-TOKEN>" "<GITLAB-SERVER1>/api/v4/projects?simple=true&per_page=65536" | jq -r "$QUERY" > repo.list
curl --request GET --header "PRIVATE-TOKEN: <PRIVATE-TOKEN>" "<GITLAB-SERVER2>/api/v3/projects?simple=true&per_page=65536" | jq -r "$QUERY" >> repo.list

Create directories for repositories

cat repo.list | cut -f1 | xargs mkdir -p

Checkout projects (with GNU parallel)

parallel --colsep '\t' --jobs 4 -a repo.list git clone {2} {1}

Build list of git repositories

find . -type d -name ".git"  | xargs realpath | xargs dirname > path.list

Report repository branch or checkout branch

cat path.list | xargs -I{} sh -c "cd {}; echo {}; git branch"
cat path.list | xargs -I{} sh -c "cd {}; echo {}; git checkout master"
cat path.list | xargs -I{} sh -c "cd {}; echo {}; git checkout develop"

Note: when you are migrating repositories you should use git clone --mirror.

Kernel Housekeeper Update Script

This post provides a bash script for Linux kernel management, specifically for those using mainline kernels on Ubuntu Bionic Beaver. The script helps in removing old kernels and installing the latest stable version.

I use this script for Kernel Housekeeping since I’m mostly on the mainline kernel. The script is currently used with Ubuntu Bionic Beaver.

#!/bin/bash

function version_gt() { test "$(echo \"$@\" | tr \" \n | sort -V | head -n 1)" != \"$1\"; }
function version_le() { test "$(echo \"$@\" | tr \" \n | sort -V | head -n 1)" == \"$1\"; }
function version_lt() { test "$(echo \"$@\" | tr \" \n | sort -rV | head -n 1)" != \"$1\"; }
function version_eq() { test "$(echo \"$@\" | tr \" \n | sort -rV | head -n 1)" == \"$1\"; }
#if version_gt $LATEST_KERNEL_VERSION_SHORT $CURRENT_KERNEL_VERSION_SHORT; then
#   echo \"$LATEST_KERNEL_VERSION_SHORT is greater than $CURRENT_KERNEL_VERSION_SHORT\"
# fi

OLD_IFS="$IFS"

IGNORE_PACKAGES="linux-headers-generic|linux-image-generic"
DUMP="$( curl -s \"http://kernel.ubuntu.com/~kernel-ppa/mainline/?C=M;O=A\" )"
HAS_INTERNET=0
if [ \"$?" -eq \"0\" ] && [ -n \"$DUMP\" ]; then
LATEST_KERNEL_VERSION="$( echo \"$DUMP\" | sed 's/<\/*[^>]*>\/*//g' | grep -E -o \"^v[^/]+\" |  sort -V  | grep -v '\-rc' | tail -1 | sed -r 's/^v//g' )"
LATEST_KERNEL_VERSION_SHORT="$( echo $LATEST_KERNEL_VERSION | tr '.-' '.' | cut -d. -f1-3 )"
HAS_INTERNET=1
else
LATEST_KERNEL_VERSION=\"Unable to resolve\"
fi

# Test if we have a generic kernel version
GENERIC_KERNEL_VERSION=$( dpkg-query -W -f'${db:Status-Abbrev} ${Package} ${Version}\n'  | grep -e '^ii' | grep 'linux-image-generic' | awk '{ print $3 }'  )
echo \"Generic Kernel Version\"
echo "======================"
if [ -n "$GENERIC_KERNEL_VERSION" ]; then
GENERIC_KERNEL_VERSION_SHORT="$( echo $GENERIC_KERNEL_VERSION | tr '.-' '.' | cut -d. -f1-3 )"
echo \"$GENERIC_KERNEL_VERSION_SHORT ($GENERIC_KERNEL_VERSION)\"
IGNORE_PACKAGES="$IGNORE_PACKAGES|linux-.*-$GENERIC_KERNEL_VERSION_SHORT-"
# echo $IGNORE_PACKAGES;
else
echo \"Not installed\".
fi
echo

CURRENT_KERNEL_VERSION=$( uname -r | cut -d- -f1,2 )
CURRENT_KERNEL_VERSION_SHORT="$( echo $CURRENT_KERNEL_VERSION | tr '.-' '.' | cut -d. -f1-3 )"
echo Current Kernel
echo "=============="
echo \"$CURRENT_KERNEL_VERSION ($CURRENT_KERNEL_VERSION_SHORT)\"
echo

echo "Latest Kernel (stable)"
echo "======================"
echo \"$LATEST_KERNEL_VERSION ($LATEST_KERNEL_VERSION_SHORT)\"
echo

IGNORE_PACKAGES="$IGNORE_PACKAGES|linux-.*$CURRENT_KERNEL_VERSION-"

# TODO add /boot/efi support
echo "Partitions "
echo "==========="
BOOT_PARTITIONS="$( df -h --output=target | grep "/boot" | tr '\n' ':' )"
if [ -z "$BOOT_PARTITIONS" ]; then
echo "No special partitions."
else
IFS=':'
for boot_partition in $BOOT_PARTITIONS; do
boot_part_total=$( df $boot_partition -h --output=size | tail -n 1 | tr -d '[:space:]' )
boot_part_avail=$( df $boot_partition -h --output=avail | tail -n 1 | tr -d '[:space:]' )
echo
echo \"$boot_partition size=$boot_part_total, free=$boot_part_avail\"
# Installed kernels

if [ ! "$( echo $boot_partition | grep efi > /dev/null 2>&1; echo $? )" -eq 0 ]; then
INSTALLED="$( test -r ${boot_partition} && ls -tr ${boot_partition}/{vmlinuz,initrd.img}-* | cut -d- -f2 | sort | uniq | tr '\n' ':'  )"
IFS=':'
for version in $INSTALLED; do
echo -n " $version taking "
SIZE=$( cd $boot_partition; ls -1  | grep "\-${version}-" | xargs du -shc | tail -1 | cut -f 1 )
echo $SIZE
done
fi;
done;
fi
echo

echo "Installed packages"
echo "=================="
echo

ALL_INSTALLED_PACKAGES=$( dpkg-query -W -f'${db:Status-Abbrev} ${Package} ${Version}\n'  | grep -e '^ii' | grep -e 'linux-image\|linux-signed-image\|linux-headers' | awk '{print $2}' | sort -V  )
echo $ALL_INSTALLED_PACKAGES
echo

echo "Removeable packages"
echo "==================="
echo

# config files
REMOVEABLE_PACKAGES="$( dpkg-query -W -f'${db:Status-Abbrev} ${Package} ${Version}\n'  | grep -e '^rc' | grep -e 'linux-image\|linux-signed-image\|linux-headers' | awk '{print $2}' | sort | grep -v -E \"$IGNORE_PACKAGES\" )"

for tag in "linux-image" "linux-headers" "linux-signed-image" "linux-image-extra"; do
# echo c="$tag-$CURRENT_KERNEL_VERSION-"
packages_to_be_removed="$( echo $ALL_INSTALLED_PACKAGES | grep $tag | sort -V | awk 'index($0,c){exit} //' c="$tag-$CURRENT_KERNEL_VERSION" | grep -v -E \"$IGNORE_PACKAGES\" )"
REMOVEABLE_PACKAGES="$( echo -e \"$REMOVEABLE_PACKAGES\n$packages_to_be_removed\" | sed '/^$/d' | sort -V | uniq )"
done;

if ! [ $(id -u) = 0 ]; then
echo \"You need to be root! Aborting...\"
exit 1
fi

if [ -z "$REMOVEABLE_PACKAGES" ]; then
echo "No packages to remove found!"
echo
else
echo $REMOVEABLE_PACKAGES
read -p "Remove (y/n)?" CHOICE
echo

case "$CHOICE" in
y|Y )
CMD="apt-get remove --purge $( echo $REMOVEABLE_PACKAGES | tr '\n' ' ' )"
# echo $CMD
eval "$CMD"
apt-get autoremove
echo
;;
* ) ;;
esac
fi

##
# Install packages from latest kernel
if [ $HAS_INTERNET == 1 ]; then

echo "Kernel upgrade"
echo "=============="
echo

LATEST_INSTALLED=$( echo $ALL_INSTALLED_PACKAGES | grep "linux-image-$LATEST_KERNEL_VERSION-" )
if [ -n "$LATEST_INSTALLED" ]; then
echo "$LATEST_KERNEL_VERSION already installed. No need to update."
echo
else
PACKAGES="$( curl -s \"http://kernel.ubuntu.com/~kernel-ppa/mainline/v$LATEST_KERNEL_VERSION_SHORT/\" |  grep -o -E 'linux-[_[:alnum:]\.\-]+\.deb' | sort | uniq )"
# echo $PACKAGES
MACHINE_TYPE=$(uname -m)
if [ ${MACHINE_TYPE} == 'x86_64' ]; then
ARCH="amd64"
else
ARCH="i386"
fi
INSTALL_PACKAGES="$( echo $PACKAGES | grep -E \"_($ARCH|all)\" |  grep -v lowlatency )"
if [ -z "$INSTALL_PACKAGES" ]; then
echo "No packages to install found. Check your script for URL errors!" 1>&2
exit 1
fi

echo "Following packages will be installed:"
echo $INSTALL_PACKAGES
echo
read -p "Continue (y/n)?" CHOICE
echo

# echo $INSTALL_PACKAGES
# exit
TMP=$( mktemp -d --suffix=.kernel-$LATEST_KERNEL_VERSION )
cd $TMP

IFS=$'\n'
URLS=""
for package in $INSTALL_PACKAGES; do
echo -n Downloading $package...
url="http://kernel.ubuntu.com/~kernel-ppa/mainline/v$LATEST_KERNEL_VERSION/$package"
wget  -q "$url"
if [ "$?" -eq 0 ]; then
echo done.
else
echo failed. aborting.
exit 1
fi
done
if [ -n "$INSTALL_PACKAGES" ]; then
dpkg -i *.deb
fi
fi
fi