Automation

Download an LFS backed file from GitLab.com without git and git-lfs installed

It is possible to download a Git LFS-backed file from GitLab.com without having git or git-lfs installed by using the GitLab API directly. This article provides two shell scripts that demonstrate how to do this.

Well, the API is there and you can do it already!

Just dig into what git is doing by a test-clone with any LFS repo:

export GIT_CURL_VERBOSE=1
export GIT_TRACE_CURL=1
git clone <my-repo> 2>&1 | tee git-clone.log

From there you are able to figure out what is happening on SSH.

Thought Experiment: Automation in Big Corporate

Thought Experiment: Automation in a Big Corporate Environment

Goal: We want to transition from owning our own fleet of cars to using a taxi service as a utility (service provider).

However, we don’t know enough about the taxi business and feel uncertain. Therefore, our first step is to hire drivers to operate our existing cars as makeshift taxis. These drivers, however, are not trained taxi drivers, they don’t know the routes, and the navigation system is a proprietary product with outdated maps. It comes from a reputable manufacturer, which is useless in practice. To keep them awake and encourage them to learn the routes, we also provide our drivers with coffee from a well-known, premium coffee chain.

GitLab: checkout all available repositories

This guide provides a set of shell commands to automate the process of checking out all available repositories from one or more GitLab instances. It leverages the GitLab API, jq, and parallel to efficiently clone projects.

Generate a private token

https://<GITLAB-SERVER1>/profile/personal_access_tokens
https://<GITLAB-SERVER2>/profile/personal_access_tokens

Checkout a list of all available repositories

QUERY=".[] | .path_with_namespace + "\t" + .ssh_url_to_repo" # JQ Query
curl --request GET --header "PRIVATE-TOKEN: <PRIVATE-TOKEN>" "<GITLAB-SERVER1>/api/v4/projects?simple=true&per_page=65536" | jq -r "$QUERY" > repo.list
curl --request GET --header "PRIVATE-TOKEN: <PRIVATE-TOKEN>" "<GITLAB-SERVER2>/api/v3/projects?simple=true&per_page=65536" | jq -r "$QUERY" >> repo.list

Create directories for repositories

cat repo.list | cut -f1 | xargs mkdir -p

Checkout projects (with GNU parallel)

parallel --colsep '\t' --jobs 4 -a repo.list git clone {2} {1}

Build list of git repositories

find . -type d -name ".git"  | xargs realpath | xargs dirname > path.list

Report repository branch or checkout branch

cat path.list | xargs -I{} sh -c "cd {}; echo {}; git branch"
cat path.list | xargs -I{} sh -c "cd {}; echo {}; git checkout master"
cat path.list | xargs -I{} sh -c "cd {}; echo {}; git checkout develop"

Note: when you are migrating repositories you should use git clone --mirror.