GitLab: checkout all available repositories

| Created | Modified

This guide provides a set of shell commands to automate the process of checking out all available repositories from one or more GitLab instances. It leverages the GitLab API, jq, and parallel to efficiently clone projects.

Generate a private token

https://<GITLAB-SERVER1>/profile/personal_access_tokens
https://<GITLAB-SERVER2>/profile/personal_access_tokens

Checkout a list of all available repositories

QUERY=".[] | .path_with_namespace + "\t" + .ssh_url_to_repo" # JQ Query
curl --request GET --header "PRIVATE-TOKEN: <PRIVATE-TOKEN>" "<GITLAB-SERVER1>/api/v4/projects?simple=true&per_page=65536" | jq -r "$QUERY" > repo.list
curl --request GET --header "PRIVATE-TOKEN: <PRIVATE-TOKEN>" "<GITLAB-SERVER2>/api/v3/projects?simple=true&per_page=65536" | jq -r "$QUERY" >> repo.list

Create directories for repositories

cat repo.list | cut -f1 | xargs mkdir -p

Checkout projects (with GNU parallel)

parallel --colsep '\t' --jobs 4 -a repo.list git clone {2} {1}

Build list of git repositories

find . -type d -name ".git"  | xargs realpath | xargs dirname > path.list

Report repository branch or checkout branch

cat path.list | xargs -I{} sh -c "cd {}; echo {}; git branch"
cat path.list | xargs -I{} sh -c "cd {}; echo {}; git checkout master"
cat path.list | xargs -I{} sh -c "cd {}; echo {}; git checkout develop"

Note: when you are migrating repositories you should use git clone --mirror.

Update: Try getting all available repositories. If you don’t get all projects and just get a 404 error, you may have an issue. Try creating the list from what you see browsing GitLab or try to get Admin-Access.