Multi, Mono, Meta, Manifest – Composite Repository?
There was this discussion about whether to use Mono- or Multi-Repositories? I won’t pick it up again.
Some cool people suggested: why not use the best of both worlds and use meta-repos?!
I Was Interested in “Meta-Repo” Tooling
I asked myself how these tools would solve problems and gave some of them a try:
- There are multi-repository management tools that work by tagging or grouping repos, like gr or mu-repo.
- Tools that just do subdirectory iteration over your repos, like gitbatch.
- Many git-extras repositories with subdirectory iteration scripts that do the same, like git multi (the one I am using ;)).
- Tools which combine multi-repo manifests with different VCS systems, like myrepos (old and mature, often found within your Linux system package management).
- Tools that try to standardize the directory layout for managing your repositories, like ghq (you should definitely use such a layout!).
- Tools which basically reassemble what
git submoduledoes, like mu-repo, git-metarepo, or meta (see below). - Tools that just check out all your organization’s repositories from GitHub, GitLab, or BitBucket, like ghorg.
- And Git itself, which added
git for-each-repowhich also does simple iteration (experimental status!). - And then there was git-slave where its master decided it was an anachronistic project and gave it up.
In the end, most tools mentioned here do not focus on workflow problems; rather, they introduce new manifests or config files. :(
But Basically git submodule is a Meta-Repo Since We Have .gitmodules!
Yes, that’s absolutely valid. .gitmodules basically behaves like a multiple repository manifest (and you can set submodules also to active:false).
- You can use
git submodule foreachto create/commit/manage multiple branches. - You could just set up a workspace or manifest repository that reassembles what other tools do.
But Isn’t There More to It?
- There are tools like repo, which is used by the Android ecosystem, or the Alibaba Golang rewrite, git-repo-go.
- They are also using a manifest repository, but you don’t need to check out everything, and it has an efficient way of managing dependencies (Android OS has 2000 repositories for different devices and drivers!).
- They add pre-/post-process actions like symlinking or command execution over multiple repositories (just like myrepos).
- They introduce the concept of a topic (your current task) and solve workflow problems like having to change several sources in different repositories for one feature, diffing over multiple repositories, and automatically creating a code review (using Gerrit).
Let’s Redefine Our Problem!
We want git submodules and an umbrella to add some workflow features. Wouldn’t that be wrong?
I did a search and came up with git-meta (not to be mistaken for other projects of the same name). They also have a nice presentation that brings up the most important thoughts (WATCH IT!).
Unfortunately, GitLab documentation did not return results on how to better solve where to put merge requests on meta-repos, etc.
And then there is the more established meta, which has 1.4K stars (only?). But they try to reinvent .gitmodules and add a ton of (un)related features, like mono-meta-repo project migration. And they even use .meta in their own repository to define plugin-dependencies #facepalm.
The main issues with meta-repos are still GitLab and how to integrate your workflow (like multiple merge requests; microservices should be decoupled and shouldn’t break!).
I don’t know what tool is better—try it out!
So, Do .gitmodules Alone Make a Repository a Meta-Repository?
Yes and no. It depends on whether you treat dependencies like 3rd party libraries or as an actual composite of multi-repos you need to work on.
Meta-Repos are a workaround for big, too-big-to-be-fun mono-repositories and a half-baked solution for composite workspaces (like those needed for microservices), since the necessary umbrella workflows are not integrated into GitHub or GitLab. If the problem doesn’t hurt, don’t try to solve it, as you’ll create new problems. You can still use Virtual Workspaces, like linking repo folders in your IDE or via symlink. Just try to stay focused on one task before having multiple workspaces open at the same time.
- Multi-Pipeline setup can be simplified if using a meta-repo (or submodules).
- Each affected service (repository) on your working topic will have a merge request.
.gitmodulesshould have CI checks (e.g., is every repository branch correctly set in.gitmodules?).
Is ‘Meta-Repo’ a Buzzword?
Yes. Basically, it’s multi-repo with a new view on the workflow issues of having a composite repository, and not much new otherwise.
You can still use git subtree in your project, but keep in mind that when a repository is shared among many projects/workspaces, it may diverge since upstreams are treated as second-class citizens, and it’s harder to maintain when branching from dev for new features.
But then again, git-meta says in their project description:
Repository for the git-meta project – build your own monorepo using Git submodules
Multi, Mono, Meta - Composite Repo?
Further Readings
- Effective Meta-Repo Pipelines in GitLab
- Mono vs. Multi-Repos. Why choose one when you can have both?
- git-meta User Guide and Architecture
- Git Submodules vs. Subtrees