Multi, Mono, Meta, Manifest – Composite Repository?

Mono or Multiple-Repositories? Meta-Repos!

There was this discussion about whether to do Mono or Multiple-Repositories? I won’t pick it up again.

Some cool guys that suggested: why not use the best of the two world and use meta-repos?!

I was interested in “meta-repo” tooling

I asked myself how they would solve problems and gave soem of them a try:

  • there are multi-repository management tools that work with by tagging or grouping repos like gr or mu-repo
  • tools that just do subdirectory iteration over your repos like gitbatch and …
  • many git-extras repisitories with subdirectory iteration scripts that do the same like git multi – the one i am using 😉
  • tools which combine multi-repo manifests with different VCS systems like myrepos – old and mature, within your linux system package management
  • tools that try to standardize the directory layout for managing your reositories like ghq – definitly you should use such layout!
  • tools which basically reassemble what git submodule does like mu-repo or git-metarepo or meta (see below)
  • tools that just checkout all your org’s repositories from GitHub, GitLab or BitBucket like ghorg
  • and git itself, who was not lazy and added git for-each-repo which also does simple iteration (experimental status!)
  • and then there was git-slave where it’s master deceided it was an anachronistic project and decided to give it up

In the end most tools mentioned here do not focus on workflow problems rather than introducing new manifests or config files 🙁

But basicallygit submodule is a meta-repo since we have .submodules!

Yes, that’s aboslutly valid. .submodules basically behaves like a multiple repository manifest (and you can set submodules also to active:false)

  • You can use git submodule foreach to create/commit/manage multiple branches
  • You could just setup a workspace or manifest repository that reassembles what other tools do

But isn’t there more to it?

  • There-repo are tools like repo which is used by Android ecosystem or the Alibaba Golang rewrite git-repo-go

  • They are using a manifest repository as well but you don’t need you to check out everything and it has an efficient way of managing dependencies
    (Android OS has 2000 repositories with repos different devices and drivers!)

  • They add pre-/post-process like symlinking or comamnd execution over mutliple repisitories (just like myrepos )
  • They introduce the concept of a topic (your current task) and solve workflow problems like having to change several sources in different repositories for one feature, diffing over mutliple repisitories and automatically creating a code review (using gerrit)

Let’s redefine our problem!

We want git submodules and an umbrella to add some workflow features wouldn’t be that wrong?

I did a search and came up with git-meta (not to be mistaken by other projects of the same name). They also have a nyce presention that bring up the mist important thoughts (WATCH IT!)

Unfortunatly GitLab docs did not return results on how to better solve where to put merge requests on meta-repos etc.

And then there is the more established meta which has 1.4K stars (only?) . But they try to reinvent .submodules and add a ton of (un)related features with like mono-meta-repo project migration. And they even use .metain their own repository to defined plugin-dependencies #facepalm.

The main issues on meta-repos are still GitLab and how to integrate your workflow (like multiple merge requests; microservices should be decoupled and should not break!).

I don’t know what tool is better – try it out! I just say i would use

So .submodules alone make a repository a meta-repository?

Yes and no – when you treat don’t treat dependencies like 3rd party libraries but an actual composite of multi-repos you need to work on.

Meta-Repos are a workaround for big too-big-to-be-fun mono-repositories and a half-baked solution for composite workspaces (like needed for microservices) since needed umbrella workflows are not integrated into GitHub or GitLab. If the problem doesn’t hurt don’t try to solve it since you create new problems. You can still use Virtual Workspaces like linking repo folders in your IDE or via symlink. Just try to stay focused on one task before having multiple workspaces open at the same time.

  • Multi-Pipeline setup can be simplified if using a meta-repo (or submodules)
  • Each affected service (repository) on your working topic will have a merge-request
  • .submodulesshould have CI checks (like is every repository-branch correctly set in .submodules ?)

Is ‘meta-repo’ a buzzword?

Yes. Basically it’s multi-repo and not much new except a new view on the workflow issues of having a composite repos.

Still you can use git subtreein your project but keep in mind when a repository is shared among many projects/workspaces it may diverge since upstreams are treated as second-class citizen and it’s harder to maintain to branch from dev on new features.

But then again: git-meta say’s in their project description:

Repository for the git-meta project — build your own monorepo using Git submodules

Multi, Mono, Meta – Composite Repo?

What now?

Further Readings

“Ternary operator” with JQ: Checking for empty values

As you noticed I use JQ commonly like RegExp. I recently used select(.!=null) to filter for non-null values. Turns out I missed something in docs:

jq -nc '{a:1},[1,23],true,null,42|values'

OUTPUT

{"a":1}
[1,23]
true
42
jq -nc '{a:1},[1,23],true,null,42|iterables'

OUTPUT

{"a":1}
[1,23]
jq -nc '{a:1},[1,23],true,null,42|nulls'

OUTPUT

null

Now that can be used to set defaults:

jq -nc '.notset?|values // "default"'

OUTPUT

"default"
$ jq -nc '.notset|values // "default"'

OUTPUT

"default"

Or just use it to do some little validation on expected input:

jq -nc '.notset|error("has no value")'

OUTPUT

jq: error (at <unknown>): has no value
jq -nc '{foo:"bar"}|(.baz|values // error("baz not set")|.'

OUTPUT

jq: error (at <unknown>): has no value
jq -nc '{foo:"bar"}|(.baz|values // error("baz not set")) as $e|.'
jq: error (at <unknown>): baz not set
jq -nc '{foo:"bar"}|(.foo|values // error("foo not set")) as $e|.'

OUTPUT

{"foo":"bar"}
jq -nc 'def chkkey(k): .[k]|values // error(k + "not set"), {foo:"bar"}|(.foo|values // error("foo is empty")) as $e|.'

OUTPUT

{"foo":"bar"}
jq -nc 'def notempty(k): .[k]|values // error(k + " is empty or not set"); {foo:"bar"}|notempty("foo")'

OUTPUT

"bar"
jq -nc 'def notempty(k): .[k]|values // error(k + " is empty or not set"); {baz:"bar"}|notempty("foo")'

OUTPUT

jq: error (at <unknown>): foo is empty or not set

Let’s turn it into a helper function:

jq -nc 'def notempty(k):
        . as $in
        |(
            (k|arrays // [k])[] as $k
            |$in[$k]|values // error($k + " is empty or not set")
        ) as $x
        |$in;

OUTPUT

{"foo":"bar","baz":"mar"}
jq -nc 'def notempty(k):
        . as $in
        |(
            (k|arrays // [k])[] as $k
            |$in[$k]|values // error($k + " is empty or not set")
        ) as $x
        |$in;
        {foo:"bar", baz: "mar"}|notempty(["meer"])'

OUTPUT

jq: error (at <unknown>): meer is empty or not set