Maintenance and migration

Why this matters

Repositories get slow for a small number of reasons, and all of them are measurable. Migration comes up whenever a company changes platform, and doing it by cloning and pushing loses more than people expect.

Measuring

$ git count-objects -vH

count: 27
size: 108.00 KiB
in-pack: 0
packs: 0
size-pack: 0 bytes
prune-packable: 0

A large size-pack is history; a large size with many loose objects usually means gc has not run recently.

To find what is actually big, list the largest objects and map them back to paths:

$ git rev-list --objects --all \
  | git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' \
  | awk '$1=="blob"' | sort -k3 -n -r | head -10

That is worth keeping somewhere: it is the standard answer to "why is this clone two gigabytes?", and the answer is almost always a handful of binaries committed years ago.

Cleaning up

Git runs gc automatically when loose objects accumulate, so most repositories need nothing. git maintenance start is the modern replacement for a cron job on a large repository, and it does incremental work rather than one long pause.

Large files that are already in history

Deleting a large file in a new commit does not shrink anything: every old commit still contains it (lesson 8.7). Removing it means rewriting history, with git filter-repo, and that has the same consequences as any rewrite, applied to every commit.

The realistic procedure:

1. Agree it with the whole team, and pick a time when nobody has unpushed work.
2. Back up: a full mirror clone, kept until everyone is satisfied.
3. Run git filter-repo with the path to remove.
4. Force-push all branches and tags.
5. Everyone re-clones. Not pulls: re-clones, because every hash has changed.
6. Move the file into LFS or out of the repository entirely, or it will come back (lesson 13.10).

Step 5 is why this is a planned operation rather than a Tuesday afternoon.

Migrating between platforms

Cloning normally and pushing copies your current branch and nothing else. A mirror copies everything:

$ git clone --mirror https://gitlab.com/northwind-trails/trailguide.git
$ cd trailguide.git
$ git remote set-url --push origin git@github.com:northwind-trails/trailguide.git
$ git push --mirror

That moves every branch, every tag and every note. What it does not move is everything that is not Git:

Both platforms have importers that carry issues and pull requests as well, and both are the right first choice: GitHub's importer for a GitLab project, and GitLab's Import project for a GitHub repository. Use --mirror when you only want the Git part, or when the importer cannot reach the source.

Two things to do before announcing the move: redirect or archive the old project, so nobody keeps pushing to it, and rewrite the remotes in everyone's clones:

$ git remote set-url origin git@github.com:northwind-trails/trailguide.git

LFS on migration

LFS objects live outside the Git repository, so a mirror push carries the pointers and not the content (lesson 13.10). They need moving explicitly:

$ git lfs fetch --all
$ git lfs push --all <new remote>

Forgetting this produces a repository that looks complete and whose images are three lines of text.

How to do it

$ git count-objects -vH
$ git gc
$ git maintenance start
$ git clone --mirror <old>
$ git push --mirror <new>
$ git remote set-url origin <new>

Common mistakes

• Deleting a large file and expecting the repository to shrink. Every old commit still contains it.
• Rewriting history without warning everyone, which leaves colleagues with incompatible clones.
• git gc --prune=now during a recovery.
• Migrating with a normal clone and push, which moves one branch.
• Forgetting the wiki, which is a separate repository.
• Forgetting LFS content, which produces a repository full of pointer files.
• Leaving the old project writable, so work continues in two places.