GIT Large File Storage

Yesterday I wanted to push a git repository to GitHub to make the contents more visible. The content is a complete set of files together with a shell script to create a bootable SD card image for the SoCrates II SoC FPGA evaluation platform from Devboards. Next to the script itself, the repository thus contains a lot of binary blobs like the tarball of the root file system, the Linux kernel and FPGA sample bitstreams. Pushing such a repository to GitHub is not possible because of the current soft- and hard file size limits of 50 MiB and 100 MiB respectively.

git-lfs to the rescue!

Git Large File Storage or LFS for short is an extension to the git version control effectively replacing large files in a git repository with links to an LFS storage server. Providing such a server is an extra service but for example GitHub and GitLab both offer this. So for me step one was to install the git extension as it was not available from the Ubuntu 16.04 LTS archives. Luckily the LFS page contains a link to a PackageCloud area and so the installation was a matter of seconds (once the script running as superuser had been checked visually):

[dzu@harry ~]$ curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh \
  | sudo bash
...
[dzu@harry ~]$ sudo apt install git-lfs
...
[dzu@harry ~]$ git lfs install

Although the tools were then available, the repository itself still could not be pushed as it still needed to be migrated into a repository using LFS. A little research showed the tool git-lfs-migrate to be the suggested way to do this. Having downloaded (and unzipped) the latest version, the actual invocation was straight forward:

[dzu@harry tmp]$ cd /tmp
[dzu@harry tmp]$ git clone --mirror ~/projects/socrates-quickstart
Cloning into bare repository 'socrates-quickstart.git'...
done.
[dzu@harry tmp]$ java -jar ~/Downloads/git-lfs-migrate.jar -s socrates-quickstart.git \
   -d socrates-quickstart-migrated.git \
   -g git@github.com:laodzu/socrates-quickstart.git \
   "*.tar.gz" "vmlinux" "uImage" "*.rbf" "u-boot.img"

Not only was the repository converted, but the large files have also already been pushed to GitHubs LFS server. Pushing the resulting repository again was a matter of seconds. Of course this repository is a complete _rewrite_ of the original repository so the latter has to be replaced with a fresh clone:

[dzu@harry tmp]$ cd ~/projects
[dzu@harry projects]$ mv socrates-quickstart socrates-quickstart-non-lfs
[dzu@harry projects]$ git clone git@github.com:laodzu/socrates-quickstart
Cloning into 'socrates-quickstart'...
remote: Counting objects: 200, done.
remote: Compressing objects: 100% (96/96), done.
remote: Total 200 (delta 104), reused 197 (delta 103), pack-reused 0
Receiving objects: 100% (200/200), 247.14 KiB | 0 bytes/s, done.
Resolving deltas: 100% (104/104), done.
Checking connectivity... done.
Downloading rootfs-socrates.tar.gz (223.52 MB)
Downloading socrates1_lab.rbf (2.11 MB)
Downloading socrates2_lab.rbf (6.68 MB)
Downloading socrates_fb_touch.rbf (2.33 MB)
Downloading socrates_ocl.rbf (1.97 MB)
Downloading u-boot.img (233.44 KB)
Downloading uImage (3.64 MB)
Downloading vmlinux (80.20 MB)
[dzu@harry projects]$

The new repository only carries the binary files needed for the current revision:

[dzu@harry projects]$ du -hs socrates-quickstart socrates-quickstart-non-lfs
643M  socrates-quickstart
2,4G  socrates-quickstart-non-lfs/
[dzu@harry projects]$

Having checked that everything works as expected, the original repository can then be removed.