.. title: GIT Large File Storage
.. slug: git-large-file-storage
.. date: 2017-03-23 17:45:30 UTC+01:00
.. tags: git, git-lfs, github
.. category: 
.. link: 
.. description: 
.. type: text

Yesterday I wanted to push a git repository to GitHub to make the
contents more visible.  The content is a complete set of files
together with a shell script to create a bootable SD card image for
the SoCrates II SoC FPGA evaluation platform from Devboards.  Next to
the script itself, the repository thus contains a lot of binary blobs
like the tarball of the root file system, the Linux kernel and FPGA
sample bitstreams.  Pushing such a repository to GitHub is not
possible because of the current soft- and hard file size limits of 50
MiB and 100 MiB respectively.

`git-lfs <https://git-lfs.github.com/>`_ to the rescue!  

.. image:: /images/git-lfs-icon.png
   :width: 200
   :alt: git-lfs banner
   :align: center

.. TEASER_END

Git Large File Storage or LFS for short is an extension to the git
version control effectively replacing large files in a git repository
with links to an LFS storage server.  Providing such a server is an
extra service but for example GitHub and GitLab both offer this.  So
for me step one was to install the git extension as it was not
available from the Ubuntu 16.04 LTS archives.  Luckily the LFS page
contains a link to a `PackageCloud <https://packagecloud.io/>`_ area
and so the installation was a matter of seconds (once the script
running as superuser had been checked visually):

.. code-block:: console

  [dzu@harry ~]$ curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh \
    | sudo bash
  ...
  [dzu@harry ~]$ sudo apt install git-lfs
  ...
  [dzu@harry ~]$ git lfs install

Although the tools were then available, the repository itself still
could not be pushed as it still needed to be migrated into a repository
using LFS.  A little research showed the tool `git-lfs-migrate
<https://github.com/bozaro/git-lfs-migrate>`_ to be the suggested way
to do this.  Having downloaded (and unzipped) the latest version, the
actual invocation was straight forward:

.. code-block:: console

  [dzu@harry tmp]$ cd /tmp
  [dzu@harry tmp]$ git clone --mirror ~/projects/socrates-quickstart
  Cloning into bare repository 'socrates-quickstart.git'...
  done.
  [dzu@harry tmp]$ java -jar ~/Downloads/git-lfs-migrate.jar -s socrates-quickstart.git \
     -d socrates-quickstart-migrated.git \
     -g git@github.com:laodzu/socrates-quickstart.git \
     "*.tar.gz" "vmlinux" "uImage" "*.rbf" "u-boot.img"

Not only was the repository converted, but the large files have also
already been pushed to GitHubs LFS server.  Pushing the resulting
repository again was a matter of seconds.  Of course this repository
is a complete _rewrite_ of the original repository so the latter has
to be replaced with a fresh clone:

.. code-block:: console

  [dzu@harry tmp]$ cd ~/projects
  [dzu@harry projects]$ mv socrates-quickstart socrates-quickstart-non-lfs
  [dzu@harry projects]$ git clone git@github.com:laodzu/socrates-quickstart
  Cloning into 'socrates-quickstart'...
  remote: Counting objects: 200, done.
  remote: Compressing objects: 100% (96/96), done.
  remote: Total 200 (delta 104), reused 197 (delta 103), pack-reused 0
  Receiving objects: 100% (200/200), 247.14 KiB | 0 bytes/s, done.
  Resolving deltas: 100% (104/104), done.
  Checking connectivity... done.
  Downloading rootfs-socrates.tar.gz (223.52 MB)
  Downloading socrates1_lab.rbf (2.11 MB)
  Downloading socrates2_lab.rbf (6.68 MB)
  Downloading socrates_fb_touch.rbf (2.33 MB)
  Downloading socrates_ocl.rbf (1.97 MB)
  Downloading u-boot.img (233.44 KB)
  Downloading uImage (3.64 MB)
  Downloading vmlinux (80.20 MB)
  [dzu@harry projects]$

The new repository only carries the binary files needed for the
current revision:

.. code-block:: console
  
  [dzu@harry projects]$ du -hs socrates-quickstart socrates-quickstart-non-lfs
  643M	socrates-quickstart
  2,4G	socrates-quickstart-non-lfs/
  [dzu@harry projects]$ 

Having checked that everything works as expected, the original
repository can then be removed.