Git Filesystem

1. Git LFS

What is LFS

Storing Large content files eventually consumes a huge amount of space on the Git Server.

Moreover, since Git is a Distributed Version Control System, every clone and every checkout or pull of this repository will have to download each version of this huge file present on the Server; unlike Centralized Systems wherein just the latest version is downloaded.

Git LFS server comes to our rescue in terms of storing huge files.

Let’s see how?

Git LFS is a system for managing and versioning large files of upto 2GM in association with a Git repository.  Instead of storing the large files within the Git repository as blobs, Git LFS stores special "pointer files" in the repository, while storing the actual file contents on a Git LFS server.  The contents of the large file are downloaded automatically when needed, for example when a Git branch containing

the large file is checked out.

Referenced: Git LFS manual page.

Summary:

Git LFS handles large files by storing references to the file in the repository, but not the actual file itself. 

Diagram: Distribution of normal files and LFS files over the remote and LFS server

Diagram: git checkout of LFS files.

Diagram: after a ‘git checkout’

How to install git LFS

Step 1: Setup git LFS on your system. Or Just for a repository.

git lfs install

hugeProj [master] $git lfs install
Updated git hooks.
Git LFS initialized.

Step 2: Choose the type of files you want to track as LFS files.

git lfs track “*.iso”

 hugeProj [master] $git lfs track "*.iso" "*.tar"
Tracking "*.iso"
Tracking "*.tar"

--List the tracked files:
hugeProj [master] $git lfs track
Listing tracked patterns
   *.iso (.gitattributes)
   *.tar (.gitattributes)
Listing excluded patterns

Step 3: Tracking information is stored in .gitattributes file, add this file and rest of the files to the repository

git add .gitattributes

Untracked files:
 (use "git add <file>..." to include in what will be committed)

.gitattributes
CentOS-7-x86_64-Minimal-1804.iso
code/

Step 4: Commit, push the files normally to the Git server.

hugeProj [master] $git add .
hugeProj [master] $git commit -m 'Add LFS objects'
hugeProj [master] $git push origin master
Uploading LFS objects: 100% (1/1), 950 MB | 0 B/s, done                                                        
Enumerating objects: 7, done.
Counting objects: 100% (7/7), done.
Delta compression using up to 4 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (6/6), 875 bytes | 875.00 KiB/s, done.
Total 6 (delta 0), reused 0 (delta 0)
To github.com:divyabhushan/hugeProject.git
   2beb247..3f8184b  master -> master

Step 5: Verify the files on the Git server and local repository.

FLS files stored on the LFS Server and not on the Git Server.

Snapshot from GitHub Server shows an LFS file is available only in the RAW form; as it is stored on a LFS Server.

hugeProj [master] $ls -lt
total 1855496
-rw-r--r--  1 Divya1 staff         13 Jan 28 12:23 README
drwxr-xr-x  3 Divya1 staff        102 Jan 28 12:22 code
-rw-r--r--@ 1 Divya1  staff 950009856 Jan 28 12:20 CentOS-7-x86_64-Minimal-1804.iso

NOTE: LFS files are denoted with an ‘@’ attribute on the client machine

-@      Display extended attribute keys and sizes in long (-l) output.

Step 6: Clone this repository as a different local repository.

Regular files are pulled/downloaded from Git Server and LFS files from the LFS Server.

git clone http://github.com/divyabhushan/hugeProject.git largeProj 

Uploading LFS objects: 100% (1/1), 950 MB | 0 B/s, done

git lfs commands:

git lfs install:

Install Git LFS configuration.

git lfs version:

Report the version number.

git lfs env:

Display the Git LFS environment.

git lfs ls-files:

Show information about Git LFS files in the index and working tree.

hugeProj [master] $git lfs ls-files
714acc0aef * CentOS-7-x86_64-Minimal-1804.iso

git lfs track “*.ext”:
View or add Git LFS paths to Git attributes.

git lfs untrack “*.ext”

Remove Git LFS paths from Git Attributes.

git lfs status

Show the status of Git LFS files in the working tree.

git lfs push:
Push queued large files to the Git LFS endpoint.

git lfs update:

Update Git hooks for the current Git repository.

--Status after ‘git add’
myProj [initialBranch] $git lfs status
On branch initialBranch
Git LFS objects to be pushed to origin/initialBranch:

Git LFS objects to be committed:

          CentOS-7-x86_64-Minimal-1804.iso

Git LFS objects not staged for commit:

CONCLUSION:

Data is increasing tremendously over the internet; with IoT(“Internet Of Things”) technology booming.

There will always be a need of a good Version system like “Git” and the filesystem like “LFS” integrated with Git to store, process and analyze the ever-growing data.

We learnt:

What is LFS how do we store are large files in the LFS cloud and checkout them to our local repository.

2. Git patch operation

Git format-patch - Prepare patches for e-mail submission

Prepare each commit with its patch in one file per commit, formatted to resemble UNIX mailbox format. The output of this command is convenient for e-mail submission

Command:

git format-patch

1) Create a patch from the commit id (HEAD)

myProj [master] $git format-patch -1
0001-Add-the-Linux-iso-image.patch

2) Create a patch from an older selected commit id:

myProj [master] $git format-patch -1 367b4ed
0001-Removing-temp-and-useless-files.patch

3) Read the patch in a diff format:

[master] $cat 0001-Add-the-Linux-iso-image.patch 
From 310ddc5504a0533ab5612d7387a97b08450851d9 Mon Sep 17 00:00:00 2001
From: divya bhushan <divya_bhushan@hotmail.com>
Date: Mon, 28 Jan 2019 12:09:12 +0100
Subject: [PATCH] Add the Linux iso image.
---
 .gitattributes                   | 2 ++
 CentOS-7-x86_64-Minimal-1804.iso | 3 +++
 2 files changed, 5 insertions(+)
 create mode 100644 .gitattributes
 create mode 100644 CentOS-7-x86_64-Minimal-1804.iso

diff --git a/.gitattributes b/.gitattributes
new file mode 100644
index 0000000..539acb8
--- /dev/null
+++ b/.gitattributes
@@ -0,0 +1,2 @@
+*.iso filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
diff --git a/CentOS-7-x86_64-Minimal-1804.iso b/CentOS-7-x86_64-Minimal-1804.iso
new file mode 100644
index 0000000..9598fdb
--- /dev/null
+++ b/CentOS-7-x86_64-Minimal-1804.iso
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:714acc0aefb32b7d51b515e25546835e55a90da9fb00417fbee2d03a62801efd
+size 950009856
-- 
2.19.0

4) To list the Statistics of the patch but do not apply it, use the command: 

git apply --stat <patch_name>

myProj [master] $git apply --stat 0001-Add-the-Linux-iso-image.patch
.gitattributes                   | 2 ++
CentOS-7-x86_64-Minimal-1804.iso |    3 +++
2 files changed, 5 insertions(+)

5) Check for errors before applying the patch; with the command:

git apply --check <patch_name>

Let’s say you received the patch via email; and you would want to apply it in your present project

cd hugeProject
--copy the patch here.
hugeProj [master] $cp ../myProj/0001-Add-the-Linux-iso-image.patch

hugeProj [master] $git apply --check 0001-Add-the-Linux-iso-image.patch
error: .gitattributes: already exists in working directory
error: CentOS-7-x86_64-Minimal-1804.iso: already exists in working directory

The patch won’t apply here as the files are already present in the project

Let’s test another patch: 0001-Removing-temp-and-useless-files.patch

--Read the patch content and the stats
hugeProj [master] $git apply --stat 0001-Removing-temp-and-useless-files.patch
mytest.mine |    2 --
runtest.t   | 0
2 files changed, 2 deletions(-)
--copy the 2 files from the 367b4ed~1 commit id of ‘myProj’ repository here
myProj [master] $checkout 367b4ed~1
myProj [(HEAD detached at d1cd657)] $cp runtest.t ../hugeProj/
myProj [(HEAD detached at d1cd657)] $cp mytest.mine ../hugeProj/
--Test if the patch would apply successfully!
--No output means a success
cd ../hugeProj
hugeProj [master] $git apply --check 0001-Removing-temp-and-useless-files.patch

hugeProj [master] $

6) Apply the patch now.

Command:

git apply <patch_name>
git apply --verbose <patch_name>

--Files mytest.mine and runtest.t will be deleted.
hugeProj [master] $git apply --verbose 0001-Removing-temp-and-useless-files.patch
Checking patch mytest.mine...
Checking patch runtest.t...
Applied patch mytest.mine cleanly.
Applied patch runtest.t cleanly.

Use the --verbose flag to see the progress report of the patch being applied.

To get a summary of the patch run: ‘git apply --summary <patch_id>

myProj [master] $git apply --summary 0001-Add-the-Linux-iso-image.patch
create mode 100644 .gitattributes
create mode 100644 CentOS-7-x86_64-Minimal-1804.iso

Divya1@Divya:myProj [master] $git apply --summary 0001-Removing-temp-and-useless-files.patch
delete mode 100644 mytest.mine
delete mode 100644 runtest.t

SUMMARY:

We learnt to create, test, and apply the patch differences of a committed snapshot of your repository work. These patches are email friendly and are shared among repository users.