Under construction

The main reference for this tutorial is the Pro Git book section on GIT internals.

This tutorial uses three libraries:

CodeMirror, released under the MIT license
sha1.js, released under the MIT license
pako 2.0.3, released under the MIT and Zlib licenses, see the project page for details.

Introduction

GIT is based on a simple model, with a lot of shorthands for common use cases. This model is sometimes hard to guess just from the everyday commands. To illustrate how GIT works, we'll implement a stripped down clone of GIT in a few lines of JavaScript. * empty lines and single closing braces excluded, a few more in total.

The Operating System's filesystem

Model of the filesystem

The Operating System's filesystem will be simulated by a very simple key-value store. In this very simple filesystem, directories are entries mapped to null and files are entries mapped to strings. The path to the current directory is stored in a separate variable.

Filesystem access functions (`read`, `write`, `mkdir`, `exists`, `cd`)

The filesystem exposes functions to read an entire file, create or replace an entire file, create a directory, test the existence of a filesystem entry, and change the current directory.

Filesystem access functions (`listdir`)

It will be handy for some operations to list the contents of a directory.

Example working directory

Our imaginary user will create a proj directory, and start filling in some files.

`git init` (creating `.git`)

The first thing to do is to initialize the GIT directory. For now, only the .git folder is needed, The rest of the function implementing git init will be written later.

Click on the eval button to see the files and directories that were created so far.

`git hash-object` (storing a copy of a file in `.git`)

The most basic element of a GIT repository is an object. Objects have a type which can be blob (individual files), tree (directories), commit (pointers to a specific version of the root directory, with a description and some metadata) and tag (named pointers to a specific commit, with a description and some metadata). When a file is added to the git repostitory, a compressed copy is stored in GIT's database, in the .git/objects/ folder. This copy is a blob object.

The compressed copy is given a unique filename, which is obtained by hashing the contents of the original file. Some filesystems have poor performance when a single directory contains a large number of files, and some filesystems have a limit on the number of files that a directory may contain. To circumvent these issues, the first two characters of the hash are used as the name of an intermediate directory: if a file's hash is 0a1bd…, its compressed copy will be stored in .git/objects/0a/1bd…

This function creates a file that looks like this:

The objects stored in the GIT database are compressed with zlib (using the "deflate" compression method). The filesystem view shows the marker deflated: followed by the uncompressed data. Click on the (un)compressed data to toggle between this pretty-printed view and the raw compressed data.

When creating some blob objects, the result could be, for example:

This function reproduces faithfully the behaviour of (a subset of the options of) the git hash-object command which can be called on a real git command-line.

Adding a file to the GIT database

So far, our GIT database does not know about any of the user's files. In order to add the contents of the README file in the database, we use git hash-object -w -t blob README, where -w tells GIT to write the object in its database, and -t blob indicates that we want to create a blob object, i.e. the contents of a file.

Click on the eval button to see the file that was created by this call.

You can notice that the database does not contain the name of the original file, only its content, stored under a unique identifier which is derived by hashing that content. Let's add the second user file to the database.

`zlib` compression

GIT compresses objects with zlib. The deflate() function used in the script above comes from the pako 2.0.3 library. To view a zlib-compressed object in your *nix terminal, simply write this declaration in your shell.

unzlib() {
  python -c \
    "import sys,zlib; \
     sys.stdout.buffer.write(zlib.decompress(open(sys.argv[1], 'rb').read()));" \
    "$1"
}

You can then inspect git objects as follows, using hexdump to view the null bytes and other non-printable bytes.

unzlib .git/objects/95/d318ae78cee607a77c453ead4db344fc1221b7 | hexdump -Cv

Storing trees (list of hashed files and subtrees)

At this point GIT knows about the contents of both of the user's files, but it would be nice to also store the filenames. This is done by creating a tree object

A tree object can contain files (by associating the blob's hash to its name), or directories (by associating the hash of other subtrees to their name). The mode (100644 for the file and 40000 for the folder) incidates the permissions, and is given in octal using the values used by *nix

In the contents of a tree, subdirectories (trees) are listed before files (blobs); within each group the entries are ordered alphabetically.

This function needs a small utility to convert hashes encoded in hexadecimal to raw bytes.

Example use of `store_tree()`

The following code, once uncommented, stores into the GIT database the trees for src and for the root directory of the GIT project.

The store_tree() function needs to be called for the contents of subdirectories first, and that result can be used to store the trees of upper directories. In the next section, we will write a function which takes a list of paths, constructs an internal representation of the hierarchy, and stores the corresponding trees bottom-up.

Storing a tree from a list of paths

Making trees out of the subfolders one by one is cumbersome. The following utility function takes a list of paths, and builds a tree from those.

function paths_to_tree(paths) {
  // This temporary mutable object will store a hierarchy of
  // subfolders and files, e.g.
  // {
  //   subfolders: { src: { subfolders: [], files: ['main.scm'] } }
  //   files: ['README']
  // }
  var hierarchy = { subfolders: {}, files: [] };

// This splits the input paths on occurrences of "/",
  // and inserts them into the "hierarchy" object.
  for (var i = 0; i < paths.length; i++) {
    var path_components = paths[i].split('/');
    var h = hierarchy;
    for (var j = 0; j < path_components.length - 1; j++) {
      if (! h.subfolders.hasOwnProperty(path_components[j])) {
        h.subfolders[path_components[j]] = {
          subfolders: {},
          files: []
        };
      }
      h = h.subfolders[path_components[j]];
    }
    h.files[h.files.length] = path_components[path_components.length - 1];
  }

// This function takes the path to a directory, e.g. "src",
  // and a hierarchy object e.g. { subfolders: [], files: ['main.scm'] }.
  // It recursively stores the tree object for that directory into
  // GIT's database.
  var to_tree = function(base_directory, hierarchy) {
    var subtrees = [];
    for (var i in hierarchy.subfolders) {
      if (hierarchy.subfolders.hasOwnProperty(i)) {
        subtrees[subtrees.length] = {
          name: i,
          hash: to_tree(join_paths(base_directory, i), hierarchy.subfolders[i])
        };
      }
    }
    return store_tree(base_directory, hierarchy.files, subtrees);
  }

// Store the trees for the whole hierarchy, starting from the
  // root directory of the GIT repository (which is represented
  // as an empty path "")
  return to_tree("", hierarchy);
}

// git add README src/main.scm
paths_to_tree(["README", "src/main.scm"]);

Storing a commit in the GIT database

Now that the GIT database contains the entire tree for the current version, a commit can be created. A commit contains

the hash of the tree object,
the hash of the previous commit, which is dubbed the parent (merge commits have two or more parents, and the initial commit has no parent commit),
information about the author (the person who initially wrote the code),
information about the committer (the person who adds the code to the GIT database, often the same person as the author, but it can be a different person e.g. when someone else rewrites the history with a rebase or applies a patch recieved by e-mail),
and a description.

The author and committer information contain

the person's name,
the person's email,
the *nix timestamp at which the version was authored or committed,
and the timezone for that timestamp.

function store_commit(tree, parents, author, committer, message) {
  var commit_contents = '';
  commit_contents += 'tree ' + tree + '\n';
  for (var i = 0; i < parents.length; i++) {
    commit_contents += 'parent ' + parents[i] + '\n';
  }
  commit_contents += 'author ' + author.name
                  + ' <' + author.email + '> '
                  + format_date(author.date) + ' '
                  + format_timezone(author.timezoneMinutes) + '\n';
  commit_contents += 'committer ' + committer.name
                  + ' <' + committer.email + '> '
                  + format_date(committer.date) + ' '
                  + format_timezone(committer.timezoneMinutes) + '\n';
  commit_contents += '\n';
  commit_contents += '' + message + (message[message.length-1] == '\n' ? '' : '\n');
  // cat commit_contents | git hash-object -w -t commit --stdin
  return hash_object(true, 'commit', true, commit_contents);
}

Storing an example commit

It is now possible to store a commit in the database. This saves a copy of the tree along with some metadata about this version. The first commit has no parent, which is represented by passing the empty list.

resolving references

The next few sections will introduce symbolic references like branch names, the special name HEAD or tag names.

Symbolic references are nothing more than regular files containing a hexadecimal hash or a string of the form ref: path/to/other/symbolic/reference. The HEAD reference is stored in .git/HEAD, and can point directly to a commit hash like 0123456789abcdef0123456789abcdef01234567, or can point to another symbolic reference, in which case the .git/HEAD file will contain e.g. refs/heads/main.

Branches are simple files stored in .git/refs/heads/name-of-the-branch and usually contain a hash like 0123456789abcdef0123456789abcdef01234567.

Tags are identical to branches in terms of representation. It seems that the only difference between tags and branches is the behaviour of git checkout and similar commands. These commands, as explained in the section about git checkout below, normally write ref: refs/heads/name-of-branch in .git/HEAD when checking out a branch, but write the hash of the target commit when checking out a tag or any other non-branch reference.

We'll start with a small utility to remove the newline at the end of a string. GIT references are usually files containing a hexadecimal hash, and following *NIX tradition these files finish with a newline byte. When reading these references, we need to get rid of the newline first.

`git symbolic-ref`

git symbolic-ref is a low-level command which reads (and in the official GIT implementation also writes and updates) symbolic references given a path relative to .git/. For example, git symbolic-ref HEAD will read the contents of the file .git/HEAD, and if that file starts with ref: , the rest of the line will be returned.

The official implementation of GIT follows references recursively and returns the path/to/file of the last file of the form ref: path/to/file. In the example below, git symbolic-ref HEAD would

read the file proj/.git/HEAD which contains ref: refs/heads/main,
follow that indirection and read the file proj/.git/refs/heads/main which contains ref: refs/heads/other
follow that indirection and read the file proj/.git/refs/heads/other which contains a hash
return the last file path that contained a ref:, i.e. return the string refs/heads/other

`git rev-parse`

console.log('hello', 'world', 3);
function git_rev_parse(ref) {
  var symbolic_ref_target = git_symbolic_ref(ref);
  if (symbolic_ref_target) {
    // symbolic ref like "ref: refs/heads/main"
    return git_rev_parse(symbolic_ref_target);
  } else if (/[0-9a-f]{40}/.test(ref)) {
    // hash like "0123456789abcdef0123456789abcdef01234567"
    return ref;
  } else if (ref == 'HEAD') {
    // user-friendly reference like "HEAD"
    return git_rev_parse(trim_newline(read(join_paths(current_directory, '.git/' + ref))));
  } else if (ref.startsWith('refs/') && exists(join_paths(current_directory, '.git/' + ref))) {
    // user-friendly reference like "refs/heads/main"
    return git_rev_parse(trim_newline(read(join_paths(current_directory, '.git/' + ref))));
  } else if (exists(join_paths(current_directory, '.git/refs/heads/' + ref))) {
    // user-friendly reference like "main" (a branch)
    return git_rev_parse(trim_newline(read(join_paths(current_directory, '.git/refs/heads/' + ref))));
  } else if (exists(join_paths(current_directory, '.git/refs/tags/' + ref))) {
    // user-friendly reference like "v1.0" (a branch)
    return git_rev_parse(trim_newline(read(join_paths(current_directory, '.git/refs/tags/' + ref))));
  } else {
    // unknown ref
    return false;
  }
}

`git branch`

A branch is a pointer to a commit, stored in a file in .git/refs/heads/name_of_the_branch. The branch can be overwritten with git branch -f. Also, as will be explained later, git commit can update the pointer of a branch.

`HEAD`

The HEAD indicates the "current" commit. It is set at first as part of the git init routine.

`git commit`

If the HEAD points to a commit hash, then git commit updates the HEAD to point to the new commit. Otherwise, when the HEAD points to a branch, then the target branch (represented by a file named .git/refs/heads/the_branch_name) is updated.

`git tag`

Tags are like branches, but are stored in .git/refs/tags/the_tag_name and a tag is not normally modified. Once created, it's supposed to always point to the same version.

GIT does offer a git tag -f existing-tag new-hash command, but using it should be a rare occurrence.

`git checkout`

Checkout, branches and other references

More importantly, the HEAD does not normally point to a tag. Although nothing actually prevents writing ref: refs/tags/v1.0 into .git/HEAD, the GIT commands will not automatically do this. For example, git checkout tag-or-branch-or-hash will put a symbolic ref: in .git/HEAD only if the argument is a branch.

Checking out files

Assert

The parsers will check that their input looks reasonably well-formed, using assert().

Reading compressed objects

Parsing tree objects

The parse_tree function above needs a small utility to convert hashes represented using raw bytes to a hexadecimal representation.

Parsing commit objects

assert(tree,      'commit lacks tree header');
  assert(author,    'commit lacks author header');
  assert(committer, 'commit lacks committer header');

return {
    tree: tree,
    parents: parents,
    author: author,
    committer: committer,
    message: message
  };
}

Parsing author and committer metadata

Example checkout

`git init`

The git init command creates the .git directory and points .git/HEAD to the default branch (a file which does not exist yet, as this branch does not contain any commit at this point).

The index

When adding files with git add, GIT does not immediately create a commit object. Instead, it adds the files to the index, which uses a binary format with lots of metadata. The mock filesystem used here lacks most of these pieces of information, so thr value 0 will be used for most fields. See this blog post for a more in-depth study of the index.

function store_index(paths) {
  var magic = 'DIRC' // DIRectory Cache
  var version = raw_bytes32(2);
  var entries = raw_bytes32(paths.length);
  var header = magic + version + entries;

index = header;

for (var i = 0; i < paths.length; i++) {
    var ctime  = raw_bytes64(0);
    var mtime  = raw_bytes64(0);
    var device = raw_bytes32(0);
    var inode  = raw_bytes32(0);
    // default permissions for files, in octal.
    var mode   = raw_bytes32(0100644);
    var uid    = raw_bytes32(0);
    var gid    = raw_bytes32(0);
    var size   = raw_bytes32(read(join_paths(current_directory, paths[i])).length);
    var hash   = hex_to_raw_bytes(hash_object(true, 'blob', false, paths[i]));
    // for this simple index, the flags (the 4 higher bits) are 0.
    assert(paths[i].length < 0xfff)
    var flags_and_file_path_length = raw_bytes16(paths[i].length)
    var file_path = paths[i] + '\0';
    entry = ctime + mtime + device + inode + mode + uid + gid + size
          + hash + flags_and_file_path_length + file_path;
    while (entry.length % 8 != 0) {
      // pad with null bytes to a multiple of 8 bytes (64-bits).
      entry += '\0';
    }

index += entry;
  }

index += hex_to_raw_bytes(sha1(index));

write(join_paths(current_directory, '.git/index'), index)
}

Playground

The implementation is now sufficiently complete to create a small repository.

By clicking on "Copy commands to recreate in *nix terminal.", it is possible to copy a series of mkdir … and printf … > … commands that, when executed, will recreate the virtual filesystem on a real system. The resulting folder is bit-compatible with the official git log, git status, git checkout etc. commands.

Conclusion

This article shows that a large part of the core of GIT can be re-implemented in a few source lines of code* (copy all the code). * empty lines and single closing braces excluded, a few more in total.

Some of the features which may appear mysterious at first sight (e.g. detached HEAD) should be clearer with the knowledge of how GIT works behind the scenes.

Furthermore, branches are often associated with an intuition (containers into which commits are added) which does not match the implementation (mutable pointers to commits).

Finally, it is tempting to think of commits as patches. While darcs tries to expose an interface which matches this intuition, it is clear that the implementation of GIT considers commits as copies of the entire repository, and are linked to the previous version solely by the parent metadata in the commit headers.

A few core commands like git diff and git apply are not described in this tutorial. They are little more than improved versions of the classical *nix commands diff and patch.

Most other commands provided by GIT are merely convenience wrappers around these commands. For example, git cherry-pick is simply a combination of git diff between the tree of a commit and the tree of its parent, followed by git apply to apply the patch and git commit to create a new commit whose diff is equivalent to the diff of the original commit. As an other example, the command git rebase performs as succession of cherry-pick operations.

By keeping in mind the internal model of GIT, it becomes easier to understand the usual commands and their quirks. By undersanding the design philosophy behind the implementation, the day-to-day usage can become, hopefully, less surprising.