:date: 2014-03-14 13:45

.. index:: git, tech, puppet, linux

GIT and Puppet
==============

.. image:: /_images/images/gallery/2014/2014-placeholder/logo-git_puppet.jpg
   :alt: git and puppet
   :scale: 100
   :align: right

`Puppet <https://puppetlabs.com/>`_ as a tool has become quite useful. Since the major upgrade from version 2.7 to 3.x, many things have changed.

But not only in Puppet, also the environment around it has changed and adopted to new challenges.

With Puppet supporting different environments and configurations connected to it, deploying changes to them can be quite challenging. The marriage of `GIT <http://en.wikipedia.org/wiki/Git_(software)>`_ with Puppet comes immediately into some ones mind and it makes actually sense. The codebase of Puppet is plain text, putting this into a versioning system is a huge benefit.

The `Puppetlabs documentation <https://docs.puppetlabs.com/>`_ describes an interesting and quite `useful way <http://puppetlabs.com/blog/git-workflow-and-puppet-environments>`_ to automatically deploy branches of a GIT repository into the file-system and Puppet environment setup. It uses the `GIT hooks <http://git-scm.com/book/th/Customizing-Git-Git-Hooks>`_ and a ruby script to trigger a repository export into the file-system.

Due to the nature of GIT, a repository always contains a sub-directory called `.git`. GIT keeps a snapshot of all files under it's control, so we can go back whenever we want.

One way of checking out the content of a GIT branch is to use a GIT hook and fetch the just checked in version to the file-system. That works very reliable and stable and makes it easy to turn changes in the repository into actual changes on a system.

Until it doesn't.

The drawback of this method becomes clear only after a while.

With each update, a complete copy of all files get deployed into the file-system. You're basically creating zillions of files. This wouldn't hurt really much, but it's unnecessary. When this constellation meets a rather small partition size as well, the number of `i-nodes <http://en.wikipedia.org/wiki/Inode>`_ on a disk get so low, that you can't create new files and can't deploy changes any more.

This is the number of i-nodes I had available, when I hit the problem. The Puppet configuration directory is under `/etc/puppet`. On that partition, though rather small of size, 63103 i-nodes are used. I could create single files, but I couldn't roll out major changes into the Puppet environments anymore.

.. code:: bash

   $ df -i
   Filesystem            Inodes   IUsed   IFree IUse% Mounted on
   /dev/sda2              65536   63103    2433   97% /
   tmpfs                 490610       1  490609    1% /dev/shm
   /dev/sda1              65536      39   65497    1% /boot
   /dev/mapper/vg0-home   65536     626   64910    1% /home
   /dev/mapper/vg0-opt    65536   10631   54905   17% /opt
   /dev/mapper/vg0-tmp    32768      12   32756    1% /tmp
   /dev/mapper/vg0-usr   192768   42689  150079   23% /usr
   /dev/mapper/vg0-var   642560  168830  473730   27% /var
   /dev/mapper/vg0-data  198800    2008  196792    2% /data

There are many ways of solving this issue.

Delete branches
---------------
As first action I listed the setup branches and removed them from the GIT repository. Especially older branches that have been created in order to develop modules,to investigate an incident or to test a feature. Usually they are created with the full history of the parent branch. Not, because it would be necessary, but because it's easy to forget to put the parameter '--orphan' in the checkout command.

.. code:: bash

   $ git checkout --orphan new-branch

Each of those branches deployed to the puppet server starts eating up the i-nodes. Just deleting them when they aren't needed any more, will free up space

.. code:: bash

   $ git -r -d|-D branch-name
   # or
   $ git push origin :branch-name


Deleting history
----------------
As a simpler version of the previous workaround, deleting the history is also an option. What you basically do is you spawn a new branch of the current one with the history and rename the new one to the old one. That way you drop all the history of the current branch and reduce that way the number of i-node you use.

.. code:: bash

   $ git push origin :old-branch-name
   $ git checkout -b --orphan new-branch-name
   $ git checkout new-branch-name
   $ git -D old-branch-name
   $ git branch -m new-branch-name old-branch-name
   $ git push origin new-branch-name

Separate Partition
------------------
While the first two solutions address more the source of the problem (GIT) and temper with the configuration there, this workaround moves the Puppet configuration onto a separate partition and gives it therefore its own set of i-<nodes.

I've created a 5GB partition in the LVM setup (also using Puppet) and mounted it into `/etc/puppet.new`. This was the partition layout when still running low on i-nodes:


.. code:: bash

   $ df -i
   Filesystem            Inodes   IUsed   IFree IUse% Mounted on
   /dev/sda2              65536   63103    2433   97% /
   tmpfs                 490610       1  490609    1% /dev/shm
   /dev/sda1              65536      39   65497    1% /boot
   /dev/mapper/vg0-home   65536     626   64910    1% /home
   /dev/mapper/vg0-opt    65536   10631   54905   17% /opt
   /dev/mapper/vg0-tmp    32768      12   32756    1% /tmp
   /dev/mapper/vg0-usr   192768   42689  150079   23% /usr
   /dev/mapper/vg0-var   642560  168830  473730   27% /var
   /dev/mapper/vg0-data  198800    2008  196792    2% /data
   /dev/mapper/vg0-puppet
                         131072   0   131072  0% /etc/puppet.new

The root partition containing `/etc/puppet` is down to ~2400 i-nodes (that means ~2400 files can be created additionally). The new partition with 5GB space has the double amount of i-nodes available and will be only used for the Puppet configuration in the GIT repository. This one is currently mounted on `/etc/puppet.new`.

The next step was simply to sync all files and attributes to the new location

.. code:: bash

   $ sudo rysnc -rvt /etc/puppet /etc/puppet.new

-----

**Update:** (2014-06-26)

It seems that even that was not enough. The new partition ran into the same problems just after a couple of weeks. Just increasing the disc space does not seem to be right, since only so little is used. Instead I have decreased the size of the inodes/number of inodes per group.

Before the change is was like this:

.. code:: bash

   $ sudo tune2fs -l /dev/mapper/vg0-puppet  | grep Inode
   Inode count:              131072
   Inodes per group:         8192
   Inode blocks per group:   512
   Inode size:           256

By taking a backup of the partition, unmounting it and re-creating the file-system on it, I was able to increase the available numbers of inodes

.. code:: bash

   # Backup already taken
   $ sudo umount /etc/puppet
   $ sudo mkfs.ext4 -I 2048 /dev/mapper/vg0-puppet
   $ sudo mount /etc/puppet
   # Restore Backup from here

The result will (hopefully) last a bit longer then just moving the files to a separate partition.

.. code:: bash

   $ df -i
   Filesystem            Inodes   IUsed   IFree IUse% Mounted on
   ...
   /dev/mapper/vg0-puppet
                        2621440  119223 2502217    5% /etc/puppet

-----

Changing the partition layout in `/etc/fstab` will help us doing the re-mount faster and permanent

.. code:: bash

   - /dev/mapper/vg0-puppet /etc/puppet.new ext3 defaults 0 0
   + /dev/mapper/vg0-puppet /etc/puppet ext3 defaults 0 0

Now just stopping the Puppetmaster and the Puppet agent, so nothing breaks while doing the magic.

.. code:: bash

   $ sudo service puppetmaster stop
   $ sudo service puppet stop

Finally, the moment of truth has come. The old configuration folders moves out of the way and the new partition will take it's place.

.. code:: bash

   $ sudo mv /etc/puppet /etc/puppet.old
   $ sudo umount /dev/mapper/vg0-puppet
   $ sudo mount -a

There's now two folders with Puppet configuration files under `/etc/`:

#. `/etc/puppet.old` #( The old configuration files on the same partition as `/`,
#. `/etc/puppet` #( The copy of the configuration files

After starting the Puppetmaster and the Puppet agent again, everything should work as usual and Puppet will continue to distribute the configuration.

The GIT deployment will also still work, because none of the files or paths have actually changed. The only thing that changed was the location, where all data is stored.

Having final look at the i-nodes over the partitions, we'll see, that this solved our problem for now.

.. code:: bash

   $ df -i
   Filesystem            Inodes   IUsed   IFree IUse% Mounted on
   /dev/sda2              65536    6123   59413   10% /
   tmpfs                 490610       1  490609    1% /dev/shm
   /dev/sda1              65536      39   65497    1% /boot
   /dev/mapper/vg0-home   65536     627   64909    1% /home
   /dev/mapper/vg0-opt    65536   10635   54901   17% /opt
   /dev/mapper/vg0-tmp    32768      12   32756    1% /tmp
   /dev/mapper/vg0-usr   192768   42689  150079   23% /usr
   /dev/mapper/vg0-var   642560  168853  473707   27% /var
   /dev/mapper/vg0-data  198800    2008  196792    2% /data
   /dev/mapper/vg0-puppet
                         131072   56993   74079   44% /etc/puppet

Final thoughts
--------------
Personally I think think that extending the Ruby-Script with some functions to clean up the partitions or to reduce the number of Inodes could have been a better solution. Even a pull of only the last version (`git pull --depth=1 origin`) could be a solution here.
However, since I'm not that fluent in Ruby and I was confident in what I had to do, switching over to the other partition, was a fast and easy way to do it.