Installing GlusterFS on the Personal Compute Cluster

michael

When I set up the personal computer cluster‘s individual nodes, the SSD was set up to have three partitions. The partition mounted at /mnt/brick will be used to set up a GlusterFS volume. This GlusterFS volume will be used to persist data for services we run on the Docker Swarm that was previously set up on the cluster.

The GlusterFS software needs to be installed on all of the nodes in the cluster. Here is here pssh makes it easy to set up all nodes at the same time. 

parallel-ssh -i -h ~/cluster/all.txt -l root "apt-get update"
parallel-ssh -i -h ~/cluster/all.txt -l root "apt-get install -y software-properties-common"
parallel-ssh -i -h ~/cluster/all.txt -l root "add-apt-repository ppa:gluster/glusterfs-6"
parallel-ssh -i -h ~/cluster/all.txt -l root "apt-get update"
parallel-ssh -i -h ~/cluster/all.txt -l root "apt-get install -y xfsprogs attr glusterfs-server glusterfs-common glusterfs-client"
parallel-ssh -i -h ~/cluster/all.txt -l root "systemctl start glusterd"
parallel-ssh -i -h ~/cluster/all.txt -l root "systemctl enable glusterd"

Now set up the GlusterFS volume from the master node and set the the master node’s firewall to enable all traffic from within the cluster’s LAN:

sudo ufw allow from 10.1.1.0/24
sudo ufw reload
sudo gluster peer probe node1
sudo gluster peer probe node2
sudo gluster peer probe node3
sudo gluster peer probe node4
sudo gluster pool list

With the last command, you should see the list of nodes all connected. node1 will be listed as localhost since you are currently on it. The next step is to create a GlusterFS volume. First a folder on the /mnt/brick partition is created on every node, then the volume is created:

parallel-ssh -i -h ~/cluster/all.txt -l root "mkdir /mnt/brick/1/"
sudo gluster volume create gfs01 \
  node1:/mnt/brick/1/ \
  node2:/mnt/brick/1/ \
  node3:/mnt/brick/1/ \
  node4:/mnt/brick/1/
sudo gluster volume list

Note that the GlusterFS volume is created to be a distributed volume. You could create it as a replicated volume in order to have data reliability, though this would create higher latency on writes. The last command will show if the volume gfs01 is available. If it was created with no problem, start the volume and grant access to it with:

sudo gluster volume start gfs01
sudo gluster volume set gfs01 auth.allow 10.1.1.1,10.1.1.2,10.1.1.3,10.1.1.4

Finally, mount the GlusterFS volume to the /mnt/gfs mount point and set it up to remount on reboots:

parallel-ssh -i -h ~/cluster/all.txt -l root "mkdir -p /mnt/gfs"
parallel-ssh -i -h ~/cluster/all.txt -l root "mount.glusterfs localhost:/gfs01 /mnt/gfs"
parallel-ssh -i -h ~/cluster/all.txt -l root "echo 'localhost:/gfs01 /mnt/gfs glusterfs defaults,_netdev,backupvolfile-server=localhost 0 0' >> /etc/fstab"
parallel-ssh -i -h ~/cluster/all.txt -l root "chown -R root:docker /mnt/gfs"
parallel-ssh -i -h ~/cluster/all.txt -l root "chmod g+w /mnt/gfs"

Check that everything mounted OK:

parallel-ssh -i -h ~/cluster/all.txt -l michael "df -h"

You should see on every node that /mnt/gfs is mounted to the GlusterFS volume gfs01. That’s it. The mount /mnt/gfs can be used like any other directory on each node except that the contents of this mount will be identical on all nodes in the cluster.

Leave a Reply