This is the documentation that I created when I installed and setup GlusterFS as the storage solution for Docker containers.
The setup that I used include: 3 Debian 10 VMs with an extra hard disk/drive each and Docker installed with Docker Swarm enabled.
Setting Up GlusterFS
First we’ll need to setup our file system
check for your added drive:
lsblk
*mine is sdb
Create the filesystem (XFS in this case)
sudo mkfs.xfs /dev/(yourDrive)
sudo mkfs.xfs /dev/sdb
Create the directory for the Gluster volumes (for each node)
sudo mkdir /gluster/bricks/1 -p
sudo mount /dev/sdb /gluster/bricks/1
sudo mkdir /gluster/bricks/1/brick
sudo mkdir /gluster/bricks/2 -p
sudo mount /dev/sdb /gluster/bricks/2
sudo mkdir /gluster/bricks/2/brick
sudo mkdir /gluster/bricks/3 -p
sudo mount /dev/sdb /gluster/bricks/3
sudo mkdir /gluster/bricks/3/brick
Edit /etc/fstab so the disks get mounted when the OS boots (for each node)
sudo nano /etc/fstab
enter this line at the end:
/dev/sdb /gluster/bricks/1 xfs defaults 0 0
/dev/sdb /gluster/bricks/2 xfs defaults 0 0
/dev/sdb /gluster/bricks/3 xfs defaults 0 0
Verify that the disks are mounted
df -h
*/dev/sdb is mounted on /gluster/bricks/1
Now we can install GlusterFS
Install GlusterFS
sudo apt install glusterfs-server
Start GlusterD on all nodes
sudo systemctl enable glusterd
sudo systemctl start glusterd
Connect to Gluster nodes
sudo gluster peer probe [otherNodeIPHere]
then check gluster peer with:
sudo gluster peer status
sudo gluster pool list
Create Gluster volume/bricks (can be done on any node)
sudo gluster volume create gfs \
replica 3 \
192.168.5.25:/gluster/bricks/1/brick \
192.168.5.26:/gluster/bricks/2/brick \
192.168.5.27:/gluster/bricks/3/brick
Start the volume and verify that it is working
sudo gluster volume start gfs
sudo gluster volume list
sudo gluster volume status gfs
sudo gluster volume info gfs
Mount the created bricks to /mnt and verify
sudo mount.glusterfs localhost:/gfs /mnt
You can test this by making a sample directory inside /mnt and see if it replicates to the other node
cd /mnt
sudo mkdir GlusterTest
Ensure the Gluster Volume is mounted on boot by editing /etc/fstab (on the main node)
sudo nano /etc/fstab
Then insert this line at the end:
localhost:/gfs /mnt glusterfs defaults,_netdev 0 0
Create the directory /etc/systemd/system/srv.mount.d/ and a file called override.conf in srv.mount.d (this tells systemd to wait for Gluster to start before mounting the volume)
sudo mkdir /etc/systemd/system/srv.mount.d/
sudo nano /etc/systemd/system/srv.mount.d/override.conf
Enter this into override.conf:
[Unit]
After=glusterfs-server.service
Wants=glusterfs-server.service
