View source for Enswitch storage on GlusterFS

This document details the procedure for migrating a multi-machine Enswitch system from NFS storage to GlusterFS storage.

The procedure has been tested on Enswitch 3.11, but should work on most other versions.  The GlusterFS servers run Ubuntu 14.04 64bit and the clients run Ubuntu 12.04 64bit.


The servers are as follows:

enswitchnfs0 - current active NFS server

enswitchnfs1 - current backup NFS server

enswitchstorage0 - New GlusterFS server 0

enswitchstorage1 - New GlusterFS server 1

The Enswitch subnet is 10.0.0.0/24


Load enswitchstorage0 and enswitchstorage1 with Ubuntu 14.04 64bit. Make a partition for the OS and leave the rest of the disk empty for the GlusterFS volume.


Update all OS packages on enswitchstorage0 and enswitchstorage1:

 sudo apt-get update
 sudo apt-get dist-upgrade
 sudo apt-get autoremove
 sudo init 6


Install additional software on enswitchstorage0 and enswitchstorage1 (optional):

 sudo apt-get install htop iotop bwm-ng tshark 


Add firewall rules on enswitchstorage0 and enswitchstorage1.  The following can be used as the base for a firewall script:

 iptables -F -m comment --comment "Clear all existing rules"
 iptables -A INPUT -m state --state RELATED,ESTABLISHED -m comment --comment "Allow packets from related and established connections" -j ACCEPT
 iptables -A INPUT -i lo -m comment --comment "Allow all on lo interface" -j ACCEPT
 iptables -A INPUT -p tcp --dport 24007 -m state --state NEW -s 10.0.0.0/24 -m comment --comment "Allow GlusterFS Daemon from Enswitch subnet" -j ACCEPT
 iptables -A INPUT -p tcp --dport 24008 -m state --state NEW -s 10.0.0.0/24 -m comment --comment "Allow GlusterFS Management from Enswitch subnet" -j ACCEPT
 iptables -A INPUT -p tcp --dport 49152 -m state --state NEW -s 10.0.0.0/24 -m comment --comment "Allow connections to GlusterFS brick #1 from Enswitch subnet" -j ACCEPT
 iptables -A INPUT -p tcp --dport 111 -m state --state NEW -s 10.0.0.0/24 -m comment --comment "Allow portmapper from Enswitch subnet" -j ACCEPT
 iptables -A INPUT -p udp --dport 111 -s 10.0.0.0/24 -m comment --comment "Allow portmapper from Enswitch subnet" -j ACCEPT
 iptables -A INPUT -m comment --comment "Log all unmatched packets" -j LOG
 iptables -A INPUT -m comment --comment "Drop all unmatched packets" -j DROP


Configure apt source for GlusterFS 3.5:

 echo "deb http://ppa.launchpad.net/semiosis/ubuntu-glusterfs-3.5/ubuntu trusty main" | sudo tee /etc/apt/sources.list.d/glusterfs.list
 echo "deb-src http://ppa.launchpad.net/semiosis/ubuntu-glusterfs-3.5/ubuntu trusty main" | sudo tee -a /etc/apt/sources.list.d/glusterfs.list


Create a file named /tmp/glusterfs-repo.key contianing the following:

 -----BEGIN PGP PUBLIC KEY BLOCK-----
 Version: SKS 1.1.4
 Comment: Hostname: keyserver.ubuntu.com
 
 mI0ETYN4SQEEALc74ea0KYx05SgG5dYcToTq47C83LZM0IHYZTIt3IstfxNZeZ8xHZGnYKtA
 6vj1PEut6W3c/iK07DSlNDaW8JJjBRki9XJuBakhciXx2/UZEnEQ2NAxaacby/4iWvX0LfCb
 lMsDipmUs6J64//OMAGp6yW+tfJEbNhYRwpNMHKZABEBAAG0GkxhdW5jaHBhZCBQUEEgZm9y
 IHNlbWlvc2lziLgEEwECACIFAk2DeEkCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJ
 EPoE05N3S6xNBOID/1Xi+6B2FxT1MBgLF/Fi0GgCXXbq+GxSSNbVXIDXoSSMxx0j+nUHIYbi
 1h0zmGfAk0xZkhkOh4J2m5qGMu8uR62AyjCSimj30Z8d4p3bUPlukLQe91mbcLr8Bj1pruA0
 KEX9xWRMgZMubkNay0gVKA7DzwEHXNHTBHxjHBAnn6Cj
 =7VYH
 -----END PGP PUBLIC KEY BLOCK-----


Import repository key:

 sudo apt-key add /tmp/glusterfs-repo.key


Install GlusterFS packages on enswitchstorage0 and enswitchstorage1:

 sudo apt-get update
 sudo apt-get install glusterfs-server ntp


Add entries to /etc/hosts for each server on enswitchstorage0 and enswitchstorage1:

 10.0.0.122   enswitchstorage0
 10.0.0.123   enswitchstorage1


Create partition for the GlusterFS volume on enswitchstorage0 and enswitchstorage1, in this example we use /dev/sda2 as the glusterfs volume:

 sudo fdisk
 sudo mkdir -p /var/glusterfs/sda2
 sudo mkfs.ext4 /dev/sda2


Mount the GlusterFS partition on enswitchstorage0 and enswitchstorage1.

Add the following to /etc/fstab:
 /dev/sda2       /var/glusterfs/sda2	ext4    noatime,nodiratime      0 0


Mount the partition manually:

 sudo mount /var/glusterfs/sda2


Create directory for enswitch volume:

 sudo mkdir /var/glusterfs/sda2/enswitch


Initialize the peer from enswitchstorage0  This command should return "peer probe: success":

 sudo gluster peer probe enswitchstorage1


Check the peer status:
 sudo gluster peer status


You should see this:

 Number of Peers: 1
 
 Hostname: enswitchstorage1
 Port: 24007
 Uuid: 2eed9049-4f5c-4e14-8d49-8935df95c9fe
 State: Peer in Cluster (Connected)


Create GlusterFS volume, the only needs to be run on enswitchstorage0:

 sudo gluster volume create enswitch replica 2 transport tcp enswitchstorage0:/var/glusterfs/sda2/enswitch enswitchstorage1:/var/glusterfs/sda2/enswitch


You should see the following if the creation was successful:

 volume create: enswitch: success: please start the volume to access data


Start the volume:

 sudo gluster volume start enswitch


Look at volume info:

 sudo gluster volume info


 Volume Name: enswitch
 Type: Replicate
 Volume ID: 3276be02-515f-41b1-ab22-a766724c8e64
 Status: Started
 Number of Bricks: 1 x 2 = 2
 Transport-type: tcp
 Bricks:
 Brick1: enswitchstorage0:/var/glusterfs/sda2/enswitch
 Brick2: enswitchstorage1:/var/glusterfs/sda2/enswitch



Configure settings for better performance:

 sudo gluster volume set enswitch performance.cache-size 1GB


Renumber libuuid user and group to free up uid 100 and gid 101.  Then change the UID/GID of the libuuid files to match their new groups:

 sudo chown libuuid:libuuid /usr/sbin/uuidd
 sudo chown libuuid:libuuid /var/lib/libuuid


Add Enswitch user and group.  There is no Enswitch code on these boxes, but this will make the file ownership show "enswitch:enswitch":

 sudo adduser --system --group --no-create-home --home /var/lib/enswitch/home --disabled-password enswitch


Create local mountpoint on enswitchstorage0 and enswitchstorage1 so that the shared files can be accessed locally.  NOTE: do not modify any files directly under /var/glusterfs/sda2/enswitch, as this will cause corruption:

 sudo mkdir /var/lib/enswitch


Add the following line to /etc/fstab on enswitchstorage0 and enswitchstorage1:


 enswitchstorage0:/enswitch /var/lib/enswitch glusterfs        direct-io-mode=disable,backupvolfile-server=enswitchstorage1,_netdev 0 0


Mount /var/lib/enswitch on enswitchstorage0 and enswitchstorage1:

 sudo mount /var/lib/enswitch


Configure apt source for GlusterFS 3.5

 echo "deb http://ppa.launchpad.net/semiosis/ubuntu-glusterfs-3.5/ubuntu precise main" | sudo tee /etc/apt/sources.list.d/glusterfs.list
 echo "deb-src http://ppa.launchpad.net/semiosis/ubuntu-glusterfs-3.5/ubuntu precise main" | sudo tee -a /etc/apt/sources.list.d/glusterfs.list
 

Create a file named /tmp/glusterfs-repo.key contianing the following:

 -----BEGIN PGP PUBLIC KEY BLOCK-----
 Version: SKS 1.1.4
 Comment: Hostname: keyserver.ubuntu.com
 
 mI0ETYN4SQEEALc74ea0KYx05SgG5dYcToTq47C83LZM0IHYZTIt3IstfxNZeZ8xHZGnYKtA
 6vj1PEut6W3c/iK07DSlNDaW8JJjBRki9XJuBakhciXx2/UZEnEQ2NAxaacby/4iWvX0LfCb
 lMsDipmUs6J64//OMAGp6yW+tfJEbNhYRwpNMHKZABEBAAG0GkxhdW5jaHBhZCBQUEEgZm9y
 IHNlbWlvc2lziLgEEwECACIFAk2DeEkCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJ
 EPoE05N3S6xNBOID/1Xi+6B2FxT1MBgLF/Fi0GgCXXbq+GxSSNbVXIDXoSSMxx0j+nUHIYbi
 1h0zmGfAk0xZkhkOh4J2m5qGMu8uR62AyjCSimj30Z8d4p3bUPlukLQe91mbcLr8Bj1pruA0
 KEX9xWRMgZMubkNay0gVKA7DzwEHXNHTBHxjHBAnn6Cj
 =7VYH
 -----END PGP PUBLIC KEY BLOCK-----


Import repository key:

 sudo apt-key add /tmp/glusterfs-repo.key


Install GlusterFS client packages:

 sudo apt-get update
 sudo apt-get install glusterfs-client ntp


On one of the current NFS boxes, mount glusterfs volume as /var/lib/enswitch2/ and rsync data:

 sudo mkdir /var/lib/enswitch2
 sudo mount enswitchstorage0:/enswitch /var/lib/enswitch2
 sudo chown enswitch:enswitch /var/lib/enswitch2
 sudo chmod 775 /var/lib/enswitch2
 sudo chmod g+s /var/lib/enswitch2
 sudo rsync -av /var/lib/enswitch/ /var/lib/enswitch2/


Add the following line to /etc/fstab on all other servers:

 enswitchstorage0:/enswitch /var/lib/enswitch glusterfs        direct-io-mode=disable,backupvolfile-server=enswitchstorage1,_netdev 0 0


Cutover procedure:

Unmount /var/lib/enswitch on all servers

 sudo umount /var/lib/enswitch


Rsync data one more time on old NFS server:

 sudo rsync -av /var/lib/enswitch/ /var/lib/enswitch2/


Mount new GlusterFS volume on all Enswitch servers:

 sudo mount /var/lib/enswitch


Restart enswitch

 sudo enswitch restart







 

References:

http://www.server-world.info/en/note?os=Ubuntu_14.04&p=glusterfs&f=2

http://www.jamescoyle.net/how-to/457-glusterfs-firewall-rules

http://www.jamescoyle.net/how-to/351-share-glusterfs-volume-to-a-single-ip-address

https://launchpad.net/~semiosis/+archive/ubuntu/ubuntu-glusterfs-3.5

https://www.howtoforge.com/creating-an-nfs-like-standalone-storage-server-with-glusterfs-3.2.x-on-ubuntu-12.10

http://www.jamescoyle.net/how-to/559-glusterfs-performance-tuning

http://toruonu.blogspot.com/2012/12/xfs-vs-ext4.html

http://joejulian.name/blog/glusterfs-bit-by-ext4-structure-change/