Enswitch storage on GlusterFS
This document details the procedure for migrating a multi-machine Enswitch system from NFS storage to GlusterFS storage.
The procedure has been tested on Enswitch 3.11, but should work on most other versions. The GlusterFS servers run Ubuntu 14.04 64bit and the clients run Ubuntu 12.04 64bit.
The servers are as follows:
enswitchnfs0 - current active NFS server
enswitchnfs1 - current backup NFS server
enswitchstorage0 - New GlusterFS server 0
enswitchstorage1 - New GlusterFS server 1
The Enswitch subnet is 10.0.0.0/24
Load enswitchstorage0 and enswitchstorage1 with Ubuntu 14.04 64bit. Make a partition for the OS and leave the rest of the disk empty for the GlusterFS volume.
Update all OS packages on enswitchstorage0 and enswitchstorage1:
sudo apt-get update sudo apt-get dist-upgrade sudo apt-get autoremove sudo init 6
Install additional software on enswitchstorage0 and enswitchstorage1 (optional):
sudo apt-get install htop iotop bwm-ng tshark
Add firewall rules on enswitchstorage0 and enswitchstorage1. The following can be used as a firewall script:
iptables -F -m comment --comment "Clear all existing rules" iptables -A INPUT -m state --state RELATED,ESTABLISHED -m comment --comment "Allow packets from related and established connections" -j ACCEPT iptables -A INPUT -i lo -m comment --comment "Allow all on lo interface" -j ACCEPT iptables -A INPUT -p tcp --dport 24007 -m state --state NEW -s 10.0.0.0/24 -m comment --comment "Allow GlusterFS Daemon from Enswitch subnet" -j ACCEPT iptables -A INPUT -p tcp --dport 24008 -m state --state NEW -s 10.0.0.0/24 -m comment --comment "Allow GlusterFS Management from Enswitch subnet" -j ACCEPT iptables -A INPUT -p tcp --dport 49152 -m state --state NEW -s 10.0.0.0/24 -m comment --comment "Allow connections to GlusterFS brick #1 from Enswitch subnet" -j ACCEPT iptables -A INPUT -p tcp --dport 111 -m state --state NEW -s 10.0.0.0/24 -m comment --comment "Allow portmapper from Enswitch subnet" -j ACCEPT iptables -A INPUT -p udp --dport 111 -s 10.0.0.0/24 -m comment --comment "Allow portmapper from Enswitch subnet" -j ACCEPT iptables -A INPUT -m comment --comment "Log all unmatched packets" -j LOG iptables -A INPUT -m comment --comment "Drop all unmatched packets" -j DROP
Configure apt source for GlusterFS 3.5:
echo "deb http://ppa.launchpad.net/semiosis/ubuntu-glusterfs-3.5/ubuntu trusty main" | sudo tee /etc/apt/sources.list.d/glusterfs.list echo "deb-src http://ppa.launchpad.net/semiosis/ubuntu-glusterfs-3.5/ubuntu trusty main" | sudo tee -a /etc/apt/sources.list.d/glusterfs.list
Create a file named /tmp/glusterfs-repo.key contianing the following:
-----BEGIN PGP PUBLIC KEY BLOCK----- Version: SKS 1.1.4 Comment: Hostname: keyserver.ubuntu.com mI0ETYN4SQEEALc74ea0KYx05SgG5dYcToTq47C83LZM0IHYZTIt3IstfxNZeZ8xHZGnYKtA 6vj1PEut6W3c/iK07DSlNDaW8JJjBRki9XJuBakhciXx2/UZEnEQ2NAxaacby/4iWvX0LfCb lMsDipmUs6J64//OMAGp6yW+tfJEbNhYRwpNMHKZABEBAAG0GkxhdW5jaHBhZCBQUEEgZm9y IHNlbWlvc2lziLgEEwECACIFAk2DeEkCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJ EPoE05N3S6xNBOID/1Xi+6B2FxT1MBgLF/Fi0GgCXXbq+GxSSNbVXIDXoSSMxx0j+nUHIYbi 1h0zmGfAk0xZkhkOh4J2m5qGMu8uR62AyjCSimj30Z8d4p3bUPlukLQe91mbcLr8Bj1pruA0 KEX9xWRMgZMubkNay0gVKA7DzwEHXNHTBHxjHBAnn6Cj =7VYH -----END PGP PUBLIC KEY BLOCK-----
Import repository key:
sudo apt-key add /tmp/glusterfs-repo.key
Install GlusterFS packages on enswitchstorage0 and enswitchstorage1:
sudo apt-get update sudo apt-get install glusterfs-server
Add entries to /etc/hosts for each server on enswitchstorage0 and enswitchstorage1:
10.0.0.122 enswitchstorage0 10.0.0.123 enswitchstorage1
Create partition for the GlusterFS volume on enswitchstorage0 and enswitchstorage1, in this example we use /dev/sda2 as the glusterfs volume:
sudo fdisk sudo mkdir -p /var/glusterfs/sda2 sudo mkfs.ext4 /dev/sda2
Mount the GlusterFS partition on enswitchstorage0 and enswitchstorage1.
Add the following to /etc/fstab:
/dev/sda2 /var/glusterfs/sda2 ext4 noatime,nodiratime 0 0
Mount the partition manually:
sudo mount /var/glusterfs/sda2
Create directory for enswitch volume:
sudo mkdir /var/glusterfs/sda2/enswitch
Initialize the peer from enswitchstorage0 This command should return "peer probe: success":
sudo gluster peer probe enswitchstorage1
Check the peer status:
sudo gluster peer status
You should see this:
Number of Peers: 1 Hostname: enswitchstorage1-test Port: 24007 Uuid: 2eed9049-4f5c-4e14-8d49-8935df95c9fe State: Peer in Cluster (Connected)
Create GlusterFS volume, the only needs to be run on enswitchstorage0:
sudo gluster volume create enswitch replica 2 transport tcp enswitchstorage0:/var/glusterfs/sda2/enswitch enswitchstorage1:/var/glusterfs/sda2/enswitch
You should see the following if the creation was successful:
volume create: enswitch: success: please start the volume to access data
Start the volume:
sudo gluster volume start enswitch
Look at volume info:
sudo gluster volume info
Volume Name: enswitch Type: Replicate Volume ID: 3276be02-515f-41b1-ab22-a766724c8e64 Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: enswitchstorage0:/var/glusterfs/sda2/enswitch Brick2: enswitchstorage1:/var/glusterfs/sda2/enswitch
Configure settings for better performance:
sudo gluster volume set enswitch performance.cache-size 1GB
Renumber libuuid user and group to free up uid 100 and gid 101. Then change the UID/GID of the libuuid files to match their new groups:
sudo chown libuuid:libuuid /usr/sbin/uuidd sudo chown libuuid:libuuid /var/lib/libuuid
Add Enswitch user and group. There is no Enswitch code on these boxes, but this will make the file ownership show "enswitch:enswitch":
sudo adduser --system --group --no-create-home --home /var/lib/enswitch/home --disabled-password enswitch
Create local mountpoint on enswitchstorage0 and enswitchstorage1 so that the shared files can be accessed locally. NOTE: do not modify any files directly under /var/glusterfs/sda2/enswitch, as this will cause corruption:
sudo mkdir /var/lib/enswitch
Add the following line to /etc/fstab on enswitchstorage0 and enswitchstorage1:
enswitchstorage0:/enswitch /var/lib/enswitch glusterfs direct-io-mode=disable,_netdev 0 0
Mount /var/lib/enswitch on enswitchstorage0 and enswitchstorage1:
sudo mount /var/lib/enswitch
Configure apt source for GlusterFS 3.5
echo "deb http://ppa.launchpad.net/semiosis/ubuntu-glusterfs-3.5/ubuntu precise main" | sudo tee /etc/apt/sources.list.d/glusterfs.list echo "deb-src http://ppa.launchpad.net/semiosis/ubuntu-glusterfs-3.5/ubuntu precise main" | sudo tee -a /etc/apt/sources.list.d/glusterfs.list
Create a file named /tmp/glusterfs-repo.key contianing the following:
-----BEGIN PGP PUBLIC KEY BLOCK----- Version: SKS 1.1.4 Comment: Hostname: keyserver.ubuntu.com mI0ETYN4SQEEALc74ea0KYx05SgG5dYcToTq47C83LZM0IHYZTIt3IstfxNZeZ8xHZGnYKtA 6vj1PEut6W3c/iK07DSlNDaW8JJjBRki9XJuBakhciXx2/UZEnEQ2NAxaacby/4iWvX0LfCb lMsDipmUs6J64//OMAGp6yW+tfJEbNhYRwpNMHKZABEBAAG0GkxhdW5jaHBhZCBQUEEgZm9y IHNlbWlvc2lziLgEEwECACIFAk2DeEkCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJ EPoE05N3S6xNBOID/1Xi+6B2FxT1MBgLF/Fi0GgCXXbq+GxSSNbVXIDXoSSMxx0j+nUHIYbi 1h0zmGfAk0xZkhkOh4J2m5qGMu8uR62AyjCSimj30Z8d4p3bUPlukLQe91mbcLr8Bj1pruA0 KEX9xWRMgZMubkNay0gVKA7DzwEHXNHTBHxjHBAnn6Cj =7VYH -----END PGP PUBLIC KEY BLOCK-----
Import repository key:
sudo apt-key add /tmp/glusterfs-repo.key
Install GlusterFS client packages:
sudo apt-get update sudo apt-get install glusterfs-client
On one of the current NFS boxes, mount glusterfs volume as /var/lib/enswitch2/ and rsync data:
sudo mkdir /var/lib/enswitch2 sudo mount enswitchstorage0:/enswitch /var/lib/enswitch2 sudo chown enswitch:enswitch /var/lib/enswitch2 sudo chmod 775 /var/lib/enswitch2 sudo chmod g+s /var/lib/enswitch2 sudo rsync -av /var/lib/enswitch/ /var/lib/enswitch2/
Add the following line to /etc/fstab on all other servers:
enswitchstorage0:/enswitch /var/lib/enswitch glusterfs direct-io-mode=disable,_netdev 0 0
Cutover procedure:
Unmount /var/lib/enswitch on all servers
sudo umount /var/lib/enswitch
Rsync data one more time on old NFS server:
sudo rsync -av /var/lib/enswitch/ /var/lib/enswitch2/
Mount new GlusterFS volume on all Enswitch servers:
sudo mount /var/lib/enswitch
Restart enswitch
sudo enswitch restart
References:
http://www.server-world.info/en/note?os=Ubuntu_14.04&p=glusterfs&f=2
http://www.jamescoyle.net/how-to/457-glusterfs-firewall-rules
http://www.jamescoyle.net/how-to/351-share-glusterfs-volume-to-a-single-ip-address
https://launchpad.net/~semiosis/+archive/ubuntu/ubuntu-glusterfs-3.5
http://www.jamescoyle.net/how-to/559-glusterfs-performance-tuning
