Enswitch storage on GlusterFS
Contents
Disclaimer
The following comes with no warranty whatsoever. I am not responsible for any data loss or other issues that may arise from following these instructions. Please make backups of all files and test this thoroughly in your lab environment before using it in production.
Overview
This document details the procedure for migrating a multi-machine Enswitch system from NFS storage to GlusterFS storage.
The procedure has been tested on Enswitch 3.11, but should work on most other versions. The GlusterFS servers run Ubuntu 14.04 64bit and the clients run Ubuntu 12.04 64bit.
The servers are as follows:
enswitchnfs0 - current active NFS server
enswitchnfs1 - current backup NFS server
enswitchstorage0 - New GlusterFS server 0
enswitchstorage1 - New GlusterFS server 1
The Enswitch subnet is 10.0.0.0/24
Server configuration
Load enswitchstorage0 and enswitchstorage1 with Ubuntu 14.04 64bit. Make a partition for the OS and leave the rest of the disk empty for the GlusterFS volume. Do not create a swap partition, a swap file will be added later.
Update all OS packages on enswitchstorage0 and enswitchstorage1:
sudo apt-get update sudo apt-get dist-upgrade sudo apt-get autoremove sudo init 6
Upgrade 14.04 kernel to 3.16 to take advantage of new metadata checksum and free inode btree options in XFS:
sudo apt-get update sudo apt-get install --install-recommends linux-generic-lts-utopic sudo apt-get autoremove sudo init 6
Create swap file:
sudo dd if=/dev/zero of=/swapfile0 bs=1M count=2048 sudo chmod 0600 /swapfile0 sudo mkswap /swapfile0
Add the following line to /etc/fstab:
/swapfile0 none swap sw 0 0
Enable swap file:
sudo swapon -a
Install additional software on enswitchstorage0 and enswitchstorage1 (optional):
sudo apt-get install htop iotop bwm-ng tshark
Add firewall rules on enswitchstorage0 and enswitchstorage1. The following can be used as the base for a firewall script:
iptables -F -m comment --comment "Clear all existing rules" iptables -A INPUT -m state --state RELATED,ESTABLISHED -m comment --comment "Allow packets from related and established connections" -j ACCEPT iptables -A INPUT -i lo -m comment --comment "Allow all on lo interface" -j ACCEPT iptables -A INPUT -p tcp --dport 24007 -m state --state NEW -s 10.0.0.0/24 -m comment --comment "Allow GlusterFS Daemon from Enswitch subnet" -j ACCEPT iptables -A INPUT -p tcp --dport 24008 -m state --state NEW -s 10.0.0.0/24 -m comment --comment "Allow GlusterFS Management from Enswitch subnet" -j ACCEPT iptables -A INPUT -p tcp --dport 49152 -m state --state NEW -s 10.0.0.0/24 -m comment --comment "Allow connections to GlusterFS brick #1 from Enswitch subnet" -j ACCEPT iptables -A INPUT -p tcp --dport 111 -m state --state NEW -s 10.0.0.0/24 -m comment --comment "Allow portmapper from Enswitch subnet" -j ACCEPT iptables -A INPUT -p udp --dport 111 -s 10.0.0.0/24 -m comment --comment "Allow portmapper from Enswitch subnet" -j ACCEPT iptables -A INPUT -m comment --comment "Log all unmatched packets" -j LOG iptables -A INPUT -m comment --comment "Drop all unmatched packets" -j DROP
Configure apt source for GlusterFS 3.5:
echo "deb http://ppa.launchpad.net/gluster/glusterfs-3.5/ubuntu trusty main" | sudo tee /etc/apt/sources.list.d/glusterfs.list echo "deb-src http://ppa.launchpad.net/gluster/glusterfs-3.5/ubuntu trusty main" | sudo tee -a /etc/apt/sources.list.d/glusterfs.list
Create a file named /tmp/glusterfs-repo.key contianing the following:
-----BEGIN PGP PUBLIC KEY BLOCK----- Version: SKS 1.1.4 Comment: Hostname: keyserver.ubuntu.com mQINBFPtYFcBEADcQMZ9aSR1ptbaEeq/8Bzu7kipaxVGh3Wcma4Lz+QPe0ofxRf+oYR22UVG mJcPnVcGFbXJ50t8BAxwtQ/TSmGdQ93bl6LORAQBZ/ud1LTr2HKpaa0F1bwpi/TAgBWqP64H u0LBGISc0G5m3/hn/bi6XxIIOzJ/L/vqLh1deVaDrYYWy5Cme8AuPtqOARKsefvVgwlpnbCt k+QaE65vgl8MXiYCaOenT07GDCq1xb7hkoVlJS4bf6F3UMJVMVy4oEyYkRw4SP7ULeT1s4yr BeDzbxhFaZRJFvGpvMW3AZxfrhX/5OpZSkQiFn5/2j4eJli4/MmptAAHpGr4tLA+s6mHmA9E 9c7wMfyFZe+wMhvangSDp09gSSZs00bqKSnYIJ/oGRjaxCllkw4SMfTOqv8l/GOxRs12yIcZ D08SSmRpoyLffrl1zElyaixtAJRenphTZyq7eRLPyQl6qEDA1XtLs3ThK5/4fghMbe7MOHiM B8MwL1RzLQkl/PU08vxfum9ki/m/LP5xpJopNHZs2L47RlX2+tq6FJWbDvQwOGoFTTnxmdDf 4EkMhlB4N+ujZw64pSMt3c08NShxty2UWpbSbc8/e7Ps4B7Lx6eq6AmqrcUChg8c9+PI2LUq j6mDbc8jxpUslvjsLU05xnq6OLv4U//pUTUz6eI8FgFadVZcoQARAQABtBlMYXVuY2hwYWQg UFBBIGZvciBHbHVzdGVyiQI4BBMBAgAiBQJT7WBXAhsDBgsJCAcDAgYVCAIJCgsEFgIDAQIe AQIXgAAKCRAT4Bt7P+hpqZ3LEACYYC4UjxwSHouV295Cxfwt9P32GcWJbFmLYtLHWVTt2vdN /M9Xb02YgVLJm/nVy2vJhqcMowSW2jO503mLq672g5mHitnIq1lh4zXcHEvP79aDRQuvkgsL EHjlk2NzYqdAsdRk3TgOLcK0SRM7Cwgwd/b/gVUtPYrX1hvQKrjGJM9VZFcCMX2RmGAS0ft3 QHzEAPZCgyamk0qB2eo8tLZYm42iMvq+ZSxGulhzi7gJkpv/wNdaP4E6o8o7KY3JIWMmxBn8 QZUKYMobze4PSBg4G4iG2ue9IrGCb8M1o+46aOSyEIc99bznF8Jrw7a8sBufVRjSZIE9A/oM EtB1pTRDn9lwx/DyYbCV16DOsk6d5x4P8cqvgdaGzl7VNLvkwmMaCH0gRFIBr937rEUbeSJH TqrVG0zXzSaUHEwXPZE0Lt2C9dEmMnT6nxC7FbJB1ATPDNx8kL7MvB4jl5HkjrD1W9Xu2y0d zwAKlg5jvzwP46MJgvm+AYK808XhOhMZjWzzt5POeDcDhGhpRSfQtAhSnRkOtKS1drMCt27h LZDEZfCp//aj7jvVL8FjamGEMfm91FLQa5LY7OoJaYoZlYUtthrXV6w5KHFjFYAKgA8tJzeb Tvc1Q9avCo2G5qWNZq6TSLxHEMo/g4gu2aGRPRrKu9w2Ibosg4OqZ/YbXC8SjA== =+Qna -----END PGP PUBLIC KEY BLOCK-----
Import repository key:
sudo apt-key add /tmp/glusterfs-repo.key
Install GlusterFS packages on enswitchstorage0 and enswitchstorage1:
sudo apt-get update sudo apt-get install glusterfs-server ntp xfsprogs
Download and install xfsprogs from a newer distribution. This is necessary to use the metadata checksum and free inode btree options:
cd /tmp wget http://mirrors.kernel.org/ubuntu/pool/main/x/xfsprogs/xfsprogs_3.2.1ubuntu1_amd64.deb sudo dpkg -i xfsprogs_3.2.1ubuntu1_amd64.deb
Add entries to /etc/hosts for each server on enswitchstorage0 and enswitchstorage1:
10.0.0.122 enswitchstorage0 10.0.0.123 enswitchstorage1
Create partition for the GlusterFS volume on enswitchstorage0 and enswitchstorage1, in this example we use /dev/sda2 as the glusterfs volume:
sudo fdisk sudo mkdir -p /var/glusterfs/sda2
If your filesystem is on a standalone hard drive use the following options:
sudo mkfs.xfs -m crc=1,finobt=1 /dev/sda2
If your filesystem is on a raid device, add options for stripe size and stripe width. Stripe size is whatever was configured in the RAID controller when you create the volume, and stripe with is the number of data disks. See http://xfs.org/index.php/XFS_FAQ#Q:_How_to_calculate_the_correct_sunit.2Cswidth_values_for_optimal_performance for more information. In this example we are using a RAID 10 volume on 6 disks with a 64K stripe size:
sudo mkfs.xfs -m crc=1,finobt=1 -d su=64k,sw=3 /dev/sda2
Mount the GlusterFS partition on enswitchstorage0 and enswitchstorage1.
If the filesystem is smaller than 1TB add the following to /etc/fstab:
/dev/sda2 /var/glusterfs/sda2 xfs defaults 0 0
If the filesystem is 1TB or larger, use the inode64 option instead. This allows inodes to be placed above 1TB which will help avoid issues. See http://xfs.org/index.php/XFS_FAQ#Q:_What_is_the_inode64_mount_option_for.3F for a more in depth explanation:
/dev/sda2 /var/glusterfs/sda2 xfs inode64 0 0
Mount the partition manually:
sudo mount /var/glusterfs/sda2
Create directory for enswitch volume:
sudo mkdir /var/glusterfs/sda2/enswitch
Initialize the peer from enswitchstorage0 This command should return "peer probe: success":
sudo gluster peer probe enswitchstorage1
Check the peer status:
sudo gluster peer status
You should see this:
Number of Peers: 1 Hostname: enswitchstorage1 Port: 24007 Uuid: 2eed9049-4f5c-4e14-8d49-8935df95c9fe State: Peer in Cluster (Connected)
Create GlusterFS volume, the only needs to be run on enswitchstorage0:
sudo gluster volume create enswitch replica 2 transport tcp enswitchstorage0:/var/glusterfs/sda2/enswitch enswitchstorage1:/var/glusterfs/sda2/enswitch
You should see the following if the creation was successful:
volume create: enswitch: success: please start the volume to access data
Start the volume:
sudo gluster volume start enswitch
Look at volume info:
sudo gluster volume info
Volume Name: enswitch Type: Replicate Volume ID: 3276be02-515f-41b1-ab22-a766724c8e64 Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: enswitchstorage0:/var/glusterfs/sda2/enswitch Brick2: enswitchstorage1:/var/glusterfs/sda2/enswitch
Configure settings for better performance:
sudo gluster volume set enswitch performance.cache-size 1GB
Renumber libuuid user and group to free up uid 100 and gid 101. Then change the UID/GID of the libuuid files to match their new groups:
sudo chown libuuid:libuuid /usr/sbin/uuidd sudo chown libuuid:libuuid /var/lib/libuuid
Add Enswitch user and group. There is no Enswitch code on these boxes, but this will make the file ownership show "enswitch:enswitch":
sudo adduser --system --group --no-create-home --home /var/lib/enswitch/home --disabled-password enswitch
Create local mountpoint on enswitchstorage0 and enswitchstorage1 so that the shared files can be accessed locally. NOTE: do not modify any files directly under /var/glusterfs/sda2/enswitch, as this will cause corruption:
sudo mkdir /var/lib/enswitch
Add the following line to /etc/fstab on enswitchstorage0 and enswitchstorage1:
enswitchstorage0:/enswitch /var/lib/enswitch glusterfs direct-io-mode=disable,backupvolfile-server=enswitchstorage1,_netdev 0 0
Mount /var/lib/enswitch on enswitchstorage0 and enswitchstorage1:
sudo mount /var/lib/enswitch sudo chown enswitch:enswitch /var/lib/enswitch sudo chmod 775 /var/lib/enswitch sudo chmod g+s /var/lib/enswitch
Client install
Configure apt source for GlusterFS 3.5
echo "deb http://ppa.launchpad.net/gluster/glusterfs-3.5/ubuntu precise main" | sudo tee /etc/apt/sources.list.d/glusterfs.list echo "deb-src http://ppa.launchpad.net/gluster/glusterfs-3.5/ubuntu precise main" | sudo tee -a /etc/apt/sources.list.d/glusterfs.list
Create a file named /tmp/glusterfs-repo.key containing the following:
-----BEGIN PGP PUBLIC KEY BLOCK----- Version: SKS 1.1.4 Comment: Hostname: keyserver.ubuntu.com mQINBFPtYFcBEADcQMZ9aSR1ptbaEeq/8Bzu7kipaxVGh3Wcma4Lz+QPe0ofxRf+oYR22UVG mJcPnVcGFbXJ50t8BAxwtQ/TSmGdQ93bl6LORAQBZ/ud1LTr2HKpaa0F1bwpi/TAgBWqP64H u0LBGISc0G5m3/hn/bi6XxIIOzJ/L/vqLh1deVaDrYYWy5Cme8AuPtqOARKsefvVgwlpnbCt k+QaE65vgl8MXiYCaOenT07GDCq1xb7hkoVlJS4bf6F3UMJVMVy4oEyYkRw4SP7ULeT1s4yr BeDzbxhFaZRJFvGpvMW3AZxfrhX/5OpZSkQiFn5/2j4eJli4/MmptAAHpGr4tLA+s6mHmA9E 9c7wMfyFZe+wMhvangSDp09gSSZs00bqKSnYIJ/oGRjaxCllkw4SMfTOqv8l/GOxRs12yIcZ D08SSmRpoyLffrl1zElyaixtAJRenphTZyq7eRLPyQl6qEDA1XtLs3ThK5/4fghMbe7MOHiM B8MwL1RzLQkl/PU08vxfum9ki/m/LP5xpJopNHZs2L47RlX2+tq6FJWbDvQwOGoFTTnxmdDf 4EkMhlB4N+ujZw64pSMt3c08NShxty2UWpbSbc8/e7Ps4B7Lx6eq6AmqrcUChg8c9+PI2LUq j6mDbc8jxpUslvjsLU05xnq6OLv4U//pUTUz6eI8FgFadVZcoQARAQABtBlMYXVuY2hwYWQg UFBBIGZvciBHbHVzdGVyiQI4BBMBAgAiBQJT7WBXAhsDBgsJCAcDAgYVCAIJCgsEFgIDAQIe AQIXgAAKCRAT4Bt7P+hpqZ3LEACYYC4UjxwSHouV295Cxfwt9P32GcWJbFmLYtLHWVTt2vdN /M9Xb02YgVLJm/nVy2vJhqcMowSW2jO503mLq672g5mHitnIq1lh4zXcHEvP79aDRQuvkgsL EHjlk2NzYqdAsdRk3TgOLcK0SRM7Cwgwd/b/gVUtPYrX1hvQKrjGJM9VZFcCMX2RmGAS0ft3 QHzEAPZCgyamk0qB2eo8tLZYm42iMvq+ZSxGulhzi7gJkpv/wNdaP4E6o8o7KY3JIWMmxBn8 QZUKYMobze4PSBg4G4iG2ue9IrGCb8M1o+46aOSyEIc99bznF8Jrw7a8sBufVRjSZIE9A/oM EtB1pTRDn9lwx/DyYbCV16DOsk6d5x4P8cqvgdaGzl7VNLvkwmMaCH0gRFIBr937rEUbeSJH TqrVG0zXzSaUHEwXPZE0Lt2C9dEmMnT6nxC7FbJB1ATPDNx8kL7MvB4jl5HkjrD1W9Xu2y0d zwAKlg5jvzwP46MJgvm+AYK808XhOhMZjWzzt5POeDcDhGhpRSfQtAhSnRkOtKS1drMCt27h LZDEZfCp//aj7jvVL8FjamGEMfm91FLQa5LY7OoJaYoZlYUtthrXV6w5KHFjFYAKgA8tJzeb Tvc1Q9avCo2G5qWNZq6TSLxHEMo/g4gu2aGRPRrKu9w2Ibosg4OqZ/YbXC8SjA== =+Qna -----END PGP PUBLIC KEY BLOCK-----
Import repository key:
sudo apt-key add /tmp/glusterfs-repo.key
Install GlusterFS client packages:
sudo apt-get update sudo apt-get install glusterfs-client ntp
Cutover procedure:
On one of the current NFS boxes, mount the GlusterFS volume as /var/lib/enswitch2/ and rsync data:
sudo mkdir /var/lib/enswitch2 sudo mount -o direct-io-mode=disable,backupvolfile-server=enswitchstorage1 enswitchstorage0:/enswitch /var/lib/enswitch2 sudo rsync -av --delete /var/lib/enswitch/ /var/lib/enswitch2/
Unmount /var/lib/enswitch on all servers
sudo umount /var/lib/enswitch
Rsync data one more time on old NFS server:
sudo rsync -av --delete /var/lib/enswitch/ /var/lib/enswitch2/
Add the following line to /etc/fstab on all other servers (commenting out the current line for the /var/lib/enswitch nfs mount):
enswitchstorage0:/enswitch /var/lib/enswitch glusterfs direct-io-mode=disable,backupvolfile-server=enswitchstorage1,_netdev 0 0
Mount new GlusterFS volume on all Enswitch servers:
sudo mount /var/lib/enswitch
Restart enswitch on all servers
sudo enswitch restart
References:
http://www.server-world.info/en/note?os=Ubuntu_14.04&p=glusterfs&f=2
http://www.jamescoyle.net/how-to/457-glusterfs-firewall-rules
http://www.jamescoyle.net/how-to/351-share-glusterfs-volume-to-a-single-ip-address
https://launchpad.net/~gluster
http://www.jamescoyle.net/how-to/559-glusterfs-performance-tuning
http://toruonu.blogspot.com/2012/12/xfs-vs-ext4.html
http://joejulian.name/blog/glusterfs-bit-by-ext4-structure-change/
