Introduction of Systemd

Introduction and work flow of systemd

systemd is a system and service manager and recent Fedora is using systemd (Fedora 18 and above, I think). System startup is managed by systemd now and all original System V (init system for starting individual services) and Upstart service has been deprecated and phased out.

The old System V was using different scripts(daemons) sequentially in different runlevels. For example, System V starts ssh service in runlevel 5 after firewall and network service because ssh service depends on network service. This sequentially running of services make a slow startup and that’s why System V has been phased out.

The new Systemd starts daemons at same time and this parallel running of daemons means a very fast startup. How come? Well, systemd uses sockets for all services. It sets up sockets for daemons and coordinates between them as they start up. So, when a daemon requires support from another daemon, systemd coordinate the data from their sockets . For example, daemon A starts and it needs a support from daemon B, however, daemon B has not started yet . Therefore, systemd writes daemon A’s request to daemon B’s socket(buffering) and it continues. So daemon A doesn’t need to wait for daemon B. As soon as daemon B starts, it will read its socket. This means parallel processing and system boots very fast. The socket activation was designed by Apple’s OS X system.

systemd is compatible with System V and actually systemd uses System V’s configurations and scripts as additional information files. If there is no any configuration file for systemd already, systemd will use System V scripts and configuration files to generate a corresponding configuration files. However, systemd configuration files always take precedence. For example, systemd uses /etc/fstab to generate corresponding systemd configuration file system or looks at /etc/rc.d directory to finds start and stop files and determines dependencies.

Configuration for systemd tasks are defined in unit files. Therefore, systemd defines different units files for different tasks and each unit file has its own configuration file. Unit files are divided to the following categories:

service

target

socket

device
mount
automount
path
snapshot
swap
timer
So, those unit files that are related to “service” have “.service” extension or unit files related to “mount” have“.mount” extension and so forth. An good example can be boot.mount or smb.service.
Now, inside each unit file, there are directives and configuration options. Some of these directives and options are common to all units and you can find it in systemd.unit man page.
man systemd.unit

According to man page, this man page(systemd.unit) lists the common configuration options of all the unit types. These options need to be configured in the [Unit] or [Install] sections of the unit files.

In addition, you can find a list of configuration options shared by services, sockets, mount and swap unit types in systemd.exec man page:
man systemd.exec

The execution specific configuration options are configured in the [Service], [Socket], [Mount], or [Swap]sections in the unit files, depending on the unit type.

Unit file syntax
Well, a very comprehensive explanation exist in man page (man systemd.unit) for your reference. Here is a short, still very useful, explanation of [Unit] section in each unit files. Remember this part is common to all of them. Some common options under [Unit] section are:

[Unit]

Description= A descriptive information along with the unit name

Documentation= A list of man pages referencing for this unit or its configuration

Requires= According to man page, this configures requirement dependencies on other units. If this unit gets activated, the units listed here will be activated as well. If one of the other units gets deactivated or its activation fails, this unit will be deactivated. Note that requirement dependencies do not influence the order in which services are started or stopped. This has to be configured independently with the After= or Before= options.

Wants= A weaker version of Requires=. A unit listed in this option will be started if the configuring unit is. However, if the listed unit fails to start up or cannot be added to the transaction this has no impact on the validity of the transaction as a whole.

Conflicts= Configures negative requirement dependencies. If a unit has a Conflicts= setting on another unit, starting the former will stop the latter and vice versa. 

Before= Configures ordering dependencies between units. If a unit foo.service contains a setting “Before=bar.service” and both units are being started, bar.service's start-up is delayed until foo.service is started up. 

After= It is the inverse of “Before=” 

AllowIsolate= If true this unit may be used with the “systemctl isolate” command. Otherwise this will be refused. 

ConditionPathExists= a file existence condition is checked before a unit is started. If the specified absolute path name does not exist the condition will fail 

ConditionPathIsDirectory= is similar to ConditionPathExists= but verifies whether a certain path exists and is a directory

Some common options under [Install] section are:

[Install] 

Alias=Additional names this unit shall be installed under. The names listed here must have the same suffix (i.e. type) as the unit file name. 

Also= Additional units to install/deinstall when this unit is installed/deinstalled 

WantedBy=, RequiredBy=  This option creates a symbolic link to the unit in a “.wants” subdirectory for the WantedBy unit. For example, assume the you enabled the rsyslog service. “WantedBy=multi-user.target” directive in syslog.service unit file creates a symbolic link in /etc/systemd/system/multi-user.target.wants/rsyslog.service to the actual rsyslog.service. Then, this unit file (rsyslog.service) is wanted/needed by multi-user.target unit file. In other words, all units that multi-user.target wants it would be in multi-user.target.wants directory. As you can see here, all WantsBy symbolic links are setup in the /etc/systemd/system directory. These symbolic links can be changed as you enable/disable a service. if you disable a service, it will remove the link.RequiredBy is a much stronger dependency.

Kernel Virtual Machine

        
Virtual Machine
1.  Creating environment for virtualization:­
a) Install the RHEL FOR 64 Bit.
b) Now check the virtualization flag by running the command
[root@example ~]# egrep '(vmx|svm)' ­­color=always /proc/cpuinfo
c) To install KVM and virtinst (a tool to create virtual machines), run the command
[root@example ~]# yum install  qemu­kvm qemu* libvirt* python­virtinst*
d) Then start the libvirt daemon:
[root@example ~]# /etc/init.d/libvirtd start
e) To check if KVM has successfully been installed, run
[root@example ~]# virsh ­c qemu:///system list
( It should display something like this )
Id Name
State
­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­
2. For graphical installation of vm :­
[root@example ~]# virt­manager
( It is a interface for managing virtual machines. It presents a summary view of running domains,
their live performance & resource utilization statistics. The detailed view graphs performance &
utilization over time.)
Then create new machine by clicking on create a new virtual machine and then entering the
following details :­
a) Name
b) iso path
c) Choosing the Ram and CPU settings
d) Size of disk image on Hard Drive
New Vm will be created.
Error :­  If virt­manager command is not working then run the command
[root@example ~]# dbus­launch ­­exit­with­session virt­manager
3. Textual installation of vm:­
For installing virtual machine by command type:­
For centos/rhel :­
virt­install ­ncentos7 ­r 1024  ­­vcpus=1 ­­disk path=/var/lib/libvirt/images/centos7.img,size=20
­­bridge=virbr1 ­l ftp://192.168.122.1/pub/centos6.3/ ­x 'console=tty0 console=ttyS0,115200n8'
Here,
a) virt­install ­   Command for installing virtual machine
b) ­ncentos7  ­  n is parameter for name and centos7 is name of the vm to be created
c) ­r 1024    ­     Ram of size 1 Gb is allocated to vm
d) ­­disk      ­      Specifies media to use as storage for the guest, with various options:­
  i) path=/var/lib/libvirt/images/centos7.img ­    It is the path on disk where image of this vm is
stored
 ii) size=20   ­    Size of 20 Gb is allocated to this vm
e) ­­bridge=virbr1     ­    Connect to a bridge device in the host
f) ­l ftp://192.168.122.1/pub/centos6.3/    ­   It specifes the location where iso is mounted. Here
centos6.3 iso is mounted on /var/ftp/pub/centos6.3/ and it is ftp path. We can also specify :­
  i) nfs path
 ii) http path
iii) Path to the local directory where iso is mounted
g) ­x 'console=tty0 console=ttyS0,115200n8'   ­   x specifies the extra argument. It is used only
with ­l option. It specifies the serial console kernel parameter which is responsible for textual
installation
For ubuntu:­
/usr/sbin/virt­install ­ncentos7 ­r 1024  ­­vcpus=1  ­­disk
path=/var/lib/libvirt/images/centos7.img,size=20 ­­bridge=virbr1 ­l
http://ca.archive.ubuntu.com/ubuntu/dists/precise/main/installer­amd64/ ­x 'console=tty0
console=ttyS0,115200n8'
We use http location for ubuntu because when we mount the iso virt­inst doesn’t find installer
and give the error:
valueerror could not find an installable distribution at location
4. Assigning IPtables rules to access vm from remote machine using different ssh port which
redirect the query to the particular VM which is define in iptables rule.
To write iptables rule we must know the  mac address. For that 1st time we have to take console
login by typing command
[root@example Downloads]#  Virsh console vmname
Where, vmname is the name given at installation time. Here it is centos7.
Give the username and password entered at installation time
After login note the mac address  and then write init0 as we can exit from console by shutting it
off
Error:­ if after login it doesn’t take ip and mac then check ethtool settings
Now we are on base machine
a) Create a directory /etc/libvirt/hooks
Create qemu file in it /etc/libvirt/hooks/qemu
( This file is executed when a QEMU guest is started, stopped )

b) For vm named centos7 file /etc/libvirt/hooks/centos7  is created which will contain  :­
********************************************************************************************************
Guest_name=centos7
Host_port=
Guest_ipaddr=192.168.122.7
Guest_port=22
if [ "\$1" = "\$Guest_name" ]; then
if [ "\$2" = "stopped" ­o "\$2" = "reconnect" ]; then








iptables ­t nat ­D PREROUTING ­p tcp ­­dport \$Host_port ­j DNAT \
­­to \$Guest_ipaddr:\$Guest_port
iptables ­D FORWARD ­d \$Guest_ipaddr/32 ­p tcp ­m state ­­state NEW \
­m tcp ­­dport \$Guest_port ­j ACCEPT
fi
if [ "\$2" = "start" ­o "\$2" = "reconnect" ]; then
iptables ­t nat ­A PREROUTING ­p tcp ­­dport \$Host_port ­j DNAT \
­­to \$Guest_ipaddr:\$Guest_port
iptables ­I FORWARD ­d \$Guest_ipaddr/32 ­p tcp ­m state ­­state NEW \
­m tcp ­­dport \$Guest_port ­j ACCEPT
fi
fi

********************************************************************************************************
where,
i) Guest_name=centos7    ­­­   Name of the vm
ii)Host_port=2207    ­­­  the port by which it will be accessed from remote machine
iii)Guest_ipaddr=192.168.122.7  ­­ vm’s ip address
Every vm will have separate file including the above data with the name vmname  in
/etc/libvirt/hooks/ which will have executable permission.
Note:­ But for the first vm we have to do entry in qemu file then after this new file will be created
with vmname in /etc/libvirt/hooks/vmname and its source will be defined in qemu file like this:­
­­­> At the end in file  /etc/libvirt/hooks/qemu add this line when any new vm is created
 source  /etc/libvirt/hooks/centos7  ( This is setting for vm named centos7 )
c) Virsh settings
[root@example Downloads]#  virsh net­destroy default
[root@example Downloads]#  virsh net­edit default
Insert this line :­
<host mac='52:54:00:34:11:54' name='centos7’' ip='192.168.122.7' />
Where, mac address is the address being noted and name is hostname and ip we can give by
our choice.
[root@example ~]# virsh net­start default
[root@example ~]# /etc/init.d/libvirtd restart
To see all the vm on base machine type
[root@example ~]# virsh list ­­all
To forcefully shutdown vm type
[root@example ~]# virsh destroy centos7
To start vm type
[root@example ~]# virsh start centos7
Now take the login from remote machine by entering command
[root@example ~]# ssh baseipaddress ­p2207 ­l username
By typing this command from remote machine this will launch us to vm by not connecting to
base machine
( Here base ipaddress is the base machine’s ip address and 2207 is port specified in vm file in
hooks )
5. Cloning of VM
Clone of any vm will make the cloned machine with the same hardware and software
configuration of the original.
→ Command for making clone machine vm2 of the original machine vm1:­
virt­clone ­­original vm1 ­­name vm2 ­­file /var/lib/libvirt/images/vm2.img
­­> We can also define the mac address which matches the range being assigned to machine
by typing command
virt­clone ­­original vm1 ­­name vm2 ­­file /var/lib/libvirt/images/vm2.img ­­mac 52:54:00:34:11:54
6. To change ram and cpu
virsh destroy test
virsh define /etc/libvirt/qemu/test.xml
virsh edit test
virsh start test
7. Script for creating vm clone of the 4 already created machines:­
#!/bin/bash
usage=" Usage: $(basename $0)  [­d <HostName>] [­t <template>]

[­r <ram>] [­c <vcpu>]







[­t ( 1. centos16 ( php version 5.2)
( 2. centos6 ( php version 5.3)
( 3. ubuntu12 ( php version 5.3)
( 4. ubuntu12­5.2 ( php version 5.2)
[­r (optional) need to pass this argument only if you need more ram than 1 gb]
[­c (optional) need to pass this argument only if you need more cpu than 1]
  Note:­ Two Options Are  Mandatory (­d && ­t )
  "
#echo $#
if [ $# ­lt 4 ];then
echo  "$usage"
exit
fi
while getopts hd:t:r:c: opts
do
case "$opts" in
h) echo "$usage"

exit

;;
d) hostname="$OPTARG"
;;
t) template="$OPTARG"

;;
r) ram="$OPTARG"

;;
c) vcpu="$OPTARG"
             ;;


esac
done
# For checking if vm with given name already exists
c=`virsh list ­­all | awk ­F" " '{print $2}'`
for v in $c
do
if [ $v = "$hostname" ];then
echo "vm with name $hostname already exists"
exit;
fi
done
# for checking if conf file already exists of the given hostname
a=`ls /etc/httpd/conf.d/ | awk ­F".conf" '{print $1}'`
for v in $a
do
if [ "$hostname" = "$v" ]; then
echo "site with name $hostname already exists"
exit;
fi
done
if [[ $template = "centos6" ||  $template = "centos16" || $template = "ubuntu12" || $template =
"ubuntu12­5.2" ]];then
mac=` awk 'NR == 1 {print $1}' /etc/vm/notused.txt`
# notused.txt file contains the unused mac address and ipaddress which are given to the cloned
machine
ipaddress=` awk 'NR == 1 {print $2}' /etc/vm/notused.txt `
# after using the addresses they are copied to used.txt
cat >> /etc/vm/used.txt << EOF
$mac, $ipaddress
EOF
port=`awk ­F"." 'NR == 1 {print $4}' /etc/vm/notused.txt`
# the used address is deleted from notused.txt
sed ­i '1d' /etc/vm/notused.txt
newport=22$port
/bin/touch
/bin/chmod
+x
/etc/libvirt/hooks/$hostname
/etc/libvirt/hooks/$hostname
cat >> /etc/libvirt/hooks/$hostname << EOF
Guest_name=$hostname
Host_port=$newport
Guest_ipaddr=$ipaddress
Guest_port=22
if [ "\$1" = "\$Guest_name" ]; then
if [ "\$2" = "stopped" ­o "\$2" = "reconnect" ]; then

iptables ­t nat ­D PREROUTING ­p tcp ­­dport \$Host_port ­j DNAT \

­­to \$Guest_ipaddr:\$Guest_port

iptables ­D FORWARD ­d \$Guest_ipaddr/32 ­p tcp ­m state ­­state NEW \

­m tcp ­­dport \$Guest_port ­j ACCEPT
fi
if [ "\$2" = "start" ­o "\$2" = "reconnect" ]; then

iptables ­t nat ­A PREROUTING ­p tcp ­­dport \$Host_port ­j DNAT \

­­to \$Guest_ipaddr:\$Guest_port

iptables ­I FORWARD ­d \$Guest_ipaddr/32 ­p tcp ­m state ­­state NEW \

­m tcp ­­dport \$Guest_port ­j ACCEPT
fi
fi
EOF
cat >> /etc/libvirt/hooks/qemu << EOF
source  /etc/libvirt/hooks/$hostname
EOF
/usr/bin/virsh destroy $template
/usr/bin/virt­clone ­­original $template ­­name $hostname ­­file
/var/lib/libvirt/images/$hostname.img ­­mac $mac
if [ ! ­z $vcpu ]; then
sed ­i "s/<vcpu>1<\/vcpu>/<vcpu>$vcpu<\/vcpu>/g" /etc/libvirt/qemu/$hostname.xml
fi
if [ ! ­z $ram  ]; then
sed ­i "s/<memory>1048576<\/memory>/<memory>$ram"048576"<\/memory>/g"
/etc/libvirt/qemu/$hostname.xml
sed ­i
"s/<currentMemory>1048576<\/currentMemory>/<currentMemory>$ram"048576"<\/currentMem
ory>/g" /etc/libvirt/qemu/$hostname.xml
fi
/usr/bin/virsh start $template
## creating virtual host file on base machine to transfer the request for $hostname to vm
/bin/touch /etc/httpd/conf.d/$hostname.conf
cat >> /etc/httpd/conf.d/$hostname.conf << EOF
<VirtualHost *:80>
ServerName $hostname.keenable.com
#ProxyRequests off
ProxyVia Full
ProxyPreserveHost On
ProxyRequests off
Proxypass /  http://$ipaddress:80/
ProxypassReverse  / http://$ipaddress:80/
</VirtualHost>
EOF
/etc/init.d/httpd reload
/usr/bin/virsh start $hostname
echo  "
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++"
echo   " NOTE:­ your machine has been created with portname ­ $newport and you can ssh
"
echo   " ssh 192.168.100.55 ­l vm ­p$newport  and passwd is vm
"
echo   " you can change password of vm sudo (user) after login
"
echo   "
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++"
else
echo "=============  ­t must have valid option ================"
echo
echo "$usage"
exit
fi
Running the script:­
sh script.sh ­d vmname ­t template
Here, vmname can be any name that you want to give and template is the already created vm to
be cloned.

GFS2 File System Administration

 Q. What is the Purpose of LVM in Clustered Environment and What are the Limitations?
The Red Hat GFS2 file system is a native file system that interfaces directly with the Linux kernel file system interface (VFS layer). Red Hat supports the use of GFS2 file systems only as implemented in Red Hat Cluster Suite.

GFS2 is based on a 64-bit architecture, which can theoretically accommodate an 8 EB file system. However, the current supported maximum size of a GFS2 file system is 25 TB.
 

Important Note from Redhat :
 Although a GFS2 file system can be implemented in a standalone system or as part of a cluster configuration, for the Red Hat Enterprise Linux 6 release Red Hat does not support the use of GFS2 as a single-node file system. Red Hat does support a number of high-performance single node file systems which are optimized for single node and thus have generally lower overhead than a cluster file system. Red Hat recommends using these file systems in preference to GFS2 in cases where only a single node needs to mount the file system. Red Hat will continue to support singlenode GFS2 file systems for mounting snapshots of cluster file systems (for example, for backup purposes).

Limitations of Traditional Filesystems ( ext2/ ext3 /ext4) in Clustered Environment

Step 1
: On node1, creating a ext4 filesystem on top of shared storage manged by CLVM

[root@node1 ~]# mkfs.ext4 /dev/vgCLUSTER/clvolume1

mke2fs 1.41.12 (17-May-2010)

Filesystem label=

OS type: Linux

Block size=1024 (log=0)

Fragment size=1024 (log=0)

Stride=0 blocks, Stripe width=0 blocks

25688 inodes, 102400 blocks

5120 blocks (5.00%) reserved for the super user

First data block=1

Maximum filesystem blocks=67371008

13 block groups

8192 blocks per group, 8192 fragments per group

1976 inodes per group

Superblock backups stored on blocks:

8193, 24577, 40961, 57345, 73729

Writing inode tables: done

Creating journal (4096 blocks): done

Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 37 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override.

Step 2

: mounting new filesystems on /clvmnt mount point

[root@node1 ~]# mkdir /clvmnt

[root@node1 ~]# mount /dev/vgCLUSTER/clvolume1 /clvmnt

 

Step 3

: On node2, directly mounting the newly created filesystem to the mount point /clvmnt

[root@node2 ~]# mkdir /clvmnt

[root@node2 ~]# mount /dev/vgCLUSTER/clvolume1 /clvmnt

Step 4

: on node1, creating a test directory under the mount point

[root@node1 ~]# cd /clvmnt

[root@node1 clvmnt]# mkdir test-under-ext4-fs

[root@node1 clvmnt]# ls -l

total 14

drwx——. 2 root root 12288 Sep 21 12:53 lost+found

drwxr-xr-x. 2 root root 1024 Sep 21 12:54 test-under-ext4-fs

Step 5

: on node2, verify, if the newly created directory visible. But it doesn’t appear

[root@node2 ~]# cd /clvmnt
[root@node2 clvmnt]# ls
lost+found

Step 6

: on gurlulclu-node2, just unmount and remount volume to reflect the actual content of the

filesystem.
 

[root@node2 ~]# umount /clvmnt

[root@node2 ~]# mount /dev/vgCLUSTER/clvolume1 /clvmnt

[root@node2 ~]# ls -l /clvmnt

total 14

drwx——. 2 root root 12288 Sep 21 12:53 lost+found

drwxr-xr-x. 2 root root 1024 Sep 21 12:54 test-under-ext4-fs

Note :

There are no supported tool/command line option to convert ext3/ext4 file system to GFS/GFS2

or existing shared GFS/GFS2 file system to ext3/ext4 file system. The only way is to, take backup of data resides at ext3 file system , create new GFS/GFS2 volume per the requirement and restore the data back to GFS/GFS2 file system.

 

                                          Inital GFS2 Filesystem Configuration

 

Identifying Required information

The Syntax to Create a GFS Filesystem is as Below

mkfs.gfs2 -p lock_dlm -t ClusterName:FSName -j NumberJournals BlockDevice

[root@node1 ~]# mkfs.gfs2 -h

Usage:

mkfs.gfs2 [options] <device> [ block-count ]

Options:

-b <bytes> Filesystem block size

-c <MB> Size of quota change file

-D Enable debugging code

-h Print this help, then exit

-J <MB> Size of journals

-j <num> Number of journals

-K Don’t try to discard unused blocks

-O Don’t ask for confirmation

-p <name> Name of the locking protocol

-q Don’t print anything

-r <MB> Resource Group Size

-t <name> Name of the lock table

-u <MB> Size of unlinked file

-V Print program version information, then exit

 

Before proceeding to creating GFS2 file system we need to Gather Below information that is required to create a new GFS file systems.

 

Identify Required Block Size: Default (4K) Blocks Are Preferred

As of the Red Hat Enterprise Linux 6 release, the mkfs.gfs2 command attempts to estimate an optimal block size based on device topology. In general, 4K blocks are the preferred block size because 4K is the default page size (memory) for Linux.

 

Identify Required No. of Journals required to Create the File system.

GFS2 requires one journal for each node in the cluster that needs to mount the file system. For example, if you have a 16-node cluster but need to mount only the file system from two nodes, you need only two journals. If you need to mount from a third node, you can always add a journal with the gfs2_jadd command. With GFS2, you can add journals on the fly.

 

Identify Required Journal Size: Default (128MB) Is Usually Optimal

When you run the mkfs.gfs2 command to create a GFS2 file system, you may specify the size of the journals. If you do not specify a size, it will default to 128MB, which should be optimal for most applications.

 

Identify Required Size and Number of Resource Groups

When a GFS2 file system is created with the mkfs.gfs2 command, it divides the storage into uniform slices known as resource groups. It attempts to estimate an optimal resource group size (ranging from 32MB to 2GB). You can override the default with the -r option of the mkfs.gfs2 command.

 

Identify the Cluster name

Below is my current configuration, and we can see my cluster is named as “GFSCluster”

[root@node1 ~]# clustat

Cluster Status for GFSCluster @ Sat Sep 21 13:15:04 2013

Member Status: Quorate

Member Name ID Status

——— — ——node1 1 Online, Local, rgmanager

node2 2 Online, rgmanager

Service Name Owner (Last) State

—— — —–———–service:HAwebService (node2) stopped

 

Identify Required Locking Protocol

We will use “lock_dlm” protocol in this case

 

Understand GFS2 Node Locking Mechanisim:

The difference between a single node file system and GFS2, then, is that a single node file system has a single cache and GFS2 has a separate cache on each node. In both cases, latency to access cached data is of a similar order of magnitude, but the latency to access uncached data is much greater in GFS2 if another node has previously cached that same data. In order to get the best performance from a GFS2 file system, each node has its own page cache which may contain some portion of the on-disk data. GFS2 uses a locking mechanism called glocks (pronounced gee-locks) to maintain the integrity of the cache between nodes. The glock subsystem provides a cache management function which is implemented using the distributed lock manager (DLM) as the underlying communication layer.

The glocks provide protection for the cache on a per-inode basis, so there is one lock per inode which is used for controlling the caching layer. If that glock is granted in shared mode (DLM lock mode: PR) then the data under that glock may be cached upon one or more nodes at the same time, so that all the nodes may have local access to the data. If the glock is granted in exclusive mode (DLM lock mode: EX) then only a single node may cache the data under that glock. This mode is used by all operations which modify the data (such as the write system call). If another node requests a glock which cannot be granted immediately, then the DLM sends a message to the node or nodes which currently hold the glocks blocking the new request to ask them to drop their locks. Dropping glocks can be (by the standards of most file system operations) a long process. Dropping a shared glock requires only that the cache be invalidated, which is relatively quick and proportional to the amount of cached data. Dropping an exclusive glock requires a log flush, and writing back any changed data to disk, followed by the invalidation as per the shared glock.

 

                                             Configuration of GFS2 File system

Step 1:

Create GFS2 files system suing the following parameters.

clustername as GFSCluster

gfs2fs as filesystem name

/dev/vgCLUSTER/clvolume1 as device path

number of device journals as 2

locking mechanisam as lock_dlm

 

[root@node1 ~]# mkfs.gfs2 -j 2 -p lock_dlm -t GFSCluster:gfs2fs /dev/vgCLUSTER/clvolume1

This will destroy any data on /dev/vgCLUSTER/clvolume1.

It appears to contain: symbolic link to `../dm-6′

Are you sure you want to proceed? [y/n] y

Device: /dev/vgCLUSTER/clvolume1

Blocksize: 4096

Device Size 0.49 GB (128000 blocks)

Filesystem Size: 0.49 GB (127997 blocks)

Journals: 2

Resource Groups: 2

Locking Protocol: “lock_dlm”

Lock Table: “GFSCluster:gfs2fs”

UUID: ee264aa4-6ab9-18ed-c994-eb577c28399a

Note:

In the above output it is showing Resource Groups as 2 ,that means total 500M volume divided into two RGs of each ~25M each. Size of RG information is important when you are planning for the file system expansion. You should minimum allocate atleast the size of one RG, whenever you want to grow the file system using gfs2_grow.

 

 

Step 2

: Configure /etc/fstabl to mount the newly create gfs2 filesystem automatically during boot

[root@node1 ~]# cat /etc/fstab

## Created by anaconda on Thu Sep 5 14:06:56 2013

##Accessible filesystems, by reference, are maintained under ‘/dev/disk’

##See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info

/dev/mapper/vg_gfsnode1-lv_root                            /          ext4     defaults     1  1

UUID=7263f27d-afbc-4a07-bf85-4354fa0651f9    /boot    ext4      defaults     1 2

/dev/mapper/vg_gfsnode1-lv_swap                          swap    swap    defaults     0 0

tmpfs                                                                      /dev/shm tmpfs    defaults     0 0

/dev/vgCLUSTER/clvolume1                                 /clvgfs     gfs2     defaults     0 0

[root@node1 ~]# mkdir /clvgfs

 

Step 3:

Make sure GFS2 and DLM modules are already loaded in the kernel, if not load with modprobe command.

[root@node1 ~]# lsmod |grep gfs

gfs2 545168 2
dlm 148231 32 gfs2
configfs 29538 2 dlm

Step 4:
Start the GFS2 Server
 

[root@node1 ~]# service gfs2 restart

Mounting GFS2 filesystem (/clvgfs): invalid device path “/dev/vgCLUSTER/clvolume1″ [FAILED]service Start up failed, so below is little troubleshooting to fix the problem.Troubleshooting check 1: :LVS doesn’t show volumes

[root@node1 ~]# lvs

connect() failed on local socket: Connection refused
Internal cluster locking initialisation failed.
WARNING: Falling back to local file-based locking.
Volume Groups with the clustered attribute will be inaccessible.
Skipping clustered volume group vgCLUSTER
Skipping volume group vgCLUSTER

LV   VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert
lv_root vg_gfsnode1 -wi-ao —3.66g
lv_swap vg_gfsnode1 -wi-ao —3.84g
Troubleshooting check 2: : There are no device path exists

[root@node1 ~]# ls “/dev/vgCLUSTER/clvolume1″

ls: cannot access /dev/vgCLUSTER/clvolume1: No such file or directory
Troubleshooting check 3: : There are no Physical PVs exists

[root@node1 ~]# pvs

connect() failed on local socket: Connection refused
Internal cluster locking initialisation failed.
WARNING: Falling back to local file-based locking.
Volume Groups with the clustered attribute will be inaccessible.
Skipping clustered volume group vgCLUSTER
Skipping volume group vgCLUSTER
PV              VG                   Fmt     Attr    PSize     PFree
/dev/vda2    vg_gfsnode1   lvm2    a–       7.51g     0

Troubleshooting check 4:: All that caused because of CLVMD not running , so I just started it.

[root@node1 ~]# service clvmd start
Starting clvmd:
Activating VG(s): 1 logical volume(s) in volume group “vgCLUSTER” now active
2 logical volume(s) in volume group “vg_gfsnode1″ now active [ OK ]
Troubleshooting check 5 :: Now All volumes, and PVS back online.

[root@node1 ~]# pvs

PV                                                  VG                      Fmt       Attr      PSize           PFree

/dev/mapper/1IET_00010002p1    vgCLUSTER      lvm2      a–        780.00m      280.00m

/dev/vda2                                        vg_gfsnode1       lvm2      a–        7.51g            0

[root@node1 ~]# lvs

LV              VG                        Attr      LSize        Pool Origin Data% Move Log Cpy%Sync Convert

clvolume1   vgCLUSTER       -wi-a    500.00m

lv_root        vg_gfsnode1        -wi-ao   3.66g

lv_swap       vg_gfsnode1        -wi-ao   3.84g

 

[root@node1 ~]# chkconfig clvmd on Start the GFS2 service again, now the service started fine.

 
[root@node1 ~]# service gfs2 restart

Mounting GFS2 filesystem (/clvgfs): [ OK ]

Step 5:

on node1, Check that the Volume Mounted properly

[root@node1 ~]# df -h /clvgfs

Filesystem                                                Size      Used      Avail     Use%     Mounted on
/dev/mapper/vgCLUSTER-clvolume1    500M    259M    242M     52%       /clvgfs

 

Step 6 :

On node2, make entries to /etc/fstab, GFS2 & DLM module, Start service and check the volume mount .

[root@node2 ~]# cat /etc/fstab
# /etc/fstab
# Created by anaconda on Thu Sep 5 14:20:11 2013
# Accessible filesystems, by reference, are maintained under ‘/dev/disk’
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info

/dev/mapper/vg_gfsnode1-lv_root                                  /                ext4              defaults        1 1
UUID=18493b92-f8aa-4107-8e63-eaa75a8c3f01          /boot         ext4              defaults        1 2
/dev/mapper/vg_gfsnode1-lv_swap                                swap         swap             defaults        0 0
/dev/vgCLUSTER/clvolume1                                        /clvgfs        gfs2              defaults       0 0
 

[root@node2 ~]# lsmod |grep gfs

gfs2 545168 0
dlm 148231 27 gfs2
configfs 29538 2 dlm

[root@node2 ~]# service gfs2 restart

Mounting GFS2 filesystem (/clvgfs): [ OK ]

 

[root@node2 ~]# df -Ph /clvgfs

Filesystem                                                  Size          Used       Avail      Use%     Mounted on

/dev/mapper/vgCLUSTER-clvolume1      500M        259M     242M      52%       /clvgfs

 

 

                                                   Verifying the Functioning of GFS2

Step 1

: On node1,create a directory under /clvgfs

[root@node1 clvgfs]# ls -lR /clvgfs

/clvgfs:

total 8

drwxr-xr-x. 2 root root 3864 Sep 21 15:07 test-gfs-fs-data

/clvgfs/test-gfs-fs-data:

total 24

-rw-r–r–. 1 root root 0 Sep 21 15:07 a

-rw-r–r–. 1 root root 0 Sep 21 15:07 b

-rw-r–r–. 1 root root 0 Sep 21 15:07 c

 

 

Step 2

: On, node2, verify the content of /clvgfs with remount

[root@node2 ~]# ls -lR /clvgfs

/clvgfs:

total 8

drwxr-xr-x. 2 root root 3864 Sep 21 15:07 test-gfs-fs-data

/clvgfs/test-gfs-fs-data:

total 24

-rw-r–r–. 1 root root 0 Sep 21 15:07 a

-rw-r–r–. 1 root root 0 Sep 21 15:07 b

-rw-r–r–. 1 root root 0 Sep 21 15:07 c

And , we see it them as it is. That concludes data visibility is real time on all the cluster nodes accessing the shared storage.

we have discussed about the introduction and initial configuration of GFS2. And now we will go through the administration tasks related to GFS2 file systems. And we doesn’t cover entire admin tasks related to GFS2 but covers major day-to-day operations related to GFS2 in Red hat Clustered environment.

Below are the Tasks I will be discussing:

 

1. GFS2 File System Expansion

2. GFS2 Journal Addition

3. Suspending and Resuming Writing Activities on GFS2 file systems

4. Repairing GFS2 file system

                                                       GFS2 File System Expansion

Step 1:

Check The Current File system Size, LV size and PV size

[root@node1 test-gfs-fs-data]# df -h /clvgfs

Filesystem                                                               Size      Used      Avail   Use%     Mounted on
/dev/mapper/vgCLUSTER-clvolume1                   500M    259M    242M   52%       /clvgfs

[root@node1 test-gfs-fs-data]# vgs

VG                   #PV   #LV   #SN     Attr          VSize           VFree
vgCLUSTER      1        1         0      wz–nc       780.00m       280.00m
vg_gfsnode         1        1         20    wz–n-        7.51g            0

[root@node1 test-gfs-fs-data]# lvs

LV               VG                      Attr             LSize Pool Origin Data% Move Log Cpy%Sync Convert
clvolume1    vgCLUSTER     -wi-ao—     500.00m
lv_root         vg_gfsnode1       -wi-ao—     3.66g
lv_swap       vg_gfsnode1       -wi-ao—     3.84g

Step 2:

Extend the LV by 200MB

[root@node1 test-gfs-fs-data]# lvextend -L +200M /dev/mapper/vgCLUSTER-clvolume1

Extending logical volume clvolume1 to 700.00 MiB

Logical volume clvolume1 successfully resized

Step 3:

Extend the GFS2 filesystem using the gfs2_grow command.

[root@node1 test-gfs-fs-data]# gfs2_grow /clvgfs

Error: The device has grown by less than one Resource Group (RG). The device grew by 200MB. One RG is 249MB

for this file system. gfs2_grow complete.

Note:

Remembered the point we discussed duringthe GFS2 filesystem creation, the command output was shown as Resource Groups = 2,that means total 500M volume divided into two RGs of each ~250M each. We should minimum allocate atleast the size of one RG, whenever we want to grow the filesystem using gfs2_grow.

[root@node1 test-gfs-fs-data]# df -h /clvgfs

Filesystem Size Used Avail Use% Mounted on

/dev/mapper/vgCLUSTER-clvolume1 500M 259M 242M 52% /clvgfs
Since gfs2_grow failes, extend the LV by 50MB more, and the try the gfs2_grow again

[root@node1 test-gfs-fs-data]# lvextend -L +50M /dev/mapper/vgCLUSTER-clvolume1

Rounding size to boundary between physical extents: 52.00 MiB
Extending logical volume clvolume1 to 752.00 MiB
Logical volume clvolume1 successfully resized

[root@node1 test-gfs-fs-data]# gfs2_grow /clvgfs

FS: Mount Point: /clvgfs
FS: Device: /dev/dm-6
FS: Size: 127997 (0x1f3fd)
FS: RG size: 63988 (0xf9f4)
DEV: Size: 192512 (0x2f000)
The file system grew by 252MB.
gfs2_grow complete.
Now we see that file system extended to 750MB.
Step4 : on, node2 , Check the filesysem size.

[root@node2 test-gfs-fs-data]# df -h /clvgfs
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vgCLUSTER-clvolume1 750M 259M 492M 35% /clvgfs

                              We can confirm that new filesystem size is visible.

                                           Repairing GFS2 Filesystems

Few points to remember:

1. We should not enable FSCK during boot time in /etc/fstab options

2. The fsck.gfs2 command must be run only on a file system that is unmounted from all nodes.
3. Pressing Ctrl+C while running the fsck.gfs2 interrupts processing and displays a prompt asking whether you would like to abort the command, skip the rest of the current pass, or continue processing.
4. You can increase the level of verbosity by using the -v flag. Adding a second -v flag increases the level again.
5. You can decrease the level of verbosity by using the -q flag. Adding a second -q flag decreases the level again.
6. The -n option opens a file system as read-only and answers no to any queries automatically. The option provides a way of trying the command to reveal errors without actually allowing the fsck.gfs2 command to take effect.

[root@node1 ~]# fsck.gfs2 -y /dev/mapper/vgCLUSTER-clvolume1

Initializing fsck
Validating Resource Group index.
Level 1 rgrp check: Checking if all rgrp and rindex values are good.
(level 1 passed)
Starting pass1
Pass1 complete
Starting pass1b
Pass1b complete
Starting pass1c
Pass1c complete
Starting pass2
Pass2 complete
Starting pass3
Pass3 complete
Starting pass4
Pass4 complete
Starting pass5
Block 191984 (0x2edf0) bitmap says 0 (Free) but FSCK saw 1 (Data)
Metadata type is 1 (data)
Fixed.
RG #127997 (0x1f3fd) free count inconsistent: is 63984 should be 63983
Resource group counts updated
Pass5 complete
The statfs file is wrong:
Current statfs values:
blocks: 191956 (0x2edd4)
free: 92656 (0x169f0)
dinodes: 24 (0~18)
Calculated statfs values:
blocks: 191956 (0x2edd4)
free: 92655 (0x169ef)
dinodes: 24 (0~18)
The statfs file was fixed.
Writing changes to disk
gfs2_fsck complete 

Redhat Linux - How to for GFS Filesystems

You can use either of the following formats to create a clustered GFS file system:
#gfs_mkfs -p lock_dlm -t ClusterName:FSName -j Number BlockDevice
#mkfs -t gfs -p lock_dlm -t LockTableName -j NumberJournals BlockDevice

You can use either of the following formats to create a local GFS file system:
#gfs_mkfs -p lock_nolock -j NumberJournals BlockDevice
#mkfs -t gfs -p lock_nolock -j NumberJournals BlockDevice

At each node, mount the GFS file systems. For more information about mounting a GFS file
Command usage:
mount BlockDevice MountPoint
mount -o acl BlockDevice MountPoint
The -o aclmount option allows manipulating file ACLs. If a file system is mounted without the -o acl mount option, users are allowed to view ACLs (with getfacl), but are not allowed to set them (with setfacl).

Formatting the logical Volume
#gfs_mkfs -p lock_dlm -t alpha:mydata1 -j 8 /dev/vg01/lvol0

The gfs_jadd command must be run on mounted file system, but it only needs to be run on one node in the cluster. All the other nodes sense that the expansion has occurred.

#lvextend -L +80G /dev/vgcl_gfs_san_eos/vol01_lv
#gfs_grow -v /dev/vgcl_gfs_san_eos/vol01_lv /db/eospr1/vol01

How To Increase GFS and LVM Size

It is assumed that you have configured the storage so that all systems in the cluster can see it prior to continuing through these steps. If you have not, then please see either the iSCSI or SAN How To.

################################################################
LVM Configuration

1. Initialize LUNs with pvcreate..

iSCSI/non-multipath:

pvcreate /dev/sd SAN/multipath:

pvcreate /dev/dm-<0-9> 2. Add a device into the Volume group

vgextend vgcl_gfs__

3. Sync the LVM Metadata
clvmd -R

4. Extend the size of the Logical Volume

lvextend -L +100G /dev/vgcl_gfs__/gfslv

[root ~]# lvextend -L +73G /dev/vgcl_gfs_san_cus/cuslv

Extending logical volume cuslv to 85.00 GB
Logical volume cuslv successfully resized5. Sync the LVM Metadata

clvmd -R

#######################################################################
GFS Configuration

6. Grow the existing GFS file system into the additional space

gfs_grow -v (device|mount)

[root ~]# gfs_grow -v /dev/mapper/vgcl_gfs_san_cus/cuslv /gfs/cus
Preparing to write new FS information...
Done.
Device has grown by less than 100 blocks.... skipping

This TEST will not do anything, but tell you what device and mount point to use.

[root ~]# gfs_grow -Tv /dev/mapper/vgcl_gfs_san_cus/cuslv /gfs/cus

Scanning SCSI DISKS in Redhat Linux

Scanning SCSI DISKS in Redhat Linux

SCAN AND CONFIGURE NEW LUNS on RHEL 5 and RHEL 6

If you have 4 fc ports on which you have assigned the storage luns.
Then you need to run the below for loop to scan the LUNS on all the 4 ports.

for i in host1 host2 host3 host0
> do
> echo "1" > /sys/class/fc_host/$i/issue_lip
> echo "- - -" > /sys/class/scsi_host/$i/scan
> done

After this you can run fdisk -l or multipath -ll to see the new luns. Once the new luns are visible , you can use the luns using LVM or raw paritioning.

IF you want to rescan the disk lun then you can run the below command.


echo 1 > /sys/block/sda/device/rescan
echo 1 > /sys/block/sdb/device/rescan
echo 1 > /sys/block/sdc/device/rescan
echo 1 > /sys/block/sdd/device/rescan