Explanation of the output of the "ntpq -p" command ?

■ Requirement : Explain output of  "ntpq -p"
■ OS Environment : Linux[RHEL, Centos]
■ Application: ntpq
■ Resolution  : 

$ ntpq -p

remote refid st t when poll reach delay offset jitter

*10.1.0.9 LOCAL(0) 11 u 186 512 377 123.355 45.573 8.832
[...]

Columns Defined:
remote: peers specified in the ntp.conf file
* = current time source
# = source selected, distance exceeds maximum value
o = source selected, Pulse Per Second (PPS) used
+ = source selected, included in final set
x = source false ticker
. = source selected from end of candidate list
- = source discarded by cluster algorithm
blank = source discarded high stratum, failed sanity

refid: remote source’s synchronization source
stratum: stratum level of the source

t: types available
l = local (such as a GPS, WWVB)
u = unicast (most common)
m = multicast
b = broadcast
- = netaddr

when: number of seconds passed since last response
poll: polling interval, in seconds, for source
reach: indicates success/failure to reach source, 377 all attempts successful
delay: indicates the roundtrip time, in milliseconds, to receive a reply
offset: indicates the time difference, in milliseconds, between the client server and source
disp/jitter: indicates the difference, in milliseconds, between two samples

Setup an iSCSI target using tgtadm in Red Hat Enterprise Linux 5?

■ Requirement : Setup an iSCSI target using tgtadm in Red Hat Enterprise Linux 5
■ OS Environment : Linux[RHEL, Centos]
■ Application: tgtadm
■ Implementation Steps : 

        SCSI uses a client-server architecture. A "client" (ie: your system) is an initiator, it initiates requests. A "server" (ie: your storage device) is a target, it has something you want and answers requests from the initiator(s). Initiators come in two varieties: software and hardware. A software initiator is just a driver that handles all requests and pairs the network interfaces driver and the SCSI drivers together to make it all work. Using a software initiator any system with an ethernet card can act as an iSCSI initiator. A hardware initiator is an iSCSI HBA, which is basically just an ethernet card with a SCSI ASIC on-board to offload all the work from the system CPU.

1. Install package : 

$yum install scsi-target-utils -y

2. Start the tgtd service:

$ service tgtd start

3. Define an iSCSI target name:

$ tgtadm --lld iscsi --op new --mode target --tid=1 --targetname iqn.2001-04.com.example:storage.disk1.amiens.sys1.xyz

4. To view the currect configuration:

$tgtadm --lld iscsi --op show --mode target

DEMO :

 here we create a 512M file as the block device:

$ dd if=/dev/zero of=/disk1 bs=1M count=512

Add it as a logical unit to the target:

$ tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /disk1

Enable the target to accept any initiators:

$ tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL

View the current configuration again:

$ tgtadm --lld iscsi --op show --mode target

Target 1: iqn.2001-04.com.example:storage.disk1.amiens.sys1.xyz

System information:
Driver: iscsi
Status: running
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: deadbeaf1:0
SCSI SN: beaf10
Size: 0
Backing store: No backing store
LUN: 1
Type: disk
SCSI ID: deadbeaf1:1
SCSI SN: beaf11
Size: 55G
Backing store: /disk1
Account information:
ACL information:
ALL

Ensure TCP port 3260 can be accessed from remote host.

Start from Red Hat Enterprise Linux 5.3, tgt-admin and /etc/tgt/targets.conf can be used for make the setup persistent over reboot:

$tgt-admin --dump > /etc/tgt/targets.conf
$  chkconfig tgtd on

In releases prior to Red Hat Enterprise Linux 5.3, you have to put the "tgtadm" commands into /etc/rc.local and enable the tgtd service:

echo >> /etc/rc.local < tgtadm --lld iscsi --op new --mode target --tid=1 --targetname iqn.2001-04.com.example:storage.disk1.amiens.sys1.xyz
tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /disk1
tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL
EOF

$ chkconfig tgtd on

Configuring iSCSI initiator on rhel 5

SCSI uses a client-server architecture. A "client" (ie: your system) is an initiator, it initiates requests. A "server" (ie: your storage device) is a target, it has something you want and answers requests from the initiator(s). Initiators come in two varieties: software and hardware. A software initiator is just a driver that handles all requests and pairs the network interfaces driver and the SCSI drivers together to make it all work. Using a software initiator any system with an ethernet card can act as an iSCSI initiator. A hardware initiator is an iSCSI HBA, which is basically just an ethernet card with a SCSI ASIC on-board to offload all the work from the system CPU.

First, install the iscsi-initiator-utils package using:

# yum install iscsi-initiator-utils

Next, start the iscsid service and enable it to start when the system boots:

# service iscsid start
# chkconfig iscsid on

Then obtain a listing of available targets from a given host (please note that ipaddress listed below must be replaced with the resolvable hostname or ip address of the system providing the port if different than default):

# iscsiadm -m discovery -t sendtargets -p ipaddress
xxx.xxx.xxx.xxx:3260,1 iqn.2010-03.com.example:tgtd

Here the "iqn.2010-03.com.example:tgtd" is the target_iqn_name.

You can view the information of your target by:

# iscsiadm -m node -T -p ipaddress

Finally, use the following command to login to the available target (again, replace target_iqn_name with one from the output of the previous command and replace ipaddress below with that of the target system):

# iscsiadm -m node -T target_iqn_name -p ipaddress -l
Logging in to [iface: default, target: iqn.2010-03.com.example:tgtd, portal: xxx.xxx.xxx.xxx,3260]
Login to [iface: default, target: iqn.2010-03.com.example:tgtd, portal: xxx.xxx.xxx.xxx,3260]: successful

Note, you can log into all the LUNs exported on the target by running:

# iscsiadm -m discovery -t st -l

This will enable the target(or targets if you logged into all) to be accessed upon reboots and it stores it in a node database found in /var/lib/iscsi that is more thoroughly described in the /usr/share/doc/iscsi-initiator-utils-VERSION/README file.

If you want to disable the target, you need log out by:

# iscsiadm -m node -T target_iqn_name -p ipaddress -u

Note, you can log out of all targets by running:

# iscsiadm -m node -U all

For a more verbose listing of possible options for the iscsiadm command in Red Hat Enterprise Linux 5 refer to the manual page using:

# man iscsiadm


For a good overview of iSCSI in Red Hat Enterprise Linux 5 refer to the documentation provided in /usr/share/doc/iscsi-initiator-utils-VERSION/README.

try:)

Configuring iSCSI initiator with multipathing ?

For configurations where both paths to the iSCSI target travel over different networks-or-subnets

1. Config the first path through one of you network interfaces:

# service iscsid start
# chkconfig iscsid on
# iscsiadm -m discovery -t st -p -P 1
# iscsiadm -m discovery -t st -p -l

2. After logging into the target you should see new SCSI block devices created, verify this by executing fdisk -l:

# partprobe
# fdisk -l
3. Config the second path through eth1:

# iscsiadm -m discovery -t st -p -P 1
# iscsiadm -m discovery -t st -p -l

For configurations where both paths to the iSCSI target travel over the same network and subnet
1. Configure the iSCSI interfaces by creating iSCSI iface bindings for all interfaces and binding by network device name (eth0, alias, vlan name, etc) or MAC address:

# service iscsid start
# chkconfig iscsid on
# iscsiadm -m iface -I iscsi-eth0 -o new
# iscsiadm -m iface -I iscsi-eth0 -o update -n iface.net_ifacename -v eth0
# iscsiadm -m iface -I iscsi-eth1 -o new
# iscsiadm -m iface -I iscsi-eth1 -o update -n iface.net_ifacename -v eth1

2. Next, verify your targets are available and log in:

# iscsiadm -m discovery -t st -p -I iscsi-eth0 -I iscsi-eth1 -P 1
# iscsiadm -m discovery -t st -p -I iscsi-eth0 -I iscsi-eth1 -l

3. After logging into the target you should see new SCSI block devices created, verify this by executing fdisk -l:

# partprobe
# fdisk -l


Each LUN has a different World Wide Identifier (WWID.) Each scsi block device with the same WWID is a different path to the same LUN. To verify the WWIDs perform the following:

# scsi_id -gus /block/sd

Configuring Multipath:

After configuring the iSCSI layer Multipath must be configured via /etc/multipath/multipath.com. Please note that different SAN vendors will their own recommendations for configuring the multipath.conf file; their recommendations should be used if they are provided. For more information on the specific settings for your NAS please contact you hardware vendor.

1. Make the following changes to /etc/multipath.conf to set up a simple Multipath configuration with default settings:



* Un-comment the "defaults" stanza by removing the hash symbols on the following lines:

defaults {
user_friendly_names yes
}

* Comment-out the "blacklist" stanza by putting hash symbols on the following lines:

# blacklist {
# devnode "*"
# }


For more information on device mapper multipath please refer to: Using Device-Mapper Multipath

2. Save the changes to multipath.conf. Start multipath and ensure that it is configured to start at boot time:

# service multipathd start
# chkconfig multipathd on

3. After starting the multipath daemon the multipath command can be used to view your multipath devices. Example output is as follows:

mpath0 (1IET_00010001) dm-4 IET,VIRTUAL-DISK
[size=10G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
\_ 6:0:0:1 sdf 8:80 [active][ready]
\_ 7:0:0:1 sdh 8:112 [active][ready]

4. Using the mpath psuedo-device for the multipathed storage, create a partition and inform the kernel of the change:

# fdisk /dev/mapper/mpath0
# partprobe

5. Use the kpartx command to inform multipath of the new partition:

# kpartx -a /dev/mapper/mapth0

6. Device mapper will then create a new mpath pseudo device. Example:

/dev/mapper/mapth0p1

7. Create a file system on the multipathed storage and mount it:

# mkfs.ext3 /dev/mapper/mpath0p1
# mount /dev/mapper/mpath0p1 /

8. With the storage mounted begin failover testing. The following is an example of failover testing via a cable-pull on eth1:

* Use the mulitpath command to verify that all paths are up. Example output:

mpath0 (1IET_00010001) dm-4 IET,VIRTUAL-DISK
[size=10G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
\_ 6:0:0:1 sdf 8:80 [active][ready]
\_ 7:0:0:1 sdh 8:112 [active][ready]

* Pull the cable on eth1. Verify the path is failed with multipath -ll. Example output:

mpath0 (1IET_00010001) dm-4 IET,VIRTUAL-DISK
[size=10G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
\_ 6:0:0:1 sdf 8:80 [active][ready]
\_ 7:0:0:1 sdh 8:112 [faulty][failed]


9. The final step in the process is tuning failover timing.

o With the default timeouts in /etc/iscsi/iscsi.conf multipath failover takes about 1.5 minutes.
o Some users of multipath and iSCSI want lower timeourts so that I/O doesn't remain queued for long periods of time.
o For more information on lowering multipathed iSCSI failover time refer to How can I improve the failover time of a faulty path when using device-mapper-multipath over iSCSI?

What is Fibre Channel?

■ Requirement : What is Fibre Channel
■ OS Environment : Linux[RHEL, Centos]
■ Application: FCP
■ Resolution : 

         Fibre Channel (FC) is a transport protocol commonly used in storage networks. A common misunderstanding is that FC and fiber optic infrastructure such as host bus adaptor cards and fiber optic cables are the one-in-the-same. This is incorrect. FC is a protocol, like TCP/IP, and can be used over fiber optic cables or over copper. FC is commonly used to transport SCSI commands over fibre optic or copper cables in a Storage Area Network (SAN.)

        A SAN is simply that: a storage network. A traditional LAN connects computers and devices via a switched infrastructure (typically over copper cables) and employs TCP/IP as a transport protocol for passing data between devices and services. In a SAN computers and storage devices are connected via copper or fiber optic cables and Fibre Channel is employed as a transport protocol for passing data between storage devices and computers.

Fibre Channel SAN Topologies :

The way that devices are connected to each other in a SAN is referred to as its topology. There are 3 topologies available in Fibre Channel:
* Point to Point
* Arbitrated Loop
* Switched Fabric

P2P: For example a workstation with an HBA in it is hooked by a fiber optic cable directly into a tape array.
Switched Fabric is the most complex and most common topology for fibre channel storage networks.

Common Fibre Channel, SCSI, and SAN Terms :

HBA : Host Bus Adaptor. An HBA can be likened to a NIC for a SAN. An HBA is a card with ports, typically fiber optic ports, that allow the system it is housed in to connect to

the SAN infrastructure. HBAs typically have multiple ports (2 or 4) to allow for multiple paths to the storage. Common HBA speeds are 2 gigabit, 4 gigabit, and 8 gigabit and the HBA speed must match the speed of the switch and fabric. Common HBA vendors are Qlogic and Emulex.
Each port on an HBA appears to the system as its own SCSI host. A 2 port card, for example, will result in the system seeing 2 SCSI hosts. You can view the information for each port under /sys/class/fc_transport/

FCoE : Fibre Channel over Ethernet. As fibre channel is a protocol, not a class of hardware, it can pass over any suitable medium. FCoE uses copper ethernet cables to transport FC data rather than fiber optical cables. FCoE cards are HBAs with ethernet ports as apposed to fiber optic ports. Just as with fibre optic HBAs FCoE ports appear to the system they are housed in as SCSI hosts. Though FCoE utilizes standard ethernet cabling FCoE switches and cards are still required to allow for operation as standard ethernet LAN equipment does not support SAN specific functionality such as zoning, masking, logins, etc

WWN : World Wide Name
WWID is a device identifier. WWIDs are the preferred device identifier to be used within the kernel.You can view the device identifier of a SCSI device by running "scsi_id -gus /block/sd" or "scsi_id -gus -p 83 /block/sd" to retrieve the WWID, if one exists.
WWPN is a port identifier, WWPN for a port can be viewed at /sys/class/fc_host/host/port_name
WWNN : World Wide Node Name. The WWNN is a unique identifier, like a MAC address, for a "node" (read: card) on a SAN. A WWNN for an HBA can be viewed at /sys/class/fc_host/host/node_name
JBOD : Just a Bunch Of Disks. A JBOD is just that, a bunch of disks. A JBOD is an array of disks with no intelligence. While full featured arrays have the intelligence to control RAID levels, striping, redundancy, replication, and carving out logical storage units a JBOD may have few or none of those features.
SP : Storage Processor. An SP can be thought of as the "brain" of an array. It is the control unit that houses the intelligence that allows modern arrays to do advanced operations like LUN allocation, striping, redundancy, etc.
Target : A target is a SCSI concept, not a fibre channel concept. A target is a device that allows for incoming connections and storage access from initiators. The target side would be the array side in a SAN.

Initiator : An initiator is a SCSI concept, not a fibre channel concept. An initiator is a device that connects to a target to access storage on that target. The initiator side would be the HBA in a SAN.

LUN : Logical Unit. A LUN is a SCSI concept, not a fibre channel concept. A LUN is analogous to a Logical Volume in LVM. The array would be the Volume Group and the LUNs would be the Logical Volumes over that. A LUN is a logical storage device carved out of a larger storage pool of aggregated physical devices.

Path: A path is a single IO connection back to a LUN or storage pool. A path typically maps to a physical connection; however, zoning and mapping must be taken into account too. A path defines the route between the system and a device and consists of 4 numbers: H:B:T:L, host, bus, target, and lun.
SAN Issue Troubleshooting Tips :-

multipath -ll

What is cluster ?

■ Requirement : Red Hat Cluster Suite
■ OS Environment : Linux[RHEL, Centos]
■Clustering software : Red Hat Cluster suite
■Storage : SCSI, SAN, NAS
■ Storage Protocols : iSCSI(pronounced "eye-scuzzy) /FCP

■ Resolution : 

cluster :  A cluster is two or more interconnected computers that create a solution to provide higher availability, higher scalability or both. The advantage of clustering computers for high availability is seen if one of these computers fails, another computer in the cluster can then assume the workload of the failed computer. Users of the system see no interruption of access.

iSCSI => protcol to connect server to storage over IP network. Needs iSCSI initiator util will be on source or server. iSCSI target util will be on storage/target machine.
FCP => fibre channel protocol to connect server to storage over optical channel. Here needs HBA(host bus adapter like NIC) cards. It driver accesses this HBA and HBA communicates to SAN switch/storage controller.(Drivers like qla2xxx of qlogic company, lpfc(of emulex) etc)

Concepts: iSCSI is a protocol whereas SCSI is storage disk. consists using initiator(software+ hardware) and target. Initiator send packet to HBA. Target resides on storage like EqualLogic, NetApp filer, EMC NS-series or a HDS HNAS computer appliance. These attched with LUN to the drives. LUN is logical unit on storage treats as device or drive.

Storage System Connection Types :

a)active/active : all paths active all time
b)active‐passive : one path is active and other is backup
c)virtual port storage system.

Multipathing and Path Failover : When transferring data between the host server and storage, the SAN uses a multipathing technique where package "device-mapper-multipath" will have to be installed on server/node. The daemon "multipathd" periodically checks the connection paths to the storage. Multipathing allows you to have more than one physical path from the Server host to a LUN(treat is a device) on a storage system. If a path or any component along the path—HBA or NIC, cable, switch or switch port, or storage processor—fails, the server selects another of the available paths. The process of detecting a failed path and switching to another is called path failover.

Installation of Red Hat Cluster Suite on RHEL 5 :

1. Register system to RHN(Needs subscription with Red Het). Make sure  it subscribes storage channel. Packages comes with DVD too. If Ignore if system is already registered :

$rhn_register

2. Use following command :

$ yum groupinstall clustering cluster-storage

To separately install it do :

• For Standard kernel :

       $ yum cman cman-kernel dlm dlm-kernel magma magma-plugins system-config-cluster rgmanager ccs fence modcluster --force

• For SMP kernel :

$ yum cman cman-kernel-smp dlm dlm-kernel-smp magma magma-plugins system-config-cluster rgmanager ccs fence modcluster --force

3. Configuring Red Hat Cluster Suite (This steps should be followed on each nodes ) :

Configuration can be achieved in three ways like :

a) Using web interface(Conga tools) ie ricci and luci.
Conga — This is a comprehensive user interface for installing, configuring, and managing Red Hat clusters, computers, and storage attached to clusters and computers.

a.  Install luci on any system which can connect to each nodes :

 $yum install luci 

b. Initialize luci like

$luci_admin init

c. Install ricci on each nodes like :

$yum install ricci -y

d. Then access A(where luci has installed) like : http://IPof_A:port. Note that you'll get the url when you'll execute

 $luci_admin init 

4. Different Clustered services (Ordered as per the manually starting queue):

a, ccsd, cman, fence, rgmanager.
b. If you use LVM with GFS : ccsd, cman, fence, clvmd, gfs, rgmanager.

5. Configuration file (will be same on each nodes):
      /etc/cluster/cluster.conf, /etc/sysconfig/cluster. While you'll configure it using web interface, it'll be automatically copied on each nodes. Make sure you have enabled all the ports in firewall or disabled the firewall on all nodes as well as on luci node.

6. Now login into LUCI web interface and create a new cluster and give a name. Then in this lcuster add each nodes one by one. In this cluster add one fail over domain like httpd.(Make sure you have installed the httpd on each nodes where all the configuration files are same.). I shall describe it later and show you the result of real fail over testing.

7. Shared Disk configure(Disk size minimum 10MB is enough) : Why it is needed ?

         AA) The shared partitions are used to hold cluster state information including "Cluster lock states", "Service states", "Configuration information". The shared disk may be on any node or on storage disk( will be connected to HBA, RAID controller(raid 1 ie mirror). This will be for shared disk(primary partition+shadow). Each minimum 10MB. Two raw devices on shared disk storage must be created for the primary shared partition and the shadow shared partition. Each shared partition must have a minimum size of 10 MB. The amount of data in a shared partition is constant; it does not increase or decrease over time. Periodically, each member writes the state of its services to shared storage. In addition, the shared partitions contain a version of the cluster configuration file. This ensures that each member has a common view of the cluster configuration. If the primary shared partition is corrupted, the cluster members read the information from the shadow (or backup) shared partition and simultaneously repair the primary partition. Data consistency is maintained through checksums, and any inconsistencies between the partitions are automatically corrected. If a member is unable to write to both shared partitions at start-up time, it is not allowed to join the cluster. In addition, if an active member can no longer write to both shared partitions, the member removes itself from the cluster by rebooting (and may be remotely power cycled by a healthy member).

BB) The following are shared partition requirements:

a)Both partitions must have a minimum size of 10 MB.
b)Shared partitions must be raw devices since file cache won't be there. They cannot contain file systems.
c)Shared partitions can be used only for cluster state and configuration information.

CC) Following are recommended guidelines for configuring the shared partitions(By Red Hat):

a)It is strongly recommended to set up a RAID subsystem for shared storage, and use RAID 1 (mirroring) to make the logical unit that contains the shared partitions highly available. Optionally, parity RAID can be used for high availability. Do not use RAID 0 (striping) alone for shared partitions.
b)Place both shared partitions on the same RAID set, or on the same disk if RAID is not employed, because both shared partitions must be available for the cluster to run.
c)Do not put the shared partitions on a disk that contains heavily-accessed service data. If possible, locate the shared partitions on disks that contain service data that is rarely accessed.

DD) Make shared partitions and attach it with the cluster :

i) initialise quorum disk once in any node

$mkqdisk -c /dev/sdX -l myqdisk

ii)Add quorum disk to cluster at the backend(In web interface it can be done. Just login into luci interface and go to cluster. You'll see "Quorum Partition" tab. click on it and proceed further to configure it.) :

#expected votes =(nodes total votes + quorum disk votes)
#Health check result is written to quorum disk every 2 secs
#if health check fails over 5 tko, 10 (2*5) secs, the node is rebooted by quorum daemon
#Each heuristic check is run very 2 secs and earn 1 score,if shell script return is 0


Note : Need to manually copy this file on each node. But if you do in web interface, you don't need to manually cop. It'll automatically done.

b)Please increase the config_version by 1 and run ccs_tool update /etc/cluster/cluster.conf.
c) Check to verify that the quorum disk has been initialized correctly : #mkqdisk -L and clustat to check its availability.
d)Please note Total votes=quorum votes=5=2+3, if quorum disk vote is less than (node votes+1), the cluster wouldn’t have survived
e) Typically, the heuristics should be snippets of shell code or commands which help determine a node’s usefulness to the cluster or clients. Ideally, you want to add traces for all of your network paths (e.g. check links, or ping routers), and methods to detect availability of shared storage. Only one master is present at any one time in the cluster, regardless of how many partitions exist within the cluster itself. The master is elected by a simple voting scheme in which the lowest node which believes it is capable of running (i.e. scores high enough) bids for master status. If the other nodes agree, it becomes the master. This
algorithm is run whenever no master is present. Here it is "ping -c1 -t1 10.65.211.86". IP may be san ip/ other nodes' IP etc.


7. Configuring Cluster Daemons :
The Red Hat Cluster Manager provides the following daemons to monitor cluster operation:
cluquorumd — Quorum daemon
clusvcmgrd — Service manager daemon
clurmtabd — Synchronizes NFS mount entries in /var/lib/nfs/rmtab with a private copy on a service's mount point
clulockd — Global lock manager (the only client of this daemon is clusvcmgrd)
clumembd — Membership daemon

8. Configuring Storage : (Either SAN/NAS - using multipath or nfs)
In luci interface click on "add a system" and then go to storage tab and assign the storage in the cluster.


To start the cluster software on a member, type the following commands in this order:

1. service ccsd start
2. service lock_gulmd start or service cman start according to the type of lock manager used
3. service fenced start
4. service clvmd start
5. service gfs start, if you are using Red Hat GFS
6. service rgmanager start

To stop the cluster software on a member, type the following commands in this order:

1. service rgmanager stop
2. service gfs stop, if you are using Red Hat GFS
3. service clvmd stop
4. service fenced stop
5. service lock_gulmd stop or service cman stop according to the type of lock manager used
6. service ccsd stop

Stopping the cluster services on a member causes its services to fail over to an active member.
=================

Testing failover domain (Making high availability):

Pre-configuration : Installed httpd on node 68 and 86.
Common home directory : /var/www/html

Configure httpd as failover domain in cluster (in luci): add failover domain > Add resources > Add service and allocate fail over domain and resource to this service.

1. First httpd_service was on 86(allotted resource is ip 67 to httpd(daemon:domain on cluster) )

ip :

[root@vm86 ~]# ip add list|grep inet
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
inet 10.65.211.86/22 brd 10.65.211.255 scope global eth0
inet 10.65.211.67/22 scope global secondary eth0
inet6 fe80::216:3eff:fe74:8d56/64 scope link
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
inet6 fe80::200:ff:fe00:0/64 scope link
[root@vm86 ~]#

2. crashed 86 server ie made down it.

3. httpd service was up : relocated on 68 : Able to access page : http://10.65.211.67:/

IP floated to 68 server : proof

[root@vm68 ~]# ip add list | grep inet
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
inet 10.65.211.68/22 brd 10.65.211.255 scope global eth0
inet 10.65.211.67/22 scope global secondary eth0
inet6 fe80::216:3eff:fe74:8d44/64 scope link
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
inet6 fe80::20

Why did I see "cluster is not quorate. refusing connection" in log file.

■ Error : "cluster is not quorate. refusing connection"
■ OS Environment : Linux[RHEL, Centos]
■ Application: RH cluster
■ Fact  : 

   I was not sure why I got this error in the system log as quorate was enabled and working fine on non-firewalled machine where SELinux was also disabled. For the two node cluster, all the cluster.conf were same. One node was connected to the cluster and other didn't.

■  Resolution :

1. Make sure : chkcofing cman off; chkcofing clvmd off; chkcofing rgmanager off;
2. Make sure all cluster.conf fules are same.
3. Check if iptables are temporary off
4. Start cman, clvmd and rgmanager manually one by one
5. Reboot all nodes altogether or start cman altogether on all nodes.

How add FTP user at backend in linux?

■ Requirement : add FTP user at backend 
■ OS Environment : Linux[RHEL, Centos]
■ Application: vsftpd
■ Implementation Steps : 

$ useradd -m testing -G users,ftp,wheel -s /bin/bash
$ passwd testing
$ getsebool ftp_home_dir
$ setsebool -P ftp_home_dir on

How to make persistent static route

■ Requirement : make route persistent 
■ OS Environment : Linux[RHEL, Centos]
■ Application: route
■ Implementation Steps : 

1. Edit /etc/sysconfig/network-scripts/route-ethX and add following :

GATEWAY=xxx.xxx.xxx.xxx
NETMASK=yyy.yyy.yyy.yyy
ADDRESS=zzz.zzz.zzz.zzz

NOTE: Replace the address here.

2. Restart network service :

$ service network restart

3. If you use bond0 device : add following entries in /etc/sysconfig/network-scripts/route-

default via X.X.X.X dev bond0
10.10.10.0/24 via X.X.X.X dev bond0

NOTE: X.X.X.X is the gateway IP address

How to create network bonding device?

■ Requirement :  Create network bonding device 
■ OS Environment : Linux[RHEL 6, Centos]
■ Application: bonding
■ Implementation Steps : 

1. Edit /etc/modprobe.conf and add following :

 alias bond bonding 

2. Edit /etc/sysconfig/network-scripts/ifcfg-bond and add following :

DEVICE=bond0
IPADDR=192.168.1.1
NETMASK=255.255.255.0
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
BONDING_OPTS=""

3. Assuming two ethernet cards are eth0 and eth1 :They will look like :

DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
MASTER=bond
SLAVE=yes
USERCTL=no
3. Make sure "bonding" kernel module is present on server: 

$ lsmod | grep bonding; 
$ modprobe bonding;

4. Restart the service network and make the bond0 up.like : 

$ service network restart

How to analysis coredump using GDB tool?

■ Requirement : How to analysis coredump using GDB tool
■ OS Environment : Linux[RHEL, Centos]
■ Application: gdb
■ Implementation Steps : 

         Usually application or kernel creates core dump/vmcore file. You need to enable coredump on the linux machine. To collect the vmcore file(memory dump using crash kernel) you need to install kexec/kdump utils in the machine.

Once setup is done, you can install GDB like :

$ yum install gdb

1. Analysis coredump generated by process/binary :

a)Install the particular package which contains that binary. Check the version to make it similar if you want to analysis the coredump file on another system.
b) Install -debuginfo- on the machine of that same package.
c) start GDBing like : 

$gdb
$run

d) Enter "bt"

at the gdb prompt to get the stack of the symbols. Now analyse these to get the clue.

2. To analysis the vmcore file you need to replace the with and with file

then type bt to get backstrace.

DEMO : Using a simple c program :


1. Program hello.c

#include

char hello[] = { "Hello, World!" };

int
main()
{
fprintf (stdout, "%s\n", hello);
return (0);
}


2. Compile the above program :

$ gcc -g -o hello hello.c

3. Run gdb on the hello binary, i.e. gdb hello.


$ gdb hello

4. Same thing can be done even before execution is started. The variable hello is global, so it can be seen even before the main procedure starts:

gdb) p hello
$1 = "Hello, World!"
(gdb) p hello[0]
$2 = 72 'H'
(gdb) p *hello
$3 = 72 'H'
(gdb)

5.Next, list the source:

(gdb) l OR gdb list

1 #include
2
3 char hello[] = { "Hello, World!" };
4
5 int
6 main()
7 {
8 fprintf (stdout, "%s\n", hello);
9 return (0);
10 }

6. The list reveals that the fprintf call is on line 8. Apply a breakpoint on that line and resume the code:

(gdb) br 8
Breakpoint 1 at 0x80483ed: file hello.c, line 8.
(gdb) r
Starting program: /home/moller/tinkering/gdb-manual/hello

Breakpoint 1, main () at hello.c:8
8 fprintf (stdout, "%s\n", hello);

7. Finally, use the “next” command to step past the fprintf call, executing it:
gdb n

What is Proxy and Reverse Proxy?

Web Proxies :

         A proxy server is a gateway for users to the Web at large. Users configure the proxy in their browser settings, and all HTTP requests are routed via the proxy. Proxies are typically operated by ISPs and network administrators, and serve several purposes: for example,

* to speed access to the Web by caching pages fetched, so that popular pages don't have to be re-fetched for every user who views them.
* to enable controlled access to the web for users behind a firewall.
* to filter or transform web content.

Reverse Proxies :

          A reverse proxy is a gateway for servers, and enables one web server to provide content from another transparently. As with a standard proxy, a reverse proxy may serve to improve performance of the web by caching; this is a simple way to mirror a website. Loadbalancing a heavy-duty application, or protecting a vulnerable one, are other common usages. But the most common reason to run a reverse proxy is to enable controlled access from the Web at large to servers behind a firewall.

The proxied server may be a webserver itself, or it may be an application server using a different protocol, or an application server with just rudimentary HTTP that needs to be shielded from the web at large. Since 2004, reverse proxying has been the preferred method of deploying JAVA/Tomcat applications on the Web, replacing the old mod_jk (itself a special-purpose reverse proxy module).

A Reverse Proxy Scenario:

           Company example.com has a website at www.example.com, which has a public IP address and DNS entry, and can be accessed from anywhere on the Internet.

The company also has a couple of application servers which have private IP addresses and unregistered DNS entries, and are inside the firewall. The application servers are visible within the network - including the webserver, as "internal1.example.com" and "internal2.example.com", But because they have no public DNS entries, anyone looking at internal1.example.com from outside the company network will get a "no such host" error.

A decision is taken to enable Web access to the application servers. But they should not be exposed to the Internet directly, instead they should be integrated with the webserver, so that http://www.example.com/app1/any-path-here is mapped internally to http://internal1.example.com/any-path-here and http://www.example.com/app2/other-path-here is mapped internally to http://internal2.example.com/other-path-here. This is a typical reverse-proxy situation.

Load following Apache Proxy Modules :

* mod_proxy: The core module deals with proxy infrastructure and configuration and managing a proxy request.
* mod_proxy_http: This handles fetching documents with HTTP and HTTPS.
* mod_proxy_ftp: This handles fetching documents with FTP.
* mod_proxy_connect: This handles the CONNECT method for secure (SSL) tunnelling.
* mod_proxy_ajp: This handles the AJP protocol for Tomcat and similar backend servers.
* mod_proxy_balancer implements clustering and load-balancing over multiple backends.
* mod_cache, mod_disk_cache, mod_mem_cache: These deal with managing a document cache. To enable caching requires mod_cache and one or both of disk_cache and mem_cache.
* mod_proxy_html: This rewrites HTML links into a proxy's address space.
* mod_xml2enc: This supports internationalisation (i18n) on behalf of mod_proxy_html and other markup-filtering modules. space.
* mod_headers: This modifies HTTP request and response headers.
* mod_deflate: Negotiates compression with clients and backends.

Most important are mod_proxy_balancer, mod_cache, mod_disk_cache, mod_mem_cache, mod_deflate

Building Apache for Proxying :

Use options during compiling apache using source code :

$ ./configure --enable-so --enable-mods-shared="proxy cache ssl all"

Using apxs tool on existing apache installation :

apxs -c -i [module-name].c

Configuring the Proxy :

Load following modules in http.conf :

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
#LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
#LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule headers_module modules/mod_headers.so
LoadModule deflate_module modules/mod_deflate.so
LoadFile /usr/lib/libxml2.so
LoadModule xml2enc_module modules/mod_xml2enc.so
LoadModule proxy_html_module modules/mod_proxy_html.so

The fundamental configuration directive to set up a reverse proxy is ProxyPass. We use it to set up proxy rules for each of the application servers in the httpd.conf file on the webserver:

ProxyPass /app1/ http://internal1.example.com/
ProxyPass /app2/ http://internal2.example.com/

       However, this is not the whole story. ProxyPass just sends traffic straight through. So when the application servers generate references to themselves (or to other internal addresses), they will be passed straight through to the outside world, where they won't work. The proxy needs to re-map the Location header to its own address space and return a valid URL. The command to enable such rewrites in the HTTP Headers is ProxyPassReverse. The Apache documentation suggests the form:

ProxyPassReverse /app1/ http://internal1.example.com/
ProxyPassReverse /app2/ http://internal2.example.com/