Why did df command throw "disk" Input output error

■ Issue/Symptom  : While DBA was starting oracle instance, it was failing. On checking FS, it was found that arch volumes are not mounted. While tried to remount them and checked through "df -h", it was throwing below error:

[root@customer-pet-oracle-3d ~]#  df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/cciss/c0d0p3      58G   48G  7.3G  87% /
/dev/cciss/c0d0p1     494M   18M  452M   4% /boot
tmpfs                  63G  232M   63G   1% /dev/shm
tmpfs                 4.0K     0  4.0K   0% /dev/vx
df: `/cinprds1_arch00': Input/output error
df: `/cinprd1_arch00': Input/output error
df: `/customerdcdp1_arch00': Input/output error
df: `/customerrptp1_arch00': Input/output error
example-prod-sea1utilnas-1a-pet:/vol/customerpet_data
                      450G  335G  116G  75% /filers/example-prod-sea1utilnas-1a-pet/customerpet_data
[root@customer-pet-oracle-3d ~]

■ OS Environment : RHEL 5.5
■ Software/Application :

DB : oracle 11.2.0.4
VxVm : VRTSvxvm-5.1.100.000-SP1_RHEL5, Symantec License Manager vxlicrep utility version 3.02.51.010
vxfs : VRTSvxfs-5.1.100.000-SP1_GA_RHEL5

■ Customer Environment : ATT PET Oracle DB
■ Investigation :
1.

$sanlun lun show|grep -i minipet_arc

customer-pet-sea1bfiler-1a:  /vol/customer_MINIPET_ARCH/lun1                /dev/sdag        host1    FCP        500.1g (536952700928)   GOOD
customer-pet-sea1bfiler-1a:  /vol/customer_MINIPET_ARCH/lun0                /dev/sdah        host1    FCP        500.1g (536952700928)   GOOD
customer-pet-sea1bfiler-1a:  /vol/customer_MINIPET_ARCH/lun2                /dev/sdai        host1    FCP          250g (268435456000)   GOOD
customer-pet-sea1bfiler-1a:  /vol/customer_MINIPET_ARCH/lun3                /dev/sdaj        host1    FCP          250g (268435456000)   GOOD

2. Check fstab entry how it is :
fstab was :

/dev/vx/dsk/minipet_arch_dg/cinprds_minipet_vol_arch00 /cinprds1_arch00 vxfs    _netdev 0 1

3. Check netfs service if running:

$/etc/init.d/netfs status

4. Search dg in log as root  :

$ awk '/arch_dg/ {print $0}' /var/log/messages.*

Sep 25 19:26:08 customer-pet-oracle-3d kernel: vxfs: msgcnt 1 mesg 037: V-2-37: vx_metaioerr - vx_inode_iodone - /dev/vx/dsk/minipet_arch_dg/rpt_minipet_vol_arch00 file system meta data write error in dev/block 0/1104
Sep 25 19:26:08 customer-pet-oracle-3d vxvm:vxconfigd: V-5-1-7935 Disk group minipet_arch_dg: update failed: Disk group has no valid configuration copies
Sep 25 19:26:08 customer-pet-oracle-3d vxvm:vxconfigd: V-5-1-7934 Disk group minipet_arch_dg: Disabled by errors
[...]
Sep 25 19:30:01 customer-pet-oracle-3d kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff811130c6aa00 on device 0xc900130 to DMP<4>vxfs: msgcnt 5 mesg 039: V-2-39: vx_writesuper - /dev/vx/dsk/minipet_arch_dg/cinprds_minipet_vol_arch00 file system super-block write error
Sep 25 19:30:01 customer-pet-oracle-3d kernel: vxfs: msgcnt 6 mesg 037: V-2-37: vx_metaioerr - vx_dirbread - /dev/vx/dsk/minipet_arch_dg/cinprds_minipet_vol_arch00 file system meta data write error in dev/block 0/1104
[...]
Sep 25 19:40:01 customer-pet-oracle-3d kernel: vxfs: msgcnt 21 mesg 039: V-2-39: vx_writesuper - /dev/vx/dsk/minipet_arch_dg/rpt_minipet_vol_arch00 file system super-block write error
Sep 25 19:40:01 customer-pet-oracle-3d kernel: vxfs: msgcnt 22 mesg 008: V-2-8: vx_direrr: vx_readdir_int_1 - /dev/vx/dsk/minipet_arch_dg/rpt_minipet_vol_arch00 file system dir inode 5 dev/block 0/150297879 dirent inode 0 error 5
Sep 25 19:40:01 customer-pet-oracle-3d kernel: vxfs: msgcnt 23 mesg 039: V-2-39: vx_writesuper - /dev/vx/dsk/minipet_arch_dg/rpt_minipet_vol_arch00 file system super-block write error
[...]
Sep 26 04:18:13 customer-pet-oracle-3d kernel: vxfs: msgcnt 334 mesg 016: V-2-16: vx_ilisterr: vx_iread - /dev/vx/dsk/minipet_arch_dg/rpt_minipet_vol_arch00 file system error reading inode 3
Sep 26 04:18:13 customer-pet-oracle-3d kernel: vxfs: msgcnt 335 mesg 039: V-2-39: vx_writesuper - /dev/vx/dsk/minipet_arch_dg/rpt_minipet_vol_arch00 file system super-block write error
Sep 26 04:18:13 customer-pet-oracle-3d kernel: vxfs: msgcnt 336 mesg 031: V-2-31: vx_disable - /dev/vx/dsk/minipet_arch_dg/rpt_minipet_vol_arch00 file system disabled

Above confirms that few blocks are corrupted in disks which are under disk group "minipet_arch_dg"

■ Permanent Solution :
1. Unmount file system if mounted. 2. Run file system check through fsck :

$fsck -F vxfs -o full 

example :

$/opt/VRTS/bin/fsck -o full -y /dev/vx/rdsk/minipet_arch_dg/rpt_minipet_vol_arch00

or 

$fsck.vxfs -o full /dev/vx/dsk/minipet_arch_dg/rpt_minipet_vol_arch00

3. OR:
or reboot system [make sure fsck is enabled in fstab]

■ Root Cause Analysis :

Error messages in system log confirm that disk blocks are corrupted. vxiod was failing to write data. vxconfigd informed kernel that it was unable to change vxfs config. 

Why does liwewise throw error like "Problem executing /opt/pbis/bin/ad-cache --delete-all >/dev/null 2>/dev/null"?

OS Environment :  Linux
Application : pbis aka likewise with version 8.3
Problem: While executing following command, it was throwing error :

/opt/likewise/bin/lwconfig --file /opt/pbis/bin/lwconfig.txt

Problem executing '/opt/pbis/bin/ad-cache --delete-all >/dev/null 2>/dev/null'

Error: Error returned by external program

Solution : As ad-cache is linked to lsa, so copy lsa binary from machine where pbis 8.0 is installed to the affected machine.

Solution : ad-cache is linked to lsa binary and this has bug. 

How to install python boto module on windows?

OS Environment : Windows 2007, 64 bit
Application : Python boto module
Implementation Steps : 

1. First Install python 3
2. Execute following steps now :

From windows command prompt :

$cd c:/
c:\>cd Python34/Scripts
c:\Python34\Scripts>pip.exe install -U boto

Output will look like below :

Downloading/unpacking boto
Installing collected packages: boto
Successfully installed boto
Cleaning up...

How to configure Postfix as a SMTP gateway?

■ Requirement: Configure postfix as a smtp gateway server
■OS Environment : Linux [RHEL 5, RHEL 6]
■Application: postfix
■ Assumption : 

•       Domain name= example.com, 
•       Internal Mail server IP = 192.168.1.3, 
•       Gateway mail server IP = 192.168.1.2, 
•       Internal postfix smtp is pre-configured. 

■Implementation Steps :

A. DMZ Mail Server Setup (or gateway mail server):  The DMZ mail server forwards the inbound mail to the internal mail server and delivers the outbound mail to internet.

1. Edit /etc/postfix/main.cf and update the lines below.

mydestination =
local_recipient_maps =
local_transport = error:local mail delivery is disabled

mynetworks = 127.0.0.0/8 192.168.1.3
relay_domains = example.com
transport_maps = hash:/etc/postfix/transport
smtpd_recipient_restrictions = permit_mynetworks
reject_unauth_destination

2. Edit the file /etc/postfix/transport and add the line below.

example.com :[192.168.1.3]

NOTE : If you would like to use multiple internal server for multiple users/domains then postfix transport maps should be changed.

e.g. in main.cf

transport_maps = /etc/postfix/transport

in /etc/postfix/transport:

user1/domain1 smtp:1-mailserver.example.com
user2/domain2 smtp:2-mailserver.example.com

3. Execute following commands to reflect above :

$ postmap /etc/postfix/transport 
$ postfix reload
NOTE : That will deliver email for user1 and user2 to [1,2]-mailserver.example.com. If you don't specify anything in the transport map, the default transport (which is usually deliver locally) will be used.

B. Configure Internal Mail Server :

The internal mail server holds the mailbox and forward all outbound mail to the DMZ mail server.

1. Edit /etc/postfix/main.cf and update the lines below : 

transport_maps = hash:/etc/postfix/transport

2. Edit file /etc/postfix/transport and add the lines below :

example.com :
.example.com :
* smtp:[192.168.3.2]

3. Create a transport database file :

$ postmap /etc/postfix/transport

4. Restart the Postfix : 

$ service postfix restart

How to install ruby on linux server?

■ Requirement : Install ruby on linux system
■ OS Environment : Linux(RHEL, Centos)
■ Implementation Steps : 

$ cd /usr/local/src
$ Download latest tar ball of ruby
$ tar xvzf ruby-XXX.tar.gz
$ cd ruby-XXX
$ ./configure
$ make
$ make install
$ ruby rubytest.rb

How to install FFmpeg, FFmpeg-PHP,Mplayer,Mencoder, flv2tool,LAME, MP3 Encoderon linux server?

■ Requirement : Install FFmpeg, FFmpeg-PHP,Mplayer,Mencoder, flv2tool,LAME, MP3 Encoderon
■ OS Environment : Linux, RHEL 5, 64 bit

■ Implementation Steps :

1. Login into server and get root access.
2. cd /usr/local/src/
3. Download following source file from appropriate vendor sites :

essential-20061022.tar.bz2
flvtool2_1.0.5_rc6.tgz
lame-3.97.tar.gz
ffmpeg-php-0.5.1.tbz2
libogg-1.1.3.tar.gz
libvorbis-1.1.2.tar.gz
MPlayer-1.0rc2.tar.bz2
ffmpeg-0.5.tar.bz2

4. Extract above modules :

$ for pkg in  lame-3.97.tar.gz libogg-1.1.3.tar.gz libvorbis-1.1.2.tar.gz flvtool2_1.0.5_rc6.tgz essential-20061022.tar.bz2 ffmpeg-php-0.5.1.tbz2 MPlayer-1.0rc2.tar.bz2 ffmpeg-0.5.tar.bz2; do tar -xvzf $pkg; done

5. Create a codecs directory :

$ mkdir /usr/local/lib/codecs/

6. Install dependent libraries :

$ yum install gcc gmake make libcpp libgcc libstdc++ gcc4 gcc4-c++ gcc4-gfortran subversion ruby ncurses-devel -y

7. Copy essentials codes in proper location :

$ cd /usr/local/src/
$ mv /usr/local/src/essential-20061022/* /usr/local/lib/codecs/
$ chmod -R 755 /usr/local/lib/codecs/

8. Install LAME :

$ cd /usr/local/src/lame-3.97
$ ./configure
$ make 
$ make install

9. Install LIBOGG:

$ cd /usr/local/src/
$ cd /usr/local/src/libogg-1.1.3
$ ./configure --enable-shared ; make ; make install
$ PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
$ export PKG_CONFIG_PATH

Put above config path in root user .bashrc file

10. Install LIBVORBIS:

$ cd /usr/local/src/
$ cd /usr/local/src/libvorbis-1.1.2
$ ./configure; make ; make install

11. Install FLVTOOL2

$ cd /usr/local/src/
$ cd /usr/local/src/flvtool2_1.0.5_rc6/
$ ruby setup.rb config
$ ruby setup.rb setup
$ ruby setup.rb install

12. Install MPLAYER

$cd /usr/local/src/
$ cd /usr/local/src/MPlayer-1.0rc2
$ ./configure; make; make install

13. Install FFMPEG:

$ cd /usr/local/src/
$ cd /usr/local/src/ffmpeg-0.5
$ ./configure --enable-libmp3lame --enable-libvorbis --disable-mmx --enable-shared
$ make
$ make install

$ export LD_LIBRARY_PATH=/usr/local/lib/

$ ln -s /usr/local/lib/libavformat.so.50 /usr/lib/libavformat.so.50
$ ln -s /usr/local/lib/libavcodec.so.51 /usr/lib/libavcodec.so.51
$ ln -s /usr/local/lib/libavutil.so.49 /usr/lib/libavutil.so.49
$ ln -s /usr/local/lib/libmp3lame.so.0 /usr/lib/libmp3lame.so.0
$ ln -s /usr/local/lib/libavformat.so.51 /usr/lib/libavformat.so.51

14. Install FFMPEG-PHP:

$ cd /usr/local/src/
$ cd /usr/local/src/ffmpeg-php-0.5.1/
$ phpize
$ ./configure
$ make
$ make install

Enable ffmpeg module in php.ini like below

echo 'extension=ffmpeg.so'

15. Now check the binaries like below :

$ for bin in lamp flvtool2l mplayer ffmpeg; do which $bin; done

Why did "ls -al" command take much time to return output?

Issue/Symptom  : "ls -al" command took much time than expected.

time ls -la /data/XXX/mboxes/tier1/01/XXX/15/67/178/XXX/ 
... (just removing the output of the ls -la command itself)

real 1m26.674s 
user 0m0.285s 
sys 0m1.627s

■ OS Environment : RHEL 6, storage netapp filer ONTAP 8
■ Software/Application : storage - FAS3240 over NFSv3, ls command[coreutils-8.4-16.el6.x86_64]
■ Investigation : Direcotry "XXX" contains around 22481 files.
■ Workaround Solution : Time consumption is expected in this case. This may not be same on other environment. Please follow RCA section.

■ Root Cause Analysis :

Above time is expected as directory has around 22481 files. "ls -al" retrieves more file attributes details than normal "ls" command. 
As per strace analysis, it is not found that any system call which took more than 1 sec. Accumulation of all calls took 1m26s as you see. Compared other user's 
directory[has around ~4000 mails], it was  not found  this much delay. It just took less than 2 sec. Test and details :

time ls -la /data/XXX/mboxes/tier1/01/XXX/15/67/178/XXX/ << no of file 22481
[...]
real 1m26.674s 
user 0m0.285s 
sys 0m1.627s

with out "-al" option :

time ls /data/XXX/mboxes/tier1/01/XXX/15/67/178/XXX/
[....]
real 0m4.678s
user 0m0.223s
sys 0m1.008s

This user has less mails :

time ls -al /data/XXX/mboxes/tier1/01/XXX/15/67/47/XXX << no of file 4442
real 0m1.010s
user 0m0.067s
sys 0m0.297s

time ls -al /data/XXX2/mboxes/tier1/01/XXX/15/67/47/XXn/
real 0m0.736s
user 0m0.070s
sys 0m0.162s

ls -al /data/XXX/mboxes/tier1/01/XXX/15/67/183/XXX/|wc -l
6468
took time :
real 0m1.406s
user 0m0.090s
sys 0m0.406s

Strace analysis :

#strace -Tttvv ls -al /data/XXX/mboxes/tier1/01/XXX/15/67/178/XXX/ &> /var/log/strace_op

#cat  /var/log/strace_op|awk '{print $NF}'|egrep -v 'msg|dat|new|bin|size|\?'|sed 's/<//'|sed 's/>//'|less 

#cat  /var/log/strace_op|awk '{print $NF}'|egrep -v 'msg|dat|new|bin|size|\?'|sed 's/<//'|sed 's/>//'|awk 'BEGIN {sum=0.0} {sum+=$NF} END {print sum}' 

If you see too much penalty in terms of consumption of retrieve time then please engage NetApp to perf analysis. Download perf tool 
and retrieve data from filers. Handover this data to Netapp to further analysis.