Sunday, June 23, 2019

Java gateway process exited before sending its port number

Problem
While creating spark sql session, received following error message.

Exception: Java gateway process exited before sending its port number

Environment

  • OS - RHEL 7
  • Jupyter notebook


Steps to reproduce

  • configure jupyter notebook
  • Start Jupyter
  • Access Jupyter webpage
  • Run program in notebook like below


from pyspark import SparkContext, SparkConf
from pyspark.sql import SparkSession


  • now click on "run"
Solution
Add following in .bashrc and restart jupyter notebook

#For jupyter notebook:
export PYSPARK_SUBMIT_ARGS="--master yarn-client pyspark-shell"
Root Cause Analysis
If you look at the code below

it says connection file is not being created in tmp directory. In my case issue was environment variable "PYSPARK_SUBMIT_ARGS" was not setup. This caused to not to create "proc"

submit_args = os.environ.get("PYSPARK_SUBMIT_ARGS", "pyspark-shell")
...
command = command + shlex.split(submit_args)
...
proc = Popen(command, **popen_kwargs)

...
# Wait for the file to appear, or for the process to exit, whichever happens first. while not proc.poll() and not os.path.isfile(conn_info_file): time.sleep(0.1) if not os.path.isfile(conn_info_file): raise Exception("Java gateway process exited before sending its port number")
...

Wednesday, November 8, 2017

Docker build encounters error "failed to unshare namespaces: Invalid argument"

Issue



_[91mnsenter: failed to unshare namespaces: Invalid argument
_[0m_[91mcontainer_linux.go:265: starting container process caused "process_linux.go:270: running exec setns process for init caused \"exit status 34\""
_[0moci runtime error: container_linux.go:265: starting container process caused "process_linux.go:270: running exec setns process for init caused \"exit status 34\""

Environment

  RHEL 7.2

Solution


Execute below commands as a root user and reboot system.

grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"
reboot

Root cause


usernamespace was enable at docker daemon level but same was not enabled in kernel level.

Docker daemon runs like this:

ExecStart=/usr/bin/dockerd --live-restore --userland-proxy=false --icc=false --disable-legacy-registry  --userns-remap=docker:docker --tlsverify --tlscacert=/var/SP/docker/certs/ca.pem --tlscert=/var/SP/docker/certs/cert.pem --tlskey=/var/SP/docker/certs/key.pem -g /var/SP/docker -H tcp://0.0.0.0:2376  -H unix://var/run/docker.sock --log-driver=syslog --log-opt syslog-address=tcp://localhost:514 --log-opt tag="DOCKER"


Docker build encounters "operation timed out"

Issue

while running docker build, we see following error.
fetch http://dl-4.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-4.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz: operation timed out
fetch http://dl-4.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-4.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz: operation timed out
ERROR: unsatisfiable constraints:

Solution 

Pass proxy as environment variable and unset at the end of Dockerfile file.
ENV http_proxy http://proxyhost.com:3128
ENV https_proxy http://proxyhost.com:3128
...
RUN unset http_proxy https_proxy

Root cause Analysis

  1. During docker build on a machine which is behind proxy server can encounter this issue. Requests do not go through  running Docker engine but go directly. hence, you need to mention proxy in Dockerfile and unset later.
  2. Second issue may happen that upstream server is not available. This is very rare. You can view with below command to see current repositories
RUN cat /etc/apk/repositories
Default values are
http://dl-cdn.alpinelinux.org/alpine/v3.4/main
http://dl-cdn.alpinelinux.org/alpine/v3.4/community

You can check through browser if you can see if domain is resolving or not. you can replace with actual domain like below
RUN sed -i 's/dl-cdn.alpinelinux.org/dl-4.alpinelinux.org/' /etc/apk/repositories




Wednesday, February 10, 2016

Why Puppet was using 96% CPU while ran as a daemon or through command prompt?

Problem :  Puppet was consuming 96% CPU while it ran as a daemon or ran as "puppet agent -t".
Environment: RHEL 6
Solution : Make sure application puppet module is tested with out any error and unwanted issue.
Root Cause Analysis: 

top has shown following:


PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11403 root 20 0 259m 140m 4516 R 96.7 1.9 9:08.77 puppet
1. Traced puppet agent which was run at command prompt while I kept daemon off.
$strace -Tvv  puppet agent -t  &> ./strace_log

2. read() system calls are spawned a lot and continuously printing following messages:
[...]
22:36:37 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096 <0 .000031="">
[...]
3. While I was tracking filename of corresponding file descriptor, I found /home/qpadm/temp was opened that had size 101GB.
4. read() call was continuously trying to read data from this file and read size was 4096 Bytes in every call. Trace looked below:
22:36:37 open("/home/qpadm/temp", O_RDONLY) = 4 <0 .000032="">
22:36:37 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 <0 .000026="">
22:36:37 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 <0 .000026="">
22:36:37 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 <0 .000027="">
22:36:37 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 <0 .000026="">
22:36:37 fstat(4, {st_dev=makedev(202, 1), st_ino=790228, st_mode=S_IFREG|0600, st_nlink=1, st_uid=1000, st_gid=1000, st_blksize=4096, st_blocks=209715224, st_size=107374182400, st_atime=2016/02/03-07:37:15, st_mtime=2016/02/03-05:11:57, st_ctime=2016/02/03-07:36:48}) = 0 <0 .000027="">
22:36:37 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6d03f3f000 <0 .000030="">
22:36:37 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096 <0 .000035="">
22:36:37 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096 <0 .000031="">
In above trace, you see file /home/qpadm/temp is opened in read mode as FD(file descriptor) is 4 and same FD is passed to read() call that returns 4096 Bytes incrementally. It supposed to take long time to read 101GB file[:) would never be finished using huge read calls those would have neen performed by kernel]. That's why CPU spiked to 60-80% consumed by puppet agent.
We need to check why above file is there and what is the usage of it and why it has large size.
This one is I think main culprit.
Another thing I noticed is either ovo was reinstalled or being migrated to newer version. In relation to this, I found following calls :
22:36:00 open("/root/ovoinstall/HPOvPacc.xml", O_RDONLY) = 4 <0 .000022="">

22:36:07 open("/root/ovoinstall/OVO-Agent-Migrate.xml", O_RDONLY) = 4 <0 .000032="">

22:36:07 fstat(4, {st_dev=makedev(202, 1), st_ino=786799, st_mode=S_IFREG|0744, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=4240, st_size=2168841, st_atime=2016/02/03-17:04:19, st_mtime=2016/01/03-22:12:54, st_ctime=2016/01/03-22:15:32}) = 0 <0 .000027="">
22:36:07 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6d03f3f000 <0 .000029="">
22:36:07 read(4, "\355\253\356\333\3\0\0\0\0\1HPOvConf-11.11.025-1\0\0"..., 4096) = 4096 <0 .000032="">
22:36:07 read(4, "\"Unable to copy file\"\nOVCSL_ERRO"..., 4096) = 4096 <0 .000029="">

Application team found that temp was not needed but was called in application puppet module. They disabled it and removed that file. It fixed the issue.


Wednesday, February 3, 2016

why vxdisk does throw error "VxVM vxdctl ERROR V-5-1-16007 Data Corruption Protection Activated - User Corrective Action Needed To recover"

Problem:

Following errors has been encountered :

[root@customer-pet-db-3a ~]# /sbin/vxdctl enable
VxVM vxdctl ERROR V-5-1-16007  Data Corruption Protection Activated - User Corrective Action Needed
To recover, first ensure that the OS device tree is up to date (requires OS specific commands).
Then, execute 'vxdisk rm' on the following devices before reinitiating device discovery:
        fas31400_85, fas31400_88, fas31400_92, fas31400_98, fas31400_93
        fas31400_96, fas31400_94, fas31400_95, fas31400_97, fas31400_99
        fas31402_41, fas31401_39
[root@customer-pet-db-3a ~]#


[root@customer-pet-db-3a ~]# /sbin/vxdisk scandisks
VxVM vxdisk ERROR V-5-1-16007  Data Corruption Protection Activated - User Corrective Action Needed
To recover, first ensure that the OS device tree is up to date (requires OS specific commands).
Then, execute 'vxdisk rm' on the following devices before reinitiating device discovery:
        fas31400_85, fas31400_88, fas31400_92, fas31400_98, fas31400_93
        fas31400_96, fas31400_94, fas31400_95, fas31400_97, fas31400_99
        fas31402_41, fas31401_39
[root@customer-pet-db-3a ~]#

Solution referred :


1. check disk :

[root@customer-pet-db-3a ~]# vxdisk -o alldgs list
DEVICE       TYPE            DISK         GROUP        STATUS
cciss/c0d0   auto:none       -            -            online invalid
fas31400_10  auto:cdsdisk    flex3a_myapp_arch  flex3a_arch_dg online thinrclm
fas31400_11  auto:cdsdisk    flex3a_dcd_arch  flex3a_arch_dg online thinrclm
fas31400_12  auto:cdsdisk    flex3a_myapps_arch  flex3a_arch_dg online thinrclm
fas31400_13  auto:cdsdisk    flex3a_rpt_arch  flex3a_arch_dg online thinrclm
fas31400_85  auto:cdsdisk    myapps_pet_data00  flex3a_myapps_pet_data_dg online thinrclm
fas31400_88  auto:cdsdisk    myapps_pet_data01  flex3a_myapps_pet_data_dg online thinrclm
fas31400_89  auto:cdsdisk    customerumprd_arch01  flex3a_pet_customerumprd_arch_dg online thinrclm
fas31400_90  auto:cdsdisk    customerumprd_u02  flex3a_pet_customerumprd_dg online thinrclm
fas31400_91  auto:cdsdisk    customerumprd_u01  flex3a_pet_customerumprd_dg online thinrclm
fas31400_92  auto:cdsdisk    -            (myapp_pet_data_dg) online thinrclm
fas31400_93  auto:cdsdisk    -            (myapp_pet_data_dg) online thinrclm
fas31400_94  auto:cdsdisk    -            (myapp_pet_data_dg) online thinrclm
fas31400_95  auto:cdsdisk    -            (myapp_pet_data_dg) online thinrclm
fas31400_96  auto:cdsdisk    -            (myapp_pet_data_dg) online thinrclm
fas31400_97  auto:cdsdisk    -            (myapp_pet_data_dg) online thinrclm
fas31400_98  auto:cdsdisk    -            (myapp_pet_data_dg) online thinrclm
fas31400_99  auto:cdsdisk    -            (myapp_pet_data_dg) online thinrclm
fas31401_39  auto:cdsdisk    myapps_pet_data02  flex3a_myapps_pet_data_dg online thinrclm
fas31402_41  auto:cdsdisk    myapps_pet_data03  flex3a_myapps_pet_data_dg online thinrclm
[root@customer-pet-db-3a ~]#

2. Removed all disk of myapp and myapps like as it is stated above :


[root@customer-pet-db-3a ~]# for i in fas31400_85 fas31400_88 fas31400_93 fas31400_94 fas31400_95 fas31400_96 fas31400_97 fas31400_98 fas31400_99 fas31401_39 fas31402_41; do vxdisk rm $i; done
VxVM vxdisk ERROR V-5-1-534 Device fas31400_85: Device is in use
VxVM vxdisk ERROR V-5-1-534 Device fas31400_88: Device is in use
VxVM vxdisk ERROR V-5-1-534 Device fas31401_39: Device is in use
VxVM vxdisk ERROR V-5-1-534 Device fas31402_41: Device is in use
[root@customer-pet-db-3a ~]# vxdg list
NAME         STATE           ID
flex3a_pet_customerumprd_arch_dg enabled,cds          1446853137.45.customer-pet-db-3a.example.net
flex3a_arch_dg enabled,cds          1376078555.40.customer-pet-db-3a.example.net
flex3a_myapps_pet_data_dg enabled,cds          1375818468.64.customer-pet-db-3c.example.net
flex3a_pet_customerumprd_dg enabled,cds          1446683837.59.customer-pet-db-3b.example.net

flex3a_myapps_pet_data_dg  is still in use. Need to deport it.

[root@customer-pet-db-3a ~]# /sbin/vxdg deport flex3a_myapps_pet_data_dg

[root@customer-pet-db-3a ~]# vxdg list
NAME         STATE           ID
flex3a_pet_customerumprd_arch_dg enabled,cds          1446853137.45.customer-pet-db-3a.example.net
flex3a_arch_dg enabled,cds          1376078555.40.customer-pet-db-3a.example.net
flex3a_pet_customerumprd_dg enabled,cds          1446683837.59.customer-pet-db-3b.example.net
[root@customer-pet-db-3a ~]#


Remove disks which were in use : 

[root@customer-pet-db-3a ~]# for i in fas31400_85 fas31400_88 fas31400_93 fas31400_94 fas31400_95 fas31400_96 fas31400_97 fas31400_98 fas31400_99 fas31401_39 fas31402_41; do vxdisk rm $i; done
VxVM vxdisk ERROR V-5-1-639 Failed to obtain locks:
        fas31400_93: no such object in the configuration
VxVM vxdisk ERROR V-5-1-639 Failed to obtain locks:
        fas31400_94: no such object in the configuration
VxVM vxdisk ERROR V-5-1-639 Failed to obtain locks:
        fas31400_95: no such object in the configuration
VxVM vxdisk ERROR V-5-1-639 Failed to obtain locks:
        fas31400_96: no such object in the configuration
VxVM vxdisk ERROR V-5-1-639 Failed to obtain locks:
        fas31400_97: no such object in the configuration
VxVM vxdisk ERROR V-5-1-639 Failed to obtain locks:
        fas31400_98: no such object in the configuration
VxVM vxdisk ERROR V-5-1-639 Failed to obtain locks:
        fas31400_99: no such object in the configuration
[root@customer-pet-db-3a ~]#

don't worry above error as we have already removed disk of myapp. That's why above errors are displayed.

check disk now:

[root@customer-pet-db-3a ~]# vxdisk -o alldgs list
DEVICE       TYPE            DISK         GROUP        STATUS
cciss/c0d0   auto:none       -            -            online invalid
fas31400_10  auto:cdsdisk    flex3a_myapp_arch  flex3a_arch_dg online thinrclm
fas31400_11  auto:cdsdisk    flex3a_dcd_arch  flex3a_arch_dg online thinrclm
fas31400_12  auto:cdsdisk    flex3a_myapps_arch  flex3a_arch_dg online thinrclm
fas31400_13  auto:cdsdisk    flex3a_rpt_arch  flex3a_arch_dg online thinrclm
fas31400_89  auto:cdsdisk    customerumprd_arch01  flex3a_pet_customerumprd_arch_dg online thinrclm
fas31400_90  auto:cdsdisk    customerumprd_u02  flex3a_pet_customerumprd_dg online thinrclm
fas31400_91  auto:cdsdisk    customerumprd_u01  flex3a_pet_customerumprd_dg online thinrclm
[root@customer-pet-db-3a ~]#

3. Enable and Scan disk again :

[root@customer-pet-db-3a ~]# /sbin/vxdctl enable
[root@customer-pet-db-3a ~]# /sbin/vxdisk scandisks

Didn't see any more error.

4. Export dg now :

[root@customer-pet-db-3a ~]# /sbin/vxdg -C -n flex3a_myapp_pet_data_dg import myapp_pet_data_dg
[root@customer-pet-db-3a ~]# vxdisk -o alldgs list
DEVICE       TYPE            DISK         GROUP        STATUS
cciss/c0d0   auto:none       -            -            online invalid
fas31400_10  auto:cdsdisk    flex3a_myapp_arch  flex3a_arch_dg online thinrclm
fas31400_11  auto:cdsdisk    flex3a_dcd_arch  flex3a_arch_dg online thinrclm
fas31400_12  auto:cdsdisk    flex3a_myapps_arch  flex3a_arch_dg online thinrclm
fas31400_13  auto:cdsdisk    flex3a_rpt_arch  flex3a_arch_dg online thinrclm
fas31400_85  auto:cdsdisk    -            (flex3a_myapps_pet_data_dg) online thinrclm
fas31400_88  auto:cdsdisk    -            (flex3a_myapps_pet_data_dg) online thinrclm
fas31400_89  auto:cdsdisk    customerumprd_arch01  flex3a_pet_customerumprd_arch_dg online thinrclm
fas31400_90  auto:cdsdisk    customerumprd_u02  flex3a_pet_customerumprd_dg online thinrclm
fas31400_91  auto:cdsdisk    customerumprd_u01  flex3a_pet_customerumprd_dg online thinrclm
fas31400_92  auto:cdsdisk    myapp_pet_data00  flex3a_myapp_pet_data_dg online thinrclm
fas31400_93  auto:cdsdisk    myapp_pet_data06  flex3a_myapp_pet_data_dg online thinrclm
fas31400_94  auto:cdsdisk    myapp_pet_data03  flex3a_myapp_pet_data_dg online thinrclm
fas31400_95  auto:cdsdisk    myapp_pet_data07  flex3a_myapp_pet_data_dg online thinrclm
fas31400_96  auto:cdsdisk    myapp_pet_data05  flex3a_myapp_pet_data_dg online thinrclm
fas31400_97  auto:cdsdisk    myapp_pet_data04  flex3a_myapp_pet_data_dg online thinrclm
fas31400_98  auto:cdsdisk    myapp_pet_data01  flex3a_myapp_pet_data_dg online thinrclm
fas31400_99  auto:cdsdisk    myapp_pet_data02  flex3a_myapp_pet_data_dg online thinrclm
fas31401_39  auto:cdsdisk    -            (flex3a_myapps_pet_data_dg) online thinrclm
fas31401_40  auto:cdsdisk    myapp_pet_data08  flex3a_myapp_pet_data_dg online thinrclm
fas31401_41  auto:cdsdisk    myapp_pet_data11  flex3a_myapp_pet_data_dg online thinrclm
fas31401_42  auto:cdsdisk    myapp_pet_data10  flex3a_myapp_pet_data_dg online thinrclm
fas31401_43  auto:cdsdisk    myapp_pet_data09  flex3a_myapp_pet_data_dg online thinrclm
fas31402_41  auto:cdsdisk    -            (flex3a_myapps_pet_data_dg) online thinrclm
fas31402_45  auto:cdsdisk    myapp_pet_data14  flex3a_myapp_pet_data_dg online thinrclm
fas31402_46  auto:cdsdisk    myapp_pet_data12  flex3a_myapp_pet_data_dg online thinrclm
fas31402_47  auto:cdsdisk    myapp_pet_data15  flex3a_myapp_pet_data_dg online thinrclm
fas31402_48  auto:cdsdisk    myapp_pet_data13  flex3a_myapp_pet_data_dg online thinrclm
[root@customer-pet-db-3a ~]#

Received error for myapps, 

[root@customer-pet-db-3a ~]# /sbin/vxdg -C -n flex3a_myapps_pet_data_dg import myapps_pet_data_dg
VxVM vxdg ERROR V-5-1-10978 Disk group myapps_pet_data_dg: import failed:
No valid disk found containing disk group
[root@customer-pet-db-3a ~]#

DG is already renamed as flex3a_myapps_pet_data_dg. Just import it simply.

[root@customer-pet-db-3a ~]# vxdg import flex3a_myapps_pet_data_dg

[root@customer-pet-db-3a ~]#


check dg now :

[root@customer-pet-db-3a ~]# vxdg list|grep -i myapp
flex3a_myapp_pet_data_dg enabled,cds          1375815375.127.customer-pet-db-3c.example.net
flex3a_myapps_pet_data_dg enabled,cds          1375818468.64.customer-pet-db-3c.example.net
[root@customer-pet-db-3a ~]#

Confirmed with DBA, they can start DB instances.


Thursday, December 24, 2015

why did kinit throw error "Preauthentication failed while getting initial credentials"?

Problem: 
While executed following command :

#kinit -k -t /root/utilscripts/nsupdateuser.keytab nsupdate@example.com

it threw error:

kinit: Preauthentication failed while getting initial credentials

Solution: Password of user may be wrong, Try to reset and test again.

Why did kinit throw error "KDC reply did not match expectations while getting initial credentials"

Problem: 

While executing following command:

#kinit username@MYDOMAIN.COM -k -t username.keytab

it threw error :

kinit: KDC reply did not match expectations while getting initial credentials

Solution : 

user doesn't have remote access to the machine. 

How to manage kerberos keytab file?

1. Use klist to display the keytab file entries:

klist -e -k -t  mykeytabfile.keytab
or klist -ekt nsupdateuser.ktab
or type command:

#ktutil     # execute this command
ktutil:     # this prompt will appear
ktutil: read_kt /etc/apache2/http.keytab   #read keytab file
ktuilt: list                             #list all princples

example:

[root@customer-prod-util-101 utilscripts]# klist -e -k -t  nsupdateuser.ktab
Keytab name: FILE:nsupdateuser.ktab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 09/08/15 21:50:45 nsupdate@site1.example.com (aes256-cts-hmac-sha1-96)
[root@customer-prod-util-101 utilscripts]#


2. Following is an example of the keytab file creation process using kerberos method :

  > ktutil
  ktutil:  addent -password -p username@example.com -k 1 -e rc4-hmac
  Password for username@example.com: [enter your password]
  ktutil:  addent -password -p username@example.com -k 1 -e aes256-cts
  Password for username@example.com: [enter your password]
  ktutil:  wkt username.keytab
  ktutil:  quit

Following is an example using Heimdal Kerberos:

> ktutil -k username.keytab add -p username@example.com -e arcfour-hmac-md5 -V 1


3. Obtain a ticket-granting ticket using the keytab for testing:

You can check that the keytab contains the appropriate encryption key by attempting to use it to obtain a ticket-granting ticket. This can be done using the kinit command:

#kinit -k -t /etc/nsupdateuser.keytab nsupdate@example.com    # here nsupdate is username exiting in AD. this has privileges to update dns records on win DNS.
#klist                <#will show if ticket is created or not

example :

[root@customer-prod-util-101 ~]#  kinit -k -t /root/utilscripts/nsupdateuser.ktab nsupdate   # uses default domain
[root@customer-prod-util-101 ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: nsupdate@site1.example.com

Valid starting     Expires            Service principal
12/24/15 05:29:49  12/24/15 15:29:49  krbtgt/site1.example.com@site1.example.com
        renew until 12/31/15 05:29:49
[root@customer-prod-util-101 ~]#


   Or try to login to test if keytab file works :

Test with out keytab file:

#kinit username@MYDOMAIN.COM
password>     // pass password of username

Test with keytab file:

#kinit username@MYDOMAIN.COM -k -t username.keytab


4. Using a keytab to authenticate scripts:

To execute a script so it has valid Kerberos credentials, use:

  > kinit username@example.com -k -t mykeytab; myscript

list out principle :

>ktutil
ktutil:  rkt nsupdateuser.ktab
ktuilt: list

5. Merging keytab files:

> ktutil
  ktutil: read_kt mykeytab-1
  ktutil: read_kt mykeytab-2
  ktutil: read_kt mykeytab-3
  ktutil: write_kt krb5.keytab
  ktutil: quit


6. Delete principle:

#ktuil
ktutil: rkt
ktuilt: list
ktutil: delete_entry slot-number
ktuilt: wkt
ktuilt: quit


7. Destroy cached ticket:

kdestroy -A  //all cache will be destroyed
kdestroy -C  //this cache will be deleted only

#kdestroy -c "FILE:/tmp/krb5cc_0"

Monday, September 28, 2015

Why did df command throw "disk" Input output error

Issue/Symptom  : While DBA was starting oracle instance, it was failing. On checking FS, it was found that arch volumes are not mounted. While tried to remount them and checked through "df -h", it was throwing below error:
[root@customer-pet-oracle-3d ~]#  df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/cciss/c0d0p3      58G   48G  7.3G  87% /
/dev/cciss/c0d0p1     494M   18M  452M   4% /boot
tmpfs                  63G  232M   63G   1% /dev/shm
tmpfs                 4.0K     0  4.0K   0% /dev/vx
df: `/cinprds1_arch00': Input/output error
df: `/cinprd1_arch00': Input/output error
df: `/customerdcdp1_arch00': Input/output error
df: `/customerrptp1_arch00': Input/output error
example-prod-sea1utilnas-1a-pet:/vol/customerpet_data
                      450G  335G  116G  75% /filers/example-prod-sea1utilnas-1a-pet/customerpet_data
[root@customer-pet-oracle-3d ~]
OS Environment : RHEL 5.5
Software/Application :
DB : oracle 11.2.0.4
VxVm : VRTSvxvm-5.1.100.000-SP1_RHEL5, Symantec License Manager vxlicrep utility version 3.02.51.010
vxfs : VRTSvxfs-5.1.100.000-SP1_GA_RHEL5
Customer Environment : ATT PET Oracle DB
Investigation :
1.
$sanlun lun show|grep -i minipet_arc

customer-pet-sea1bfiler-1a:  /vol/customer_MINIPET_ARCH/lun1                /dev/sdag        host1    FCP        500.1g (536952700928)   GOOD
customer-pet-sea1bfiler-1a:  /vol/customer_MINIPET_ARCH/lun0                /dev/sdah        host1    FCP        500.1g (536952700928)   GOOD
customer-pet-sea1bfiler-1a:  /vol/customer_MINIPET_ARCH/lun2                /dev/sdai        host1    FCP          250g (268435456000)   GOOD
customer-pet-sea1bfiler-1a:  /vol/customer_MINIPET_ARCH/lun3                /dev/sdaj        host1    FCP          250g (268435456000)   GOOD
2. Check fstab entry how it is :
fstab was :
/dev/vx/dsk/minipet_arch_dg/cinprds_minipet_vol_arch00 /cinprds1_arch00 vxfs    _netdev 0 1
3. Check netfs service if running:
$/etc/init.d/netfs status
4. Search dg in log as root  :
$ awk '/arch_dg/ {print $0}' /var/log/messages.*

Sep 25 19:26:08 customer-pet-oracle-3d kernel: vxfs: msgcnt 1 mesg 037: V-2-37: vx_metaioerr - vx_inode_iodone - /dev/vx/dsk/minipet_arch_dg/rpt_minipet_vol_arch00 file system meta data write error in dev/block 0/1104
Sep 25 19:26:08 customer-pet-oracle-3d vxvm:vxconfigd: V-5-1-7935 Disk group minipet_arch_dg: update failed: Disk group has no valid configuration copies
Sep 25 19:26:08 customer-pet-oracle-3d vxvm:vxconfigd: V-5-1-7934 Disk group minipet_arch_dg: Disabled by errors
[...]
Sep 25 19:30:01 customer-pet-oracle-3d kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff811130c6aa00 on device 0xc900130 to DMP<4>vxfs: msgcnt 5 mesg 039: V-2-39: vx_writesuper - /dev/vx/dsk/minipet_arch_dg/cinprds_minipet_vol_arch00 file system super-block write error
Sep 25 19:30:01 customer-pet-oracle-3d kernel: vxfs: msgcnt 6 mesg 037: V-2-37: vx_metaioerr - vx_dirbread - /dev/vx/dsk/minipet_arch_dg/cinprds_minipet_vol_arch00 file system meta data write error in dev/block 0/1104
[...]
Sep 25 19:40:01 customer-pet-oracle-3d kernel: vxfs: msgcnt 21 mesg 039: V-2-39: vx_writesuper - /dev/vx/dsk/minipet_arch_dg/rpt_minipet_vol_arch00 file system super-block write error
Sep 25 19:40:01 customer-pet-oracle-3d kernel: vxfs: msgcnt 22 mesg 008: V-2-8: vx_direrr: vx_readdir_int_1 - /dev/vx/dsk/minipet_arch_dg/rpt_minipet_vol_arch00 file system dir inode 5 dev/block 0/150297879 dirent inode 0 error 5
Sep 25 19:40:01 customer-pet-oracle-3d kernel: vxfs: msgcnt 23 mesg 039: V-2-39: vx_writesuper - /dev/vx/dsk/minipet_arch_dg/rpt_minipet_vol_arch00 file system super-block write error
[...]
Sep 26 04:18:13 customer-pet-oracle-3d kernel: vxfs: msgcnt 334 mesg 016: V-2-16: vx_ilisterr: vx_iread - /dev/vx/dsk/minipet_arch_dg/rpt_minipet_vol_arch00 file system error reading inode 3
Sep 26 04:18:13 customer-pet-oracle-3d kernel: vxfs: msgcnt 335 mesg 039: V-2-39: vx_writesuper - /dev/vx/dsk/minipet_arch_dg/rpt_minipet_vol_arch00 file system super-block write error
Sep 26 04:18:13 customer-pet-oracle-3d kernel: vxfs: msgcnt 336 mesg 031: V-2-31: vx_disable - /dev/vx/dsk/minipet_arch_dg/rpt_minipet_vol_arch00 file system disabled
Above confirms that few blocks are corrupted in disks which are under disk group "minipet_arch_dg"

Permanent Solution :
1. Unmount file system if mounted. 2. Run file system check through fsck :
$fsck -F vxfs -o full 
example :
$/opt/VRTS/bin/fsck -o full -y /dev/vx/rdsk/minipet_arch_dg/rpt_minipet_vol_arch00

or 

$fsck.vxfs -o full /dev/vx/dsk/minipet_arch_dg/rpt_minipet_vol_arch00
3. OR:
or reboot system [make sure fsck is enabled in fstab]

Root Cause Analysis :

Error messages in system log confirm that disk blocks are corrupted. vxiod was failing to write data. vxconfigd informed kernel that it was unable to change vxfs config. 

Tuesday, August 11, 2015

Why does liwewise throw error like "Problem executing /opt/pbis/bin/ad-cache --delete-all >/dev/null 2>/dev/null"?

OS Environment :  Linux
Application : pbis aka likewise with version 8.3
Problem: While executing following command, it was throwing error :

/opt/likewise/bin/lwconfig --file /opt/pbis/bin/lwconfig.txt
Problem executing '/opt/pbis/bin/ad-cache --delete-all >/dev/null 2>/dev/null'

Error: Error returned by external program

Solution : As ad-cache is linked to lsa, so copy lsa binary from machine where pbis 8.0 is installed to the affected machine.
Solution : ad-cache is linked to lsa binary and this has bug. 

Wednesday, January 28, 2015

How to install python boto module on windows?

OS Environment : Windows 2007, 64 bit
Application : Python boto module
Implementation Steps : 

1. First Install python 3
2. Execute following steps now :

From windows command prompt :

$cd c:/
c:\>cd Python34/Scripts
c:\Python34\Scripts>pip.exe install -U boto

Output will look like below :

Downloading/unpacking boto
Installing collected packages: boto
Successfully installed boto
Cleaning up...

Saturday, January 24, 2015

How to configure Postfix as a SMTP gateway?

■ Requirement: Configure postfix as a smtp gateway server
OS Environment : Linux [RHEL 5, RHEL 6]
Application: postfix
■ Assumption : 

  •       Domain name= example.com, 
  •       Internal Mail server IP = 192.168.1.3, 
  •       Gateway mail server IP = 192.168.1.2, 
  •       Internal postfix smtp is pre-configured. 

Implementation Steps :

A. DMZ Mail Server Setup (or gateway mail server):  The DMZ mail server forwards the inbound mail to the internal mail server and delivers the outbound mail to internet.

1. Edit /etc/postfix/main.cf and update the lines below.

mydestination =
local_recipient_maps =
local_transport = error:local mail delivery is disabled

mynetworks = 127.0.0.0/8 192.168.1.3
relay_domains = example.com
transport_maps = hash:/etc/postfix/transport
smtpd_recipient_restrictions = permit_mynetworks
reject_unauth_destination


2. Edit the file /etc/postfix/transport and add the line below.

example.com :[192.168.1.3]

NOTE : If you would like to use multiple internal server for multiple users/domains then postfix transport maps should be changed.

e.g. in main.cf

transport_maps = /etc/postfix/transport

in /etc/postfix/transport:

user1/domain1 smtp:1-mailserver.example.com
user2/domain2 smtp:2-mailserver.example.com


3. Execute following commands to reflect above :

$ postmap /etc/postfix/transport 
$ postfix reload
NOTE : That will deliver email for user1 and user2 to [1,2]-mailserver.example.com. If you don't specify anything in the transport map, the default transport (which is usually deliver locally) will be used.

B. Configure Internal Mail Server :

The internal mail server holds the mailbox and forward all outbound mail to the DMZ mail server.

1. Edit /etc/postfix/main.cf and update the lines below : 

transport_maps = hash:/etc/postfix/transport

2. Edit file /etc/postfix/transport and add the lines below :

example.com :
.example.com :
* smtp:[192.168.3.2]


3. Create a transport database file :

$ postmap /etc/postfix/transport

4. Restart the Postfix : 

$ service postfix restart

Friday, January 23, 2015

How to install ruby on linux server?

■ Requirement : Install ruby on linux system
■ OS Environment : Linux(RHEL, Centos)
■ Implementation Steps : 

$ cd /usr/local/src
$ Download latest tar ball of ruby
$ tar xvzf ruby-XXX.tar.gz
$ cd ruby-XXX
$ ./configure
$ make
$ make install
$ ruby rubytest.rb

How to install FFmpeg, FFmpeg-PHP,Mplayer,Mencoder, flv2tool,LAME, MP3 Encoderon linux server?

■ Requirement : Install FFmpeg, FFmpeg-PHP,Mplayer,Mencoder, flv2tool,LAME, MP3 Encoderon
■ OS Environment : Linux, RHEL 5, 64 bit

■ Implementation Steps :

1. Login into server and get root access.
2. cd /usr/local/src/
3. Download following source file from appropriate vendor sites :

essential-20061022.tar.bz2
flvtool2_1.0.5_rc6.tgz
lame-3.97.tar.gz
ffmpeg-php-0.5.1.tbz2
libogg-1.1.3.tar.gz
libvorbis-1.1.2.tar.gz
MPlayer-1.0rc2.tar.bz2
ffmpeg-0.5.tar.bz2

4. Extract above modules :

$ for pkg in  lame-3.97.tar.gz libogg-1.1.3.tar.gz libvorbis-1.1.2.tar.gz flvtool2_1.0.5_rc6.tgz essential-20061022.tar.bz2 ffmpeg-php-0.5.1.tbz2 MPlayer-1.0rc2.tar.bz2 ffmpeg-0.5.tar.bz2; do tar -xvzf $pkg; done

5. Create a codecs directory :

$ mkdir /usr/local/lib/codecs/

6. Install dependent libraries :

$ yum install gcc gmake make libcpp libgcc libstdc++ gcc4 gcc4-c++ gcc4-gfortran subversion ruby ncurses-devel -y

7. Copy essentials codes in proper location :

$ cd /usr/local/src/
$ mv /usr/local/src/essential-20061022/* /usr/local/lib/codecs/
$ chmod -R 755 /usr/local/lib/codecs/

8. Install LAME :

$ cd /usr/local/src/lame-3.97
$ ./configure
$ make 
$ make install

9. Install LIBOGG:

$ cd /usr/local/src/
$ cd /usr/local/src/libogg-1.1.3
$ ./configure --enable-shared ; make ; make install
$ PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
$ export PKG_CONFIG_PATH

Put above config path in root user .bashrc file

10. Install LIBVORBIS:

$ cd /usr/local/src/
$ cd /usr/local/src/libvorbis-1.1.2
$ ./configure; make ; make install

11. Install FLVTOOL2

$ cd /usr/local/src/
$ cd /usr/local/src/flvtool2_1.0.5_rc6/
$ ruby setup.rb config
$ ruby setup.rb setup
$ ruby setup.rb install

12. Install MPLAYER

$cd /usr/local/src/
$ cd /usr/local/src/MPlayer-1.0rc2
$ ./configure; make; make install

13. Install FFMPEG:

$ cd /usr/local/src/
$ cd /usr/local/src/ffmpeg-0.5
$ ./configure --enable-libmp3lame --enable-libvorbis --disable-mmx --enable-shared
$ make
$ make install

$ export LD_LIBRARY_PATH=/usr/local/lib/

$ ln -s /usr/local/lib/libavformat.so.50 /usr/lib/libavformat.so.50
$ ln -s /usr/local/lib/libavcodec.so.51 /usr/lib/libavcodec.so.51
$ ln -s /usr/local/lib/libavutil.so.49 /usr/lib/libavutil.so.49
$ ln -s /usr/local/lib/libmp3lame.so.0 /usr/lib/libmp3lame.so.0
$ ln -s /usr/local/lib/libavformat.so.51 /usr/lib/libavformat.so.51

14. Install FFMPEG-PHP:

$ cd /usr/local/src/
$ cd /usr/local/src/ffmpeg-php-0.5.1/
$ phpize
$ ./configure
$ make
$ make install

Enable ffmpeg module in php.ini like below

echo 'extension=ffmpeg.so'

15. Now check the binaries like below :

$ for bin in lamp flvtool2l mplayer ffmpeg; do which $bin; done




Why did "ls -al" command take much time to return output?

Issue/Symptom  : "ls -al" command took much time than expected.

time ls -la /data/XXX/mboxes/tier1/01/XXX/15/67/178/XXX/ 
... (just removing the output of the ls -la command itself)
real 1m26.674s 
user 0m0.285s 
sys 0m1.627s



OS Environment : RHEL 6, storage netapp filer ONTAP 8 
Software/Application : storage - FAS3240 over NFSv3, ls command[coreutils-8.4-16.el6.x86_64]
Investigation : Direcotry "XXX" contains around 22481 files.
Workaround Solution : Time consumption is expected in this case. This may not be same on other environment. Please follow RCA section.

Root Cause Analysis :


Above time is expected as directory has around 22481 files. "ls -al" retrieves more file attributes details than normal "ls" command. 
As per strace analysis, it is not found that any system call which took more than 1 sec. Accumulation of all calls took 1m26s as you see. Compared other user's 
directory[has around ~4000 mails], it was  not found  this much delay. It just took less than 2 sec. Test and details :
time ls -la /data/XXX/mboxes/tier1/01/XXX/15/67/178/XXX/ << no of file 22481
[...]
real 1m26.674s 
user 0m0.285s 
sys 0m1.627s
with out "-al" option :
time ls /data/XXX/mboxes/tier1/01/XXX/15/67/178/XXX/
[....]
real 0m4.678s
user 0m0.223s
sys 0m1.008s
This user has less mails :
time ls -al /data/XXX/mboxes/tier1/01/XXX/15/67/47/XXX << no of file 4442
real 0m1.010s
user 0m0.067s
sys 0m0.297s
time ls -al /data/XXX2/mboxes/tier1/01/XXX/15/67/47/XXn/
real 0m0.736s
user 0m0.070s
sys 0m0.162s
ls -al /data/XXX/mboxes/tier1/01/XXX/15/67/183/XXX/|wc -l
6468
took time :
real 0m1.406s
user 0m0.090s
sys 0m0.406s

Strace analysis :

#strace -Tttvv ls -al /data/XXX/mboxes/tier1/01/XXX/15/67/178/XXX/ &> /var/log/strace_op
#cat  /var/log/strace_op|awk '{print $NF}'|egrep -v 'msg|dat|new|bin|size|\?'|sed 's/<//'|sed 's/>//'|less 
#cat  /var/log/strace_op|awk '{print $NF}'|egrep -v 'msg|dat|new|bin|size|\?'|sed 's/<//'|sed 's/>//'|awk 'BEGIN {sum=0.0} {sum+=$NF} END {print sum}' 

If you see too much penalty in terms of consumption of retrieve time then please engage NetApp to perf analysis. Download perf tool 
and retrieve data from filers. Handover this data to Netapp to further analysis. 

Why inodes is almost full on file system?

Issue/Symptom  : dfm: Warning event on filer:/(Inodes Almost Full)
OS Environment : Netapp OnTAP 8.1
Software/Application : Netapp DFM[5] sends alert that inode is full.
Environment : Applicable to all customer who uses netapp filers
Investigation : Huge number of small files are put on volumes

Workaround Solution :

Check how is the usage of inodes : 

filer> df -i 

Check currently maximum setup :

filer> maxfiles 

Calculate maximum number of inodes it can hold [4KB size of each inode]

Find maximum size of volume :

filer> df -h  

If maximum size is  XY GB, then maximum supported inodes :

= (XY*1024*1024)/4

Set new inode value :

filer>maxfiles   
 
Permanent solution  : Same as workaround solution. 
Root Cause Analysis : No required, is known.

How to clean unused semaphore

Issue/Symptom  : Sometime you'll see semaphore usage is full.
OS Environment : Linux or RHEL
Software/Application : HP ovo sends semaphore usage alerts
Investigation : Unsed semaphores are not cleared by kernel
Workaround Solution : Use below scripts to clean unused semaphore.


cat clean-unused-semaprhore.sh

#Developed By Kamal maiti, 
#check if root can run it.
if [[ $EUID -ne 0 ]]; then
   exit 1
 else

#collect all semaphore ID
  for SEMID in `ipcs -s|egrep -v -e "Semaphore|key"|sed '/^$/d'|awk '{print $2}'|sort -u`
     do
     #GETPID of semaphore
   PID=`ipcs -s -i $SEMID|tail -2|head -1|awk '{print $NF}'`
     #GET PROCESS ID
    #Ignore process ID 0 which is main process & test PID greater than 0
if [ $PID -gt 0 ]; then

#Test of PID exists in process list, if exits then don't do anything.

  if ps -p $PID > /dev/null
    then
  #running process are
     echo "$SEMID   $PID" &>/dev/null      else
# dead process are, kill corresponding semaphore of related PID is not exisitng.

    echo "$SEMID   $PID" &>/dev/null 

  #cleaning semaphore of dead process :
  ipcrm -s $SEMID
 fi
fi
 done
fi


RUN :chmod +x clean-unused-semaprhore.sh; ./clean-unused-semaprhore.sh

High CPU usage, server was not accessible over ssh

■ Issue/Symptom : High load on server, not accessible over ssh
OS Environment : RHEL 5.5
■ Background Information  :
  • Infra was running test
  • Server was intermittently highly loaded
  • ssh was failing :
    • [usera@user01lxv ~]$ ssh 10.57XXX
    • Password:
    • Connection closed by 10.57.XXX
  • console shows "lockd: rejected NSM callback from 7f000001:30001" and sometimes NFS is not ok
Investigation :
  • iowait was very high and fluctuating.
  • All the cpu were busy to serve i/o bound operations
$ mpstat -P ALL 1

Linux 2.6.18-128.el5 (xxxxxxx) 11/19/2014
10:17:29 PM CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s
10:17:30 PM all 0.00 0.00 0.00 75.00 0.00 0.00 0.00 25.00 182.18
10:17:30 PM 0 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 182.18
10:17:30 PM 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00
  • top had shown high load or no process took too much cpu
top - 22:18:03 up 50 days, 22:20, 4 users, load average: 25.19, 26.68, 30.74
Tasks: 235 total, 2 running, 231 sleeping, 0 stopped, 2 zombie
Cpu(s): 2.0%us, 0.8%sy, 0.0%ni, 0.0%id, 96.8%wa, 0.0%hi, 0.4%si, 0.0%st
Mem: 3866480k total, 2916884k used, 949596k free, 12440k buffers
Swap: 8385920k total, 498424k used, 7887496k free, 350200k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20841 cw 21 0 5606m 2.1g 5040 S 4.0 56.8 291:05.56 /opt/cw/jre/bin/java -Duser.timezone=America/Mexico_City -Xms2560m -Xmx2560m -XX:MaxPermSize=128m
  • Found that there were lot of "D" stated processes which didn't appear on nso-102, 101
$ ps aux |awk '{print $1 " " $8 " " $NF }'|grep D

USER STAT COMMAND
root D< [kjournald]
root Ds 0
root Ds /var/run/vmware-guestd.pid
nobody DN /usr/bin/log2mysql-nso-tomcat-writer
nobody DN /usr/bin/log2mysql-nso-tomcat-spooler
root D
  • In above output, system thread kjournald is also in D state which looked bad from kernel perspective. Journalling would have stopped.
■ Workaround Solution :
Shutdown VM and power on again.[D stated processes can't be killed unless system is rebooted]

Permanent Solution :
Shutdown VM and power on again. .[D stated processes can't be killed unless system is rebooted]
Root Cause Analysis :
  • IOwait was mainly taking place as there were high number of D stated processes.

why amazon cloud load balancer was flapping between two instances?

Issue/Symptom :
  • Why SMOKETEST LB in amazon aws cloud was flapping between qpass-prod-smktst-201.dub1.qpass.net & qpass-prod-smktst-101.dub1.qpass.net. States changed between "InService" & "OutofService" ?
OS Environment :
  • Both nodes has RHEL 6, LB is provided by Amazon
Investigation :
  • LB is mapped to above two nodes. Incomming port is 443, destination port is 80. It is found that applications are listening port 80 on both nodes. Server Health check timeout has been increased in LB, but issue still persisted.
Permanent Solution :

For the LB, at Health check section, Ping target will be TCP:80, Timeout set 5 seconds, Interval 30 seconds, Unhealthy Threshold 2
Healthy Threshold  10


Root Cause Analysis : 
  • It was found that Ping target was HTTP:80, Ping path was /ping.html. Though webbased ping returns OK[200 status code] but it does't work properly.

why netapp dfm does send "Clock Skewed" alert from filer?

Issue/Symptom  : DFM sent alert like "Dfm: Error event on Clock Skewed"

OS Environment : Netapp ONTAP

Investigation : Not performed

Workaround Solution :

"options timed.enable off"
"options timed.enable on"

Permanent Solution : 
  • Unknown.
Root Cause Analysis :
unknown