tag:blogger.com,1999:blog-43336134279669520102024-03-17T20:03:47.473-07:00open-sys-servicesThis is dedicated to the linux users, system admins, open source enthusiastic, techs whoever is looking for solution, tricks & concept etc. Reader will apply concept or execute command at their own risk. Owner of these article is not responsible for any impact, damages or errors. Playing with Linux.http://www.blogger.com/profile/01147934604287011206noreply@blogger.comBlogger247125tag:blogger.com,1999:blog-4333613427966952010.post-9508753304554325242019-06-23T22:38:00.000-07:002019-06-23T22:38:00.616-07:00Java gateway process exited before sending its port number<div dir="ltr" style="text-align: left;" trbidi="on">
<b>Problem</b><br />
While creating spark sql session, received following error message.<br />
<br />
<blockquote class="tr_bq" style="background: white; vertical-align: baseline; word-break: break-all;">
<span style="color: #e75c58; font-family: "Courier New"; font-size: 10.5pt;">Exception</span><span style="font-family: "Courier New"; font-size: 10.5pt;">: Java gateway
process exited before sending its port number<o:p></o:p></span></blockquote>
<span style="background-color: white; color: #032f62; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;"><br /></span>
<span style="background-color: white; color: #032f62; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;"><b>Environment</b></span><br />
<br />
<ul style="text-align: left;">
<li><span style="color: #032f62; font-family: SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace;"><span style="background-color: white; font-size: 12px; white-space: pre;">OS - RHEL 7</span></span></li>
<li><span style="color: #032f62; font-family: SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace;"><span style="background-color: white; font-size: 12px; white-space: pre;">Jupyter notebook</span></span></li>
</ul>
<br />
<span style="color: #032f62; font-family: SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace;"><span style="background-color: white; font-size: 12px; white-space: pre;"><br /></span></span>
<span style="color: #032f62; font-family: SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace;"><span style="background-color: white; font-size: 12px; white-space: pre;"><b>Steps to reproduce</b></span></span><br />
<br />
<ul style="text-align: left;">
<li><span style="color: #032f62; font-family: SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace;"><span style="background-color: white; font-size: 12px; white-space: pre;">configure jupyter notebook</span></span></li>
<li><span style="color: #032f62; font-family: SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace;"><span style="background-color: white; font-size: 12px; white-space: pre;">Start Jupyter</span></span></li>
<li><span style="color: #032f62; font-family: SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace;"><span style="background-color: white; font-size: 12px; white-space: pre;">Access Jupyter webpage</span></span></li>
<li><span style="color: #032f62; font-family: SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace;"><span style="background-color: white; font-size: 12px; white-space: pre;">Run program in notebook like below</span></span></li>
</ul>
<br />
<span style="color: #032f62; font-family: SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace;"><span style="background-color: white; font-size: 12px; white-space: pre;"><br /></span></span>
<blockquote class="tr_bq">
<span style="color: #032f62; font-family: SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace;"><span style="background-color: white; font-size: 12px; white-space: pre;">from pyspark import SparkContext, SparkConf</span></span></blockquote>
<blockquote class="tr_bq">
<span style="color: #032f62; font-family: SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace;"><span style="background-color: white; font-size: 12px; white-space: pre;"></span></span><span style="color: #032f62; font-family: SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace;"><span style="background-color: white; font-size: 12px; white-space: pre;">from pyspark.sql import SparkSession</span></span></blockquote>
<span style="color: #032f62; font-family: SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace;"><span style="background-color: white; font-size: 12px; white-space: pre;"><br /></span></span>
<br />
<ul>
<li>now click on "run"</li>
</ul>
<div>
<b>Solution</b></div>
<div>
Add following in .bashrc and restart jupyter notebook</div>
<div>
<br /></div>
<div>
<blockquote class="tr_bq">
#For jupyter notebook:<br />export PYSPARK_SUBMIT_ARGS="--master yarn-client
pyspark-shell"</blockquote>
<div class="MsoNormal">
<o:p></o:p></div>
<div class="MsoNormal">
<o:p></o:p></div>
<b>Root Cause Analysis</b></div>
<div>
If you look at the code below</div>
<div>
<a href="https://github.com/apache/spark/blob/master/python/pyspark/java_gateway.py#L104">https://github.com/apache/spark/blob/master/python/pyspark/java_gateway.py#L104</a></div>
<div>
<br /></div>
<div>
it says connection file is not being created in tmp directory. In my case issue was environment variable "PYSPARK_SUBMIT_ARGS" was not setup. This caused to not to create "proc"</div>
<div>
<br /></div>
<div>
<span style="background-color: white; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;">submit_args </span><span class="pl-k" style="background-color: white; box-sizing: border-box; color: #d73a49; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;">=</span><span style="background-color: white; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;"> os.environ.get(</span><span class="pl-s" style="background-color: white; box-sizing: border-box; color: #032f62; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;"><span class="pl-pds" style="box-sizing: border-box;">"</span>PYSPARK_SUBMIT_ARGS<span class="pl-pds" style="box-sizing: border-box;">"</span></span><span style="background-color: white; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;">, </span><span class="pl-s" style="background-color: white; box-sizing: border-box; color: #032f62; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;"><span class="pl-pds" style="box-sizing: border-box;">"</span>pyspark-shell<span class="pl-pds" style="box-sizing: border-box;">"</span></span><span style="background-color: white; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;">)</span></div>
<div>
<span style="background-color: white; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;">...</span></div>
<div>
<span style="background-color: white; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;"> command </span><span class="pl-k" style="background-color: white; box-sizing: border-box; color: #d73a49; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;">=</span><span style="background-color: white; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;"> command </span><span class="pl-k" style="background-color: white; box-sizing: border-box; color: #d73a49; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;">+</span><span style="background-color: white; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;"> shlex.split(submit_args)</span></div>
<div>
<span style="background-color: white; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;">...</span></div>
<div>
<span style="background-color: white; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;">proc </span><span class="pl-k" style="background-color: white; box-sizing: border-box; color: #d73a49; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;">=</span><span style="background-color: white; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;"> Popen(command, </span><span class="pl-k" style="background-color: white; box-sizing: border-box; color: #d73a49; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;">**</span><span style="background-color: white; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;">popen_kwargs)</span></div>
<div>
<span style="background-color: white; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;"><br /></span></div>
<div>
<span style="background-color: white; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;">...</span></div>
<div>
<span style="background-color: white; font-size: 12px; white-space: pre;"><span style="color: #24292e; font-family: SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace;"> # Wait for the file to appear, or for the process to exit, whichever happens first.
while not proc.poll() and not os.path.isfile(conn_info_file):
time.sleep(0.1)
if not os.path.isfile(conn_info_file):
raise Exception("Java gateway process exited before sending its port number")</span></span></div>
<div>
<span style="background-color: white; font-size: 12px; white-space: pre;"><span style="color: #24292e; font-family: SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace;">...</span></span></div>
</div>
Playing with Linux.http://www.blogger.com/profile/01147934604287011206noreply@blogger.com4tag:blogger.com,1999:blog-4333613427966952010.post-28507384485588143882017-11-08T05:20:00.005-08:002017-11-08T05:20:45.992-08:00Docker build encounters error "failed to unshare namespaces: Invalid argument"<div dir="ltr" style="text-align: left;" trbidi="on">
<h2 style="text-align: left;">
<b>Issue</b></h2>
<br />
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin-top:0in;
margin-right:0in;
margin-bottom:8.0pt;
margin-left:0in;
line-height:107%;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{font-family:"Calibri",sans-serif;}
.MsoPapDefault
{margin-bottom:8.0pt;
line-height:107%;}
/* Page Definitions */
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
-->
</style>
<br />
<div class="WordSection1">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; text-autospace: none;">
<span style="color: #333333; font-family: "Courier New"; font-size: 10.0pt;">_[91mnsenter: failed to unshare namespaces: Invalid argument<br />
_[0m_[91mcontainer_linux.go:265: starting container process caused
"process_linux.go:270: running exec setns process for init caused
\"exit status 34\""<br />
_[0moci runtime error: container_linux.go:265: starting container process
caused "process_linux.go:270: running exec setns process for init caused
\"exit status 34\""</span></div>
<h2 style="line-height: normal; margin-bottom: 0.0001pt; text-align: left;">
<span style="color: #333333; font-family: Courier New;"><span style="font-size: 13.3333px;">Environment</span></span></h2>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; text-autospace: none;">
<span style="color: #333333; font-family: "Courier New"; font-size: 10.0pt;"> RHEL 7.2</span></div>
<h2 style="line-height: normal; margin-bottom: 0.0001pt; text-align: left;">
<span style="color: #333333; font-family: "Courier New"; font-size: 10.0pt;">Solution</span></h2>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; text-autospace: none;">
<span style="color: #333333; font-family: "Courier New"; font-size: 10.0pt;"><br /></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; text-autospace: none;">
<span style="color: #333333; font-family: "Courier New"; font-size: 10.0pt;">Execute below commands as a root user and reboot system.</span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; text-autospace: none;">
<span style="color: #333333; font-family: "Courier New"; font-size: 10.0pt;"><br /></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; text-autospace: none;">
<span style="color: #333333; font-family: Courier New;"><span style="font-size: 13.3333px;">grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"</span></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; text-autospace: none;">
<span style="color: #333333; font-family: Courier New;"><span style="font-size: 13.3333px;">reboot</span></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; text-autospace: none;">
<span style="color: #333333; font-family: Courier New;"><span style="font-size: 13.3333px;"><br /></span></span></div>
<h2 style="line-height: normal; margin-bottom: 0.0001pt; text-align: left;">
<span style="color: #333333; font-family: Courier New;"><span style="font-size: 13.3333px;">Root cause</span></span></h2>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; text-autospace: none;">
<span style="color: #333333; font-family: Courier New;"><span style="font-size: 13.3333px;"><br /></span></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; text-autospace: none;">
<span style="color: #333333; font-family: Courier New;"><span style="font-size: 13.3333px;">usernamespace was enable at docker daemon level but same was not enabled in kernel level.</span></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; text-autospace: none;">
<span style="color: #333333; font-family: Courier New;"><span style="font-size: 13.3333px;"><br /></span></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; text-autospace: none;">
<span style="color: #333333; font-family: Courier New;"><span style="font-size: 13.3333px;">Docker daemon runs like this:</span></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; text-autospace: none;">
<span style="color: #333333; font-family: Courier New;"><span style="font-size: 13.3333px;"><br /></span></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; text-autospace: none;">
<span style="color: #333333; font-family: Courier New;"><span style="font-size: 13.3333px;">ExecStart=/usr/bin/dockerd --live-restore --userland-proxy=false --icc=false --disable-legacy-registry --userns-remap=docker:docker --tlsverify --tlscacert=/var/SP/docker/certs/ca.pem --tlscert=/var/SP/docker/certs/cert.pem --tlskey=/var/SP/docker/certs/key.pem -g /var/SP/docker -H tcp://0.0.0.0:2376 -H unix://var/run/docker.sock --log-driver=syslog --log-opt syslog-address=tcp://localhost:514 --log-opt tag="DOCKER"</span></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; text-autospace: none;">
<span style="color: #333333; font-family: Courier New;"><span style="font-size: 13.3333px;"><br /></span></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; text-autospace: none;">
<span style="color: #333333; font-family: Courier New;"><span style="font-size: 13.3333px;"><br /></span></span></div>
</div>
</div>
Playing with Linux.http://www.blogger.com/profile/01147934604287011206noreply@blogger.com0tag:blogger.com,1999:blog-4333613427966952010.post-29595056859243043982017-11-08T05:13:00.004-08:002017-11-08T05:13:55.099-08:00Docker build encounters "operation timed out"<div dir="ltr" style="text-align: left;" trbidi="on">
<h1 style="border-bottom-color: rgb(153, 153, 153); font-family: Arial, sans-serif; font-size: 24px; font-weight: normal; line-height: 1.25; margin: 30px 0px 0px;">
Issue</h1>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; margin-top: 10px; word-wrap: break-word;">
while running docker build, we see following error.</div>
<table class="wysiwyg-macro" data-macro-body-type="PLAIN_TEXT" data-macro-id="b229c907-5cb3-47cd-8cfe-949a00e5b1e0" data-macro-name="code" data-macro-parameters="theme=Midnight|title=errors" data-macro-schema-version="1" data-mce-style="background-image: url('https://confluence.sp.vodafone.com/plugins/servlet/confluence/placeholder/macro-heading?definition=e2NvZGU6dGhlbWU9TWlkbmlnaHR8dGl0bGU9ZXJyb3JzfQ&locale=en_GB&version=2'); background-repeat: no-repeat;" style="background-color: #f0f0f0; background-image: url("/plugins/servlet/confluence/placeholder/macro-heading?definition=e2NvZGU6dGhlbWU9TWlkbmlnaHR8dGl0bGU9ZXJyb3JzfQ&locale=en_GB&version=2"); background-position: 0px 0px; background-repeat: no-repeat; border: 1px solid rgb(221, 221, 221); color: #333333; cursor: move; font-family: Arial, sans-serif; font-size: 14px; margin-top: 10px; padding: 24px 2px 2px; width: 1474.9px;"><tbody>
<tr><td class="wysiwyg-macro-body" style="background-color: white; border: 1px solid rgb(221, 221, 221); cursor: text; margin: 0px; padding: 10px; white-space: pre-wrap;"><pre style="tab-size: 4; white-space: pre-wrap;">fetch http://dl-4.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-4.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz: operation timed out
fetch http://dl-4.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-4.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz: operation timed out
ERROR: unsatisfiable constraints:
</pre>
</td></tr>
</tbody></table>
<h1 style="border-bottom-color: rgb(153, 153, 153); font-family: Arial, sans-serif; font-size: 24px; font-weight: normal; line-height: 1.25; margin: 30px 0px 0px;">
Solution </h1>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; margin-top: 10px; word-wrap: break-word;">
Pass proxy as environment variable and unset at the end of Dockerfile file.</div>
<table class="wysiwyg-macro" data-macro-body-type="PLAIN_TEXT" data-macro-id="ced525da-b1d8-4a1c-8c3b-b65b448bad2f" data-macro-name="code" data-macro-parameters="theme=Midnight|title=proxy" data-macro-schema-version="1" data-mce-style="background-image: url('https://confluence.sp.vodafone.com/plugins/servlet/confluence/placeholder/macro-heading?definition=e2NvZGU6dGhlbWU9TWlkbmlnaHR8dGl0bGU9cHJveHl9&locale=en_GB&version=2'); background-repeat: no-repeat;" style="background-color: #f0f0f0; background-image: url("/plugins/servlet/confluence/placeholder/macro-heading?definition=e2NvZGU6dGhlbWU9TWlkbmlnaHR8dGl0bGU9cHJveHl9&locale=en_GB&version=2"); background-position: 0px 0px; background-repeat: no-repeat; border: 1px solid rgb(221, 221, 221); color: #333333; cursor: move; font-family: Arial, sans-serif; font-size: 14px; margin-top: 10px; padding: 24px 2px 2px; width: 1474.9px;"><tbody>
<tr><td class="wysiwyg-macro-body" style="background-color: white; border: 1px solid rgb(221, 221, 221); cursor: text; margin: 0px; padding: 10px; white-space: pre-wrap;"><pre style="tab-size: 4; white-space: pre-wrap;">ENV http_proxy http://proxyhost.com:3128
ENV https_proxy http://proxyhost.com:3128
...
RUN unset http_proxy https_proxy</pre>
</td></tr>
</tbody></table>
<h1 style="border-bottom-color: rgb(153, 153, 153); font-family: Arial, sans-serif; font-size: 24px; font-weight: normal; line-height: 1.25; margin: 30px 0px 0px;">
Root cause Analysis</h1>
<ol style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; margin: 10px 0px 0px;">
<li>During docker build on a machine which is behind proxy server can encounter this issue. Requests do not go through running Docker engine but go directly. hence, you need to mention proxy in Dockerfile and unset later.</li>
<li>Second issue may happen that upstream server is not available. This is very rare. You can view with below command to see current repositories</li>
</ol>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; margin-top: 10px; word-wrap: break-word;">
RUN cat /etc/apk/repositories</div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; margin-top: 10px; word-wrap: break-word;">
Default values are</div>
<table class="wysiwyg-macro" data-macro-body-type="PLAIN_TEXT" data-macro-id="2283e93e-7142-41d3-9dac-33e893b1a036" data-macro-name="code" data-macro-parameters="theme=Midnight|title=default repo of alpine" data-macro-schema-version="1" data-mce-style="background-image: url('https://confluence.sp.vodafone.com/plugins/servlet/confluence/placeholder/macro-heading?definition=e2NvZGU6dGhlbWU9TWlkbmlnaHR8dGl0bGU9ZGVmYXVsdCByZXBvIG9mIGFscGluZX0&locale=en_GB&version=2'); background-repeat: no-repeat;" style="background-color: #f0f0f0; background-image: url("/plugins/servlet/confluence/placeholder/macro-heading?definition=e2NvZGU6dGhlbWU9TWlkbmlnaHR8dGl0bGU9ZGVmYXVsdCByZXBvIG9mIGFscGluZX0&locale=en_GB&version=2"); background-position: 0px 0px; background-repeat: no-repeat; border: 1px solid rgb(221, 221, 221); color: #333333; cursor: move; font-family: Arial, sans-serif; font-size: 14px; margin-top: 10px; padding: 24px 2px 2px; width: 1474.9px;"><tbody>
<tr><td class="wysiwyg-macro-body" style="background-color: white; border: 1px solid rgb(221, 221, 221); cursor: text; margin: 0px; padding: 10px; white-space: pre-wrap;"><pre style="tab-size: 4; white-space: pre-wrap;">http://dl-cdn.alpinelinux.org/alpine/v3.4/main
http://dl-cdn.alpinelinux.org/alpine/v3.4/community</pre>
</td></tr>
</tbody></table>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; margin-top: 10px; word-wrap: break-word;">
<br /></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; margin-top: 10px; word-wrap: break-word;">
You can check through browser if you can see if domain is resolving or not. you can replace with actual domain like below</div>
<table class="wysiwyg-macro" data-macro-body-type="PLAIN_TEXT" data-macro-id="669a1b11-d075-4b17-9e73-435b7d031221" data-macro-name="code" data-macro-parameters="theme=Midnight|title=alternative alpine repo" data-macro-schema-version="1" data-mce-style="background-image: url('https://confluence.sp.vodafone.com/plugins/servlet/confluence/placeholder/macro-heading?definition=e2NvZGU6dGhlbWU9TWlkbmlnaHR8dGl0bGU9YWx0ZXJuYXRpdmUgYWxwaW5lIHJlcG99&locale=en_GB&version=2'); background-repeat: no-repeat;" style="background-color: #f0f0f0; background-image: url("/plugins/servlet/confluence/placeholder/macro-heading?definition=e2NvZGU6dGhlbWU9TWlkbmlnaHR8dGl0bGU9YWx0ZXJuYXRpdmUgYWxwaW5lIHJlcG99&locale=en_GB&version=2"); background-position: 0px 0px; background-repeat: no-repeat; border: 1px solid rgb(221, 221, 221); color: #333333; cursor: move; font-family: Arial, sans-serif; font-size: 14px; margin-top: 10px; padding: 24px 2px 2px; width: 1474.9px;"><tbody>
<tr><td class="wysiwyg-macro-body" style="background-color: white; border: 1px solid rgb(221, 221, 221); cursor: text; margin: 0px; padding: 10px; white-space: pre-wrap;"><pre style="tab-size: 4; white-space: pre-wrap;">RUN sed -i 's/dl-cdn.alpinelinux.org/dl-4.alpinelinux.org/' /etc/apk/repositories</pre>
</td></tr>
</tbody></table>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; margin-top: 10px; word-wrap: break-word;">
<br /></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; margin-top: 10px; word-wrap: break-word;">
<br /></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; margin-top: 10px; word-wrap: break-word;">
<br /></div>
<div style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; margin-top: 10px; word-wrap: break-word;">
<br /></div>
</div>
Playing with Linux.http://www.blogger.com/profile/01147934604287011206noreply@blogger.com0tag:blogger.com,1999:blog-4333613427966952010.post-57924901796708187622016-02-10T04:27:00.002-08:002016-02-10T04:37:38.689-08:00Why Puppet was using 96% CPU while ran as a daemon or through command prompt?<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<b>Problem :</b> Puppet was consuming 96% CPU while it ran as a daemon or ran as "puppet agent -t".<br />
<b>Environment:</b> RHEL 6<br />
<b>Solution : </b>Make sure<b> </b>application puppet module is tested with out any error and unwanted issue.<br />
<b>Root Cause Analysis: </b><br />
<br />
top has shown following:<br />
<br />
<code></code><br />
<span style="background-color: white; color: #333333; font-family: "arial" , sans-serif; font-size: 14px; line-height: 22.2222px;">PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND</span><br />
<span style="background-color: white; color: #333333; font-family: "arial" , sans-serif; font-size: 14px; line-height: 22.2222px;">11403 root 20 0 259m 140m 4516 R 96.7 1.9 9:08.77 puppet</span><br />
<span style="color: #333333; font-family: "arial" , sans-serif;"><span style="background-color: white; font-size: 14px; line-height: 22.2222px;"></span></span></div>
<div style="background-color: #f0f0f0; color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 22.2222px; margin-top: 10px; padding: 0px;">
1. Traced puppet agent which was run at command prompt while I kept daemon off.</div>
<div style="background-color: #f0f0f0; color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 22.2222px; margin-top: 10px; padding: 0px;">
<code></code></div>
<div style="background-color: #f0f0f0; color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 22.2222px; margin-top: 10px; padding: 0px;">
$strace -Tvv puppet agent -t &> ./strace_log</div>
<div style="background-color: #f0f0f0; color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 22.2222px; margin-top: 10px; padding: 0px;">
</div>
<br />
2. read() system calls are spawned a lot and continuously printing following messages:<br />
<div class="code panel" style="background: rgb(245, 245, 245); border-radius: 3px; border: 1px solid rgb(204, 204, 204); color: #333333; font-family: monospace; font-size: 12px; line-height: 1.33333; margin: 9px 0px; padding: 0px;">
<div class="codeContent panelContent" style="margin: 0px; padding: 9px 12px;">
<pre class="code-java" style="max-height: 30em; overflow: auto; padding: 0px; white-space: pre-wrap; word-wrap: normal;"><span style="background-color: #f0f0f0; font-family: "arial" , sans-serif; font-size: 14px; line-height: 22.2222px; white-space: normal;"><code></code></span></pre>
<pre class="code-java" style="max-height: 30em; overflow: auto; padding: 0px; white-space: pre-wrap; word-wrap: normal;">[...]
22:36:37 read(4, <span class="code-quote" style="color: #009100;">"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"</span>..., 4096) = 4096 <0 .000031="">
[...]
<!--0--><!--0--></0></pre>
<pre class="code-java" style="max-height: 30em; overflow: auto; padding: 0px; white-space: pre-wrap; word-wrap: normal;"><span style="background-color: #f0f0f0; font-family: "arial" , sans-serif; font-size: 14px; line-height: 22.2222px; white-space: normal;"></span></pre>
</div>
</div>
<div style="background-color: #f0f0f0; color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 22.2222px; margin-top: 10px; padding: 0px;">
3. While I was tracking filename of corresponding file descriptor, I found /home/qpadm/temp was opened that had size 101GB.<br />
4. read() call was continuously trying to read data from this file and read size was 4096 Bytes in every call. Trace looked below:</div>
<div class="code panel" style="background: rgb(245, 245, 245); border-radius: 3px; border: 1px solid rgb(204, 204, 204); color: #333333; font-family: monospace; font-size: 12px; line-height: 1.33333; margin: 9px 0px; padding: 0px;">
<div class="codeContent panelContent" style="margin: 0px; padding: 9px 12px;">
<pre class="code-java" style="max-height: 30em; overflow: auto; padding: 0px; white-space: pre-wrap; word-wrap: normal;"><span style="background-color: #f0f0f0; font-family: "arial" , sans-serif; font-size: 14px; line-height: 22.2222px; white-space: normal;"><code></code></span></pre>
<pre class="code-java" style="max-height: 30em; overflow: auto; padding: 0px; white-space: pre-wrap; word-wrap: normal;">22:36:37 open(<span class="code-quote" style="color: #009100;">"/home/qpadm/temp"</span>, O_RDONLY) = 4 <0 .000032="">
22:36:37 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 <0 .000026="">
22:36:37 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 <0 .000026="">
22:36:37 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 <0 .000027="">
22:36:37 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 <0 .000026="">
22:36:37 fstat(4, {st_dev=makedev(202, 1), st_ino=790228, st_mode=S_IFREG|0600, st_nlink=1, st_uid=1000, st_gid=1000, st_blksize=4096, st_blocks=209715224, st_size=107374182400, st_atime=2016/02/03-07:37:15, st_mtime=2016/02/03-05:11:57, st_ctime=2016/02/03-07:36:48}) = 0 <0 .000027="">
22:36:37 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6d03f3f000 <0 .000030="">
22:36:37 read(4, <span class="code-quote" style="color: #009100;">"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"</span>..., 4096) = 4096 <0 .000035="">
22:36:37 read(4, <span class="code-quote" style="color: #009100;">"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"</span>..., 4096) = 4096 <0 .000031="">
<!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--></0></0></0></0></0></0></0></0></0></pre>
<pre class="code-java" style="max-height: 30em; overflow: auto; padding: 0px; white-space: pre-wrap; word-wrap: normal;"><span style="background-color: #f0f0f0; font-family: "arial" , sans-serif; font-size: 14px; line-height: 22.2222px; white-space: normal;"></span></pre>
</div>
</div>
<div style="background-color: #f0f0f0; color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 22.2222px; margin-top: 10px; padding: 0px;">
In above trace, you see file /home/qpadm/temp is opened in read mode as FD(file descriptor) is 4 and same FD is passed to read() call that returns 4096 Bytes incrementally. It supposed to take long time to read 101GB file<span class="error">[:) would never be finished using huge read calls those would have neen performed by kernel]</span>. That's why CPU spiked to 60-80% consumed by puppet agent.</div>
<div style="background-color: #f0f0f0; color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 22.2222px; margin-top: 10px; padding: 0px;">
We need to check why above file is there and what is the usage of it and why it has large size.</div>
<div style="background-color: #f0f0f0; color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 22.2222px; margin-top: 10px; padding: 0px;">
This one is I think main culprit.</div>
<div style="background-color: #f0f0f0; color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 22.2222px; margin-top: 10px; padding: 0px;">
Another thing I noticed is either ovo was reinstalled or being migrated to newer version. In relation to this, I found following calls :</div>
<div class="code panel" style="background: rgb(245, 245, 245); border-radius: 3px; border: 1px solid rgb(204, 204, 204); color: #333333; font-family: monospace; font-size: 12px; line-height: 1.33333; margin: 9px 0px; padding: 0px;">
<div class="codeContent panelContent" style="margin: 0px; padding: 9px 12px;">
<pre class="code-java" style="max-height: 30em; overflow: auto; padding: 0px; white-space: pre-wrap; word-wrap: normal;"><span style="background-color: #f0f0f0; font-family: "arial" , sans-serif; font-size: 14px; line-height: 22.2222px; white-space: normal;"><code></code></span></pre>
<pre class="code-java" style="max-height: 30em; overflow: auto; padding: 0px; white-space: pre-wrap; word-wrap: normal;">22:36:00 open(<span class="code-quote" style="color: #009100;">"/root/ovoinstall/HPOvPacc.xml"</span>, O_RDONLY) = 4 <0 .000022="">
22:36:07 open(<span class="code-quote" style="color: #009100;">"/root/ovoinstall/OVO-Agent-Migrate.xml"</span>, O_RDONLY) = 4 <0 .000032="">
22:36:07 fstat(4, {st_dev=makedev(202, 1), st_ino=786799, st_mode=S_IFREG|0744, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=4240, st_size=2168841, st_atime=2016/02/03-17:04:19, st_mtime=2016/01/03-22:12:54, st_ctime=2016/01/03-22:15:32}) = 0 <0 .000027="">
22:36:07 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6d03f3f000 <0 .000029="">
22:36:07 read(4, <span class="code-quote" style="color: #009100;">"\355\253\356\333\3\0\0\0\0\1HPOvConf-11.11.025-1\0\0"</span>..., 4096) = 4096 <0 .000032="">
22:36:07 read(4, <span class="code-quote" style="color: #009100;">"\"</span>Unable to copy file\<span class="code-quote" style="color: #009100;">"\nOVCSL_ERRO"</span>..., 4096) = 4096 <0 .000029=""><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--></0></0></0></0></0></0></pre>
<pre class="code-java" style="max-height: 30em; overflow: auto; padding: 0px; white-space: pre-wrap; word-wrap: normal;"><span style="background-color: #f0f0f0; font-family: "arial" , sans-serif; font-size: 14px; line-height: 22.2222px; white-space: normal;"></span></pre>
</div>
</div>
<pre class="code-java" style="max-height: 30em; overflow: auto; padding: 0px; white-space: pre-wrap; word-wrap: normal;"></pre>
<pre class="code-java" style="max-height: 30em; overflow: auto; padding: 0px; white-space: pre-wrap; word-wrap: normal;">Application team found that temp was not needed but was called in application puppet module. They disabled it and removed that file. It fixed the issue.</pre>
<pre class="code-java" style="max-height: 30em; overflow: auto; padding: 0px; white-space: pre-wrap; word-wrap: normal;"></pre>
<pre class="code-java" style="max-height: 30em; overflow: auto; padding: 0px; white-space: pre-wrap; word-wrap: normal;"></pre>
</div>
Playing with Linux.http://www.blogger.com/profile/01147934604287011206noreply@blogger.com0tag:blogger.com,1999:blog-4333613427966952010.post-56284184695009326422016-02-03T01:38:00.000-08:002016-02-03T01:38:13.377-08:00why vxdisk does throw error "VxVM vxdctl ERROR V-5-1-16007 Data Corruption Protection Activated - User Corrective Action Needed To recover"<div dir="ltr" style="text-align: left;" trbidi="on">
<b>Problem:</b><br />
<br />
Following errors has been encountered :<br />
<br />
[root@customer-pet-db-3a ~]# /sbin/vxdctl enable<br />
VxVM vxdctl ERROR V-5-1-16007 Data Corruption Protection Activated - User Corrective Action Needed<br />
To recover, first ensure that the OS device tree is up to date (requires OS specific commands).<br />
Then, execute 'vxdisk rm' on the following devices before reinitiating device discovery:<br />
fas31400_85, fas31400_88, fas31400_92, fas31400_98, fas31400_93<br />
fas31400_96, fas31400_94, fas31400_95, fas31400_97, fas31400_99<br />
fas31402_41, fas31401_39<br />
[root@customer-pet-db-3a ~]#<br />
<br />
<br />
[root@customer-pet-db-3a ~]# /sbin/vxdisk scandisks<br />
VxVM vxdisk ERROR V-5-1-16007 Data Corruption Protection Activated - User Corrective Action Needed<br />
To recover, first ensure that the OS device tree is up to date (requires OS specific commands).<br />
Then, execute 'vxdisk rm' on the following devices before reinitiating device discovery:<br />
fas31400_85, fas31400_88, fas31400_92, fas31400_98, fas31400_93<br />
fas31400_96, fas31400_94, fas31400_95, fas31400_97, fas31400_99<br />
fas31402_41, fas31401_39<br />
[root@customer-pet-db-3a ~]#<br />
<br />
<b>Solution referred :</b><br />
<b><br /></b>
<br />
<b>1. check disk :</b><br />
<br />
[root@customer-pet-db-3a ~]# vxdisk -o alldgs list<br />
DEVICE TYPE DISK GROUP STATUS<br />
cciss/c0d0 auto:none - - online invalid<br />
fas31400_10 auto:cdsdisk flex3a_myapp_arch flex3a_arch_dg online thinrclm<br />
fas31400_11 auto:cdsdisk flex3a_dcd_arch flex3a_arch_dg online thinrclm<br />
fas31400_12 auto:cdsdisk flex3a_myapps_arch flex3a_arch_dg online thinrclm<br />
fas31400_13 auto:cdsdisk flex3a_rpt_arch flex3a_arch_dg online thinrclm<br />
fas31400_85 auto:cdsdisk myapps_pet_data00 flex3a_myapps_pet_data_dg online thinrclm<br />
fas31400_88 auto:cdsdisk myapps_pet_data01 flex3a_myapps_pet_data_dg online thinrclm<br />
fas31400_89 auto:cdsdisk customerumprd_arch01 flex3a_pet_customerumprd_arch_dg online thinrclm<br />
fas31400_90 auto:cdsdisk customerumprd_u02 flex3a_pet_customerumprd_dg online thinrclm<br />
fas31400_91 auto:cdsdisk customerumprd_u01 flex3a_pet_customerumprd_dg online thinrclm<br />
fas31400_92 auto:cdsdisk - (myapp_pet_data_dg) online thinrclm<br />
fas31400_93 auto:cdsdisk - (myapp_pet_data_dg) online thinrclm<br />
fas31400_94 auto:cdsdisk - (myapp_pet_data_dg) online thinrclm<br />
fas31400_95 auto:cdsdisk - (myapp_pet_data_dg) online thinrclm<br />
fas31400_96 auto:cdsdisk - (myapp_pet_data_dg) online thinrclm<br />
fas31400_97 auto:cdsdisk - (myapp_pet_data_dg) online thinrclm<br />
fas31400_98 auto:cdsdisk - (myapp_pet_data_dg) online thinrclm<br />
fas31400_99 auto:cdsdisk - (myapp_pet_data_dg) online thinrclm<br />
fas31401_39 auto:cdsdisk myapps_pet_data02 flex3a_myapps_pet_data_dg online thinrclm<br />
fas31402_41 auto:cdsdisk myapps_pet_data03 flex3a_myapps_pet_data_dg online thinrclm<br />
[root@customer-pet-db-3a ~]#<br />
<b><br /></b>
<b>2. Removed all disk of myapp and myapps like as it is stated above :</b><br />
<br />
<br />
[root@customer-pet-db-3a ~]# for i in fas31400_85 fas31400_88 fas31400_93 fas31400_94 fas31400_95 fas31400_96 fas31400_97 fas31400_98 fas31400_99 fas31401_39 fas31402_41; do vxdisk rm $i; done<br />
VxVM vxdisk ERROR V-5-1-534 Device fas31400_85: Device is in use<br />
VxVM vxdisk ERROR V-5-1-534 Device fas31400_88: Device is in use<br />
VxVM vxdisk ERROR V-5-1-534 Device fas31401_39: Device is in use<br />
VxVM vxdisk ERROR V-5-1-534 Device fas31402_41: Device is in use<br />
[root@customer-pet-db-3a ~]# vxdg list<br />
NAME STATE ID<br />
flex3a_pet_customerumprd_arch_dg enabled,cds 1446853137.45.customer-pet-db-3a.example.net<br />
flex3a_arch_dg enabled,cds 1376078555.40.customer-pet-db-3a.example.net<br />
flex3a_myapps_pet_data_dg enabled,cds 1375818468.64.customer-pet-db-3c.example.net<br />
flex3a_pet_customerumprd_dg enabled,cds 1446683837.59.customer-pet-db-3b.example.net<br />
<br />
flex3a_myapps_pet_data_dg is still in use. Need to deport it.<br />
<br />
[root@customer-pet-db-3a ~]# /sbin/vxdg deport flex3a_myapps_pet_data_dg<br />
<br />
[root@customer-pet-db-3a ~]# vxdg list<br />
NAME STATE ID<br />
flex3a_pet_customerumprd_arch_dg enabled,cds 1446853137.45.customer-pet-db-3a.example.net<br />
flex3a_arch_dg enabled,cds 1376078555.40.customer-pet-db-3a.example.net<br />
flex3a_pet_customerumprd_dg enabled,cds 1446683837.59.customer-pet-db-3b.example.net<br />
[root@customer-pet-db-3a ~]#<br />
<br />
<br />
R<b>emove disks which were in use : </b><br />
<br />
[root@customer-pet-db-3a ~]# for i in fas31400_85 fas31400_88 fas31400_93 fas31400_94 fas31400_95 fas31400_96 fas31400_97 fas31400_98 fas31400_99 fas31401_39 fas31402_41; do vxdisk rm $i; done<br />
VxVM vxdisk ERROR V-5-1-639 Failed to obtain locks:<br />
fas31400_93: no such object in the configuration<br />
VxVM vxdisk ERROR V-5-1-639 Failed to obtain locks:<br />
fas31400_94: no such object in the configuration<br />
VxVM vxdisk ERROR V-5-1-639 Failed to obtain locks:<br />
fas31400_95: no such object in the configuration<br />
VxVM vxdisk ERROR V-5-1-639 Failed to obtain locks:<br />
fas31400_96: no such object in the configuration<br />
VxVM vxdisk ERROR V-5-1-639 Failed to obtain locks:<br />
fas31400_97: no such object in the configuration<br />
VxVM vxdisk ERROR V-5-1-639 Failed to obtain locks:<br />
fas31400_98: no such object in the configuration<br />
VxVM vxdisk ERROR V-5-1-639 Failed to obtain locks:<br />
fas31400_99: no such object in the configuration<br />
[root@customer-pet-db-3a ~]#<br />
<br />
<b>don't worry above error as we have already removed disk of myapp. That's why above errors are displayed.</b><br />
<b><br /></b>
<b>check disk now:</b><br />
<br />
[root@customer-pet-db-3a ~]# vxdisk -o alldgs list<br />
DEVICE TYPE DISK GROUP STATUS<br />
cciss/c0d0 auto:none - - online invalid<br />
fas31400_10 auto:cdsdisk flex3a_myapp_arch flex3a_arch_dg online thinrclm<br />
fas31400_11 auto:cdsdisk flex3a_dcd_arch flex3a_arch_dg online thinrclm<br />
fas31400_12 auto:cdsdisk flex3a_myapps_arch flex3a_arch_dg online thinrclm<br />
fas31400_13 auto:cdsdisk flex3a_rpt_arch flex3a_arch_dg online thinrclm<br />
fas31400_89 auto:cdsdisk customerumprd_arch01 flex3a_pet_customerumprd_arch_dg online thinrclm<br />
fas31400_90 auto:cdsdisk customerumprd_u02 flex3a_pet_customerumprd_dg online thinrclm<br />
fas31400_91 auto:cdsdisk customerumprd_u01 flex3a_pet_customerumprd_dg online thinrclm<br />
[root@customer-pet-db-3a ~]#<br />
<br />
<b>3. Enable and Scan disk again :</b><br />
<br />
[root@customer-pet-db-3a ~]# /sbin/vxdctl enable<br />
[root@customer-pet-db-3a ~]# /sbin/vxdisk scandisks<br />
<br />
<b>Didn't see any more error.</b><br />
<br />
<b>4. Export dg now :</b><br />
<br />
[root@customer-pet-db-3a ~]# /sbin/vxdg -C -n flex3a_myapp_pet_data_dg import myapp_pet_data_dg<br />
[root@customer-pet-db-3a ~]# vxdisk -o alldgs list<br />
DEVICE TYPE DISK GROUP STATUS<br />
cciss/c0d0 auto:none - - online invalid<br />
fas31400_10 auto:cdsdisk flex3a_myapp_arch flex3a_arch_dg online thinrclm<br />
fas31400_11 auto:cdsdisk flex3a_dcd_arch flex3a_arch_dg online thinrclm<br />
fas31400_12 auto:cdsdisk flex3a_myapps_arch flex3a_arch_dg online thinrclm<br />
fas31400_13 auto:cdsdisk flex3a_rpt_arch flex3a_arch_dg online thinrclm<br />
fas31400_85 auto:cdsdisk - (flex3a_myapps_pet_data_dg) online thinrclm<br />
fas31400_88 auto:cdsdisk - (flex3a_myapps_pet_data_dg) online thinrclm<br />
fas31400_89 auto:cdsdisk customerumprd_arch01 flex3a_pet_customerumprd_arch_dg online thinrclm<br />
fas31400_90 auto:cdsdisk customerumprd_u02 flex3a_pet_customerumprd_dg online thinrclm<br />
fas31400_91 auto:cdsdisk customerumprd_u01 flex3a_pet_customerumprd_dg online thinrclm<br />
fas31400_92 auto:cdsdisk myapp_pet_data00 flex3a_myapp_pet_data_dg online thinrclm<br />
fas31400_93 auto:cdsdisk myapp_pet_data06 flex3a_myapp_pet_data_dg online thinrclm<br />
fas31400_94 auto:cdsdisk myapp_pet_data03 flex3a_myapp_pet_data_dg online thinrclm<br />
fas31400_95 auto:cdsdisk myapp_pet_data07 flex3a_myapp_pet_data_dg online thinrclm<br />
fas31400_96 auto:cdsdisk myapp_pet_data05 flex3a_myapp_pet_data_dg online thinrclm<br />
fas31400_97 auto:cdsdisk myapp_pet_data04 flex3a_myapp_pet_data_dg online thinrclm<br />
fas31400_98 auto:cdsdisk myapp_pet_data01 flex3a_myapp_pet_data_dg online thinrclm<br />
fas31400_99 auto:cdsdisk myapp_pet_data02 flex3a_myapp_pet_data_dg online thinrclm<br />
fas31401_39 auto:cdsdisk - (flex3a_myapps_pet_data_dg) online thinrclm<br />
fas31401_40 auto:cdsdisk myapp_pet_data08 flex3a_myapp_pet_data_dg online thinrclm<br />
fas31401_41 auto:cdsdisk myapp_pet_data11 flex3a_myapp_pet_data_dg online thinrclm<br />
fas31401_42 auto:cdsdisk myapp_pet_data10 flex3a_myapp_pet_data_dg online thinrclm<br />
fas31401_43 auto:cdsdisk myapp_pet_data09 flex3a_myapp_pet_data_dg online thinrclm<br />
fas31402_41 auto:cdsdisk - (flex3a_myapps_pet_data_dg) online thinrclm<br />
fas31402_45 auto:cdsdisk myapp_pet_data14 flex3a_myapp_pet_data_dg online thinrclm<br />
fas31402_46 auto:cdsdisk myapp_pet_data12 flex3a_myapp_pet_data_dg online thinrclm<br />
fas31402_47 auto:cdsdisk myapp_pet_data15 flex3a_myapp_pet_data_dg online thinrclm<br />
fas31402_48 auto:cdsdisk myapp_pet_data13 flex3a_myapp_pet_data_dg online thinrclm<br />
[root@customer-pet-db-3a ~]#<br />
<br />
<b>Received error for myapps, </b><br />
<br />
[root@customer-pet-db-3a ~]# /sbin/vxdg -C -n flex3a_myapps_pet_data_dg import myapps_pet_data_dg<br />
VxVM vxdg ERROR V-5-1-10978 Disk group myapps_pet_data_dg: import failed:<br />
No valid disk found containing disk group<br />
[root@customer-pet-db-3a ~]#<br />
<br />
<b>DG is already renamed as flex3a_myapps_pet_data_dg. Just import it simply.</b><br />
<br />
[root@customer-pet-db-3a ~]# vxdg import flex3a_myapps_pet_data_dg<br />
<br />
[root@customer-pet-db-3a ~]#<br />
<br />
<br />
<b>check dg now :</b><br />
<br />
[root@customer-pet-db-3a ~]# vxdg list|grep -i myapp<br />
flex3a_myapp_pet_data_dg enabled,cds 1375815375.127.customer-pet-db-3c.example.net<br />
flex3a_myapps_pet_data_dg enabled,cds 1375818468.64.customer-pet-db-3c.example.net<br />
[root@customer-pet-db-3a ~]#<br />
<br />
Confirmed with DBA, they can start DB instances.<br />
<br />
<br /></div>
Playing with Linux.http://www.blogger.com/profile/01147934604287011206noreply@blogger.com1tag:blogger.com,1999:blog-4333613427966952010.post-36973476992861280212015-12-24T01:53:00.001-08:002015-12-24T01:53:04.326-08:00why did kinit throw error "Preauthentication failed while getting initial credentials"?<div dir="ltr" style="text-align: left;" trbidi="on">
<b>Problem: </b><br />
While executed following command :<br />
<br />
#kinit -k -t /root/utilscripts/nsupdateuser.keytab nsupdate@example.com<br />
<br />
it threw error:<br />
<br />
kinit: Preauthentication failed while getting initial credentials<br />
<br />
<b>Solution</b>: Password of user may be wrong, Try to reset and test again.</div>
Playing with Linux.http://www.blogger.com/profile/01147934604287011206noreply@blogger.com0tag:blogger.com,1999:blog-4333613427966952010.post-9797207693007282602015-12-24T01:51:00.000-08:002015-12-24T01:51:07.531-08:00Why did kinit throw error "KDC reply did not match expectations while getting initial credentials"<div dir="ltr" style="text-align: left;" trbidi="on">
<b>Problem: </b><br />
<br />
While executing following command:<br />
<br />
#kinit username@MYDOMAIN.COM -k -t username.keytab<br />
<br />
it threw error :<br />
<br />
kinit: KDC reply did not match expectations while getting initial credentials<br />
<br />
<b>Solution : </b><br />
<br />
user doesn't have remote access to the machine. </div>
Playing with Linux.http://www.blogger.com/profile/01147934604287011206noreply@blogger.com0tag:blogger.com,1999:blog-4333613427966952010.post-21433135653404118802015-12-24T01:49:00.001-08:002015-12-24T01:49:04.997-08:00How to manage kerberos keytab file?<div dir="ltr" style="text-align: left;" trbidi="on">
<b>1. Use klist to display the keytab file entries:</b><br />
<code></code><br />
klist -e -k -t mykeytabfile.keytab<br />
or klist -ekt nsupdateuser.ktab<br />
</div>
<b>or type command:</b><br />
<code></code><br />
#ktutil # execute this command<br />
ktutil: # this prompt will appear<br />
ktutil: read_kt /etc/apache2/http.keytab #read keytab file<br />
ktuilt: list #list all princples<br />
<br />
example:<br />
<br />
[root@customer-prod-util-101 utilscripts]# klist -e -k -t nsupdateuser.ktab<br />
Keytab name: FILE:nsupdateuser.ktab<br />
KVNO Timestamp Principal<br />
---- ----------------- --------------------------------------------------------<br />
1 09/08/15 21:50:45 nsupdate@site1.example.com (aes256-cts-hmac-sha1-96)<br />
[root@customer-prod-util-101 utilscripts]#<br />
<br />
<b><br /></b>
<b>2. Following is an example of the keytab file creation process using kerberos method :</b><br />
<code></code><br />
> ktutil<br />
ktutil: addent -password -p username@example.com -k 1 -e rc4-hmac<br />
Password for username@example.com: [enter your password]<br />
ktutil: addent -password -p username@example.com -k 1 -e aes256-cts<br />
Password for username@example.com: [enter your password]<br />
ktutil: wkt username.keytab<br />
ktutil: quit<br />
<br />
<b>Following is an example using Heimdal Kerberos:</b><br />
<code></code><br />
> ktutil -k username.keytab add -p username@example.com -e arcfour-hmac-md5 -V 1<br />
<br />
<br />
<b>3. Obtain a ticket-granting ticket using the keytab for testing:</b><br />
<br />
You can check that the keytab contains the appropriate encryption key by attempting to use it to obtain a ticket-granting ticket. This can be done using the kinit command:<br />
<code></code><br />
#kinit -k -t /etc/nsupdateuser.keytab nsupdate@example.com # here nsupdate is username exiting in AD. this has privileges to update dns records on win DNS.<br />
#klist <#will show if ticket is created or not<br />
<br />
<b>example :</b><br />
<code></code><br />
[root@customer-prod-util-101 ~]# kinit -k -t /root/utilscripts/nsupdateuser.ktab nsupdate # uses default domain<br />
[root@customer-prod-util-101 ~]# klist<br />
Ticket cache: FILE:/tmp/krb5cc_0<br />
Default principal: nsupdate@site1.example.com<br />
<br />
Valid starting Expires Service principal<br />
12/24/15 05:29:49 12/24/15 15:29:49 krbtgt/site1.example.com@site1.example.com<br />
renew until 12/31/15 05:29:49<br />
[root@customer-prod-util-101 ~]#<br />
<br />
<br />
Or try to login to test if keytab file works :<br />
<b><br /></b>
<b>Test with out keytab file:</b><br />
<code></code><br />
#kinit username@MYDOMAIN.COM<br />
password> // pass password of username<br />
<br />
<b>Test with keytab file:</b><br />
<code></code><br />
#kinit username@MYDOMAIN.COM -k -t username.keytab<br />
<br />
<br />
<b>4. Using a keytab to authenticate scripts:</b><br />
<br />
To execute a script so it has valid Kerberos credentials, use:<br />
<code></code><br />
> kinit username@example.com -k -t mykeytab; myscript<br />
<br />
<b>list out principle :</b><br />
<code></code><br />
>ktutil<br />
ktutil: rkt nsupdateuser.ktab<br />
ktuilt: list<br />
<br />
<b>5. Merging keytab files:</b><br />
<code></code><br />
> ktutil<br />
ktutil: read_kt mykeytab-1<br />
ktutil: read_kt mykeytab-2<br />
ktutil: read_kt mykeytab-3<br />
ktutil: write_kt krb5.keytab<br />
ktutil: quit<br />
<br />
<br />
<b>6. Delete principle:</b><br />
<code></code><br />
#ktuil<br />
ktutil: rkt <keytabfile></keytabfile><br />
ktuilt: list<br />
ktutil: delete_entry slot-number<br />
ktuilt: wkt <keytabfile></keytabfile><br />
ktuilt: quit<br />
<br />
<br />
<b>7. Destroy cached ticket:</b><br />
<code></code><br />
kdestroy -A //all cache will be destroyed<br />
kdestroy -C <cache-name> //this cache will be deleted only</cache-name><br />
<br />
#kdestroy -c "FILE:/tmp/krb5cc_0"<br />
Playing with Linux.http://www.blogger.com/profile/01147934604287011206noreply@blogger.com0tag:blogger.com,1999:blog-4333613427966952010.post-12677532596740050042015-09-28T02:33:00.004-07:002015-09-28T02:33:54.335-07:00Why did df command throw "disk" Input output error<div dir="ltr" style="text-align: left;" trbidi="on">
■ <b>Issue/Symptom </b> : While DBA was starting oracle instance, it was
failing. On checking FS, it was found that arch volumes are not mounted. While
tried to remount them and checked through "df -h", it was throwing below error:
<br />
<pre><span style="background-color: yellow;">[root@customer-pet-oracle-3d ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/cciss/c0d0p3 58G 48G 7.3G 87% /
/dev/cciss/c0d0p1 494M 18M 452M 4% /boot
tmpfs 63G 232M 63G 1% /dev/shm
tmpfs 4.0K 0 4.0K 0% /dev/vx
df: `/cinprds1_arch00': Input/output error
df: `/cinprd1_arch00': Input/output error
df: `/customerdcdp1_arch00': Input/output error
df: `/customerrptp1_arch00': Input/output error
example-prod-sea1utilnas-1a-pet:/vol/customerpet_data
450G 335G 116G 75% /filers/example-prod-sea1utilnas-1a-pet/customerpet_data
[root@customer-pet-oracle-3d ~]</span>
</pre>
■ <b>OS Environment</b> : RHEL 5.5 <br />
■ <b>Software/Application</b> : <br />
<pre>DB : oracle 11.2.0.4
VxVm : VRTSvxvm-5.1.100.000-SP1_RHEL5, Symantec License Manager vxlicrep utility version 3.02.51.010
vxfs : VRTSvxfs-5.1.100.000-SP1_GA_RHEL5
</pre>
■ <b>Customer Environment</b> : ATT PET Oracle DB <br />
■ <b>Investigation</b> : <br />
1. <br />
<pre><span style="background-color: yellow;">$sanlun lun show|grep -i minipet_arc
customer-pet-sea1bfiler-1a: /vol/customer_MINIPET_ARCH/lun1 /dev/sdag host1 FCP 500.1g (536952700928) GOOD
customer-pet-sea1bfiler-1a: /vol/customer_MINIPET_ARCH/lun0 /dev/sdah host1 FCP 500.1g (536952700928) GOOD
customer-pet-sea1bfiler-1a: /vol/customer_MINIPET_ARCH/lun2 /dev/sdai host1 FCP 250g (268435456000) GOOD
customer-pet-sea1bfiler-1a: /vol/customer_MINIPET_ARCH/lun3 /dev/sdaj host1 FCP 250g (268435456000) GOOD</span>
</pre>
2. Check fstab entry how it is : <br />
fstab was : <br />
<pre>/dev/vx/dsk/minipet_arch_dg/cinprds_minipet_vol_arch00 /cinprds1_arch00 vxfs _netdev 0 1
</pre>
3. Check netfs service if running: <br />
<pre>$/etc/init.d/netfs status
</pre>
4. Search dg in log as root : <br />
<pre><span style="background-color: yellow;">$ awk '/arch_dg/ {print $0}' /var/log/messages.*
Sep 25 19:26:08 customer-pet-oracle-3d kernel: vxfs: msgcnt 1 mesg 037: V-2-37: vx_metaioerr - vx_inode_iodone - /dev/vx/dsk/minipet_arch_dg/rpt_minipet_vol_arch00 file system meta data write error in dev/block 0/1104
Sep 25 19:26:08 customer-pet-oracle-3d vxvm:vxconfigd: V-5-1-7935 Disk group minipet_arch_dg: update failed: Disk group has no valid configuration copies
Sep 25 19:26:08 customer-pet-oracle-3d vxvm:vxconfigd: V-5-1-7934 Disk group minipet_arch_dg: Disabled by errors
[...]
Sep 25 19:30:01 customer-pet-oracle-3d kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff811130c6aa00 on device 0xc900130 to DMP<4>vxfs: msgcnt 5 mesg 039: V-2-39: vx_writesuper - /dev/vx/dsk/minipet_arch_dg/cinprds_minipet_vol_arch00 file system super-block write error
Sep 25 19:30:01 customer-pet-oracle-3d kernel: vxfs: msgcnt 6 mesg 037: V-2-37: vx_metaioerr - vx_dirbread - /dev/vx/dsk/minipet_arch_dg/cinprds_minipet_vol_arch00 file system meta data write error in dev/block 0/1104
[...]
Sep 25 19:40:01 customer-pet-oracle-3d kernel: vxfs: msgcnt 21 mesg 039: V-2-39: vx_writesuper - /dev/vx/dsk/minipet_arch_dg/rpt_minipet_vol_arch00 file system super-block write error
Sep 25 19:40:01 customer-pet-oracle-3d kernel: vxfs: msgcnt 22 mesg 008: V-2-8: vx_direrr: vx_readdir_int_1 - /dev/vx/dsk/minipet_arch_dg/rpt_minipet_vol_arch00 file system dir inode 5 dev/block 0/150297879 dirent inode 0 error 5
Sep 25 19:40:01 customer-pet-oracle-3d kernel: vxfs: msgcnt 23 mesg 039: V-2-39: vx_writesuper - /dev/vx/dsk/minipet_arch_dg/rpt_minipet_vol_arch00 file system super-block write error
[...]
Sep 26 04:18:13 customer-pet-oracle-3d kernel: vxfs: msgcnt 334 mesg 016: V-2-16: vx_ilisterr: vx_iread - /dev/vx/dsk/minipet_arch_dg/rpt_minipet_vol_arch00 file system error reading inode 3
Sep 26 04:18:13 customer-pet-oracle-3d kernel: vxfs: msgcnt 335 mesg 039: V-2-39: vx_writesuper - /dev/vx/dsk/minipet_arch_dg/rpt_minipet_vol_arch00 file system super-block write error
Sep 26 04:18:13 customer-pet-oracle-3d kernel: vxfs: msgcnt 336 mesg 031: V-2-31: vx_disable - /dev/vx/dsk/minipet_arch_dg/rpt_minipet_vol_arch00 file system disabled</4></span>
</pre>
Above confirms that few blocks are corrupted in disks which are under disk
group "minipet_arch_dg" <br />
<br />■ <b>Permanent Solution</b> : <br />
1. Unmount file system if mounted. 2. Run file system check through fsck :
<br />
<pre><span style="background-color: yellow;">$fsck -F vxfs -o full <mountpont>
</mountpont></span></pre>
<span style="background-color: yellow;">example : </span><br />
<pre><span style="background-color: yellow;">$/opt/VRTS/bin/fsck -o full -y /dev/vx/rdsk/minipet_arch_dg/rpt_minipet_vol_arch00
or
$fsck.vxfs -o full /dev/vx/dsk/minipet_arch_dg/rpt_minipet_vol_arch00</span>
</pre>
3. OR: <br />
or reboot system [make sure fsck is enabled in fstab] <br />
<br />■ <b>Root Cause Analysis</b> : <br />
<br />
Error messages in system log confirm that disk blocks are corrupted. vxiod
was failing to write data. vxconfigd informed kernel that it was unable to
change vxfs config. </div>
Playing with Linux.http://www.blogger.com/profile/01147934604287011206noreply@blogger.com1tag:blogger.com,1999:blog-4333613427966952010.post-73459643092954947672015-08-11T23:31:00.001-07:002015-08-11T23:31:03.717-07:00Why does liwewise throw error like "Problem executing /opt/pbis/bin/ad-cache --delete-all >/dev/null 2>/dev/null"?<div dir="ltr" style="text-align: left;" trbidi="on">
<b style="background-color: white; color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.1999998092651px; line-height: 18.4799995422363px;">OS Environment : Linux</b><br style="background-color: white; color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.1999998092651px; line-height: 18.4799995422363px;" /><b style="background-color: white; color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.1999998092651px; line-height: 18.4799995422363px;">Application : pbis aka likewise with version 8.3</b><br style="background-color: white; color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.1999998092651px; line-height: 18.4799995422363px;" /><b style="background-color: white; color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.1999998092651px; line-height: 18.4799995422363px;">Problem: While executing following command, it was throwing error :</b><br />
<b style="background-color: white; color: #222222; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13.1999998092651px; line-height: 18.4799995422363px;"><br /></b>
<div class="MsoNormal">
<span style="color: #44546a;">/opt/likewise/bin/lwconfig
--file /opt/pbis/bin/lwconfig.txt<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="background: yellow; color: #44546a; mso-highlight: yellow;">Problem executing '/opt/pbis/bin/ad-cache --delete-all >/dev/null
2>/dev/null'</span><span style="color: #44546a;"><o:p></o:p></span></div>
<br />
<div class="MsoNormal">
<span style="color: #44546a;">Error: Error returned by
external program<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="color: #44546a;"><br /></span></div>
<div class="MsoNormal">
<span style="color: #44546a;"><b>Solution : </b>As </span><span style="color: #44546a;">ad-cache is linked to lsa, so copy lsa binary from machine where pbis 8.0 is installed to the affected machine.</span></div>
<div class="MsoNormal">
<span style="color: #44546a;"><b>Solution :</b> ad-cache is linked to lsa binary and this has bug. </span></div>
<div class="MsoNormal">
<br /></div>
</div>
Playing with Linux.http://www.blogger.com/profile/01147934604287011206noreply@blogger.com3tag:blogger.com,1999:blog-4333613427966952010.post-28285724791585945292015-01-28T21:37:00.002-08:002015-01-28T21:37:36.276-08:00How to install python boto module on windows?<div dir="ltr" style="text-align: left;" trbidi="on">
<b>OS Environment : Windows 2007, 64 bit</b><br />
<b>Application : Python boto module</b><br />
<b>Implementation Steps : </b><br />
<br />
1. First Install python 3<br />
2. Execute following steps now :<br />
<br />
From windows command prompt :<br />
<br />
<span style="color: blue;">$cd c:/</span><br />
<span style="color: blue;">c:\>cd Python34/Scripts</span><br />
<span style="color: blue;">c:\Python34\Scripts>pip.exe install -U boto</span><br />
<br />
Output will look like below :<br />
<br />
<span style="color: blue;">Downloading/unpacking boto</span><br />
<span style="color: blue;">Installing collected packages: boto</span><br />
<span style="color: blue;">Successfully installed boto</span><br />
<span style="color: blue;">Cleaning up...</span></div>
Playing with Linux.http://www.blogger.com/profile/01147934604287011206noreply@blogger.com6tag:blogger.com,1999:blog-4333613427966952010.post-40236337823874763462015-01-24T10:40:00.000-08:002015-01-24T10:40:11.467-08:00How to configure Postfix as a SMTP gateway?<div dir="ltr" style="text-align: left;" trbidi="on">
<b>■ </b><b>Requirement: Configure postfix as a smtp gateway server</b><br />
<b>■</b><b>OS Environment : Linux [RHEL 5, RHEL 6]</b><br />
<b>■</b><b>Application: postfix</b><br />
<b>■ Assumption : </b><br />
<br />
<ul style="text-align: left;">
<li><b> Domain name= example.com, </b></li>
<li><b> Internal Mail server IP = 192.168.1.3, </b></li>
<li><b> Gateway mail server IP = 192.168.1.2, </b></li>
<li><b> Internal postfix smtp is pre-configured. </b></li>
</ul>
<br />
<b>■</b><b>Implementation Steps :</b><br />
<br /><b>A. DMZ Mail Server Setup (or gateway mail server): </b> The DMZ mail server forwards the inbound mail to the internal mail server and delivers the outbound mail to internet.<br />
<br /><b>
1. Edit /etc/postfix/main.cf and update the lines below.</b><br />
<br /><span style="color: blue;">
mydestination = <br />
local_recipient_maps = <br />
local_transport = error:local mail delivery is disabled<br />
<br />
mynetworks = 127.0.0.0/8 192.168.1.3<br />
relay_domains = example.com<br />
transport_maps = hash:/etc/postfix/transport<br />
smtpd_recipient_restrictions = permit_mynetworks<br />
reject_unauth_destination</span><br /><br /><b>
2. Edit the file /etc/postfix/transport and add the line below.</b><br />
<br /><span style="color: blue;">
example.com :[192.168.1.3]</span><br />
<br />
NOTE : If you would like to use multiple internal server for multiple users/domains then postfix transport maps should be changed.<br />
<br />
e.g. in main.cf<br />
<br />
transport_maps = /etc/postfix/transport<br />
<br />
in /etc/postfix/transport:<br />
<br /><span style="color: blue;">
user1/domain1 smtp:1-mailserver.example.com<br />
user2/domain2 smtp:2-mailserver.example.com</span><br />
<br /><b>3. Execute following commands to reflect above :</b><br />
<br />
<span style="color: blue;">$ postmap /etc/postfix/transport </span><br />
<span style="color: blue;">$ postfix reload</span>
<br />NOTE : That will deliver email for user1 and user2 to [1,2]-mailserver.example.com. If you don't specify anything in the transport map, the default transport (which is usually deliver locally) will be used.<br /><br />
<b>B. Configure Internal Mail Server : </b><br />
<br />
The internal mail server holds the mailbox and forward all outbound mail to the DMZ mail server.<br />
<br /><b>
1. Edit /etc/postfix/main.cf and update the lines below : </b><br />
<br /><span style="color: blue;">
transport_maps = hash:/etc/postfix/transport</span><br />
<br /><b>
2. Edit file /etc/postfix/transport and add the lines below :</b><br />
<br /><span style="color: blue;">
example.com :<br />
.example.com :<br />
* smtp:[192.168.3.2]</span><br />
<br /><b>
3. Create a transport database file :</b><br />
<br /><span style="color: blue;">$ postmap /etc/postfix/transport</span><br />
<br /><b>
4. Restart the Postfix : </b><br />
<br /><span style="color: blue;">$ service postfix restart</span><br /><!--------><!-----------------><!----------><!-----------></div>
Playing with Linux.http://www.blogger.com/profile/01147934604287011206noreply@blogger.com0tag:blogger.com,1999:blog-4333613427966952010.post-19791180196739218642015-01-23T11:56:00.004-08:002015-01-25T00:40:40.304-08:00How to install ruby on linux server?<div dir="ltr" style="text-align: left;" trbidi="on">
<b>■ Requirement : Install ruby on linux system</b><br />
<b>■ OS Environment : Linux(RHEL, Centos)</b><br />
<b>■ Implementation Steps : </b><br />
<br />
<span style="color: blue;">$ cd /usr/local/src</span><br />
<span style="color: blue;">$ Download latest tar ball of ruby</span><br />
<span style="color: blue;">$ tar xvzf ruby-XXX.tar.gz</span><br />
<span style="color: blue;">$ cd ruby-XXX</span><br />
<span style="color: blue;">$ ./configure</span><br />
<span style="color: blue;">$ make</span><br />
<span style="color: blue;">$ make install</span><br />
<span style="color: blue;">$ ruby rubytest.rb</span></div>
Playing with Linux.http://www.blogger.com/profile/01147934604287011206noreply@blogger.com0tag:blogger.com,1999:blog-4333613427966952010.post-79563602358205835762015-01-23T11:07:00.002-08:002015-01-23T11:07:33.513-08:00How to install FFmpeg, FFmpeg-PHP,Mplayer,Mencoder, flv2tool,LAME, MP3 Encoderon linux server?<div dir="ltr" style="text-align: left;" trbidi="on">
■ <b>Requirement :</b> Install FFmpeg, FFmpeg-PHP,Mplayer,Mencoder, flv2tool,LAME, MP3 Encoderon<br />
■ <b>OS Environment :</b> Linux, RHEL 5, 64 bit<br />
<br />
■ <b>Implementation Steps :</b><br />
<br />
<b>1. Login into server and get root access.</b><br />
<b>2. cd /usr/local/src/</b><br />
<b>3. Download following source file from appropriate vendor sites :</b><br />
<br />
<span style="color: blue;">essential-20061022.tar.bz2</span><br />
<span style="color: blue;">flvtool2_1.0.5_rc6.tgz</span><br />
<span style="color: blue;">lame-3.97.tar.gz</span><br />
<span style="color: blue;">ffmpeg-php-0.5.1.tbz2</span><br />
<span style="color: blue;">libogg-1.1.3.tar.gz</span><br />
<span style="color: blue;">libvorbis-1.1.2.tar.gz</span><br />
<span style="color: blue;">MPlayer-1.0rc2.tar.bz2</span><br />
<span style="color: blue;">ffmpeg-0.5.tar.bz2</span><br />
<br />
<b>4. Extract above modules :</b><br />
<br />
$ <span style="color: blue;">for pkg in lame-3.97.tar.gz libogg-1.1.3.tar.gz libvorbis-1.1.2.tar.gz flvtool2_1.0.5_rc6.tgz essential-20061022.tar.bz2 ffmpeg-php-0.5.1.tbz2 MPlayer-1.0rc2.tar.bz2 ffmpeg-0.5.tar.bz2; do tar -xvzf $pkg; done</span><br />
<br />
<b>5. Create a codecs directory :</b><br />
<br />
<span style="color: blue;">$ mkdir /usr/local/lib/codecs/</span><br />
<br />
<b>6. Install dependent libraries :</b><br />
<br />
$ <span style="color: blue;">yum install gcc gmake make libcpp libgcc libstdc++ gcc4 gcc4-c++ gcc4-gfortran subversion ruby ncurses-devel -y</span><br />
<br />
<b>7. Copy essentials codes in proper location :</b><br />
<br />
<span style="color: blue;">$ cd /usr/local/src/</span><br />
<span style="color: blue;">$ mv /usr/local/src/essential-20061022/* /usr/local/lib/codecs/</span><br />
<span style="color: blue;">$ chmod -R 755 /usr/local/lib/codecs/</span><br />
<br />
<b>8. Install LAME :</b><br />
<br />
<span style="color: blue;">$ cd /usr/local/src/lame-3.97</span><br />
<span style="color: blue;">$ ./configure</span><br />
<span style="color: blue;">$ make </span><br />
<span style="color: blue;">$ make install</span><br />
<br />
<b>9. Install LIBOGG:</b><br />
<br />
<span style="color: blue;">$ cd /usr/local/src/</span><br />
<span style="color: blue;">$ cd /usr/local/src/libogg-1.1.3</span><br />
<span style="color: blue;">$ ./configure --enable-shared ; make ; make install</span><br />
<span style="color: blue;">$ PKG_CONFIG_PATH=/usr/local/lib/pkgconfig</span><br />
<span style="color: blue;">$ export PKG_CONFIG_PATH</span><br />
<br />
Put above config path in root user .bashrc file<br />
<br />
<b>10. Install LIBVORBIS:</b><br />
<br />
<span style="color: blue;">$ cd /usr/local/src/</span><br />
<span style="color: blue;">$ cd /usr/local/src/libvorbis-1.1.2</span><br />
<span style="color: blue;">$ ./configure; make ; make install</span><br />
<br />
<b>11. Install FLVTOOL2</b><br />
<br />
<span style="color: blue;">$ cd /usr/local/src/</span><br />
<span style="color: blue;">$ cd /usr/local/src/flvtool2_1.0.5_rc6/</span><br />
<span style="color: blue;">$ ruby setup.rb config</span><br />
<span style="color: blue;">$ ruby setup.rb setup</span><br />
<span style="color: blue;">$ ruby setup.rb install</span><br />
<br />
<b>12. Install MPLAYER</b><br />
<br />
<span style="color: blue;">$cd /usr/local/src/</span><br />
<span style="color: blue;">$ cd /usr/local/src/MPlayer-1.0rc2</span><br />
<span style="color: blue;">$ ./configure; make; make install</span><br />
<br />
<b>13. Install FFMPEG:</b><br />
<br />
<span style="color: blue;">$ cd /usr/local/src/</span><br />
<span style="color: blue;">$ cd /usr/local/src/ffmpeg-0.5</span><br />
<span style="color: blue;">$ ./configure --enable-libmp3lame --enable-libvorbis --disable-mmx --enable-shared</span><br />
<span style="color: blue;">$ make</span><br />
<span style="color: blue;">$ make install</span><br />
<span style="color: blue;"><br /></span>
<span style="color: blue;">$ export LD_LIBRARY_PATH=/usr/local/lib/</span><br />
<span style="color: blue;"><br /></span>
<span style="color: blue;">$ ln -s /usr/local/lib/libavformat.so.50 /usr/lib/libavformat.so.50</span><br />
<span style="color: blue;">$ ln -s /usr/local/lib/libavcodec.so.51 /usr/lib/libavcodec.so.51</span><br />
<span style="color: blue;">$ ln -s /usr/local/lib/libavutil.so.49 /usr/lib/libavutil.so.49</span><br />
<span style="color: blue;">$ ln -s /usr/local/lib/libmp3lame.so.0 /usr/lib/libmp3lame.so.0</span><br />
<span style="color: blue;">$ ln -s /usr/local/lib/libavformat.so.51 /usr/lib/libavformat.so.51</span><br />
<br />
<b>14. Install FFMPEG-PHP:</b><br />
<br />
<span style="color: blue;">$ cd /usr/local/src/</span><br />
<span style="color: blue;">$ cd /usr/local/src/ffmpeg-php-0.5.1/</span><br />
<span style="color: blue;">$ phpize</span><br />
<span style="color: blue;">$ ./configure</span><br />
<span style="color: blue;">$ make</span><br />
<span style="color: blue;">$ make install</span><br />
<br />
Enable ffmpeg module in php.ini like below<br />
<br />
echo 'extension=ffmpeg.so'<br />
<br />
<b>15. Now check the binaries like below :</b><br />
<br />
<span style="color: blue;">$ for bin in lamp flvtool2l mplayer ffmpeg; do which $bin; done</span><br />
<br />
<br />
<br />
<br /></div>
Playing with Linux.http://www.blogger.com/profile/01147934604287011206noreply@blogger.com0tag:blogger.com,1999:blog-4333613427966952010.post-60345673406178001762015-01-23T00:42:00.001-08:002015-01-23T11:13:08.928-08:00Why did "ls -al" command take much time to return output? <div dir="ltr" style="text-align: left;" trbidi="on">
<b>Issue/Symptom </b> : "ls -al" command took much time than expected. <br />
<br />
<pre style="background-color: ivory; color: green;">time ls -la /data/XXX/mboxes/tier1/01/XXX/15/67/178/XXX/
... (just removing the output of the ls -la command itself)</pre>
<pre style="background-color: ivory; color: green;">real 1m26.674s
user 0m0.285s
sys 0m1.627s
</pre>
<pre style="background-color: ivory; color: green;">
</pre>
<pre style="background-color: ivory; color: green;"></pre>
<pre style="background-color: ivory; color: green;"></pre>
<pre style="background-color: ivory; color: green;"></pre>
■ <b>OS Environment</b> : RHEL 6, storage netapp filer ONTAP 8 <br />
■ <b>Software/Application</b> : storage - FAS3240 over NFSv3, ls
command[coreutils-8.4-16.el6.x86_64] <br />
■ <b>Investigation</b> : Direcotry "XXX" contains around 22481
files.<br />
■ <b>Workaround Solution</b> : Time consumption is expected in this case.
This may not be same on other environment. Please follow RCA section. <br />
<br />
■ <b>Root Cause Analysis</b> : <br />
<br />
<br />
<pre style="background-color: ivory; color: green;">Above time is expected as directory has around 22481 files. "ls -al" retrieves more file attributes details than normal "ls" command.
As per strace analysis, it is not found that any system call which took more than 1 sec. Accumulation of all calls took 1m26s as you see. Compared other user's
directory[has around ~4000 mails], it was not found this much delay. It just took less than 2 sec. Test and details :</pre>
<pre style="background-color: ivory; color: green;">time ls -la /data/XXX/mboxes/tier1/01/XXX/15/67/178/XXX/ << no of file 22481
[...]
real 1m26.674s
user 0m0.285s
sys 0m1.627s</pre>
<pre style="background-color: ivory; color: green;">with out "-al" option :</pre>
<pre style="background-color: ivory; color: green;">time ls /data/XXX/mboxes/tier1/01/XXX/15/67/178/XXX/
[....]
real 0m4.678s
user 0m0.223s
sys 0m1.008s</pre>
<pre style="background-color: ivory; color: green;">This user has less mails :</pre>
<pre style="background-color: ivory; color: green;">time ls -al /data/XXX/mboxes/tier1/01/XXX/15/67/47/XXX << no of file 4442
real 0m1.010s
user 0m0.067s
sys 0m0.297s</pre>
<pre style="background-color: ivory; color: green;">time ls -al /data/XXX2/mboxes/tier1/01/XXX/15/67/47/XXn/
real 0m0.736s
user 0m0.070s
sys 0m0.162s</pre>
<pre style="background-color: ivory; color: green;">ls -al /data/XXX/mboxes/tier1/01/XXX/15/67/183/XXX/|wc -l
6468
took time :
real 0m1.406s
user 0m0.090s
sys 0m0.406s
Strace analysis :
#strace -Tttvv ls -al /data/XXX/mboxes/tier1/01/XXX/15/67/178/XXX/ &> /var/log/strace_op</pre>
<pre style="background-color: ivory; color: green;">#cat /var/log/strace_op|awk '{print $NF}'|egrep -v 'msg|dat|new|bin|size|\?'|sed 's/<//'|sed 's/>//'|less </pre>
<pre style="background-color: ivory; color: green;">#cat /var/log/strace_op|awk '{print $NF}'|egrep -v 'msg|dat|new|bin|size|\?'|sed 's/<//'|sed 's/>//'|awk 'BEGIN {sum=0.0} {sum+=$NF} END {print sum}'
If you see too much penalty in terms of consumption of retrieve time then please engage NetApp to perf analysis. Download perf tool
and retrieve data from filers. Handover this data to Netapp to further analysis.
</pre>
</div>
Playing with Linux.http://www.blogger.com/profile/01147934604287011206noreply@blogger.com0tag:blogger.com,1999:blog-4333613427966952010.post-52238680886077312762015-01-23T00:37:00.003-08:002015-01-23T11:10:53.741-08:00Why inodes is almost full on file system?<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="mw-content-ltr" dir="ltr" lang="en">
<b>Issue/Symptom </b> : dfm: Warning event on filer:/<volume>(Inodes
Almost Full) </volume><br />
■ <b>OS Environment</b> : Netapp OnTAP 8.1 <br />
■ <b>Software/Application</b> : Netapp DFM[5] sends alert that inode is full. <br />
■ <b>Environment</b> : Applicable to all customer who uses netapp
filers <br />
■ <b>Investigation</b> : Huge number of small files are put on volumes <br />
<br />
■ <b>Workaround Solution</b> : <br />
<br />
<pre style="color: green;">Check how is the usage of inodes :
filer> df -i <volume>
Check currently maximum setup :
filer> maxfiles <volume>
Calculate maximum number of inodes it can hold [4KB size of each inode]
Find maximum size of volume :
filer> df -h <volume>
If maximum size is XY GB, then maximum supported inodes :
= (XY*1024*1024)/4
Set new inode value :
filer>maxfiles <volume> </volume></volume></volume></volume></pre>
<pre style="color: green;"><volume><volume><volume><volume> <new value="">
</new></volume></volume></volume></volume></pre>
<pre style="color: green;"></pre>
■ <b>Permanent solution </b> : Same as workaround solution. <br />
■ <b>Root Cause Analysis</b> : No required, is known. <br />
<!--
NewPP limit report
Preprocessor node count: 4/1000000
Post-expand include size: 0/2097152 bytes
Template argument size: 0/2097152 bytes
Expensive parser function count: 0/100
--><!-- Saved in parser cache with key qube:pcache:idhash:40015-0!*!*!*!*!*!* and timestamp 20150123083641 --></div>
<div class="printfooter">
<br /></div>
</div>
Playing with Linux.http://www.blogger.com/profile/01147934604287011206noreply@blogger.com0tag:blogger.com,1999:blog-4333613427966952010.post-56958404711242173332015-01-23T00:36:00.002-08:002015-01-23T00:36:29.775-08:00How to clean unused semaphore<div dir="ltr" style="text-align: left;" trbidi="on">
<b>Issue/Symptom </b> : Sometime you'll see semaphore usage is full. <br />
■ <b>OS Environment</b> : Linux or RHEL <br />
■ <b>Software/Application</b> : HP ovo sends semaphore usage alerts <br />
■ <b>Investigation</b> : Unsed semaphores are not cleared by kernel<br />
■ <b>Workaround Solution</b> : Use below scripts to clean unused semaphore.
<br />
<br />
<br />
<pre style="color: green;">cat clean-unused-semaprhore.sh
#Developed By Kamal maiti,
#check if root can run it.
if [[ $EUID -ne 0 ]]; then
exit 1
else
#collect all semaphore ID
for SEMID in `ipcs -s|egrep -v -e "Semaphore|key"|sed '/^$/d'|awk '{print $2}'|sort -u`
do
#GETPID of semaphore
PID=`ipcs -s -i $SEMID|tail -2|head -1|awk '{print $NF}'`
#GET PROCESS ID
#Ignore process ID 0 which is main process & test PID greater than 0
if [ $PID -gt 0 ]; then
#Test of PID exists in process list, if exits then don't do anything.
if ps -p $PID > /dev/null
then
#running process are
echo "$SEMID $PID" &>/dev/null else
# dead process are, kill corresponding semaphore of related PID is not exisitng.
echo "$SEMID $PID" &>/dev/null
#cleaning semaphore of dead process :
ipcrm -s $SEMID
fi
fi
done
fi
</pre>
<pre style="color: green;">
</pre>
<pre style="color: green;">RUN :chmod +x clean-unused-semaprhore.sh; ./clean-unused-semaprhore.sh</pre>
</div>
Playing with Linux.http://www.blogger.com/profile/01147934604287011206noreply@blogger.com2tag:blogger.com,1999:blog-4333613427966952010.post-41931050081441514472015-01-23T00:34:00.004-08:002015-01-23T12:03:56.994-08:00High CPU usage, server was not accessible over ssh<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="mw-content-ltr" dir="ltr" lang="en">
<span style="font-size: large;">■ <b>Issue/Symptom </b>:</span> High load on server, not accessible over ssh <br />
<span style="font-size: large;">■ <b>OS Environment</b> :</span> RHEL 5.5<br />
<span style="font-size: large;">■ <b>Background Information </b> :</span><br />
<ul>
<li>Infra was running test
</li>
<li>Server was intermittently highly loaded
</li>
<li>ssh was failing :
<ul>
<li>[usera@user01lxv ~]$ ssh 10.57XXX</li>
<li>Password:
</li>
<li>Connection closed by 10.57.XXX</li>
</ul>
</li>
<li>console shows "lockd: rejected NSM callback from 7f000001:30001" and
sometimes NFS is not ok </li>
</ul>
<span style="font-size: large;">■ <b>Investigation</b> : </span><br />
<ul>
<li>iowait was very high and fluctuating.
</li>
<li>All the cpu were busy to serve i/o bound operations </li>
</ul>
<pre><span style="color: blue;">$ mpstat -P ALL 1</span></pre>
<pre><span style="color: blue;">
Linux 2.6.18-128.el5 (xxxxxxx) 11/19/2014
10:17:29 PM CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s
10:17:30 PM all 0.00 0.00 0.00 75.00 0.00 0.00 0.00 25.00 182.18
10:17:30 PM 0 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 182.18
10:17:30 PM 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00</span><span style="color: red;">
</span></pre>
<ul>
<li>top had shown high load or no process took too much cpu </li>
</ul>
<pre><span style="color: blue;">top - 22:18:03 up 50 days, 22:20, 4 users, load average: 25.19, 26.68, 30.74
Tasks: 235 total, 2 running, 231 sleeping, 0 stopped, 2 zombie
Cpu(s): 2.0%us, 0.8%sy, 0.0%ni, 0.0%id, 96.8%wa, 0.0%hi, 0.4%si, 0.0%st
Mem: 3866480k total, 2916884k used, 949596k free, 12440k buffers
Swap: 8385920k total, 498424k used, 7887496k free, 350200k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20841 cw 21 0 5606m 2.1g 5040 S 4.0 56.8 291:05.56 /opt/cw/jre/bin/java -Duser.timezone=America/Mexico_City -Xms2560m -Xmx2560m -XX:MaxPermSize=128m</span><span style="color: red;">
</span></pre>
<ul>
<li>Found that there were lot of "D" stated processes which didn't appear on
nso-102, 101 </li>
</ul>
<pre><span style="color: blue;">$ ps aux |awk '{print $1 " " $8 " " $NF }'|grep D</span></pre>
<pre><span style="color: blue;">
USER STAT COMMAND
root D< [kjournald]
root Ds 0
root Ds /var/run/vmware-guestd.pid
nobody DN /usr/bin/log2mysql-nso-tomcat-writer
nobody DN /usr/bin/log2mysql-nso-tomcat-spooler
root D</span><span -="" -nlp="" auxwwww="" bin="" d="" deployer="" ds="" jconsole="" jobscheduler.properties="" monitor="" opt="" perf="" pre="" root="" s="" scopeux=""><strike style="color: red;">
</strike><ul>
<li>In above output, system thread kjournald is also in D state which looked bad
from kernel perspective. Journalling would have stopped. </li>
</ul>
</span><div style="text-align: left;">
<span -="" -nlp="" auxwwww="" bin="" d="" deployer="" ds="" jconsole="" jobscheduler.properties="" monitor="" opt="" perf="" pre="" root="" s="" scopeux=""><span style="font-size: large;"><b>■ Workaround Solution</b><b> :</b></span><span style="color: red;"><span style="font-size: large;"> </span>
</span></span></div>
<span -="" -nlp="" auxwwww="" bin="" d="" deployer="" ds="" jconsole="" jobscheduler.properties="" monitor="" opt="" perf="" pre="" root="" s="" scopeux=""><span style="color: red;">
</span><pre style="color: green;">Shutdown VM and power on again.[D stated processes can't be killed unless system is rebooted]
<span style="font-size: large;">
</span></pre>
<span style="font-size: large;"><b>■ </b><b>Permanent Solution</b><b> :</b></span><span style="color: red;"><span style="font-size: large;"> </span>
</span><pre style="color: green;">Shutdown VM and power on again. .[D stated processes can't be killed unless system is rebooted]
</pre>
<span style="font-size: large;"><span style="color: red;">
</span><b>■ </b><b>Root Cause Analysis</b><b> :</b></span><span style="color: red;"> </span><ul>
<li>IOwait was mainly taking place as there were high number of D stated
processes.<span style="color: red;"> </span></li>
<span style="color: red;">
</span><strike style="color: red;">
</strike></ul>
<span style="color: red;">
</span><strike style="color: red;">
</strike><!--
NewPP limit report
Preprocessor node count: 16/1000000
Post-expand include size: 0/2097152 bytes
Template argument size: 0/2097152 bytes
Expensive parser function count: 0/100
--><!-- Saved in parser cache with key qube:pcache:idhash:39894-0!*!*!*!*!*!* and timestamp 20150123083318 --></span></pre>
</div>
<div class="printfooter">
<br /></div>
</div>
Playing with Linux.http://www.blogger.com/profile/01147934604287011206noreply@blogger.com0tag:blogger.com,1999:blog-4333613427966952010.post-26702185437366144632015-01-23T00:33:00.002-08:002015-01-23T11:19:14.152-08:00why amazon cloud load balancer was flapping between two instances?<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="mw-content-ltr" dir="ltr" lang="en">
<b><span id="goog_507077534"></span><span id="goog_507077535"></span>Issue/Symptom </b>: <br />
<ul>
<li>Why SMOKETEST LB in amazon aws cloud was flapping between qpass-prod-smktst-201.dub1.qpass.net & qpass-prod-smktst-101.dub1.qpass.net. States changed between "InService" & "OutofService" ? </li>
</ul>
■ <b>OS Environment</b> : <br />
<ul>
<li>Both nodes has RHEL 6, LB is provided by Amazon </li>
</ul>
■ <b>Investigation</b> :<br />
<ul>
<li>LB is mapped to above two nodes. Incomming port is 443, destination port is 80. It is found that applications are listening port 80 on both nodes. Server Health check timeout has been increased in LB, but issue still persisted. </li>
</ul>
■ <b>Permanent Solution</b> : <br />
<br />
<pre style="color: green;">For the LB, at Health check section, Ping target will be TCP:80, Timeout set 5 seconds, Interval 30 seconds, Unhealthy Threshold 2
Healthy Threshold 10
</pre>
<pre style="color: green;"></pre>
<pre style="color: green;"></pre>
<pre style="color: green;"></pre>
■ <b>Root Cause Analysis</b> : <br />
<ul>
<li>It was found that Ping target was HTTP:80, Ping path was /ping.html. Though webbased ping returns OK[200 status code] but it does't work properly. </li>
</ul>
<!--
NewPP limit report
Preprocessor node count: 4/1000000
Post-expand include size: 0/2097152 bytes
Template argument size: 0/2097152 bytes
Expensive parser function count: 0/100
--><!-- Saved in parser cache with key qube:pcache:idhash:39792-0!*!*!*!*!*!* and timestamp 20150123083202 --></div>
<div class="printfooter">
<br /></div>
</div>
Playing with Linux.http://www.blogger.com/profile/01147934604287011206noreply@blogger.com1tag:blogger.com,1999:blog-4333613427966952010.post-31286790605055126152015-01-23T00:31:00.001-08:002015-01-23T11:20:21.973-08:00why netapp dfm does send "Clock Skewed" alert from filer?<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="mw-content-ltr" dir="ltr" lang="en">
<b>Issue/Symptom </b> : DFM sent alert like "Dfm: Error event on <filer>
Clock Skewed" </filer><br />
<br />
■ <b>OS Environment</b> : Netapp ONTAP <br />
<br />
■ <b>Investigation</b> : Not performed <br />
<br />
■ <b>Workaround Solution</b> : <br />
<br />
<pre style="color: green;">"options timed.enable off"
"options timed.enable on"
</pre>
<pre style="color: green;">
</pre>
<pre style="color: green;"></pre>
■ <b>Permanent Solution</b> : <br />
<ul>
<li>Unknown. </li>
</ul>
■ <b>Root Cause Analysis</b> : <br />
<dl>
<dd>unknown </dd></dl>
<!--
NewPP limit report
Preprocessor node count: 4/1000000
Post-expand include size: 0/2097152 bytes
Template argument size: 0/2097152 bytes
Expensive parser function count: 0/100
--><!-- Saved in parser cache with key qube:pcache:idhash:39885-0!*!*!*!*!*!* and timestamp 20150123083106 --></div>
<div class="printfooter">
<br /></div>
</div>
Playing with Linux.http://www.blogger.com/profile/01147934604287011206noreply@blogger.com0tag:blogger.com,1999:blog-4333613427966952010.post-32341939443380145662015-01-23T00:30:00.006-08:002015-01-23T11:22:02.648-08:00Why it was failed to run command ‘/bin/bash’ and was throwing error "No such file or directory" during configuring chrooted ftp?<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="mw-content-ltr" dir="ltr" lang="en">
<b>Issue/Symptom </b> : Receiving following error while execute below command
<br />
<br />
<pre>$chroot /chroot
"Editing Chroot: failed to run command ‘/bin/bash’: No such file or directory
</pre>
■ <b>OS Environment</b> : RHEL 5,6 <br />
■ <b>Involved Software/Application</b> : openssh, coreutils, rssh <br />
■ <b>Investigation</b> : Found that /bin/bash is present inside /chroot
directory.<br />
■ <b>Permanent Solution</b> : <br />
<br />
<pre style="color: green;">1. $ldd /chroot/bin/bash to see dependent libraries.
2. Copy missing libraries from /lib64 or /usr/lib64 to /chroot/lib64
</pre>
<pre style="color: green;">
</pre>
<pre style="color: green;"></pre>
■ <b>Root Cause Analysis</b> : Libraries of /bin/bash were missing or were
partially present. <br />
<!--
NewPP limit report
Preprocessor node count: 7/1000000
Post-expand include size: 0/2097152 bytes
Template argument size: 0/2097152 bytes
Expensive parser function count: 0/100
--><!-- Saved in parser cache with key qube:pcache:idhash:39900-0!*!*!*!*!*!* and timestamp 20150123082957 --></div>
<div class="printfooter">
<br /></div>
</div>
Playing with Linux.http://www.blogger.com/profile/01147934604287011206noreply@blogger.com0tag:blogger.com,1999:blog-4333613427966952010.post-64600830402606466142015-01-23T00:26:00.001-08:002015-01-23T11:24:50.920-08:00why it was throwing error "550 Permission denied" while uploading file on vsftpd server, but no issue was reported to download file?<div dir="ltr" style="text-align: left;" trbidi="on">
<b>Issue :</b> Receiving "550 Permission denied" while uploading file on vsftpd server, no issue to download file.<br />
<br />
<b>Kernel :</b> 2.6.9-55.ELsmp<br />
<b>OS:</b> RHEL 4<br />
<b>vsftpd : </b>vsftpd-2.0.1-5.el4.5, vsftpd-2.0.1-9.el4.i386.rpm<br />
<br />
<br />
<b>Solution :</b><br />
<br />
<span style="background-color: white; color: blue;">enabling "chown_uploads=YES" caused problem. Disabled it like :</span><br />
<span style="background-color: white; color: blue;"><br />
#chown_uploads=YES</span><br />
<span style="background-color: white; color: blue;"><br />
Also enabled below :</span><br />
<span style="background-color: white; color: blue;"><br />
anonymous_enable=YES</span><br />
<br />
<b>Troubleshooting : </b><br />
<br />
<b>from client : following steps had been performed :</b><br />
<br />
ftp> put testfile.txt<br />
local: testfile.txt remote: testfile.txt<br />
---> TYPE I<br />
200 Switching to Binary mode.<br />
ftp: setsockopt (ignored): Permission denied<br />
---> PASV<br />
227 Entering Passive Mode (10,57,71,14,181,149)<br />
---> STOR testfile.txt<br />
550 Permission denied.<br />
ftp> exit<br />
<br />
<br />
<b>from server, we saw following messages in /var/log/vsftpd.log</b><br />
<br />
Thu Jan 22 02:24:17 2015 [pid 9478] [test122] FTP command: Client "10.57.71.126", "PASV"<br />
Thu Jan 22 02:24:17 2015 [pid 9478] [test122] FTP response: Client "10.57.71.126", "227 Entering Passive Mode (10,57,71,14,24,213)"<br />
Thu Jan 22 02:24:17 2015 [pid 9478] [test122] FTP command: Client "10.57.71.126", "STOR testfile.txt"<br />
Thu Jan 22 02:24:17 2015 [pid 9478] [test122] FTP response: Client "10.57.71.126", "550 Permission denied."<br />
<div>
<br /></div>
<div>
<b>successful upload log shows below messages [in /var/log/xferlog]: </b></div>
<div>
<br /></div>
<div>
<div>
Thu Jan 22 04:20:26 2015 1 10.57.71.126 12 /home/test122/upload/testfile.txt b _ i r test122 ftp 0 * c</div>
<div>
Thu Jan 22 04:21:01 2015 1 10.57.71.126 12 /home/test122/upload/testfile.txt b _ o r test122 ftp 0 * c</div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<br /></div>
Playing with Linux.http://www.blogger.com/profile/01147934604287011206noreply@blogger.com0tag:blogger.com,1999:blog-4333613427966952010.post-64328661951003439632013-04-28T12:47:00.001-07:002015-01-23T11:35:10.049-08:00How cisco UCS makes easy to manage IT environemnt and provides optimized usage of resources?<div dir="ltr" style="text-align: left;" trbidi="on">
CISCO launched UCS (unified computing system) few years back and this makes really easy to manage IT environment. CISCO UCS 5k chassis (containing 8 blades), fabric interconnect(6k) and fabric extender (2k) ie FEX form UCS environment where FCOE protocol is used. In a typical setup, two FEX will be connected to one chassis. Each FEX has 8 ethernets and 2 FC ports. Two downlink connection will be for single blade, ie each will be from FEX. It is not required to manual cabling. It is required to insert FEX module. This is also called IO module.<br />
<br />
Vendor has improved unified communication link, there will have one FCOE uplink to Fabric(A and B, for redundancy) to each CISCO fabric interconnect switch. Both A and B Fab can be connected in order to maintain failover. Number cables dramatically reduced. Hence, power consumption less.<br />
<br />
In order to send out traffic to uplink, Nexus 5k switch can be used and this supports FCoE and other features too. In DC, generally we prefer Nexus 5k. All zoning, vlan and trunking are done in this switch. There is also redundancy here ie NX A and NX B. Your storage systems will be connected to this NexUS switch. Even backup will be connected to it. In order to send-out traffic from NEXUS, there are other switches. Cat 6K can be used. NexUS 7k can be also used. I heard somewhere MDS switch is also used. This is basically core switch which interfaces to ISP routers.<br />
<br />
In order to manage UCS, there is UCS manager which is accessible through GUI. One can manage everything through backend. It is required to login into primay fabric interconnect switch and execute commands. From installation to patching, upgrading(FW and software) can be done. UCSM has nice features.<br />
<br />
If vmware ESX is setup on blades and if they form ESX cluster with FT enabled, then your IT environment will 100% up. </div>
Playing with Linux.http://www.blogger.com/profile/01147934604287011206noreply@blogger.com0tag:blogger.com,1999:blog-4333613427966952010.post-44200394981132451662013-04-10T12:58:00.000-07:002015-01-23T11:52:51.811-08:00How server to storage data paths failover works using multipath?<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
<span style="font-size: large;"><b>■ Purpose : </b></span>How server to storage data path failover works using multipath?<br />
<span style="background-color: white;"><b><span style="font-size: large;">■ Environment :</span></b> RHEL 6 </span><br />
<span style="background-color: white;"><br /></span>
<span style="background-color: white; font-size: large;"><b>1. Default multipath.conf configuration looks like :</b></span><br />
<span style="background-color: white;"><br /></span>
<span style="background-color: white;">#multipath.conf<br />#NetApp recommended settings<br /><br /><br /><span style="color: blue;">defaults<br />{<br /> user_friendly_names yes<br /> max_fds max<br /> queue_without_daemon no<br /> bindings_file "/var/lib/multipath/bindings"<br /> uid=500<br /> gid=500<br />}<br />blacklist<br />{<br /> wwid DevId<br /> devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"<br /> devnode "^hd[a-z]"<br /> devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]"<br />}<br />devices<br />{<br /> device<br /> {<br /> vendor "NETAPP"<br /> product "LUN"<br /> getuid_callout "/sbin/scsi_id -g -u -s /block/%n"<br /> prio_callout "/sbin/mpath_prio_ontap /dev/%n"<br /> features "1 queue_if_no_path"<br /> hardware_handler "0"<br /> path_grouping_policy group_by_prio<br /> failback immediate<br /> rr_weight uniform<br /> rr_min_io 128<br /> path_checker directio<br /> flush_on_last_del yes<br /> }<br />}</span><br /><br /><b><span style="font-size: large;">2. View paths :</span></b><br /><br />$ multipath -ll</span><br />
<span style="background-color: white;"><br /><span style="color: blue;">mini_p (360a98000572d45394b34715579354446) dm-23 NETAPP,LUN<br />[size=1.0T][features=1 queue_if_no_path][hwhandler=0][rw]<br />\_ round-robin 0 [prio=8][active]<br /> \_ 1:0:0:0 sda 8:0 [active][ready]<br /> \_ 2:0:1:0 sdca 68:224 [active][ready]<br />\_ round-robin 0 [prio=2][enabled]<br /> \_ 2:0:2:0 sdct 70:16 [active][ready]<br /> \_ 1:0:1:0 sdq 65:0 [active][ready]</span></span><br />
<span style="background-color: white;"><br /><b><span style="font-size: large;"> 3. Explanations are stated below :</span></b></span><br />
<span style="background-color: white;"><br /><span style="color: blue;">mini_p (360a98000572d45394b34715579354446) dm-23 NETAPP,LUN<br />------ --------------------------------- ---- --- ---------------<br /> | | | | |-------> Product<br /> | | | |------------------> Vendor<br /> | | |-----------------------> sysfs name<br /> | |-------------------------------------------------> WWID of the device<br /> |-----------------------------------------------------------------> User defined Alias<br /><br />[size=1.0T][features=1 queue_if_no_path][hwhandler=0][rw]<br /> --------- --------------------------- ----------------<br /> | | |--------------------> Hardware Handler <br /> | |---------------------------------------------> Features supported<br /> |---------------------------------------------------------------> Size of the DM</span><br /><br />Path Group 1:</span><br />
<span style="background-color: white;"><br /><span style="color: blue;">\_ round-robin 0 [prio=8][active]<br />-- ------------- ------ ------<br /> | | | |----------------------------------------> Path group state<br /> | | |-----------------------------------------------> Path group priority<br /> | |--------------------------------------------------------------> Path selector<br /> |-------------------------------------------------------------------> Path group level</span><br /><br />First path on Path Group 1:</span><br />
<span style="background-color: white;"><br /> <span style="color: blue;">\_ 1:0:0:0 sda 8:0 [active][ready]<br /> -------- --- ---- ------ -----<br /> | | | | |---------------------------------> Physical Path state<br /> | | | |----------------------------------------> DM Path state<br /> | | |-------------------------------------------------> Major, minor numbers<br /> | |-------------------------------------------------------> Linux device name<br /> |--------------------------------------------------------------> host,chan,scsiid,lun</span><br /><br />Second path on Path Group 1:</span><br />
<span style="background-color: white;"><br /> <span style="color: blue;">\_ 2:0:1:0 sdca 68:224 [active][ready]</span><br /><br />Path Group 2:</span><br />
<span style="background-color: white;"><br /> <span style="color: blue;">\_ 2:0:2:0 sdct 70:16 [active][ready]<br /> \_ 1:0:1:0 sdq 65:0 [active][ready]</span></span><br />
<br />
<b><span style="font-size: large;">4. Meaning various parameters are discussed below : </span></b><br />
<br />
<span style="background-color: white; color: blue;">a. polling_interval : </span>Specifies the interval between two path checks in seconds. <br />
<span style="background-color: white; color: blue;">b. udev_dir :</span> The directory where udev device nodes are created. The default value is /dev. <br />
<span style="background-color: white;"><span style="color: blue;">c. multipath_dir :</span> </span>/var/lib/multipath/bindings, The directory where the dynamic shared objects are stored. <br />
<span style="background-color: white;"><span style="color: blue;">d. path_selector :</span> </span>Specifies the default algorithm to use in determining what path to use for the next I/O operation.<br />
<br />
<b>Possible values include:</b><br />
<br />
<span style="background-color: #ffe599;"> round-robin 0:</span> Loop through every path in the path group, sending the same amount of I/O to each.<br />
<span style="background-color: #ffe599;"> queue-length 0: </span>Send the next bunch of I/O down the path with the least number of outstanding I/O requests.<br />
<span style="background-color: #ffe599;"> service-time 0:</span> Send the next bunch of I/O down the path with the shortest estimated service time, which is determined <br />
<br />
by dividing the total size of the outstanding I/O to each path by its relative throughput.<br />
The default value is round-robin 0.<br />
<br />
<span style="background-color: white; color: blue;">e. path_grouping_policy : </span>Specifies the default path grouping policy to apply to unspecified multi. paths. Possible values include:<br />
<br />
<span style="background-color: #ffe599;"> failover:</span> 1 path per priority group.<br />
<span style="background-color: #ffe599;"> multibus:</span> all valid paths in 1 priority group.<br />
<span style="background-color: #ffe599;">group_by_serial: </span>1 priority group per detected serial number.<br />
<span style="background-color: #ffe599;">group_by_prio: </span>1 priority group per path priority value. Priorities are determined by callout programs specified as global, per-controller, or per-multipath options.<br />
<span style="background-color: #ffe599;">group_by_node_name:</span> 1 priority group per target node name. Target node names are fetched in/sys/class/fc_transport/target*/node_name.<br />
<br />
The default value is failover. <br />
<br />
<span style="background-color: white; color: blue;">f. getuid_callout :</span><br />
<br />
Specifies the default program and arguments to call out to obtain a unique path identifier. An absolute path is required.<br />
The default value is /lib/udev/scsi_id --whitelisted --device=/dev/%n.<br />
<br />
<span style="background-color: white; color: blue;">g. prio : </span>Specifies the default function to call to obtain a path priority value. For example, the ALUA bits in SPC-3 <br />
<br />
provide an exploitable prio value. Possible values include:<br />
<span style="background-color: #ffe599;">const:</span> Set a priority of 1 to all paths.<br />
<span style="background-color: #ffe599;"> emc:</span> Generate the path priority for EMC arrays.<br />
alua: Generate the path priority based on the SCSI-3 ALUA settings.<br />
tpg_pref: Generate the path priority based on the SCSI-3 ALUA settings, using the preferred port bit.<br />
<span style="background-color: #ffe599;"> ontap:</span> Generate the path priority for NetApp arrays.<br />
rdac: Generate the path priority for LSI/Engenio RDAC controller.<br />
hp_sw: Generate the path priority for Compaq/HP controller in active/standby mode.<br />
<span style="background-color: #ffe599;"> hds:</span> Generate the path priority for Hitachi HDS Modular storage arrays.<br />
The default value is const. <br />
<br />
<span style="background-color: white; color: blue;">h, path_checker :</span><br />
<br />
Specifies the default method used to determine the state of the paths. Possible values include:<br />
<span style="background-color: #ffe599;">readsector0</span>: Read the first sector of the device.<br />
t<span style="background-color: #ffe599;">ur:</span> Issue a TEST UNIT READY to the device.<br />
e<span style="background-color: #ffe599;">mc_clariion:</span> Query the EMC Clariion specific EVPD page 0xC0 to determine the path.<br />
<span style="background-color: #ffe599;">hp_sw</span>: Check the path state for HP storage arrays with Active/Standby firmware.<br />
<span style="background-color: #ffe599;"> rdac: </span>Check the path stat for LSI/Engenio RDAC storage controller.<br />
<span style="background-color: #ffe599;"> directio:</span> Read the first sector with direct I/O.<br />
The default value is directio. <br />
<br />
<span style="background-color: white; color: blue;">i. failback : </span><br />
Manages path group failback.<br />
<span style="background-color: #ffe599;"> immediate :</span> A value of immediate specifies immediate failback to the highest priority path group that contains <br />
<br />
active paths.<br />
<span style="background-color: #ffe599;">manual : </span>A value of manual specifies that there should not be immediate failback but that failback can happen only <br />
<br />
with operator intervention.<br />
<span style="background-color: #ffe599;"> followover :</span> A value of followover specifies that automatic failback should be performed when the first path of a <br />
<br />
path group becomes active. This keeps a node from automatically failing back when another node requested the failover.<br />
A numeric value greater than zero specifies deferred failback, expressed in seconds.<br />
The default value is manual. <br />
<br />
<br />
<span style="background-color: white; color: blue;">j. rr_min_io : </span> Specifies the number of I/O requests to route to a path before switching to the next path in the current <br />
<br />
path group. This setting is only for systems running kernels older than 2.6.31. Newer systems should use rr_min_io_rq. The <br />
<br />
default value is 1000.<br />
<br />
<span style="background-color: white; color: blue;">k. rr_min_io_rq : </span> Specifies the number of I/O requests to route to a path before switching to the next path in the current <br />
<br />
path group, using request-based device-mapper-multipath. This setting should be used on systems running current kernels. On <br />
<br />
systems running kernels older than 2.6.31, use rr_min_io. The default value is 1.<br />
<br />
<br />
<span style="background-color: white; color: blue;">l. rr_weight : </span> If set to priorities, then instead of sending rr_min_io requests to a path before calling path_selector to <br />
<br />
choose the next path, the number of requests to send is determined by rr_min_io times the path's priority, as determined by <br />
<br />
the prio function. If set to uniform, all path weights are equal. The default value is uniform.<br />
<br />
<span style="background-color: white; color: blue;">m. no_path_retry : </span>A numeric value for this attribute specifies the number of times the system should attempt to use a <br />
<br />
failed path before disabling queueing.<br />
<br />
<span style="background-color: #ffe599;"> fail :</span> A value of fail indicates immediate failure, without queueing.<br />
<span style="background-color: #ffe599;">queue:</span> A value of queue indicates that queueing should not stop until the path is fixed.<br />
The default value is 0.<br />
<br />
<span style="background-color: white;"><span style="color: blue;">n. user_friendly_names :</span></span>If set to yes, specifies that the system should use the /etc/multipath/bindings file to assign a <br />
<br />
persistent and unique alias to the multipath, in the form of mpathn. If set to no, specifies that the system should use the <br />
<br />
WWID as the alias for the multipath. In either case, what is specified here will be overridden by any device-specific <br />
<br />
aliases you specify in the multipaths section of the configuration file. The default value is no.<br />
<br />
<span style="background-color: white; color: blue;">o. queue_without_daemon : </span>If set to no, the multipathd daemon will disable queueing for all devices when it is shut down. The <br />
<br />
default value is no.<br />
<br />
<span style="background-color: white; color: blue;">p. flush_on_last_del : </span>If set to yes, the multipathd daemon will disable queueing when the last path to a device has <br />
<br />
been deleted. The default value is no.<br />
<br />
<span style="background-color: white; color: blue;">q. max_fds : </span> Sets the maximum number of open file descriptors that can be opened by multipath and the multipathd daemon. <br />
<br />
This is equivalent to the ulimit -n command. As of the Red Hat Enterprise Linux 6.3 release, the default value is max, <br />
<br />
which sets this to the system limit from /proc/sys/fs/nr_open. For earlier releases, if this is not set the maximum number <br />
<br />
of open file descriptors is taken from the calling process; it is usually 1024. To be safe, this should be set to the <br />
<br />
maximum number of paths plus 32, if that number is greater than 1024.<br />
<br />
<span style="background-color: white; color: blue;">r. checker_timeout :</span> The timeout to use for path checkers that issue SCSI commands with an explicit timeout, in <br />
<br />
seconds. The default value is taken from sys/block/sdx/device/timeout.<br />
<br />
<span style="background-color: white; color: blue;">s. fast_io_fail_tmo : </span> The number of seconds the SCSI layer will wait after a problem has been detected on an FC remote <br />
<br />
port before failing I/O to devices on that remote port. This value should be smaller than the value of dev_loss_tmo. <br />
<br />
Setting this to off will disable the timeout. The default value is determined by the OS.<br />
<br />
<span style="background-color: white; color: blue;">t. dev_loss_tmo : </span> The number of seconds the SCSI layer will wait after a problem has been detected on an FC remote port <br />
<br />
before removing it from the system. Setting this to infinity will set this to 2147483647 seconds, or 68 years. The default <br />
<br />
value is determined by the OS.</div>
Playing with Linux.http://www.blogger.com/profile/01147934604287011206noreply@blogger.com0tag:blogger.com,1999:blog-4333613427966952010.post-5801799296514164292013-04-10T12:13:00.001-07:002015-01-23T12:37:04.630-08:00How to verify UDP packet communication between two linux system?<div dir="ltr" style="text-align: left;" trbidi="on">
■ <b>Purpose: </b>verify UDP packet communication between two linux system<br />
■ <b>OS Environment :</b> Linux<br />
■ <b>Required Application :</b> nc<br />
■ <b>Real Example :</b><br />
<br />
We had to check UDP packet communication between linux and a windows system. Main purpose was to make all linux clients send/forward udp packets to central windows server which has arcsight. remote log forwarding was enabled on client.<br />
<br />
■ <b>Troubleshooting Steps :</b><br />
<br />
<b>1. send UDP packets from client :</b><br />
<br />
<span style="background-color: white;"><span style="color: blue;">$nc -uv IP_of_system_where_UDP_is_sending port_number_to_which_UDP_is_receiving</span></span><br />
<span style="background-color: white; color: blue;">Hello</span><br />
<span style="background-color: white; color: blue;">This is test UDP packet</span><br />
<span style="background-color: white; color: blue;">Are you capturing it</span><br />
<span style="background-color: white; color: blue;">Please let me know</span><br />
<br />
Example :<br />
<br />
<span style="background-color: white; color: blue;">$ nc 192.1.2.10 514</span><br />
<span style="background-color: white; color: blue;">Hello</span><br />
<span style="background-color: white; color: blue;">This is test UDP packet</span><br />
<span style="background-color: white; color: blue;">Are you capturing it</span><br />
<span style="background-color: white; color: blue;">Please let me know</span><br />
<br />
If you capture the packets using wireshark / tshark or tcpdump, you'll see above packets on windows system/linux system. If your system wants to listen or receive UDP packets on<span style="background-color: #ffd966;"></span> any linux box, you can execute this :<br />
<br />
Open UDP port on system A :<br />
<br />
<span style="background-color: white; color: blue;">SystemA$ nc -luv port</span><br />
<span style="background-color: #6aa84f;"><br /></span>
<span style="background-color: white;">Send UDP packaets from system B :</span><br />
<span style="background-color: white;"><br /><span style="color: blue;"><b>SystemB</b>$ nc -uv <ipofsystema> 514</ipofsystema></span></span><br />
<span style="background-color: white; color: blue;">>>type anyting, you'll see it on SystemA </span><br />
<span style="background-color: #6aa84f;"><br /></span>
<br />
<br />
<br /></div>
Playing with Linux.http://www.blogger.com/profile/01147934604287011206noreply@blogger.com0