How to use strace command in linux?

■ Requirement : strace usage
■ OS Environment : Linux[RHEL, Centos]
■ Application: strace 
■ Implementation Steps :  

             strace is a useful diagnostic, instructional, and debugging tool. System administrators, diagnosticians and trouble-shooters will find it invaluable for solving problems with programs for which the source is not readily available since they do not need to be recompiled in order to trace them.

Arguments and returned value :

Each line in the trace contains the system call name, followed by its arguments in parentheses and its return value. An example from stracing the command ''cat /dev/null'' is:

open("/dev/null", O_RDONLY) = 3

Errors (typically a return value of -1) have the errno symbol and error string appended.

open("/foo/bar", O_RDONLY) = -1 ENOENT (No such file or directory)


Examples :

strace -s ls
strace -o output_file ls
strace -o outputfile -d ls
strace -e expr

like
strace -e trace=set
strace -e trace=open
strace -e trace=read
strace -e trace=file
strace -e trace=process
strace -e trace=network
strace -e trace=signal
strace -e trace=ipc
strace -e trace=desc //descriptors
strace -e read=set

For example, to see all input activity on file descriptors 3 and 5 use
-e read=3,5
For example, to see all output activity on file descriptors 3 and 5 use
-e write=3,5

-p pid //Attach to the process with the process ID pid and begin tracing

example : strace -e trace=network -p 4009

-u username //Run command with the user ID , group ID , and supplementary groups of username
===============

Special Use :

==================
1. Find out which config files a program reads on startup :-

strace php 2>&1 | grep php.ini
strace -e open php 2>&1 | grep php.ini

2. Why does this program not open my file?

$ strace -e open,access 2>&1 | grep your-filename

Look for an open() or access() syscall that fails

3. What is that process doing RIGHT NOW?

Ever had a process suddenly hog lots of CPU? Or had a process seem to be hanging?

strace -p 15427

4. What is taking time?

strace -c -p 11084

5. Can't I connect to that server?

strace -e poll,select,connect,recvfrom,sendto nc www.yahoo.com 80
================

Note :

-------------------------
ps -e -o pcpu,cpu,nice,state,cputime,args --sort pcpu | sed '/^ 0.0 /d' //
List processes by % cpu usage

ps -C firefox-bin -L -o pid,tid,pcpu,state //
List all threads for a particular process

watch -n.1 'cat /proc/interrupts' //
Watch changeable data continuously
--------------------------