Memory Profiler:
Purify
The Third Degree
Valgrind
fuser
sar
ps
top
netstat
hwconfig
lsdev
lspci
pnpdump
dmesg
sum
lsof
stat
closefd
cfengine
GAP
stow
netstat
tcpstat
ipcs
cpan2rpm
slurp //watch any program install
setup.sh
CPANPLUS
savecore
sys
ngrep
netcat
ntop
ptrace
proc(4)
time
trace
truss
strace
file(1)
od(1)
hexdump(1)
strings(1)
sudo strings /var/log/wtmp | grep -V tty | sort | uniq
/etc/syslogd.conf
tcpdump
tcpflow
ethereal
ping
traceroute
nmap
sudo tcpdump -i eth0 -l -n port 53 | tee dns_log
1. Reproducibility. First, find a way to reliably reproduce the error, which in itself, often points you straight at the problem through deductive reasoning. The bug may happen in precisely one circumstance, which can only happen in one place in the code. A bug that appears randomly is essentially unsolvable unless you have a leap of insight. You need a guarantee of cause and effect to make inferences about changes you introduce. A change in the code the fixes the problem may or may not really fix it, as the problem randomly appears and disappears.
2. Reduction. Reduce the problem to its essence. Determine the smallest input that will cause the bug. The simpler the data or path to the bug, the more easily you can deduce or track down the problem. A large data set introduce a great deal of noise that camouflages the essential cause of the trouble. If you have a large data input file that causes the problem, do a binary search type reduction. Cut the file in half, throw out the last half. If you still have problem, then you can ignore the final half. If the problem goes away, then start whittling down the final half, as it must contain the input that causes the problem.
3. Deduction. This is your primary weapon once you have a small input that reliably cause a problem. What is the general path through the program used by the input? What components might be the problem or mangle the data so that a future component fails? What is the difference between the input that doesn't work and some other input that does? Try to reduce the scope of possibilities by forming and eliminating hypotheses.
4. Experimentation. You must change the conditions of the test to see if your bug disappears. If you correct the bug with a change, that change might tell you what the problem is, or at least give you a big hint about what is going on. You form hypotheses with your logic and deductive reasoning skills and then filter them by experimentation and observation.
5. Experience. Experience has no substitute. To become a good programmer, apprentice yourself to a good programmer or fumble your way through it yourself for a few years (making lots of mistake either way). Experience helps in the debugging process in two ways: first, you hone your ability to execute the previous four elements; and second, you might have seen a similar bug or just plain know more about a particular problem. Borrowing the experience of other developers is also important. Searching the web and talking to other developers can save you a huge amount of effort by leveraging other people's experience. Interesting, just explaining the problem to another developer can line things up properly in your head so that simple deduction tells you where the problem lies.
6. Tenacity. All bugs are caused by computers doing exactly what they are told; absolutely no mystery in that, and hence all problems are solvable. You must begin with the attitude that you will never give up no matter how long it takes to find the problem and correct it. You will become more and more confident each time you solve a problem. You must be tenacious. In the process of solving insidious bugs, you will become a much better programmer as you start to anticipate errors and to code in a manner less likely to produce mysterious bugs.
http://www-128.ibm.com/developerworks/web/library/wa-debug.html
Race conditions
Race conditions are bugs that only occur under certain circumstances. These circumstances usually involve the time at which certain events correlate with other events. Using the debugger to debug these situations is not always possible, because the act of using the debugger may change the timing of the events in the program. This can cause a symptom to occur without the debugger, but while using the debugger, the symptom may disappear. The bug isn't gone, it just isn't being 'tickled'. There really isn't any stock method to get rid of race conditions. Usually, an intense analysis of the algorithms is necessary. Finite-state diagrams may also be useful if you have the patience for it.
Timestamps on file
What is the difference between modification time and inode change time? When a file is modified, both the modification time and inode change time is updated. The inode change time reflects the time that permission or ownership was changed. If we use chmod or chown on a file, the inode change time would be different from the modification time.
To see the various timestamp on a file: stat filename
