Backend Performance Tuning

Gather statistics. Know where the problem is, before you can diagnose. Various tools such as nagios (network monitoring), top, ps, free, and strace give you relevant information on each machine. If both the web servers and the backend database server are busy, then you should look into optimizing the backend database (use MySQL query cache, use memcache, add indexes, add more memory, etc).

1. Put a squid proxy in front of your mod_perl / Tomkat servers if possible.
2. Consider using a CDN (akamai) if possible.
3. Use persistent technologies such as mod_perl, Tomkat, share memory, opcode caching / optimizer / analyzer.
4. Reduce load on backend database by using memcache.
5. Optimize SQL queries and database machines.
6. Reduce system calls.
7. Consider using threads or event driven programming.
8. Consider using share memory so that mod_perl/apache does not have to handle sending data.
9. Consider using 64bit operating system, add more memory, and using hardware RAID
10. Reduce the number of data transfer on network interface (compress it on the fly, or store it on the file system already compressed)
11. Reduce the amount of data written to disk if we can.

The database server should never be on the same box as the web server, even when the box does not have a lot of hits. This makes it easy to isolate resource contention problem, and do performance tuning.

tuning Apache
tuning PHP
tuning MySQL

Things to keep in mind

Operating system impose certain limits on your program.

1. On Linux, you can only have approximately 32,000 files inside a directory. That is a lot of files, but in reality, if you have more than 1000 files per directory, you should see performance degraded. If you have a lot of files to be stored on the same filesystem, split them up, and store them in nested directories where the names of the directories are 2 or 3 letters long.
2. The same limitation mentioned above also dictate how many tables your mysql database can have. For each table, mysql use 3 files. So at the max, you can only have 32000 / 3 tables.

pmap - reports memory map for a process. Check out this blog for some explaination.

Clustering

How to cluster mail servers?
http://virtualthreads.blogspot.com/2006/02/understanding-memory-usage-on-linux.html
http://virtualthreads.blogspot.com/2006/01/tuning-apache-part-1.html
Solaris 2.x - Tuning your TCP/IP Stack and More
http://people.redhat.com/alikins/system_tuning.html
TCP-Friendly Unicast Rate-Based Flow Control
A HOWTO on Optimizing PHP
Linux Virtual Server
Perlbal
HAProxy