- Put a squid proxy in front of your mod_perl / Tomkat servers if possible.
- Consider using a CDN (akamai) if possible.
- Use persistent technologies such as mod_perl, Tomkat, share memory, opcode caching / optimizer / analyzer.
- Reduce load on backend database by using memcache.
- Optimize SQL queries and database machines.
- Use indexes
- Faster SAN (Hitachi)
- Reduce system calls.
- Consider using threads or event driven programming.
- Consider using share memory so that mod_perl/apache does not have to handle sending data.
- Consider using 64bit operating system, add more memory, and using hardware RAID
- Reduce the number of data transfer on network interface (compress it on the fly, or store it on the file system already compressed)
- Reduce the amount of data written to disk if we can.
- Traditional 3-tier architecture with hardware load balancer in front of databases
- Clusters based on types / features: ad, app, photo, monitoring, gallery, profile, userinfo, IM status cache, message, testinomial, friends, graph servers, gallery search, object cache
- No persistent database connection
- Don't go after the biggest problem first. Tackle the low hanging fruit first. Or tackle both in parallel if your team have sufficient man power.
- Optimize without downtime
- Move sorting query types into the application and limit the amount of data returned to the browser.
- Reduce range
- Range on primary keys.
- Benchmark, make change, benchmark, make change (cycle of improvement)
- Always have a plan to rollback.
- Incremental rollout (rollout to a small set of users before rolling it out to all users).
- Horizontal scalability. Rather than buying big, centralized servers, buy a lot of thin, cheap servers. If one fails, rollover to another servers. Spawn up a new server and join the cluster. Use the AWS auto-scale approach.
- Work with a team. Access the current problems. Define the issues. Determine available hardware.
- Have a road map.
The database server should never be on the same box as the web server, even when the box does not have a lot of hits. This makes it easy to isolate resource contention problem, and do performance tuning.
Things to keep in mind
Operating system impose certain limits on your program.
- On Linux, you can only have approximately 32,000 files inside a directory. That is a lot of files, but in reality, if you have more than 1000 files per directory, you should see performance degraded. If you have a lot of files to be stored on the same filesystem, split them up, and store them in nested directories where the names of the directories are 2 or 3 letters long.
- The same limitation mentioned above also dictate how many tables your mysql database can have. For each table, mysql use 3 files. So at the max, you can only have 32000 / 3 tables.
pmap - reports memory map for a process. Check out this blog for some explaination.
How to cluster mail servers?
Solaris 2.x - Tuning your TCP/IP Stack and More
TCP-Friendly Unicast Rate-Based Flow Control
A HOWTO on Optimizing PHP
Linux Virtual Server
Friendster Architecture 2005
MySpace Architecture 2009
Boost application performance using asynchronous I/O
The Linux Page Cache and pdflush:Theory of Operation and Tuning for Write-Heavy Loads
Linux Disk Performance/File IO per process
Disk I/O Analysis - Determining I/O Bottlenecks
Troubleshooting Memory Usage
I/O statistics per process
How to Check Memory Usage in Linux based Server
Is there a way to find out what application using most of bandwidth in Linux?
Linux find the memory used by a program / process using pmap command
Choosing the right Linux File System Layout using a Top-Bottom Process
Introduction to nginx.conf scripting
MessagePack: A cross-language efficient binary-based serialization library—“It’s like JSON, but very fast and small”. Claims to outperform protocol buffers for at least some benchmarks.
ZeroMQ: Modern & Fast Networking Stack
Friendster: 1 billion queries per day.
MySpace - 2009: Windows ASP.NET, IIS, SQL Server. 300 millions users. 100 gigabitss/second out to the Internet. 10GB/sec is HTML content. 4500+ web servers, 1200+ cache servers running 64-bit Windows 2003 with 16GB of object cache in RAM. 500+ database servers running 64-bit Windows and SQL Server 2005.