mod_php Apache Memory Usage

For me, one of the most rewarding things in systems architecture is taking something that is slow and laggy and making it way faster. Seeing an HTML call drop from 800ms to 200ms is probably a lot more exciting than really it should be. But there’s real danger in just “optimizing everything” without real knowledge or regard to how those optimized things work together.

There’s reasons why defaults are made the defaults, and one of the primary reasons is that the defaults are most likely to play nice with ohter pieces of your architecture. I’ve seen systems that are much slower *after* optimization because of conflicting optimizations, and sometimes people don’t even realize that they are slower since the didn’t get a good first initial benchmark. They think they’ve “tried everything”, and maybe they just end up just throwing more hardware at the problem.

I recently inflicted this kind of bad optimization on myself on an EC2 AWS Linux server with a pretty standard LAMP stack with one bad default setting. My server was running out of memory and the kernel OOM killer was on the loose terrorizing the countryside.

After some digging I was able to fix the problem, but along the way I made some (to me at least) interesting realizations.

Realization #1 – EC2 servers usually don’t (and probably shouldn’t) have swap.
Disk swap can be a nice safety blanket sometimes. If you’re swapping a lot your system might be terribly slow, but it might prevent a crash. All of a sudden you might be in the red on all your system speed monitors, but better red than dead, right?
Every EC2 instance beyond micro has available instance ephemeral storage that you could borrow for swap if you’ve provisioned them (which guess what, you probably haven’t since it’s not a part of most AMIs). But before you go re-provision your instance or even worse using EBS for swap, consider that Amazon will charge you for all that IO usage. If your system is doing serious swapping you can really increase your instance spend, maybe even to the point it would be cheaper to just upgrade to more RAM.

Realization #2 – Apache httpd pre-fork MPM can have a huge memory footprint.
The apache pre-fork model with mod_php is a very standard way to use PHP. Maybe not the best way in many cases these days, but that’s how my particular AWS AMI came out of the box, and it generally works pretty well. But this was the default:
ServerLimit 256
MaxClients 256

So apache can run 256 child processes. Say your server has 4GB available to be used by apache after the kernel, database, user stuff, etc., etc. That’s only 16M available per child if your system is under heavy web request load and needs all its children.

That might be ok with a bare-bones httpd setup, but we’re talking PHP here. My child processes were up to 150M each under load, so at that rate as soon as apache was trying to do more than about 27 children it was running out of system memory.

Realization #3 – You can pretty easily cut down Apache memory usage in a systematic way.
150M is pretty big for an httpd child… those were some chunky kids, no doubt, but what are they getting so fat on?

# ps ax -o rss,command,pid | grep httpd | sort -rn | head -1
157092 /usr/sbin/httpd            16034
(that’s my biggest httpd process, currently resident at 150+ with PID# 16034)

# pmap -qx 16034 | awk ‘{print $3 ” ” $6 $7 $8}’ | sort -rn | head
65508 apc.5wroru(deleted)
65120 [anon]
11296 [anon]
6108 [anon]
1684 libphp5.so
660 libc-2.17.so
620 libnss3.so
448 libsqlite3.so.0.8.6
340 libphp5.so
248 libcurl.so.4.3.0
(that’s the memory map of that process, using awk to show the relevant rows of the extended output.)

[anon] in this case is just memory that the child has (m)allocated, not part of any named library.. so nothing we can do about that without some hard-core debugging, but safe to assume it’s probably our general site usage and some greedy PHP scripts.

Or visually, this:

Apache Memory Map Usage

Apache Memory Map Usage

The first advice you might read about apache memory optimization is to shave the modules you aren’t using. Reasonable advice, but for me that would mean going after little slices of that little cyan “other” slice in the pie chart. If I was trying to squeeze every last K out of a micro instance then I might, but I’m more concerned about the APC usage (and the [anon], but again not much I can do about that easily).

I could have pared down the APC shared memory size (www.php.net/manual/en/apc.configuration.php), but if I did that I’d lose a lot of the meager hit rate I was achieving anyways.

Looking more at APC made me realize that while it was occasionally getting me a decent hit rate, it was still at 64M much too small to prevent it from fragmenting and spending a lot of effort ejecting items from the cache. One of those times that if I had used a tool like ApacheBench (ab) and hit the same URL a bunch of times I would have thought was great, but under real world use with a bunch of different sites and URLs just didn’t work.

In turning off APC I was able to up my maximum apache servers back to something I wasn’t afraid of hitting, but was still within the range that even at full usage I didn’t have to worry about running out of RAM.

So good luck tuning your severs, but don’t get too aggressive dropping in every bit of optimization advice you see, and make sure to always check against a real-world benchmark after you make the change.

No comments yet.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.