Category Archives: Apache

ApacheCon EU 2008 is Over

Posted on by

ApacheCon EU 2008 has come to an end, and I think it’s been a really good show. Exactly what defines a conference as “good” is sometimes hard to capture: I think a lot is in the intangibles like who was there, what was done and what was discussed over the tables in the common room.

There were good talks too: I think dividing the conference up in tracks worked really well. It made it easy to find related sessions and follow them. A logical next step might be to work with the presenters to make sure that the contents of a particular track are cohesive, with minimal overlap. The planners may want to assign each track a volunteer “editor” from that particular field who gets to work with the individual presenters.

I spent a significant amount of time in the Systems Administration and Security tracks because that’s where I was presenting. On Friday, my talks done, I sat in on some sessions about projects and technologies with which I have never worked, and found them really inspiring. During Ate’s talk on enterprise portals I downloaded the installer of Jetspeed 2, and found myself with a fully functional portal on my laptop, ready for corporate branding and custom portlets. That’s cool stuff.

I guess that I’ll also be looking at interesting stuff like Serf and the Waka as they develop and (hopefully) are documented. Will I play a role in the development of Apache 3.0? Ma-a-a-a-a-a-y-be… in my copious spare time. Will there be an Apache 3.0? Only time and dev@httpd.apache.org will tell.

I feel tired, yet refreshed, with lots of interesting things to check out. And that is a good feeling to have after a conference.

PGP Keysigning at ApacheCon EU 2008

Posted on by

AC EU 2008 Speaker buttonWe’re doing a PGP Keysigning again at ApacheCon Europe 2008, but the Apache Wiki is down today so I can’t update its PGPKeySigning page. Several folks have already sent me their key: thank you very much!

The Keysigning session will happen at the tail end of the Welcome Reception, and hopefully be done before the BOFs start. If you would like to participate, send your public key to sctemme at apache dot org before 3PM on Wednesday. I will compile the key list after I’m done with my talks.

Smart Tar

Posted on by

It’s fully ensconced in my muscle memory: tar -xzf somepackage.tar.gz to untar a gzip-compressed tarball. But what if the tarball is compressed with Bzip2? You have to use tar -xjf or tar won’t understand the compression format.

Or, on MacOSX, just use tar -xf on either compression format and tar figures it out for itself. And that’s the way it should be: smart software that can find out on what file format it operates.

But my fingers still want to tar -xzf…

Reading List for ApacheCon Scaling Out Presentation

Posted on by

I gave my Apache Performance Tuning talks last week at ApacheCon in Atlanta. The first one (Scaling Up) has a reading list at the end of the slide deck, but I noticed as I was presenting that the second slide deck (Scaling Out) does not. Two good books on web site scalability are Theo Schlossnagle’s Scalable Internet Architectures and Cal Henderson’s Building Scalable Websites.

Look at That 800 Pound Apache Hiding in the Corner

Posted on by

800 Pound ApacheWindows is not usually the operating system that comes to mind when deploying Apache. Not all of us, however, have a choice in what operating system we run, and the fact that open source software like Apache and PHP run on the Windows platform allows more people to get acquainted with the open source stack without having to switch operating systems.

Still, sites that run Apache in Production on Windows are few and far between. This means that Windows-specific parts of the Apache code receive far less attention and use than code specific to platforms like Linux, which leads to interesting and hard to analyze bugs. For instance, there’s this Windows box running Apache, MySQL and PHP to support a couple of popular PHP applications like Joomla and Gallery. No, I won’t tell you where it is, even though it’s all better now. It was crashing on a regular basis, say every half hour or so. Normally, it’s not so bad when an Apache child process crashes: especially with the Prefork MPM (still the one recommended by the PHP folks), only one client connection will die and there are plenty of other children available so there is no interruption in service. The Windows MPM only has one child process, so when that crashes the server is offline for a couple of seconds while the parent spins up a new child. This is very frustrating and clearly not acceptable in a production situation. But how to debug?

The account that runs the Apache service (perhaps I should put up a deployment best practices guide at some point) was not allowed to write crash dumps. Even though I enlisted the help of a very experienced Windows programmer, we were not able to make Apache dump core. I did observe though that the child process never seemed to grow beyond 256Mb before it wrote a whole bunch of out-of-memory errors to the PHP error log and then crashed with either an access violation or a terse message about how the zend_mm_heap was corrupted. This led me to a crazed and unsuccessful Google search for process memory limits on Windows 2003, and equally unsuccessful attempts to recycle the process before it crashed by setting MaxRequestsPerChild.

So how did I solve the problem? I didn’t solve it, but made it go away (which is something different although the immediate result is the same) by lowering the ThreadsPerChild value from the configuration file default of 256 to a more conservative 100. This lower number must have prevented PHP memory management from stepping on its own toes, and the result was that the server stayed up for 19 days straight before it was manually restarted. Better? You bet! The only, slightly worrisome thing: the child process ballooned to a working set of 800Mb of RAM, and has even been up to 1.2Gb before settling down. Good thing the server has 2Gb installed. Since the server ran for 19 days, I am convinced that the situation was stable, and even if there were a slow leak I could always put MaxRequestsPerChild back in. It just goes to show that PHP applications like Joomla are very large, and cause Apache/PHP to allocate an enormous amount of thread-local storage.

I would still like to know what caused the crashes, but making them go away is almost just as good as actually solving the problem.

Getting WAMP to Talk Amongst Themselves

Posted on by

Have you, as I was just now, been repeatedly confounded by PHP flat-out refusing to load its MySQL module on Windows? I have just finished banging my head against this particular wall for a day or so and would like to share the Apache Way to solving the issue.

The problem is, as is so frequently the case, DLL Hell. The php_mysql.dll module loads libmysql.dll, but apparently it has to be the version against which it was compiled. The required DLL is bundled with PHP, but what if an older, incompatible version of that DLL is found earlier in the search path? You’re hosed, that’s what if, and the module load fails with a message to the log file like so:

PHP Warning: PHP Startup: Unable to load dynamic library 'D:\\php-5.2.3\\ext\\php_mysql.dll' - The specified procedure could not be found.\r\n in Unknown line 0

I suppose that would be a missing symbol in the underlying library on other platforms. A quick Google put me on the right track and sure enough the indispensable Process Explorer utility tells me that httpd.exe has C:\WINDOWS\system32\libmysql.dll loaded. Yup, that’s our problem. No, I didn’t build that box. That box must have been built by someone who thought copying crap into the Windows directory is ever a good idea.

How do we solve this the Apache Way? Not by copying DLLs around, that’s for sure. The Apache configuration language has the LoadFile directive for this particular purpose. Loading the correct DLL right before the PHP module:

LoadFile "d:/php-5.2.3/libmysql.dll"
LoadModule php5_module "d:/php-5.2.3/php5apache2_2.dll"

makes PHP pick up the right symbols to run with MySQL.

How Henning Made Me Actually Do It

Posted on by

Like yachtsmen, programmers are lazy. Being a little bit of both, I guess that makes me doubly lazy, and a byproduct of this seems to be that I just don’t get around to signing the keys that emerge from the ApacheCon Keysigning sessions I organize.

Henning the Fussbal FanFor programmers, laziness manifests itself in the wish to write programs to perform repetitive tasks, not infrequently spending more time writing the program than it would have taken to just perform the task by hand. Of course the merit of writing such a program is that others can use it to perform the same repetitive task, without having to write the program first. A couple of days ago, Apache’s Henning Schmiedehausen posted PGPSigner, a utility that helps you sign all the keys on your list from the Keysigning session.

This is absolutely great, it just helped me do in five minutes what I had already put off for close to a month, and the keys from the last keysigning session are now signed, uploaded to the keyservers (pgpkeys.mit.edu and minsky.surfnet.nl) and mailed to their owners insofar their mail got through. I used the Signing Party Keyring that contains the keys of all the attendants.

Small patch to make the startup script suck in the jars in the lib directory:


Index: pgpsigner.sh
===================================================================
--- pgpsigner.sh (revision 1009)
+++ pgpsigner.sh (working copy)
@@ -31,4 +31,8 @@
exit 1
fi

-java -jar target/${APPNAME}-${APPVERSION}.jar "$@"
+for j in `ls lib/*.jar` ; do
+ CLASSPATH="$CLASSPATH:$j"
+done
+
+java -classpath $CLASSPATH -jar target/${APPNAME}-${APPVERSION}.jar "$@"

This makes it easier to run the program from the command line as opposed to from within Eclipse. (: Thank you Henning.

ApacheCon EU 2007 Keysigning

Posted on by

Yep, we’re doing it again. Wednesday night May 2 at ApacheCon Europe, we’ll be having a PGP KeySigning. All Apache committers and all conference attendees are invited to participate.

Why do we have a PGP Keysigning session at ApacheCon? At the Apache Software Foundation, we sign our releases with PGP. Every release archive is accompanied by a signature file (name ends in .asc) and a hash file (name ends in .md5) that you can use to verify the integrity of the release.

Continue reading →