Registration is now open for ApacheCon US 2008. There will be an Early Bird discount, so register early and Save! The schedule is up and I’m very happy to see the return of the schedule grid that shows the entire conference on one page. J. Aaron has done a great job on the site.
I will be presenting one session at the conference: Hardening Enterprise Apache Installations Against Attacks will discuss security issues with the Apache Web Server and how the developer team reacts to issues as they are found. We’ll also talk about protecting applications that are served by the Apache server and may be the target of attacks that do not subvert the web server itself, but the code behind it.
The first time I did this talk, at AacheCon EU 2008, I ran out of time. There’s so much to talk about! The feedback forms submitted by the attendees did, however, identify some spots I can tighten up, so I’m looking forward to present a new, updated version of the talk this fall.
Hope to see you, first week of November, in New Orleans!
This has already been Slashdotted, but it’s so cool I just had to mention it. Michael Ogawa at UC Davis has graphed the development activity of a number of open source projects, including the Apache HTTP Server. If you run the video, you’ll see developer’s names float in and out as their activity level rises and subsides:
If you’re seeing this in syndication, the embedded video may not come across so do click on the post title, or go directly to the movie site. There is a sound control in the movie pane to turn down the perky electrobeat, although it fits the datastream perfectly.
ApacheCon EU 2008 has come to an end, and I think it’s been a really good show. Exactly what defines a conference as “good” is sometimes hard to capture: I think a lot is in the intangibles like who was there, what was done and what was discussed over the tables in the common room.
There were good talks too: I think dividing the conference up in tracks worked really well. It made it easy to find related sessions and follow them. A logical next step might be to work with the presenters to make sure that the contents of a particular track are cohesive, with minimal overlap. The planners may want to assign each track a volunteer “editor” from that particular field who gets to work with the individual presenters.
I spent a significant amount of time in the Systems Administration and Security tracks because that’s where I was presenting. On Friday, my talks done, I sat in on some sessions about projects and technologies with which I have never worked, and found them really inspiring. During Ate’s talk on enterprise portals I downloaded the installer of Jetspeed 2, and found myself with a fully functional portal on my laptop, ready for corporate branding and custom portlets. That’s cool stuff.
I guess that I’ll also be looking at interesting stuff like Serf and the Waka as they develop and (hopefully) are documented. Will I play a role in the development of Apache 3.0? Ma-a-a-a-a-a-y-be… in my copious spare time. Will there be an Apache 3.0? Only time and dev@httpd.apache.org will tell.
I feel tired, yet refreshed, with lots of interesting things to check out. And that is a good feeling to have after a conference.
We’re doing a PGP Keysigning again at ApacheCon Europe 2008, but the Apache Wiki is down today so I can’t update its PGPKeySigning page. Several folks have already sent me their key: thank you very much!
The Keysigning session will happen at the tail end of the Welcome Reception, and hopefully be done before the BOFs start. If you would like to participate, send your public key to sctemme at apache dot org before 3PM on Wednesday. I will compile the key list after I’m done with my talks.
It’s fully ensconced in my muscle memory: tar -xzf somepackage.tar.gz to untar a gzip-compressed tarball. But what if the tarball is compressed with Bzip2? You have to use tar -xjf or tar won’t understand the compression format.
Or, on MacOSX, just use tar -xf on either compression format and tar figures it out for itself. And that’s the way it should be: smart software that can find out on what file format it operates.
La’s photos from ApacheCon in Atlanta are now online. There are some shots from the Keysigning, and some of Santiago singing Karaoke at the 24h Metro Bar & Diner.
I gave my Apache Performance Tuning talks last week at ApacheCon in Atlanta. The first one (Scaling Up) has a reading list at the end of the slide deck, but I noticed as I was presenting that the second slide deck (Scaling Out) does not. Two good books on web site scalability are Theo Schlossnagle’s Scalable Internet Architectures and Cal Henderson’s Building Scalable Websites.
I have just put up the Key Ring with all the keys that were on the list for the PGP Keysigning last night. These are all the people who signed up: not all of them actually made it to the Keysigning.
We did have a lot of people who had never attended a Keysigning before: welcome to the web of trust folks!
Windows is not usually the operating system that comes to mind when deploying Apache. Not all of us, however, have a choice in what operating system we run, and the fact that open source software like Apache and PHP run on the Windows platform allows more people to get acquainted with the open source stack without having to switch operating systems.
Still, sites that run Apache in Production on Windows are few and far between. This means that Windows-specific parts of the Apache code receive far less attention and use than code specific to platforms like Linux, which leads to interesting and hard to analyze bugs. For instance, there’s this Windows box running Apache, MySQL and PHP to support a couple of popular PHP applications like Joomla and Gallery. No, I won’t tell you where it is, even though it’s all better now. It was crashing on a regular basis, say every half hour or so. Normally, it’s not so bad when an Apache child process crashes: especially with the Prefork MPM (still the one recommended by the PHP folks), only one client connection will die and there are plenty of other children available so there is no interruption in service. The Windows MPM only has one child process, so when that crashes the server is offline for a couple of seconds while the parent spins up a new child. This is very frustrating and clearly not acceptable in a production situation. But how to debug?
The account that runs the Apache service (perhaps I should put up a deployment best practices guide at some point) was not allowed to write crash dumps. Even though I enlisted the help of a very experienced Windows programmer, we were not able to make Apache dump core. I did observe though that the child process never seemed to grow beyond 256Mb before it wrote a whole bunch of out-of-memory errors to the PHP error log and then crashed with either an access violation or a terse message about how the zend_mm_heap was corrupted. This led me to a crazed and unsuccessful Google search for process memory limits on Windows 2003, and equally unsuccessful attempts to recycle the process before it crashed by setting MaxRequestsPerChild.
So how did I solve the problem? I didn’t solve it, but made it go away (which is something different although the immediate result is the same) by lowering the ThreadsPerChild value from the configuration file default of 256 to a more conservative 100. This lower number must have prevented PHP memory management from stepping on its own toes, and the result was that the server stayed up for 19 days straight before it was manually restarted. Better? You bet! The only, slightly worrisome thing: the child process ballooned to a working set of 800Mb of RAM, and has even been up to 1.2Gb before settling down. Good thing the server has 2Gb installed. Since the server ran for 19 days, I am convinced that the situation was stable, and even if there were a slow leak I could always put MaxRequestsPerChild back in. It just goes to show that PHP applications like Joomla are very large, and cause Apache/PHP to allocate an enormous amount of thread-local storage.
I would still like to know what caused the crashes, but making them go away is almost just as good as actually solving the problem.
Have you, as I was just now, been repeatedly confounded by PHP flat-out refusing to load its MySQL module on Windows? I have just finished banging my head against this particular wall for a day or so and would like to share the Apache Way to solving the issue.
The problem is, as is so frequently the case, DLL Hell. The php_mysql.dll module loads libmysql.dll, but apparently it has to be the version against which it was compiled. The required DLL is bundled with PHP, but what if an older, incompatible version of that DLL is found earlier in the search path? You’re hosed, that’s what if, and the module load fails with a message to the log file like so:
PHP Warning: PHP Startup: Unable to load dynamic library 'D:\\php-5.2.3\\ext\\php_mysql.dll' - The specified procedure could not be found.\r\n in Unknown line 0
I suppose that would be a missing symbol in the underlying library on other platforms. A quick Google put me on the right track and sure enough the indispensable Process Explorer utility tells me that httpd.exe has C:\WINDOWS\system32\libmysql.dll loaded. Yup, that’s our problem. No, I didn’t build that box. That box must have been built by someone who thought copying crap into the Windows directory is ever a good idea.
How do we solve this the Apache Way? Not by copying DLLs around, that’s for sure. The Apache configuration language has the LoadFile directive for this particular purpose. Loading the correct DLL right before the PHP module:
