Author Archives: Sander

Look at That 800 Pound Apache Hiding in the Corner

Posted on by

800 Pound ApacheWindows is not usually the operating system that comes to mind when deploying Apache. Not all of us, however, have a choice in what operating system we run, and the fact that open source software like Apache and PHP run on the Windows platform allows more people to get acquainted with the open source stack without having to switch operating systems.

Still, sites that run Apache in Production on Windows are few and far between. This means that Windows-specific parts of the Apache code receive far less attention and use than code specific to platforms like Linux, which leads to interesting and hard to analyze bugs. For instance, there’s this Windows box running Apache, MySQL and PHP to support a couple of popular PHP applications like Joomla and Gallery. No, I won’t tell you where it is, even though it’s all better now. It was crashing on a regular basis, say every half hour or so. Normally, it’s not so bad when an Apache child process crashes: especially with the Prefork MPM (still the one recommended by the PHP folks), only one client connection will die and there are plenty of other children available so there is no interruption in service. The Windows MPM only has one child process, so when that crashes the server is offline for a couple of seconds while the parent spins up a new child. This is very frustrating and clearly not acceptable in a production situation. But how to debug?

The account that runs the Apache service (perhaps I should put up a deployment best practices guide at some point) was not allowed to write crash dumps. Even though I enlisted the help of a very experienced Windows programmer, we were not able to make Apache dump core. I did observe though that the child process never seemed to grow beyond 256Mb before it wrote a whole bunch of out-of-memory errors to the PHP error log and then crashed with either an access violation or a terse message about how the zend_mm_heap was corrupted. This led me to a crazed and unsuccessful Google search for process memory limits on Windows 2003, and equally unsuccessful attempts to recycle the process before it crashed by setting MaxRequestsPerChild.

So how did I solve the problem? I didn’t solve it, but made it go away (which is something different although the immediate result is the same) by lowering the ThreadsPerChild value from the configuration file default of 256 to a more conservative 100. This lower number must have prevented PHP memory management from stepping on its own toes, and the result was that the server stayed up for 19 days straight before it was manually restarted. Better? You bet! The only, slightly worrisome thing: the child process ballooned to a working set of 800Mb of RAM, and has even been up to 1.2Gb before settling down. Good thing the server has 2Gb installed. Since the server ran for 19 days, I am convinced that the situation was stable, and even if there were a slow leak I could always put MaxRequestsPerChild back in. It just goes to show that PHP applications like Joomla are very large, and cause Apache/PHP to allocate an enormous amount of thread-local storage.

I would still like to know what caused the crashes, but making them go away is almost just as good as actually solving the problem.

E-mail from Paypal…

Posted on by

So this e-mail message from Paypal landed in my Hotmail. What makes this blog-worthy is that it actually, really came from Paypal. It was not a spam zombie attack, phishing scheme, virus or other type of malware, but a real, honest missive about their privacy policy. Go figure.

Front and Center at MIT

Posted on by

Today the research project of our friend Oliver is featured on the front page of the MIT website. He recently received his Ph.D. in Physics from MIT and there is an article on his research into twodimensional electron systems. Congratulations, Oliver.

Image sourced from MIT website.

The Great Scouring of Space Has Begun

Posted on by

“Giant Telescope Begins Scouring Space” writes the Associated Press in a story picked up unchanged by numerous news sources. My first thought: “Space will be so clean when it’s done.” Let’s go to the dictionary definition and have a look:

  1. a : to rub hard especially with a rough material for cleansing b : to remove by rubbing hard and washing — How does one rub a void?
  2. archaic : to clear (a region) of enemies or outlaws — SDI Redux!
  3. to clean by purging : PURGE
  4. to remove dirt and debris from (as a pipe or ditch) — Ooh! Ooh! We’re finally going to take care of space debris!
  5. to free from foreign matter or impurities by or as if by washing <scour wool> — Once we are rid of the space junk, there better be none of that left up there.
  6. to clear, dig, or remove by or as if by a powerful current of water

OK, so the other definition of scouring is “to go through or range over in or as if in a search”, which is clearly the one the author intended to use. But it’s still funny!

Now Rat’s Adorable

Posted on by

We saw Ratatouille the other day and had a great time. Wonderful to look at, great graphics and a good story. The humor is funny without being mean, and the rats are really well done. The way they move is just like you’d expect rats to scurry about. They are truly revolting, not by being gross, but by playing on our reflexive fear of rats in the kitchen. Go see it! Spoilers below…

Continue reading →

Getting WAMP to Talk Amongst Themselves

Posted on by

Have you, as I was just now, been repeatedly confounded by PHP flat-out refusing to load its MySQL module on Windows? I have just finished banging my head against this particular wall for a day or so and would like to share the Apache Way to solving the issue.

The problem is, as is so frequently the case, DLL Hell. The php_mysql.dll module loads libmysql.dll, but apparently it has to be the version against which it was compiled. The required DLL is bundled with PHP, but what if an older, incompatible version of that DLL is found earlier in the search path? You’re hosed, that’s what if, and the module load fails with a message to the log file like so:

PHP Warning: PHP Startup: Unable to load dynamic library 'D:\\php-5.2.3\\ext\\php_mysql.dll' - The specified procedure could not be found.\r\n in Unknown line 0

I suppose that would be a missing symbol in the underlying library on other platforms. A quick Google put me on the right track and sure enough the indispensable Process Explorer utility tells me that httpd.exe has C:\WINDOWS\system32\libmysql.dll loaded. Yup, that’s our problem. No, I didn’t build that box. That box must have been built by someone who thought copying crap into the Windows directory is ever a good idea.

How do we solve this the Apache Way? Not by copying DLLs around, that’s for sure. The Apache configuration language has the LoadFile directive for this particular purpose. Loading the correct DLL right before the PHP module:

LoadFile "d:/php-5.2.3/libmysql.dll"
LoadModule php5_module "d:/php-5.2.3/php5apache2_2.dll"

makes PHP pick up the right symbols to run with MySQL.

How Henning Made Me Actually Do It

Posted on by

Like yachtsmen, programmers are lazy. Being a little bit of both, I guess that makes me doubly lazy, and a byproduct of this seems to be that I just don’t get around to signing the keys that emerge from the ApacheCon Keysigning sessions I organize.

Henning the Fussbal FanFor programmers, laziness manifests itself in the wish to write programs to perform repetitive tasks, not infrequently spending more time writing the program than it would have taken to just perform the task by hand. Of course the merit of writing such a program is that others can use it to perform the same repetitive task, without having to write the program first. A couple of days ago, Apache’s Henning Schmiedehausen posted PGPSigner, a utility that helps you sign all the keys on your list from the Keysigning session.

This is absolutely great, it just helped me do in five minutes what I had already put off for close to a month, and the keys from the last keysigning session are now signed, uploaded to the keyservers (pgpkeys.mit.edu and minsky.surfnet.nl) and mailed to their owners insofar their mail got through. I used the Signing Party Keyring that contains the keys of all the attendants.

Small patch to make the startup script suck in the jars in the lib directory:


Index: pgpsigner.sh
===================================================================
--- pgpsigner.sh (revision 1009)
+++ pgpsigner.sh (working copy)
@@ -31,4 +31,8 @@
exit 1
fi

-java -jar target/${APPNAME}-${APPVERSION}.jar "$@"
+for j in `ls lib/*.jar` ; do
+ CLASSPATH="$CLASSPATH:$j"
+done
+
+java -classpath $CLASSPATH -jar target/${APPNAME}-${APPVERSION}.jar "$@"

This makes it easier to run the program from the command line as opposed to from within Eclipse. (: Thank you Henning.

Bugzilla 3.0 Has Landed

Posted on by

I am currently playing around with the recently released Bugzila 3.0. Before I upgrade my production instance, I’m doing a clean installation on my desktop box to see what it looks like, and whether or where any upgrade difficulties might be lurking.

Bugzilla uses quite a few Perl modules, and their install guide lists these by their Perl name. However, on my Ubuntu system, I’d like to install these modules through the packages instead of CPAN, so I keep a somewhat clean system. Here is a table that translates between the Debian Package name and the Perl Module name:

Required Modules
Ubuntu Package Perl Module Notes
CGI perl-modules  
Date::Format libtimedate-perl Found the package name through Bugzilla’s checksetup.pl --check-modules, which calls it TimeDate
DBI libdbi-perl  
DBD::mysql  
File::Spec perl-base checksetup.pl speaks of PathTools, which may be a bundle containing this module
Template libtemplate-perl  
Email::Send   No package, installed via CPAN. Note its dependencies Module::Pluggable and Return::Value do have packages as libmodule-pluggable-perl and libreturn-value-perl respectively.
Email::MIME::Modifier libemail-mime-modifier-perl  
Optional Modules
Ubuntu Package Perl Module Notes
GD libgd-perl  
Template::Plugin::GD::Image libtemplate-perl Not in separate package
Chart::Base libchart-perl  
GD::Graph libgd-graph-perl Brings along libgd-text-perl as a dependency
GD::Text libgd-text-perl Sucked in by libgd-graph-perl as a dependency
XML::Twig libxml-twig-perl  
MIME::Parser libmime-perl  
LWP::UserAgent libwww-perl  
PatchReader No Package Installed through CPAN
Image::Magick perlmagick  
Net::LDAP libnet-ldap-perl:  
SOAP::Lite libsoap-lite-perl  
HTML::Parser libhtml-parser-perl:  
HTML::Scrubber libhtml-scrubber-perl  
Email::MIME::Attachment::Stripper No Package Installed via CPAN
Email::Reply No Package Installed from CPAN. Dependency Email::Abstract has a package (libemail-abstract-perl), but dependency Email::MIME::Creator has not and would have to be installed from CPAN. However, this is where we run into trouble: the package does not test cleanly. Perhaps this is because CPAN builds packages in /usr/local, and the rest of Email::MIME was installed in /usr as per usual under Linux packaging systems. Rather than forcing the issue, Email::Reply was not installed.
mod_perl2 libapache2-mod-perl2 This package works with the Apache server installed by the apache2 package
CGI N/A Uprev 3.11 or above required by mod_perl, not available under packaging system which installs 3.10
Apache::DBI libapache-dbi-perl  

A great help for finding already installed modules is dpkg –search . This will tell you which package a given file on your machine belongs to. For instance, it tells me that CGI.pm belongs to perl-modules, which is the default module collection on the system:


sctemme@sarlacc:~$ dpkg --search /usr/share/perl/5.8.7/CGI.pm
perl-modules: /usr/share/perl/5.8.7/CGI.pm

To get the name of a module not yet installed, you can use apt-cache search, and keep in mind that modules are typically named after the Perl module except all lowercase, like TimeDate becomes libtimedate-perl. If the package name proves elusive, grepping through the output of apt-cache search ".*-perl$" should help. When all else fails, go to CPAN.

Another neat tool is apt-file. This doesn’t appear to be part of the standard tools on Ubuntu, but can be installed as a package. First run apt-file update (as root), then you can run apt-file search <pattern>, with as pattern the name of a file you know should be part of a package. It will return the names of any package that has such a file in it.

What do we learn from this?

  1. All but one required module have a corresponding package, and Email::Send installs cleanly
  2. Inbound e-mail can’t be done with the Ubuntu/Debian packages at the moment, since one of the dependencies is broken
  3. mod_perl operation requires a slightly higher level of the CGI interface than is delivered by the packaging system. Since my production system doesn’t use the Ubuntu packaged Apache, I am hardly a mod_perl expert and we are currently doing fine without, I’m not going to worry about mod_perl at this time. Perhaps in the future

So, what does Bugzilla 3.0 look like? Haven’t found out yet, but stay tuned.