This is a short overview of what happened on Friday August 28 2009
to the services.  A more detailed post will come at a later
time after we complete the audit of all machines involved.

On August 27th, starting at
about 18:00 UTC an account used for automated backups for the ApacheCon
website hosted on a 3rd party hosting provider was used to upload files
to  The account was accessed using SSH key
authentication from this host.

To the best of our knowledge at this time, no end users were affected by this incident,  and the attackers were not able to escalate their
privileges on any machines.

While we have no evidence that downloads were affected, users are always advised to check digital
signatures where provided. runs
FreeBSD 7-STABLE and is more widely known as 
Minotaur serves as the seed host for most websites, in
addition to providing shell accounts for all Apache committers.

attackers created several files in the directory containing files for, including several CGI scripts.  These files were then
rsynced to our production webservers by automated processes.  At about
07:00 on August 28 2009 the attackers accessed these CGI scripts over
HTTP, which spawned processes on our production web services.

At about 07:45 UTC we noticed these rogue processes on, the Solaris 10 machine that normally serves our websites.

Within the next 10 minutes we decided to shutdown all machines involved as a precaution.

an initial investigation we changed DNS for most services to, a machine not affected and provided a basic downtime

After investigation, we determined that our European fallover and backup machine,, was not affected.   While
the some files had been copied to the machine by automated rsync
processes, none of them were executed on the host, and we restored from
a ZFS snapshot to a version of all our websites before any accounts
were compromised.

At this time several machines remain offline, but most user facing websites and services are now available.

We will provide more information as we can.