UPDATE 1: The instructions below are incomplete. The proper upgrade command is "apt-get install openssl libssl1.0.0". If you've just updated openssl, please go back and update libssl as well.

UPDATE 2: Adding instructions for VMWare-hosted System VMs (Thanks to Geoff Higginbottom at ShapeBlue)

UPDATE 3: Added instructions for verification of correct versions installed

UPDATE 4: Apache CloudStack 4.0-4.1 not vulnerable, they use older Debian/openssl.

Earlier this week, a security vulnerability was disclosed in OpenSSL, one of the software libraries that Apache CloudStack uses to encrypt data sent over network network connections. As the vulnerability has existed in OpenSSL since early 2012, System VMs in Apache CloudStack versions 4.1.1-4.3 are running software using vulnerable versions of OpenSSL. This includes CloudStack's Virtual Router VMs, Console Proxy VMs, and Secondary Storage VMs.

We are actively working on creating updated System VM templates for each recent version of Apache CloudStack, and for each of the hypervisor platforms which Apache CloudStack supports. Due to our testing and QA processes, this will take several days. In the meantime, we want to provide our users with a temporary workaround for currently running System VMs.

If you are running Apache CloudStack 4.0.0-incubating through the recent 4.3 release, the the following steps will help ensure the security of your cloud infrastructure until an updated version of the System VM template is available:

For KVM/Xen hosted systems

  1. As an administrator in the CloudStack web UI, navigate to Infrastructure->System VMs
  2. For each System VM listed, note the host it is running on, and it’s “Link Local IP address."
  3. With that data, perform the following steps for each System VM:
    1. ssh into that host as root
    2. From the host, ssh into the SSVM via it’s link local IP address: (e.g. ssh -i /root/.ssh/id_rsa.cloud -p 3922 169.254.3.33)
    3. On the System VM, first run "apt-get update"
    4. Then run apt-get install "openssl libssl1.0.0". If a dialog appears asking to restart programs, accept it’s request.
    5. Next, for Secondary Storage VMs, run /etc/init.d/apache2 restart
    6. Log out of the System VM and host server
  4. Back in the CloudStack UI, now navigate to Infrastructure->Virtual Routers. For each VR, host it's running on and it's link local IP address, and then repeat steps a-f above.

For VMWare hosted systems

  1. As an administrator in the CloudStack web UI, navigate to Infrastructure->System VMs
  2. For each System VM listed, note it's management IP address
  3. With that data, perform the following steps for each System VM:
    1. From the Management Server, ssh to the System VM via it's management IP: (eg ssh -i /var/lib/cloud/management/.ssh/id_rsa -p 3922 root@10.40.50.8)
    2. On the System VM, first run "apt-get update"
    3. Then run apt-get install "openssl libssl1.0.0". If a dialog appears asking to restart programs, accept it’s request.
    4. Next, for Secondary Storage VMs, run /etc/init.d/apache2 restart
    5. Log out of the System VM
  4. Back in the CloudStack UI, now navigate to Infrastructure->Virtual Routers. For each VR, host it's running on and it's link local IP address, and then repeat steps a-f above.

Verification

On each System VM, you can test if it has non-vulnerable openssl packages installed by listing installed packages and looking at the installed versions of openssl and libssl. As in the example below, for a system to be non-vulnerable, the packages need to be at or above version 1.0.1e-2+deb7u6:

root@v-14-VM:~# dpkg -l|grep ssl
ii  libssl1.0.0:i386                     1.0.1e-2+deb7u6                  i386         SSL shared libraries
ii  openssl                              1.0.1e-2+deb7u6                  i386         Secure Socket Layer (SSL) binary and related cryptographic tools

We realize that for larger installations where System VMs are being actively created and destroyed based on customer demand, this is a very rough stop-gap. The Apache CloudStack security team is actively working on a more permanent fix and will be releasing that to the community as soon as possible.

For Apache CloudStack installations that secure the web-based user-interface with SSL, these may also be vulnerable to HeartBleed, but that is outside the scope of this blog post. We recommend testing your installation with [1] to determine if you need to patch/upgrade the SSL library used by any web servers (or other SSL-based services) you use.

1: http://filippo.io/Heartbleed/