Azure bug bounty Root to storage account administrator

In my previous blog post Azure bug bounty Pwning Red Hat Enterprise Linux I detailed how it was possible to get administrative access to the Red Hat Update Infrastructure consumed by Red Hat Enterprise Linux virtual machines booted from the Microsoft Azure Marketplace image. In theory, if exploited one could have gained root access to all virtual machines consuming the repositories by releasing an updated version of a common package and waiting for virtual machines to execute yum update.

As an attacker, this would have granted access to every piece of data on the compromised virtual machines. Sadly, the attack vector is actually much more widespread than this. Given some poor implementation within the mandatory Microsoft Azure Linux Agent (WaLinuxAgent) one is able to obtain the administrator API keys to the storage account used by the virtual machine for debug log shipping purposes, at the time of research this storage account defaulted to one shared by multiple virtual machines.

At the time of research, the Red Hat Enterprise Linux image available on the Microsoft Azure Marketplace came with WaLinuxAgent 2.0.16. When a virtual machine was created with the “Linux diagnostic extension” enabled the API key for access to the specified storage account was written to /var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9007/xmlCfg.xml.

Once acquired one can simply use the Azure Xplat-CLI to interact the storage account:

export AZURE_STORAGE_ACCOUNT="storage_account_name_as_per_xmlcfg"
export AZURE_STORAGE_ACCESS_KEY="storage_account_access_key_as_per_xmlcfg"
azure storage container list # acquire some container name
azure storage blob list # provide the container name
# Copy, download, upload, delete any blobs available across any containers you can access.

If the storage account was used by multiple virtual machines there is potential to download their virtual hard disks.

Azure bug bounty Pwning Red Hat Enterprise Linux

TL;DR Acquired administrator level access to all of the Microsoft Azure managed Red Hat Update Infrastructure that supplies all the packages for all Red Hat Enterprise Linux instances booted from the Azure marketplace.

I was tasked with creating a machine image of Red Hat Enterprise Linux that was compliant to the Security Technical Implementation guide defined by the Department of Defense.

This machine image was to be used for both Amazon Web Services and Microsoft Azure. Both of which offer marketplace images which had a metered billing pricing model[1][2]. Ideally, I wanted my custom image to be billed under the same mechanism, as such the virtual machines would be able to consume software updates from a local Red Hat Enterprise Linux repository owned and managed by the cloud provider.

Both Amazon Web Services and Microsoft Azure utilise a deployment of Red Hat Update Infrastructure for supplying this functionality.

This setup requires two main parts:

Red Hat Update Appliance

There is only one Red Hat Update Appliance per Red Hat Update Infrastructure installation, however, both Amazon Web Services and Microsoft Azure create one per region.

The Red Hat Update Appliance is responsible for:

The Red Hat Update Appliance does not need to be exposed to the repository clients.

Content Delivery server

The content delivery server(s) provide the yum repositories that clients connect to for updated packages.

Achieving metered billing

Both Amazon Web Services and Microsoft Azure use SSL certifications for authentication against the repositories.

However, these are the same SSL certificates for every instance.

On Amazon Web Services having the SSL certificates is not enough, you must have booted your instance from an AMI that had an associated billing code. It is this billing code that ensures you pay the extra premium for running Red Hat Enterprise Linux.

On Azure it remains undefined how they manage to track billing. At the time of research, it was possible to copy the SSL certificates from one instance to another and successfully authenticate. Additionally, if you duplicated a Red Hat Enterprise Linux virtual hard disk and created a new instance from it all billing association seemed to be lost but repository access was still available.

Where Azure Failed

On Azure to setup repository connectivity, they provide an RPM with the necessary configuration. In the older version of their agent, it is responsible for this task [3]. The installation script it references comes from the following archive. If you expand this archive you will find the client configuration for each region.

By browsing the metadata of the RPMs we can discover some interesting information:

$ rpm -qip RHEL6-2.0-1.noarch.rpm
Name        : RHEL6                        Relocations: (not relocatable)
Version     : 2.0                               Vendor: (none)
Release     : 1                             Build Date: Sun 14 Feb 2016 06:40:54 AM UTC
Install Date: (not installed)               Build Host: westeurope-rhua.cloudapp.net
Group       : Applications/Internet         Source RPM: RHEL6-2.0-1.src.rpm
Size        : 20833                            License: GPLv2
Signature   : (none)
URL         : http://redhat.com
Summary     : Custom configuration for a cloud client instance
Description :
Configurations for a client to connect to the RHUI infrastructure

As you can see, the build host enables us to discover all of the Red Hat Update Appliances:

$ host westeurope-rhua.cloudapp.net
westeurope-rhua.cloudapp.net has address 104.40.209.83

$ host eastus2-rhua.cloudapp.net
eastus2-rhua.cloudapp.net has address 13.68.20.161

$ host southcentralus-rhua.cloudapp.net
southcentralus-rhua.cloudapp.net has address 23.101.178.51

$ host southeastasia-rhua.cloudapp.net
southeastasia-rhua.cloudapp.net has address 137.116.129.134

At the time of research, all of servers were exposing their REST APIs over HTTPs.

The URL to the archive containing these RPMs was discovered a package labeled PrepareRHUI on available on any Red Hat Enterprise Linux Box running on Microsoft Azure.

$ yumdownloader PrepareRHUI
$ rpm -qip PrepareRHUI-1.0.0-1.noarch.rpm
Name        : PrepareRHUI                  Relocations: (not relocatable)
Version     : 1.0.0                             Vendor: Microsoft Corporation
Release     : 1                             Build Date: Mon 16 Nov 2015 06:13:21 AM UTC
Install Date: (not installed)               Build Host: rhui-monitor.cloudapp.net
Group       : Unspecified                   Source RPM: PrepareRHUI-1.0.0-1.src.rpm
Size        : 770                              License: GPL
Signature   : (none)
Packager    : Microsoft Corporation <xiazhang@microsoft.com>
Summary     : Prepare RHUI installation for Redhat client
Description :
PrepareRHUI is used to prepare RHUI installation for before making a Redhat image.

The build host is interesting rhui-monitor.cloudapp.net, at the time of research running a port scan revealed an application running on port 8080.

Microsoft Azure RHUI Monitoring tool

Despite the application requiring username and password based authentication, It was possible to execute a run of their “backend log collector” on a specified content delivery server. When the collector service completed the application supplied URLs to archives which contain multiple logs and configuration files from the servers.

Included within these archives was an SSL certificate that would grant full administrative access to the Red Hat Update Appliances [4].

Pulp admin keys for Microsoft Azure's RHUI

At the time of research all Red Hat Enterprise Linux virtual machines booted from the Azure Marketplace image had the following additional repository configured:

[rhui-PA]
name=Packages for Azure
mirrorlist=https://eastus2-cds1.cloudapp.net/pulp/mirror/PA
enabled=1
gpgcheck=0
sslverify=1
sslcacert=/etc/pki/rhui/ca.crt

Given no gpgcheck is enabled, with full administrative access to the Red Hat Enterprise Linux Appliance REST API one could have uploaded packages that would be acquired by client virtual machines on their next yum update.

The issue was reported in accordance to the Microsoft Online Services Bug Bounty terms. Microsoft agreed it was a vulnerability in their systems. Immediate action was taken to prevent public access to rhui-monitor.cloudapp.net. Additionally, they eventually prevented public access to the Red Hat Update Appliances and they claim to have rotated all secrets.

[1] https://azure.microsoft.com/en-in/pricing/details/virtual-machines/red-hat/

[2] https://aws.amazon.com/partners/redhat/

[3] https://github.com/Azure/azure-linux-extensions/blob/master/Common/WALinuxAgent-2.0.16/waagent#L2891

[4] https://fedorahosted.org/pulp/wiki/Certificates

Hello World

resource "null_resource" "hello_world" {
    provisioner "local-exec" {
        command = "echo 'Hello World'"
    }
}

Interested in automation and the HashiCorp suite of tools? If so you’ll love this blog. Through different posts we will explore lots different automation tasks utilising both public and private cloud with the HashiCorp toolset.

Thanks, Ian.