Azure bug bounty Pwning Red Hat Enterprise LinuxNov 26, 2016 · 5 minute read · Comments
TL;DR Acquired administrator level access to all of the Microsoft Azure managed Red Hat Update Infrastructure that supplies all the packages for all Red Hat Enterprise Linux instances booted from the Azure marketplace.
I was tasked with creating a machine image of Red Hat Enterprise Linux that was compliant to the Security Technical Implementation guide defined by the Department of Defense.
This machine image was to be used for both Amazon Web Services and Microsoft Azure. Both of which offer marketplace images which had a metered billing pricing model. Ideally, I wanted my custom image to be billed under the same mechanism, as such the virtual machines would be able to consume software updates from a local Red Hat Enterprise Linux repository owned and managed by the cloud provider.
This setup requires two main parts:
Red Hat Update Appliance
The Red Hat Update Appliance is responsible for:
- Downloading new packages from the Red Hat CDN. It is the only component that requires a connection back to Red Hat.
- Copying new packages to each content delivery server
- Supplying a REST API and command line interface for management of software repositories
The Red Hat Update Appliance does not need to be exposed to the repository clients.
Content Delivery server
The content delivery server(s) provide the yum repositories that clients connect to for updated packages.
Achieving metered billing
However, these are the same SSL certificates for every instance.
On Amazon Web Services having the SSL certificates is not enough, you must have booted your instance from an AMI that had an associated billing code. It is this billing code that ensures you pay the extra premium for running Red Hat Enterprise Linux.
On Azure it remains undefined how they manage to track billing. At the time of research, it was possible to copy the SSL certificates from one instance to another and successfully authenticate. Additionally, if you duplicated a Red Hat Enterprise Linux virtual hard disk and created a new instance from it all billing association seemed to be lost but repository access was still available.
Where Azure Failed
On Azure to setup repository connectivity, they provide an RPM with the necessary configuration. In the older version of their agent, it is responsible for this task . The installation script it references comes from the following archive. If you expand this archive you will find the client configuration for each region.
By browsing the metadata of the RPMs we can discover some interesting information:
$ rpm -qip RHEL6-2.0-1.noarch.rpm Name : RHEL6 Relocations: (not relocatable) Version : 2.0 Vendor: (none) Release : 1 Build Date: Sun 14 Feb 2016 06:40:54 AM UTC Install Date: (not installed) Build Host: westeurope-rhua.cloudapp.net Group : Applications/Internet Source RPM: RHEL6-2.0-1.src.rpm Size : 20833 License: GPLv2 Signature : (none) URL : http://redhat.com Summary : Custom configuration for a cloud client instance Description : Configurations for a client to connect to the RHUI infrastructure
As you can see, the build host enables us to discover all of the Red Hat Update Appliances:
$ host westeurope-rhua.cloudapp.net westeurope-rhua.cloudapp.net has address 184.108.40.206 $ host eastus2-rhua.cloudapp.net eastus2-rhua.cloudapp.net has address 220.127.116.11 $ host southcentralus-rhua.cloudapp.net southcentralus-rhua.cloudapp.net has address 18.104.22.168 $ host southeastasia-rhua.cloudapp.net southeastasia-rhua.cloudapp.net has address 22.214.171.124
At the time of research, all of servers were exposing their REST APIs over HTTPs.
$ yumdownloader PrepareRHUI $ rpm -qip PrepareRHUI-1.0.0-1.noarch.rpm Name : PrepareRHUI Relocations: (not relocatable) Version : 1.0.0 Vendor: Microsoft Corporation Release : 1 Build Date: Mon 16 Nov 2015 06:13:21 AM UTC Install Date: (not installed) Build Host: rhui-monitor.cloudapp.net Group : Unspecified Source RPM: PrepareRHUI-1.0.0-1.src.rpm Size : 770 License: GPL Signature : (none) Packager : Microsoft Corporation <firstname.lastname@example.org> Summary : Prepare RHUI installation for Redhat client Description : PrepareRHUI is used to prepare RHUI installation for before making a Redhat image.
The build host is interesting
rhui-monitor.cloudapp.net, at the time of research running a port scan revealed an application running on port 8080.
Despite the application requiring username and password based authentication, It was possible to execute a run of their “backend log collector” on a specified content delivery server. When the collector service completed the application supplied URLs to archives which contain multiple logs and configuration files from the servers.
Included within these archives was an SSL certificate that would grant full administrative access to the Red Hat Update Appliances .
At the time of research all Red Hat Enterprise Linux virtual machines booted from the Azure Marketplace image had the following additional repository configured:
[rhui-PA] name=Packages for Azure mirrorlist=https://eastus2-cds1.cloudapp.net/pulp/mirror/PA enabled=1 gpgcheck=0 sslverify=1 sslcacert=/etc/pki/rhui/ca.crt
Given no gpgcheck is enabled, with full administrative access to the Red Hat Enterprise Linux Appliance REST API one could have uploaded packages that would be acquired by client virtual machines on their next yum update.
The issue was reported in accordance to the Microsoft Online Services Bug Bounty terms. Microsoft agreed it was a vulnerability in their systems. Immediate action was taken to prevent public access to
rhui-monitor.cloudapp.net. Additionally, they eventually prevented public access to the Red Hat Update Appliances and they claim to have rotated all secrets.