Docker

software-development

Videos
Articles

https://hub.docker.com/
https://hub.docker.com/_/amazonlinux/

Bitbucket

sudo docker exec -i -t 665b4a1e17b6 /bin/bash

yum install docker.x86_64 docker-storage-setup.noarch
sudo service docker start
sudo usermod -a -G docker ec2-user // logout and log back in for this to be effective
docker run hello-world

fallocate -l 20G /usr/local/DockerDrive.img
sudo mkfs.btrfs -f /usr/local/DockerDrive.img
Add to /etc/fstab:
/usr/local/DockerDrive.img      /var/lib/docker btrfs   defaults        0       0
Run mount -a to mount the new entry without rebooting the system
Check to see if the new entry got mounted: df -h

I think to force it you could also add -s=aufs to the default options in /etc/default/docker
Solution is to switch either to aufs or overlayfs drivers.

Try the latest official package from docker
Delete the /var/lib/docker and try a different storage driver
Try installing linux-image-extra and use aufs

/etc/docker/daemon.json:

{
"debug": true,
"hosts": ["tcp://127.0.0.1:5000", "unix:///var/run/docker.sock"]
}

> cat Dockerfile

FROM fedora

RUN yum install myapp

EXPOSE 8080

CMD /user/bin/myapp

> docker build -t vbatts/myapps

...

> docker push vbatts/myapps

What is Docker?

Docker separates applications from infrastructure using container technology similar to how virtual machine separate the operating system from bare metal.

Docker virtualize the operating system.

Docker is a tool that allows developers, sys-admins to easily deploy their applications in a sandbox (called containers) to run on the host operating system. They key benefit of Docker is that it allows users to package an application with all of its dependencies into a standard unit. Unlike Virtual Machines (VM), containers do not have the high overhead and therefore enable more efficient usage of the underlying system and resources.

The industry standard today is to use Virtual Machines (VMs) to run software applications. VMs run applications inside a guest Operating System, which runs on virtual hardware powered by the server’s host OS.

VMs are great at providing full process isolation for applications: there are very few ways a problem in the host operating system can affect the software running in the guest operating system, and vice-versa. But this isolation comes at great cost — the computational overhead spent virtualizing hardware for a guest OS to use is substantial.

Containers take a different approach: by leveraging the low-level mechanics of the host operating system, containers provide most of the isolation of virtual machines at a fraction of the computing power.

Does Docker work on Windows?

Yes.

What platforms does Docker support?

Linux, Mac, and Windows.

How can we install Docker?

Docker works on Mac, Linux, and Windows

How can we validate the Docker was installed successfully?

docker run hello-world

The above command should displays "Hello from Docker". This message shows that your installation appears to be working.

What might Docker depends on?

  1. Python. To check if we have Python installed, run: python —version.
  2. pip. To check if we have pip installed, run: pip —version

What does the 'docker pull' command do?

docker pull busybox

The pull command fetches the busybox image ( https://hub.docker.com/busybox/ ) from the Docker registry (https://hub.docker.com/explore/ ) and save it to our system.

What does the 'docker images' command do?

docker images

Displays a list of docker images on our system.

What should we do if we face a 'permission denied' message when using docker commands?

If we are on a Mac, make sure that the Docker engine is running. If we are on a Linux, prefix our docker commands with sudo. Alternatively, we can create a docker group ( https://docs.docker.com/engine/installation/linux/linux-postinstall/ ) to get rid of this issue.

What does the 'docker run' command do?

docker run busybox

The above command create a running instance (or a container) from the busybox image. Wait, nothing happened! Is that a bug? Well, no. Behind the scenes, a lot of stuff happened. When you call run, the Docker client finds the image (busybox in this case), loads up the container and then runs a command in that container. When we run docker run busybox, we didn't provide a command, so the container booted up, ran an empty command and then exited. Well, yeah - kind of a bummer. Let's try something more exciting.

docker run busybox echo "hello from busybox"

In this case, the Docker client dutifully ran the echo command in our busybox container and then exited it. If you've noticed, all of that happened pretty quickly. Imagine booting up a virtual machine, running a command and then killing it. Now you know why they say containers are fast!

What does the 'docker ps' command do?

docker ps

The 'docker ps' command displays all containers that are currently running.

What does the 'docker ps -a' command do?

The 'docker ps -a' command displays all containers, including the ones that exited, but have not been garbage-collected.

What is the purpose of the '-it' option of the 'docker run' command?

docker run -it busybox sh

Running the 'docker run' command with the --it flags attach us to an interactive tty in the container. Now we can run as many commands in the container as we want.

What happens if we accidentally remove the bin directory?

If you're feeling particularly adventurous you can try rm -rf bin in the container. Make sure you run this command in the container and not in your laptop. Doing this will not make any other commands like ls, echo work. Once everything stops working, you can exit the container (type exit and press Enter) and then start it up again with the docker run -it busybox sh command. Since Docker creates a new container every time, everything should start working again.

What is the difference between 'docker ps' and 'docker ps -a'?

The 'docker ps' command displays a list of containers that is currently running. The 'docker ps -a' command displays a list of containers that is currently running, and the ones that exited but have not been garbage-collected. As we run docker instances, we are leaving stray containers, which eats up disk space. We should clean up containers when we are done with them. To do that, we can run the 'docker rm' command. Just copy the container IDs from the output of 'docker ps -a' and paste them:

docker rm 305297d7a235 ff0a5c3750b9

What does the 'docker rm' command do?

The 'docker ps' command displays a list of containers that is currently running. The 'docker ps -a' command displays a list of containers that is currently running, and the ones that exited but have not been garbage-collected. As we run docker instances, we are leaving stray containers, which eats up disk space. We should clean up containers when we are done with them. To do that, we can run the 'docker rm' command. Just copy the container IDs from the output of 'docker ps -a' and paste them:

docker rm 305297d7a235 ff0a5c3750b9

How can we delete all instances that exited in one go?

If we have a bunch of containers to delete in one go, copy-pasting IDs can be tedious. In that case, we can simply run:

docker rm $(docker ps -a -q -f status=exited)

What is the purpose of the --rm flag of the 'docker run' command?

This tells docker to automatically delete the container after it exited. For one off docker runs, the --rm flag is very useful.

What does the 'docker rmi' command do?

It removes the image.

What is a docker image?

It is the blueprint for the container.

What is a container?

Containers created from images and run the actual application. We create a container using the 'docker run' command.

What is the Docker Daemon?

The Docker Daemon is the background service running on the host that manages building, running, and distributing Docker containers. The daemon is the process that runs in the operating system which the clients talk to.

What is the Docker client?

The command line tool that allows the user to interact with the daemon. More generally, there can be other forms of clients too, such as Kitematic which provide a GUI to the users.

What is Docker Hub?

A registery of Docker images. If required, we can host our own Docker registry.

How can we run a docker instance in such a way that our terminal is not attached to it?

Specify the -d flag to the 'docker run' command:

docker run -d -P --name static-site prakhar1989/static-site

What is the purpose of the -P option of the 'docker run' command?

The -P option publish all exposed ports to random ports. We can see the ports by running:

docker port [CONTAINER]
docker port static-site

80/tcp -> 0.0.0.0:32769
443/tcp -> 0.0.0.0:32768

In the above example, port 80 inside the container is mapped to port 32769 on the host machine.

How can we get the IP address of the docker instance if we are using docker-toolbox?

If we are using docker-toolbox, then you might need to use docker-machine ip default to get the IP.

How can we specify custom port to which the client will forward connections to the container?

Specify the -p option of the 'docker run' command:

docker run -p 8888:80 prakhar1989/static-site

In the above example, we specify that port 8888 on the host machine should be forwarded to port 80 inside the container.

How can we stop a detached container?

Run the 'docker stop' command with a container ID

How can we deploy a docker image?

To deploy a docker image on a real server, we would just install Docker, and run the 'docker run' command.

Why is docker image akin to a git repository?

For simplicity, we can think of docker image akin to a git repository. Images can be committed with changes and have multiple versions. If we do not provide a specific version number, the client default to the latest. For example, we can pull the specific version of the ubuntu image:

docker pull ubuntu:12.04

How can we pull a specific version of an image?

docker pull ubuntu:12.04

The above command pulls version 12.04 of the ubuntu image.

How can we find docker images?

There are tens of thousands of images available on Docker Hub. We can also search for images directly from the command line using the 'docker search' command.

What is a base image?

Base images are images that have no parent image.

What is a child image?

Child images are images that build on base images and add additional functionality.

What are official images?

Official images are images that are officially maintained and supported by the folks at Docker. These are typically one word long.

What are user images?

User images are images created and shared by users like you and me. Typically these are formatted as user/image-name.

What is the onbuild version?

These images include multiple ONBUILD triggers, which should be all we need to bootstrap most applications. The build will COPY a requirements.txt file, RUN pip install, and then copy the current directory to /usr/src/app

In other words, the onbuild version of the images includes helpers that automate the boring parts of getting an app running.

What is the purpose of the Dockerfile file?

The Dockerfile is a simple text-file that contains a list of commands that the Docker client calls while creating an image. It's a simple way to automate the image creation process. The best part is that the commands you write in a Dockerfile are almost identical to their equivalent Linux commands. This means you don't really have to learn new syntax to create your own dockerfiles.

We start with specifying our base image. Use the FROM keyword to do that:

FROM python:3-onbuild

The next thing we need to the specify is the port number that needs to be exposed. Since our app is running on port 5000, that's what we'll indicate.

EXPOSE 5000

Next, we will add the command for starting the application, which is 'python ./app.py'. We use the CMD command:

CMD ["python", "./app.py"]

The primary purpose of CMD is to tell the container which command it should run when it is started. With that, our Dockerfile is now ready. This is how it looks like:

# our base image
FROM python:3-onbuild

# specify the port number the container should expose
EXPOSE 5000

# run the application
CMD ["python", "./app.py"]

What does the 'docker build' command do?

Now that we have our Dockerfile, we can build our image. The docker build command does the heavy-lifting of creating a Docker image from a Dockerfile.

docker build -t username/image-name .

The docker build command takes an optional tag name with the -t option, and a location of the directory containing the Dockerfile. Notice the dot at the end of the above command.

Now, run the image and see if it actually work:

docker run -p 8888:5000 username/image-name

Can we host our own Docker registry?

Yes. https://docs.docker.com/registry/deploying/

How can we publish our image to Docker Hub?

docker push username/image-name

Do we have to publish our image?

No. It is not imperative to host your image on a public registry (or any registry) in order to deploy to AWS. In case you're writing code for the next million-dollar unicorn startup you can totally skip this step. The reason why we're pushing our images publicly is that it makes deployment super simple by skipping a few intermediate configuration steps.

How can we run Docker on Amazon Elastic Beanstalk?

AWS Elastic Beanstalk (EB) is a PaaS (Platform as a Service) offered by AWS. If you've used Heroku, Google App Engine etc. you'll feel right at home. As a developer, you just tell EB how to run your app and it takes care of the rest - including scaling, monitoring and even updates. In April 2014, EB added support for running single-container Docker deployments which is what we'll use to deploy our app. Although EB has a very intuitive CLI, it does require some setup, and to keep things simple we'll use the web UI to launch our application.

To follow along, you need a functioning AWS account. If you haven't already, please go ahead and do that now - you will need to enter your credit card information. But don't worry, it's free and anything we do in this tutorial will also be free! Let's get started. Here are the steps:

  1. Login to your AWS console.
  2. Click on Elastic Beanstalk. It will be in the compute section on the top left. Alternatively, just click here to access the EB console.
  3. Click on "Create New Application" in the top right
  4. Give your app a memorable (but unique) name and provide an (optional) description
  5. In the New Environment screen, choose the Web Server Environment.
  6. The following screen is shown below. Choose Docker from the predefined configuration. You can leave the Environment type as it is. Click Next.
  7. This is where we need to tell EB about our image. Open the Dockerrun.aws.json file located in the flask-app folder and edit the Name of the image to your image's name.
  8. When you are done, click on the radio button for "upload your own" and choose this file.
  9. Next up, choose an environment name and a URL. This URL is what you'll share with your friends so make sure it's easy to remember.
  10. For now, we won't be making changes in the Additional Resources section. Click Next and move to Configuration Details.
  11. In this section, all you need to do is to check that the instance type is t1.micro. This is very important as this is the free instance by AWS. You can optionally choose a key-pair to login. If you don't know what that means, feel free to ignore this for now. We'll leave everything else to the default and forge ahead.
  12. We also don't need to provide any Environment Tags and Permissions, so without batting an eyelid, you can click Next twice in succession. At the end, the screen shows us the Review page. If everything looks good, go ahead and press the Launch button.
  13. The final screen that you see will have a few spinners indicating that your environment is being set up. It typically takes around 5 minutes for the first-time setup.

While we wait, let's quickly see what the Dockerrun.aws.json file contains. This file is basically an AWS specific file that tells EB details about our application and docker configuration.

{
  "AWSEBDockerrunVersion": "1",
  "Image": {
    "Name": "prakhar1989/catnip",
    "Update": "true"
  },
  "Ports": [
    {
      "ContainerPort": "5000"
    }
  ],
  "Logging": "/var/log/nginx"
}

The file should be pretty self-explanatory, but you can always reference the official documentation for more information. We provide the name of the image that EB should use along with a port that the container should open.

Hopefully by now, our instance should be ready. Head over to the EB page and you should a green tick indicating that your app is alive and kicking. Go ahead and open the URL in your browser and you should see the application in all its glory. Feel free to email / IM / snapchat this link to your friends and family.

See http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/docker-singlecontainer-deploy.html

Why should we have a multi-container environments?

In the last section, we saw how easy and fun it is to run applications with Docker. We started with a simple static website and then tried a Flask app. Both of which we could run locally and in the cloud with just a few commands. One thing both these apps had in common was that they were running in a single container.

Those of you who have experience running services in production know that usually apps nowadays are not that simple. There's almost always a database (or any other kind of persistent storage) involved. Systems such as Redis and Memcached have become de riguer of most web application architectures. Hence, in this section we are going to spend some time learning how to Dockerize applications which rely on different services to run.

In particular, we are going to see how we can run and manage multi-container docker environments. Why multi-container you might ask? Well, one of the key points of Docker is the way it provides isolation. The idea of bundling a process with its dependencies in a sandbox (called containers) is what makes this so powerful.

Just like it's a good strategy to decouple your application tiers, it is wise to keep containers for each of the services separate. Each tier is likely to have different resource needs and those needs might grow at different rates. By separating the tiers into different containers, we can compose each tier using the most appropriate instance type based on different resource needs. This also plays in very well with the whole microservices movement which is one of the main reasons why Docker (or any other container technology) is at the forefront of modern microservices architectures.

That way if our app becomes popular, we can scale it by adding more containers depending on where the bottleneck lies.

Another Dockerfile example:

# start from base
FROM ubuntu:14.04
MAINTAINER Prakhar Srivastav <prakhar@prakhar.me>

# install system-wide deps for python and node
RUN apt-get -yqq update
RUN apt-get -yqq install python-pip python-dev
RUN apt-get -yqq install nodejs npm
RUN ln -s /usr/bin/nodejs /usr/bin/node

# copy our application code
ADD flask-app /opt/flask-app
WORKDIR /opt/flask-app

# fetch app specific deps
RUN npm install
RUN npm run build
RUN pip install -r requirements.txt

# expose port
EXPOSE 5000

# start app
CMD [ "python", "./app.py" ]

The yqq flag is used to suppress output and assumes "Yes" to all prompt. We also create a symbolic link for the node binary to deal with backward compatibility issues.

We then use the ADD command to copy our application into a new volume in the container - /opt/flask-app. This is where our code will reside. We also set this as our working directory, so that the following commands will be run in the context of this location. Now that our system-wide dependencies are installed, we get around to install app-specific ones. First off we tackle Node by installing the packages from npm and running the build command as defined in our package.json file. We finish the file off by installing the Python packages, exposing the port and defining the CMD to run as we did in the last section.

How can we tell one container about the other container and get them to talk to each other?

Before we talk about the features Docker provides especially to deal with such scenarios, let's see if we can figure out a way to get around the problem. Hopefully this should give you an appreciation for the specific feature that we are going to study.

Okay, so let's run docker ps and see what we have.

So we have one ES container running on 0.0.0.0:9200 port which we can directly access. If we can tell our Flask app to connect to this URL, it should be able to connect and talk to ES, right? Let's dig into our Python code and see how the connection details are defined.

es = Elasticsearch(host='es')

To make this work, we need to tell the Flask container that the ES container is running on 0.0.0.0 host (the port by default is 9200) and that should make it work, right? Unfortunately that is not correct since the IP 0.0.0.0 is the IP to access ES container from the host machine i.e. from my Mac. Another container will not be able to access this on the same IP address. Okay if not that IP, then which IP address should the ES container be accessible by?

Now is a good time to start our exploration of networking in Docker. When docker is installed, it creates three networks automatically.

docker network ls

NETWORK ID          NAME                DRIVER
075b9f628ccc        none                null
be0f7178486c        host                host
8022115322ec        bridge              bridge

The bridge network is the network in which containers are run by default. So that means that when I ran the ES container, it was running in this bridge network. To validate this, let's inspect the network:

docker network inspect bridge

[
    {
        "Name": "bridge",
        "Id": "8022115322ec80613421b0282e7ee158ec41e16f565a3e86fa53496105deb2d7",
        "Scope": "local",
        "Driver": "bridge",
        "IPAM": {
            "Driver": "default",
            "Config": [
                {
                    "Subnet": "172.17.0.0/16"
                }
            ]
        },
        "Containers": {
            "e931ab24dedc1640cddf6286d08f115a83897c88223058305460d7bd793c1947": {
                "EndpointID": "66965e83bf7171daeb8652b39590b1f8c23d066ded16522daeb0128c9c25c189",
                "MacAddress": "02:42:ac:11:00:02",
                "IPv4Address": "172.17.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        }
    }
]

You can see that our container e931ab24dedc is listed under the Containers section in the output. What we also see is the IP address this container has been allotted - 172.17.0.2. Is this the IP address that we're looking for? Let's find out by running our flask container and trying to access this IP.

Although we have figured out a way to make the containers talk to each other, there are still two problems with this approach:

  1. We would need to a add an entry into the /etc/hosts file of the Flask container so that it knows that es hostname stands for 172.17.0.2. If the IP keeps changing, manually editing this entry would be quite tedious.
  2. Since the bridge network is shared by every container by default, this method is not secure.

The good news that Docker has a great solution to this problem. It allows us to define our own networks while keeping them isolated. Let's first go ahead and create our own network:

docker network create foodtrucks
docker network ls

NETWORK ID          NAME                DRIVER
1a3386375797        foodtrucks          bridge
8022115322ec        bridge              bridge
075b9f628ccc        none                null
be0f7178486c        host                host

The network create command creates a new bridge network, which is what we need at the moment. There are other kinds of networks that you can create, and you are encouraged to read about them in the official docs.

Now that we have a network, we can launch our containers inside this network using the --net flag. Let's do that - but first, we will stop our ES container that is running in the bridge (default) network.

docker ps
docker stop e931ab24dedc
docker run -dp 9200:9200 --net foodtrucks --name es elasticsearch
docker network inspect foodtrucks

docker run -it --rm --net foodtrucks prakhar1989/foodtrucks-web bash
docker run -d --net foodtrucks -p 5000:5000 --name foodtrucks-web prakhar1989/foodtrucks-web

What is Docker Machine?

https://docs.docker.com/machine/. Create Docker hosts on your computer, on cloud providers, and inside your own data center.

What is Docker Swarm?

https://docs.docker.com/swarm/. A native clustering solution for Docker.

What is Docker Compose?

https://docs.docker.com/compose/. A tool for defining and running multi-container Docker applications. Compose is a tool that is used for defining and running multi-container Docker apps in an easy way. It provides a configuration file called docker-compose.yml that can be used to bring up an application and the suite of services it depends on with just one command.

Let's see if we can create a docker-compose.yml file for our SF-Foodtrucks app and evaluate whether Docker Compose lives up to its promise.

The first step, however, is to install Docker Compose. If you're running Windows or Mac, Docker Compose is already installed as it comes in the Docker Toolbox. Linux users can easily get their hands on Docker Compose by following the instructions on the docs. Since Compose is written in Python, you can also simply do pip install docker-compose. Test your installation with:

docker-compose version

Now that we have it installed, we can jump on the next step i.e. the Docker Compose file docker-compose.yml. The syntax for the yml is quite simple and the repo already contains the docker-compose file that we'll be using:

version: "2"
services:
  es:
    image: elasticsearch
  web:
    image: prakhar1989/foodtrucks-web
    command: python app.py
    ports:
      - "5000:5000"
    volumes:
      - .:/code

At the parent level, we define the names of our services - es and web. For each service, that Docker needs to run, we can add additional parameters out of which image is required. For es, we just refer to the elasticsearch image available on the Docker Hub. For our Flask app, we refer to the image that we built at the beginning of this section.

Via other parameters such as command and ports we provide more information about the container. The volumes parameter specifies a mount point in our web container where the code will reside. This is purely optional and is useful if you need access to logs etc. Refer to the online reference to learn more about the parameters this file supports.

Note: You must be inside the directory with the docker-compose.yml file in order to execute most Compose commands.

Great! Now the file is ready, let's see docker-compose in action. But before we start, we need to make sure the ports are free. So if you have the Flask and ES containers running, lets turn them off.

docker stop $(docker ps -q)

Now we can run docker-compose. Navigate to the food trucks directory and run docker-compose up:

docker-compose up

Head over to the IP to see your app live. That was amazing wasn't it? Just few lines of configuration and we have two Docker containers running successfully in unison. Let's stop the services and re-run in detached mode.

docker-compose up -d
docker-compose ps

docker-compose stop

docker network rm foodtrucks
docker network ls

docker-compose up -d
docker ps
docker network ls

You can see that compose went ahead and created a new network called foodtrucks_default and attached both the new services in that network so that each of these are discoverable to the other. Each container for a service joins the default network and is both reachable by other containers on that network, and discoverable by them at a hostname identical to the container name.

With Docker Compose, you can also pause your services, run a one-off command on a container and even scale the number of containers. I also recommend you checkout a few other use-cases of Docker compose. Hopefully I was able to show you how easy it is to manage multi-container environments with Compose.

How can we use Docker with AWS Elastic Container Service?

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License