UNICLOUD
Docker Use at Unidata

An example of having followed these instructions can be found with these TDS and RAMADDA demonstration servers running on the Azure cloud.

Install docker-machine on your local computer. docker-machine is a command line tool that gives users the ability to create Docker VMs on your local computer or on a cloud provider such as Azure.

The following docker-machine command will create a Docker VM on Azure for running various Unidata Docker containers. Replace the environment variables with your choices. This command will take a few minutes to run (between 5 and 10 minutes). You will have to supply azure-subscription-id and azure-subscription-cert path. See the Azure docker-machine instructions, if you have questions about this process. Also set the size of the VM (e.g., Small, ExtraLarge) and supply the name of the Azure Docker host.

# Create Azure VM via docker-machine
docker-machine \
  -D create \
  -d azure \
  --azure-subscription-id=$AZURE_ID \
  --azure-subscription-cert=$AZURE_CERT \
  --azure-size=$AZURE_SIZE $AZURE_HOST

Execute the following eval command on your local computer shell environment to ensure docker commands will be run with the newly created Docker host.

# Ensure docker commands will be run with new host
eval "$(docker-machine env $AZURE_HOST)"

If you see an error message pertaining to docker-machine regenerate-certs, see the Certificate Regeneration section of the Appendix.

Mysteriously, when you ssh (see next section) into the fresh VM, you are immediately told to restart it so let's preempt that message by doing that now.

# immediately restart VM, according to Azure
docker-machine restart $AZURE_HOST
# Again, ensure docker commands will be run with new host
eval "$(docker-machine env $AZURE_HOST)"

At the very least, we will need unzip on the Azure Docker host. The Unix tree command can also be handy. docker is already installed on Azure by default.

# update and install packages
sudo apt-get -qq update
sudo apt-get -qq upgrade
sudo apt-get -qq install unzip tree

# Add ubuntu to docker group
sudo usermod -G docker ubuntu

# Restart docker service
sudo service docker restart

In Unix, when adding a user to a group, it is simply easiest to logout and log back in for this change to be recognized. Do that by exiting the VM and logging back in with docker-machine ssh command.

docker-compose is a tool for defining and running multi-container Docker applications. In our case, we will be running the LDM, TDS, TDM (THREDDS Data Manager) and RAMADDA so docker-compose is perfect for this scenario. Install docker-compose on the Azure Docker host.

You may have to update version (currently at 1.5.2).

  # Get docker-compose
  curl -L \
  https://github.com/docker/compose/releases/download/1.5.2/\
docker-compose-`uname -s`-`uname -m` > docker-compose
  sudo mv docker-compose /usr/local/bin/
  sudo chmod +x /usr/local/bin/docker-compose

We have done the preliminary legwork to tackle the next step in this process. We will now want to clone two repositories that will allow us to configure and start running the the LDM, TDS, and RAMADDA. In particular, we will be cloning:

• github.com/Unidata/Unidata-Dockerfiles
• github.com/Unidata/TdsConfig

With that background information out of the way, let's clone those repositories by creating ~/git directory where our repositories will live and issuing some git commands.

# Get the git repositories we will want to work with
mkdir -p /home/ubuntu/git
git clone https://github.com/Unidata/Unidata-Dockerfiles \
    /home/ubuntu/git/Unidata-Dockerfiles
git clone https://github.com/Unidata/TdsConfig /home/ubuntu/git/TdsConfig

Let's first display the free disk space with the df command.

df -H

Create a /data directory where the LDM can write data soft link to the /mnt directory. Also, create a /repository directory where RAMADDA data will reside.

# Set up data directories
sudo ln -s /mnt /data
sudo mkdir /mnt/ldm/
sudo chown -R ubuntu:docker /data/ldm
sudo mkdir /home/ubuntu/repository/
sudo chown -R ubuntu:docker /home/ubuntu/repository

These directories will be used by the LDM, TDS, and RAMADDA docker containers when we mount directories from the Docker host into these containers.

Note the TDM is an application that works in conjunction with the TDS. It creates indexes for GRIB data in the background, and notifies the TDS via port 8443 when data have been updated or changed. See here to learn more about the TDM.

The ADDE port 112 is for future use since we have not dockerized ADDE, yet.

When you start RAMADDA for the very first time, you must have a password.properties file in the RAMADDA home directory which is /home/ubuntu/repository/. See RAMADDA documentation for more details on setting up RAMADDA. Here is a pw.properties file to get you going. Change password below to something more secure!

# Create RAMADDA default password
echo ramadda.install.password=changeme! > \
  /home/ubuntu/repository/pw.properties

When the TDM communicates to the TDS concerning changes in data it observes with data supplied by the LDM, it will communicate via the tdm tomcat user. Edit the docker-compose.yml file and change the TDM_PW to MeIndexer. This is not as insecure as it would seem since the tdm user has few privileges. Optimally, one could change the password hash for the TDM user in the tomcat-users.xml file. Also endure TDS_HOST is pointing to the correct Azure Docker host (e.g., http://unidata-server.cloudapp.net).

As we are approaching completion, let's ensure all files in /home/ubuntu are owned by the ubuntu user in the docker group.

sudo chown -R ubuntu:docker /home/ubuntu

You are almost ready to run the whole kit and caboodle. But first pull the relevant docker images to make this easier for the subsequent docker-compose command.

# Docker pull all relavant images
docker pull unidata/ldmtds:latest
docker pull unidata/tdm:latest
docker pull unidata/tds:latest
docker pull unidata/ramadda:latest

We are now finally ready to start the LDM, TDS, TDM, RAMADDA with the following docker-compose command.

# Start up all images
docker-compose -f ~/git/Unidata-Dockerfiles/ams2016/docker-compose.yml up -d

From the shell where you started docker-machine earlier you can execute the following docker ps command to list the containers on your docker host. It should look something like the output below.

docker ps --format "table {{.ID}}\t{{.Image}}\t{{.Status}}"

If you used the configuration described herein, you will have a /data/ldm directory tree that looks something like this created by the LDM:

tree --charset=ASCII  -L 3  /data/ldm -d -I '*2015*|*2016*|current'

/data/ldm
`-- pub
    `-- native
        |-- grid
        |-- radar
        `-- satellite

5 directories

Poke around for GRIB2 data.

find /data/ldm -name *.grib2 | awk 'BEGIN { FS = "/" } ; { print $NF }' | head

RR_CONUS_13km_20151216_2200.grib2
RR_CONUS_13km_20151216_2100.grib2
RR_CONUS_13km_20151216_2000.grib2
RR_CONUS_13km_20151216_2300.grib2
GFS_Global_onedeg_20151216_1800.grib2
Level3_Composite_N0R_20151217_0000.grib2
Level3_Composite_N0R_20151217_0005.grib2
Level3_Composite_N0R_20151217_0010.grib2
Level3_Composite_N0R_20151216_2155.grib2
Level3_Composite_N0R_20151216_2315.grib2

Search for GRIB index files (gbx9). If you do not see them, see the section about a finicky TDM in the in the Appendix.

find /data/ldm -name *.gbx9 | awk 'BEGIN { FS = "/" } ; { print $NF }' | head

RR_CONUS_13km_20151216_2200.grib2.gbx9
RR_CONUS_13km_20151216_2300.grib2.gbx9
RR_CONUS_13km_20151216_2100.grib2.gbx9
RR_CONUS_13km_20151216_2000.grib2.gbx9
GFS_Global_onedeg_20151216_1800.grib2.gbx9
Level3_Composite_N0R_20151217_0005.grib2.gbx9
Level3_Composite_N0R_20151217_0000.grib2.gbx9
Level3_Composite_N0R_20151216_2205.grib2.gbx9
Level3_Composite_N0R_20151216_2315.grib2.gbx9
Level3_Composite_N0R_20151216_2330.grib2.gbx9

Verify what you have the TDS and RAMADDA running by, for example, navigating to: http://unidata-server.cloudapp.net/thredds/catalog.html and http://unidata-server.cloudapp.net:8081/repository. If you are going to RAMADDA for the first time, you will have to do some RAMADDA set up.

Another way to verify your work is run the Unidata Integrated Data Viewer.

For problems, help, questions, please submit an issue on github.

• National Science Foundation (Grant NSF-1344155)
• Microsoft "Azure for Research" program
• Tom Yoksas for Unidata operations expertise