In this post, we’re going to look at how to get the CoreOS Clair container security tool running from the command line, with a view to integrating it into a CI/CD workflow using the Klar CLI tool.

The way most people are using Docker is as a “better VM” in that they’re taking Ubuntu LTS releases and deploying on top of them. This is as opposed to using a “stripped down”/minimal image like Alpine Linux. The advantage of the minimal images is that they often remove a whole bunch of software that’s not needed in a container, reducing the potential attack surface and making the images smaller and easier to work with.

A few of the thorny issues which are not always considered in this scenario is how to upgrade/patch the “OS” (libraries etc…) running in Docker and how to check whether a Docker image is missing patches.

To solve this, CoreOS released a tool called Clair which allows you to “scan” Docker images in order to ensure that all patches/upgrades have been applied.

tldr;

Install CoreOS Clair onto Minikube using the Helm chart. Ignore the PostgeSQL part of the chart as the PV doesn’t work with Minikube. Ensure it can talk to DB.

Expose Clair to the cluster. Download and install Klar, run it, pointing at the Clair service and pass it an image to scan (needs docker locally?).

Ignore v1 API error message from Clair logs

Installing CoreOS Clair onto Minikube

Installing Clair onto Minikube is fairly straight forward thanks to Clair providing a Helm chart in GitHub, which you can find here. Personally, I found an issue with the “PostgreSQL” container within this helm chart, which caused it to fail to start, due to something to do with PhysicalVolumes:

Warning Failed 12h (x675 over 2d) kubelet, minikube Error: lstat /tmp/hostpath-provisioner/pvc-16c78bd3-12c1-11e8-8c48-080027fa3e9c: no such file or directory
Normal SuccessfulMountVolume 9m kubelet, minikube MountVolume.SetUp succeeded for volume "pvc-16c78bd3-12c1-11e8-8c48-080027fa3e9c"
Normal SuccessfulMountVolume 9m kubelet, minikube MountVolume.SetUp succeeded for volume "default-token-5v2tq"
Warning Failed 7m (x12 over 9m) kubelet, minikube Error: lstat /tmp/hostpath-provisioner/pvc-16c78bd3-12c1-11e8-8c48-080027fa3e9c: no such file or directory

Therefore, I just ran the “vanilla” PostgreSQL image and exposed it as a service with:

kubectl run postgres --image postgres
kubectl expose deployment postgres --port 5432 --target-port 5432 --type NodePort

and pointed Clair at the PostgreSQL instance using this bit of configuration in the custom values Helm YAML:

postgresURI: "postgres://postgres@postgres:5432/postgres?sslmode=disable"

(the full file can be found here)

Next up, you can install Clair itself from the Helm chart with our custom values:

helm dependency update clair
helm install clair -f mycustomvalues.yaml

NOTE: This assumes you’ve got Helm and Tiller installed and setup (have run helm init etc…)

Assuming it’s all gone well, you should see the following output when running “kubectl get pods”:

and the Clair pod logs should look something like the following:

Now you need to grab the port that the Clair service is running on in our Minikube cluster with the “kubectl get services” command:

The value we’re after is the port mapped onto 6060 (in the above screenshot 32527).

Assuming you’ve installed Klar (instructions here) you can now run Klar against a test image with the following command (update port and Minikube ip for you config):

CLAIR_ADDR=http://192.168.99.100:32527 CLAIR_OUTPUT=High CLAIR_THRESHOLD=10 klar postgres:9.5.1

The output currently will likely be “Found 0 vulnerabilities” which is a bit of an issue (it should be > 0). The reason for this false report is that the Clair “updaters” take a while to download the full list of vulnerabilities and lists from the Ubuntu/Debian/Alpine etc… databases. The solution is to wait a few minutes for the updater to get the full list of vulnerabilities (can tail the logs for the “vuln db updated” or whatever message)

Integrating into Continuous Integration workflow

So, at this point, you can automate the above to fail our build if the newly built/pushed image has an unacceptable level of vulnerabilities. The Klar command exit value is determined by whether the vulnerabilities exceeded the configured thresholds (0 = all good, 1 = fail). Therefore, it’s trivial to integrate into the build workflow.

You do need to sit down and work out where to set the thresholds at (i.e. what constitutes an unacceptable level of risk).

Conclusion and Further work

It would have been great if Klar supported scanning images locally (before they were pushed to a registry) as it would somewhat further reduce the risk that an “insecure” image might be deployed.

Also, we didn’t cover any alerting or notifications when Clair does find security issues in an image (i.e. who should be alerted? what should they do?).

Finally, there was no discussion of scanning already deployed images or any kind of periodic “background” scanning of a deployed environment.

For other options for Clair integration instead of Klar, have a look at the Clair integrations page.