Arm processors, used in Raspberry Pi’s and maybe even in a future Mac, are gaining in popularity due to their reduced cost and improved power efficiency over more traditional x86 offerings. As Arm processor adoption accelerates the need for Docker images that support both x86 and Arm will become more and more a necessity. Luckily, recent releases of Docker are capable of building images for multiple architectures. In this post I will cover one way to achieve this by combining a recent release of Gitlab (12+), k3s and the buildx plugin for Docker.

I am taking inspiration for this post from two places. First, this excellent writeup was a great help in getting things start – This post was also instrumental in getting this going –

I assume you already have a working installation of Gitlab with the container registry configured. Optionally, you can use Docker Hub but I won’t cover that in detail. Using Docker Hub involves changing the repository URL and then logging into Docker Hub. You will also need some system available capable of running k3s that is using at least Linux 4.15+. For this you can use either Ubuntu 18.04+ or CentOS 8. There may be other options but I know these two will work. The kernel version is a hard requirement and is something that caused me some headache. If I had just RTFM I could have saved myself some time. For my setup I installed k3s onto a CentOS 8 VM and then connected it to Gitlab. For information on how to setup k3s and connecting it to Gitlab please see this post.

Once you are running k3s on a system with a supported kernel you can start building multi-arch images using buildx. I have created an example project available at that you can import into Gitlab to get you started. This example project targets a runner tagged as kubernetes to perform the build. Here is a breakdown of what the .gitlab-ci.yml file is doing:

  • Installs buildx from GitHub ( as a Docker cli plugin
  • Registers qemu binaries to emulate whatever platform you request
  • Builds the images for the requested platforms
  • Pushes resulting images up to the Gitlab Docker Registry

Unlike the linked to posts I also had to add in a docker buildx inspect --bootstrap to make things work properly. Without this the new context was never active and the builds would fail.

The example .gitlab-ci.yml builds multiple architectures. You can request what architectures to build using the --platform flag. This command, docker buildx build --push --platform linux/amd64,linux/arm64,linux/arm/v7,linux/arm/v6 -t ${CI_REGISTRY_URL}:${CI_COMMIT_SHORT_SHA} . will cause images to be build for the listed architectures. If you need a list of available architectures you can target you can add docker buildx ls right before the build command to see a list of supported architectures.

Once the build has completed you can validate everything using docker manifest inspect. Most likely you will need to enable experimental features for your client. Your command will look similar to this DOCKER_CLI_EXPERIMENTAL=enabled docker manifest inspect <REGISTRY_URL>/drue/buildx-example:9ae6e4fb. Be sure to replace the path to the image with your image. Your output will look similar to this if everything worked properly:

   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 527,
         "digest": "sha256:611e6c65d9b4da5ce9f2b1cd0922f7cf8b5ef78b8f7d6d7c02f793c97251ce6b",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 527,
         "digest": "sha256:6a85417fda08d90b7e3e58630e5281a6737703651270fa59e99fdc8c50a0d2e5",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 527,
         "digest": "sha256:30c58a067e691c51e91b801348905a724c59fecead96e645693b561456c0a1a8",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 527,
         "digest": "sha256:3243e1f1e55934547d74803804fe3d595f121dd7f09b7c87053384d516c1816a",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v6"

You should see multiple architectures listed.

I hope this is enough to get you up and running building multi-arch Docker images. If you have any questions please open an issue on Github and I’ll try to get it answered.

Not too long ago I wrote about using Packer to build VM templates for Proxmox and created a Github project with the files. In the end I provided basic information on how to setup cloud-init within the Proxmox GUI. This time we’re going to dive a bit deeper into using cloud-init within Proxmox and customize it as needed.

First, lets quickly cover what cloud-init is. Cloud-init is a system for configuring an operating system on first boot. It is always used on cloud based systems like AWS, Azure, OpenStack and can be used on non-cloud based systems like Proxmox, VirtualBox or any system where you can present the info as a CD-ROM. Using cloud-init you can pass in instance meta-data information, network configuration and user information. As part of the user information you can also provide commands to be run. It is the ability to run commands on initial boot that we’re going to tap into.

Out of the box, Proxmox provides a basic cloud-init system that you can enable through the web interface that works well if all you need is to create a user with an SSH key and configure the network. But if you want to customize it you will need to ensure you have snippets enabled and visit the cli of your Proxmox system.

Continue reading

Have you ever wanted to write out a large, templated config file using only shell script code? Maybe you are working with a small IoT device with limited power or some other device and you want to avoid additional dependencies for single task. In these situations using a larger config management system tool can be too heavy or just not practical. In this post I’ll explore the envsubst utility as a way to write out a config file from a template. In the end you’ll see that envsubst is a great and lightweight utility that can be used to create config files.

Continue reading

If you work with AWS using CLI tools I highly recommend aws-vault to help keep your AWS keys secure. Be sure to visit the usage guide for full details on setup. I configured my copy to be unlocked when I am actively using my computer. It’s also a good idea to ensure your storage is encrypted.

A while back I took the time to learn a bit of OpenStack’s Disk Image Builder. Recently I decided to give Packer a try to build templates for Proxmox and I decided to release the results as a Github repo. You can find the repo at The project allows you to build a mostly empty CentOS 7 or CentOS 8 template for Proxmox. You can further customize the image by expanding the provisioner section of the packer.json files.

diagram showing how this site is hosted

A co-worker recently discovered a fun project called diagrams that allows you to create diagrams from code. Documentation and how to install diagrams is available at The image you see above was generated with some simple code. The code used to generate the graph looks like this:

from diagrams import Diagram, Cluster
from diagrams.oci.edge import Cdn
from import Nginx
from diagrams.onprem.compute import Server
from diagrams.onprem.database import Mariadb
from diagrams.onprem.inmemory import Memcached
from diagrams.onprem.client import Users

with Diagram("", show=False):
  cloudflare = Cdn("CloudFlare")
  users = Users("users")

  with Cluster("web server"):
    nginx = Nginx("nginx")
    php = Server("php")

  with Cluster("database server"):
    mariadb = Mariadb("mariadb")
    memcached = Memcached("memcached")
  users - cloudflare
  cloudflare - nginx
  nginx - php
  php - mariadb
  php - memcached

Using diagrams is an easy way to quickly create and track changes to diagrams.

RancherOS, available at, is a lightweight container operating system. It is easy to install and easy to configure but a bit light on documentation for some specific use cases. Here, I will describe how I setup RancherOS (1.5.5 as of this writing) for use with my locally installed Rancher 2.x based bare metal cluster. I will also touch on using cloud-config to configure RancherOS at boot to include the iSCSI subsystem and auto join my cluster.

I run my nodes on a Proxmox based hypervisor and have FreeNAS based storage providing NFS and iSCSI. I’m not going to cover the installation of Rancher, Proxmox or FreeNAS but just focus on basic configuration of RancherOS.

RancherOS itself is able to accept configure information using a cloud-config file. Using a cloud-config file allows you to configure a number of things during the first boot up. I take advantage of this to configure some persistent volumes, add my ssh key, enable the iSCSI subsystem and even automatically join my cluster. Here is what the file looks like, with some values removed/shortened:

# cloud-config

# create an rc.local which will cause this system to join the cluster. Replace required values for your server URL and your token
  - path: /etc/rc.local
    permissions: "0755"
    owner: root
    content: |
      if [ ! -f /opt/init-done ]; then
        docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:v2.3.5 --server <your rancher server url> --token <your rancher token> --worker --node-name $(ip ro | grep default | awk '{print $7}')
        touch /opt/init-done

  # in my setup I use iSCSI to provide block storage to pods, for this to work on RancherOS the iSCSI subsystem must be enabled
    open-iscsi: true
  # setup some local persistent storage for a few important volumes
  # this ensures Kubernetes works properly across reboots
        - /home:/home
        - /opt:/opt
        - /var/lib/kubelet:/var/lib/kubelet
        - /etc/kubernetes:/etc/kubernetes
  - <paste your ssh public key here>

For my setup I saved this file onto a web host accessible within my network. Below you will see how we tell RancherOS about the file during the setup process. You can find more configuration options at

Please note that the most important settings are the persistent mount options. You should at least use those if you plan to connect the RancherOS instance to a Rancher based Kubernetes cluster.

With the cloud-config file created we can now install RancherOS. There are a few options for installing RancherOS but for my setup I am simply using the basic iso file. For my target machine, a unibody 2008 MacBook, I had to burn the image to CDR. I booted the ISO and waited for it to finish the boot process. Once it was ready, I entered my install command:

sudo ros install -d /dev/sda -c http://<hostname>/rancheros.yaml

This command will instruct the installer to download the config file specified, save it locally (into /var/lib/rancher/conf) and then get everything ready on /dev/sda. I answer y to the reboot question and the system reboots into RancherOS. After a while the system will join your cluster and be ready for use.

That’s it. Your RancherOS node should now be ready to for use and will support iSCSI based block storage. In future posts I will try to discuss setting up other aspects of a bare metal Kubernetes cluster (where bare metal basically refers to running it anywhere but some cloud provider). If you have questions please reach out to me via Twitter.

Using iscsi on RancherOS

Once in awhile I like to read about what kind of software and utilities other people are using on their system to make their lives easier. It’s always interesting to see what mix of tools people are using and often times I learn about a new tool I hadn’t heard of before. Today I thought I’d do the same as I’ve started using a number of new tools on a regular basis just in the past six months.

As a systems engineer that is also familiar with programming I have what may be a unique mix of software and tools on my computer. Let’s take a look.

Operating System(s)

I have been using macOS full time since about 2008. I use macOS because it is a mix Unix and a GUI (NeXT if you’re keeping score) which gives me a familiar and robust command line environment with an excellent desktop environment.

I also use Linux heavily but almost never as a desktop or workstation. I have a laptop that I can dual boot between Linux and macOS for testing. I also run multiple Linux systems to run Proxmox for virtualization. Proxmox is a great way to get use out of otherwise retired computers. In fact, my Proxmox cluster is an older HP desktop with a quad core processor mixed with a pair of old MacBooks. I have written about Proxmox before and you can find it here.

I have one Windows PC that exists mostly because of games but also some business software.

Software Tools

When it comes to software these are the tools I use most frequently.

  • Code Editing and Runtimes/Languages
  • DevOps Type Stuff
  • Kuberenetes
    • kubectx/kubens for easy cluster and namespace switching
    • k9s for a text based UI to Kubernetes
  • Utilities
    • Brew
    • Patterns tool for working with regular expressions. Been using it for years but several tools now exist like it
    • iTerm 2 superior to the default terminal available in macOS
  • Other
    • Spotify for music
    • VirtualBox for testing Ansible roles
    • Twitter client
    • RamBox for chat
    • Bear for notes

Quick list of software tools that I find make using Kubernetes even better. I consider these tools must haves.

In 2004 I took delivery of a new car that was equipped with a CD changer. Until then I had only ever had cars with a single disc player so stepping up to a deck with a six disc changer was incredible. No longer did I need to keep a sleeve of CDs in the car that would get scratched or lost, I could just keep what I was listening to at the time right in the deck. It was still a time where creating and burning playlists to a burnable CD was totally acceptable and all was well in the world.

I kept that car for about ten years and during that time we saw the iPod and other music players gain tremendous popularity. And why wouldn’t they? You could put as much music onto the device as it would hold and carry it around with you anywhere you went! The original iPod even had this slick wheel based interface for getting around quickly and easily. Amazing! Unfortunately, when my car was designed, these types of players weren’t common yet and there was no way to interface an iPod or any type of music player with my deck because I didn’t have an AUX input. Bluetooth connectivity was even less of a thing at the time so that option was out as well.

So I kept on making CDs of the music I wanted to listen to and feeding them into the changer knowing that some day I would sell the car and pick one up that had a Bluetooth interface. I thought, one day I’ll finally be able to listen to all of my music at anytime and it’ll be great.

Well, as it turns out it wasn’t all rainbows and unicorns.

A few years ago I picked up a newer Mazda, one with Bluetooth and iPod connectivity, an AUX port and even Pandora! The possibilities before me seemed perfect and I got to working figuring out which method would work best for me. After much tinkering I settled on using Bluetooth because it offered wireless connectivity and worked with whatever music app I wanted to use. I loaded up Spotify with downloaded music and that was that.

After awhile though the flaws in this new system started to appear. I discovered that Mazda’s Bluetooth implementation was less than ideal. It takes a lot of time for it to connect to my phone and start playing music, sometimes over a minute. I can no longer just hop in the car and have it resume where it left off just moments after starting the car. Other times it connects but can’t tell me what is playing or just refuses to play anything at all until I visit Spotify and select something from there.

And herein lies the primary issue and why I miss the venerable CD changer. It isn’t because Mazda’s Bluetooth implementation is bad (and it is really bad), it’s that the process of selecting music is so much more involved. To select music, I have to get my phone, unlock it, open the Spotify app and go digging for the playlist or album I want…while driving. It turns out that having a large selection of music requires changes to how you interface and interact with that music. It requires that you look at a screen to scroll and make selections. All of these interactions are fine when you can spend the time doing them but in the car speeding down the highway is not the right time.

So why is the CD changer the better option here? It’s because interacting with a CD changer is fundamentally different than a music app on your phone, even if your vehicle has a stellar deck and you are able to interact with the music app using steering wheel controls or the touch screen you still need to look at a screen to know where you are. Not so with a CD changer. You put six discs into a changer and you know what slot they are in. You know, using your ears, which track you are listening, which CD it is on and from that you know what slot is it in. If you want to listen to a different disc you know how many times to press the disc change button. Listening to Taylor Swift on disc 1 and now you want to listen to the third song on your new Metallica album in slot 3? Press the disc change button twice and then press the next track button a couple of times. Done and you didn’t even have to take your hands off the steering wheel. Such an interaction isn’t an option anymore. In a music app the interface is 100% on the device, with a CD changer half the interface is in your head.

In the end, it isn’t really the CD changer I miss. It’s actually the “interface” that CD changers provided. There is no equivalent, that I’m aware of, in today’s music apps that emulates the CD changer interface. I believe the ideal solution would be to allow a user to configure a set of playlists as “slots” like a CD changer that are in a locked order. Controls are then offered on screen and the steering wheel to switch between playlists in a locked order, just like a CD changer.

I like the progress that has been made with technology. I appreciate being able to put more music than I could possibly listen to in a year in my pocket. I just wish this progress didn’t come at the expense of usability. Burning a CD was a hassle but when it was done it was done but interacting with your player happens every time.