In this new series of articles we will explore our brand new Screenly 2 Player in detail. This particular article will focus the operating system and security. These are two areas that are hard to get right, but they are essential building a sustainable digital signage platform.

To fully understand the reasoning for the decisions we have made in Screenly 2 Player, we first need to take a trip down memory lane and go back to where we started with the very first Screenly release.

How Screenly came to be (technical background)

At that time, you only had one choice as far as operating system went and that was the official Raspberry Pi Linux distribution called Raspbian (which is a fork of the popular Linux distribution Debian). Raspbian has a lot of benefits. It’s quick to get up and running with and perhaps most importantly, it has a massive community around it. As a result, if you run into any kind of problem, chances are you will find a blog post or forum post from someone who ran into the exact same issue in the past along with a solution.

Starting out, Raspbian served us well. We were able to rapidly build Screenly OSE, and get it out to the market and receive feedback. Given that it was an open source project that people ran and managed by themselves, the task of managing updates and maintaining those devices fell the user. There was no central management system at all. Things just worked and we were able to rapidly grow the install base of Screenly OSE.

After launching Screenly OSE, we started to receive customer request for a commercially backed version of Screenly. The main theme of the requests were the ability to have a central management interface and to offer support. This seemed like sensible features so we got busy building what’s now known as Screenly. Since part of the value proposition was to offer a managed solution, this meant that we had to figure out how to keep those devices up to date both in regards to security updates and for our actual software.

What we really wanted was some kind of configuration management tool that allowed us to manage these devices remotely in a central fashion. We started evaluate various options, and in the end decided on using Puppet, which at the time in 2013 was arguably the most innovative configuration management tool. Getting Puppet to work well on a low-powered device such as the early Raspberry Pis ended up being a challenge. The agent (which was written in Ruby) was insanely slow. A single run could take minutes and would consume all resources, which in turn impacted the overall performance of the playback. After a lot of system tweaks and Linux magic, we ended up getting it to a state where it worked well enough to solve the need and to this date, it is what serves updates for the Screenly 1 Player.

As we scaled up the deployment base, we started to run into other Raspbian related issues. One of the early issues we had to address was the fact that there would be breaking changes in apt packages. Since we had no way to properly pin package versions[1], we ended up having to setup our own apt mirror in order to control when updates were being rolled out. That indeed did the trick, but it created a fair amount of overhead to manage and roll out updates[2].

Another issue that we ran into with Raspbian was related to the disk images. Since we deliver Screenly in the form of a disk image, we needed to frequently build these images to ensure that the updates were applied and that the latest firmwares were in place to support the latest Raspberry Pi boards. The crux however was that until recently, the documentation for building the Raspbian disk image (the one you can download from the official source) was close to non-existing. That has now improved, and it is now possible to build you own images in a relatively straight-forward fashion, but this is a relatively recent development. Yet, what we wanted was to have these disk images to be automatically built on our CI system such that all we needed to do if we wanted to generate a new image was to push a button rather than having to involve a sequence of manual operations (which is always error prone) [3].

Lastly, we recognized that we’re not an operating system company. Building and managing a Linux operating system at the level that is required for mission critical operations such as digital signage, is not a task to take lightly. Raspbian is a great operating system for hobbyists, but when you start to get up to the scale we operate at Screenly, you start to run into its limitations. Our customers frequently deploy these devices in places where they are hard to reach (in the ceiling, public places, remote areas etc). They need to just work. We need to have built-in abilities for devices to automatically roll-back to known-good state if for whatever reason an update fails. Doing this with Raspbian would perhaps be possible, but it would require rewriting large parts of the operating system.

In 2016 when we started to the work on what is today known as the Screenly 2 Player, we knew we needed to move away from Rasbian. What we didn’t know where to move to. At the time, Rasbian was really the only widely used Linux distribution for the Raspberry Pi. So we did what geeks do: we started to spec out what out ultimate operating system would look like for this particular use case. We came up with the following criteria (heavily inspired by CoreOS):

• The operating system should be lightweight with only the absolute minimal pieces we need. This was important both because it reduces the attack vector, but also to reduce the disk footprint.
• We wanted to distribute the operating system as one big file and the application(s) as a separate one. Hence, you wouldn’t be able to get ‘partial update.’ The delivery methods for this would ideally be a binary diff delivered over HTTPS (with gpg verification or similar) to reduce the bandwidth consumption.
• We would like to have a kind of A/B scheme (much like CoreOS), with known-good states to roll back to if for whatever reason the upgrade fails (both for operating system and applications).
• It should boot blazing fast (ideally less than 5s from boot to content showing).
• The system should be fully locked down. There should be no unnecessary processes running and all services shall run at the lowest possible permission (ideally through some kind of isolation, such as Docker).
• The ability to automatically build disk images on CI.

We were reluctant to build this system ourselves, but at the time we felt that we didn’t really have a choice. We started working on a prototype based on a BusyBox environment where we got an early prototype of the Screenly 2 Player working (that met the boot criteria).

Enter Ubuntu Core

Shortly after we began this work, we heard rumors about Ubuntu Core (then known as Snappy). The design philosophy of Core aligned very well with our criteria, but perhaps the best part was that we didn’t need to develop our own operating system (something we were very reluctant to do). We decided it was our best option. We have since been working closely with Ubuntu and their engineers, and we are very happy that we did not go down this path ourselves. Building an operating system from scratch is not easy. Even with arguably some of the most talented operating system engineers, it has taken Ubuntu a significant amount of time to get to the state where we could fully run Screenly on top of Ubuntu Core. If we would have tried doing this ourselves, it could very well have starved us to (financial) death.

Let us revisit our criteria with compare it to Ubuntu Core.

Lightweight

Check. The disk image is tiny and it doesn’t include anything we do not need (well, almost at least). Currently our disk image (with Screenly installed) is a bit larger than Raspbian Lite. However, we intend to remove some additional components in the future to reduce the footprint even further.

Transactional updates

Check. Ubuntu Core does not use apt at all. Instead, it uses something called Snaps. This is used both for applications and for the operating system. Perhaps more importantly, this is fully transactional. If an update fails, the system will simply revert to a known-good state.

Blazing fast boot

Not yet. The boot is not as fast as we would like it at this point. This is however something that we are working with Ubuntu’s engineers on and we still aspire to get to that 5s boot time.

Fully locked down

Check. The system is completely locked down. Each package (snap) needs to request permission when installed. As such, even if the Screenly viewer component would be compromised for some reason, it could not mess with the operating system. Moreover, the entire content of the snap is mounted as read-only, making it very challenging to use as an attack vector.

Automatic disk image build

Check. Using the Ubuntu Core’s toolkit, we are able to build images that we can ship to customers. Moreover, we can set up multiple channels and promote versions across those channels.

Additional benefits

In addition to our the our initial criteria, we also get the following benefits:

• Commercial support.
• Automatic security updates.
• A service for distributing updates to the fleet.

All in all, the value proposition for Ubuntu Core makes a lot of sense to us. Unfortunately, it took Ubuntu a lot longer than anticipated to get Ubuntu Core ready for prime time. This played a big part in the large delays we had in shipping the Screenly 2 Player. Yet, we are still sold on the benefits for our use case.

We’re clearly just scraping on the surface of the security topic in this article as it is a massive area. If you want to take a deep dive into Ubuntu Core’s security, there’s a whitepaper available. There’s also a good overview video on Ubuntu Core and Snaps available here.

Closing thoughts

We are confident that our move to Ubuntu Core will improve the overall user experience. Security is an essential part of Screenly’s value proposition, and by piggybacking on Ubuntu Core, we are able to get much further than we would be able to do ourselves. Also, by not having to write and manage our own operating system, we can free up resources to build the best digital signage platform on the market and build value for our users.

In the following articles, we will explore other parts of the Screenly 2 Player.

The upgrade to Screenly 2 Player is a free upgrade for all our users. You can find the instructions on how to upgrade here.

Footnotes

[1] It is technically possible to pin packages using Puppet. However, if you don’t control the apt mirror, the pinned version will at some point be deleted form the mirror and upgrades will start breaking.

[2] Due to the dependency situation of apt packages, it is hard to only update one package. Instead, you are forced to updated far more packages than you would like (unless you build your own apt packages).

[3] We mitigated as much of the manual process as possible and the build process involved fetching the latest Raspbian disk image and then apply a number of bash script, which we then turned into a disk image.