Filter results by

Containers for IoT-Scale Workloads: Part 2

Last week we introduced some of the challenges in providing a solid, secure and agile IT environment for a modern IoT service like SAMI. Today we’ll explore how we addressed these issues in our transition to Mesos and Docker.

Read Part 1 of this article.

From the get-go, we decided to build an automation pipeline, which would enable us to do the following:

  • Build a fault-tolerant, self-healing infrastructure.
  • Use a modern cluster-management/distributed-init system, which ensures the defined app replicas are always running.
  • Use Git as single source of truth: all job configurations will be stored in Git, which enables us to easily build a change-management/audit system.
  • Package apps as Docker images for faster deploys.
  • Build a fast and reactive orchestration system.
  • Wire our existing infrastructure into the pipeline.
  • Finally, put together a Continuous Delivery platform that is fast and robust enough to allow engineering (including QA) to perform production deployments! This was our mission—to free ourselves from daily operational overhead and complexities, and focus instead on developing new services and improving processes, thus adding more value to the organization.

With the above goals in mind, we came up with the following design:

  • A Highly Available (HA) Mesos cluster will provide abstraction at the datacenter (DC) level, just like a typical OS will provide abstraction at a workstation level. DC is the new form factor.
  • Mesos will act as a DC-wide resource manager.
  • Build system will start packaging applications as Docker images. These images would then be pushed to an internal Docker Registry.
  • Marathon will act as a DC-wide init system/process manager. All long-running jobs are deployed and managed via Marathon.
  • Chronos will act as a DC-wide cron system. All short-lived/batch jobs will be scheduled and managed via Chronos.
  • All Marathon and Chronos job configurations will be checked into Git. Any changes to infrastructure will then be traceable.
  • Git2Consul will be used to sync Git repositories to Consul’s KV store, and Consul handlers watching for changes to KV will post the updates to Marathon/Chronos via REST API.
  • Consul will continue to be our Service Registry. Use Registrator to listen for Docker events and register changes to Consul. This will eventually be replaced by Mesos-DNS.

Our workflow now roughly follows the steps below:

Containers workflow

  1. Upon a developer commit, a Git Post-Receive Hook kicks off a Jenkins build.
  2. Jenkins will employ maven-docker plugin to create, tag and push Docker images to the private Docker Registry. It also updates the image tag in a Consul’s KV.
  3. A Consul Handler watches for changes to this KV and updates the Git repo containing Marathon job configurations.
  4. Gi2Consul picks up these changes and syncs the repo to another KV in Consul.
  5. A second handler watching this KV will post the job configurations to Marathon. If the service is not already registered, it’s created. Else it is updated.
  6. Marathon acts as a cluster-manager and a distributed init system.
  7. A small service called Registrator listens for Docker events and updates Consul’s service catalog.
  8. Any changes to service registry are picked up by Consul-Template, which refreshes all configurations (most notably HAProxy) and reloads dependent services.

As you can see, without any manual intervention, applications are automatically built, packaged and deployed to various environments. And if you look more closely, you will see an audit trail: each step is recorded in Git. This is extremely important, especially when you have a large team and a complex stack with many moving parts. Add to this the requirement for anyone in engineering to do multiple releases of multiple services in a single day, and you can see why a thorough record of deployments is critical!

Notifications are now sent to teams instantaneously (via emails, chatrooms, etc.), informing them of success/failures of their jobs. This results in a faster feedback loop, completely eliminating Ops from the equation!

With Mesos, we are able to run our infrastructure more efficiently, keeping our cloud expenses predictable. We are able to achieve high resource utilization, and thanks to resource isolation, we are able to pack multiple services (running in Docker containers) onto a single host, creating a truly multi-tenant deployment, where applications can co-exist with backends like Cassandra and Kafka.

Looking back at our implementation

When all was said and done, it only took us about three months to go from inception to implementation to production! Yes, we move FAST! But there were many challenges and hurdles along the way, both from a management and technical perspective.

A new user needs to understand many new concepts and nuances in the Mesos/Docker world. It takes some effort to break away from the traditional way of thinking about applications and hosts and datacenters. This required some internal evangelism to get various different groups to buy into this new concept.

From a technical standpoint, the ecosystem around containers is still not fully mature, although there is tremendous work happening in this space. As such, we took this into account when designing the system, ensuring that organizational parts of the pipeline can change. It’s important to design your infrastructure in a modular fashion, allowing you to easily swap tools and technologies in and out as needed.


Given that modern applications are comprised of many microservices and backends, the way we approach application deployment needed a rework. We believe the new container-based technology, along with a cluster-management service like Mesos, fits this use case perfectly. Containers enable us to run services at scale and gracefully handle all the inter-dependencies. They also enable us to employ complex deployment strategies involving hybrid clouds.

With the new pipeline I’ve described above, we can ship code faster, scale easily, achieve multi-tenancy thanks to true resource isolation and ensure optimum resource utilization. We are able to achieve close to 80-percent resource utilization during heavy workloads in some of our pre-production stacks. This is incredible, given that the industry-standard datacenter resource utilization is around 8 percent!

As an example, this is how the resource utilization of one of our Mesos clusters in a pre-production stack under heavy load looks:


Overall, the SAMI team is very excited about this new paradigm shift. But it involved plenty of research, learning and hard work to get us this far. For those of you who plan to go down this path, keep in mind the new system will require changes that are different from the way you’ve done things in the past. But we’re excited about the results so far, and we believe this change is taking us in the right direction!

Read Part 1 of this article.

Top image: Louis Vest

Get the ARTIK Newsletter

You like your news fresh! Sign up now and you will be the first to know about our latest software releases, coding tips, upcoming events, blog posts, datasheet updates, and more.

* By checking either box, you may receive notifications by phone, email, text, and/or other electronic means from Samsung Semiconductor, Inc. and its affiliates. If you choose to receive partner notifications, we may forward your contact information to our partners. You may unsubscribe from these services at any time by clicking on the unsubscribe link in our communications or by submitting a request here. Please see our Privacy Policy and Terms of Use for more information about how your data is stored and used.