Implementing eBay's Jenkins + Mesos + Docker Design

I attended the 2014 Jenkins User Conference in San Francisco primarily to get some face-to-face time with some Jenkins gurus, and I was hopeful that it would help me to come up with some ideas as to how to solve some real problems my team has encountered in rolling out our instance across the organization.

Our cluster had become quite unwieldy and difficult to maintain. We had too many servers, too many vm's, too many Linux Containers to maintain, etc...

I started looking into solutions to replace/supplant the LXC containers I had been maintaining, but was thinking that I would eventually wind up running into a lot of the same problems I was already dealing with:

• How do I manage this enormous fleet of containers?
• How do I know how many containers I need for each physical host?
• How well does this scale when things get bursty?
• How can I treat a job, or pipeline as higher priority than the rest?
• How can I avoid having to write a bunch of specialized Nagios stuff to keep the systems up and running.

In a nutshell: How do I take some of the Ops out of my regular DevOps duties, and keep my phone from making the horrible-work-noise?

Enter the dudes from eBay.

They talked of generating Jenkins Masters and Slaves on demand from a cloud of resources using Mesos, Marathon, Zookeeper, and Docker (see Part 1 and Part 2 of their blog on the subject).

As an aside, it was a serendipitous accident that I happened to have been in their preso at all, because I was actually looking for another demo that KK and company were putting on at the time.

At any rate, their solution pretty much exactly fit the problem I was trying to solve:

Racks of expensive servers sitting idle is a very bad thing.

So I brought this back with me, and began talking to my colleagues about how we could make use of it for our particular needs.

We didn't really need to spawn entire instances on demand (yet), and our containers are totally disposable, so we sluiced the parts from the eBay model that we needed, and came up with a workable solution in which produces on-demand Jenkins Slaves, each running on a Docker container running on a Mesos Slave.

Using the official Mesosphere packages, we wound up having to recompile the plugin to work correctly.

Some issues we have currently which I haven't had the chance to root out are:

• It takes a really long time for the slaves to be created. My colleague has found some stuff on this, and it appears that this is a known issue and is being worked on. UPDATE: This is greatly improved in recent versions of the Jenkins Mesos Plugin as of 4-16-2015
• We are periodically seeing an issue where slaves just seems to disappear (they are reported as being LOST in the Mesos Master Dashboard).  The symptoms look like this in the Jenkins console log:
  

  FATAL: channel is already closed
  hudson.remoting.ChannelClosedException: channel is already closed