- Developer Tools
Phil Whelan, August 13, 2013
To submit a talk proposal for DevOps Days Vancouver please visit
At the Agile conference in Nashville last week, I attended Brandon Burton’s DevOps talk, “Enabling DevOps - the road to a better culture”. This is a summary of that talk.
Who is Brandon Burton?
Brandon is involved in Web Operations at Mozilla and has been a part of the Los Angeles DevOps movement. He also mentioned that he is an infamous LOLcat enthusiast.
Mozilla is best known for the web-browser Firefox, but they are also involved in many things, such as the upcoming “firefoxos”. As Brandon describes Mozilla, “Mostly, we’re the open web”.
History of IT at Mozilla
Originally, when Mozilla started it was just a single small team managing their IT infrastructure. As that grew, they organized into teams. Each team has a distinct focus, but there is good overlap between teams enabling a good level of shared knowledge. Brandon describes stage with a good visual of a 3-wheeled car. “We had a lot of PHP and lots of snow-flake servers”.
Then came clusters of servers and they started to work with Puppet. They also started to work more with Python. Teams turned into more teams and they began to do a lot more automation. Continuing the car theme, things at Mozilla are now starting to look more like a Pimp-My-Ride hot-rod car. The WebOps team grew from 2 to where it is now at 12 people. There are currently 700 people at Mozilla.
The demand for new web sites and resources is now constantly growing. The list of programming languages, frameworks, databases and related technologies is growing. Apache, Django, MySQL, Memcached, RabbitMQ, Zeus are the primary tools that Mozilla websites are built with.
The culture at Mozilla is about continuous improvement and passionate people. Hiring passionate people is a necessary part.
Similar to ActiveState and many people we met related to DevOps and agile practices, Brandon’s team have been inspired by the Phoenix Project, written by Gene Kim, Kevin Behr, and George Spafford. They are now looking to use Kanban to help prioritize tasks, by incorporating it into their use of Bugzilla. Currently, this is just at the experimentation stage. The idea they are playing with is to have Bugzilla Bugs relate to Kanban cards and different states of the work flow.
A core focus of Brandon’s team is “self-service”, so that developers have the flexibility to get things done without having to request time and resources from Ops teams. This started by implementing continuous-integration (CI) with Jenkins. Anyone with an LDAP login is able to login to Jenkins and create their own jobs and run them.
They built a couple of one-click deployment tools. The first, Chief, which was an older and “bespoke” application. This started as a small project written by a developer, various small features have been added, some by devs and some by ops, but it’s never had a real owner. Because of that, a joint project called Captain Shove was started to provide a more flexible and feature rich one-click deployment pipeline. This was made up of two halves, Caption and Shove. Caption is the web interface and Shove is a daemon.
Other self-service tools that they began to use were Graphite and New Relic. In the case of New Relic, WebOps only had to deploy a service to each web server it manages and then anyone to instrument any part of their code, just by including the client libraries in their code. They can also easily send and store time-series data in Graphite by adding a small statsd library to their project.
Logstash is another such service that they are in process of building out. This is similar to Splunk. It can receive any logs files as streams and provide a better centralized management of the logs and features such as search.
The above technologies are all technologies for the IT platform. It is important for Mozilla to be able to take a project all the way from an idea to the customer as effectively as possible. The value should flow from stage to stage, from department to department. IT is the platform that supports all these stages and departments.
In keeping with their philosophy of developer self-service and efficient flow from idea to customer, Mozilla decided that PaaS was the way to go. They tried several different PaaS solutions, but decided that Stackato was the only one that was ready for real use and could be used without them having to get too deep into maintaining it. The support for multiple programming languages, multiple frameworks and multiple data-services was also appealing. They would be able to work across a broad range of technologies and this would give their developers a lot of flexibility.
Currently, Stackato is used in development, testing, staging at Mozilla. It is the earlier adopters who are jumping in a seeing the benefit, but everyone sees the promise. “Change is difficult for busy people”.
Developers like being able to do everything from the command-line and it is also important for them to have an API to integrate with. This gives them a lot of control in both their application and for automation. For some, the web console is a big plus. This can be used as an alternative to the command-line. “Not everyone is a command-line jockey. Especially Windows guys”, notes Brandon.
Brandon also mentioned they liked the ability to pass environment variables to any application, which follows what Heroku do.
Within Mozilla, there is currently a lot of DevOps brain-storming happening related to Stackato. Pushing to Stackato from Jenkins? Managing shared secrets? How do developers share API keys or database keys?
They are still finalizing details around how to move to production with Stackato. Their model for production needs to be aligned with Stackato’s workflow. Their Ops teams currently have a well established process for hardening RedHat machines, which is their basis for production. The human resources that manage and maintain this need to have to time start working with and understanding Stackato. This will probably happen over the next 6 months. Once it does, adoption of Stackato across Mozilla will move much faster. Stackato is designed to be used as a consistent environment from development through to production. If it can only be used as far as staging, then developers still hove to figure out how their applications will deploy in production, so the incentive is lost. Once they know that what are do in development will work seamlessly in production, Brandon thinks that many more developers will jump on board.
Relating to security in production, Brandon talked about the fact that Mozilla is a big target and people are constantly attempting to hack their systems. Hackers are even paid to find vulnerabilities by people wanting to gain access to their systems.
An interested concept that Brandon talked about was “PaaS Patterns”. This referred to the ability to fit existing workflows and toolsets in and around a PaaS architecture. Again, this related back to the principle of “self-service”. For instance, maybe you are already using Chef to build your application. How can you incorporate that with Stackato?
Other existing tools would be around metrics and monitoring. What would be a common PaaS pattern that developers could adopt with Stackato that would easily allow them to continue to use these tools in their workflow? For instance, Stackato has no built-in support for Graphite. They are looked at whether adding a Stackato service for this would make sense. Stackato does have out-the-box New Relic support, which is something they do use.
A few questions were asked at the end of the talk.
Q: How do developers get access to Mozilla’s Stackato PaaS, so that they can start pushing apps?
A: Access to is tied into Mozilla’s LDAP system. If you have an account on the LDAP you can log straight into the PaaS and start pushing applications.
Q: What does Stackato use for managing permissions across teams?
A: Stackato have the concept of Groups. Groups allow many users to share resources.
Q: Is Mozilla considered creating any backend services for use with Stackato apps, such as for handling push-notifications?
A: Nothing like that has come up yet. We do consider also consider 3rd party services in such situations. It’s the debate of “do you build or do you buy”?
Q: Do Development and Ops teams work together as one team?
A: I don’t think to do DevOps you need everyone under one management structure. They just need to work closely together.
I am very interested to see the progress that Mozilla makes with PaaS over the next 6 months. Once they rollout to production, adoption should move quickly. Hopefully, we will see more “PaaS patterns” emerge, as they integrate more DevOps tools into the workflow and enable further self-service for Mozilla developers. Mozilla is a very open organization, and folks like Brandon are involved in broader DevOps conversation that reaches far outside of Mozilla. Some of places you can find Brandon sharing his experience are…
You can find the slides from the talk here.
ActiveState is a sponsor of the DevOps Days Vancouver 2013 conference taking place 25th-26th October in Vancouver, BC, Canada.
DevOps Days is a self-organizing conference for DevOps practitioners that happens around the globe.
To submit a talk proposal for DevOps Days Vancouver please visit
Subscribe to ActiveState Blogs by Email
Tags: agile2013 , brandon burton , devops , IT , mozilla , ops , PaaS , self-service , solarce , webops