- Developer Tools
John Wetherill, July 22, 2014
Years ago a colleague of mine at Sun had an annoying habit of interrupting technical discussions at engineering meetings with the phrase “We're in the weeds!”
It was annoying because he would often disrupt fascinating and mind-expanding discussions on, say, the structure of TCP packets, or on message bus implementations. These were always interesting debates that we were reluctant to stop.
Years later I came to realize the wisdom of his visionary approach of focusing on the big picture and ensuring that we didn't get bogged down in implementation details, premature optimization, and other impediments to getting the product out the door. I soon found myself saying “Get out of the weeds” at engineering meetings, yet I was continuously hampered due to the overall complexity of software development. As much as I wanted to untangle myself from the weeds, I was unable to do so.
Fast-forward a couple of decades to the introduction of PaaS, which finally provides the capabilities needed to eliminate much of the complexity of software delivery, and truly Get out of the Weeds.
This blog collects a handful of situations I’ve encountered over the years where having access to a PaaS would have saved tremendous amounts of time, effort, complexity, cost, and overall stress for all involved.
Implement SSO across multiple applications
At one company, a coworker was tasked with implementing Single Sign-on (SSO) across a number of internal applications. He dove into this job with gusto, first researching the current state of SSO solutions available, building a handful of prototypes, finally choosing Spring Security and a few other frameworks as the basis, and went to work. Three months, and countless thousands of lines of extra code later he finally shipped it. While it was mostly working, “mostly working” doesn’t cut it in the security world.
Compare this to the effort involved to enable SSO across apps in Stackato: click a checkbox. This takes all of 3 seconds instead of 3 months, which is...15 million percent faster! Then there’s the reduction in code complexity, test suites, external application changes.
And, it actually works.
Onboarding a new engineer can be a nightmare. I’ve witnessed, and worked at, companies where bringing an engineer up to speed can take weeks. Much of this time is taken setting up, providing access to, and understanding complex development and deployment environments consisting of multiple servers, services, languages, runtimes, frameworks, containers, etc.. Then there’s the complexity of the processes involved in requesting resources, running test suites, interacting with other teams, etc.
How many times have you seen an engineer go through the long and sometimes painful onboarding process, only to jump ship immediately after? That sure doesn’t come cheap.
Most of this agony, cost, and time can be eliminated by using a Private PaaS like Stackato to set up the developer environments. I’m a big fan of the “micro cloud,” a facility Stackato provides allowing the entire cloud application suite, including all services and dependencies, to be provisioned on a single developer laptop quickly. New developers can access a fully-configured stack, on their own laptop, in hours, or even minutes after first logging in. A developer would be capable of deploying, running and testing large complex application stacks before they even learn where the nearest break room is.
I hate to think of all the time I’ve wasted waiting for access to a database instance. At first I thought this was an anomaly of one team I was working on, but then encountered the same lag time at the next team. And the next company. More recently, talking to engineering teams at a wide range of large and small organizations, I’ve discovered it’s still all too common to have to wait days or more for a simple db instance to be provisioned.
This lag time has several negative effects. First, it disrupts the flow. When I request a service instance for a problem I’m working on, all my focus and creativity is focused on this problem. But after having to wait days to get the service provisioned, my focus is almost certainly elsewhere and it takes effort to change gears again back to the old flow.
Another negative effect of this delay is that it often causes disharmony between the IT and developer teams. This disharmony is a common theme at devops conferences, and is extremely costly. This is where Stackato shines: it allows the IT team to deliver a reliable, scalable, self-service, on-demand developer platform with very little intervention required.
Now, instead of submitting a ticket and waiting, a developer needing a new database instance simply clicks a button, and is able to make use of the new service in minutes. This enables creativity, experimentation, and more efficient flow, while also greatly improving the harmony across the IT and developer teams.
Differences in configuration across developer machines can be incredibly costly. One time, inconsistency in a configuration file came close to sinking the company.
Without getting into too many details I’ll just say that the massive amounts of data involved, the intermittency of the problem, and the complexity of the application, resulted in the ship date slipping by 3 weeks, during which several stressed out developers were working around the clock, poring over over 4 page printouts of crazy SQL queries, parsing hundreds of thousand of lines of logs (logs - don’t get me started!), following misdirection after misdirection, and trying in vain to reproduce the problem. This three week slip date almost cost us our biggest customer, and probably shaved five years off my overall lifespan.
At the end of the day it turned out the problem was caused by an inconsistency between a configuration file on QA and in production. On QA a service was referenced by IP address in QA, and by hostname on the customer site. The DNS round-robining was occasionally resulting in a stale database being referenced.
I’ve seen countless other examples like this, with similar results. A backslash instead of a forward-slash in a config file on one developer system takes a whole morning to fix. Or a trailing space in a property file in QA results in a two-day slip.
The point here is that if QA, dev, and production had been using Stackato as the basis, configuration between the environments is minimized, as almost all aspects of the configuration is in common, whether running on a developer laptop, or on multiple racks spanning data centers.
Of course not all configuration is identical between various environments, but when using a PaaS as a foundation, most differences go away, minimizing the potential for issues caused by the differences.
Log Files - Don’t get me Started!
Too late, I already started. As mentioned, the above scenario resulted in countless hours poring over log files, most of them from Spring, where a single exception could easily generate over a hundred log lines. That didn’t include the up-front effort coordinating these logs in the first place, fiddling with tomcat’s notion of rotation, dealing with multiple agents capturing and forwarding log messages, ensuring the agents always stay alive, making sure disks don’t fill up.
In short, dealing with logs is a pain in the butt. With client/server architectures, multiple log files, inconsistent log message formats, disparate servers, disk capacities constraints, log rotation, notification requirements, time sync issues, correlation challenges, multiple user and system tasks, multiple interacting apps and processes are coming and going across servers, data centers, and continents, and more, it’s a wonder we’re able to deal with logs at all.
But, much of this pain goes away when using Stackato to manage logs. Like the SSO example above, Stackato allows configuration of log aggregation with a single operation or command. All application logs, from multiple app instances running across multiple availability zones, can be easily captured and redirected to tools or apps that are highly capable of dealing with them, like Loggly or Splunk.
Get out of the Weeds
I could go on with examples of situations where Stackato would have saved countless time in scaling, zero-downtime upgrades, security, and countless general development activities.
The point is, if you find that you or your team is mired in implementation details, constantly optimizing and re-optimizing, and basically tangled up, I recommend you take Stackato for a spin. Chances are you’ll find that it will also allow you to Get Out of the Weeds.
Share this post:
Image courtesy of Michael Wilfall
Learn more about Stackato. Find out how our private PaaS has helped organizations reduce deployment time from weeks to minutes, and manage their cloud applications more efficiently. If you would like to get hands-on experience with Stackato, you can get Stackato by downloading the free micro cloud, requesting a customized demonstration or signing up for a POC.
Subscribe to ActiveState Blogs by Email
Share this post:
Tags: consistent environment, logging, PaaS, platform-as-a-service, security, service provisioning, sso, stackato