ActiveBlog

No More Leaks: Detecting Memory Leaks in Cloud Applications with Plumbr and Stackato
by John Wetherill

John Wetherill, March 4, 2014

stackato-plumbr-no-leaks-in-cloud A few months ago at JavaOne I had the pleasure of meeting Nikita Salnikov-Tarnovski, founder of Plumbr, an exciting startup from Estonia. Nikita and his team have created a tool which I believe should be categorized as “essential” for cloud developers that build apps based on the JVM. This article describes Plumbr, and shows how to quickly enable Plumbr memory leak detection in any Stackato-deployed Java or JVM cloud application.

 

 

Why Avoid Leaks?

First let’s talk about leaks. Memory leaks are: 

  • unpredictable, often showing up at the worst possible time.
  • evasive, difficult to track down, and even more difficult to reproduce
  • costly, resulting in countless hours of wasted productivity, lost revenue, and decreased moral
  • damaging to reputation, brand, inter-team and intra-team relationships, revenue
  • dangerous: the wrong leak at the wrong time and place can sink a ship or a company.

So leaks are bad, but they’re also very common. According to Nikita, 47% of Java applications contain memory leaks.

We are fortunate that memory leaks, like most chronic problems in the software world, have attracted the attention of The People Who Like To Fix Things, who over the years have generously given us many ingenious tools and practices to help combat leaks.

Detecting Leaks

Tools and practices to detect and prevent leaks have been around for a long time. By necessity a cloud developer needs a selection of tools to combat leaks, and they come in many forms, including:

Static code analysers

Klocwork or JetBrain’s On-the-fly code analyser included with IntelliJ IDEA.

Platform Capabilities

Java’s garbage collection (GC) is often considered as a platform capability that can help reduce memory leaks by automating the release of unreferenced objects. But GC is not a silver bullet. In fact it can be argued that GC distances the developer from thinking about leaks, which then causes leaks. Certainly Java’s GC doesn’t prevent leaks, which is the whole reason behind tools like Plumbr.

Language Constructs and Idioms

Consider this Ruby code snippet:

File.open(“textfile”, ‘w’) {|f| f.write(“hi”) }

This ensures that the allocated resource (a file in this case) is immediately and automatically released when its reference goes out of scope. Clojure and many other languages provide similar constructs that help developers “remember” to close or free resources.

Coding conventions and practices

When coding, a good habit to get into is to type the code to close a resource immediately after opening it, before typing any code that consumes the resource.

Testing frameworks, tools and libraries

Loads of tools and libraries are available to help detect and minimize leaks. jMeter and apache bench are handy ways to stress-test an application scare out any leaks that might be lurking.

Monitoring Tools, Instrumentation Libraries

Products like New Relic, Nagios, Riemann, and facilities like JMX are invaluable for leak and performance monitoring.

Developer Tools

The popular profiler YourKit can help discover and fix leaks in Java and .NET code.

Logging

Logging, the old standby, can be helpful to help discover and fix leaks. To use logging effectively in the cloud, a log aggregation capabilities and tools like Logyard, PaaS, and Loggly are invaluable.

Assembly Required

Each of the above tools/practices takes some investment to employ, such as the time to learn, or configure, or implement, or educate, or automate the use of such tools.

Not so Plumbr. This amazing tool asks so little yet provides so much. It installs in seconds, and will monitor any Java or JVM-based app with ease. When a leak is detected it notifies the deployer, providing general or detailed information about the leak. I’ll cover steps to use Plumbr with Stackato below, but first it’s illustrative to look at the inner workings.

How does Plumbr work?

Plumbr makes use of the handy, but not widely used, java.lang.instrument package which tweaks the bytecode of compiled Java and JVM apps to enable instrumentation and measurement of various app characteristics.

Plumbr uses this package to observe the lifecycle of all of the objects created by an app, including when/where they were created, when they were freed by GC, and their memory consumption characteristics over time. But Plumbr is much more than just tracking objects and memory: they also have a sophisticated cloud-based analytics system which learns over time the signature and patterns of common leaks across large classes of applications, then uses this information to further refine their leak-recognition. In other words, Plumbr gets better over time.

After a leak is detected they then provide a detailed root-cause analysis of the leak, with specifics on how to fix it.
Nikita made it clear to me that Plumbr doesn’t show false negatives: i.e. Plumbr will detect all leaks. It might also report leaks in code where the increase in memory usage is by design (i.e., false positives) but these are usually easily recognized by the application developer. Also, as mentioned above, Plumbr is getting smarter over time so presumably the false positives are decreasing.

Leak Detection in the Cloud With Stackato

Cloud apps that scale are particularly sensitive to leaks, and therefore cloud app developers will want to use any tools at their disposal to detect and be aware of any leaks before they disrupt the application.

Fortunately this process is trivially simple with Plumbr and Stackato. Here are the steps. Please try this at home.

  1. Sign up to the Plumbr portal. Only two things are required: an email address and your choice of password.
  2. Download the agent. The important part of this is the “plumbr.jar” file, which conveniently includes your specific auto-generated Plumbr API keys.
  3. This is one of several ways that Plumbr strives to simplify life for the developer. Many other APIs and services rely on API keys, but require copying/pasting the key into a script or app to use it. With Plumbr auto-generates your key when you signup, and embeds this key in the jarfile, so the key creation is completely seamless and hidden.
     
  4. Add the jarfile from step 2 to your application directory structure.
     
  5. As with any Stackato-deployed app, create a stackato.yml file, and add this directive to enable Plumbr
    env: JAVA_OPTS: -javaagent:plumbr.jar

  6. Now deploy your app to Stackato with
    stackato push -n

    That’s all there is to it. Now the app running in Stackato is reporting back to Plumbr with its memory consumption details. To see these:Plumbr Leak Report

     

    Plumbr Dashboard

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

  7. Reload the portal page in your browser and you’ll see the reports of any leaks


Private Plumbr

Stackato is a Private PaaS, meaning it runs anywhere, whether on-premise behind a firewall, or on a public cloud like HPCloud, or on a laptop for that matter. Many enterprises choose Private PaaS for this reason: they’re unwilling or unable to push applications or sensitive data to the public cloud.

For these folks, Plumbr as described above isn’t a viable solution, since it doesn’t make sense to send memory consumption details, as cryptic as they are, to an offsite cloud service.

Fortunately Plumbr has this covered too: they offer “hosted” version of Plumbr where the entire Plumbr stack runs on premise, behind the firewall.

Summary

Cloud development tools are evolving and advancing at an unprecedented rate. One of the biggest challenges is keeping up with and understanding all the latest developments, a never-ending task that can take considerable investment in time. But it’s an investment that pays off, particularly in the case of Plumbr with Stackato, where the time investment is so low and the return is so high.

If you’re a Java or JVM cloud developer, check out how Plumbr simplifies cloud leak detection and download the Stackato micro cloud to try it yourself.

Image courtesy of megan@flickr

Subscribe to ActiveState Blogs by Email

Share this post:

Category: stackato
About the Author: RSS

John Wetherill ActiveState's Developer Evangelist. Originally from Canada, John has spent much of his career designing and building software at a handful of startups, at Sun Microsystems, NeXT Inc., and more recently in the smart grid and energy space. His biggest passion is for developer tools, or more generally any tool, language, process, or system that improves developer productivity and quality of life. Without question, Stackato is one such tool and the reason why he is here. No stranger to technology evangelism, John spent several years in the late 1990's on Sun's Technology Evangelism Team spreading the Java Gospel across the globe and focusing on the prolific number of Java technologies. Now John is now returning to his roots, as a technology evangelist working for a Canadian company, albeit remotely from Santa Cruz.