- Developer Tools
Troy Topnik, October 30, 2012
If you're using the Stackato Sandbox, you may have already noticed some differences in the Management Console's Logs interface. They hint at bigger changes behind the scenes, coming in Stackato 2.4:
Application logs can now be aggregated by a new process called Logyard, which streams log data to a (configurable) rolling buffer. From there, it can be retrieved with an API call for viewing with the 'stackato' client or the Management Console.
System logs are similarly aggregated, but can also be forwarded to log analysis software like Splunk, or third-party services like Loggly and Papertrail.
The Problem with Log Files
In the pre-cloud days, it was OK for logs to simply be file resources. If all of your computing is happening 'locally' (ie. on a single system) it's easy to know where to look for the logs of any processes. Linux systems have conventions on where log files are stored, and usually have sane mechanisms for rotating and archiving the log files.
If you have a single system to worry about, or even a limited number, you can find and manage your log files the old-fashioned way. When you need to examine them, you use built-in system tools. When you need make more room, you delete the old ones or make the automated log rotation systems more aggressive.
As we start to move to more complex, "cloudy" systems, a number of problems present themselves:
- accessing multiple hosts becomes a hassle
- log volume becomes too large to manage
- it becomes hard to analyze all relevant log files together
A whole sub-industry has grown up to deal with these problems. Cloud logging software and services have sprung up to help sysadmins manage log analysis, aggregation, monitoring and storage.
Not wanting to re-invent the wheel, or lock our users into a Stackato-specific log analysis system, we chose instead to focus on making the data available via the Stackato API and allow people to use their preferred tools.
What Logyard Does
Logyard treats logs as streams rather than files. Treating logs as event streams makes the system event information more portable, accessible, and malleable.
Logs are streamed from all Stackato VMs into a centralized data store which keeps a rolling window of log data. The Stackato admin can configure how many lines this buffer contains. The system and application log files are all still there if you have to dig for an older event.
For end users running applications on the system, this means quick access to all the logs the application emits through one interface, command, or API call.
In the Management Console, the View Logs button shows an interleaved (by time stamp) log stream that can be filtered by source, file name, or application instance. The stackato logs command offers the same information at the command line using the same API.
The previous mechanism used the 'files' API calls, which required live application instances to fetch logs from. Not very helpful if you were trying to investigate why an application was not running. With the new system, logs are persisted -- available even if the application itself is not running.
To maintain API compatibility with Cloud Foundry (and previous Stackato releases) the stackato client will automatically revert to using the old file API when communicating with these systems. "Log Files" links for each application instance are also still available in the Management Console if you need them.
System Logs and Output Filters
Admins can view Stackato system logs with the kato tail command. Like the 'stackato logs' command, you can easily fetch log streams by process, or view them all together.
In Stackato 2.4 you'll be able to use output plugins (Beta) to forward these system-level log streams to third party services and software. There are plugins available for Loggly and Papertrail as well as the more generic 'archive' and 'tail' plugins. The README file for each plugin provides information on how to use it.
These plugins are extremely simple, and can be readily adapted to forward to other log analysis software, such as Splunk or logstash. Bear in mind that this is a Beta feature, and the plugin spec is still changing (i.e. you might have to rewrite those plugins for subsequent releases).
So please, try out the new Logs interface on the Stackato Sandbox and let us know what you think. Keep your eyes open for the Stackato 2.4 release, and have fun with the new output filters when you get your hands on a new VM.
Trackback URL for this post:
Share this post: