There has been a lot of interest over the last 12 months in products based on open source for monitoring and management. In the area of log analysis, Elasticsearch has been a player which has strengthened with the growing investments in the space. The awareness has been greatly increased in the past year. While the popular Kibana frontend to Elasticsearch has been the main GUI. These two projects are paired with Logstash for ingest, combined these make up the ELK stack. There is another great open source project to take a look at. The focus of this weeks write-up is on this alternative to ELK.
The company behind Graylog is Torch out of Hamburg Germany (https://www.torch.sh/) they do consulting around the product. The open source site is https://www.graylog2.org/ the project is an ElasticSearch based product, but unlike Kibana it also has additional features:
- Take inputs directly into the Graylog server processes
- Output from the server to multiple backends based on output plugins, right now the main one is for ElasticSearch
- Alerting based on matching or other criteria are integrated into the Graylog project along with a stream processing capability
The supported data comes in the form of plugins which include syslog or GELF (Graylog Extended Log Format) or other plugins. GELF allows for several enhancement from typical syslog.
- No length limitations for messages (syslog is 1024 bytes)
- Data types (string, number)
- Variation in syslog implementation
- Compression via gzip or zlib
The nice thing is that you don’t need to do any extractions once the messages have been added via GELF. They have 72 such plugins including many GELF libraries (See: https://www.graylog2.org/supported-sources?perPage=100)
On the site you can sign up for a self-service trial of the software, I did this in early November, there has been another release since then. These screenshots may be a little out of date:
There can be multiple backend nodes connected to the frontend. There is some good management within the GUI of the connections. The main dashboard when you login shows you information about the cluster, components, and the status. There is a query box.
Some other administrative views. Many of the log management tools, especially in open source neglect the day to day maintenance and administration. Being a systems and operations person myself I always dig into the internals needed for day to day administration. Graylog has a lot of what’s been missing across open source ElasticSearch management tools. Some additional views:
They have a data generator in the demo so you’ll see there are plenty of events in the data store.
Here is a query for smtp in the last 30 minutes.
You can also see inside the queries being sent to ElasticSearch, here are the JSON objects being passed to the engine:
Value breakdowns of the results quickly
Graylog has the notion of stream as illustrated below
What these are is a way to pass realtime rules against the data coming into the Graylog server before they are committed to elasticsearch, this real time processing provides a differentiator to Kibana based systems
Some sample sinks of what you can do with a proper eventing system, such as alerting:
The requisite dashboarding for any monitoring tool. Everyone loves dashboards, users are always asking for more dashboards, and they clearly do sell monitoring products. The value they provide are typically pretty limited. If the actual analytics in our software were better the computer would be doing the analysis versus a user looking at graphical displays of data. I digress…
You cannot share the same backend between Kibana/Logstash and Graylog since they use a different schema for the log data in ElasticSearch. Hence you’ll have to make a decision which tool you want to use when setting up ElasicSearch. Please leave comments or questions below on @jkowall on Twitter.