Building dashboards with splunk, twig and bootstrap

There won’t be much code with this one because it was an internal project, but it’s been interesting enough that I wanted to do a post. We’re using a monitoring package called AppManager by Zoho corp. It uses a hub-and-spoke architecture with remote “managed servers” rolling data up to a “management server”. Monitors are configured on “managed servers” to monitor systems at their local site. Individual monitors are placed into groups, and those groups can be further grouped, so that alarm states bubble up through the management groups.

My challenge was to create a simple stoplight dashboard showing the ships and the status of various monitor groups. Management wanted it to look like the Twillio service status dashboard. The software has a decent REST API returning JSON or XML. The initial dashboard was a snap, crawl the JSON returned and populate an HTML table. This was so easy that I decided it was time to learn Twig.

Twig is actually pretty slick, and I can easily see the benefits of using a template engine. I re-worked my code to populate an array of data, and passed that data into the twig render function. Twig lets you nest templates inside other templates, and pass data down to the “child”. It’s pretty slick, and I think going forward all my future work will be run through it.

I also wanted to play with Bootstrap, since my UIs are usually pretty bad. Bootstrap is super easy, looks great and is well documented. I’ve officially said goodbye to jQuery UI and hello to bootstrap.

The problem with AppManager is that the management server doesn’t keep monitor data, only monitor status. The other problem is that their dashboards aren’t very pretty. We already have a Splunk installation, so I figured this was a good time to play with Splunk.

I installed nodeJS on every ship, along with the splunk forwarder. At regular intervals the nodeJS script runs against the local AppManager server, gets details about specific monitors (anything in a group called ‘splunkforward’) and writes the JSON data out to the filesystem. The Splunk universal forwarder then pics up those files and sends them to the indexer. Splunk parses JSON data pretty well, and we have a Splunk wizard on site to help carve it up. Splunk also has great graphing features which did a lot of my work for me.

Finally, I wanted to get the monitor details back into my dashboard. Splunk has a PHP SDK which lets you easily retrieve saved queries and re-execute them. The query returns a job ID, and you can either go into a polling loop checking it’s status, or just execute it in blocking mode. Since the data I fed into Splunk initially was just JSON data, this is what I get back out. Those JSON documents can then be parsed with json_decode.

The Splunk bit was also really exciting to me. The alternative would be to parse the data and write it out to some RDBMS and query it out with SQL. I’m learning the data as I work through the project, am changing it, adding fields on the fly, and dealing with some differences in the JSON layout from AppManager (based on monitor type). Using Splunk has freed me from having to battle with SQL. I just feed it JSON files, query them out later and re-parse them before feeding them back into Twig. Fun!

This whole thing might seem overly complex, but consider that it’s expanded out for 24 ships, and each ship is connected with a high latency, low bandwidth satellite link which will occasionally fail. Splunk provided guaranteed delivery of data, and a convenient way to store and access it.

The finished product!

The finished product!

 

The path for getting AppManager data from the ship to HQ and finally a dashboard.

The path for getting AppManager data from the ship to HQ and finally a dashboard.

 

Showing a higher level and to ships sending data. In reality it's 24.

Showing a higher level and to ships sending data. In reality it’s 24.

Advertisements

About robertlabrie
DevOps Engineer at The Network Inc in metro Atlanta. Too many interests to list here, check out my posts, or look me up on LinkedIn

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: