Elections in the democratic republic of Splunk

If you can’t have fun with your technology, then throw it out and get new technology. The product I interact most with at work is Splunk. It’s very simple in some ways and very complicated in others, but underlying it all is the spirit of fun.

Watching a Splunk instance start up gives some insight into the culture at the company. Startup messages contain gems like:

  • Splunk> All batbelt. No tights.
  • Splunk> Finding your faults, just like mom.
  • Splunk> See your world. Maybe wish you hadn’t.

Or my all time favourite:

  • Splunk> Take the sh out of IT.

For some reason it strikes me as even funnier that you can turn these messages off by setting the OFFENSIVE=less directive in the configuration. One clever Splunker asked if is an OFFENSIVE=more directive which I have not tried yet.

That’s gimme fun. Other fun, you gotta make yourself.

Newer versions of Splunk use something called search head clustering (SHC) which allows the search heads to talk directly to each other. This replaced the notoriously poor search head pooling method which used shared disk space to replicate bundles between search heads. Yuk. So. Yuk.

For SHC to work, one cluster member has to be the captain. The captain runs the show and hands out jobs to his hapless, slacking minions. The cluster members decide amongst themselves who should be captain and hold their own elections whenever they need to. Splunk is now sentient.

In a normal world, this would be fine but I am a despot dictator and cannot stand the idea of free elections. So we played with the delay settings for elections on a per-host basis and have successfully figured out how to ensure that only carefully bred hosts from the “right families” win the captain elections.

To validate this worked as expected, we set up election alerting (in Splunk!) and quickly came to realize that all was not well in the democratic republic of Splunk. The populace seems restless with their leadership and they have elections almost daily. However, thanks to our meddling in the affairs of Splunk, our carefully chosen hosts continue to win the elections and maintain their iron grip on the masses.

I fully expect to wake up one day to an alert that tells me some unintended host has been elected captain and a corresponding start up message:

  • Splunk> Viva la resistance!