Centralizing logs with Papertrail

Sysadmins have a love/hate relationship with logs. We spend hours and hours every day diving through them looking for clues about what happened that shouldn’t have, what didn’t happen that should have, what systems and people are actually doing, and gauging capacity for the future.

It’s one thing to look at one log for one particular issue; but some complex issues lead a merry chase through many logs or many servers which can get very complicated very fast. To ease that burden, all but the simplest of setups should employ some form of log centralization. Centralized logs are easier to access en masse and they’re easier to bring analytical tools to bear to pry out their secrets.

My personal infrastructure is just a few servers running a few things and even I use log centralization. After looking at many offerings which were total overkill for my needs, I settled on Paper Trail and have never looked back. Here’s why:

Paper Trail is easy to set up

There are literally two steps to setting it up.

  • Update whatever logger your system uses with one line
  • restart it.

There’s an optional third step if you also want app logs and blammo…logs just start appearing in your Paper Trail dashboard like magic.

It is easy to query logs in Paper Trail

Once your logs start flowing, the dashboard (pictured in the header of this post) fills with log entries in (basically) real time. From there, you can click around to see only specific logs, or logs from a specific system. The ‘All Events’ dashboard is a good launching pad for event-based issues because all your logs, from all your systems are there in chronological order. Events that span systems or span logs end up grouped together which can be a huge troubleshooting help.

Simply clicking around isn’t going to help with the more complex problems so Paper Trail provides a boolean search syntax that allows concise queries to “find this but not that”. I admit it took a little getting used to because I am heavily dependent on regex in my daily life, but once I got a grip on it, I appreciate the beauty of boolean queries such as program:mutil sender:bbs1 "Unable to extract bundle" which looks for Fidonet bundle extraction error messages in the Mystic BBS mutil.log on my bbs1 host. Another example is program:mutil.log Results: echo -("0 echo") which alerts me when new Echo Mail comes in to my BBS. I’ve set this up as an alert (next section) because BBS’s predate email alerts by a few decades.

Paper Trail is hooked into many third party services

Beyond simply searching for stuff, Paper Trail can do things when it finds something that matches your query. It can actually do a lot of things but the two I use most are email and StatHat.

For example, I am a paranoid guy and even though I have my boxes locked down to disallow root login, I’d like to know as soon as possible if someone managed to penetrate that. This little gem looks in the logs every minute for "session opened for user root" -CRON and if it find it, it sends me an email. Note the -CRON bit; that’s to reduce false positives surrounding how the cron daemon’s activities are logged.

root-login

How about some stats with that? I send instances of this query to Stat Hat program:server_binkp "Connect:" which means I get a nice email daily from Stat Hat showing me how many Binkley mailer connections there were to my system in the last 24 hours. Again, pre-Internet BBSes don’t really tell you when they’ve broken so this daily graph gives me peace of mind that things are still humming along. It’s also shocking to find out that I get over 100 Binkley connections a day from belonging to only 5 or so Fidonet-like networks!

binkp

There are way more Paper Trail integrations than I will ever use, but since we all have unique needs there’s probably something for everyone in this list.

I have 8 alerts set up in Paper Trail and most of them fall in to the category of extending what my systems cannot do for themselves. If a service I am running can’t tell me when something happens, but it logs it, then I configure an alert in Paper Trail and call it a day. It’s become a very valuable tool in my kit.

Disclosure: I am (obviously) a Paper Trail customer, but I am on the free tier and received no consideration from Paper Trail for this post.