OpenSSH v7 and DSA key support. AKA “Permission denied (publickey)”

sshI have a little personal server farm with a handful of hosts that run things like my websites, a BBS and my VPN server. I recently upgraded my desktop to Kubuntu 16.04 and suddenly my SSH key was no longer working. I started seeing this when I tried to log in:

I began troubleshooting to determine what was wrong with my key. I had just upgraded my workstation, after all, I could have restored the wrong keys from my backup. That is when I discovered that I was able to log in to some of my servers, just not all of them. That puzzled me because I knew I had not touched the servers and I thought that they all used the same key. What could cause my key to work on some servers, but not others? It had to be a client-side issue but I didn’t know what.

I have a continuity plan to access my servers if anything like this should happen so I implemented it and it allowed me to look in the auth.log. I saw these messages in the server logs:

Continue reading “OpenSSH v7 and DSA key support. AKA “Permission denied (publickey)””

A primer on your phone, I mean…your website.

The world of things is grouped into three categories for me. There’s things I know, things I don’t know, and things that in order for me to understand they even exist you’d have to go back to the Big Bang to give me enough context to get a grip on. I think that most of us think that most people know what we know, or at least have enough context to get up to speed pretty quickly. Recently, however, I find myself talking to a lot of end-user website owners and I’ve come to realize that is not so. I’ve had to have many Big Bang conversations with website owners in order to explain what I felt were pretty fundamental pieces of the Internet. So, I thought I’d try to lay out the basic things that I think everyone that owns a website needs to know.

Confused

Many of the people that cross my path daily are legitimately trying to understand all the moving parts of their website; but there is a sub-community that promotes willful ignorance as well. In some circles it has become chic to be incompetent with technology. We wouldn’t dream of saying things like “I take my car to work but I have no bloody idea how to drive” or “Lawnmower? Not a clue how it works, when it runs out of gas I just throw it out and buy a new one because I have no idea where the gas goes in”. But it is somehow OK, and in fact fashionable, to say “my website? Not a clue how it works. When it stops working I just scream and yell at random people until someone fixes it”.

So here’s my attempt to help.

Continue reading “A primer on your phone, I mean…your website.”

How to securely update WordPress

WordPress has a lot of great features that allow non-technical people to (mostly) manage their own blogs. One of those features is the ability to perform WordPress upgrades and install plugins right from the admin interface. People who have trouble understanding how FTP works, or who aren’t very successful at fumbling around on a command line can make use of these features without having to become sysadmins. This obfuscation of technology is one of the reasons that WordPress has become so successful and in such widespread use.

But, as the saying goes “Security or Convenience: pick one”.

Continue reading “How to securely update WordPress”

Defeating poor port knocking configurations

I was thinking about port knocking the other day (yep, that’s how I roll) and while I consider it to be a valid security layer, it occurred to me that it would be pretty easy to set up a poor implementation of it that was susceptible to being gamed. Here’s how that thought process went.

Caveat: This is a proof of concept and has many points against it which I outline at the end of this post.

For the uninitiated, port knocking is a process whereby some port on a server can be fire-walled off until some pre-determined set of ports are ‘knocked’ on, and then the firewall can be reconfigured to open some other port. A practical example is a server where you need SSH access, but you don’t want to leave the SSH daemon running wide open to the world all the time. You can use a port knocking daemon like knockd, coupled with an IPTables firewall to protect that port. The normal configuration would be to have the SSH daemon running on some arbitrary port and have the firewall dropping connections to that port until a valid set of ports are knocked on, and then the IPTables would be rewritten, usually temporarily, to allow connections to the SSH port.

Continue reading “Defeating poor port knocking configurations”

Centralizing logs with Papertrail

Sysadmins have a love/hate relationship with logs. We spend hours and hours every day diving through them looking for clues about what happened that shouldn’t have, what didn’t happen that should have, what systems and people are actually doing, and gauging capacity for the future.

It’s one thing to look at one log for one particular issue; but some complex issues lead a merry chase through many logs or many servers which can get very complicated very fast. To ease that burden, all but the simplest of setups should employ some form of log centralization. Centralized logs are easier to access en masse and they’re easier to bring analytical tools to bear to pry out their secrets.

Continue reading “Centralizing logs with Papertrail”

Fun with Curl

Curl is one of those quintessential *nix tools that adheres beautifully to the “one tool, one task” philosophy. curl exists to give us the ability to issue requests against web servers. As sysadmins we’re usually concerned with how the web server responds to requests rather than how the actual page renders so a CLI tool like curl is quick and easy. It also lets us spoof things like user agents and referers in case we want to see how the web site responds to different browsers or different referers.

Let’s look at this site:

$ curl http://slumpedoverkeyboarddead.com | head

Continue reading “Fun with Curl”

Elections in the democratic republic of Splunk

If you can’t have fun with your technology, then throw it out and get new technology. The product I interact most with at work is Splunk. It’s very simple in some ways and very complicated in others, but underlying it all is the spirit of fun.

Watching a Splunk instance start up gives some insight into the culture at the company. Startup messages contain gems like:

  • Splunk> All batbelt. No tights.
  • Splunk> Finding your faults, just like mom.
  • Splunk> See your world. Maybe wish you hadn’t.

Or my all time favourite:

  • Splunk> Take the sh out of IT.

Continue reading “Elections in the democratic republic of Splunk”