If you can't have fun with your technology, then throw it out and get new technology. The product I interact most with at work is Splunk. It's very simple in some ways and very complicated in others, but underlying it all is the spirit of fun.

Watching a Splunk instance start up gives some insight into the culture at the company. Startup messages contain gems like:

• Splunk> All batbelt. No tights.
• Splunk> Finding your faults, just like mom.
• Splunk> See your world. Maybe wish you hadn't.

Or my all time favourite:

• Splunk> Take the sh out of IT.

For some reason it strikes me as even funnier that you can turn these messages off by setting the OFFENSIVE=less directive in the configuration. One clever Splunker asked if is an OFFENSIVE=more directive which I have not tried yet.

That's gimme fun. Other fun, you gotta make yourself.

Newer versions of Splunk use something called search head clustering (SHC) which allows the search heads to talk directly to each other. This replaced the notoriously poor search head pooling method which used shared disk space to replicate bundles between search heads. Yuk. So. Yuk.

For SHC to work, one cluster member has to be the captain. The captain runs the show and hands out jobs to his hapless, slacking minions. The cluster members decide amongst themselves who should be captain and hold their own elections whenever they need to. Splunk is now sentient.

In a normal world, this would be fine but I am a despot dictator and cannot stand the idea of free elections. So we played with the delay settings for elections on a per-host basis and have successfully figured out how to ensure that only carefully bred hosts from the "right families" win the captain elections.

To validate this worked as expected, we set up election alerting (in Splunk!) and quickly came to realize that all was not well in the democratic republic of Splunk. The populace seems restless with their leadership and they have elections almost daily. However, thanks to our meddling in the affairs of Splunk, our carefully chosen hosts continue to win the elections and maintain their iron grip on the masses.

I fully expect to wake up one day to an alert that tells me some unintended host has been elected captain and a corresponding start up message:

• Splunk> Viva la resistance!

Today I ran across what is likely the most elemental example of logic ever devised. Aristotle is credited with its earliest forms and I am therefore sure that it is taught to every first year philosophy student on the planet; but that does not take away its simplistic beauty.

It is a syllogism.

Much like the best scientific theories are simple, the best examples of logic are simple. So here we go.

A syllogism is an example of using deductive reasoning to reach a conclusion.

Deductive reasoning starts from the general to produce claims about the specific. Inductive reasoning starts from specific observations to derive a general claim.

The syllogistic form is two or more premises that, if true, must mean the conclusion is true. For example:

All men are human
Jon is a man
Therefore, Jon is human

As with all simple things, it is only simple because the hard work has already been done by other people. There are actually two complicated things going on in this tiny piece of prose that make it a valid example of logic.

The two premises have a function: the major premise is the premise "All men are human". It is major because it states the general claim about all men. The second premise "Jon is a man" is the minor premise. It is minor because it states the specific case, about Jon in particular, and how it relates to the major premise. In order to logically relate these two things, the minor premise must be shown to be related to the major premise. The reason for that follows.

The second thing that is going is called "distribution of the middle term". The middle term is the thing that joins the major and minor premise. It's easy to pick out because it appears (in some form) in both the major and minor premise. In this case, the middle term is man/men.

Distribution of the term "man" means that the major premise makes a claim about all members of that category. In this case, "All men are human". And since the minor premise has been linked to the major premise via the middle term, logic dictates that the minor premise takes on all the characteristics that have been claimed for the major premise.

If all those criteria are present, you have a logical argument. A major premise makes a statement about all members of population (distributed) and the minor statement claims the specific case is included in the population of the major premise (middle term), so therefore the conclusion must be true.

That's a nice text book answer, but let's look at some examples when things go wrong.

What happens if the major premise does not distribute the middle term?

Some men are human
Jon is a man
Therefore, Jon is human

This is a logic error

Since the middle term is not distributed (ie, only 'some' men are human, not 'all' men) it becomes obvious the conclusion is now suspect. Jon may or may not be human in this case.

What happens if the middle term is omitted?

All men are human
Jon is tall
Therefore, Jon is human

This is a logic error

Again, it becomes very obvious how the conclusion becomes invalid if the rules of logic are not followed. The lack of a middle term means the two premises are not tied together in any way, so the required relationship fails.

My final example deals with the distinction between logic and truth. It is entirely possible to construct a valid logical argument that is not true. Using the same model, it may look like this:

All puppies can fly
Fido is a puppy
Therefore, Fido can fly

This is a factual error

This is a perfectly logical statement. It follows all of the rules. I stated a major premise, I ensured the existence of a middle term between the major and minor premise and I distributed it properly therefore my conclusion must be correct. Sadly, it's patently wrong, but truth is not the province of logic so the truth of the argument does nothing to invalidate its perfect logic.

In real life, things are rarely so obvious. But the tendency to equate logic with truth sometimes allows us to accept, or arrive at, incorrect conclusions because of coherent logical arguments even if they contain untrue premises. I can think of examples in social circles where conclusions are drawn about people based on the community groups they belong to. However, sometimes framing those arguments in a syllogism can highlight either the logical fallacies of them, or the untrue premises the conclusion relies upon.

Title image from xkcd

Reductionism is a philosophy that provides some interesting thought exercises. When applied to science, the basic theory is that all science can be revealed as some subset of some other part of science, therefore reducing the subset into the other discipline. In short, systems are no more than the sum of their parts. Followed to its logical conclusion, we would eventually discover that there is, in fact, only a single branch of science that could explain everything.

By way of a small example (that doesn't work fully), we can take Kepler's laws of planetary motion. Kepler explained how planets move. A few years later, Newton came up with his laws of dynamics which explain how everything moves. Newton's laws explained how the planets moved as well as everything else, therefore Kepler's work was "reduced" to become part of the general understanding of how things move.

This is understandable and even logical. As our knowledge of the natural world expands, we frequently find that previous work, usually smaller in scope, fits into a larger picture. Incidentally, this is the hallmark of a good theory; good theories allow predictions.

Let's expand it a bit. How far can the discipline of biology be reduced? Living things are made of cells, which are made of molecules, which are made of atoms which are made of particles. Following the reductionist point of view, biology can therefore be reduced to be a branch of physics. That's much harder to wrap your head around. Fortunately, there is another philosophy that we can use in a scientific framework to help us understand why this is so hard to grasp.

Emergence is the philosophy that some systems exhibit behaviours that cannot be explained by any of its parts. In short, the sum is greater than its parts. When applying these philosophies to science, the criteria for valid reductionism is that it is a two-way street. Meaning that in order to be considered a valid reduction, the sum does not exhibit emergent traits and the parts are all accounted for. Taking the Kepler/Newtonian example above; this works. All of Kepler's laws are accounted for in Newton's (reduction) and Newton's "sum" does not behave in ways that cannot be predicted by its component parts (emergence).

However, when applying the emergence criteria to biology it seems that the sum has a great many behaviours and characteristics that cannot be explained by the parts. Since our understanding of particle physics does not provide us with a way to understand why Jimmy likes chocolate ice cream and Jenny does not, we cannot reduce human biology to particle physics. Somewhere in there is the religious argument for creation, but it will more likely turn out to be either ignorance on our part about physics, or invalidation of the emergence/reduction philosophies as scientifically useful. For the moment, however, they provide interesting 'what if' scenarios and playful theorizing is a valid way to pursue science and uncover new ideas.

As part of my undergrad studies we need to gain an understanding of what science is, what it is not, and - if it is science - how to determine if it is good science. Not all scientific theories are created equal and there are ways to evaluate how good a theory is. Some of the ways involve hard criteria such as leveraging probability and statistical analysis, and some are softer. It's the softer ones that interest me today.

In biological science, the mother of all theories is Darwin's theory on the origin of the species through natural selection (usually referred to as 'evolution'). I'm going to use this theory as a framework to provide examples of what makes a good theory. The theory of evolution through natural selection is a good theory, and here's why.

Good theories are simple.

Complicated theories are not desirable because they do not easily explain phenomenon properly and that makes it less available for other scientists to validate and build upon. It also makes it much harder to explain in the pub.

Darwin's theory of how evolution occurs is this: every time cells are copied, a small number are copied imperfectly . This means offspring are slightly different from the sum of their parents and that can mean three possible outcomes:

• If this difference is detrimental, it will likely prevent the offspring from mating so the change will be lost forever.
• If this difference is neutral, it will not hinder the offspring's ability to mate so the change will probably be carried on into future generations at a normal rate.
• If this difference is beneficial, that's when fun stuff starts: this guy will out compete its peers for food (so they starve and die) and has a better chance of reproducing more. This means this change will be carried into future generations at an accelerated rate (more offspring AND less competition)

The last two examples imply that species change over generations. Simplicity itself.

Keep in mind that in Darwin's time nobody had any real clue about genetics and the world was 50 years away from even discovering DNA existed, never mind understanding its purpose which did not come until over a century later. In the intervening time until today, many scientists have brought modern tools to bear and have validated Darwin's theory time and again.

Good theories are general.

If more than one theory can explain a phenomenon then the one that explains more should be chosen.

Darwin came upon his theory by observing finches in the Galapagos islands in the early 1800s. He noticed that although there were finches on all the islands, their beaks were very different. The finches that lived on islands where bugs were the main food supply had long thin beaks which were effective at extracting bugs from crevasses. The finches that lived on islands where nuts were the most populous food had shorter beaks more able to crack nuts.

Once Darwin had properly fleshed out his theory he realized that every living thing in the planet could have evolved in the same way. The beaks were just examples of advantageous mutations that were preserved and the finches born with unsuitable beaks for their island perished and their mutations lost.

If Darwin's theory only explained why finches had different beaks in the Galapagos, that would be a very specific and therefore less useful theory. The fact that it can explain the current structure of all life makes it a very general and therefore very useful theory.

Good theories allow predictions.

It is a hallmark of human intelligence to ponder "if this means that, then it must also mean this other thing. Let's check." In Darwin's case, if his theory of how finches got their beaks is correct, then we should be able to predict and find examples of generational adaptation in other life on Earth. In practice this is a terribly hard thing to do because adaptations take millions of generations to manifest enough to be measurable. For us long-lived humans that is a long time span. However, there are some short-lived life forms on Earth that can help us.

Mosquitoes in London turned out to be a good example. During WW II, the London tube was used for air strike protection. When the air raid sirens sounded, everyone went down into the tube and the mosquitoes down there had a feast and were able to multiply healthily and rapidly. Fast forward to 2015 and the population of mosquitoes in the London tube are now a different species than the mosquitoes above ground.(Seriously, Check it out).

Because these mosquitoes were isolated from others of their species, they evolved differently and are now different mosquitoes altogether. Their food is different, their environmental tolerances are different and they can no longer even reliably mate with the above ground mosquitoes. These are all evolutionary changes that Darwin's theory predicted.

Science knows it does not have all the answers. In fact, many scientists will tell you that science doesn't have a single answer. But if we understand concepts well enough to use them, to forge ahead with knowledge, then that is enough. That is why it is critical to have criteria and tools with which to evaluate theories. When all you have to choose from are "good enough" theories, you want the best "good enough" there is.

I went to a Christmas craft show today and a large part of the space was devoted to antiques and collectibles. The vendors had very large spaces with lots of old stuff ranging from typewriters to old door locks, to china and Polariod cameras. These guys weren't just cleaning out their attics, they were definitely "in the business".

Picking through old mechanical stuff is a great joy of mine. It doesn't have to necessarily work as long as it retains enough of its parts that I can see how it used to work. I don't have much use for old magazines and china, but I love old machines; or, at least, the things that preceded our machines of today. I must not be alone because the vendors there obviously know there is business in collecting and selling this stuff. That got me thinking about the reasons why we love old stuff.

I am a knowledge worker (Linux syadmin) and the size of the plant footprints I have been working with has steadily increased over the yeas. I now work in a globally distributed plant with innumerable servers. As my career progressed I came to realize that in very short order things become too complex for one person to handle or, in some cases, to even comprehend. Very quickly it becomes obvious that the only way very complex things happen is when we are able to build upon work from other people in the past.

The Open Source Software community is a great example. The days when one person could make a significantly useful and polished piece of software are all but gone. Even in the case when a single developer may own a project today, she most certainly relies heavily on frameworks and tools built by others. In the same way, 100 year-old door locks hold a fascination for me because they tell me a story about how today's locks came to be. 19th century locks have pitifully tiny springs in them that cause the action to work and they don't last very long in every day use. Today's locks are much more robust, but in order to get that way, we had to figure out what didn't work very well and, in some cases, we had to wait for the technology to be invented to make its successor.

It would not be possible to simply produce a reliable, working lock out of the blue. It's necessary to go through iterations to develop something good. If we did not have door locks today and built one from scratch, it would surely resemble the antiquated locks of the past instead of the highly advanced, bluetooth controlled keypads we have today.

There is value in iteration. In the book The Cathedral and the Bazaar, Eric Raymond includes the reason for this value in one of his rules for creating good software:

Plan to throw one [version] away; you will, anyhow.

This rule is an acknowledgement that we don't know what we don't know until we try to solve the problem. Going into a problem with the hubris to think you fully understand all its complexity has proven time and again to only lead to folly.

I think that is why old things speak to me. My work has provided me with a very keen awareness that making things is not easy. Antiques that are left behind for me to look at show me the path to how we arrived at what we have today. An old typewriter that was built without a number 1 key makes me realize that the lower-case letter L was good enough until, for some reason, it wasn't. Later typewriters that have correction ribbons show me how our world-view was still paper-centric and the solution to typos was, basically, to paint over them in some fashion. Finally, looking at computers today we have a "standard" keyboard set at somewhere between 103-105 keys and it includes several modifier keys that allow a wide variety of characters to be typed. And, of course, the paper-paradigm is long gone because we've decoupled typing and printing, so we are able to correct at will. All of these things happened because people wanted them and until the current typewriter did not meet their needs, there was no way to know what modifications were needed. This is the value of iteration and building upon work from the past.

Brute force hacking is the easiest, least effective, and messiest method of all the ways to attempt to gain access to a system. It leaves a really obvious trail, and it's fairly easy to stop unless you've become the target of large organization that really is out to get you.

By definition, brute force hack attempts are simply some variation of just trying to guess a proper username and password combination. I will look at attempts to break in to a Linux box via SSH, but the principals are the same regardless of the attack target.

Linux tracks log ins in two files and there are two separate commands to read them: last and lastb. The last command shows successful logins and system reboots, whereas the lastb command shows unsuccessful logins. Let's take a look at the latter.

The format of the output of both files is like so:

# lastb | head -n5
root     ssh:notty    218.18.37.100    Tue Nov  3 04:06 - 04:06  (00:00)    
root     ssh:notty    14.153.223.37    Tue Nov  3 03:08 - 03:08  (00:00)    
root     ssh:notty    93.90.222.136    Tue Nov  3 02:28 - 02:28  (00:00)    
root     ssh:notty    59.45.79.116     Tue Nov  3 01:58 - 01:58  (00:00)    
root     ssh:notty    59.45.79.116     Tue Nov  3 01:58 - 01:58  (00:00) 

The columns are username, terminal, originating IP address, and the login time, logout time and session duration. Since these are failed logins the last three columns together should represent no session time at all. If you look at the out put of the last command, you'll see what successful sessions look like.

Now that we know what file looks like, let's start tearing it down:

# lastb | awk '{print $3}' | uniq -c | sort -u
1 108-252-199-66.l
1 115.195.175.30
1 14.153.223.37
1 173-9-123-253-ne
1 176.31.128.45
1 218.18.37.100
1 5.8.66.90
1 59.45.79.116
1 93.90.222.136
2 193.104.41.54
5 108-252-199-66.l
15 66.48.26a9.ip4.s
22 91.210.104.86
29 23.95.21.136
1217 59.45.79.116
1326 176.31.128.45
1391 176.31.128.45
2760 59.45.79.116
2771 59.45.79.116

It becomes fairly obvious what is going on. The bottom 5 lines represent 2 unique IPs that have collectively failed to log in almost 10,000 times. For context, this lastb log only covers 48 hours. These are definitely brute force attempts, so let's see what they're doing:

The first and last guy has been solely trying to brute force the root account. This won't work on my system because I do not allow password logins through SSH:

# lastb | grep 59.45.79.116 | wc -l 
6749
# lastb | grep 59.45.79.116 | awk '{print $1}' | sort -u
root

The next guy is doing more of a dictionary type attack. He is trying to stumble across a valid username and password combination by going through an alphabetic list of names (looks like 450 unique names that he tries multiple times. This also will not work on my system because I have no users and, as mentioned, I do not allow password logins:

# lastb | grep 176.31.128.45 | awk '{print $1}' | sort -u | wc -l
450
# lastb | grep 176.31.128.45 | awk '{print $1}' |  wc -l
2718
# lastb | grep 176.31.128.45 | awk '{print $1}' | sort -u | head  
Clara
Claudia
Jana
achim
adelbert
adele
adrian
albert
albrecht
alex
... snip ...
werner
wiebke
wilfried
wilhelm
willi
wilma
wolf
wolfgang
xavier
yvonne

So now you know your enemy. What can you do to stop them?

Turn off password logins

vim /etc/ssh/sshd_config
set: PasswordAuthentication no
service ssh restart

Move SSH off the standard port 22.

This is controversial because it is essentially an attempt at security through obscurity, but if you take the time to do this you will notice a drastic drop in brute force attacks. I attribute this to the fact that most brute force attacks are done en masse and therefore the low hanging fruit become the targets of opportunity. Simply moving off port 22 will drastically reduce the attention you get.

vim /etc/ssh/sshd_config
# What ports, IPs and protocols we listen for
Port 22
service ssh restart

Use an IP blocker

If you simply must leave SSH on port 22 and accepting passwords, you can look at a package like Fail2Ban. This package tails logs and when it finds X number of failed password attempts, it does some preconfigured action. Typically, changes your IP tables rules to ban that IP either permanently or for some set period of time. I've used it in the past and it performed well.

Update

Nothing beats empirical data, so I moved my SSH daemon off port 22 on Saturday Nov 14th and watched to see what would happen to the stats. Below is a graph of failed log in attempts. I think it's pretty obvious when I changed ports. This is a really startling example of how a simple change like moving off a well known port can drop your exposure to drive by attacks. This will not be enough to prevent attacks from someone who is specifically targetting you, but it does get you off the low-hanging fruit list.

One of the things I love most about science is hearing other people call science a "thing". "Science says the planets are round". "Science says vaccines reduce the spread of herd diseases". While these conclusions are true, the way in which they are spoken belies some level of ignorance as to what science is.

We live in a society exquisitely dependent on science and technology, in which hardly anyone knows anything about science and technology.- Carl Sagan

Science is a way of thinking. It is a process that has been developed over generations to support the human endeavour to figure out how things work. The scientific process is not perfect and it certainly has led to some incorrect conclusions from time to time. But a big strength of the scientific process is that it is self-correcting. If you've ever spent any time in the open source community, you know how it works. The same organized scepticism and peer review that keep open source projects churning out good code are the same facets of the scientific process that keep good ideas flowing.

This leads me to my point. Because science is a process, it must be taught. Most people do not have any training in the cornerstones of the scientific process which are the ability to apply reason and the desire to think critically about ideas. Therefore, most people will accept virtually anything as evidence. Nowhere is this more apparent than in people's insatiable appetite for complex information boiled down into simple graphs.

Graphs show the correlation between two or more things: "How much do I weigh as I age?" "How many cars are red over time?" What graphs do not show is causation: "Ageing causes people to weigh more." "The spinning of the world produces red cars." Everything that you can think of can be related in some way and reduced to a couple of lines on a graph but that does not mean thing A caused thing B. The common mistake of people who do not take the time to reason things through is that they conclude that the correlation of two things must mean one causes the other.

The most entertaining example of putting two unrelated things together on a graph in an effort to correlate them has got to be Tyler Vigen's effort to build Spurious Correlations. That is, correlations that have no causal effect on each other, but seem to when plotted together on a graph.

Take a run through. It's a good laugh. And a few of them actually seem legit.

This is my first post with Ghost and since it contains code snippets and command line goodies I thought it would be a good test for Ghost's markdown language. Let's see how it goes.

The sheer number of bad people on the planet mean that there's a really good chance your website has at least been probed to see if it is a good attack platform. It may also mean that your website has already been compromised and is doing bad things for some other person as we speak. Some people I talk to say things like "well, if I get hacked, I'll deal with it then". But that's dumb. It's dumb because when someone compromises your website, they're not going to put a big banner on it letting you know. It may be days, weeks or months before you notice.

Without trying to sort your way through the myriad of 'security services', what can a lowly sysadmin do to protect herself? Access logs are a treasure trove of information and are an often overlooked resource. A fun activity is to take a gander through your Apache access logs now and again to see what people are hitting.

First, check out the referrers. Where are people coming from?

# awk '{print $11}' access.log | sort -u
"-"
"http://buttons-for-website.com"
"http://mydomain.com/"
"http://www.mydomain.com/"
"http://www.mydomain.com/phoenix_room.php"
"http://www.mydomain.com/reservations.php"
"http://www.mydomain.com/rooms.php"
"http://www.mydomain.com/styles/phoenix.css"
"http://www.mydomain.com/thehollow.php"
"https://www.google.ca/"

So far so good. Most of the requests are coming from links on my own site. I don't know why buttons-for-website.com would have a link to me, but there's more to look at before we can start drilling down. For example, what pages are these guys looking for and what IP addresses are they coming from?

Probes generally result in 404 response codes. This is because the bad guy is hitting a specific URL in order to determine if you're running a piece of software, or if you are running a specific version of some software. So check out any 404s and what the request was for:

# awk '$9==404 {print $7}' access.log
/xmlrpc.php
/robots.txt`

The robots.txt file is not a smoking gun. All search engines are supposed to look for it. However, the random request for an xmlrpc.php end point is curious. That is the xml endpoint for a Wordpress blog and this domain has never run an Wordpress installation. This smells like someone sniffing around. Let's look more closely at that one.

# for ip in  $(grep 2015 access.log  | awk '{print $1}' | sort -u);do host $ip; done

160.100.208.82.in-addr.arpa domain name pointer 82-208-100-160.pg-nat-    pool.mts-nn.ru.

A Russian IP. There is a ton of really bad traffic coming out of the eastern block these days so a probe for a non-existent URL from a Russian IP raises eyebrows.

It's also worth checking out all the non-404's in the log for two reasons:

• If this WAS a Wordpress installation, I would not see the xmlrpc.php file 404'ing. That means someone could be exploiting my xml endpoint under my radar.
• Bad guy may have successfully placed a bad file on your server and is hitting it to do her nefarious bidding. Those also would likely be returning non-404 responses.

There's not enough in this log to worry me, but if there were more suspicious entries, I'd start looking at how much traffic this IP was using to get a sense if he was successfully exploiting the site and then block it.

Checks like this are not very in-depth, but they take about 5 minutes to do and can really shine a light on anomalies worth investigating further.

I was assigned a Fidonet node number tonight. That will mean almost nothing to anyone, but to me it's an accomplishment. It took me 3 days of hacking around to get this thing set up.

Fidonet was the predominant pre-Internet message network which was created in 1984 and had an initial 12 nodes (BBSes), peaking in 1996 with ~40,000 nodes and down to about ~2,000 now.

I've been a Linux hacker (the good kind) all my adult life and have therefore been surrounded by other hackers of varying degrees of competency. Through those people I've learned that anyone can run a script, but the true mark of a craftsman in any trade is the ability to apply abstract knowledge to new situations.

I consider bringing up 20 year old tech in a day when almost nobody understands that tech any more to be the true mark of a craftsman. There's almost nobody left who knows how to do this so you have to rely heavily on intuition and experience to get the pieces working right.

Since day one, Fidonet Sysops have always had to prove their skills. The only way to join Fidonet is to set up a Fidonet capable BBS and then send a Fidonet netmail to your network coordinator asking for a node number. The node number is the only way to become part of the Fidonet network and asking for it via netmail proves you have your system set up correctly. Even today, the network coordinator in charge of my area is happy to help me set things up via Internet email or voice, but will STILL only issue me a node number if he receives my request via old school netmail.

I am now Jon Watson at node 1:249/207

One of my gifts for Christmas was a Pebble "classic" smart watch. I've wanted one for a while because the idea of smart watches and other wearable computing devices is interesting to me, but having never had any experience with one, it was hard for me to determine if I'd actually like it. The Pebble Classic is cheap enough ($109 here in Canada, generally) that it's worth the risk. I've had my Pebble for a little over a week now and here are my thoughts.

The Concept

The million dollar questions is why would anyone need yet another device to tell them when they have an email or a text message? It's a good question and part of the reason why I was not 100% sold on the idea, but here's what I thought the advantages would be and so far it has worked out as I expected.

The Pebble apps on my phone essentially just forward any notifications that already appear on my phone to my watch. My watch quietly vibrates and I take a look to see if it is something I care about right now. If it is a phone call, I pull out my phone and answer it. If it is almost anything else, it can probably wait until I am sitting down somewhere. Prior to the Pebble, I would have to pull my phone out pretty much every time it went off to ascertain the cause. Some hackey things like assigning different alert tones or vibration patterns to specific apps helped some, but in general I either would not feel the distinct vibration or remember what sound went with what. Pebble allows me to just quickly glance down and be done with it.

My phone now stays in 'quiet' mode (doesn't even vibrate) in my pocket and only comes out when I actually want to do something. An added unexpected bonus to this is that my phone's battery life has improved because I don't pull it out and turn on the big ol' 6-inch HD screen a hundred times a day any more.

Warning: App Overload

I've done this with every phone, tablet, and computer I've ever had and my Pebble was no different. I have this inherent (and usually mistaken) belief that native apps are going to be somehow inferior to third party, community developed apps. I therefore usually spend the first few days with a new device loading it up with third party apps to replace the native apps instead of spending the time to learn what the native apps actually bring to the table. This almost always means I end up with a wonky, unstable, slow device and the first few days with my Pebble were no exception.

Stage two of this process is to hard-reset the device and then only reload the few apps that I actually need or want and let the nice stable native apps do their job.

Pebble has made their SDK widely available so there are tons of developers and hackers making apps and watch faces for it. The Pebble app store is not curated as far as I can tell, so like the Google Play Store for Android, there are a lot of crap apps to sift through in order to find the few diamonds.

Hacker's Delight: The SDK

I am a sysadmin, not a developer. While I certainly write lots of scripts and I have taken programming in school, it's not my area of expertise. Therefore, when I hear that some device has an SDK, I don't get very excited about it because I know from experience that getting a development setup working is usually a huge pain in the butt and takes tons of my time fumbling around with things I am not experienced with before I can even attempt to write my first Hello World program.

This is not the Pebble world. In literally 10 minutes, I was writing code and deploying it to my Pebble using the online Cloud Pebble.

Cloud Pebble has an online IDE, online compiler and - get this - one-click compile and deploy to your phone. The deployment uses the Developer Server in your Andoird (and iOS?) app to push code to the phone and run it. It is incredibly painless. The most painful part for me was reaching waaay back in my memory to remember how C handles strings (or doesn't, as the case is) because C pre-dates even me by a few years. I took C++ in school which is a whole different beast.

I wrote my first watch face and called it Wordy Face (that is an actual screenshot of it running on my Pebble). If you want to laugh at my terrible C code you can grab it from Github here, or you can go ahead and use the watch face on your own Pebble by installing it from My Pebble Faces here.

My App List

The apps I ended up deciding I could not live without are as follows (all are available from the Pebble app store):

• YANC (Yet Another Notification Center): There are a ton of third-party notification apps to replace the native Pebble one. They are all clunky and slow and load a lot of useless stuff into the watch, but YANC is actually pretty good. It allows a very broad range of control over things like the text size and icons of notifications sent to the watch and I have chosen it over "Notification Center" and "Pebble Navication"
• Pebblets: You can only load 8 applications to the Pebble at any given time. I rarely go over 5, but Pebblets crams 6 different apps into one app "slot" so it is indispensible if you want things like a timer, a calendar and a calculator at your finger tips.
• Music Boss: The Pebble music app doesn't seem to know how to talk to Spotify despite having a setting to select Spotify as the music app on the phone. Music Boss addresses this short coming and allows you to remote control pretty much any music app on your phone with your Pebble. This is handy for me when I am running because I don't have to pull out my phone to skip a song, pause it, or adjust the volume.
• Dashboard: The whole point of the Pebble is to reduce the amount of times I have to pull out my phone to do stuff. Dashboard is essentially a utility remote control for the phone that allows me to do things like turn off the wifi and lock the screen from my wrist. Two things I do surprisingly a lot.
• Wordy Face: I wrote it...I gotta use it, right?

I'm into month two with my Pebble Classic and in that time I've gone through almost every app and watch face in the Pebble App store. That has enabled me to gain a lot of insight into how I use my Pebble and the surprise ending is that although notifications are a big part of its usefulness, I use it for much more than that.

I decided to write this post after reading David Breger's post on LinkedIn about why he does not wear his Pebble any more. I tried to comment on the post but the Submit button would not enable for me, so I ended up thinking about his post and about how my experience with the Pebble so deeply differed from his. I finally came to realize that David's post solely focuses around notifications and he has completely missed, or at least did not talk about, the rest of the Pebble ecosystem. So I aim to fix that.

I have found my Pebble to be a very useful tool for reasons that have nothing to do with notifications. My first concept of the Pebble was totally centered around notifications because that is how it is marketed. I didn't really know how the Pebble worked until I had one in my hands, but I quickly came to realize that it basically forwards notifications from my phone to my watch. If an app does not create notifications on my phone to begin with (ie - I've turned notifications off for some app) then it would also not be able to notify my watch. That led me into the realization that the Pebble is more of a remote screen than anything else.

Then I started rummaging through the app store and I found apps that let me go the other way. I found apps such as Music Boss and Dashboard that let me send commands back to the phone. That created a whole new frame of mind for me because I now realized that this Pebble on my wrist was actually not only a remote screen, it was also a remote control for my phone. That opened up a world of ideas and here are some of the things that are now just part of my every day life with my Pebble.

Controlling my music

I use the Music Boss app to control the music on my phone. I can do rudimentary things like stop/pause/skip and control the volume. That in itself is nice - when I am running or working out at the gym I don't need to pull my phone out to mess with my music. If someone wants to talk to me, I simply press the middle button on my watch and my music pauses.

The next step in this evolution is the Bluetooth connection in my car. Since I own a Pebble, the Bluetooth on my phone is always on. Therefore, when I step into my car, my phone hooks up to my car stereo immediately. What can I do at this point? Press that middle button on my Pebble and presto - my music is playing in my car! I have not touched my phone or my stereo. In fact my phone is still buried in my pocket somewhere. All I had to do was press a button on my watch.

Securing my phone

Both the Pebble and the Android Pebble App exhibit separation anxiety when they cannot find each other. My phone knows when the connection to my watch dies so it seems like a no-brainer to turn that into something more useful than a battery draining whine fest. Enter the Dislock app.

Dislock is an app that locks my phone when the connection to my Pebble is lost. Under normal circumstances I just have a very unsecure swipe lock set on my phone, no PIN or password required. However, once I forget my phone on the table and start to walk home, my phone locks and now it requires a PIN to log in. As an added bonus, I used the Canvas application to create a watch face that vibrates when my Pebble disconnects from my phone so I become aware that my phone is no longer near me.

Keeping up on appointments

I don't want to become the Canvas app spokesperson here, but it really is an amazing app. With zero technical skill, anyone can create very intricate watch faces for their Pebble. I have a custom watch face that, among other things, shows me my next calendar appointment. Given more room I could put more than one, but I choose to put other useful things like battery status and day of the week on my watch face instead, so one is all I can fit in. However, that one is very useful to me as it is a constant reminder of the next timed appointment I need to work around.

Any review of a Pebble that focuses solely around notifications is incomplete. There is much more depth to the device than simply vibrating when you get an email and these abilities only become clear after living with the watch for a while and plundering the app store to get a sense of what it can do. I am not saying that a smart watch is for everyone, but I am saying that you need to get past the novelty of notifications before you can really understand how the Pebble can become a useful part of your life instead of just The Next Big Thing (tm) strapped on your wrist.

I recently bought a Chromebook. Over the years I have had a short, unimpressive experience with one of those "Netbooks" that tried to create a place in the market so I was prepared to be a little disappointed. However, the critical role this thing had to fulfill is to be a backup computer to RDP into work if my primary system died so I was willing to put up with some limitations as long as it could plug that hole.

The first thing I learned during this process is that customer reviews from Chromebook users are almost totally useless. They mostly consist of incredibly naive and clearly non-technical people who were shocked and dismayed that their $250 "laptop" did not run Windows or MS Office. I doubt the critical thinking skills of these people because if it were possible to produce such a beast at that price point, it seems obvious to me that the market would be flush with them. Having said that, there are some low end $350 full-blown laptops out there from Acer and HP so the market is pretty close.

Once you get rid of the cruft, the remaining reviews from bloggers and tech sites don't really reveal any information that isn't written right on the box. Chromebooks are such simple machines that it takes about 5 seconds to rattle off the specs and sockets and you're done. Watch - sockets: 1 joint mic/headphone jack, 1 HDMI port, 1 USB 2.0 port, 1 USB 3.0 port, one multi-SD Card reader, 1 webcam. Now specs: 1 dual core 1.7 ARM CPU, 2GB RAM, Webcam, keyboard, trackpad. Done. So the real magic, the real reviews, the real information that I was looking for is not what is it but rather, what can I DO with it? That information is hard to come by.

Chromebooks run Google's ChromeOS which is essentially a big ol' browser. Therefore, the only apps you can run are extensions built for the Chrome browser, or apps built specifically for ChromeOS itself. If you want to know what a using a Chromebook is like and you happen to be running Windows 8, put your Chrome browser into Windows 8 mode. That is exactly what running a Chromebook is like except you can't minimize it and go do something else. The browser is the OS.

So what does this mean to you? That depends entirely on how you use your computer. For me, switching to ChromeOS was a non-event. I don't use local storage anymore except for a few documents I don't want in the cloud and despite the relatively small storage space on my Chromebook (16GB), I am not even close to running out of space. For me, this is a fully functional laptop because pretty much the only thing I use my computer for anymore is as a connection point to the Internet. My life already is a browser.

On the flip side, if you make heavy use of your local machine (like you're a gamer or you need to run specific applications) then you're going to have a tough time and this machine is not for you. Chromebook critics like to point out that you can't run applications that you may need and they almost always throw out Microsoft Office as an app you won't be able to use. I have to ask myself, in 2014, who really needs MS Office anyhow? We're past specific apps now and we're well into functionality. In other words, we've gone from "I have to use MS Office" to "I have to create a document". The application used to create documents, spreadsheets, slide shows, etc is irrelevant to virtually everyone but a small percentage of people who are locked into the Office product due to work or some other weird situation. I also have to wonder who actually creates documents and spreadsheets and slide shows anymore. I am definitely on the techie side of the world's population and I have one spreadsheet to do some budgeting on and that is about it. I just don't make documents anymore - if I am writing something down it is because someone else needs to see it so I write an email or blog post instead.

If you happen to be a technologist, you can drastically extend the Chromebook's abilities. I mentioned at the top of my post that one of the critical roles I needed this device to fill was to be able to RDP into work. Generally, except in some very specific setups at work, this cannot be done with a Chromebook. However, being a Google device, Chromebooks are very easily put into "developer mode" and from there it is child's play to use a tool like Crouton to install a full blown Linux distro side-by-side with ChromeOS. This setup is far and away the best part of my Chromebook for a couple of reasons. First, it is not dual-boot; I simply fire up my Chromebook, open a shell with CTRL+SHIFT+T and start up Ubuntu in a chroot. I can flip back and forth from my running Ubuntu instance to my Chromebook and it runs them side by side with ease. The second good part is that putting your Chromebook into developer mode is not like rooting or jail-breaking your phone. It is a totally reversible process so if you decide you don't want it in this state any more, or if you sell your Chromebook, you can easily take it out of Developer Mode without leaving a trace.

I'm happy to answer questions about what can and cannot be done with a Chromebook if anyone is looking to buy one and wants some answers before hand.

If you don't already know what AWK is, you're going to find this blog post really, really boring. Eyes glazed over, drooling a little bit, head bobbingly boring.

This is you if you are a banana and don't know what AWK is while reading this post.

This is you if you are POTUS and you know what AWK is while reading this post

Are we clear? If you're the banana, you should leave now. This post is not for bananas.

Comparisons

While I knew that AWK could parse text files with amazing ease, I did not know until recently that it could do comparisons on the fly. I recently wanted to pull out the number of occurrences of various http response codes from a log file.

Given a log file format like this:

198.103.249.251 - - [11/Jan/2013:14:50:34 -0400] "GET /images/gallery/pause.gif HTTP/1.1" 304 - "http://www.phoenixhollow.com/" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Tablet PC 2.0; .NET CLR 1.1.4322; .NET4.0C)"

Let's see how many 200s there are and print the IP addresses of those requests:

awk '{ if ( $8 == 200 ) { print $1} }' access.log

Now let your little sysadmin mind go wild with all of the possibilities of this. Pipe the output into sort and you can start to get some really interesting data very quickly.

Looping

Want to kill every damn process on the system? AWK up the PIDs and kill em:

for pid in ${ps -ef | grep http | awk '{print $2}'};do kill $pid; done

OK, so this is more shell-ey than AWK-ey, but it would be pretty damned hard to do without the amazing AWK.

Change Field Delimiters

So far my two examples are great for text files that have their fields delimited by spaces. That's not the whole world though, my friend. There are lots of delimeters out there and sometimes even when you're working with a space delimited file, the data you want might not be perfectly delimited by spaces.

Back to this log. Let's suppose I want to get the Internet Explorer version (MSIE 8.0 in this case).

198.103.249.251 - - [11/Jan/2013:14:50:34 -0400] "GET /images/gallery/pause.gif HTTP/1.1" 304 - "http://www.phoenixhollow.com/" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Tablet PC 2.0; .NET CLR 1.1.4322; .NET4.0C)"

Change the delimeter to the semi-colon with this invocation.

awk -F\; '{print $2}' access.log

Like anything in *nix, the real power is in stringing these simple tools like AWK together. These three little tricks comprise about 85% of my AWK usage and make my life a whole lot simpler.

Did anyone turn into a banana?

Web encryption is at the top of the discussion list these days in geek circles and with good cause. The revelations over the past few months that many countries are collecting wide swaths of Internet data on their own and foreign citizens has made us all stop and re-think things. We used to think it was fairly near impossible to collect every email and every web session that passes through the Internet but that assumption is now being challenged. Even the security of our encrypted web sessions using the tried and true Secure Sockets Layer (SSL) technology has now been revealed to be orders of magnitude less secure from government prying than we thought. As we work our way through this maelstrom of blows to the head it's becoming clear that the only answer to true privacy on the Internet is Trust No One (TNO) encryption.

Who uses encryption, anyhow?

The short answer is that we all do, we're just aware of it in degrees varying from really aware to not a clue it's happening. Joe netizen who does things like Internet banking uses SSL encryption every session but the browser takes care of it all so Joe has no idea it's happening. If websites did not use SSL encryption, then Joe would be giving away his username, his password, his bank account numbers, and everything else every time he used his bank's website. These days many websites other than banks force SSL encryption in order to protect their users because the bad guys have shown that they can wreak havoc with just usernames and passwords, never mind more sensitive data.

What is this encryption you speak of?

I think we all understand that encryption means something is encoded in such a way that only the intended recipient can unscramble it and read it. If I want to send you something encrypted, then I have to scramble it in some way that you will know how to unscramble it. The method that you use to unscramble it is called the "key". Anyone with the key to the scramble will be able to read the message.

Up until somewhere around the 1970's all crypto was symmetrical meaning that there was only a single key and it was used to encode as well as decode the message. This symmetrical crypto process was very secure and it still in wide use today in industrial and military grade crypto, but it has one serious drawback. In order for me to encrypt something to send to you, you and I would have had to meet prior in order to share the symmetrical key with each other. That really limits the usefulness of crypto because you can't go running to your bank to get a crypto key in order to use their website; then run to facebook to get a key to use their site, then run to your employer to get a key to use their web mail, etc, etc. Another problem with symmetrical crypto is that once you have the key to encrypt messages, you can also decrypt all the messages sent to whomever gave you the key. In short, you need to give away the farm with symmetrical crypto. What we really need is a method to share a key without any prior meeting between the two parties. This is called asymmetrical crypto and now forms the basis for what we call the Public Key Infrastructure (PKI).

Steven Levy's book, Crypto, is a really great book on how public key encryption came about and how the US government tried to stop it.

In PKI, the encryption key and the decryption key are not the same (hence why it is called asymmetrical crypto). The encryption key is called the public key and, as the name suggests, it's not secret. It can be shared with the public in plain view without degrading the integrity of the encrypted message. The decryption key is called the private key and it is a secret. It must be held in confidence by the person who will need to decrypt the incoming messages. This means that if you want to send me an encrypted message, you simply go get my public key which I can just send you via unsecured means in plain view of everyone, and use it to encrypt the message.

Here's the critical points of PKI:

• The public key can be shared with everyone. It is usually uploaded to a public key server and PKI programs used to encrypt messages know this and can find public keys when necessary.
• When used in web server discussions, the term key is usually replaced with certificate.
• The private key is the only key that can decrypt a message encrypted with the associated public key
• Keypairs (meaning the set of public and private keys) usually expire. Once a keypair expires, it is no longer considered valid and usually cannot be used. How do web servers use PKI?

As I mentioned above, websites are secured using SSL. SSL is an encrypted socket (meaning an IP address/Port pair, for those of you who care) and it is encrypted with symmetrical encryption. But knowing what we now know about symmetrical crypto, this is not a workable solution because you'd have to have some way to get your hands on the symmetrical key prior to visiting the website. What really happens is that PKI is used to encrypt a symmetrical keypair which is then used for that session. At a very high level, an SSL session is set up like this:
Your web browser connects to the secure web server
The web server sends its public key and a random number to your web browser
Your web browser generates a key with that random secret and encrypts it with the web servers public key and sends it back. Now you're set up to engage in a secure web session using that (usually smaller) symmetrical key.

This process has been in use for years and works very well. Since PKI is used initially, there is no need for you to have any of the web server's crypto before your first connection to the site and it is used to generate a nice safe symmetrical key to protect your data. So what's the problem? Private key expiry is the problem.

Since we now know that many countries are now collecting Internet data and storing it, we can infer that they have a lot of encrypted data stored away that they cannot read. These countries have two choices: either try to break the crypto or get their hands on the decryption key.

What's "breaking crypto" mean?

"Breaking" crypto means, essentially, to guess the key. When dealing with computers we're necessarily talking about binary so the number of possible keys (called the keyspace) for any given key length is roughly 2^key length (leaving some bits for headers and control data). Therefore, a 2048-bit keyspace has a number of possible keys so large that I can't find a calculator to give me a result for 2^2048 other than "infinity". Backing it down a lot by way of example, an easily broken 64-bit keyspace has a mere 10,000,000,000,000,000,000 possible combinations (that's 19 zeroes). Therefore, "guessing" a key in this arena means powerful computers that are able to try thousands of keys per second. There are some algorithms around that seek to limit the number of guesses required but again "limiting" in this scope means years rather than decades. At this point, there is no practical scenario where breaking a 2048-bit key is useful for anyone on the planet. It would take years, perhaps decades, so whatever data you were trying to get at is likely not useful anymore. Having said that, anyone can get lucky and stumble across the key first try. Also, in time even large keys like will be breakable. There was a time when 56 and 64-bit keyspaces were considered unbreakable but now those small keys are trivial to break because of the increase in computing power available today.

There's a popular idea floating around that organizations like the NSA can just break the crypto on any message and forcibly decrypt anything they want. This myth stems from the fact that Hollywood knows nothing about crypto and also has exactly zero interest in providing anything even remotely factual if it gets in the way of a plot. The scenes where the hacker-guy or hacker-gurl sits down and 'cracks the encryption' in a few minutes is completely, utterly ludicrous. Let's put this in perspective: a key's strength (meaning its resistance to being cracked) is based on the length which refers to how many bits the key is comprised of. A 1024-bit key is very common and in many cases web servers are now using 2048-bit keys.

A computer powerful enough to crack a single 1024-bit key would cost $1,000,000 to build and take one full year to crack the key. link

So what does this mean? It means crypto is really, really effective and even the three letter agencies world-wide have virtually no chance of cracking even a single email message in any reasonable time frame. When you take into account that they are collecting literally millions of encrypted messages every day, it's obvious that they would never be able to catch up. So what's the solution? Bypass the whole cracking problem by just getting the private key.

Recall that keys expire. When they expire, web server companies generate new ones and discard the old ones. Once a key is expired, there has never really been a lot of concern about what happens to it because it has been used only for transient sessions. All of the traffic that key encrypted is old, it's gone, that session is over, there's nothing lying around to decrypt. However, in the new PRISM world, that's no longer the case. The NSA almost certainly has stockpiles of encrypted data that they've collected off the Internet so if it were able to lay its hands on those expired private keys that were used to encrypt that data....well, bingo. That's a hell of a lot easier than trying to crack millions and millions of keys. It's not totally clear yet whether the named companies in the PRISM documents are, in fact, handing over expired keys to the NSA, but the possibility certainly exists. Congress in the US has passed a number of laws in the past that unprotect previously protected information after a set period of time. For example, after 6 months, email is not longer considered private therefore the hoops law enforcement agencies have to go through to get old email are significantly less than newer email. I'm not sure how expired SSL keys fit into this framework, but...well....? The NSA may have to wait a year or so for the key to expire, but eventually they may be able to warrant or just talk the big web providers into giving them the expired keys.

So what do we do?

The fundamental weak point in all Internet encryption is that it happens on the Internet. This means that the provider you are using (Gmail, Facebook, Hotmail, etc) has the private key that can be used to decrypt your traffic. You are putting all of your trust in those providers to not allow their private key to be compromised, subpoenaed or other otherwise fall into the wrong hands. The only true way to know for sure that your data cannot be decrypted is to encrypt it yourself before it enters the Internet. This process has the fledgling acronym PIE (Pre-Internet Encryption). PIE is just one practical application of the well established TNO (Trust No-One) approach to security we all learned thanks to Agent Maulder.

There's not much you can do about your website usage since it's not possible to encrypt your own traffic and send it to a web server. That part remains vulnerable to examination if the expired private key can be had. However, for email, chat, and cloud storage there are viable solutions out there that provide end-to-end encryption meaning that nobody between the sender and the recipient has the decryption key.

For some starting points and alternative programs that use PIE:

Email: Check out Thunderbird with the Enigmail add-on
Chat: Check out Threema
Dropbox: Check out BitTorrent Sync

Here's a non-concept for you: secure email. There's a lot of media frenzy surrounding the recent shuttering of Lavabit and Silent Mail and most of it is unwarranted (see what I did, there? Warranted?) While any security is certainly better than no security, the media is presenting the loss of these services as something that matters and honestly, it really doesn't. Email is so inherently insecure and the laws of most countries allow law enforcement to warrant emails anyhow, so there's almost no advantage to using a secure email service if your intention is to be bad. In short, there is no such thing as secure email.

I had never heard of Silent Mail before a few days ago but I have both a Lavabit (had) and a Hush Mail account; both provide encryption bundled into their email service and from the press surrounding Silent Mail, I assume it offered a similar service.

Why doesn't secure email exist?

The Simple Mail Transfer Protocol (SMTP) that is used to send email around the Internet does not support security. Over the years, security processes like Transport Security Layer (TLS) and Secure Sockets Layer (SSL) have been bolted on to SMTP to offer some semblance of security during transmission across the Internet, but most Internet Service Provider (ISPs) and many of the large email providers do not use these security add ons. When you send an email, your server connects to the server belonging to whomever you're sending the email to and says "what encryption do you support?" If the receiving end says "none", then your server says "OK, I will just send this to you unencrypted" and sends your email in plain text across the rather hostile Internet. Supporting encryption is expensive (meaning it takes time and computer resources to keep setting up and tearing down secure connections) so the vast majority of email providers do not support it. Therefore, we can extrapolate that a good chunk of email on the planet is sent in the clear.

So what do these secure email services offer?

They offer encryption "at rest". This means that once your email is received on their servers (when it is at rest), they encrypt it for you until you pick it up. When you log in to pick it up, they decrypt it and send it to you. If you're using POP to pick up your email, that's the end of it: the unencrypted email is sent down the pipe to your email client and typically erased from the server. If you're using IMAP or webmail, then the email generally stays on the server so it is decrypted many times - every time you call up an email to look at it - and an encrypted copy stays on the server until such time as you purposely delete it.

How do they encrypt my email?

When you create an account on these services they generate an encryption keypair for you as part of the signup process. The keypair consists of a public and a private key which are two separate files. The private key is the valuable one because it can be used to decrypt your emails and they will protect that key by locking it with the password or passphrase that you supplied when you signed up. Your private key is necessarily stored on the mail server because if it were not, then they could not decrypt your email before sending it to you when you log in to view it. However, what is NOT on the server is the password or passphrase required to unlock that private key so that it can be put into use. Your private key cannot be used to decrypt your email until you log in at which time the server then knows your passphrase to unlock your private key. A passphrase is essentially the same thing as a password except that it can consist of many words as the name suggest. A password is "thisisagreatpassword" a passphrase is "On December 12th, 1857 there was a huge explosion in Norway" (probably not a fact).

So if my email is encrypted it's protected from prying eyes, right?

No. Not all prying eyes.

If Lavabit or Hushmail or Silent Mail were to be hacked or if they lost a hard drive during an upgrade or something like that, your email would be protected because the person who gained access to the hard drive or system would presumably not know your individual passphrase. Therefore, all they would see when they looked in your mail directories is a bunch of random encrypted gibberish. "Psuedo-random noise" as us geeks like to call it.

However, if someone with legal authority came along and said "give me all of Jon's email" then there is simply no protection. Lavabit claims that their "system [is] so secure that even our administrators can’t read your e-mail." which is disingenuous at best. Keep in mind that in order for you to be able to read your email when you log in, these secure email providers have to decrypt the email for you first. Since your passphrase is not permanently present on the server, the claim that administrator can't read your email teeters on the bad side of disingenuous but doesn't quite tip over the line.

But let's get real; when Joe FBI shows up and this secure email provider says "Gosh, Joe FBI, there's just no way I can decrypt that email for you, sorry" nobody is going to believe that. Since these secure email providers must, by necessity, be able to use your private key on their server it's quite obvious that they can indeed decrypt your email under some circumstances so Joe FBI says "the next time Jon logs on, trap his passphrase and use it to decrypt his email and send it to me". There's no way that any of these secure email providers can convince anyone that this is not completely possible and, let's face it, pretty trivial to do. I am reasonably sure that this is why Lavabit and Silent Mail shut down: once faced with the very real specter of being forced to decrypt, they realized there was simply no way to avoid being forced to do it. Given the information at hand it would appear that Silent Mail went through this exact same thought process and shut down pre-emptively. On the other hand, given Lavabit's abrupt shutdown and very vaguely worded explanation, it seems likely that they were actually served with a warrant or a National Security Letter which typically carry a gag order along with it.

But didn't Edward Snowden use Lavabit?

Maybe. One media outlet reported that he might have and now every media outlet says he did so it's not really easy to tell. What IS really easy to tell is that if Snowden did use Lavabit, he did it with eyes wide open. There's no way an NSA analyst who spent his waking hours trolling through millions of captured emails does not know how email works and what its security limits are.

So secure email is a waste of money?

It depends what you want from it. If you want privacy for your every day life then there is some value. If you want protection from the government for illegal activities, you should stop using email altogether. And hang your head in shame.

I use Hushmail which is a Canadian based private email company (note how they use the word private and not the word secure). I am more than happy to pay for the service for a few reason:

Full disclosure: I am just a happy Hushmail customer. Sadly, they have given me no money, anything of any value or provided me any benefit above what I am normally entitled to due to my paid account for writing this next bit.

• I am not attempting to conduct any illegal activity. This is critical because if I'm not involved in anything illegal it'll be harder than hell to get a warrant to get at my email.

• Hushmail is Canadian and I am Canadian. Typically, your government is not allowed to spy on its own citizens so it's usually better to deal with companies in your own country where possible. Citizens are usually afforded a slightly higher level of protection than foreign nationals. I realize there are some large and angry discussions about how the U.S. is handling this at the moment, but in the rest of the free world, it's better to be a citizen than not. If someone wanted my emails, they would have to get a warrant from a Canadian court and give it to my Canadian company to get my Canadian email. That's a better deal than storing my email in the U.S. where I am just another foreign national with little protection.

• I know exactly what protections I am getting so I can evaluate what I am paying for. Hushmail is very transparent about what protections I can expect and they have a handy-dandy chart that essentially explains in a simple way what I explained in a very complicated way above.

• I know exactly how they will handle a court order because they tell me exactly how they handle court orders.

• I can send encrypted email to people who do not have a Hushmail account, people who have never heard of encryption, and people that have no idea how to do anything other than load a web page.

• I have a variety of really neat advanced Hushtools which makes life easier for me and for other people who have to deal with my insistence on encrypting my email.

• Hushmail's spam filters actually work. I'm accustomed to GMail's great spam filters but every time I use another email provider I am always amazed at how much spam ends up in my inbox. Hushmail's spam filters are as good as GMail's. This is not Hushmail specific, but after 6 years of using GMail, it's really nice to use a provider that does not shuffle through my email in order to put ads in front of me.This is the difference between paying for email and using a free service. If you aren't paying for something, then you are the product. I like not being the product.

There are some really good reasons for using a private email service, but they're not the reasons the media is spewing out. Like any product, it's a good fit for some uses and a poor fit for others. For me, Hushmail's tools, their transparency about both the product itself and their action to information requests, and the very reasonable pricing makes it a no-brainer for me. My paranoid tin-foil hat wearing mind loves it. Your mileage may vary.

Final thoughts

I find it very weird that both Lavabit and Silent Mail shut down with basically the same story. Lavabit's vagueness seems to indicate they had been served with a warrant of some kind, and Silent Circle outright says that they shut down Silent Mail because they came to realize that "Email as we know it with SMTP, POP3, and IMAP cannot be secure" and they did not want to be faced with the prospect of being forced to turn over customer emails. The weird part is not that these guys figured out they could not resist being compelled to turn over customer email; the weird part is that they apparently did not plan for how they would react to these warrants when they came. Co-operating with law enforcement is required by humans and businesses alike; any business needs to have some sort of plan or policy in place about how they are going to handle their legal obligation to cooperate with law enforcement. For two services to simply throw up their hands and say "Gosh! I never saw THIS coming" shows a really short-sighted approach to doing business in my humble opinion. There are no anarchistic businesses, my friends. Anarchy is for humans.