Obituary, for Ray Tomlinson and Email

March 7, 2016

Several news outlets, for example The Verge, have just run obituaries for Ray Tomlinson. Tomlinson was involved at BBN in the days of ARPANET, or you might say the infantile days of the internet, and implemented some functionality that would eventually morph into modern-day email.

The Verge calls Tomlinson out for being “the savior of the @ sign,” which was relatively uncommon in everyday use (being found mostly in accounting to indicate unit prices) before his idea to use it for user addresses. This is a cute chapter in the history of a now ubiquitous symbol, but it also calls out one of the most powerful properties of email, which goes back to Tomlinson’s very first version. When you send an email to someone, you indicate not just who it should be delivered to, but where. That where, the host name, might be a computer at the person’s employer or university, it might be a computer that they own personally, or, as is usually the case today, it might be a computer run by their ISP or a dedicated email provider. To you, to the sender, it doesn’t matter. Like a letter carrier might consult a route map, your mail server will consult the DNS to determine exactly where to connect in order to deliver the message. It will initiate this connection using SMTP, an open and standardized protocol universally supported by mail servers regardless of the specific mail transfer agent, operating system, or even processor architecture they run. The recipient’s mail server will then hold the message for them to retrieve, using a mail user agent (MUA) on their computer or phone, or a web interface.

All of this detail is hidden from users. To them, you just send an email to someone. They give you this string, this “user@host” address, and you type it into your MUA or webmail or whatever you use, and you click the Send button, and then they get it. As another deceased computer pioneer loved to say, it just works. This concept in computer systems is referred to as federation. That is, you can say that email works so well because a huge number of distinct email providers have all federated together in order to exchange messages.

The incredible thing about email is just how rare this is.

Compare this to modern messaging systems, which my generation increasingly uses instead of email to keep in touch. The popular new messaging tools are completely non-federated: Telegram, WhatsApp, Skype, Viber, etc. all require that everyone you want to communicate with sign up with the exact same service providers that you use. Facebook Messenger, which I suspect is the #1 messaging service in the United States, requires that everyone you want to communicate with have an entire Facebook account, which is not a small investment of time or information. Viewed from this abstract perspective, it is clear that these modern services are inferior to the comparatively archaic technology of email.

Tomlinson’s innovation was, in hindsight, obvious. Even in foresight, I think, it was not viewed as particularly revolutionary by any of the people involved. Tomlinson discusses email as having been quite boring. The idea was simple, the technology was simple. Tomlinson didn’t really invent email. He just put together the first generation of software for an obvious use of the large-scale network that was being invented.

Like Tomlinson, though, email is dead. It is still walking, yes, but the pace of innovation has stopped. As technology, it is disposable. No one cares. No one gets excited about email.

The more I look at these industries and services, and the technology and business behind them, the more I suspect that the idea of a federated messaging platform like email is fundamentally a product of its time: The early days of the internet (and ARPANET), when everything was built by universities and defense contractors for use by employees of the same. When the internet was not really a business, but a research venture, quietly backed by the profitable business of war.

We’ve seen this happen over and over. XMPP is a prominent example, as is SIP. These standards are still around and in use, but not in anywhere near the way that their creators had envisioned. A usable system of federated XMPP servers just never happened, and even the organizations that had used XMPP within their own platforms (Facebook, Google) have phased it out.

Every computing professional should think very, very seriously about this question: why is a system that was fairly simple to create in 1971 apparently impossible today?

Business Purposes

A lot of the problem lies in the motivation of modern software developers. Tomlinson did his work in a now hard to imagine golden age where the internet ran on magic and no one talked about ROI. Email was designed the way it was because it was the best solution anyone could think of. It was not designed that way because it worked as a business.

Today, before a company will put in the significant investment of engineer-hours that it requires to develop a modern messaging system, they will want to have a monetization scheme in mind. And sure, there are the SnapChats and Yos, services that make it big with no apparent revenue model. But even they understand that federation means losing control. When you have central control of a messaging service, it might just be hard to figure out how to bolt on advertising. When you abandon all control of that service, you just don’t have the option any more. It’s hard to justify from a business perspective.

This problem is even bigger when you consider that building a succesful federated system requires that kind of investment from multiple parties.

Critical Mass

To have a succesful federated system, you can’t have just one company or person that believes in the project. You need a whole federation worth of them. Consider Diaspora, an attempt at a federated social network. Sure, Diaspora still exists, and I’m sure all six users love it, but it’s a complete failure. The critical mass just never happened, in terms of users (especially important to a social network) or infrastructure.

A lot of this is a product of the competition that now exists. Why is your organization, your ISP, or whoever going to implement a server for X protocol when it’s competing with a dozen other products and there’s no assurance they won’t have to put out something else in a few months? This might be a fundamental problem with the maturity of the internet now. There just isn’t a clear single choice for most real applications, like there was back when many of these applications were being developed for the first time.

Inherent Complexity

And there is, unfortunately, an underlying technical reason as well. Creating a federated system with many stakeholders (such as many mail server operators) immediately introduces a huge degree of complexity into a system. There are few places where this is as clear as email.

Ask any mail server operator about this and they will emit a sound befitting the Eldritch horror that email has become. Even now, email is not 8-bit clean. I’d bet you that if you asked four out of five - no, nineteen out of twenty - CS majors today what “8-bit clean” meant, they wouldn’t be able to tell you. It’s just not a problem that we worry about in the modern world. But email still suffers from it!

Okay, I know it’s killing you, here’s what it means: ASCII was originally 7-bit (this was just the fewest bits the standards committee could make work). The email standards were specified for 7-bit ASCII, so even though all modern computers user 8-bit bytes, there is no guarantee when sending an email that the 8th bit of each byte will make it through intact, because there could be a machine in there somewhere that is still strictly expecting 7-bit characters. For this reason, all 8-bit content in emails goes through base64, which maps it to 7-bit ASCII characters.

And that’s just a little taste of the things wrong with email. Throw in modern spam controls (DKIM, DMARC, Detc.) and you have a system built out of layers and layers of cruft bound together with twine and Perl.

And at the very base of this stack of mattresses is the underlying pea: federated systems are hard to change. Any time that a change is made to the underlying protocols of email, every single mail server has to be upgraded to accomodate that change. This wasn’t so bad when the total number of mail servers (and computers on the internet!) numbered in the hundreds or even thousands, especially on ARPANET where there was a good bet any given mail server administrator had personally met every other mail server administrator and could show up at their homes with a weapon if they kept not fixing their open relay. But today, it’s simply impossible.

So instead, compatibility is always preserved. Each new standard is layered on top of the old. Unicode is base64 encoded into ASCII. Calendar items are attached as files. Crypto signatures are added as headers, or attachments, or as part of the body, depending on how the sender configured their MUA. Most of the time crypto signatures are just missing, though, because no one’s ever made a sufficiently standard system that mere humans could use.

The end of an era

So shed a tear for Ray Tomlinson and shed a few tears for email. We’ll be using it for a long time to come, but like the fax machine, the wind has just gone out of its sails. The future looks very different now than it did back the ’70s. The internet is no longer a global platform for communications between people, it’s a global platform for communications between people and the central services that they use.

To the users, this isn’t much different. In fact, it’s better in many ways, these central services are much more agile and competitive. But something has definitely been lost. Most people would point to privacy and reliability as huge concerns around centralizing communications services, and this is absolutely true. But I think there’s something else, as well.

I think a lot of the fun has just gone out of it.