Federation: The future of open online services, and the war against it

To clarify, by “federation”, I mean federation in contrast to a client-server or P2P models. Specifically, a collection of independent servers, each serving a number of users, all using a standardized, federated protocol.

Let’s go over the basics real quick. At first, there was the client-server model.

client-server-modelThere is a server that communicates directly with a bunch of clients. For instance, when you open up Facebook, you’re using the client-server model (yes, Facebook has multiple servers, but they’re all owned and controlled by Facebook, so in this context, it’s essentially a single server).

Then, around the time of Napster, we realized that it might be a good idea to take servers out of the equation. This introduced the peer-to-peer model.

peer-to-peer-modelThis distributed model offered us decentralization — a network that can’t be destroyed by removing a single server out of the middle. It’s a network that’s a collection of peers with no central authority. Protocols like BitTorrent use the P2P model. However, there is a downside to the P2P model: each of the peers have a fairly high processing cost, and are usually expected to be constantly connected to the network.

So what’s the in-between? Federation: a collection of independent servers, each serving a number of clients.

federation-model“Independent” is the important word here: the idea is that anyone can host their own server, and can join it to the network of servers by using an agreed-upon, or “federated”, protocol. This allows us to have an open network (unlike, say, Facebook’s servers) while not burdening the clients with all the processing. An excellent example of the federation model is email: an email server can be run by your ISP, your company, an online ad-supported service, or you can run one yourself. Multiple clients connect to each server (ie all of your ISP’s customers), and the servers can talk to each other via an established protocol (SMTP). There is no central authority in the email system: your little home server has, by design, the same “say” in the network as Gmail’s servers.

The federated model, while being old tech, is still the best compromise between client-server and P2P models. It enforces an open network, gives us the option to completely own our data, while still leaving room for our corporate peers. However, there’s been an increasing trend away from federated models by several large service providers.

<a href="https://en.wikipedia have a peek at these guys.org/wiki/XMPP” target=”_blank”>XMPP is an instant messaging protocol (it’s actually a lot more than just an IM protocol, but that’s not important here) which uses the federated model. Users connect to the server for their domain, and they can chat with users on different domains via server-to-server communication. Google Talk (aka Hangouts) implemented XMPP support in 2006. The idea was that a user on Google Talk, say, “user1@gmail.com” could chat with a remote user, like “user2@jabber.com” without user2 having to create a Gmail account. This was a Good Thing, because it let users have choice of IM providers, while still letting users on different networks chat with each other. In 2010, Facebook Chat added XMPP support. This was also a good thing, for an additional 400 million accounts could be reached via XMPP. It looked like XMPP was going to get as popular as email.

But then… it all fell apart. Both Google and Facebook dropped XMPP support in late 2014 / early 2015. There was never much of an explanation from either corporation, just something along the lines of “We’re switching to X new API and we didn’t bother adding XMPP support” and “We promise we might eventually one day look at maybe adding something resembling XMPP support. Maybe.”

evil-corp-noir

So what actually happened? They realized the business value of vendor lock-in. Effectively, “we know that user1 wants to chat with user2. Why make it easy for user2 to chat externally? We can just force them to join to chat with user1, giving us more product a new client!” You want to chat with grandma over Facebook Chat? Too bad, you’ll have to make an FB account now… even though perfectly good tech exists to let you chat with her from wherever.

But that’s just XMPP. We have slightly bigger issues to worry about.

It’s becoming increasingly more difficult to run your own email server. If you set up a new email server on a dedi/VPS somewhere, and follow all the usual recommended practices (PTR, SPF records, DKIM), your emails will be put in the Spam bin on both Gmail and Microsoft inboxes. Gmail will direct you to read their Bulk Sender Guidelines (even if you only sent a single email), and Microsoft will give you a place to “register” your server for a chance to avoid the spam bin. In order to avoid the spam bin on Gmail, however, you’ll need to build up a “reputation”, by having conversations with numerous Gmail accounts, and having them mark you as “Not Spam”. Here are some HN threads griping about this issue. To make matters worse, having some reputation with Gmail doesn’t guarantee that your email will get delivered: the Gmail servers will accept your email, but they may still end up in the user’s spam bin or disappear completely, especially if this is the first time you’ve emailed this particular user.

This is a ridiculous amount of effort for a user who just wants to run their own email server because, oh I dunno, maybe they want to actually own their emails? This is also extremely concerning, because email is the original example of an open and decentralized system. I suspect that, within the next few years, it’ll gradually become impossible for anyone to run a small email server. Eventually, you’ll use your work email to talk to your work colleageues, your Gmail to talk to your friends on Gmail, and your Outlook to talk to your friends on Outlook. Here’s vendor lock-in again — why let you run your own server when you can maintain three accounts (and, of course, see ads from all three providers).