Federation: The future of open online services, and the war against it

To clarify, by “federation”, I mean federation in contrast to a client-server or P2P models. Specifically, a collection of independent servers, each serving a number of users, all using a standardized, federated protocol.

Let’s go over the basics real quick. At first, there was the client-server model.

client-server-modelThere is a server that communicates directly with a bunch of clients. For instance, when you open up Facebook, you’re using the client-server model (yes, Facebook has multiple servers, but they’re all owned and controlled by Facebook, so in this context, it’s essentially a single server).

Then, around the time of Napster, we realized that it might be a good idea to take servers out of the equation. This introduced the peer-to-peer model.

peer-to-peer-modelThis distributed model offered us decentralization — a network that can’t be destroyed by removing a single server out of the middle. It’s a network that’s a collection of peers with no central authority. Protocols like BitTorrent use the P2P model. However, there is a downside to the P2P model: each of the peers have a fairly high processing cost, and are usually expected to be constantly connected to the network.

So what’s the in-between? Federation: a collection of independent servers, each serving a number of clients.

federation-model“Independent” is the important word here: the idea is that anyone can host their own server, and can join it to the network of servers by using an agreed-upon, or “federated”, protocol. This allows us to have an open network (unlike, say, Facebook’s servers) while not burdening the clients with all the processing. An excellent example of the federation model is email: an email server can be run by your ISP, your company, an online ad-supported service, or you can run one yourself. Multiple clients connect to each server (ie all of your ISP’s customers), and the servers can talk to each other via an established protocol (SMTP). There is no central authority in the email system: your little home server has, by design, the same “say” in the network as Gmail’s servers.

The federated model, while being old tech, is still the best compromise between client-server and P2P models. It enforces an open network, gives us the option to completely own our data, while still leaving room for our corporate peers. However, there’s been an increasing trend away from federated models by several large service providers.

XMPP is an instant messaging protocol (it’s actually a lot more than just an IM protocol, but that’s not important here) which uses the federated model. Users connect to the server for their domain, and they can chat with users on different domains via server-to-server communication. Google Talk (aka Hangouts) implemented XMPP support in 2006. The idea was that a user on Google Talk, say, “user1@gmail.com” could chat with a remote user, like “user2@jabber.com” without user2 having to create a Gmail account. This was a Good Thing, because it let users have choice of IM providers, while still letting users on different networks chat with each other. In 2010, Facebook Chat added XMPP support. This was also a good thing, for an additional 400 million accounts could be reached via XMPP. It looked like XMPP was going to get as popular as email.

But then… it all fell apart. Both Google and Facebook dropped XMPP support in late 2014 / early 2015. There was never much of an explanation from either corporation, just something along the lines of “We’re switching to X new API and we didn’t bother adding XMPP support” and “We promise we might eventually one day look at maybe adding something resembling XMPP support. Maybe.”

evil-corp-noir

So what actually happened? They realized the business value of vendor lock-in. Effectively, “we know that user1 wants to chat with user2. Why make it easy for user2 to chat externally? We can just force them to join to chat with user1, giving us more product a new client!” You want to chat with grandma over Facebook Chat? Too bad, you’ll have to make an FB account now… even though perfectly good tech exists to let you chat with her from wherever.

But that’s just XMPP. We have slightly bigger issues to worry about.

It’s becoming increasingly more difficult to run your own email server. If you set up a new email server on a dedi/VPS somewhere, and follow all the usual recommended practices (PTR, SPF records, DKIM), your emails will be put in the Spam bin on both Gmail and Microsoft inboxes. Gmail will direct you to read their Bulk Sender Guidelines (even if you only sent a single email), and Microsoft will give you a place to “register” your server for a chance to avoid the spam bin. In order to avoid the spam bin on Gmail, however, you’ll need to build up a “reputation”, by having conversations with numerous Gmail accounts, and having them mark you as “Not Spam”. Here are some HN threads griping about this issue. To make matters worse, having some reputation with Gmail doesn’t guarantee that your email will get delivered: the Gmail servers will accept your email, but they may still end up in the user’s spam bin or disappear completely, especially if this is the first time you’ve emailed this particular user.

This is a ridiculous amount of effort for a user who just wants to run their own email server because, oh I dunno, maybe they want to actually own their emails? This is also extremely concerning, because email is the original example of an open and decentralized system. I suspect that, within the next few years, it’ll gradually become impossible for anyone to run a small email server. Eventually, you’ll use your work email to talk to your work colleageues, your Gmail to talk to your friends on Gmail, and your Outlook to talk to your friends on Outlook. Here’s vendor lock-in again — why let you run your own server when you can maintain three accounts (and, of course, see ads from all three providers).

5 thoughts on “Federation: The future of open online services, and the war against it”

    1. I lost a disk and that whole blog went with it. That was right after win10 came out though — there are better write-ups on the subject now, so nothing of value was lost.

  1. Legendary! This is just incredible! Not merely top quality, however additionally important details. Which is unusual ahead by presentlies! I need to state that I am actually pleased and also will surely return once more if you maintain the top quality as well as value of the material at this degree, or maybe acquire it on the next level. Truly, from all-time low of my heart, thank you for your time! God bless!

  2. Fuck all you hack job weirdo lames u watch people threw there phone u fucks don’t have anything better to do, oh yeah like sell child poronography threw Facebook and terioize people. Well check it out u stupid fucks I give a dam dout what u try to believe that its untraceable how u nerds r beating the system, foreal? U pittyful people, u think the man is going to make something that he gives to us that’s smarter than him. Lame duck please ur done ur caught if it transmits it recevieved …so have fun in prison, cause my phone is hacked by butches just like u if not u. Thanks can u say done, washed will do ,fucku die in PC, protective custody that is. Bitches

Leave a Reply

Your email address will not be published.