On March 11th, 2005, a small group of nerds studying in the overly boring city of Blois, France, decided to buy a domain to play with self hosting DNS and email.

pict2331.jpg

10 years later, linuxwall.info is still (mostly) self-hosted and we are (mostly) nerds. It has been, and still is, a lot of fun and I believe the learning process has to progress two to three times faster than my peers who don't self-host.

(on the left: probably the first server to host linuxwall.info, back in my apartment in Blois)

10 years is probably long enough of an experiment to draw lessons on the state of self-hosting. So let's start with the conclusion: self-hosting is entirely possible, but only if you invest the time and energy to do it right. There is no good way to self-host without spending time on it. Symptoms of poorly done self-hosting include loss of email, website headaches, connectivity issues and angry spouse.

What follows is a rapid overview of what I learned doing self-hosting for a decade, starting with the most frustrating aspect of it...

 

Don't count on your ISP for help

Over the last 10 years, the people who own the pipes have made absolutely zero effort to facilitate self-hosting. I've hosted linuxwall.info on Free.fr, Neuf/SFR, Comcast and Verizon. The only one who provided me with a static IP was Free.fr. Comcast went as far as blocking tcp/25 inbound without providing a way to disable the blocking. ISPs are completely uncooperative with self-hosters, and for no particular reason other than pushing people toward "business" class of bandwidth (same exact thing, but with a static IP and an extra $100 on your monthly bill).

Uplink is probably what self-hosters fear the most. Will it be enough bandwidth? Will it slow down the Internet for the rest of the house? Here is the truth: you need almost no bandwidth to self-host. For many years, linuxwall.info was hosted with 128kbps of uplink, and it worked fine. The handful of inbound email, http connections, xmpp chats or DNS requests will fit into a tiny percentage of your uplink, and you won't even notice it. Just make sure you don't run offsite backups while your spouse is watching netflix... (or use QoS).

 

Self-hosters need friends

Don't self-host alone, that won't work. The chances of your internet connection going down are too great. Self-hosting must be done the way the internet was built: if one endpoint goes down, there must be another endpoint that takes over. For DNS, that's easy enough with slaves in multiple locations. For web or email, it's more difficult.linuxwall_crew_2005.jpg

(on the right: the linuxwall.info crew: steph, franck, christophe, myself and jerome, circa 2005)

Hosting email servers is definitely the hardest, because building an IMAP distributed cluster with something like dovecot or cyrus-imap is far from trivial. However, it is fairly easy to build a secondary MX with postfix that buffers inbound emails when the primary is down, and just forwards them to the primary when it comes back online.

That's how linuxwall's MX operates. smtp.linuxwall.info is a primary email server (described here), and if it goes down, smtp2.linuxwall.info will receive the inbound mail, cache it, and forward it to the primary when back online. SMTP2 is really just running Postfix with a very basic transport configuration, so we avoid the madness of synchronizing IMAP datastores across servers.

$ dig +short MX linuxwall.info
20 smtp2.linuxwall.info.
10 smtp.linuxwall.info.
root@smtp2:/etc/postfix # cat transport
linuxwall.info  relay:smtp.linuxwall.info:25

DNS is another one that Nowadays, linuxwall's DNS servers host more than one domain. Besides the main one, there is 1nw.eu, chatonly.org, frimousse-action.com, insecure.ws, lesroutesduchocolat.{fr,com}, tzib.net, necto.org and a few more. The root DNS is hosted in my house, but on a dynamic IP, so it is only used as an authority to the 3 slaves that have static IPs. DNS outages are rare because of the distributed nature of the platform, but it does require that several people participate in the network. Don't self-host alone!

linuxwall_dns.svg

 

Do hard things and write about it

DNS, Email, XMPP, distributed storage, VPN, public websites, ... The hardest things to self-host are also the most interesting. After many years of internet presence, I have enough public websites to warrant paying for a hosted server at online.net. Most of the critical parts of my infra (email, dns slave, websites) are now hosted on linux containers on that host. For a while though, everything that touched the linuxwall.info domain was hosted on personal ADSL connections (the french equivalent to cable).

Linuxwall.info was created to experiment with all sorts of cool technologies that we simply didn't have access to as students. One of the goal was to write about these technologies, and logically the first site we created was http://wiki.linuxwall.info. Over the years, we've written about dozens of tools, setups, successes and failures and some of these articles have become quite popular:

As it turns out, most of the tech we experimented with is identical to what is deployed in "professional" environments. Same Apache conf, same postfix, same VPN, same DKIM, same RAID, etc... Sometimes, self-hosting is even more advanced that what you'd find in your typical LAMP stack at work, because the constraints on hosting stuff at home require a good amount of engineering creativity.

Most certainly, self-hosting helped me become a better engineer. It helped me acquire experience much more quickly than if I had waited for exposure at work. It does come with the cost of many and many weekends spent deploying, fixing, upgrading and operating an infrastructure that could just as well been hosted elsewhere. For free. But without the satisfaction or making it all work myself.

 

Control the traffic

QoS is perhaps the most interesting networking challenge a self-hoster can take upon. I have spent hours tweaking the QoS of my gateway server, and wrote a lengthy analysis of Linux's QoS stack at Journey to the Center of the Linux Kernel: Traffic Control, Shaping and QoS. Studying QoS not only helps understand the details of network protocols, but also made me appreciate minimalistic designs: almost all network services that run on the interbits_sent-1week.pngnet today can be hosted with just a few kilobytes per seconds of bandwidth. The myth of gigabits internet connections for residences is built by ISPs as a marketing tool. Even at pick times, self-hosting at home while doing video-conferences does not make my uplink use more than one megabit of traffic. And I verizon pay for 25 of those!

(on the left: a week of uplink from my house, using up to 720kbps)

I can't remember a time when I filled up an uplink during normal operations. Sure, there are times when a large uplink is pleasant - for example when recovering 20GB of email backups from a friend's server - but 99% of the time a basic QoS policy will ensure that your services have the necessary bandwidth without annoying all the users in the house.

 

The takeaway

Most people get exhausted of self-hosting after a few years, give up and move everything to "the cloud". We haven't, and it's been 10 years. The secret is to take the time to do things right, and not do it alone.

For sure, there will be times of reading your emails directly from the cache of Postfix, because the IMAP server is down Or times when you arrive at a nice vacation spot only to realize that the IP of your home server has changed, and you can't update the DNS for a week. Those times suck. But they also help you think about reliability and high-availability. Stuff that, once you've acquired the knowledge, people will pay a lot of money for.

tl;dr: self-hosting is awesome. 10 years in and going strong, Linuxwall.info is reading for the next 10!

Screenshot_-_03112015_-_11_41_57_AM.png