As for almost everything else, I run my own XMPP server using ejabberd. The setup has been stable for years, and I've never had any problem with it. Up until a few months ago when my contacts at gmail.com stopped being reachable.

I didn't really look into the issue up until recently, assuming that gmail had changed their s2s policy and did not peer with random xmpp hosts anymore. But today, I decided to finally look into it, and review the configuration of the ejabberd server, as well as DNS.

The XMPP server at linuxwall.info uses port 5269 for s2s connections. But that's for incoming traffic only. For outgoing traffic, the ejabberd server simply calls the s2s server of the recipient's domain. After checking that all sockets were open and listening, I decided to fire up my favorite tool: tcpdump. I dumped all tcp traffic on port 5269 on the ejabberd server, while adding a gmail.com buddy in my roster. And this is what I saw:

Request:

17:37:13.242041
  IP (tos 0x0, ttl 64, id 49542, offset 0, flags [DF], proto TCP (6), length 221)
  192.168.0.1.56084 > 173.194.78.125.5269:
    Flags [P.], cksum 0xba80 (correct), seq 847045431:847045600, ack 2715810876,
    win 92, options [nop,nop,TS val 1835999932 ecr 1080996834], length 169
    E.....@.@.........N}....2|.7...<...\.......
    mo".@n..<?xml version='1.0'?>
    <stream:stream xmlns:stream='http://etherx.jabber.org/streams'
    xmlns='jabber:server' xmlns:db='jabber:server:dialback'
    to='gmail.com' version='1.0'>

Response:

17:37:13.326198
  IP (tos 0x0, ttl 47, id 24874, offset 0, flags [none], proto TCP (6), length 297)
  173.194.78.125.5269 > 192.168.0.1.56084
    Flags [P.], cksum 0x27aa (correct), seq 2715811017:2715811262, ack 847045669,
    win 239, options [nop,nop,TS val 1080996919 ecr 1835999943], length 245
    E..)a*../.l...N}............2|.%....'......
    @n.7mo".
    <stream:error>
      <undefined-condition xmlns="urn:ietf:params:xml:ns:xmpp-streams"/>
      <str:text xmlns:str="urn:ietf:params:xml:ns:xmpp-streams">
        linuxwall.info is a Google Apps Domain with Talk service enabled.
      </str:text>
    </stream:error></stream:stream>

"linuxwall.info is a Google Apps Domain with Talk service enabled."...

This is when it hit me: months ago, I registered linuxwall.info as a google app domain ! Our MX was down for a week, and I redirected all traffic to google app. I didn't bother deleting the domain afterward, in case this would happen again. But Google has its own security policy, and if a domain is registered in google app, it assumes that it should not send XMPP traffic to any other server that itself. It kind of makes sense.

Deleting the google app account fixed the issue, and ejabberd is now happily exchanging s2s traffic with google.

And, once again, tcpdump saved my day !