Hosting Go code on Github with a custom import path

We host MIG at https://github.com/mozilla/mig, but while I have tons of respect for the folks at Github, I can't guarantee that we won't use another hosting provider in the future. Telling people to import MIG packages using something of the form "github.com/mozilla/mig/<package>" bothers me, and I've been looking for a better solution for a while.

I bought the domain mig.ninja with the intention to use that as the base import path. I initially tried to use HAProxy to proxy github.com, and somewhat succeeded, but it involved a whole bunch of rewrites that were frankly ugly.

Thankfully, Russ Cox got an even better solution merged into Go 1.4 and I ended up implementing it. Here's how.

Understanding go get

When asked to fetch a package, go get does a number of checks. If the target is on a known hosting site, it fetches the data using a method that is hardcoded (git for github.com, hg for bitbucket, ...). But when using your own domain, go get has no way to know how to fetch the data. To work around that, go lets you specify the vcs method in the import path: import mig.ninja/mig.git The .git indicates to go get that git should be used to retrieve the package. It also doesn't interfere with the code: in the code that import the package, .git is ignored and the package content is accessed using mig.Something.

But that's ugly. No one wants to suffix .git to their import path. There is another, cleaner, solution that use a HTML file to tell go get where the package is located, and which protocol should be used to retrieve it. The file is served from the location of the import path. For an example, let's curl https://mig.ninja/mig:

$ curl https://mig.ninja/mig
<!DOCTYPE html>
<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
        <meta name="go-import" content="mig.ninja/mig git https://github.com/mozilla/mig">
        <meta http-equiv="refresh" content="0; url=http://mig.mozilla.org">
    </head>
    <body>
        Nothing to see here; <a href="http://mig.mozilla.org">move along</a>.
    </body>
</html>

The key here is in the <meta> tag named "go-import". When go get requests https://mig.ninja/mig, it hits that HTML file and knows that "mig.ninja/mig" must be retrieved using git from https://github.com/mozilla/mig.

One great aspect of this method, aside from removing the need for .git in the import path, is that "mig.ninja/mig" can now be hosted anywhere as long as the meta tag continues to indicate the authoritative location (in this case: github). It also works nicely with packages under the base repository, such that go get mig.ninja/mig/modules/file works as expected as long as the file is served from that location as well. Note that go get will retrieve the entire repository, not just the target package.

Serving the meta tag from HAProxy

Creating a whole web server for the sole purpose of serving an 11 lines of HTML isn't very appealing. So I reused an existing server that already hosts various things, including this blog, and is powered by HAProxy.

HAProxy can't serve files, but here's the trick, it can serve a custom response at a monitoring uri. I created a new HTTPS backend for mig.ninja that monitors /mig and serves a custom HTTP 200 response.

frontend https-in
        bind 62.210.76.92:443
        mode tcp
        tcp-request inspect-delay 5s
        tcp-request content accept if { req_ssl_hello_type 1 }
        use_backend jve_https if { req_ssl_sni -i jve.linuxwall.info }
        use_backend mig_https if { req_ssl_sni -i mig.ninja }

backend mig_https
        mode tcp
        server mig_https 127.0.0.1:1666

frontend mig_https
        bind 127.0.0.1:1666 ssl no-sslv3 no-tlsv10 crt /etc/certs/mig.ninja.bundle
        mode http
        monitor-uri /mig
        errorfile 200 /etc/haproxy/mig.ninja.200.http
        acl mig_pkg url /mig
        redirect location https://mig.ninja/mig if !mig_pkg

The configuration above uses SNI to serve multiple HTTPS domains from the same IP. When a new connection enters the https_in frontend, it inspects the server_name TLS extension and decides which backend should handle the request. If the request is for mig.ninja, it sends it to the mig_https backend, which forward it to the mig_https frontend. There, the request URI is inspect. If it matches /mig, the file mig.ninja.200.http is returned. Otherwise, a HTTP redirect is returned to send the caller back to https://mig.ninja/mig (in case a longer path, such as mig.ninja/mig/module, was requested).

mig.ninja.200.http is a complete HTTP response, with HTTP headers, body and proper carriage return. HAProxy doesn't process the file at all, it just globs it and sends it back to the client. It looks like this:

HTTP/1.1 200 OK
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<meta name="go-import" content="mig.ninja/mig git https://github.com/mozilla/mig">
<meta http-equiv="refresh" content="0; url=http://mig.mozilla.org">
</head>
<body>
Nothing to see here; <a href="http://mig.mozilla.org">move along</a>.
</body>
</html>

Building MIG

With all that in place, retrieving mig is now as easy as go get mig.ninja/mig. Icing on the cake, the clients can now be retrieved with go get as well:

$ go get -u mig.ninja/mig/client/mig
$ mig help
$ go get -u mig.ninja/mig/client/mig-console
$ mig-console

Telling everyone to use the right path

We just this in place, you can't guarantee that users of your packages won't directly reference the github import path.

Except that you can! As outlined's in rsc's document, and in go help importpath, it is possible to specify the authoritative location of a package directly in the package itself. This is done by adding a comment with the package location right next to the package name:

package mig /* import "mig.ninja/mig" */

Using the import comment, Go will enforce retrieval from the authoritative source. Other tools will use it as well, for example try accessing MIG's doc from Godoc using the github URL, and you will notice the redirection: https://godoc.org/github.com/mozilla/mig

Trust Github, it's a great service, but controlling your import path is the way to Go ;)

Quelques digressions sous GPL