bundesliga2go with mongodb, zeromq, gevent, web.py and websockets

bundesliga2go (switch to mongo branch) is a project which I started with Vlad so that we could look at the german Bundesliga results on a smartphone. It was originally our Hackweek 6 project. The results are provided by the webservice available at OpenLigaDB. They are provided in XML format.

The first thing we noticed was that it was going to be difficult and slow to just write a mobile web client which would query OpenLigaDB every time that a result or update was required. That is because when matches are in progress (typically Saturday at 15:30 german time) OpenLigaDB can be quite sluggish. Also, it seemed ridiculous to query already available and final results over a live internet connection with a mobile phone. On top, it can get expensive. We figured we could provide a better solution using modern smartphone browsers' offline mode.

So, the first thing to do was to cache the responses from OpenLigaDB. This was done by creating a suds (SOAP client for Python) synchronise script which simply asked OpenLigaDB for match data and stored this in a SQLite database. We then created an API (Python) for accessing the local database and for ensuring it was kept up to date with OpenLigaDB (we used APScheduler to schedule updates), especially when matches were in progress. The mobile client frontend was done with jQuery mobile. The actual web server itself was done with web.py.

Before long it became apparent that SQLite (or any relational database) might not be the most appropriate storage solution. The kind of data we were storing was more suited to a document type storage form so we moved to mongodb. This was helpful because we could now use techniques such as map/reduce to quickly compile the Bundesliga table and top scorers.

We implemented a first working draft by having the web.py server return JSON to the mobile browser. The browser stored the data using localStorage. This meant that it was only necessary to talk to the web.py server when updates were requested. Obviously, we needed to get around the security fence that browsers erect around remote JSON. We thought about using jsonp to do this but that seemed clumsy. What we came up with was CORS - the client first sends a HTTP OPTIONS request to the web.py server. The server authorises the client to use JSON returned over standard GET or POST and the client follows up immediately with the actual GET/POST request whereupon the web.py server returns the JSON data. The client then parses, stores and displays the data.

This worked fairly well until we hit the issue of live updating. On match day it is of course highly desirable to get live updates (something like 'push'). As the mobile application is HTML5 based, websockets are the obvious solution - except that they are not supported by every browser. We figured we'd provide websockets where available and try to degrade gracefully to long polling or the CORS method if necessary.

With respect to the actual live updates, what we essentially wanted to do is to push the updates to the client as soon as we stored them in the local database. This screams for an event based, non-blocking solution, so we chose zeromq for the messaging and gevent with gevent-websocket for the non-blocking websocket part. Some preliminary testing shows that actually does work as a broadcast server. More testing is necessary, though.

With websockets we are running into problems with some browsers. Testing on desktop machines, only Opera 11 supports version 7 of the websocket protocol. Firefox 7 and Chrome are already at version 8 of the protocol. The gevent-websocket library has not yet been updated to support version 8. The author is actively working on it though.

The code is available on github. Check out the mongo branch. This will be merged into master at some stage but I'm not really up to speed with git at all and I'd rather invest time in getting the application done than in having a super clean git repository.