Making the seeeduino stalker (v1) work with avrdude and a Makefile

I got one of these Seeeduino Stalker v1 boards recently, for $22 as they were having a fire sale on them over on Seeeduino Studio to make way for version 2.0. The board carries an Atmel 328 and has an RTC chip plus a micro SD card already soldered on.

When I got the board, I got an UARTSBee programmer to go with it. This is basically just a USB to Serial adapter (FTDI chip). I got the version 4.0 board which unfortunately has 6 pins outgoing, which makes it incompatible with the 5 pin input of the Stalker. It is, however, easily possible to solder 5 pins onto the holes provided directly behind the 6 pin output. Then the programmer can be stuck straight on to the Stalker's input port.

To get started with programming the Stalker I downloaded the arduino software from the CrossAVR repository on the openSUSE build service. This is a java based GUI which simply provides an editor for text files and uses gcc to compile and avrdude to upload the software to the arduino (the Stalker). Using the arduino GUI was quite easy and uploading worked without a problem.

However, I wanted to use a standard Makefile and avrdude to flash the software to the Stalker. This turned out to be a bit more complicated than I thought because the arduino GUI knew that it was necessary to pull the DTR before flashing the software. This had the effect of resetting the Stalker, putting it into the bootloader. Only then will the Stalker accept a firmware flash. What confused me was that by running strace on the arduino GUI I saw that it was calling avrdude with -p stk200v1. When I tried this directly with avrdude it didn't work (because avrdude wasn't pulling on the DTR line to reset the Stalker so it would spring into the bootloader and accept a flash). It turns out that avrdude has already thought of this and provides a -p arduino option. Using this worked out of the box. Below is a video of the Stalker with the UARTSBee programmer stuck on vertically. Attached to it is a breadboard with some LEDS just to show the pins set to output and blinking.

bundesliga2go with mongodb, zeromq, gevent, web.py and websockets

bundesliga2go (switch to mongo branch) is a project which I started with Vlad so that we could look at the german Bundesliga results on a smartphone. It was originally our Hackweek 6 project. The results are provided by the webservice available at OpenLigaDB. They are provided in XML format.

The first thing we noticed was that it was going to be difficult and slow to just write a mobile web client which would query OpenLigaDB every time that a result or update was required. That is because when matches are in progress (typically Saturday at 15:30 german time) OpenLigaDB can be quite sluggish. Also, it seemed ridiculous to query already available and final results over a live internet connection with a mobile phone. On top, it can get expensive. We figured we could provide a better solution using modern smartphone browsers' offline mode.

So, the first thing to do was to cache the responses from OpenLigaDB. This was done by creating a suds (SOAP client for Python) synchronise script which simply asked OpenLigaDB for match data and stored this in a SQLite database. We then created an API (Python) for accessing the local database and for ensuring it was kept up to date with OpenLigaDB (we used APScheduler to schedule updates), especially when matches were in progress. The mobile client frontend was done with jQuery mobile. The actual web server itself was done with web.py.

Before long it became apparent that SQLite (or any relational database) might not be the most appropriate storage solution. The kind of data we were storing was more suited to a document type storage form so we moved to mongodb. This was helpful because we could now use techniques such as map/reduce to quickly compile the Bundesliga table and top scorers.

We implemented a first working draft by having the web.py server return JSON to the mobile browser. The browser stored the data using localStorage. This meant that it was only necessary to talk to the web.py server when updates were requested. Obviously, we needed to get around the security fence that browsers erect around remote JSON. We thought about using jsonp to do this but that seemed clumsy. What we came up with was CORS - the client first sends a HTTP OPTIONS request to the web.py server. The server authorises the client to use JSON returned over standard GET or POST and the client follows up immediately with the actual GET/POST request whereupon the web.py server returns the JSON data. The client then parses, stores and displays the data.

This worked fairly well until we hit the issue of live updating. On match day it is of course highly desirable to get live updates (something like 'push'). As the mobile application is HTML5 based, websockets are the obvious solution - except that they are not supported by every browser. We figured we'd provide websockets where available and try to degrade gracefully to long polling or the CORS method if necessary.

With respect to the actual live updates, what we essentially wanted to do is to push the updates to the client as soon as we stored them in the local database. This screams for an event based, non-blocking solution, so we chose zeromq for the messaging and gevent with gevent-websocket for the non-blocking websocket part. Some preliminary testing shows that actually does work as a broadcast server. More testing is necessary, though.

With websockets we are running into problems with some browsers. Testing on desktop machines, only Opera 11 supports version 7 of the websocket protocol. Firefox 7 and Chrome are already at version 8 of the protocol. The gevent-websocket library has not yet been updated to support version 8. The author is actively working on it though.

The code is available on github. Check out the mongo branch. This will be merged into master at some stage but I'm not really up to speed with git at all and I'd rather invest time in getting the application done than in having a super clean git repository.

OpenId login with web.py fast easy simple example

#!/usr/bin/env python
import web
from web import webopenid
urls = (
'/', 'index',
'/openid', 'webopenid.host',
)

app = web.application(urls,globals())

def form(openid_loc):
  oid = status()
  if oid:
    return '''
<form action="%s" method="post">

<img alt="OpenID" src="http://openid.net/login-bg.gif"
/>
<b>%s</b>
<input name="action" type="hidden" value="logout" />
<input name="return_to" type="hidden" value="%s" />
<button type="submit">log out</button></form>
''' % (openid_loc, oid, web.ctx.fullpath)
  else:
    return '''
<form action="%s" method="post">
<input name="openid" style="background:
url("http://openid.net/login-bg.gif")
no-repeat scroll 0% 0% transparent;
padding-left: 18px;" type="text" value="" />
<input name="return_to" type="hidden" value="%s"
/>
<button type="submit">log in</button></form>
''' % (openid_loc, web.ctx.fullpath)

class index:
  def GET(self):
    oid = webopenid.status()
    if not oid:
      return 'please log in: ' + \
        webopenid.form('/openid')
    else:
      return 'you are logged in as:' + \
              webopenid.form('/openid')

if __name__ == '__main__':
  app.run()

Some thoughts on startiing hacking on the AVR Butterfly

Connecting the Butterfly
Without knowing much about the Butterfly, it was difficult to get things working at first. The Butterfly comes with pre-installed firmware which shows off the features. There are options like entering and displaying your name on the LED screen (the Butterfly actually comes with a  safety pin so you can use it as a name label on your shirt), displaying temperature (I think) and changing and displaying the date.

I figured it would be best to make an initial back up copy of the original Butterfly firmware. This wasn't particularly easy. I used a Fritzl ASP initially (modified by jw to handle shifting the TTL +5v +0v to a quasi RS232 +12v -12v). With avrdude I was able to retrieve and save the original firmware. The command and options were:

avrdude -p atmega169 -b 19200 -c butterfly -P /dev/ttyUSB0 -U flash:r:flash.hex:i

But in order to be able to actually talk to the Butterfly, you need to get it into a state where the bootloader is active (i.e. that the Butterfly doesn't just jump into the firmware's main()). The trick here is to reset the Butterfly (short the two contacts on the extreme bottom right of the Butterfly as you look at it from the front) and then immediately press the joystick down (take care not to waggle it here - otherwise it won't work) and within a couple of seconds enter the avrdude command above in the console. avrdude should tell you that it connected successfully to the programmer and was able to save the flash.hex file (you may need to touch the flash.hex file beforehand).

Things to watch out for
When connecting using a serial cable over the Butterfly UART pins, I followed the diagram in the Butterfly User's Manual. You need to remember that the serial connector shown in that diagram is the 'male' connector - the one with the pins. If you (like me) have the other connector - the 'female' side, then you are going to have to 'mirror' the diagram. You would therefore have the GND at the extreme left of the upper row (where the upper row has 5 holes) and you would have to find out accordingly where the RX and TX holes are.

Another thing to look out for when connecting the serial connector to the UART pins is that some serial cables are 'crossover' cables, designed to connect PC to PC. If this is the case, you will have to swap RX and TX when connecting to UART. If nothing is working, try changing the connections around. That's how I got it working (after about two hours wondering why nothing was happening).


A quick first firmware for the Butterfly

1  #define F_CPU 1000000UL // how fast is CPU?
2  #include
3  #include
4  int main()
5  {    DDRB |=(1<
6  for(;;)
7    {
8      PORTB &= ~(1<
9      _delay_ms(200.0);
10      PORTB |= (1<
11     _delay_ms(200.0);
12    }
13    return 0;
14 }

TurboGears 2.1b1 released and updated on the openSUSE build service

One-Click Install TurboGears 2.1?

You can now One-Click Install TurboGears 2.1 Beta 1 directly from the openSUSE build service. Just click here!

2.1b1: (**January, 25 2009**) ¶

• Deprecated default in favor of _default
• Fixed handling of Unicode parameters
• Added disable_request_extensions flag to configuration to allow users to ignore the request extension dispatch bits of object dispatch.
• Increased length of Permission.permission_name to 63 chars.
• Fixed GET requests on nested RestControllers?.
• Fixed case-sensitive incorrectness in quickstart when using mako template option.
• Fixed numerous URL routing bugs, consilidating the RC and TGC controller base.
• Fixed eroneous tg.url call inside the quickstart template.
• Added use_dotted_templatenames support for Genshi.
• Added ignore_parameters setting in the config
• Added ability to have sa_auth.cookie_secret in .ini file
• Added cookie_secret to quickstart template
• Added tgext template
• Added backwards compat for TurboJson?

File uploads using with ajax using jquery.form and TurboGears 2.1

While trying to follow the ajax file upload example from http://jquery.malsup.com/form/#file-upload I was running into trouble with the server response. I was expecting the server to return json, which I wanted to parse in a callback function. What happened was that the server did its job perfectly and returned content-type application/json. However, the browser (in this case Firefox 3.5) prompted me to download the response, rather than to parse it in the callback function. This is obviously useless.

The answer was to return content-type text/plain instead of application/json. To do this in TurboGears 2.1, I followed an example from Pylons. My controller now looks like this:

   @expose()
    def uploadFile(self,**kwargs):
        """ Simple controller to handle uploaded files

        """
        # do stuff such as handle file upload...
        response.headers['Content-type'] = "text/plain"
        return dict(info={'filesize':50000,'filename':'whatever')

Note the response.headers['Content-type'] = "text/plain" - and that I left out the expose('json') which you would normally use in TurboGears to return json...

TurboGears2 application using Apache and mod_wsgi

Here are the steps I went through to get a TurboGears2 application running using Apache and mod_wsgi. It assumes that apache2 and mod_wsgi are configured and properly set up (don't forget to a2enmod wsgi in order to load the wsgi module). There is a good example documented at http://www.turbogears.org/2.0/docs/main/Deployment/modwsgi+virtualenv.html

Install the required packages:

zypper in apache2
zypper in apache2-mod_wsgi
zypper in python-wsgideploy

Create a new turbogears application using paster. This one is called labum

paster quickstart labum

Go into the newly created application root folder

cd labum

Create an apache wsgi setup for the application

paster modwsgi_deploy

Go into the newly created folder 'apache'

cd apache

Edit the labum.wsgi file to suit your environment. I quickstarted the labum application in the
folder /home/cfarrell/Dokumente/labum, so I need to remove the default references to /usr/local/turbogears. I do this in vi using

:%s/\/usr\/local\/turbogears7\/home\/cfarrell\/Dokumente

Copy the the file 'labum' in the 'apache' directory to the apache2 configuration directory. On openSUSE this worked (all *.conf files in /etc/apache2/vhosts.d/ are auto loaded by apache):

cp /home/cfarrell/Dokumente/labum/apache/labum /etc/apache2/vhosts.d/labum.conf

Go back into your turbogears application directory:

cd /home/cfarrell/Dokumente/labum

Copy your development.ini file to production.ini and remove the debugging by setting
debug = False in the first couple of lines

Reload apache and go to http://localhost/labum. If everything worked, you should now be able to see your TurboGears application. If it didn't, you should see what happened by looking at

/var/log/apache2/errors.log