Excel, Python and Python-Fu

My current place of work uses a good deal of Excel to store and represent results from experiments. It’s pretty common when you see Excel used like this to realise that the right application is really a database, but a proper RDMS just doesn’t give the flexibility or friendliness of randomly adding columns and hoiking groups of cells here and there with the mouse.

Anyway, scripting this stuff usually involves dumping it as comma separated value (another poor man’s database) importing and then playing with it. I can see I’m going to be a CSV master within a pretty short space of time.

My latest user-time-saving hack is to try and figure out how to take an Excel document with many “sheets” (or tabs) and dump each one into an individual CSV file, for separate. Naturally, there is a newsgroup dedicated to this kind of hackery, and (thank the maker) it’s Python based. I used xlrd for this, which is as intuitive as I have come to expect from Python libraries.

The resulting code works fine, but is probably prohibitively slow. The files we work with are around 70MB, which takes 30 seconds for xlrd to read. The discussion on python-excel also led me to psycho a Python speed-up magic bullet that no programmer has a right to expect. As it turns out I can’t expect it, because it doesn’t run on 64 bit systems. The successor pypy looks interesting (scary) but not quite the same.

My discussion on python-excel once again brought to my attention to bizarre willingness of strangers to improve and extend my code for me. Also, the willingness of people on Ubuntu forums to debug problems with my machine I could probably fix myself, except I’m too lazy.

Am I supposed to “give something back to the community” at some point? Will telling my dad I think Ubuntu is pretty cool count?


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: