Archive for August, 2008

Excel, Python and Python-Fu

August 29, 2008

My current place of work uses a good deal of Excel to store and represent results from experiments. It’s pretty common when you see Excel used like this to realise that the right application is really a database, but a proper RDMS just doesn’t give the flexibility or friendliness of randomly adding columns and hoiking groups of cells here and there with the mouse.

Anyway, scripting this stuff usually involves dumping it as comma separated value (another poor man’s database) importing and then playing with it. I can see I’m going to be a CSV master within a pretty short space of time.

My latest user-time-saving hack is to try and figure out how to take an Excel document with many “sheets” (or tabs) and dump each one into an individual CSV file, for separate. Naturally, there is a newsgroup dedicated to this kind of hackery, and (thank the maker) it’s Python based. I used xlrd for this, which is as intuitive as I have come to expect from Python libraries.

The resulting code works fine, but is probably prohibitively slow. The files we work with are around 70MB, which takes 30 seconds for xlrd to read. The discussion on python-excel also led me to psycho a Python speed-up magic bullet that no programmer has a right to expect. As it turns out I can’t expect it, because it doesn’t run on 64 bit systems. The successor pypy looks interesting (scary) but not quite the same.

My discussion on python-excel once again brought to my attention to bizarre willingness of strangers to improve and extend my code for me. Also, the willingness of people on Ubuntu forums to debug problems with my machine I could probably fix myself, except I’m too lazy.

Am I supposed to “give something back to the community” at some point? Will telling my dad I think Ubuntu is pretty cool count?

Sound cards and Linux

August 22, 2008

I’ve been working with Linux for about 10 years now and it’s come on a lot since then. My new machine for my job here in Pathology is a spiffy Dell machine running Ubuntu 64 and so far setting it up has been largely painless.

The one problem I’ve had is with the high-end Creative X-Fi Sound Card. I haven’t looked into it much up to now, but today I found a page which explained where to find the drivers and get them to work. It was almost reassuring to find the old Linux I remember from the late 90s is still alive and well.

The instructions on getting the driver to work are found on a random Wiki here:

http://gentoo-wiki.com/HOWTO_Creative_Labs_X-Fi

They start by saying what a great card this is and with all these features, then almost immediately tells you that most of these features will never work under Linux. It then presents the following mild boilerplate:

THIS IS A COMPLETE HACK!
Warning: You will most likely screw up some part of your system using these drivers and configuring them the way this guide suggests. The drivers are still unstable, there have been reports of crashing X, ALSA, and locking up the whole computer. Using these drivers is like waving a red flag at a bull…..

The instructions themselves are about 8 pages long, mostly hacks to kernel modules, configure and makefiles. Even after many years working with Linux several of them made me gasp in horror. Here is one extract:

LinuxSys.c

Modify the LinuxSys driver file (drivers/src/ossrv/LinuxSys.c).

Remove the following line (~line 1581):


set_fs(fs);

Modify and Add the following lines from Vasko (~line 1558):


HRESULT sysGetFileSize(CTVOID *This, CTDWORD *pdwSize, CTHANDLE hFile)
{
stFileHdl *pFileHdl = (stFileHdl *) hFile

to


HRESULT
sysGetFileSize(CTVOID *This, CTDWORD *pdwSize, CTHANDLE hFile)
{
stFileHdl *pFileHdl = (stFileHdl *) hFile; // Comments By Vasko
if(!pFileHdl)
{
printk(KERN_WARNING "sysGetFileSize: no file handle\n");
CTRETURN(E_INVALIDARG);
}
if(!pdwSize)
{
printk(KERN_WARNING "sysGetFileSize: no pointer to return value (pdwSize)\n");
CTRETURN(E_INVALIDARG);
}
}

Marvellous. I’ll be going without sound then.

Hello world!

August 21, 2008

I would change the default title, but since I’m a programmer person, I think I’ll leave it.

Hello Blogging world!