Kai's Development Blog

Python Regexes: Named Groups. Cool Bananas

2011-12-08T09:55:00.001+01:00

I'm currently writing a Python parser for GenBank files. I know BioPython has one, and it doesn't even suck, but BioPython requires a bunch of C extensions, so I can't go and just ship it with my Python application. So I'm creating a BioPython-compatible API for the classes that I need for antiSMASH, without the dependency tail BioPython forces on me. Having contributed to BioPerl before, I do like to use regular expressions for token-based parsers, especially as I'm not too fond of lexers in Python. Now, a GenBank header is a peculiar thing that stems from the punch card ages, with a fixed-width format. Unfortunately, at some point that format changed, and things moved around. And there's a ton of programs out there that produce GenBank files that are slightly off. So parsing the header using a token-based approach seems like a good thing. Now, let's look at the first line. That has a bunch of interesting information.

LOCUS       SCU49845     5028 bp    DNA             PLN       21-JUN-1999

The 'LOCUS' tag explains what this line is about, then there's the accession number for the sequence, the length of the sequence, the type of the molecule, some classification where the sequence comes from, and the date the sequence last saw a major update. The regex to parse this is pretty straightforward (yeah, right):

LOCUS\s+([\w.]+)\s+(\d+)\sbp\s(ss-|ds-|ms-|\s{3,3})(\S{2,4})\s+(linear\s\s|circular|\s{7,7})\s+(\w{3,3})\s+(\d\d-\w{3,3}-\d{4,4})

Incidently, this is a great example for why people dislike regular expressions. Now, in both Perl and Python, there's a way to define verbose regular expressions, so you can restate the regular expression as:

LOCUS\s+        # Header line starts with LOCUS tag followed by multiple spaces
(               # accession number regex:
  [\w.]+        # any alphanumeric character or '.'
)
\s+             # skip over whitespace
(               # sequence length
  \d+           # digits only
)\sbp\s         # skip ' bp ' string
(               # single, double or mixed stranded or nothing
  ss-|ds-|ms-|\s\s\s # can be all spaces
)
(               # Molecule type DNA, RNA, rRNA, mRNA, uRNA
  \S+
)
\s+
(               # linear, circular or seven spaces
  linear|circular|\s{7,7}
)\s+
(               # division code, three characters
 \w{3,3}
)
\s+
(               # date, in dd-MMM-yyyy
  \d\d-\w{3,3}-\d{4,4}
)

This is already pretty decent, but Python can do one better. With the current version of the regex, I need to remember that the molecule type is the 4th group, so match.group(3) will be what I'm looking for. Python decided to extend the Perl extension syntax a bit more, to add named groups. With ?P<name> you can name groups. and then call match.group('name') to access them later. So the final version of the parser regex turns into

LOCUS\s+        # Header line starts with LOCUS tag followed by multiple spaces
(?P<accession>  # accession number regex:
  [\w.]+        # any alphanumeric character or '.'
)
\s+             # skip over whitespace
(?P<length>     # sequence length
  \d+           # digits only
)\sbp\s         # skip ' bp ' string
(?P<stranded>   # single, double or mixed stranded or nothing
  ss-|ds-|ms-|\s\s\s # can be all spaces
)
(?P<molecule>   # Molecule type DNA, RNA, rRNA, mRNA, uRNA
  \S+
)
\s+
(?P<formation>  # linear, circular or seven spaces
  linear|circular|\s{7,7}
)\s+
(?P<division>   # division code, three characters
 \w{3,3}
)
\s+
(?P<date>       # date, in dd-MMM-yyyy
  \d\d-\w{3,3}-\d{4,4}
)

and you can use speaking names to access the group's contents later. Great to make the code more readable. Combined with a bunch of tests, that should stay maintainable.

From the frontline, day 5

2011-08-12T09:16:00.002+02:00

Another day, another piece of testing mayhem. I've completed the 0.1 version of my Flask-Downloader helper class. With this, I could complete my web app. Now, the downloader itself has a bunch of tests to make sure it's working as expected, but I was also going to test the corresponding code paths in the web app's tests.
The user can provide the input either by uploading a file or by giving an accession number. Testing for the file uploads was easy, as the Flask test client accepts file-like objects as data input for POST requests. So testing the app will do the right thing is as easy as:

def test_upload(self):
    file_handle = open(tmp_filename)
    data = dict(file=file_handle)
    rv = self.client.post('/upload', data=data)
    assert "upload succeeded" in rv.data

Assuming your upload function listens on '/upload' and returns a page that contains "upload ducceeded", of course.

Testing file downloads is a bit more elaborated, because I don't actually want my downloader to connect to the internet during a test run. Minimock to the rescue! I can fake the download helper and create the same kind of output to fool the application code.

from minimock import Mock
from werkzeug import FileStore
def test_download(self):
    data = dict(id="FAKE")
    # now create the fake downloader
    tmp_file = open(tmp_file_path)
    dl.download = Mock('dl.download')
    dl.download.mock_returns = FileStore(stream=tmp_file)
     rv = self.client.post('/download', data=data)
    assert "download succeeded" in rv.data

With similar assumptions as in the example before, and also the idea that you have a pre-existing file in tmp_file_path. A StringIO file-like object should do the trick as well.

With all the tests in place and a test coverage of 100%, I declare this campaign a success. I still need to deploy the new web app on my test server instead of the old one, but I'm going to do that next week. I will also continue my war on legacy code, now tackling the pieces that do the actual work. No war is over as quick as you'd initially hope after all. Also, I'm pretty sure the 100% code coverage don't mean there's not plenty of places for bugs to hide in, just that at least all of the code is looked at by the interpreter once. Still, it's a good conclusion to a busy week. Testing rocks.

From the frontline, day 4

2011-08-11T13:24:00.006+02:00

Today, I decided to go for the downloader component that can download files on the behalf of the users. While looking at how to test this, I actually noticed that the mm_unit functionality has been merged into the minimock package. Sweet.
I wanted to keep this modular, a downloader sounds like a tool I could use in a couple of projects. So I created a Flask extension. There's a nice wizard script that automates the creation of the boilerplate files. Using the wizard, I created Flask-Downloader. It's pretty straightforward to use. There's a download(url) function that will return a werkzeug.FileStorage instance, just like the flask upload hander. I'll also add a save(url) function that'll save the url's contents to a file without returning a file-like object.

Not too much to write about, spent a lot of time researching stuff today. Hope to get done with my changes tomorrow. Let's see how that'll work out

From the frontline, day 3

2011-08-10T09:21:00.004+02:00

Today, I decided to go and restructure my webapp into a package as recommended by the Flask "Larger Applications" pattern. Thanks to my existing test suite, the move was quick and pain-free. I had to fix imports in one of the tests, but apart from that, the only thing I had to do was splitting up the webapp.py file correctly into webapp/__init__.py, webapp/views.py and webapp/models.py.

I then started playing with implementing the actual functionality for uploading files and creating database entries for the jobs submitted. Took a while to get this right, never done database stuff with Flask before. But again, pretty easy to set up tests for all this. Also, I discovered Flask-Testing, making flask unit testing even more comfy. Just had to fix up the Twill module Flask-Testing comes with to not use the md5 and sha modules, triggering deprecation warnings. Will continue to write the last tests for the job submission form tomorrow, and then see how to deal with making the web app download data from elsewhere for the user.

From the frontline, day 2

2011-08-09T09:41:00.006+02:00

Looks like my system strikes back. A fitting thing to happen for an episode 2, I guess. Turns out that nosetests and virtualenv need some extra care and feeding when kept together. Installing another copy of nosetests into my virtualenv fixed the test failures I was seeing. Thanks to the folks on #python and #pocoo for pointing me the right way.

Of course this broke the code coverage. Nothing a pip install --upgrade coverage wouldn't fix, though. As an added bonus, the coverage html output now looks much nicer. I guess it was redesigned between whatever my system got and the version pip grabbed.

Of course, after spending quite some time writing tests for my email sending module, the #pocoo folks point me at the already existing Flask-Mail extension, that integrates into the Flask test harness (as in, if you're testing, it won't send email) already. Oh well. Switched, ditched quite some code and corresponding tests. Even less stuff I have to maintain myself.

Unfortunately, Flask-Mail doesn't seem to like it when you switch on app.config['TESTING'] = True after initialization. Fortunately, you can still fiddle with the value used so it doesn't try sending emails, like so:

def setUp(self):
    webapp.app.config['TESTING'] = True
    webapp.mail.suppress = True

The key here is the mail.suppress = True setting. Once that's done, all the testing options work as expected. You can even have a look at the msg objects that would have been sent using the following snippet:

def test_sent_mail(self):
    """Test if emails were generated and sent correctly"""
    with webapp.mail.record_messages() as outbox:
        rv = self.app.post('/send-email',
                 data=dict(message="hello world"))
        assert len(outbox) == 1
        msg = outbox[0]
        assert "hello world" in msg.body

I like it, this really gets all the stuff I do under test in a very straightforward manner.

From the frontline, day 1

2011-08-08T11:17:00.004+02:00

I've decided to start the campaign by ditching the existing PHP web app. I lost all confidence in it last Friday, when I found that it only worked by accident, due to a well-placed typo.
As I'm rewriting the web app anyway, I thought I could also ditch PHP altogether. Not that it's necessarily a bad language for web apps, but the rest of the code is perl or python, so getting rid of php means one less language to get confused by.

Because this is the War on Legacy Code, I'm not going to write untested code in this campaign. So first I need to brush up my python unit testing skills. I do have parts of a python version of the web app already (untested, that won't work later), but it's missing a user feedback form.
I don't want to send an email for every run of the test suite, so I need to mock up smtplib.SMTP. After some web research, I'll be using Ian Bicking's minimock to provide my mock objects. As I don't just run doctests (even though they're pretty cool), I decided to also throw in MiniMockUnit, which makes minimock print the output to a StringIO buffer instead of stdout. That way, you can easily put it in a normal unit test.

I usually run my tests using nosetests. Turns out, nosetests allows me to run both vanilla unit tests and doctests, and it also has a code coverage plugin. Thus,

nosetests -v --with-doctest --with-coverage --cover-html --cover-package="testmodule"

will get the module "testmodule" tested using available unit tests, doctests and the test coverage will be reported in html in the cover/ directory. The --cover-package part seems to be needed to stop the coverage code from trying (and failing) to create coverage information files in the standard lib paths.

To sum up, I didn't actually see much battle.. er.. code today but my arsenal is filled with testing tools, and I'm well prepared to jump into the fray tomorrow. Also, thanks to some help from the folks on #gsoc IRC on freenode, I now have decent syntax highlighting and formatting on my blog, so I might be able to post code samples for real now. Life is good so far.

War on Legacy Code

2011-08-08T08:04:00.005+02:00

Following the hallowed US American tradition of declaring war on whatever things you don't like, I've decided to declare war on legacy code this week.
By legacy code, I mostly mean untested code, following the definition of Michael Feathers' book Working with Legacy Code. That's a great read, by the way. If you're working with old code, you should go read it. I found that I've been doing most of the things mentioned already, but it's nice to see a systematic write-up about it.
My chosen battlefield in this war is the code at my day job, mostly because it's in a much worse shape than the code I deal with in the various Open Source projects I'm involved in. Seeing how my day job code is a Frankenstein's monster of Perl, PHP and Python parts, some of the work will be to get some of the tests done twice. In particular, I really want to get rid of the PHP parts.
I won't delve into the particulars of the code too much, it's published under the GPLv3 if anyone is interested. I will however try to post some daily news from the front lines, with things that I have thought about during that particular day of the battle.

Geeky stage props made easy

2011-06-19T22:29:00.004+02:00

Currently I'm involved in building the stage and props for an amateur theater, the Brechtbau Theater at my university. We're currently preparing for an Agatha Cristie play, "And then there were none". As a great opportunity, we are able to perform this piece in the city's professional theater, the Landestheater Tübingen (LTT).

Needless to say, we're really excited about this. Of course having to build a stage for a 300 seat theater is a bit different to building a stage for the 80 seat theater we've got at university. Also, we only have about four hours to set up the stage, and after the last night, we have to clear out immediately. While the way to build a modular stage design probably is worth a blog post on it's own, I want to talk about the electronics behind stage a bit today.

Without wanting to spoil some of the surprises we have in store for our audience, we're working on a stage design with lots of big cogwheels and other moving parts. We will power some of these with the cheapest and most readily available power source we have available: actors. But some of the stage has parts that are just out of reach, or need to be positioned more precisely. For this, I'm currently planning to use a combination of stepper motors and hobby servos, run by an Arduino Uno.

I'm still heavy in the prototyping stage, but I just wanted to share my discovery on how easy it is to do stuff like this with the Arduino. My current test setup looks a bit like

Using an L293D (left) and an L293NE (right) IC, I'm running two Trinamic bipolar steppers, and I'm also controlling four servo motors. For making all of these move forwards and backwards, I had to write about 50 lines of code, including whitespace and some comments. Arguably, moving a couple of motors forward and backward in a loop isn't that intersting, but the amout of work the Arduino default libraries already take care of is just great.

Next, I'll have to figure out how to build a lamellar transport belt and move it with one of the steppers, while converting the circular movement of the other stepper to a linear movement. Never played with elaborate hardware stuff before, this is fun.

Packaging python modules, the really easy way

2011-02-17T16:24:00.004+01:00

I just had to package up a python package to make installation of a software we use at work easier. The systems we run here are Suse- and Ubuntu-based, so I got to package RPMs and debs. I did package the odd perl package and some bioinformatics tool before, but I haven't looked at it for a while, so I was pretty rusty. I use the OpenSuse Build Service to build RPMs, and the nice folks in FreeNode's #opensuse-buildservice pointed me a py2pack, a truly amazing piece of software that creates a .spec file for you from package information in pypi. Now, I had to package pysvn, which happens to not link to the downloadable file from pypi. As it turns out, this just stopped me from using py2pack for downloading the file, py2pack generate works fine. Filling in the missing License and Description information was easy, and my package was building on OBS in under five minutes. For Ubuntu, Python packaging guide was a bit less comfortable, but a short while afterwards my Ubuntu package was built on Launchpad as well. Life is good. At least so far, now I get to make all of that work for system #3 in the department. This happens to be an OS sold by a company from the northern west coast of the USA, and it doesn't have an execute bit and can't deal with hashbang lines. Oh well, two out of three isn't too bad.

Defensive programming to the rescue

2011-01-21T10:14:00.015+01:00

I've just fixed a bug in a web app that I'm working on with a couple of colleagues, a bug that is a great example of a hard to find, easy to fix bug that could have been avoided by some defensive programming. I thought I'd share this titbit to illustrate why defensive programming is a good idea.

The application we're working on consists of three parts:

a web front-end that accepts jobs (php based)
a database keeping the job queue (SQL)
the back-end that runs the jobs (python)

The job is actually an external script, which is important for this bug as well.

On the web front-end, the user gets to choose a couple of options for the job using check-boxes.

The bug we were seeing was that if the user just clicked one checkbox (and that wasn't the "all" checkbox), the external script would die with an "invalid options" error.

The options are passed into the SQL database as a comma-separated string, and our first suspicion was that for single options, there was a trailing comma left. A quick look at the database dump showed that this was not the case.

The next idea was that the back-end was not constructing the command line for the external script correctly. The code looked sane, but just to be sure I decided to do some printf debugging. In the printout of the command line, I finally found the bug. It was in the web front-end after all. Let's look at the code (changed a bit for brevity).

// 1 is "all"
if($_POST["1"] == "on" ){
  $options = "1";
} else {
  for ($i = 2; $i <= 10; $i++) {
      if($_POST["$i"] == "on") {
          $options .= " " . $i . ",";
      }
  }
  $suffix = strripos($options, ",");
  $options = substr($options, 0, $suffix);
}

Can you spot the problem?

$options .= " " . $i . ",";

is the culprit. It adds a leading white-space to the $options string. Looking at the database dump, that's easy to miss.

Now, why is this a problem? Let's look at the back-end code (changed for clarity again):

job = get_next_job_from_work_queue()
args = ['./do_stuff.py', job.filename]
args += ['--options', job.options != None and \
                      jobs.options or '1']
subprocess.call(args)

The back-end uses an array for it's command line arguments to avoid having to call out to a shell first. This has the side effect that all the arguments are passed to the called script verbatim. Thus, a leading space character is kept and passed to the called application. This is defensive programming fail #1.

Still, this doesn't explain why the white-space is a problem. For that, we need to look at do_stuff.py (Changed for clarity again).

value = options[options.index(i) + 1]
if i == "--options":
    if "," not in value and value not in ["1","2","3","4","5",\
                                          "6","7","8","9","10"]:
        invalidoptions(i)

Assuming we clicked on the #7 check-box in the front-end, " 7" is passed to do_stuff.py. " 7" is not in ["1","2","3","4","5","6","7","8","9","10"], so we fail. Defensive programming fail #2 is in the value = options[options.index(i) + 1] line. Adding a strip() here would have avoided the bug. This was not found in manual testing, as the shell takes care of stripping the white-space characters for us. Still, a bit of defensive programming would have helped to avoid the issue.

If I ever teach a programming course, one of the assignments will be finding and fixing a bug like this.

Update: 2011-08-08 Pretty-print code.

My Samba status 11/1

2011-01-17T07:36:00.004+01:00

Hi folks, you will have noticed that I failed to post any of my "On the way to Samba4" reports in December last year. That was because I failed to do any Samba work in December, spending all of my time on work-related things. A few co-workers and me had to rush to get a web server up and running that allows biologists to figure out what secondary metabolites like antibiotics might be produced by their bacterium/fungus. After pulling a 90-hour-week to get finished between Christmas and New Year's Eve, I had to take some time off not staring at a computer screen. Batteries recharged now, I'm ready to get into action. Over the weekend, I've been getting the skeleton for some DNS torture tests set up, I'm hoping to flesh this out a bit more during the week. Cheers, Kai

On the way to Samba 4, week 47

2010-12-01T23:58:00.001+01:00

Not much to show for this week. I spent the week at a conference, and while I learned many interesting things, I didn't have time to get Samba-related work done. Let's try again for week 48.

On the way to Samba 4, week 46

2010-11-24T23:05:00.003+01:00

Really not much time to work on Samba this week. So I didn't get around to really debug the failing echo test for quite a while. Then tridge pointed out that I was trying to use a connected UDP socket as if it was not connected, and things started to work like a charm. It doesn't look like I'll have much more time during week 47. Hopefully I will get around to clean up the patches and push my example anyway.

On the way to Samba 4, week 45

2010-11-17T23:25:00.003+01:00

A bit late, as usual, but I finally have something productive again. I've got a Samba4 UDP echo server and async client library (including torture test suite), and they're almost working. :) I'm saying almost because I can only get it to work manually right now, not from "make test". I'm sure it's some small thing I'm missing, I'm expecting this to be fixed soon. The code also still lacks some explanatory comments, but should be ready for the mainline for the week 46 report. The client library and torture test also runs against netcat, so two "implementations" of echo servers are supported for sure. ;)

On the way to Samba 4, week 44

2010-11-10T21:11:00.006+01:00

I usually hate it when real life stops me from coding, but in this case I had a really fun weekend off the computer. The downside is that the echo server still is not done. So not much happening on the Samba 4 front from my side. One related news item though is the release of a new version of the Yocto Project. Quoting from their website:

The Yocto Project™ is an open source collaboration project that provides templates, tools and methods to help you create custom Linux-based systems for embedded products regardless of the hardware architecture.

Based on OpenEmbedded, one of it's supported platforms is the Beagle board. This looks like a pretty good platform to play with to get a bunch of embedded Samba images to work. This goes straight on my todo list, as if that wasn't pretty full already. Still, maybe a nice setup to demo at the next SambaXP conference. So, that's it with this week's installment of my Samba 4 blog post. Unless real life happens again, maybe next week I can show the world how a well-designed Samba 4 task looks like.

On the way to Samba 4, week 43

2010-11-02T20:07:00.003+01:00

Last week, I didn't really get to spend much time on Samba 4. About the only thing I managed was working on a patch that renamed the s4 "net" binary to "samba-tool" while at the GSoC Mentor Summit 2010 with Jelmer. The rename reduces the number of executables with the same name in s3 and s4, getting us a step closer to a fully merged build. Renaming the s4 net tool hopefully inconveniences less admins out there. If you've been bitten by this, sorry about that. That's all there is for now for this week. With any luck, next week's post will be a bit longer.

On the way to Samba 4, week 42

2010-10-27T08:19:00.003+02:00

This is the first installment of my hopefully regular progress reports on the things I did last week to get Samba 4 closer to a stable release. The biggest piece of work certainly was finally pushing the DNS server implementation. Yes, that's correct, Samba 4 now has it's own DNS server. So far, all it does is serve out resource records from AD that it's authoritative for. You need to sync over the LDAP entries from another AD DC as well. Still, that's a good start, and thanks to the existing infrastructure in Samba4, it was possible to do all of this in about 1000 lines of code. Features still missing from the DNS server so far:

Recursive query support (needs a DNS resolver library, working on that)
Support for update requests so clients can update their own entries
On provisioning, we need to pre-load the database with a couple of DNS records

You can see a broad overview of the todo items for the DNS server in source4/dns_server/TODO.

WOMBAT 0.5.0 released

2010-01-07T15:40:00.002+01:00

The Wombat team is happy to release Wombat 0.5.0

Compared to the 0.4.0 release, Wombat 0.5.0 sports a lot of new features.

Notable differences are:

Now based on Pylons 0.9.7 and SQLAlchemy 0.5
No support for Python 2.4 anymore due to Pylons requirements
Users can now upload artwork
Uploaded artwork will be committed to the media repository after moderation by an administrator or lead artist
Tag-based browsing and other new UI features
A new subversion backend uses pysvn instead of executing the svn shell commands
Automated database cleanups

Changes since the 0.4.5 release include:

Art upload features as mentioned above
Security hardening
- More resistance against command injection
- More complicated password algorithm to slow down rainbow-table attacks
Multiple bug fixes

As always, the Wombat demo site has the latest version for you to test.

Wombat demo sites back up

2009-11-24T08:55:00.003+01:00

It's aliiiiiiive! Thanks to Thomas Ingham, a fellow Wombat developer, http://wombat.worldforge.org/ has a new home, ending a couple of months of downtime while I was looking for new hosting. Tom's box has enough power to allow running two instances of Wombat. Following the next release, the Wombat demo site will be running the latest released version of Wombat, while the Wombat development site will be running the tip of the development tree, on an example media repository. We're currently dealing with the teething pains of rolling out Wombat on a distro that is different from the one I develop on, but everything should be running fine in a couple of days. Thanks again to Tom and his sysop Josh for giving two cute little Wombat web-apps a new home.

Guten Morgen net

2009-08-11T09:50:00.002+02:00

After a couple of days of grinding through the code, the "net" tool from the Samba suite is finally ready to speak languages other than English. A bit over 1700 messages are now waiting for translators. Note to self: Next time I work on i18n, I want combat pay.

Huston, we have a login

2009-06-05T10:23:00.003+02:00

After some initial issues, I have just joined my first XP box to Samba4 running on snoopy, my ARM-based system-on-a-chip home server. So now I have a spiffy Active Directory setup running on a system that makes no noise, barely heats up, and that peaks at 2W power usage. If that's not green IT, I don't know what is.

WOMBAT 0.4.5 released

2009-05-21T13:02:00.003+02:00

The WOMBAT team is happy to announce the release of 0.4.5 WOMBAT 0.4.5 is mostly a bugfix release. Several backend and usability issues have been resolved. Features include

Tag cloud now really is a "cloud", not just a list of tags
In directory view, clicking on the expanded information no longer collapses the extended information (bug #291724)
Tag-based browsing now filters on multiple tags if desired

As always, check out the WOMBAT live demo.

WOMBAT 0.4.4 released.

2009-03-23T09:11:00.002+01:00

The WOMBAT team is happy to announce the release of 0.4.4 WOMBAT 0.4.4 introduces preliminary tag support for the beginning of a new browsing experience. Features include

Tags for assets and collections
Tag cloud replaces "unassigned file view"
Bug fixes for upgrade issues missed in 0.4.3

As always, check out the WOMBAT live demo.

WOMBAT 0.4.3 released.

2009-03-17T15:14:00.003+01:00

The WOMBAT team is happy to announce the release of 0.4.3 Different to other releases, this is a point release to follow the updated versions of Pylons 0.9.7 and SQLAlchemy 0.5 and contains no other changes. This version does not support Python 2.4 anymore, WOMBAT 0.4.2 was the last version to support this. As always, check out the WOMBAT live demo.

WOMBAT 0.4.2 released

2009-01-01T16:53:00.002+01:00

Another year, with a brand-new WOMBAT release. The biggest feature of the 0.4.2 release is an automated update function that keeps WOMBAT in sync with the media repository. Other features include automated database cleanup and the return of the searchbox. As always, check out the WOMBAT live demo.

WOMBAT 0.4.1 released

2008-10-31T17:56:00.003+01:00

The WOMBAT team is happy to announce the release of 0.4.1 Based on the UI revamp in release 0.4.0, new UI features in this release include:

Custom icons that reflect the content of the media, if possible
New stock icons from the Tango project
Display total number of files under a directory to simplify browsing

As always, the WOMBAT demo page has the current version for you to test.

WOMBAT 0.4.0 released

2008-10-29T01:25:00.003+01:00

The WOMBAT team is happy to announce the release of 0.4.0 New features in this version include:

Completely revamped look and feel
Support for users and roles
Actions that change the server's data

As always, the WOMBAT demo page has the current version for you to test.

WOMBAT 0.3.0 released

2008-08-13T13:08:00.002+02:00

The WOMBAT team is happy to announce the release of WOMBAT 0.3.0 New features for this version include:

Completely redesigned data backend based on SQLAlchemy
Support for scanning and updating the repository without running an extra script to generate the metadata
Preliminary support for Assets and Collections, the WebUI still needs some work.

As always, the WOMBAT demo site runs the latest version for you to test.

WOMBAT status

2008-08-03T23:19:00.002+02:00

Hi folks, development around WOMBAT hasn't really slowed down, despite the lack of releases recently. We're currently working hard on bringing the back-end data model into a shape that will allow us to add more features to WOMBAT more easily. I'm currently busy writing a svn xml parser that will load the metadata into a SQL database, instead of relying on files like we used to. This, together with a major overhaul of the overal design of the data model, gives us better integration with the SVN backend. A next step in this design will be tracking svn updates file by file instead of forcing a re-scan of the whole data set. Stay tuned for updates.

WOMBAT 0.2.1 released.

2008-07-19T16:29:00.002+02:00

The WOMBAT development team is happy to release WOMBAT 0.2.1 This is mostly a bug fix release fixing an uncaught exception when trying to create thumbnails for interlaced PNG files. It also has some visual updates, displaying all available files for the root directory and allowing to narrow down the number of files by browsing into subdirectories, scrollable divs to better display text files in place and a couple of others. Note that the wombat demo website moved to http://wombat.worldforge.org/, thanks to Anders (aka. Demitar) for hosting.

WOMBAT 0.2.0 released.

2008-07-15T19:44:00.003+02:00

The WOMBAT team is happy to announce the release of WOMBAT 0.2.0 This second alpha release of WOMBAT is the first to feature integration with Subversion.

Other Features Include:

Advanced Searching: Search by author, file type or modified date from within the repository.
A new stylish logo from Orin Shepherd at Coalmarch Productions (not a feature, but we're counting it as a definite addition.)
Thumbnail rendering for "most" image assets.
Inline file-detail viewing for quick side-by-side comparisons.
Application status screen with helpful statistics.
A better browsing experience on the whole.

Of course the live demo shows the current status.

WOMBAT 0.1.4 released.

2008-07-15T10:29:00.002+02:00

New features for this release include:

Thumbnail preview for images, provided the PIL module is available.
Advanced search for things like "author", "file extension" and "creation date"
Various bug fixes.

As always, have a look at the live demo to see what's hot.

WOMBAT 0.1.3 released.

2008-07-12T23:39:00.002+02:00

Features of this release include:

Loading revision data from a prepared metadata dir, vastly improving scan speed.
Many more search options
Status information on the front page.
Various bug fixes

As always, the demo site awaits your tests.

WOMBAT 0.1.2 released.

2008-07-10T00:37:00.003+02:00

I'm happy to announce WOMBAT 0.1.2 with Subversion integration. While I'm not too happy with the speed the integration gives us, it's a start. 0.1.2 also features some nifty AJAX magic to make the directory details go away when you don't need them and other visual improvements. Oh, and as you can see I moved the announcements of WOMBAT micro version changes to my personal blog, in order to not spam the WorldForge website. As always, you can try out the current release of WOMBAT.

Introducing WOMBAT

2008-07-01T08:50:00.003+02:00

Yesterday I got around to release WOMBAT, the WorldForge Open Media Browser/Archive Tool. So finally I'm coding for WorldForge again after my last project stalled a little. Like plunger, WOMBAT is written in Python. WOMBAT makes use of the Pylons webapp framework. Apart from the brain-dead egg distribution mechanism, it's really neat. From the developer side of things, the framework gives you a lot of help to work on features, not web glue. I still have to get the FCGI connection to Apache working, but so far the demo site on my home box works fine standalone. Thanks to NAT, I can even pretend it's a real web server.

Getting rid of warnings

2007-12-14T10:06:00.000+01:00

If you have ever built Samba4, you will have noticed that gcc spits tons of warnings. As fixing little things all over the code is a good and easy way to get an idea what's done where, I have decided to go and fix as many of those warnings as possible.

Improving Samba4 winbind, a look back

2007-09-19T10:01:00.000+02:00

What the project is about Samba4 contains a basic winbind implementation, but it is still lacking many features. The goal of my project was to improve Samba4 winbind so that the nsswitch and pam functionality provided by Samba3's winbindd would be present in Samba4, too. Simply copying code over from Samba3 would not do, of course, as the underlying architecture in Samba4 is different. Also, the goal was to improve readability of the code, as opposed to the more organically "grown" look of the Samba3 winbindd code. In Samba4, there is a library taking care of user/group management called libnet. Samba4 winbind uses this library extensively. What was done so far

Porting nsstest to Samba4 nsstest is a binary that test basic functionality of a nsswitch library. Once all the tests in nsstest work for libnss_winbind from Samba4, the winbind implementation is useable.
Getting information about users / SIDs This is the basic nsswitch getpwnam/getpwuid functionality returning a pwent structure. It is also possible to query for AD / NT domain information about the user/sid.
User listing / enumeration nsswitch provides a set of functions (setpwent/getpwent/endpwent) to iterate over a password database. The classic database is the /etc/passwd file, but of course using nsswitch, it's possible to use a directory like LDAP or AD.
Mapping SIDs to user ids and back This one is only stubbed out, as Samba4 doesn't handle that mapping yet. However, functions for this were needed to make winbind work, so I had to stub these out. The advantage is that the other code will automagically start to work correctly once these functions are implemented for real.
Mapping SIDs to group ids and back Much the same applies here, once idmapping is supported in Samba4, this will be replaced by real code.

What is left to do

Group enumeration The libnet functions for group enumeration were not implemented by the time GSoC was up. Now these functions are in, so support for groups identical to the user functions will follow soon.
NTLM caching Due to time constraints, caching of NTLM blobs was discarded. The nsswitch/pam functionality was regarded as core importance.
PAC/info3 caching As with NTLM caching, PAC/info3 caching was discarded. Caching is only interesting once the other features are working and will be implemented eventually.
Automated tests Currently the only way to test all of the functionality is to wrap the wbinfo binary and let that take care of constructing the necessary winbind queries. This is a bit clumsy. Jerry Carter is currently working on a winbind client library that will allow to access the functionality of wbinfo without a wrapper. The tests will be implemented using that API once it is in the tree.

Future (related) work First of all, the features still left on the TODO list will be implemented. Group functions first, testing next if possible. There is more to winbind than this GSoC project was about, so the more missing features will be implemented. The caching will follow once the other features are working and tested. An improved winbind will help Samba4 to not only act as an AD controller but also as a domain member. A look back Complying with long-standing computer science tradition, I underestimated the amount of work that had to be done before I could start working on the actual features I was planning to implement. In the end I had to prioritize features and drop the least important ones to get finished in time. I did not expect to spend so much time figuring out my way around the libnet code. However, the foundation for implementing the dropped features is laid, so I do not feel too bad about it. Samba4 winbind already works better than before. Pending group support and id mapping, it will be usable for simple scenarios. Conclusions I still need to be more careful about the scheduling of projects and estimating the amount of work required to get features to work. Still, the only way to improve is to try and adjust the estimations accordingly. I feel more confident around the Samba4 code now, thanks to Metze, Jelmer and Andrew's help. Of course thanks to all the other team members for the help and advice offered, on IRC and the mailing lists. Last but not least I would like to thank Google in general and Leslie Hawthorn in particular for running the third Summer of Code program in an efficient manner, making this a really enjoyable experience.

Having a Campus Party

2007-07-27T13:56:00.000+02:00

¡Buenas tardes, compañeros!

Spending all of this week in Valencia, Spain for Campus Party 2007, I'm having loads of fun hanging out with the other Summer of Code students and all the nice people from Google.

I have tried hard to get some work done for my Summer of Code project this week, but it's rather hard to really concentrate when you're in a room with thousands of people playing computer games, having shouting competitions and generally a lot of noise. I wish I had noise-canceling earphones.
Still, I'm currently working on some code that should make my life easier next week. Writing utility functions is lots of work, but usually worth it.

Yesterday, I had a fun time delivering my presentation about experiences with Google Summer of Code and tips how to get into Google Summer of Code in 2008. The small lecture room that Google uses for the talks and tutorials was pretty full, so I hope some of these people will actually get into Google Summer of Code next year. I did not really get into details on my current and previous projects but rather talked about personal experiences and strategies to get your proposal accepted.

Apart from writing code for my SoC project, preparing and giving the talk, I'm having fun with the Googlers and SoCers, hanging out at the beach at 3 in the night, having nice food and generally enjoying myself. Sleep is a bit short, but there's always something.

¡Hasta luego!

En route to Valencia

2007-07-23T11:05:00.000+02:00

Finally on my way to Campus Party 2007. I really need to get my talk for thursday done, hopefully I'll be able to finish it before my plane touches down in Valencia. (I'm currently waiting for my first flight to depart, not blogging from a plane.) Originally I was planning to take care of my talk last weekend, but some nice virus decided that I'd rather spend all weekend in bed, sick. Oh well, this won't be the first just-in-time presentation. I'm just hoping my stomach will settle down a bit so I can actually enjoy all the nice Spanish food. I'm looking forward to a nice week in Spain, even though I will have to spend my time working on my GSoC project, as well as my LinuxConf.EU talk. I will be blogging from the event, too.

LinuxConf.EU

2007-07-12T10:23:00.000+02:00

The talk I proposed for LinuxConf.EU was accepted, so I'm going to Cambridge early September. Unfortunately this means that I actually have to write the talk I proposed to give. Also, as I need to hand in my talk by 30th of July, I'll probably need to write it while I'm at Campus Party in Valencia. Oh well. I'm looking forward to another nice conference.

Traveling to Spain

2007-07-04T07:37:00.000+02:00

Google invited me and three other students to join them for the 2007 version of Campus Party. This means that I will travel to one of Spain's largest technology events to meet some other GSoCers and give a presentation about my GSoC project and my experience with GSoC in general. It looks like this is going to be lots of fun. Time to go and work on my presentation.

plunger moves to Google project hosting.

2007-06-30T10:06:00.000+02:00

So far, plunger has existed in the WorldForge cvs tree out of convenience. However, as there are quite some people interested in plunger from outside the WorldForge project, I've decided to provide more infrastructure than what WorldForge has right now. This includes an issue tracker to allow people to more easily create bug reports and give feedback. After spending some hours to move the code from cvs to google's svn, plunger now lives at http://code.google.com/p/plunger/ Expect some changes while I sort out the infrastructure I want to have set up at the new site.

Hello World

2007-06-30T09:51:00.000+02:00

This being the traditional first program for any programming language, I think it's a fitting title for the first post to a development blog. Now, what is this blog all about? I'm not a big fan of blogging in general, but I have to admit that they are a good way to provide news and status updates for projects. I'm developing for a couple of different projects and I'd like to have a single place to write about this, instead of having to go to a different blog site for each.