Now the clocks have gone back it is, I'm sure you'll have noticed, getting dark even earlier in the afternoon. Naturally, this is the ideal time for my desk lamp to break.

Last week I wondered aloud whether my current SAX based approach to building an XSLT engine was going to work. Specifically the piece of spec text I was worried about was section 2.6.2 which says

The xsl:import element is only allowed as a top-level element. The xsl:import element children must precede all other element children of an xsl:stylesheet element, including any xsl:include element children. When xsl:include is used to include a stylesheet, any xsl:import elements in the included document are moved up in the including document to after any existing xsl:import elements in the including document.

Section 2.6.2 continues with a discussion of import precedence, which gave me the clue I needed. It says

An xsl:stylesheet element in the import tree is defined to have lower import precedence than another xsl:stylesheet element in the import tree if it would be visited before that xsl:stylesheet element in a post-order traversal of the import tree.

...
Each definition and template rule has import precedence determined by the xsl:stylesheet element that contains it.

For example, suppose

• stylesheet A imports stylesheets B and C in that order;
• stylesheet B imports stylesheet D;
• stylesheet C imports stylesheet E.

Then the order of import precedence (lowest first) is D, B, E, C, A.

While the spec says imports are "moved up", that doesn't mean they have to be actually moved around and processed before the remainder of the stylesheet. In the example above, consider how you'd worked out the import of precedence of B. You need to know not only about B, but also C and E. However, you need to have looked into C to know about E, to work out the precedence of B. Makes my head hurt anyway. But you don't need to do it in the way the spec seems to be leading you. In fact, you can do anything you like so long as the result achieved is equivalent.

Turns out, processing imports can be delayed until the very end of the stylesheet. All you need to do is keep a list of the stylesheets to be imported as you go. Once you hit the end of the initial stylesheet, you work through the list you've made in reverse order (post-order traversal, remember) and load each extra stylesheet. Should one of those stylesheet import a further stylesheet, push it on to the end of the list, which automatically makes it the next stylesheet you pull in. Keep going until your list is empty. Magically, you've pulled in the imported stylesheets in exactly the order required by the spec. Run through the example above, and you'll see it just drops out. Fab.

I was quite startled when I realised how straightforward it was. Initial implementation only took an hour or so, and I've just committed it to svn.

Had a pair of jays spend half an hour or so hopping round the garden yesterday lunchtime. That was top.

No bird excitement at the park - too dark - although those hilarious throw-the-benches-in-the-pond japesters had been back after a few months of absence. What they do is really terrifically funny. They take the benches and then, you'll never guess, they drop them in the pond. It near breaks my heart to pull the benches out and put them back where they should be beause comedy genius, it really is. Well, perhaps you had to be there.

Just committed an initial run at xsl:include. It's incomplete in approximately 19 ways, but for a super-trivial example

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
  <xsl:include href="identity.xsl"/>
</xsl:stylesheet>

Promoted LexicalHandler and DeclHandler to be full members of XMLReader, rather than properties set via the rather tortuous setProperty call. XMLFilter and XMLFilterImpl have been extended to provide support for Lexical~ and DeclHandler. DefaultHandler now provides default, do nothing, implementations of Lexical~ and DeclHandler. Consequently DefaultHandler2 is now deprecated and will be removed in due course.

An XSLT processor has two distinct pieces: a compiler, which reads the stylesheets and builds an executable model of some sort (the transformer); and the compiled transformer which you run against a target document. Obviously the compiler needs to know about the transformer and how to build it, but the transformer need know nothing about how it sprang into being.

This is good for me, because it means I might only have half the job to redo. Up to now, I've been compiling stylesheets in a streaming mode using a big pile of SAX content handlers. As I encounter an xsl:element, say, I connect the SAX event stream to an xsl:element handler which creates the xsl:element object, populates, validates it, adds it to the containing object, finally creates the next handler and connects that. It's a valid (and I think under-utilised) approach to building object graphs from XML documents - in each handler the context is well defined, the housekeeping is straightforward, the memory requirements are low. I knew there was a possibility I'd have to build the stylesheet as a DOM, if only to satisfy the document('') function. I'd hoped that maybe I could be a bit clever and only do that if that function was actually used.

Re-reading the spec last night (reading a spec is always a good idea when trying to implement it) I realised I'd coded myself into a dead end, and it was time to turn around. Two paragraphs in particular changed my mind. In section 2.6.2 it says

The xsl:import element is only allowed as a top-level element. The xsl:import element children must precede all other element children of an xsl:stylesheet element, including any xsl:include element children. When xsl:include is used to include a stylesheet, any xsl:import elements in the included document are moved up in the including document to after any existing xsl:import elements in the including document.

Both xsl:variable and xsl:param are allowed as top-level elements. A top-level variable-binding element declares a global variable that is visible everywhere.

Do you see the problem? To implement these requirements correctly requires out of order processing. For a top-level variable to be visible everywhere, all the top-level variables must be processed before anything that might reference them. For imports to be moved up, you need to know the surrounding context.

You might still be able to deal with this using streaming processing, but it becomes much more complicated. You'd have to make one pass to build the object model, the make a pass over the model itself to validate it, perhaps defer some processing, and it all starts to look a little hairy.

Using a DOM, this is all much more straightforward. You'd parse the stylesheet into a DOM, and walk over that using XPath. It would make other things, like include handling, more straightforward too.

When I started writing this, I'd decided to rewrite what I'd done using a DOM, but now I'm not so sure. I think maybe I could get the SAX handling to work after all. The rearranging and reordering only needs to happen at the top level of the document. Hmm, perhaps I need to reread the spec again :)

Modes now work. Yay.

A little after half past six this morning, sat screeching in a tree by St Martin de Porres School on the corner of Forest Road and Oakland Road. I could hear it as I left my house at the far end of Forest Road, and it kept going pretty much non-stop, except as I passed by. Once I got further down Woodbridge Road the sound of the pub's bottle bins being emptied drowned it out.

Ok, I didn't see it, but it was definitely there. It's quite exciting actually. You don't expect to see such definingly country birds in the city. It's even more exciting than the time I saw a skylark smack in the middle of Milton Keynes.

Hooked up namespaces declared in the stylesheet to the XPath compiler, so template matches and XPath expressions can now use namespaces as normal.

Committed first cut of inline element generation - where you have XML literals within the stylesheet. It's ok, although namespaces are not hooked up yet and attributes get dropped on the floor.

Just committed an initial implementation of xsl:call-template. It does the right thing for correct XSLT. At the moment, though, it can do the wrong thing for incorrect XSLT - if you give an unknown name it raises a run-time error not a compile-time error.

Things are actually getting quite useful now. If you'd like to have a play, grab the mangle branch from Subversion

svn co svn://jezuk.dnsalias.net/jezuk/arabica/branches/mangle

Spent two or three hours over the last couple of days implementing xsl:variable, xsl:param and xsl:with-param. In that order. Doing xsl:variable is easy, xsl:param is essentially the same except you're looking to see if something of the same name already exists, while xsl:with-param is putting that thing there beforehand (if you see what I mean).

Currently they all implement the @select behaviour. I haven't done child content yet, and that'll probably wait a while. There are more interesting things to do :)

[Anna - it's ok that this doesn't make any sense to you.]

My recent development rush has wound down so I've packaged a new Mango release. Online Javadoc is up-to-date. Other documentation currently lacks some of the new things.

It's ACCU conference proposal time again. As ever, I'm clanging up hard against the deadline, but managed to dash off three ideas anyway. As you do. Well, I do. Doing a conference session is hard work, but jolly good fun. Plus it means you can swagger about the place like you own it (since I'm ACCU chair, I'll get to double swagger). You get subbed in too. The ACCU Conference is deliberately priced to be affordable (it's half the price of something like DevWeek, and at least twice as useful), but I won't deny that the financial motive for sticking in a proposal isn't there. On the other hand, I'm slightly wary of how it might look if I am on the programme - might look like I wasn't there on merit. Just have to make sure I don't cock it up, I guess.

Anyway, half-formed proposals below ...

Doing interesting things with them, by way of bringing C++ STL style algorithms to Java.

C++ has the STL, and it's considered a crucial part of the Standard Library. It's also considered something of an icon of good library design. Java's standard library has nothing comparable. Why not? And why isn't anyone complaining?

Includes stuff about library design - Iterator has three methods and they're all wrong :) Comparing the Java and C++ ideas of an iterator, and how that affects the way we can write algorithm libraries.

Also looking at the difference in approach between Java libraries and C++ libraries. To my eye, Java libraries tend to come in lumps while C++ libraries tend to come in lots of little pieces. Also considering the difference the C++ doing things at compile time, compared to Java doing things are runtime.

Most of the discussion also applies to C# enumerators. I suspect Python's list comprehensions are relevant here too.

This is perhaps the best developed idea, especially as I have code written to back it up. I've also just finished a job, written in Java, where pretty much was written using STL style algorithms (and not just for_each :p ), so it's all fresh and churning. Need to think of a better title, though.

I've done a lot of mixed language development - Java/SQL/XSLT, C++/XPath, Java/C++ across a COM boundary, Java APIs that wrap C++ classes, laying C# across C libraries, that kind of thing. I've also recently been working with IKVM to cross-compile Java to .NET. Talking about the problems that arise - the "impedance mismatch" - and ways to deal with it. Adapting vs. translating, object-model mismatches, that kind of thing.

We all know about unit testing. Call the function, check the result, pass or fail. That's great for straightforward, synchronous systems, which admittedly is most things.

Asynchronous functions, functions which fire callbacks, futures, that kind of thing, are much more difficult to test. A parser, for example, generates a series of tokens. Altering a value on a spreadsheet could cause a cascade of recalculations. Choosing a different child window might result in a half-a-dozen menu and toolbar reconfigurations. These things can be crucial to the success of a system, but are among the most difficult to test in an automated way.

That's the problem area :) I'm not sure I have a solution though, so this is a bit of weak proposal at the moment. At least not a solution I've implemented yet - basically I think you have to capture the event stream, and compare it to a known good stream. The compare could be interesting though - is timing crucial, is ordering important, can some events be ignored, etc, etc? Motivating example here is the XSLT engine I'm writing.

Just rehung my bird feeder. Everytime I refill it, it gets attached by squirrels within a day or two. We don't usually see squirrels in the garden, so I don't know how they know, but they do. And when I say squirrels, I mean several squirrels at the same time. They're like a little squirrel commando unit, scambling up over the fence, across the lawn and into the holly tree. Then one or two of them hang onto the feeder and swing around until the bit of wire it hangs on unwraps from the branch. Squirrels and feed fall to the floor, if they're lucky the lids comes off and, wahey, peanut feast all around. In the most recent raid, one of them spent several minutes secreting nuts around the garden afterwards, before stuffing his fat little squirrelface.

I'm slightly irritated the blue tits didn't put up more resistance. It's their peanuts being nicked, after all.

Don't really know if I'm allowed to mention this, since previous copies have had Not for review written in biro on the front, but a new issue of Gavin Burrow's occasional perzine arrived in the post this morning. I don't know why Gav started sending them to me, but I do enjoy receiving them. I've only skimmed it, and it appears full of reviews and articles about art exhibitions I haven't seen and bands I've never heard of. I'm looking forward to settling down later in the week and giving it some of the time it deserves.

If you know, know of, or knew Gavin nee Gav nee Captain Courageous, I suspect he'd probably send you a copy too if you asked nicely. Drop me a line if you need contact details.

Today I am mostly writing WinCVS Python macros. For bonus points (although not, unfortunately, bonus pay) I will be flinging in a bit of Tk gui action, and at least one external Java program. Yay, go me. No, really.

You might have heard of Freecycle. It's a network of local email lists which connect people with stuff to give away with people who can give that stuff a new home. It means you can shift that old sofa/box of kids toys/enormous pile of books/whatever without having it end up in landfill. Over the past few months, as we've smashed our way through the house, I've disposed of a number of things on the local Freecycle list, including bathroom suites, old bits of furniture, and (actually most fun of all) most of my comics collection.

Most offers receive a number of replies, often very quickly indeed. So how do you choose? Initially, I simply went for the first reply I got. In recent months, as more and more people have joined the list, I've got more replies and even more quickly. I now employ an entirely subjective set of criteria including:

• Do not ask me to phone you. This is an email transaction. You've just emailed me remember? If I'm going to choose you, you need to answer my emails.
• Try, please try, not to email me from a Yahoo account with a stupid pseudonym. For one item I offered the first two replies were from "ryan_giggs" and "michealowen" - two gentlemen I suspect can afford their own shower rail. Not an absolute rule - I did quite like bananaisland@...
• Talk fucking English! I took the time to craft my offer email in proper sentences (more or less), accurately describing the item in question, while also trying convey my generally good humoured and easy going nature. Replies that look lik thev bin typd by a drunkj caat wolking on the kbaord are, frankly, just bloody rude.
• Don't ask me to deliver. I've already said you have to collect.
• A bit of backstory helps. Why, exactly, do you want my old piece of flatpack furniture?
• A thank you does wonders.

My trusty old Iiyama VisionMaster monitor has made a massive electrical cracking sparking noise and gone completely black. Arse. On the plus side, this is the first time I haven't had to buy a replacement monitor in a panic :)