Arabica XML Toolkit in C++

Visual Studio, how I curse your useless warning C4800

'type' : forcing value to bool 'true' or 'false' (performance warning)
Performance warning, my arse.

XSLT: Implementing position matches[2]

Revisiting position matches at the moment. I've described how position matches need to be written, and the code works and works well.

Except when it doesn't. It actually fails for the less common cases, and it took me a little while to work out why.

Here's the pattern from the test case that showed the problem

foo[@att1='c'][2]

It wants to match the second node in the set of foo elements with an att1 attribute containing 'c'.

Arabica rewriting finds the positional predicate and applies its incorrect magic. The rewritten pattern is equivalent to

foo[2][@att1='c']

which picks out the second foo node if it has an att1 attribute containing 'c'. The difference isn't immediately clear, even when you have the both the incorrect output and the expected output sitting in front of you.

My small crumb of comfort is that if you do want foo[2][@att1='c'], Arabica does do the right thing. That gave me the clue. Arabica implements XSLT match patterns by rewriting them as XPath expressions.

foo[2][@att1='c']

is rewritten as an XPath along the lines of

self::foo[. = parent::*/foo[2]][@att1='c']

My faulty rewriting of

foo[@att1='c'][2]

was

self::foo[@att1='c'][. = parent::*/foo[2]]

which you should be able to see is logically identical to the above. What I need is

self::foo[. = parent::*/foo[@att='c'][2]]

I had to work quite hard to see that this is what it should be, despite being pretty familiar with XPath and XSLT use and implementation. It's only been part of my working toolkit for the last 8 years or so, after all. When rewriting a positional match, any preceding predicates must be folded into the rewritten expression. Now I see it, it's pretty obvious.

Taggle: Parameterised on string_type

The Taggle parser in subversion is now parameterised on string_type and string_adaptor, in exactly the same way as the usual Arabica XMLReader class. The two are now equivalent, which means that all the SAX filters, the DOM builder, XPath, and so on can be applied to Taggle.

Moments before I'm about to go to bed, I discover Taggle fails (in a coredumpy way) for documents which have a DOCTYPE declaration

Moments before I'm about to go to bed, I discover Taggle fails (in a coredumpy way) for documents which have a DOCTYPE declaration. Will have a look at a fix in the morning.

Taggle: Building the code

If you've grabbed the code from subversion:

svn co svn://jezuk.dnsalias.net/jezuk/arabica/branches/tagsoup-port

you might be wondering how to build it.

For Visual Studio 2005 users, open up the vs8\taggle.sln project and build away. It should just work. If it doesn't, then check the project build notes for information on setting up search paths and things.

For Unixy types, you will need a mighty three steps:

autoreconf - to create the configure script
./configure - to dig out where the various bits and pieces Arabica needs are, and to create the Makefiles
make - to, erm, make everything

Problems, questions, issues? Get in touch.

Taggle: And there it is ...

Taggle, Arabica's port of the TagSoup HTML parser, now builds and runs. It dodges pretty much every encoding issue on the planet, but as a first go it's really quite pleasing. Give it this -

This is bold, bold italic, italic, normal text

and get this

This is bold, bold italic, italic, normal text (Ok, you have to squint a bit at the indenting, but that's a separate issue.)

If you want to have a play, check out the tagsoup-port branch from subversion:

svn co svn://jezuk.dnsalias.net/jezuk/arabica/branches/tagsoup-port

In examples/Taggle, there's a little command line application that read HTML documents and prints the corrected markup to the console.

I'll merge this back into the trunk in the next few days.

Taggle: Bringing HTML Parsing to Arabica

After a rather intense return work after Christmas, I'm taking a bit of a break from Arabica's XSLT development for something a bit lighter - porting John Cowan's excellent TagSoup package to C++ and Arabica. TagSoup, if your not familiar with it, is

a SAX-compliant parser written in Java that, instead of parsing well-formed or valid XML, parses HTML as it is found in the wild: poor, nasty and brutish, though quite often far from short. TagSoup is designed for people who have to process this stuff using some semblance of a rational application design. By providing a SAX interface, it allows standard XML tools to be applied to even the worst HTML.

Obviously, if you have a SAX parser you can apply all your standard XML techniques - not only SAX filters, but building a DOM, applying XPaths, or XSLT transformations as well.

Cowan describes what TagSoup does as

TagSoup is designed as a parser, not a whole application; it isn't intended to permanently clean up bad HTML, as HTML Tidy does, only to parse it on the fly. Therefore, it does not convert presentation HTML to CSS or anything similar. It does guarantee well-structured results: tags will wind up properly nested, default attributes will appear appropriately, and so on.

The semantics of TagSoup are as far as practical those of actual HTML browsers. In particular, never, never will it throw any sort of syntax error: the TagSoup motto is "Just Keep On Truckin'". But there's much, much more. For example, if the first tag is LI, it will supply the application with enclosing HTML, BODY, and UL tags. Why UL? Because that's what browsers assume in this situation. For the same reason, overlapping tags are correctly restarted whenever possible: text like:

This is bold, bold italic, italic, normal text

gets correctly rewritten as:

This is bold, bold italic, italic, normal text.

Looks simple, doesn't it? Well, that's a simple example and it's still a tricky and awkward result in practice. Cowan's patience in persuing this and what looks like a rather elegant solution is to be applauded. Porting his code to C++ has been pretty quick and painless so far, and I expect the new piece, which I've called Taggle, to be finished pretty soon. Arabica will be stronger for it - thanks John!

XSLT: Implementing position matches

Earlier this week, I outlined the equivalent XPath expressions for XSLT matches which use positions. I've bene avoiding implementing this for a while for a couple of reasons. Firstly, I thought it would take more time than I generally have in one Arabica sitting, i.e. more than an hour. Secondly, I wasn't quite sure how I'd actually go about it. I knew the equivalent expression were correct, I just didn't know how I was going to arrive at them in the code.

Happily, writing it out and then reading it back again later triggered the little flash I needed, and it turns out to be really quite easy. I've spent a bit of time on it this morning, having wrapped up the paying work for Christmas, and I've got the first pass working.

Within Arabica, each match pattern is represented as one or more steps, where each step in the pattern is represented as a TestStepExpression. A TestStepExpression contains some kind of node test (obviously) and one or more predicates. The predicate is the bit in square brackets. A match pattern like this

a[3]

would be compiled into a TestStepExpression contain a NameNodeTest checking for 'a', and a single predicate '3'.

To find expressions that need rewriting is simply

foreach step in the steplist foreach predicate in the step predicatelist if predicate is type NUMBER predicate = rewrite(predicate)

The code is here, if you're interested.

This kind of transformation could be done earlier on, on the AST produced by Arabica's XPath parser. It's easier, however, to operate on the compiled version. For instance, in the case above the numeric predicate could be a number literal, the result of a function, or the result of an arbitrarily complex calculation. Detecting all those cases is actually quite tricky at the AST level. Once the compiled objects are generated, we can just check the predicates return type.

Finding predicates containing calls to the last() or position() functions is going be slightly more work, and probably slightly fiddlier. I'm off to have a crack at that now.

[An hour or so later] ... and I think that's it. Each predicate is an expression, and an expression may contain other expressions and so on. An expression might model a comparision operator, for instance, which would contain expression each for the left and right hand operands. I put together a little walker to zoom through this expression tree looking for instances of the last or position functions. If it finds one, then I just generate the rewrite expression in exactly the same way as before.

XSLT: Position matches

Almost exactly two years ago (glurk!), I wrote a little item about XSLT match patterns and gave a clue to how I implement them. A pattern like

chapter/para

is equivalent to an XPath expression like

boolean(self::para/parent::chapter)


That's how Arabica actually implements it, rewriting the match pattern as an XPath expression during compilation.

This works in every case, except those involving positional matches,
chapter/para[1]
for instance.  In the simple case above, the rewriting is actually quite simple.  You can push each step, here chapter then para, onto a LIFO stack and then pull them off again adding the extra bit of wrapping as you go.  For positional matches, the rewriting is a little more involved.  A match like 
chapter/para[last()] needs to be rewritten as something along the lines of 
self::para[. = parent::*/para[last()]]/parent::chapter although you could probably simplify it to 
self::para[. = parent::chapter/para[last()]]

Arabica::XSLT doesn't do this rewriting at the moment, which is, I feel, the largest single hole in it.  I'm aiming to have a go at in the next few days.  If I can crack it relatively quickly, maybe by this time next year I'll be looking for a new project :)
Positional matches must by the way, at least as far as I can see, be evaluated in this way (even if the underlying implementation is different).  This is why matches like this can be quite expensive in terms of runtime, and are generally discouraged.



XSLT: Test case results spreadsheet
 Uploaded the XSLT test results spreadsheet to Google Docs and, should you be so motivated, the latest results will always be available here.