Tuesday 28 August, 2007
My ongoing XSLT development uses the OASIS XSLT Conformance test cases. I found creating meaningful XPath tests hard enough, but XSLT is a magnitude or two above that and I just didn't fancy it. There's also the danger, of course, that you construct the test case according to your reading of the spec, rather than to the meaning of the spec. Whichever way round, I couldn't have done without them.
The test cases are in two large chunks, one provided by Xalan/IBM/Lotus, the other by Microsoft. Those chunks are then further subdivided. I'm currently running the Xalan tests, for no particular reason than they're listed first in the catalogue. The results, using this morning's Subversion head, are as follows :
|attribvaltemplate||Run: 12||Failures: 0||Errors: 0||Skips: 1|
|axes||Run: 130||Failures: 0||Errors: 0||Skips: 2|
|boolean||Run: 90||Failures: 0||Errors: 0||Skips: 1|
|conditional||Run: 23||Failures: 0||Errors: 0||Skips: 0|
|conflictres||Run: 35||Failures: 0||Errors: 0||Skips: 1|
|copy||Run: 62||Failures: 0||Errors: 0||Skips: 8|
|dflt||Run: 4||Failures: 0||Errors: 0||Skips: 0|
|expression||Run: 6||Failures: 0||Errors: 0||Skips: 6|
|extend||Run: 4||Failures: 0||Errors: 0||Skips: 4|
|impincl||Run: 29||Failures: 6||Errors: 0||Skips: 2|
|lre||Run: 22||Failures: 11||Errors: 0||Skips: 0|
|match||Run: 32||Failures: 14||Errors: 0||Skips: 1|
|math||Run: 107||Failures: 1||Errors: 0||Skips: 0|
|mdocs||Run: 18||Failures: 0||Errors: 0||Skips: 7|
|message||Run: 16||Failures: 2||Errors: 0||Skips: 2|
|modes||Run: 17||Failures: 1||Errors: 0||Skips: 0|
|namedtemplate||Run: 19||Failures: 2||Errors: 0||Skips: 1|
|namespace||Run: 133||Failures: 39||Errors: 0||Skips: 0|
|node||Run: 21||Failures: 2||Errors: 0||Skips: 0|
|output||Run: 108||Failures: 78||Errors: 0||Skips: 1|
|position||Run: 111||Failures: 7||Errors: 0||Skips: 15|
|predicate||Run: 58||Failures: 0||Errors: 0||Skips: 0|
|processorinfo||Run: 1||Failures: 1||Errors: 0||Skips: 0|
|reluri||Run: 11||Failures: 9||Errors: 0||Skips: 2|
|select||Run: 85||Failures: 2||Errors: 0||Skips: 6|
|sort||Run: 37||Failures: 7||Errors: 0||Skips: 10|
|string||Run: 133||Failures: 4||Errors: 0||Skips: 8|
|variable||Run: 70||Failures: 7||Errors: 0||Skips: 0|
|ver||Run: 5||Failures: 0||Errors: 0||Skips: 4|
|whitespace||Run: 22||Failures: 1||Errors: 0||Skips: 10|
|Total:||Run: 1421||Failures: 194||Errors: 0||Skips: 92|
In total, 1329 tests run with 194 failures, for a success rate of 85.4% (or fail rate of 14.6% if you prefer).
A fail means the output Arabica::XSLT generated didn't match the expected output. An error means Arabica::XSLT threw an exception somewhere, didn't compile the stylesheet, or didn't produce any output for some other reason. A skip means that the test wasn't run for some reason - either it was marked not compilable, as compilable but not executable, or just not even touched. A test that's expected not to compile but does, or not to run but does is actually flagged as a fail. (It makes sense if you think about it :)).
The majority of the skips are because of unimplemented XSLT elements or functions (xsl:number, xsl:strip-space, xsl:preserve-space, xsl:key, key(), id()), or because the test output is HTML which Arabica:XSLT doesn't support. The biggest set of fails is the output tests (78 out of 108 tests fail), mainly because the output is text rather than XML. The test harness can't actually handle text output cases yet, so tests which generate text output will always appear as failures.
I haven't added these numbers up for a while. I've got to say I'm pleased.