[RSS 0.91]
Tuesday 28 August, 2007
#XSLT: Test case state of play

My ongoing XSLT development uses the OASIS XSLT Conformance test cases. I found creating meaningful XPath tests hard enough, but XSLT is a magnitude or two above that and I just didn't fancy it. There's also the danger, of course, that you construct the test case according to your reading of the spec, rather than to the meaning of the spec. Whichever way round, I couldn't have done without them.

The test cases are in two large chunks, one provided by Xalan/IBM/Lotus, the other by Microsoft. Those chunks are then further subdivided. I'm currently running the Xalan tests, for no particular reason than they're listed first in the catalogue. The results, using this morning's Subversion head, are as follows :
attribvaltemplate Run: 12 Failures: 0 Errors: 0 Skips: 1
axes Run: 130 Failures: 0 Errors: 0 Skips: 2
boolean Run: 90 Failures: 0 Errors: 0 Skips: 1
conditional Run: 23 Failures: 0 Errors: 0 Skips: 0
conflictres Run: 35 Failures: 0 Errors: 0 Skips: 1
copy Run: 62 Failures: 0 Errors: 0 Skips: 8
dflt Run: 4 Failures: 0 Errors: 0 Skips: 0
expression Run: 6 Failures: 0 Errors: 0 Skips: 6
extend Run: 4 Failures: 0 Errors: 0 Skips: 4
impincl Run: 29 Failures: 6 Errors: 0 Skips: 2
lre Run: 22 Failures: 11 Errors: 0 Skips: 0
match Run: 32 Failures: 14 Errors: 0 Skips: 1
math Run: 107 Failures: 1 Errors: 0 Skips: 0
mdocs Run: 18 Failures: 0 Errors: 0 Skips: 7
message Run: 16 Failures: 2 Errors: 0 Skips: 2
modes Run: 17 Failures: 1 Errors: 0 Skips: 0
namedtemplate Run: 19 Failures: 2 Errors: 0 Skips: 1
namespace Run: 133 Failures: 39 Errors: 0 Skips: 0
node Run: 21 Failures: 2 Errors: 0 Skips: 0
output Run: 108 Failures: 78 Errors: 0 Skips: 1
position Run: 111 Failures: 7 Errors: 0 Skips: 15
predicate Run: 58 Failures: 0 Errors: 0 Skips: 0
processorinfo Run: 1 Failures: 1 Errors: 0 Skips: 0
reluri Run: 11 Failures: 9 Errors: 0 Skips: 2
select Run: 85 Failures: 2 Errors: 0 Skips: 6
sort Run: 37 Failures: 7 Errors: 0 Skips: 10
string Run: 133 Failures: 4 Errors: 0 Skips: 8
variable Run: 70 Failures: 7 Errors: 0 Skips: 0
ver Run: 5 Failures: 0 Errors: 0 Skips: 4
whitespace Run: 22 Failures: 1 Errors: 0 Skips: 10
Total:Run: 1421 Failures: 194 Errors: 0 Skips: 92

In total, 1329 tests run with 194 failures, for a success rate of 85.4% (or fail rate of 14.6% if you prefer).

A fail means the output Arabica::XSLT generated didn't match the expected output. An error means Arabica::XSLT threw an exception somewhere, didn't compile the stylesheet, or didn't produce any output for some other reason. A skip means that the test wasn't run for some reason - either it was marked not compilable, as compilable but not executable, or just not even touched. A test that's expected not to compile but does, or not to run but does is actually flagged as a fail. (It makes sense if you think about it :)).

The majority of the skips are because of unimplemented XSLT elements or functions (xsl:number, xsl:strip-space, xsl:preserve-space, xsl:key, key(), id()), or because the test output is HTML which Arabica:XSLT doesn't support. The biggest set of fails is the output tests (78 out of 108 tests fail), mainly because the output is text rather than XML. The test harness can't actually handle text output cases yet, so tests which generate text output will always appear as failures.

I haven't added these numbers up for a while. I've got to say I'm pleased.


[Add a comment]

SourceForge Project Page

Jez Higgins