After moving our Hudson server to a new server, we started to experience a very confusing test failure in the Java Language Levels module.
The test that failed was
AdvancedLevelTest.testShouldBeErrors. It compiles the files in
javalanglevels/testFiles/forAdvancedLevelTest/shouldBreak/ and makes sure that they all convert with errors.
I had tried to figure out what causes this, since we haven’t changed the code base, but I failed to see the connection. Here is a summary of the facts:
- The test failed on denmark.cs on the local partition.
- The test passed on denmark.cs on the network file system.
- The test passed on finland.cs on the network file system.
- The test passed on finland.cs on the /local partition.
From all I can tell, the software (JDK, Ant, JavaCC, etc.) we are using on denmark.cs and finland.cs is the same. It definitely is the same when I switch on denmark.cs from
In the end, it turned out to be a badly written test. It didn’t create a new
LanguageLevelConverter for each file to be converted, so the (expected) errors from a previous conversion counted towards the errors of following conversions. There were three files that converted without errors.
For some reason, the order in which the files in
javalanglevels/testFiles/forAdvancedLevelTest/shouldBreak/ are enumerated and compiled differs on denmark’s local file system. On denmark’s local file system, one of those files that converts without error is processed first, letting the test fail. On other systems, the three tests aren’t processed until later, and the test succeeds even though it shouldn’t.
I fixed the test, but I don’t know the JLL well enough to decide if this is just a problem with the test, or if there is something wrong with the converter.
This was a very confusing, almost mysterious test failure. It was made so hard to figure out by the combination of a badly written test and the unexpected difference in file enumeration order. The latter really should not matter, but the flaw in the test case made it significant.
Hudson 1.328 is available now, and I have already updated.