mandag den 4. april 2011

Optimization by Release build making nunit test fail?

A strange failure on the build server today: 


Given the test
        [Test]
        public void TestAcceptsDatetime()
        {
          Assert.That(validatorUnderTest.IsValid(DateTime.Now), Is.True);
        }
Which tests the method 

        public bool IsValid(object o)
        {
          return o is DateTime;
        }
I got a passing test when compiling in debug mode. But when submitted to the build server building and executing in release mode, the tests fails. 

The bastard defied manual debugging, since transforming the method being tested to produce a debug print also made the test pass. That is after making the following change
        public bool IsValid(object o)
        {
          Console.WriteLine("o is a " + o.GetType());
          return o is DateTime;
        }

made the test pass and indeed also produced the expected output "o is a System.DateTime".

To me the only reasonable explanation is that the compiler or runtime somehow optimizes away the type information on the object in the method without debug printing, since it sees that the object is not really used. The compiled version seemed to be behaving correctly. 

The test was made to pass by assigning the Datetime.Now to a value in the test. This seem to make the assumptions about the is operator correct, but it is a strange resolution. 

        [Test]
        public void TestAcceptsDatetime()
        {
          var testvalue = DateTime.Now;
 
          Assert.That(validatorUnderTest.IsValid(testvalue), Is.True);
        }

2 kommentarer:

  1. Interesting. It seems it is, indeed, an error. Not in the compiler but the JIT.

    After reproducing it, I took the liberty of digging a bit deeper and ultimately discuss it at Stack Overflow here:

    http://stackoverflow.com/questions/5543692/can-c-is-operator-suffer-under-release-mode-optimization-on-net-4/

    SvarSlet
  2. Thanks for posting on SO.

    It should of course be mentioned that the problems only were related to the tests. Everywhere the simple "is DateTime" method is actually invoked the results was correct; so even though this may be a bug it sn't very severe. The JIT only seem to be making the optimization when the object being tested has scope of the test method invokation.

    SvarSlet