Test report is wrong when `fork in Test := true`

[tkawachi]

Following test unexpectedly succeeds when fork in Test := true.

import org.scalacheck.Properties
object Spec1 extends Properties("Spec1") {
  property("prop 1") = false
  property("prop 2") = true
}

[info] ! Spec1.prop 1: Falsified after 0 passed tests.
[info] + Spec1.prop 2: OK, proved property.
[info] Passed: Total 0, Failed 0, Errors 0, Passed 0
[success] Total time: 4 s, completed Jul 21, 2015 10:45:18 PM

I confirmed this issue was introduced by 8204261 using git bisect.

Tested by sbt 0.13.8 and scalacheck 1.12.4.

[xuwei-k]

+1

[rickynils]

Weird. Will look into this after my vacation.

[k0ala]

I also observe this with version 1.12.5

[dwijnand]

People interested in this bug might want to see if sbt 0.13.13-RC1 fixes it for them: sbt/sbt#2730

[Philippus]

this bug is still there for me in sbt 0.13.13

[lihaoyi]

This sounds a lot like com-lihaoyi/utest#25 caused by sbt/sbt#3510 (uTest's workaround is linked from the utest issue)

[retronym]

We ran into this in the Scala build as well and I disabled forking. Turns out that took us out of the frying pan and into the fire, as we now get the wrong version of scala-library on the Scalacheck classpath.

So its time to fix this bug :)

When forking tests, the SBT process setup up a server that listens on a stream for events sent by the forked JVM.

This code in the server:

  @annotation.tailrec def react(): Unit = is.readObject match {
    case `Done` =>
      os.writeObject(Done); os.flush()
    case Array(`Error`, s: String) =>
      log.error(s); react()
    case Array(`Warn`, s: String) =>
      log.warn(s); react()
    case Array(`Info`, s: String) =>
      log.info(s); react()
    case Array(`Debug`, s: String) =>
      log.debug(s); react()
    case t: Throwable =>
      log.trace(t); react()
    case Array(group: String, tEvents: Array[Event]) =>
      listeners.foreach(_ startGroup group)
      val event = TestEvent(tEvents)
      listeners.foreach(_ testEvent event)
      val suiteResult = SuiteResult(tEvents)
      results += group -> suiteResult
      listeners.foreach(_ endGroup (group, suiteResult.result))
      react()
  }

Assumes that all events for a given group will be sent in a single batch (results is a Map, not a MultiMap)

Since 8204261, the Scalacheck Framework seems to violate this assumption by creating a different Task for each property. This calls handleEvent, which is implemented by the SBT framework to batch up events from the task and send them up to the server after task completion.

The test interface isn't documented, AFAICT, so this sort of misunderstanding isn't surprising.

I'm going to try to modify Scalacheck to use coarser grained tasks. I might also try to patch SBT to use a multi-map to just deal with the way events are coming.