Performance considerations

[mpizenberg]

Currently, the design of the supervisor and runners is very straightforward and generic: give one test to each worker. When a worker sends a result, give it another test and transfer the result to reporter.

This design has the potential drawback of much message passing between supervisor and workers (two per test basically). Node-test-runner is using another strategy consisting of splitting the tests in subsets and give each subset to a worker. Workers only respond when finished. This also has drawbacks, namely code complexity and potential unbalanced work load between subsets. Using a work-stealing queue could also reduce the amount of messages (but not guaranteed since communication is needed to transfer work).

Another optimization that would be cool to do is the case of --workers 0 (or --workers 1?). It could be a special case where everything is run in the main thread and no worker is spawned. It has multiple performance advantages for small to medium tests suites, and also probably for CI. Only huge tests suites with many fuzz tests should benefit from a multi-worker architecture in my opinion. To be verified.

1. The tests can be run in one go. No need to exchange with supervisor and wait for another order.
2. The TestResult do not have to be serialized and de-serialized between runner and reporter.
3. Only one Elm main is compiled instead of two (runner and reporter).

But as with all performance matters, measure first!