Add a safety timeout (with a reasonably large time, not infringing on user-specified timeouts under "normal execution")

[jvoigtlaender]

Probably somewhere around here (either in this function or at its call site):

haskell-template-task/raw/embedded/src/TestHarness.hs

Lines 66 to 76 in 0d393c8

	runTestText :: ShowS -> Test -> IO (Counts, ShowS)
	runTestText = performTest reportStart reportError reportFailure
	where
	reportStart _ = return
	reportError = reportProblem "Error:" "Error in: "
	reportFailure = reportProblem "Failure:" "Failure in: "
	reportProblem p0 p1 _ msg ss us = return $ \rest -> us (line ++ '\n' : rest)
	where
	line = "### " ++ kind ++ path' ++ "\n" ++ msg
	kind = if null path' then p0 else p1
	path' = showPath [x | x@(Label _) <- path ss]

[jvoigtlaender]

Probably in cases where the test suite is of the form [[Test]] (instead of [Test]), this would mean that the timeout applies to each of the inner [Test] lists as a whole, not to each of the "basic" tests therein individually. If one wanted more fine-grained behavior at that point, probably pattern-matching on the Test argument of runTestText would be needed (to unpack TestList cases from this data type: https://hackage.haskell.org/package/HUnit-1.6.2.0/docs/Test-HUnit-Base.html#t:Test).

[marcellussiegburg]

A test case for this issue is: https://autotool.fmi.uni-due.de/route/einsendung/430/590 (without timeouts for each test case)

[jvoigtlaender]

@patritzenfeld, would this solve the "endless computation" issue we encountered this term?

[patritzenfeld]

I'm not sure actually. Tests with timeouts just ran endlessly when used on unproductive recursion of my Picture type. The same kind of tests abort when used on other types, e.g. Lists.

So it could also be the case that this global timeout will be ignored aswell and there is something wrong with HInt or the Picture type itself.

[marcellussiegburg]

If hint keeps running (ignoring timeouts). The resolution would be to sandbox the hint call itself. Call hint in a separate thread/process and kill it when the timeout is over (if it has not finished executing then). [Probably using https://hackage.haskell.org/package/async]

[jvoigtlaender]

for reference: #10 (comment)