Safe Haskell | None |
---|---|
Language | Haskell2010 |
API for test runners
Synopsis
-
data
TestTree
- = forall t. IsTest t => SingleTest TestName t
- | TestGroup TestName [ TestTree ]
- | PlusTestOptions ( OptionSet -> OptionSet ) TestTree
- | forall a. WithResource ( ResourceSpec a) ( IO a -> TestTree )
- | AskOptions ( OptionSet -> TestTree )
- | After DependencyType Expr TestTree
- foldTestTree :: forall b. Monoid b => TreeFold b -> OptionSet -> TestTree -> b
-
data
TreeFold
b =
TreeFold
{
- foldSingle :: forall t. IsTest t => OptionSet -> TestName -> t -> b
- foldGroup :: OptionSet -> TestName -> b -> b
- foldResource :: forall a. OptionSet -> ResourceSpec a -> ( IO a -> b) -> b
- foldAfter :: OptionSet -> DependencyType -> Expr -> b -> b
- trivialFold :: Monoid b => TreeFold b
- data ResourceSpec a = ResourceSpec ( IO a) (a -> IO ())
-
newtype
Traversal
f =
Traversal
{
- getTraversal :: f ()
-
newtype
Ap
f a =
Ap
{
- getApp :: f a
-
data
Ingredient
- = TestReporter [ OptionDescription ] ( OptionSet -> TestTree -> Maybe ( StatusMap -> IO ( Time -> IO Bool )))
- | TestManager [ OptionDescription ] ( OptionSet -> TestTree -> Maybe ( IO Bool ))
- type Time = Double
- tryIngredients :: [ Ingredient ] -> OptionSet -> TestTree -> Maybe ( IO Bool )
- ingredientOptions :: Ingredient -> [ OptionDescription ]
- ingredientsOptions :: [ Ingredient ] -> [ OptionDescription ]
- consoleTestReporter :: Ingredient
- listingTests :: Ingredient
- newtype ListTests = ListTests Bool
- testsNames :: OptionSet -> TestTree -> [ TestName ]
- parseOptions :: [ Ingredient ] -> TestTree -> IO OptionSet
- optionParser :: [ OptionDescription ] -> ([ String ], Parser OptionSet )
- suiteOptionParser :: [ Ingredient ] -> TestTree -> ([ String ], Parser OptionSet )
- defaultMainWithIngredients :: [ Ingredient ] -> TestTree -> IO ()
-
data
Status
- = NotStarted
- | Executing Progress
- | Done Result
- data Result = Result { }
- data Outcome
- data FailureReason
- resultSuccessful :: Result -> Bool
- data Progress = Progress { }
- type StatusMap = IntMap ( TVar Status )
- launchTestTree :: OptionSet -> TestTree -> ( StatusMap -> IO ( Time -> IO a)) -> IO a
-
newtype
NumThreads
=
NumThreads
{
- getNumThreads :: Int
- data DependencyException = DependencyLoop
- suiteOptions :: [ Ingredient ] -> TestTree -> [ OptionDescription ]
- coreOptions :: [ OptionDescription ]
- newtype TestPattern = TestPattern ( Maybe Expr )
- parseExpr :: String -> Maybe Expr
- parseTestPattern :: String -> Maybe TestPattern
- noPattern :: TestPattern
- type Path = Seq String
- exprMatches :: Expr -> Path -> Bool
- testPatternMatches :: TestPattern -> Path -> Bool
- formatMessage :: String -> IO String
- forceElements :: [a] -> ()
- installSignalHandlers :: IO ()
- newtype SignalException = SignalException CInt
- timed :: IO a -> IO ( Time , a)
- getTime :: IO Time
Working with the test tree
The main data structure defining a test suite.
It consists of individual test cases and properties, organized in named groups which form a tree-like hierarchy.
There is no generic way to create a test case. Instead, every test
provider (tasty-hunit, tasty-smallcheck etc.) provides a function to
turn a test case into a
TestTree
.
Groups can be created using
testGroup
.
forall t. IsTest t => SingleTest TestName t |
A single test of some particular type |
TestGroup TestName [ TestTree ] |
Assemble a number of tests into a cohesive group |
PlusTestOptions ( OptionSet -> OptionSet ) TestTree |
Add some options to child tests |
forall a. WithResource ( ResourceSpec a) ( IO a -> TestTree ) |
Acquire the resource before the tests in the inner tree start and
release it after they finish. The tree gets an
|
AskOptions ( OptionSet -> TestTree ) |
Ask for the options and customize the tests based on them |
After DependencyType Expr TestTree |
Only run after all tests that match a given pattern finish
(and, depending on the
|
:: forall b. Monoid b | |
=> TreeFold b |
the algebra (i.e. how to fold a tree) |
-> OptionSet |
initial options |
-> TestTree |
the tree to fold |
-> b |
Fold a test tree into a single value.
The fold result type should be a monoid. This is used to fold multiple
results in a test group. In particular, empty groups get folded into
mempty
.
Apart from pure convenience, this function also does the following useful things:
-
Keeping track of the current options (which may change due to
PlusTestOptions
nodes) - Filtering out the tests which do not match the patterns
Thus, it is preferred to an explicit recursive traversal of the tree.
Note: right now, the patterns are looked up only once, and won't be affected by the subsequent option changes. This shouldn't be a problem in practice; OTOH, this behaviour may be changed later.
An algebra for folding a
TestTree
.
Instead of constructing fresh records, build upon
trivialFold
instead. This way your code won't break when new nodes/fields are
indroduced.
TreeFold | |
|
trivialFold :: Monoid b => TreeFold b Source #
trivialFold
can serve as the basis for custom folds. Just override
the fields you need.
Here's what it does:
-
single tests are mapped to
mempty
(you probably do want to override that) - test groups are returned unmodified
- for a resource, an IO action that throws an exception is passed (you want to override this for runners/ingredients that execute tests)
data ResourceSpec a Source #
ResourceSpec
describes how to acquire a resource (the first field)
and how to release it (the second field).
ResourceSpec ( IO a) (a -> IO ()) |
Monoid generated by
*>
Traversal | |
|
Instances
Applicative f => Semigroup ( Traversal f) Source # | |
Applicative f => Monoid ( Traversal f) Source # | |
Monoid generated by
liftA2
(
<>
)
Starting from GHC 8.6, a similar type is available from Data.Monoid . This type is nevertheless kept for compatibility.
Ingredients
data Ingredient Source #
Ingredient
s make your test suite tasty.
Ingredients represent different actions that you can perform on your test suite. One obvious ingredient that you want to include is one that runs tests and reports the progress and results.
Another standard ingredient is one that simply prints the names of all tests.
Similar to test providers (see
IsTest
), every ingredient may specify
which options it cares about, so that those options are presented to
the user if the ingredient is included in the test suite.
An ingredient can choose, typically based on the
OptionSet
, whether to
run. That's what the
Maybe
is for. The first ingredient that agreed to
run does its work, and the remaining ingredients are ignored. Thus, the
order in which you arrange the ingredients may matter.
Usually, the ingredient which runs the tests is unconditional and thus should be placed last in the list. Other ingredients usually run only if explicitly requested via an option. Their relative order thus doesn't matter.
That's all you need to know from an (advanced) user perspective. Read on if you want to create a new ingredient.
There are two kinds of ingredients.
The first kind is
TestReporter
. If the ingredient that agrees to run
is a
TestReporter
, then tasty will automatically launch the tests and
pass a
StatusMap
to the ingredient. All the ingredient needs to do
then is to process the test results and probably report them to the user
in some way (hence the name).
TestManager
is the second kind of ingredient. It is typically used for
test management purposes (such as listing the test names), although it
can also be used for running tests (but, unlike
TestReporter
, it has
to launch the tests manually if it wants them to be run). It is
therefore more general than
TestReporter
.
TestReporter
is provided
just for convenience.
The function's result should indicate whether all the tests passed.
In the
TestManager
case, it's up to the ingredient author to decide
what the result should be. When no tests are run, the result should
probably be
True
. Sometimes, even if some tests run and fail, it still
makes sense to return
True
.
TestReporter [ OptionDescription ] ( OptionSet -> TestTree -> Maybe ( StatusMap -> IO ( Time -> IO Bool ))) |
For the explanation on how the callback works, see the
documentation for
|
TestManager [ OptionDescription ] ( OptionSet -> TestTree -> Maybe ( IO Bool )) |
tryIngredients :: [ Ingredient ] -> OptionSet -> TestTree -> Maybe ( IO Bool ) Source #
Run the first
Ingredient
that agrees to be run.
If no one accepts the task, return
Nothing
. This is usually a sign of
misconfiguration.
ingredientOptions :: Ingredient -> [ OptionDescription ] Source #
Return the options which are relevant for the given ingredient.
Note that this isn't the same as simply pattern-matching on
Ingredient
. E.g. options for a
TestReporter
automatically include
NumThreads
.
ingredientsOptions :: [ Ingredient ] -> [ OptionDescription ] Source #
Like
ingredientOptions
, but folds over multiple ingredients.
Standard console ingredients
NOTE: the exports in this section are deprecated and will be removed in the future. Please import Test.Tasty.Ingredients.Basic if you need them.
Console test reporter
consoleTestReporter :: Ingredient Source #
A simple console UI
Tests list
listingTests :: Ingredient Source #
The ingredient that provides the test listing functionality
This option, when set to
True
, specifies that we should run in the
«list tests» mode
Instances
Eq ListTests Source # | |
Ord ListTests Source # | |
Defined in Test.Tasty.Ingredients.ListTests |
|
IsOption ListTests Source # | |
Defined in Test.Tasty.Ingredients.ListTests |
testsNames :: OptionSet -> TestTree -> [ TestName ] Source #
Obtain the list of all tests in the suite
Command line handling
parseOptions :: [ Ingredient ] -> TestTree -> IO OptionSet Source #
Parse the command-line and environment options passed to tasty.
Useful if you need to get the options before
defaultMain
is called.
Once within the test tree,
askOption
should be used instead.
The arguments to this function should be the same as for
defaultMainWithIngredients
. If you don't use any custom ingredients,
pass
defaultIngredients
.
optionParser :: [ OptionDescription ] -> ([ String ], Parser OptionSet ) Source #
Generate a command line parser from a list of option descriptions, alongside any related warning messages.
Since: 1.3
suiteOptionParser :: [ Ingredient ] -> TestTree -> ([ String ], Parser OptionSet ) Source #
The command line parser for the test suite, alongside any related warnings.
Since: 1.3
defaultMainWithIngredients :: [ Ingredient ] -> TestTree -> IO () Source #
Parse the command line arguments and run the tests using the provided ingredient list.
When the tests finish, this function calls
exitWith
with the exit code
that indicates whether any tests have failed. See
defaultMain
for
details.
Running tests
Current status of a test
NotStarted |
test has not started running yet |
Executing Progress |
test is being run |
Done Result |
test finished with a given result |
A test result
Result | |
|
Outcome of a test run
Note: this is isomorphic to
. You can use the
Maybe
FailureReason
generic-maybe
package to exploit that.
Success |
test succeeded |
Failure FailureReason |
test failed because of the
|
Instances
Show Outcome Source # | |
Generic Outcome Source # | |
type Rep Outcome Source # | |
Defined in Test.Tasty.Core
type
Rep
Outcome
=
D1
('
MetaData
"Outcome" "Test.Tasty.Core" "tasty-1.4.3-KbMFbPlj5bE7mefj1W94Ui" '
False
) (
C1
('
MetaCons
"Success" '
PrefixI
'
False
) (
U1
::
Type
->
Type
)
:+:
C1
('
MetaCons
"Failure" '
PrefixI
'
False
) (
S1
('
MetaSel
('
Nothing
::
Maybe
Symbol
) '
NoSourceUnpackedness
'
NoSourceStrictness
'
DecidedLazy
) (
Rec0
FailureReason
)))
|
data FailureReason Source #
If a test failed,
FailureReason
describes why
TestFailed |
test provider indicated failure of the code to test, either because the tested code returned wrong results, or raised an exception |
TestThrewException SomeException |
the test code itself raised an exception. Typical cases include missing example input or output files.
Usually, providers do not have to implement this, as their
|
TestTimedOut Integer |
test didn't complete in allotted time |
TestDepFailed |
a dependency of this test failed, so this test was skipped. |
Instances
Show FailureReason Source # | |
Defined in Test.Tasty.Core |
Test progress information.
This may be used by a runner to provide some feedback to the user while a long-running test is executing.
Progress | |
|
type StatusMap = IntMap ( TVar Status ) Source #
Mapping from test numbers (starting from 0) to their status variables.
This is what an ingredient uses to analyse and display progress, and to detect when tests finish.
:: OptionSet | |
-> TestTree | |
-> ( StatusMap -> IO ( Time -> IO a)) |
A callback. First, it receives the
After this callback returns, the test-running threads (if any) are terminated and all resources acquired by tests are released.
The callback must return another callback (of type
|
-> IO a |
Start running the tests (in background, in parallel) and pass control to the callback.
Once the callback returns, stop running the tests.
The number of test running threads is determined by the
NumThreads
option.
newtype NumThreads Source #
Number of parallel threads to use for running tests.
Note that this is
not
included in
coreOptions
.
Instead, it's automatically included in the options for any
TestReporter
ingredient by
ingredientOptions
, because the way test
reporters are handled already involves parallelism. Other ingredients
may also choose to include this option.
Instances
data DependencyException Source #
Exceptions related to dependencies between tests.
DependencyLoop |
Test dependencies form a loop. In other words, test A cannot start until test B finishes, and test B cannot start until test A finishes. |
Instances
Show DependencyException Source # | |
Defined in Test.Tasty.Run |
|
Exception DependencyException Source # | |
Defined in Test.Tasty.Run |
Options
suiteOptions :: [ Ingredient ] -> TestTree -> [ OptionDescription ] Source #
All the options relevant for this test suite. This includes the options for the test tree and ingredients, and the core options.
coreOptions :: [ OptionDescription ] Source #
The list of all core options, i.e. the options not specific to any
provider or ingredient, but to tasty itself. Currently contains
TestPattern
and
Timeout
.
Patterns
newtype TestPattern Source #
TestPattern ( Maybe Expr ) |
Instances
Eq TestPattern Source # | |
Defined in Test.Tasty.Patterns (==) :: TestPattern -> TestPattern -> Bool Source # (/=) :: TestPattern -> TestPattern -> Bool Source # |
|
Show TestPattern Source # | |
Defined in Test.Tasty.Patterns |
|
IsOption TestPattern Source # | |
Defined in Test.Tasty.Patterns |
testPatternMatches :: TestPattern -> Path -> Bool Source #
Utilities
formatMessage :: String -> IO String Source #
Catch possible exceptions that may arise when evaluating a string. For normal (total) strings, this is a no-op.
This function should be used to display messages generated by the test suite (such as test result descriptions).
forceElements :: [a] -> () Source #
Force elements of a list ( https://ro-che.info/articles/2015-05-28-force-list )
installSignalHandlers :: IO () Source #
Install signal handlers so that e.g. the cursor is restored if the test
suite is killed by SIGTERM. Upon a signal, a
SignalException
will be
thrown to the thread that has executed this action.
This function is called automatically from the
defaultMain*
family of
functions. You only need to call it explicitly if you call
tryIngredients
yourself.
This function does nothing on non-UNIX systems or when compiled with GHC older than 7.6.
newtype SignalException Source #
This exception is thrown when the program receives a signal, assuming
installSignalHandlers
was called.
The
CInt
field contains the signal number, as in
Signal
. We don't use that type synonym, however,
because it's not available on non-UNIXes.
Instances
Show SignalException Source # | |
Defined in Test.Tasty.Runners.Utils |
|
Exception SignalException Source # | |
Defined in Test.Tasty.Runners.Utils |