294 lines
9.1 KiB
Markdown
294 lines
9.1 KiB
Markdown
# Testing ScriptFu using the testing framework
|
|
|
|
## Quick start
|
|
|
|
0. Rebuild PIKA.
|
|
The build must be a non-stable build (nightly/development version.)
|
|
1. View the Pika Error Console dockable
|
|
2. Open the SF Console
|
|
3. Enter '(testing:load-test "tinyscheme.scm")'
|
|
|
|
Expect to finally see a report of testing in the SF Console.
|
|
Also expect to see "Passed" messages, as progress indicators,
|
|
in the Pika Error Console.
|
|
You may also see much extraneous data in the SF Console,
|
|
since as a REPL, it prints the value yielded by each expression.
|
|
|
|
Some extreme test cases may take about a minute.
|
|
If you see a "Pika is not responding" dialog, choose Wait.
|
|
|
|
"tinyscheme.scm" tests the embedded interpreter.
|
|
You can also try "pdb.scm" to test the PDB.
|
|
Or another test script to test a smaller portion.
|
|
|
|
## Organization and naming
|
|
|
|
The test language itself does not name a test.
|
|
|
|
The test scripts are in the repo at /plug-ins/script-fu/test/tests.
|
|
|
|
The filesystem of the repo organizes and names the tests.
|
|
The name of a file or directory indicates what is tested.
|
|
The tests don't know their own names.
|
|
|
|
A test script is usually many tests of one PIKA or ScriptFu object or function.
|
|
There may be many test script files for the same object.
|
|
|
|
Tests and test groups can be organized in directories in the source repo.
|
|
A directory of tests can be named for the PIKA object under test.
|
|
|
|
The leaf files and directories
|
|
are coded into larger test files.
|
|
The larger test files simply load all the files for a PIKA object.
|
|
Loading a file executes the tests and alters testing state.
|
|
|
|
The test files when installed are flattened into one directory.
|
|
Thus a test file that loads many tests loads them from the same top directory.
|
|
|
|
### Major test groups
|
|
|
|
1. PDB: Tests ScriptFu binding to the PIKA PDB.
|
|
2. tinyscheme: Tests the embedded TinyScheme interpreter.
|
|
3. other: Special test programs, often contributed with a new feature of ScriptFu.
|
|
|
|
## Testing State
|
|
|
|
The process of testing produces a state in the testing framework and in Pika.
|
|
|
|
### Testing framework state
|
|
|
|
The test framework state is the count of tests and info about failed tests.
|
|
It accumulates over a session of Gimp
|
|
(more precisely, over a session of ScriptFu Console
|
|
or over a session of any plugin that loads the testing framework.)
|
|
|
|
The tests themselves do not usually reset the test state using '(testing:reset)'.
|
|
|
|
You can get a boolean of the total testing framework state
|
|
using the predicate (testing:all-passed?) .
|
|
|
|
### Pika State
|
|
|
|
Pika state includes open images, installed resources, the selection, etc.
|
|
Testing has side effects on Pika state.
|
|
|
|
To ensure tests succeed, you should test a new install of Pika.
|
|
If you don't mind a few failed tests,
|
|
you can test later than a new install.
|
|
|
|
Tests may require that PIKA be newly started:
|
|
|
|
1. PDB tests may hardcode certain constant ID's and rely on PIKA
|
|
to consistently number ID's.
|
|
|
|
Tests may require that PIKA be newly installed:
|
|
|
|
1. PDB tests may depend on the initial set of Pika resources in ~/.config/PIKA
|
|
|
|
## Building for testing
|
|
|
|
### Non stable build
|
|
|
|
The test framework and test scripts are only installed in a non-stable build.
|
|
|
|
### Line numbers in error messages
|
|
|
|
The test scripts are intended to be portable across platforms
|
|
and robust to changes in the test scripts.
|
|
When testing error conditions (using assert-error)
|
|
the testing framework compares expected prefix of error messages
|
|
with actual error messages.
|
|
To do that requires either that TinyScheme be built without the compile option
|
|
to display file and line number in error messages,
|
|
OR that TinyScheme puts details such as line number as the suffix of error message.
|
|
|
|
In other words, the testing of error conditions is not exact,
|
|
only a prefix of the error message is compared.
|
|
When you are writing such a test,
|
|
write an expected error string that is a prefix that omits details.
|
|
|
|
In libscriptfu/tinyscheme/scheme.h :
|
|
```
|
|
# define SHOW_ERROR_LINE 0
|
|
```
|
|
## Test flavors
|
|
|
|
The testing framework can test normal operation and some error detection.
|
|
The test framework does not test detection of syntax errors because parsing errors
|
|
prevent the test framework from starting.
|
|
|
|
### Unit tests of small fragments
|
|
|
|
1. Normal operation: "assert"
|
|
2. Expected runtime errors: "assert-error"
|
|
|
|
|
|
|
|
### Functional tests of plugins
|
|
|
|
The tests are plugins themselves.
|
|
They are not usually automated, but require manual running and visual inspection.
|
|
They are found in /scripts/test
|
|
|
|
## Testing framework features
|
|
|
|
The "testing.scm" framework is simple.
|
|
Mostly it keeps stats for tests passed/failed
|
|
and some information about failed tests.
|
|
|
|
This section describes the "testing.scm" framework.
|
|
In the future, other test frameworks may coexist.
|
|
|
|
Some contributed tests have their own testing code
|
|
e.g. "byte IO".
|
|
|
|
### Tests are not embedded in the tested source
|
|
|
|
Any tests of Scheme code are NOT annotations
|
|
in the Scheme code they test.
|
|
Tests are separate scripts.
|
|
|
|
### Tests are declarative
|
|
|
|
Tests are declarative, short, and readable.
|
|
They may be ordered or have ordered expressions,
|
|
especially when they test side effects on the Pika state.
|
|
|
|
### Tests can be order independent and repeated
|
|
|
|
Often, you can run tests in any order and repeat tests, up to a point.
|
|
Then test objects that have accumulated
|
|
might start to interfere with certain tests.
|
|
|
|
Tests generally should not hardcode PIKA ID's that PIKA assigns.
|
|
|
|
In general, run a large test, such as pdb.scm or tinyscheme.scm.
|
|
But you can also run a small test such as layer-new.scm.
|
|
Just be aware that if you run tests in an order of your choice,
|
|
and if you repeat tests in the same session,
|
|
you might start to see more errors than on the first run of a test
|
|
after a fresh start of Pika.
|
|
|
|
### Some tests require a clean install
|
|
|
|
Tests of resources may try to create a resource (e.g. brush)
|
|
that a prior run of the test already created
|
|
and that was saved by Pika as a setting.
|
|
|
|
For such tests, you may need to test only after a fresh install of Gimp
|
|
(when the set of resources is the set that Pika installs.)
|
|
|
|
### The test framework does not name or number tests
|
|
|
|
The filesystem names the test files.
|
|
|
|
You identify a test by the code it executes and its order in a file.
|
|
|
|
### Progress
|
|
|
|
The test framework logs progress to the PIKA Error Console
|
|
using pika-message.
|
|
|
|
The test framework displays failures, but not successes, as they occur.
|
|
Display is usually to the SF Console.
|
|
|
|
### History of test results
|
|
|
|
The test framework does not keep a permanent history of test results.
|
|
The test framework does not write into the file system.
|
|
|
|
It does not alter the testing scripts,
|
|
so you can load test scripts by name from a git repo
|
|
without dirtying the repo.
|
|
|
|
Test scripts may test Pika features that write the file system.
|
|
|
|
### Known to fail tests
|
|
|
|
The test framework does not have a feature to ignore tests that fail.
|
|
That is, the framework does not support a third category of test result: known-to-fail.
|
|
Other frameworks might report success even though a known-to-fail test did fail.
|
|
|
|
You can comment out tests that fail.
|
|
|
|
### Tests cannot catch syntax errors
|
|
|
|
The test framework can not test detection of syntax errors
|
|
because parsing errors
|
|
prevent the test framework from starting.
|
|
|
|
## Writing tests
|
|
|
|
See /test/frameworks/testing.scm for more explanation of the testing language.
|
|
|
|
### Writing tests from examples
|
|
|
|
In the "MIT Scheme Reference" you might see examples like:
|
|
|
|
```
|
|
(vector 'a 'b 'c) => #(a b c)
|
|
```
|
|
|
|
The '=>' symbol should be read as 'yields.'
|
|
|
|
You can convert to this test:
|
|
```
|
|
(assert '(equal?
|
|
(vector 'a 'b 'c)
|
|
#(a b c)))
|
|
```
|
|
|
|
Note the left and right hand sides of the MIT spec
|
|
go directly into the test.
|
|
|
|
### Equality in tests
|
|
|
|
The testing framework does not choose the notion of equality.
|
|
|
|
You can choose from among equal? string=? and other predicates.
|
|
Generally you should prefer equal?
|
|
since it tests for object sameness, component by component,
|
|
instead of pointer equality.
|
|
|
|
Often you don't need an equality predicate,
|
|
when the test expression itself has a boolean result.
|
|
|
|
### Quoting in tests
|
|
|
|
Note the use of backquote ` (backtick) and unquote , (comma).
|
|
When writing tests,
|
|
you must often do this to make certain expressions evaluate later,
|
|
after the assert statement starts and installs an error-hook.
|
|
|
|
The backquote makes an expression into data to pass to assert,
|
|
which will evaluate the expression.
|
|
Otherwise, if the expression is evaluated before passing, an error may come before the assert function starts,
|
|
and the test is not properly caught or logged.
|
|
|
|
The unquote undoes the effect of the backquote: it makes the unquoted expression evaluate before passing it to an assert statement.
|
|
|
|
### Defining symbols outside a test expression
|
|
|
|
You can define symbols (say a variable or a function)
|
|
before a test expression
|
|
and refer to that symbol in the test expression
|
|
but you might need to unquote it so it evaluates
|
|
before the test expression function (assert or assert-error)
|
|
is evaluated.
|
|
|
|
## Internationalization
|
|
|
|
We intend the tests are independent of locale
|
|
(the user's preferred language.)
|
|
|
|
There is no test that changes the locale
|
|
as part of the test process.
|
|
(There is no API such as pika-set-locale.)
|
|
|
|
To test that ScriptFu properly internationalizes,
|
|
you must change the locale and retest.
|
|
The printing of numbers is known to fail in German.
|
|
|
|
|
|
|
