PIKApp/plug-ins/script-fu/test/tests

Testing ScriptFu using the testing framework

Quick start

0. Rebuild PIKA. The build must be a non-stable build (nightly/development version.)
1. View the Pika Error Console dockable
2. Open the SF Console
3. Enter '(testing:load-test "tinyscheme.scm")'

Expect to finally see a report of testing in the SF Console. Also expect to see "Passed" messages, as progress indicators, in the Pika Error Console. You may also see much extraneous data in the SF Console, since as a REPL, it prints the value yielded by each expression.

Some extreme test cases may take about a minute. If you see a "Pika is not responding" dialog, choose Wait.

"tinyscheme.scm" tests the embedded interpreter. You can also try "pdb.scm" to test the PDB. Or another test script to test a smaller portion.

Organization and naming

The test language itself does not name a test.

The test scripts are in the repo at /plug-ins/script-fu/test/tests.

The filesystem of the repo organizes and names the tests. The name of a file or directory indicates what is tested. The tests don't know their own names.

A test script is usually many tests of one PIKA or ScriptFu object or function. There may be many test script files for the same object.

Tests and test groups can be organized in directories in the source repo. A directory of tests can be named for the PIKA object under test.

The leaf files and directories are coded into larger test files. The larger test files simply load all the files for a PIKA object. Loading a file executes the tests and alters testing state.

The test files when installed are flattened into one directory. Thus a test file that loads many tests loads them from the same top directory.

Major test groups

1. PDB: Tests ScriptFu binding to the PIKA PDB.
2. tinyscheme: Tests the embedded TinyScheme interpreter.
3. other: Special test programs, often contributed with a new feature of ScriptFu.

Testing State

The process of testing produces a state in the testing framework and in Pika.

Testing framework state

The test framework state is the count of tests and info about failed tests. It accumulates over a session of Gimp (more precisely, over a session of ScriptFu Console or over a session of any plugin that loads the testing framework.)

The tests themselves do not usually reset the test state using '(testing:reset)'.

You can get a boolean of the total testing framework state using the predicate (testing:all-passed?) .

Pika State

Pika state includes open images, installed resources, the selection, etc. Testing has side effects on Pika state.

To ensure tests succeed, you should test a new install of Pika. If you don't mind a few failed tests, you can test later than a new install.

Tests may require that PIKA be newly started:

1. PDB tests may hardcode certain constant ID's and rely on PIKA to consistently number ID's.

Tests may require that PIKA be newly installed:

1. PDB tests may depend on the initial set of Pika resources in ~/.config/PIKA

Building for testing

Non stable build

The test framework and test scripts are only installed in a non-stable build.

Line numbers in error messages

The test scripts are intended to be portable across platforms and robust to changes in the test scripts. When testing error conditions (using assert-error) the testing framework compares expected prefix of error messages with actual error messages. To do that requires either that TinyScheme be built without the compile option to display file and line number in error messages, OR that TinyScheme puts details such as line number as the suffix of error message.

In other words, the testing of error conditions is not exact, only a prefix of the error message is compared. When you are writing such a test, write an expected error string that is a prefix that omits details.

In libscriptfu/tinyscheme/scheme.h :

# define SHOW_ERROR_LINE 0

Test flavors

The testing framework can test normal operation and some error detection. The test framework does not test detection of syntax errors because parsing errors prevent the test framework from starting.

Unit tests of small fragments

1. Normal operation: "assert"
2. Expected runtime errors: "assert-error"

Functional tests of plugins

The tests are plugins themselves. They are not usually automated, but require manual running and visual inspection. They are found in /scripts/test

Testing framework features

The "testing.scm" framework is simple. Mostly it keeps stats for tests passed/failed and some information about failed tests.

This section describes the "testing.scm" framework. In the future, other test frameworks may coexist.

Some contributed tests have their own testing code e.g. "byte IO".

Tests are not embedded in the tested source

Any tests of Scheme code are NOT annotations in the Scheme code they test. Tests are separate scripts.

Tests are declarative

Tests are declarative, short, and readable. They may be ordered or have ordered expressions, especially when they test side effects on the Pika state.

Tests can be order independent and repeated

Often, you can run tests in any order and repeat tests, up to a point. Then test objects that have accumulated might start to interfere with certain tests.

Tests generally should not hardcode PIKA ID's that PIKA assigns.

In general, run a large test, such as pdb.scm or tinyscheme.scm. But you can also run a small test such as layer-new.scm. Just be aware that if you run tests in an order of your choice, and if you repeat tests in the same session, you might start to see more errors than on the first run of a test after a fresh start of Pika.

Some tests require a clean install

Tests of resources may try to create a resource (e.g. brush) that a prior run of the test already created and that was saved by Pika as a setting.

For such tests, you may need to test only after a fresh install of Gimp (when the set of resources is the set that Pika installs.)

The test framework does not name or number tests

The filesystem names the test files.

You identify a test by the code it executes and its order in a file.

Progress

The test framework logs progress to the PIKA Error Console using pika-message.

The test framework displays failures, but not successes, as they occur. Display is usually to the SF Console.

History of test results

The test framework does not keep a permanent history of test results. The test framework does not write into the file system.

It does not alter the testing scripts, so you can load test scripts by name from a git repo without dirtying the repo.

Test scripts may test Pika features that write the file system.

Known to fail tests

The test framework does not have a feature to ignore tests that fail. That is, the framework does not support a third category of test result: known-to-fail. Other frameworks might report success even though a known-to-fail test did fail.

You can comment out tests that fail.

Tests cannot catch syntax errors

The test framework can not test detection of syntax errors because parsing errors prevent the test framework from starting.

Writing tests

See /test/frameworks/testing.scm for more explanation of the testing language.

Writing tests from examples

In the "MIT Scheme Reference" you might see examples like:

(vector 'a 'b 'c)  =>  #(a b c)

The '=>' symbol should be read as 'yields.'

You can convert to this test:

(assert '(equal?
            (vector 'a 'b 'c)
            #(a b c)))

Note the left and right hand sides of the MIT spec go directly into the test.

Equality in tests

The testing framework does not choose the notion of equality.

You can choose from among equal? string=? and other predicates. Generally you should prefer equal? since it tests for object sameness, component by component, instead of pointer equality.

Often you don't need an equality predicate, when the test expression itself has a boolean result.

Quoting in tests

Note the use of backquote ` (backtick) and unquote , (comma). When writing tests, you must often do this to make certain expressions evaluate later, after the assert statement starts and installs an error-hook.

The backquote makes an expression into data to pass to assert, which will evaluate the expression. Otherwise, if the expression is evaluated before passing, an error may come before the assert function starts, and the test is not properly caught or logged.

The unquote undoes the effect of the backquote: it makes the unquoted expression evaluate before passing it to an assert statement.

Defining symbols outside a test expression

You can define symbols (say a variable or a function) before a test expression and refer to that symbol in the test expression but you might need to unquote it so it evaluates before the test expression function (assert or assert-error) is evaluated.

Internationalization

We intend the tests are independent of locale (the user's preferred language.)

There is no test that changes the locale as part of the test process. (There is no API such as pika-set-locale.)

To test that ScriptFu properly internationalizes, you must change the locale and retest. The printing of numbers is known to fail in German.