Large suites¶
Four features keep a big app's tests manageable. A small app needs none of them.
${NAME} values: secrets and per-machine values¶
Anywhere in a test file, ${NAME} is replaced by the value of NAME:
- from the
.envfile next to the test file, or - from the environment (for CI secrets).
A name that isn't set is an error before anything runs. A name set in both places to different values is also an error: jevtest won't guess which one you meant.
OPENROUTER_API_KEY=sk-or-...
EMAIL=qa@example.com
PASSWORD=correct-horse
ANDROID_DEVICE=Pixel 8
device: { android: "${ANDROID_DEVICE}" }
tests:
- name: Sign in
fresh: true
steps:
- do: Sign in with email "${EMAIL}" and password "${PASSWORD}"
see: Welcome
Where values go. In app and device, values are filled in when the file loads. In steps, a value is filled in only when the app needs it: the text typed, the text compared, the element searched for, the URL opened. Logs, reports and the goals sent to Jev keep ${NAME}.
What Jev can see
Jev reads the screen. A password field shows only dots, but text the app displays (a typed email, a greeting with the user's name) is part of the screen Jev reads, and so part of the lockfile. Keep secrets in password fields.
.env format: one KEY=value per line; export KEY=value and quoted values ("…" or '…') are fine; lines starting with # are comments. Anything else, or a key set twice, is an error with its line number.
include: shared tests¶
A library file holds tests other files use. It has only tests: (and its own include:):
tests:
- name: Sign in
fresh: true
steps:
- do: Sign in with email "${EMAIL}" and password "${PASSWORD}"
see: Welcome
app: build/app.apk
device: { android: Pixel 8 }
include: shared/auth.yaml
tests:
- name: Buy one item
fresh: true
steps:
- use: Sign in
- do: Add the first item to the cart and check out
expect: The order is confirmed
- Tests in an included file don't run on their own; files that include it can
use:them. - Paths are relative to the including file. Libraries can include libraries; a loop is an error.
- Test names must be unique across a file and everything it includes.
${NAME}values in a library come from the.envnext to the test file being run.
Folders¶
runs every test file in tests/ and its subfolders, in name order. Every YAML file in the folder must be either:
- a test file (it has
app:), which runs, or - a library that one of those test files includes.
Anything else is an error that names the file, so a test file that lost its app: line is never silently skipped.
Each file keeps its own lockfile. --test NAME (repeatable) picks tests from any of the files. There is one exit code and one junit.xml for the whole folder. Files run one after another; within a file, devices run at the same time.
Several devices at once¶
List devices to split a platform's tests across them:
- Tests are dealt out in order, one group per device in turn.
- A
fresh: falsetest stays on the device of the test before it, since it continues from where that one left the app. - Each platform in
app:also runs at the same time as the others. - A device left with nothing to run says so:
[android · emulator-5556] no tests left for this device. - While several devices run, each test's log is printed in one piece, tagged with its device, so output never interleaves:
[android · Pixel 8] ▶ Checkout
[android · Pixel 8] ✓ tap: Buy (0.4s) — on button 'Buy'
[android · Pixel 8] PASS Checkout (6.1s)
Each device writes its own report under results/<run>/<platform>/<device>/.
Measured. The 17-test demo suite: Android emulator and iOS simulator one after the other took 104 s + 212 s. The emulator, a Pixel 4a and the simulator at once took 218 s for all 34 test runs.