Skip to content

Python API

jevtest is a command-line tool, and the command line is its supported interface. The Python modules below are documented for contributors and for anyone embedding jevtest; they follow the architecture.

Domain

steps

Tests and their steps: one action, then the checks that must hold after it.

Each kind of action is its own type, so a step can only carry the values that kind needs.

Action module-attribute

Action = (
    Do
    | Use
    | Touch
    | Clear
    | TypeText
    | Scroll
    | Swipe
    | ScrollTo
    | Key
    | Wait
    | Background
    | Rotate
    | Location
    | OpenUrl
    | DarkMode
    | Grant
    | Network
    | Screenshot
    | Launch
    | Stop
    | Restart
    | ClearData
    | Reinstall
    | Back
    | Home
    | HideKeyboard
)

Everything a step can do.

Check module-attribute

Check = Expect | See | NotSee

Something that must be true after a step's action.

Do dataclass

Reach a plain-English goal: Jev picks the actions. Quoted values are what it may type.

Use dataclass

Run another test's steps here.

Touch dataclass

Tap, double-tap or long-press the element a target names.

Clear dataclass

Erase a text field.

TypeText dataclass

Type text, into the named field or (without into) the focused one. Typed exactly as written.

Scroll dataclass

Scroll the content.

Swipe dataclass

Swipe across the screen, or on the element a target names.

ScrollTo dataclass

Scroll until the text is on screen.

Key dataclass

Press a named key (enter, delete, ...).

Wait dataclass

Wait a fixed time.

Background dataclass

Send the app to the background for a while, then bring it back.

Rotate dataclass

Rotate the device.

Location dataclass

Set the GPS location.

OpenUrl dataclass

Open a deep link or URL.

DarkMode dataclass

Switch dark appearance on or off.

Grant dataclass

Grant the app a runtime permission.

Network dataclass

Switch Wi-Fi and mobile data on or off.

Screenshot dataclass

Save a screenshot.

Launch dataclass

Launch the app.

Stop dataclass

Stop the app.

Restart dataclass

Stop and launch the app.

ClearData dataclass

Stop the app and clear its data.

Reinstall dataclass

Uninstall and install the build again.

Back dataclass

Go back.

Home dataclass

Go to the home screen.

HideKeyboard dataclass

Close the on-screen keyboard.

Expect dataclass

Jev judges the statement true of the screen.

See dataclass

The text is on screen.

NotSee dataclass

The text is not on screen.

Step dataclass

One action (or none), then the checks that must hold after it.

Attributes:

Name Type Description
action Action | None

What to do, or None for a step that only checks.

checks tuple[Check, ...]

What must be true afterwards, in order.

settings Settings

The settings this step runs with: the file's, with any the step sets for itself.

source object

The step as written in the test file, for reports.

Test dataclass

A named list of steps.

Attributes:

Name Type Description
name str

Unique across a test file and everything it includes.

fresh bool

Start from a clean install (True) or carry on from where the previous test left the app (False).

steps tuple[Step, ...]

The steps, in order.

Suite dataclass

One test file, ready to run.

Attributes:

Name Type Description
path Path

The test file.

apps Mapping[Platform, Path]

The build for each platform, in the order the file lists them.

devices Mapping[Platform, tuple[str, ...]]

The devices for each platform; tests are split across several.

tests tuple[Test, ...]

The tests to run, in order.

library Mapping[str, Test]

Every test use: can name: this file's and those of the files it includes.

variables Mapping[str, str]

The ${NAME} values the file uses.

includes tuple[Path, ...]

The library files it includes, directly or not.

settings Settings

The file's settings. Each step carries its own copy, with the step's own changes.

screen

The screen of the app under test, as a flat list of elements.

Bounds module-attribute

Bounds = tuple[int, int, int, int]

An element's rectangle: left, top, right, bottom, in the device's own units.

Point module-attribute

Point = tuple[int, int]

A point on the screen, in the device's own units.

Element dataclass

One thing on the screen: a button, a text field, a label, a switch, ...

Attributes:

Name Type Description
kind str

What it is: button, text_field, password_field, text, switch, checkbox, image, cell, ...

text str

Its visible label, content description or value.

hint str

Placeholder or hint text.

resource_id str

The Android resource id or iOS accessibility identifier.

bounds Bounds

Where it is on the screen.

enabled bool

Whether it responds to input.

editable bool

Whether it takes typed text.

clickable bool

Whether it responds to a tap.

scrollable bool

Whether its content scrolls.

focused bool

Whether it has input focus.

checked bool | None

On or off for a switch or checkbox; None for anything that can't be checked.

selected bool

Whether it is selected.

value str

A text field's current contents (what clearing it must delete).

id str

Its id on this screen (e1, e2, ...), assigned by Screen.

center property
center: Point

The middle of the element: where a tap lands.

end property
end: Point

A point just inside the right edge: tapping there puts the text cursor after the text.

label
label() -> str

A short description for logs and for Jev, e.g. button 'Sign in'.

Screen dataclass

Everything on the screen at one moment.

Attributes:

Name Type Description
width int

Screen width, in the device's own units.

height int

Screen height, in the device's own units.

elements tuple[Element, ...]

The elements, in reading order. Each gets an id (e1, e2, ...) here.

keyboard_visible bool

Whether the on-screen keyboard is up.

keyboard_top int

Where the keyboard starts (0 when there is none).

content_height property
content_height: int

The height not covered by the keyboard: where page gestures belong.

editable property
editable: tuple[Element, ...]

The elements that take typed text.

by_id
by_id(element_id: str) -> Element

The element with this id.

Raises:

Type Description
KeyError

No element has that id.

texts
texts() -> list[str]

Every visible text and hint on the screen.

shows
shows(text: str) -> bool

Whether any text or hint on the screen contains text, ignoring case.

region
region(el: Element) -> str

Where an element is, in words (top-left ... bottom-right): Jev reads words better than numbers.

decisions

What Jev decided to do next while reaching a do: goal.

Each move is its own type, so a move that acts on an element always has one. describe() is also what Jev is shown as the actions already taken, so its wording is part of every recorded decision: change it and recorded lockfiles no longer match.

Move module-attribute

Move = (
    Finished
    | Impossible
    | WaitForScreen
    | GoBack
    | PressEnter
    | CloseKeyboard
    | ScrollPage
    | TouchElement
    | SwipeElement
    | TypeInto
    | ClearField
)

Every move Jev can pick.

Finished dataclass

The goal is reached.

describe
describe() -> str

done.

Impossible dataclass

The goal can't be reached from this screen.

describe
describe() -> str

impossible.

WaitForScreen dataclass

The screen is still loading: wait for it to change.

describe
describe() -> str

wait.

GoBack dataclass

Go back, or dismiss a dialog or menu.

describe
describe() -> str

back.

PressEnter dataclass

Press Enter to submit what was typed.

describe
describe() -> str

press enter.

CloseKeyboard dataclass

Close the on-screen keyboard.

describe
describe() -> str

hide keyboard.

ScrollPage dataclass

Scroll the page to reveal more content.

describe
describe() -> str

E.g. scroll down.

TouchElement dataclass

Tap, double-tap or long-press an element.

describe
describe() -> str

E.g. tap button 'Sign in'.

SwipeElement dataclass

Swipe on one element, e.g. to delete a list row.

describe
describe() -> str

E.g. swipe_left cell 'Item 3'.

TypeInto dataclass

Type one of the goal's quoted values into a field.

describe
describe() -> str

E.g. type "${EMAIL}" into text_field 'Email'.

ClearField dataclass

Erase a text field.

describe
describe() -> str

E.g. clear text_field 'Email'.

Decision dataclass

A move, with how sure Jev was of it.

Attributes:

Name Type Description
move Move

What to do.

confidence float

Jev's confidence in the chosen action (0 to 1).

probabilities Mapping[str, float]

Jev's probability for each action it was offered.

model

The questions jevtest asks a decision model, and the answers it gets back.

A decision model (Jev) never writes free text: every question offers options and it picks one (Choice), or it gives the probability that something is true (YesNo). These types are what the application layer speaks; the adapter turns them into the model's wire format.

Question module-attribute

Question = Choice | YesNo

A question for the model.

Answer module-attribute

Answer = Picked | Probability

An answer from the model.

State module-attribute

State = Mapping[str, object]

What the model is shown alongside the questions: the screen, and whatever else the question needs.

Choice dataclass

Pick one of options (option id -> description).

YesNo dataclass

The probability that the instructions' question is true.

Picked dataclass

The answer to a Choice: one of its option ids.

Probability dataclass

The answer to a YesNo: the probability of yes (0 to 1).

ModelCall dataclass

One request to the model, for reports and -v output.

Attributes:

Name Type Description
state State

What the model was shown.

questions Mapping[str, Question]

What it was asked.

answers Mapping[str, Answer]

What it answered.

recorded bool

Whether the answers came from the lockfile rather than the model.

ms int

How long the model took (0 when recorded).

cost float

What the request cost in US dollars (0 when recorded).

served_by str | None

The model version that answered.

results

What happened when tests ran: plain records, built by the runner, read by reports and the console.

CheckResult dataclass

How one check went.

Attributes:

Name Type Description
check Check

The check.

status Status

Pass or fail.

detail str | None

What was found (e.g. Jev's probability), or why it failed.

model_calls tuple[ModelCall, ...]

The model requests the check made.

StepResult dataclass

How one step went.

Attributes:

Name Type Description
step Step

The step.

status Status

Pass or fail.

seconds float

How long it took.

detail str | None

What the action did (e.g. which element it tapped), or why it failed.

decisions tuple[Decision, ...]

For a do: step, the moves Jev chose, in order.

model_calls tuple[ModelCall, ...]

The model requests the action made (the checks keep their own).

checks tuple[CheckResult, ...]

The checks that ran, in order; the first failure stops them.

steps tuple[StepResult, ...]

For a use: step, the used test's step results.

screenshot str | None

The screenshot taken when this step failed the test.

failure property
failure: str | None

Plain-English reason for the step's first failure, or None if it passed.

TestResult dataclass

How one test went.

Attributes:

Name Type Description
name str

The test's name.

status Status

Pass or fail.

seconds float

How long it took.

steps tuple[StepResult, ...]

The step results, up to and including the first failure.

start_failure str | None

Why the app could not be started, if it couldn't.

screenshot str | None

The screenshot taken when the app could not be started.

failure property
failure: str | None

Plain-English reason the test failed, or None if it passed.

RunResult dataclass

The results of every test a device ran.

passed property
passed: int

How many tests passed.

failed property
failed: int

How many tests failed.

seconds property
seconds: float

How long the tests took, added up.

failures

The things that can go wrong, as jevtest's own exception types.

Adapters catch the exceptions of the tools they wrap (subprocess, HTTP, YAML, ...) and raise one of these instead, so nothing outside an adapter ever handles a tool's exception. Every message says what to fix.

JevtestError

Bases: Exception

Base class: something jevtest could not do. The message says why and what to fix.

TestFileError

Bases: JevtestError

A test file, a file it includes, its .env, or the command line is wrong.

DeviceError

Bases: JevtestError

The device, its tools, or jevtest's on-device agent couldn't do something.

ModelError

Bases: JevtestError

The decision model (Jev) couldn't be asked, or answered something unusable.

StepFailed

Bases: JevtestError

A step or check didn't get the result the test expects: the test fails, the run goes on.

ports

The interfaces jevtest's core needs from the outside world.

The application layer depends only on these. Adapters implement them; the command line wires implementations to them in one place (jevtest.cli.main), which is also what lets tests pass fakes instead.

Device

Bases: Protocol

A phone, emulator or simulator with the app under test installed on it.

Every method raises DeviceError when the device can't do it.

install
install(app: Path) -> str

Install the build and return its package or bundle id.

check_ready
check_ready() -> None

Raise DeviceError if the device can't be tested right now (e.g. it's locked).

launch
launch() -> None

Launch the app and wait until it is in the foreground.

resume
resume() -> None

Bring the app back to the foreground without restarting it.

stop
stop() -> None

Stop the app.

clear_data
clear_data() -> None

Clear the app's data, as if newly installed.

reinstall
reinstall() -> None

Uninstall and install the build again.

app_state
app_state() -> AppState

Where the app is. Cheap: it doesn't read the screen.

screen
screen() -> Screen

What's on the screen now.

screenshot
screenshot(path: Path) -> None

Save a PNG of the screen.

wait_idle
wait_idle(
    timeout: float, quiet: float | None = None
) -> None

Return once the screen has stopped changing (for quiet seconds), or after timeout seconds.

wait_change
wait_change(timeout: float) -> None

Return as soon as the screen may have changed, or after timeout seconds.

tap
tap(x: int, y: int) -> None

Tap a point.

double_tap
double_tap(x: int, y: int) -> None

Double-tap a point.

long_press
long_press(x: int, y: int) -> None

Press and hold a point.

swipe
swipe(
    direction: Direction,
    element: Element | None = None,
    screen: Screen | None = None,
) -> None

Swipe on an element, or across the part of the screen the keyboard doesn't cover.

scroll
scroll(
    direction: Direction, screen: Screen | None = None
) -> None

Scroll the content so more of it in direction comes into view.

type_text
type_text(text: str, at: Point | None = None) -> None

Type into the focused field, or first focus the field at at.

clear_text
clear_text(element: Element) -> None

Erase a text field.

key
key(name: str) -> None

Press a named key.

back
back() -> None

Go back.

home
home() -> None

Go to the home screen.

hide_keyboard
hide_keyboard() -> None

Close the on-screen keyboard.

rotate
rotate(orientation: Orientation) -> None

Rotate the device. Put back when the device is closed.

set_location
set_location(latitude: float, longitude: float) -> None

Set the GPS location.

open_url
open_url(url: str) -> None

Open a deep link or URL.

dark_mode
dark_mode(on: bool) -> None

Switch dark appearance on or off. Put back when the device is closed.

grant
grant(permission: str) -> None

Grant the app a runtime permission.

network
network(on: bool) -> None

Switch Wi-Fi and mobile data on or off. Put back when the device is closed.

restore
restore() -> None

Put back what steps changed (orientation, appearance, network, location).

Called before each fresh test, so one test's changes never leak into the next.

close
close() -> None

Put back what steps changed and release everything the device started.

DecisionModel

Bases: Protocol

The model that makes jevtest's judgement calls (Jev), behind its lockfile.

calls property
calls: Sequence[ModelCall]

Every request made so far, in order.

ask
ask(
    state: State, questions: Mapping[str, Question]
) -> Mapping[str, Answer]

Ask several questions about one state; get one answer per question.

Raises:

Type Description
ModelError

The model couldn't be asked, or answered outside the options.

Clock

Bases: Protocol

Time, so tests can use a fake one.

now
now() -> float

Seconds on a monotonic clock.

sleep
sleep(seconds: float) -> None

Wait.

RunListener

Bases: Protocol

Told what happens as tests run, so it can be shown as it happens.

test_started
test_started(test: Test) -> None

A test is starting.

start_failed
start_failed(test: Test, reason: str) -> None

The app could not be started for a test.

use_started
use_started(name: str, depth: int) -> None

A use: step is running another test's steps.

step_done
step_done(result: StepResult, depth: int) -> None

A step's action finished (its checks follow).

check_done
check_done(result: CheckResult, depth: int) -> None

A check finished.

test_done
test_done(result: TestResult) -> None

A test finished.

settings

Settings: what a test file may tune, the defaults that suit most apps, and the limits that keep a test honest.

A test file needs no settings at all. Its settings: block changes a default for every step in the file, and a step can change timeout, settle, max_actions, max_scrolls or confidence for itself. Every value must be within LIMITS: a limit is where a setting stops tuning a test and starts hiding a problem in the app.

JEV_MODELS module-attribute

JEV_MODELS = 'typesafe/jev-'

jevtest uses Jev only: a model name must start with this.

LIMITS module-attribute

LIMITS: Mapping[str, tuple[float, float]] = {
    "timeout": (1, 300),
    "settle": (1, 30),
    "max_actions": (1, 50),
    "max_scrolls": (1, 500),
    "confidence": (0.5, 0.99),
}

The lowest and highest value of each numeric setting, inclusive.

WHOLE module-attribute

WHOLE = frozenset({'max_actions', 'max_scrolls'})

Settings that count something, so take whole numbers.

STEP_SETTINGS module-attribute

STEP_SETTINGS = frozenset(LIMITS)

Settings a step can change for itself. model is for the whole file: one file, one Jev.

DEFAULTS module-attribute

DEFAULTS = Settings()

The settings of a file without a settings: block.

Settings dataclass

How steps wait, how far they go, and how sure Jev must be.

Attributes:

Name Type Description
model str

The Jev version that decides and judges. Part of every lockfile key, so changing it re-asks Jev.

timeout float

Seconds a step waits for what it looks for (an element, a check).

settle float

Most seconds to wait for the screen to stop changing after an action. The wait ends as soon as the screen is still, so a higher value only costs time on screens that never stop moving.

max_actions int

Actions a do: goal may take.

max_scrolls int

Scrolls a scroll_to: may make. It also stops at the end of the content.

confidence float

An expect: passes when Jev's probability that the statement is true is above this.

kinds

Small closed sets of values, as enums instead of loose strings.

Platform

Bases: StrEnum

A mobile platform jevtest can test on.

AppState

Bases: StrEnum

Where the app under test is, as the device reports it.

Direction

Bases: StrEnum

A direction to scroll or swipe.

Orientation

Bases: StrEnum

A device orientation.

Gesture

Bases: StrEnum

A touch on one point of the screen.

Status

Bases: StrEnum

The outcome of a test, step or check.

variables

${NAME} values: written as names in test files, filled in only where the app needs them.

VARIABLE module-attribute

VARIABLE = re.compile('\\$\\{([A-Za-z_][A-Za-z0-9_]*)\\}')

A ${NAME} reference.

names_in

names_in(text: str) -> list[str]

The variable names text refers to, in order.

fill

fill(text: str, variables: Mapping[str, str]) -> str

text with each ${NAME} replaced by its value.

Used only for what the app sees (typed text, compared text, searched-for elements, URLs). Logs, reports and Jev's goals keep the names, so secrets stay out of them.

Raises:

Type Description
KeyError

text names a variable that variables doesn't have.

Application

runner

Running tests: each step's action on the device, then its checks, until the first failure.

The runner reaches the device, the model and time only through the domain's ports, reports progress to a RunListener as it goes, and returns plain result records. It prints nothing and writes no files except the screenshots it is asked to take.

APP_MAY_LEAVE module-attribute

APP_MAY_LEAVE = (Stop, ClearData, Reinstall, Home, OpenUrl)

Actions after which the app may be closed or in the background.

NO_SETTLE module-attribute

NO_SETTLE = (
    Stop,
    ClearData,
    Reinstall,
    Home,
    Wait,
    Screenshot,
    ScrollTo,
    Do,
    Use,
)

Actions that do their own waiting, or change nothing on screen.

LAUNCH_QUIET module-attribute

LAUNCH_QUIET = 0.5

Seconds without a change that count as "the app has finished starting" (apps pause longer while starting).

END_OF_CONTENT module-attribute

END_OF_CONTENT = 2

Scrolls in a row that must move nothing before scroll_to: calls it the end. One isn't enough: a real phone's web view sometimes ignores a single scroll.

TestRunner

Runs a suite's tests on one device.

Parameters:

Name Type Description Default
suite Suite

The test file: its tests, the tests use: can name, and its ${NAME} values.

required
device Device

The device, with the app installed.

required
brain Brain

jevtest's judgement, backed by the decision model.

required
screenshots Path

Where failure screenshots (and screenshot: steps) are saved.

required
clock Clock

Time.

required
listener RunListener

Told about each test, step and check as it finishes.

required
run
run() -> RunResult

Run every test in the suite, in order.

run_test
run_test(test: Test) -> TestResult

Start the app, then run the test's steps until one fails.

brain

The questions jevtest asks its decision model, and what it does with the answers.

The model only picks from options it is given, so every question is a Choice over things jevtest can actually do on the current screen, or a YesNo. Text to type always comes from the test file, never from the model.

The wording of the questions and of the screen description is part of every recorded decision: change it and recorded lockfiles no longer match.

Brain

jevtest's judgement: the next move toward a goal, which element a target means, and checks.

Parameters:

Name Type Description Default
model DecisionModel

Where the questions go (Jev behind its lockfile, or a fake).

required
next_action
next_action(
    goal: str, screen: Screen, actions_taken: Sequence[str]
) -> Decision

One model request: the next move toward goal from screen.

The target, field and value are asked in the same request (they're only used when the chosen action needs them), so each move costs one round trip.

locate
locate(
    target: str,
    screen: Screen,
    candidates: Sequence[Element] | None = None,
) -> Element | None

The element the test file names, or None if it isn't on the screen.

Text is matched in code first: an exact label, then an element containing the text. Only a description that isn't on-screen text goes to the model, which picks an element and then must confirm it (a Choice always picks the closest option, even when the target isn't there).

check
check(statement: str, screen: Screen) -> float

The model's probability that statement is true of the screen.

quoted_values

quoted_values(goal: str) -> list[str]

The values a goal puts in quotes: the only text the model may have typed.

describe

describe(screen: Screen) -> list[dict[str, object]]

The screen as the model reads it: one short record per element, positions in words.

planning

Splitting a file's tests across several devices.

shard

shard(
    tests: Sequence[Test], devices: int
) -> list[tuple[Test, ...]]

Deal the tests out to devices devices, in order, one group per device in turn.

A fresh: false test goes where the test before it went, since it carries on from where that one left the app. A device can end up with nothing to run.