Python API¶
jevtest is a command-line tool, and the command line is its supported interface. The Python modules below are documented for contributors and for anyone embedding jevtest; they follow the architecture.
Domain¶
steps ¶
Tests and their steps: one action, then the checks that must hold after it.
Each kind of action is its own type, so a step can only carry the values that kind needs.
Action
module-attribute
¶
Action = (
Do
| Use
| Touch
| Clear
| TypeText
| Scroll
| Swipe
| ScrollTo
| Key
| Wait
| Background
| Rotate
| Location
| OpenUrl
| DarkMode
| Grant
| Network
| Screenshot
| Launch
| Stop
| Restart
| ClearData
| Reinstall
| Back
| Home
| HideKeyboard
)
Everything a step can do.
Check
module-attribute
¶
Something that must be true after a step's action.
Do
dataclass
¶
Reach a plain-English goal: Jev picks the actions. Quoted values are what it may type.
Use
dataclass
¶
Run another test's steps here.
Touch
dataclass
¶
Tap, double-tap or long-press the element a target names.
Clear
dataclass
¶
Erase a text field.
TypeText
dataclass
¶
Type text, into the named field or (without into) the focused one. Typed exactly as written.
Scroll
dataclass
¶
Scroll the content.
Swipe
dataclass
¶
Swipe across the screen, or on the element a target names.
ScrollTo
dataclass
¶
Scroll until the text is on screen.
Key
dataclass
¶
Press a named key (enter, delete, ...).
Wait
dataclass
¶
Wait a fixed time.
Background
dataclass
¶
Send the app to the background for a while, then bring it back.
Rotate
dataclass
¶
Rotate the device.
Location
dataclass
¶
Set the GPS location.
OpenUrl
dataclass
¶
Open a deep link or URL.
DarkMode
dataclass
¶
Switch dark appearance on or off.
Grant
dataclass
¶
Grant the app a runtime permission.
Network
dataclass
¶
Switch Wi-Fi and mobile data on or off.
Screenshot
dataclass
¶
Save a screenshot.
Launch
dataclass
¶
Launch the app.
Stop
dataclass
¶
Stop the app.
Restart
dataclass
¶
Stop and launch the app.
ClearData
dataclass
¶
Stop the app and clear its data.
Reinstall
dataclass
¶
Uninstall and install the build again.
Back
dataclass
¶
Go back.
Home
dataclass
¶
Go to the home screen.
HideKeyboard
dataclass
¶
Close the on-screen keyboard.
Expect
dataclass
¶
Jev judges the statement true of the screen.
See
dataclass
¶
The text is on screen.
NotSee
dataclass
¶
The text is not on screen.
Step
dataclass
¶
One action (or none), then the checks that must hold after it.
Attributes:
| Name | Type | Description |
|---|---|---|
action |
Action | None
|
What to do, or None for a step that only checks. |
checks |
tuple[Check, ...]
|
What must be true afterwards, in order. |
settings |
Settings
|
The settings this step runs with: the file's, with any the step sets for itself. |
source |
object
|
The step as written in the test file, for reports. |
Test
dataclass
¶
A named list of steps.
Attributes:
| Name | Type | Description |
|---|---|---|
name |
str
|
Unique across a test file and everything it includes. |
fresh |
bool
|
Start from a clean install (True) or carry on from where the previous test left the app (False). |
steps |
tuple[Step, ...]
|
The steps, in order. |
Suite
dataclass
¶
One test file, ready to run.
Attributes:
| Name | Type | Description |
|---|---|---|
path |
Path
|
The test file. |
apps |
Mapping[Platform, Path]
|
The build for each platform, in the order the file lists them. |
devices |
Mapping[Platform, tuple[str, ...]]
|
The devices for each platform; tests are split across several. |
tests |
tuple[Test, ...]
|
The tests to run, in order. |
library |
Mapping[str, Test]
|
Every test |
variables |
Mapping[str, str]
|
The |
includes |
tuple[Path, ...]
|
The library files it includes, directly or not. |
settings |
Settings
|
The file's settings. Each step carries its own copy, with the step's own changes. |
screen ¶
The screen of the app under test, as a flat list of elements.
Bounds
module-attribute
¶
An element's rectangle: left, top, right, bottom, in the device's own units.
Element
dataclass
¶
One thing on the screen: a button, a text field, a label, a switch, ...
Attributes:
| Name | Type | Description |
|---|---|---|
kind |
str
|
What it is: button, text_field, password_field, text, switch, checkbox, image, cell, ... |
text |
str
|
Its visible label, content description or value. |
hint |
str
|
Placeholder or hint text. |
resource_id |
str
|
The Android resource id or iOS accessibility identifier. |
bounds |
Bounds
|
Where it is on the screen. |
enabled |
bool
|
Whether it responds to input. |
editable |
bool
|
Whether it takes typed text. |
clickable |
bool
|
Whether it responds to a tap. |
scrollable |
bool
|
Whether its content scrolls. |
focused |
bool
|
Whether it has input focus. |
checked |
bool | None
|
On or off for a switch or checkbox; None for anything that can't be checked. |
selected |
bool
|
Whether it is selected. |
value |
str
|
A text field's current contents (what clearing it must delete). |
id |
str
|
Its id on this screen (e1, e2, ...), assigned by |
Screen
dataclass
¶
Everything on the screen at one moment.
Attributes:
| Name | Type | Description |
|---|---|---|
width |
int
|
Screen width, in the device's own units. |
height |
int
|
Screen height, in the device's own units. |
elements |
tuple[Element, ...]
|
The elements, in reading order. Each gets an id (e1, e2, ...) here. |
keyboard_visible |
bool
|
Whether the on-screen keyboard is up. |
keyboard_top |
int
|
Where the keyboard starts (0 when there is none). |
content_height
property
¶
The height not covered by the keyboard: where page gestures belong.
by_id ¶
The element with this id.
Raises:
| Type | Description |
|---|---|
KeyError
|
No element has that id. |
shows ¶
Whether any text or hint on the screen contains text, ignoring case.
region ¶
Where an element is, in words (top-left ... bottom-right): Jev reads words better than numbers.
decisions ¶
What Jev decided to do next while reaching a do: goal.
Each move is its own type, so a move that acts on an element always has one. describe() is also what Jev is
shown as the actions already taken, so its wording is part of every recorded decision: change it and recorded
lockfiles no longer match.
Move
module-attribute
¶
Move = (
Finished
| Impossible
| WaitForScreen
| GoBack
| PressEnter
| CloseKeyboard
| ScrollPage
| TouchElement
| SwipeElement
| TypeInto
| ClearField
)
Every move Jev can pick.
Impossible
dataclass
¶
The goal can't be reached from this screen.
WaitForScreen
dataclass
¶
The screen is still loading: wait for it to change.
PressEnter
dataclass
¶
Press Enter to submit what was typed.
ScrollPage
dataclass
¶
Scroll the page to reveal more content.
TouchElement
dataclass
¶
Tap, double-tap or long-press an element.
SwipeElement
dataclass
¶
Swipe on one element, e.g. to delete a list row.
TypeInto
dataclass
¶
Type one of the goal's quoted values into a field.
ClearField
dataclass
¶
Erase a text field.
model ¶
The questions jevtest asks a decision model, and the answers it gets back.
A decision model (Jev) never writes free text: every question offers options and it picks one (Choice), or it
gives the probability that something is true (YesNo). These types are what the application layer speaks; the
adapter turns them into the model's wire format.
State
module-attribute
¶
What the model is shown alongside the questions: the screen, and whatever else the question needs.
Choice
dataclass
¶
Pick one of options (option id -> description).
YesNo
dataclass
¶
The probability that the instructions' question is true.
Picked
dataclass
¶
The answer to a Choice: one of its option ids.
Probability
dataclass
¶
The answer to a YesNo: the probability of yes (0 to 1).
ModelCall
dataclass
¶
One request to the model, for reports and -v output.
Attributes:
| Name | Type | Description |
|---|---|---|
state |
State
|
What the model was shown. |
questions |
Mapping[str, Question]
|
What it was asked. |
answers |
Mapping[str, Answer]
|
What it answered. |
recorded |
bool
|
Whether the answers came from the lockfile rather than the model. |
ms |
int
|
How long the model took (0 when recorded). |
cost |
float
|
What the request cost in US dollars (0 when recorded). |
served_by |
str | None
|
The model version that answered. |
results ¶
What happened when tests ran: plain records, built by the runner, read by reports and the console.
CheckResult
dataclass
¶
StepResult
dataclass
¶
How one step went.
Attributes:
| Name | Type | Description |
|---|---|---|
step |
Step
|
The step. |
status |
Status
|
Pass or fail. |
seconds |
float
|
How long it took. |
detail |
str | None
|
What the action did (e.g. which element it tapped), or why it failed. |
decisions |
tuple[Decision, ...]
|
For a |
model_calls |
tuple[ModelCall, ...]
|
The model requests the action made (the checks keep their own). |
checks |
tuple[CheckResult, ...]
|
The checks that ran, in order; the first failure stops them. |
steps |
tuple[StepResult, ...]
|
For a |
screenshot |
str | None
|
The screenshot taken when this step failed the test. |
failure
property
¶
Plain-English reason for the step's first failure, or None if it passed.
TestResult
dataclass
¶
How one test went.
Attributes:
| Name | Type | Description |
|---|---|---|
name |
str
|
The test's name. |
status |
Status
|
Pass or fail. |
seconds |
float
|
How long it took. |
steps |
tuple[StepResult, ...]
|
The step results, up to and including the first failure. |
start_failure |
str | None
|
Why the app could not be started, if it couldn't. |
screenshot |
str | None
|
The screenshot taken when the app could not be started. |
failures ¶
The things that can go wrong, as jevtest's own exception types.
Adapters catch the exceptions of the tools they wrap (subprocess, HTTP, YAML, ...) and raise one of these instead, so nothing outside an adapter ever handles a tool's exception. Every message says what to fix.
JevtestError ¶
Bases: Exception
Base class: something jevtest could not do. The message says why and what to fix.
TestFileError ¶
DeviceError ¶
Bases: JevtestError
The device, its tools, or jevtest's on-device agent couldn't do something.
ModelError ¶
Bases: JevtestError
The decision model (Jev) couldn't be asked, or answered something unusable.
StepFailed ¶
Bases: JevtestError
A step or check didn't get the result the test expects: the test fails, the run goes on.
ports ¶
The interfaces jevtest's core needs from the outside world.
The application layer depends only on these. Adapters implement them; the command line wires implementations to
them in one place (jevtest.cli.main), which is also what lets tests pass fakes instead.
Device ¶
Bases: Protocol
A phone, emulator or simulator with the app under test installed on it.
Every method raises DeviceError when the device can't do it.
check_ready ¶
Raise DeviceError if the device can't be tested right now (e.g. it's locked).
wait_idle ¶
Return once the screen has stopped changing (for quiet seconds), or after timeout seconds.
wait_change ¶
Return as soon as the screen may have changed, or after timeout seconds.
swipe ¶
swipe(
direction: Direction,
element: Element | None = None,
screen: Screen | None = None,
) -> None
Swipe on an element, or across the part of the screen the keyboard doesn't cover.
scroll ¶
Scroll the content so more of it in direction comes into view.
type_text ¶
Type into the focused field, or first focus the field at at.
rotate ¶
Rotate the device. Put back when the device is closed.
dark_mode ¶
Switch dark appearance on or off. Put back when the device is closed.
network ¶
Switch Wi-Fi and mobile data on or off. Put back when the device is closed.
restore ¶
Put back what steps changed (orientation, appearance, network, location).
Called before each fresh test, so one test's changes never leak into the next.
DecisionModel ¶
Bases: Protocol
The model that makes jevtest's judgement calls (Jev), behind its lockfile.
ask ¶
Ask several questions about one state; get one answer per question.
Raises:
| Type | Description |
|---|---|
ModelError
|
The model couldn't be asked, or answered outside the options. |
Clock ¶
settings ¶
Settings: what a test file may tune, the defaults that suit most apps, and the limits that keep a test honest.
A test file needs no settings at all. Its settings: block changes a default for every step in the file, and a
step can change timeout, settle, max_actions, max_scrolls or confidence for itself. Every value must be
within LIMITS: a limit is where a setting stops tuning a test and starts hiding a problem in the app.
JEV_MODELS
module-attribute
¶
jevtest uses Jev only: a model name must start with this.
LIMITS
module-attribute
¶
LIMITS: Mapping[str, tuple[float, float]] = {
"timeout": (1, 300),
"settle": (1, 30),
"max_actions": (1, 50),
"max_scrolls": (1, 500),
"confidence": (0.5, 0.99),
}
The lowest and highest value of each numeric setting, inclusive.
WHOLE
module-attribute
¶
Settings that count something, so take whole numbers.
STEP_SETTINGS
module-attribute
¶
Settings a step can change for itself. model is for the whole file: one file, one Jev.
Settings
dataclass
¶
How steps wait, how far they go, and how sure Jev must be.
Attributes:
| Name | Type | Description |
|---|---|---|
model |
str
|
The Jev version that decides and judges. Part of every lockfile key, so changing it re-asks Jev. |
timeout |
float
|
Seconds a step waits for what it looks for (an element, a check). |
settle |
float
|
Most seconds to wait for the screen to stop changing after an action. The wait ends as soon as the screen is still, so a higher value only costs time on screens that never stop moving. |
max_actions |
int
|
Actions a |
max_scrolls |
int
|
Scrolls a |
confidence |
float
|
An |
kinds ¶
Small closed sets of values, as enums instead of loose strings.
Platform ¶
Bases: StrEnum
A mobile platform jevtest can test on.
AppState ¶
Bases: StrEnum
Where the app under test is, as the device reports it.
Direction ¶
Bases: StrEnum
A direction to scroll or swipe.
Orientation ¶
Bases: StrEnum
A device orientation.
Gesture ¶
Bases: StrEnum
A touch on one point of the screen.
Status ¶
Bases: StrEnum
The outcome of a test, step or check.
variables ¶
${NAME} values: written as names in test files, filled in only where the app needs them.
VARIABLE
module-attribute
¶
A ${NAME} reference.
fill ¶
text with each ${NAME} replaced by its value.
Used only for what the app sees (typed text, compared text, searched-for elements, URLs). Logs, reports and Jev's goals keep the names, so secrets stay out of them.
Raises:
| Type | Description |
|---|---|
KeyError
|
|
Application¶
runner ¶
Running tests: each step's action on the device, then its checks, until the first failure.
The runner reaches the device, the model and time only through the domain's ports, reports progress to a
RunListener as it goes, and returns plain result records. It prints nothing and writes no files except the
screenshots it is asked to take.
APP_MAY_LEAVE
module-attribute
¶
Actions after which the app may be closed or in the background.
NO_SETTLE
module-attribute
¶
Actions that do their own waiting, or change nothing on screen.
LAUNCH_QUIET
module-attribute
¶
Seconds without a change that count as "the app has finished starting" (apps pause longer while starting).
END_OF_CONTENT
module-attribute
¶
Scrolls in a row that must move nothing before scroll_to: calls it the end. One isn't enough: a real
phone's web view sometimes ignores a single scroll.
TestRunner ¶
Runs a suite's tests on one device.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
suite
|
Suite
|
The test file: its tests, the tests |
required |
device
|
Device
|
The device, with the app installed. |
required |
brain
|
Brain
|
jevtest's judgement, backed by the decision model. |
required |
screenshots
|
Path
|
Where failure screenshots (and |
required |
clock
|
Clock
|
Time. |
required |
listener
|
RunListener
|
Told about each test, step and check as it finishes. |
required |
brain ¶
The questions jevtest asks its decision model, and what it does with the answers.
The model only picks from options it is given, so every question is a Choice over things jevtest can actually
do on the current screen, or a YesNo. Text to type always comes from the test file, never from the model.
The wording of the questions and of the screen description is part of every recorded decision: change it and recorded lockfiles no longer match.
Brain ¶
jevtest's judgement: the next move toward a goal, which element a target means, and checks.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
DecisionModel
|
Where the questions go (Jev behind its lockfile, or a fake). |
required |
next_action ¶
One model request: the next move toward goal from screen.
The target, field and value are asked in the same request (they're only used when the chosen action needs them), so each move costs one round trip.
locate ¶
locate(
target: str,
screen: Screen,
candidates: Sequence[Element] | None = None,
) -> Element | None
The element the test file names, or None if it isn't on the screen.
Text is matched in code first: an exact label, then an element containing the text. Only a description that isn't on-screen text goes to the model, which picks an element and then must confirm it (a Choice always picks the closest option, even when the target isn't there).
check ¶
The model's probability that statement is true of the screen.
quoted_values ¶
The values a goal puts in quotes: the only text the model may have typed.
describe ¶
The screen as the model reads it: one short record per element, positions in words.