Skip to content

Changelog

0.6.0

Settings. A test file can now tune how steps wait and how far they go, with defaults that suit most apps and limits that keep a test from hiding a broken app. A file without settings runs exactly as before.

  • A settings: block sets timeout, settle, max_actions, max_scrolls, confidence (how sure Jev must be for an expect: to pass) and model for every step in the file.
  • A step can set timeout, settle, max_actions (on do:), max_scrolls (on scroll_to:) or confidence (with expect:) for itself. A setting on a step it means nothing for is an error.
  • Every value has limits (for example timeout 1–300 s, confidence 0.5–0.99). timeout: 0 is no longer accepted.

0.5.0

Rebuilt on a clean architecture. Test files, commands and output are unchanged, and recorded lockfiles still match.

  • Four layers with dependencies pointing inward: domain (types and ports, standard library only), application (the runner), adapters (devices, Jev, files, reports) and cli (the composition root). The rule is checked on every commit by import-linter.
  • Steps, decisions, results and screens are frozen, typed values: one class per action and per move, so a step or a move can only carry what it needs.
  • The package is fully typed (mypy --strict, py.typed), and every public class and function is documented; the docs have a generated Python API page and an Architecture page.
  • Tools' exceptions stop at the adapters; the rest of jevtest sees four failure types.
  • A fresh: true test now also puts back device changes an earlier test made (rotate:, dark_mode:, network:, location), so one failing test can't leave the device rotated for the rest.
  • An iOS app state XCUITest doesn't define is an error, not a guess.
  • scroll_to calls it the end of the content only after two scrolls in a row move nothing: a real phone's web view sometimes ignores one.
  • iOS: while a system alert is up, touches go to SpringBoard, and an agent call may take 150 s, since XCUITest waits up to 60 s for SpringBoard to settle before touching on a real iPhone.
  • Requires Python 3.11 or newer.

0.4.0

No settings. A test file is just app, device, include and tests.

  • settings: is gone. Each step waits up to 10 seconds for what it looks for; a step can say timeout: 30.
  • The rest are fixed rules of jevtest, listed in the docs: 10 actions per do: goal, expect: passes above 0.5, and the Jev model is pinned per release (typesafe/jev-1.13).
  • scroll_to scrolls until the text appears, stopping at the end of the content or after 50 scrolls; it also checks after its last scroll now.
  • A real iPhone needs no team setting: jevtest signs its agent with the team that signed your app.
  • The per-step max_actions and max_scrolls options are gone.

0.3.0

Nothing is assumed any more. Every value a run uses comes from the test file, the .env next to it or the command line; anything missing or wrong is an error that says what to fix.

  • device: is required for every platform, and names must match exactly one device.
  • settings: is required with every value: model, max_actions, max_scrolls (new), timeout, settle, threshold.
  • ios_team is required for a real iPhone; the error lists the teams signed into Xcode.
  • Every test needs fresh: true or fresh: false.
  • Steps are exact: a bare word must be an action (- bakc is an error, not a goal); numbers, text and on/off must be the right type; options must belong to their action; scroll_to needs direction; location is [lat, lon]; grant needs the full Android permission name; key names are exact.
  • The whole file is checked at once and every problem is reported together.
  • .env is read only from next to the test file; a value set differently there and in the environment is an error.
  • jevtest run requires --lock record|frozen|refresh|off (replacing --frozen, --refresh-lock, --no-lock) and --out.
  • In a folder, a YAML file that is neither a test file nor an included library is an error.
  • New --prune-lock removes recorded decisions a full passing run didn't use.
  • Jev retries are printed.
  • rotate, dark_mode, network and simulator location are put back when the run ends, on every platform.
  • Results are always under <platform>/<device>/.

0.2.0

  • ${NAME} values from .env or the environment, kept out of logs, reports and Jev's goals.
  • include: library files for sharing tests.
  • jevtest run takes several files and folders.
  • Several devices per platform, with the tests split across them and run at the same time; platforms run at the same time.
  • Real iPhones: signing, installing and driving the agent over the USB tunnel.

0.1.0

  • First version: YAML tests with actions and checks, Jev through OpenRouter, the lockfile, Android and iOS drivers with on-device agents, JUnit and JSON reports.