Agent sessions: run a list of steps

Part of the Agent sessions reference, which lists every call.

Run a list of steps

POST /v1/agent-sessions/{id}/steps

Runs one to eight steps you write, in order, exactly as written: no AI planning, no AI read-back of the page, and none of the session’s AI budget used. Use it when your program already knows what to do — the same steps a message’s intents list, sent directly. Each step has the AgentIntent shape of the OpenAPI document, the same shape a turn reports in results[i].intent.

This is how you drive a hosted session step by step. Every hosted session runs through a proxy of yours, and the step-by-step operations of /v1/sessions (navigate, interact, wait and the rest) answer 503 driver-not-integrated on such a session. Use an agent session instead: POST /message for an AI turn, POST /steps for steps you write.

{
  "steps": [
    { "kind": "navigate", "url": "https://example.com/search" },
    { "kind": "interact", "action": "type", "selector": "#q", "value": "blue shoes" },
    { "kind": "interact", "action": "press", "value": "Enter" },
    { "kind": "capture", "capture": "screenshot" }
  ]
}

What each step looks like

Copy the field names exactly:

A wait waits for something, not for a length of time. It needs a condition: idle (the page settles) or selector_visible (the element named by selector shows). timeoutMs is optional, and is the longest to wait, in milliseconds. There is no ms or seconds field, and a field a step does not take is dropped, not refused: { "kind": "wait", "ms": 1500 } is a wait with no condition. A wait with no valid condition, or a selector_visible wait with no selector, does not run: its row is a failure with diagnosis.category invalid_request, its intent reads { "kind": "wait", "condition": "selector_visible" } whatever you sent, and the next step still runs. To pause for a set time, send a behavioral_pause with duration_ms:

curl -sS -X POST "https://api.driftstack.dev/v1/agent-sessions/$ID/steps" \
  -H "Authorization: Bearer $DRIFTSTACK_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{
    "steps": [
      { "kind": "navigate", "url": "https://example.com/" },
      { "kind": "wait", "condition": "idle", "timeoutMs": 5000 },
      { "kind": "behavioral_pause", "duration_ms": 1500 },
      { "kind": "extract", "selector": "h1" }
    ]
  }'

How steps run

steps uses the same vocabulary as intents on a message’s answer (navigate, interact, wait, capture, scroll, behavioral_pause, back, extract, tap_at, dialog). Every step goes through the same checks a message’s steps go through, and is done the way a person on the phone does it: a tap is a touch, typing uses the on-screen keyboard.

A press step names its key in value, not key. It presses one key of the iPhone’s on-screen keyboard: Enter (or Return), Backspace, Space (each in any letter case), or one printable character, as in { "kind": "interact", "action": "press", "value": "Backspace" }. The key is read only from value: key is not a field a press takes, and a press with no value fails. An iPhone keyboard has no Tab, Escape, arrow, Home, End, Page Up, Page Down or forward Delete key and no modifier keys (Shift, Control, Alt or Option, Command), so a press of any of those is a failed row and nothing is sent, as on the phone. To move to the next field, tap it; to submit a form, press Enter.

A type step with text adds to what the field holds. Sending the same list again under a new Idempotency-Key types the text again after what is already there. A type step with an empty value clears the field: send an empty one, then the new text, to replace what the field held, or type the whole value once. A field holding over 200 characters, or one whose page puts characters back as they are deleted (a fixed prefix such as +31 , or an input mask), cannot be cleared this way: the step fails with nothing typed (see Message). A step that types a saved credential or a session secret clears the field first. A saved credential or a session secret is typed with its placeholder ({{credential:cred_…}} or {{credential:sec_…}}), and only on one of its websites.

A native drop-down list (<select>) is chosen from with a type step. Its selector names the list and its value the option’s text (or its value), as in { "kind": "interact", "action": "type", "selector": "#country", "value": "Canada" }, on a session whose browser has the iPhone’s list picker. A tap on the list or on one of its options is a failed row and nothing is sent. A list inside an embedded frame and a list that takes several choices (multiple) cannot be chosen from yet. How the option is matched: Driving by selector.

After a navigation, the first tap, tap_at, type or key press waits for the page to settle: before the first of those steps on a page that is new since the last one (a navigate, a back, or a page load the browser reported), the session waits until the page has finished loading and its network is quiet, for at most 3 seconds (5 with the time to hear back from the browser), then runs the step either way. A page whose document has been read and shows content, but which is still fetching other files, counts as loaded 3 seconds after its document was read, for this wait and for an idle wait step (rolling out: on only for the accounts it has been turned on for so far; on every other account such a page counts as loaded only once it has finished loading). A session’s first list of steps, and any list after the page changed (an AI turn that navigated, for one), gets this wait before its first such step even with no navigate in it. An idle wait step of yours that the browser answers as come true counts as the page having settled, and no wait of the session’s follows it on that page (a selector_visible wait does not count: an element can show before the page’s script is ready). A change of view with no page load (a single-page app’s own route change) gets none: add a wait step for it. When that wait ran out and the step it preceded is a plain type step whose field held its text, one more wait of the same length follows that step, and none after that: two waits, 10 seconds, per new page at worst. A browser that has stopped answering holds each of these waits for those 5 seconds and no longer; on a session whose browser cannot cancel a command it has started, the step after each such wait can then be held up to 60 seconds more before it is sent, and its own checks before the tap or the keys may be skipped. These waits use the turn’s own time, so a list that runs close to the turn’s time limit can stop a step earlier than it did before. On the page a session was created at, while that page is still loading, an idle wait of yours is answered at once, so the session’s own wait still follows it.

A type step in a list says what the field did, and is never failed or typed again for this check. A plain type step (text of at most 1,000 characters with no line break, no secret, no saved credential or {{credential:…}} placeholder, not inside a frame) reads its field back where the browser can read a field’s live value. Two sentences can close its summary, and the step succeeds either way:

• “the field held the typed text and then no longer showed it; if the text is missing, the page likely re-rendered after loading: add an idle wait with a longer timeout, or a selector_visible wait for something that appears only once the page is ready, before typing” — the field showed the text right after the keys and, after that second wait, holds only what it held before them. A field that uses its text and empties itself (a code field that submits, a coupon that applies itself) reads the same way, so check what the page did before you type the text again;
• “the field did not show the typed text; it may have used it at once, e.g. a tag or code field” — the field held only what it held before the keys as soon as they were typed.

The text itself is never quoted.

The answer is the same shape as a message’s: kind: "plan-executed" with intents (every step in order, the ones not run included), results, ok, and notice / notice_reason when the run ended early for a reason no step says (the session paused, or stopped answering automated steps) — or kind: "stopped" when you called Stop. There is no usage or answer: no model is called.

Each successful step says which page the session reported: page. When enabled on this deployment, a successful row of a list of steps carries page, the url and title the session last reported when that step’s result arrived — what Page state would have answered at that moment (on a session with several tabs, the last report from any of them). A saved credential or session secret typed in the session shows as its placeholder, as in the step’s summary (see what that does not cover):

{
  "kind": "success",
  "intent": { "kind": "extract", "selector": "h1" },
  "summary": "…",
  "page": { "url": "https://example.com/search?q=blue+shoes", "title": "blue shoes - Example" }
}

url or title is null when the session has reported none yet. page is absent when the session has reported no page state, its last report was a loading or stalled one too old to trust, live page state is not available on this deployment, the step’s text was withheld (withheld: true), the address and title could not be checked for a typed saved credential in time, or the session typed a saved credential whose value is no longer known; a failed row never carries it. A page load a step starts can be reported a moment after the step’s result, so after a tap that opens another page, add an idle wait before relying on page. The title is page text: treat it as data. Like values, page is returned in this response and never stored: the transcript and a replay of an Idempotency-Key carry the step without it. A message’s rows (POST /message) do not carry it.

A step that cannot run is a failed row, not a refused request. A step outside the vocabulary, an address that is not a full http:// or https:// address, one on this computer or a private or link-local network (such as 169.254.169.254, or a numeric form of one of those), a name that only a private network answers (one with no dot, such as http://intranet/, one with an empty label, such as http://a..b/, or one ending .internal, .local, .localdomain, .home.arpa, .lan, .corp, .home, .intranet, .private or .svc, or another name that is always this computer, such as localtest.me or lvh.me) or a nip.io, sslip.io, xip.io or traefik.me name that stands for such an address in the forms listed (127.0.0.1.nip.io, 7f000001.nip.io) or for no address at all, one carrying a user name or password, a key no iPhone keyboard has (Tab, Escape, an arrow), an interact swipe (in the vocabulary, but never a step a list takes), a tap on a native list (one is chosen from by a type step naming the option, on a session whose browser has the list picker; a type into one on any other session fails the same way), a frame the page does not have, a frame step this session’s browser cannot do, a frame named by its selector (a frame inside a shadow root) on a session whose browser cannot enter one, or a frame found by its address or name on a session whose browser cannot find one that way — each is a failed row at its place in results whose reason says why. Nothing after it is sent, the steps after it are listed in intents with no result, and ok is false. A wait that times out does not stop the run: its row fails and the next step runs, as in a message.

A capture is a screenshot or a dom_snapshot. pdf is in the step schema, but no session can make a PDF yet: a capture with pdf is a failed row whose reason says so, and nothing after it is sent. A screenshot is the screen as the phone shows it, not the whole page.

A selector must be CSS, on a wait as on a tap. A wait for selector_visible names its element by a CSS selector, as a tap, a type and an extract do. Text pseudo-classes such as :has-text() or :contains(), Playwright >> chaining, a locator with an engine prefix (text=, css=, id=, xpath=, data-testid=), XPath, a # or . with no name after it (a., #, .-) and an escaped id or class that starts with a digit (.1\:x) are not CSS, and a step that uses one is refused before it runs: its row fails with a reason that names the part that is not CSS, diagnosis.category invalid_request and retryable: false, and the step is never sent to the browser. Match on an attribute or on the page’s structure instead (button[type="submit"], form > button:first-child). A test id is an attribute like any other: [data-testid="checkout"]. So is a field’s hint text: [placeholder="Email"] matches only a placeholder that is exactly Email, letter case and spaces included, and [placeholder*="mail"] matches one that contains mail. A refused tap, type or extract stops the run like any step that cannot run; a refused wait does not, and the next step runs. A wait that names a frame is judged on its selector first: one that is not CSS is refused before the session looks for the frame or waits for it.

A comma list and a >>> path in one selector are refused. A selector with a comma at its top level and a >>> (host-a, host-b >>> button) fails before it is sent: diagnosis.category invalid_request, retryable: false, and a reason that is the step’s name followed by “a comma list cannot be combined with >>> yet: split it into separate steps, or use :is() without >>>”. A refused step other than a wait ends the list. Split it into one step per branch, or write the alternatives as :is(…) without >>>. A comma inside parentheses, […] or quotes, or escaped as \,, is not a top-level comma. A scroll by selector on a browser that cannot scroll to a selector is a plain scroll and is not refused.

How long a step waits for a missing element

To check whether an element is there, pass a short timeoutMs on the wait (for example 2000): without one, a wait for an element that never shows holds the list for 30 seconds, or 60 with a frame. On the page itself, an extract by selector answers the same question without waiting. A wait that times out fails its row and the next step runs.

Preview steps (eval accounts)

Some steps are a preview: they run only on an account the preview is switched on for (the evaluation accounts first). The AI plans with them on such an account only.

• A scroll to the side. direction may also be left or right, for a page wider than the screen: { "kind": "scroll", "direction": "right" }. It moves the page itself sideways; a page no wider than the screen does not move, and the step fails with the page did not scroll. On any other account such a step is a failed row, as it always was (does not say whether to scroll up or down), and nothing after it is sent.
• How far, in words. amount instead of amount_px: one_screen (a little less than the screen’s height, or its width for left and right, so some of what was showing stays in view), to_top (all the way up, only with direction: "up") or to_bottom (all the way down, only with direction: "down"), as in { "kind": "scroll", "direction": "down", "amount": "to_bottom" }. A step naming both amount and amount_px, an amount that is not one of these three, or a to_top or to_bottom with another direction is refused with a 400 naming steps.<i>.amount, and nothing is sent. On any other account an amount that is one of these words is a 400 naming steps.<i>.amount too (before, the word was ignored and the step ran as if it named no amount). The step’s line says how far it was asked to go (scrolled down to the bottom); a page longer than the browser’s longest single scroll is not reached in one step, and the line says (capped). A to_top on a page already at the top (or a to_bottom at the bottom) succeeds with scrolled 0px — the page is already at the top.
• A wait for the page address. condition: "url_matches" with url waits until the address of the page contains that text, as in { "kind": "wait", "condition": "url_matches", "url": "/order/confirmed", "timeoutMs": 10000 }. The text is matched exactly as written: letter case counts, and it is plain text, never a pattern (.* matches only an address that holds a dot and a star). Write it as the browser writes the address: printable ASCII only, a host in its ASCII form (xn--…), a space or a letter outside ASCII percent-encoded (/caf%C3%A9, not /café). Some printable characters (", <, >, `, {, }, |, ^) are accepted but the browser percent-encodes them in a path or fragment, so a text holding one written plainly never matches there and the wait times out. Pick a text the address does not hold yet: one it already holds ends the wait at once. url is 1 to 4,096 characters (counted in UTF-16 units; an empty one would match at once), and the wait takes no selector and no frame: each of those, a url with a space or a character outside printable ASCII, and url on any other condition, is refused with a 400 naming the field, and nothing is sent. The address is read by the browser itself; nothing runs in the page. timeoutMs is rounded up to whole seconds; without it the browser waits up to 30 seconds, and never longer than 300. Like any wait, one that times out fails its row (the page address still does not contain the text asked for) and the next step runs. The stored copy of the step keeps no url, so a step list taken from a stored receipt and sent again names no url and is refused with a 400 on an account in the preview: send the url again with it, as for a type step’s text. On any other account such a wait is a failed row, as it always was (while STEP_UNKNOWN_FIELD_REFUSAL is on, its url is a 400 naming steps.<i>.url, as any field a wait does not take).
• A tap or a type by label. On an account this preview is switched on for, and in a list of steps only, a tap or a type may name its control by the label you see instead of a selector: { "kind": "interact", "action": "type", "text_match": "City", "value": "Lyndhurst" }. Send exactly one of selector and text_match: both, or neither, is refused with a 400 (text_match_with_selector, text_match_or_selector_missing). text_match is 1 to 120 characters once spaces at either end are trimmed, on one line (text_match_invalid otherwise); it is refused on any other step (text_match_wrong_step), with a frame (text_match_in_frame) and with double (text_match_double_tap). When the step runs, the page’s controls are read once and the label is matched exactly, ignoring letter case and runs of white space — never a part of a label — against the controls the page shows: a hidden control is never picked, and a type looks only at text fields. The label each control is matched by is the name the browser reads for it: its aria-labelledby or aria-label, else its <label>, else its placeholder, else its own text. So a field with both a <label> and a placeholder is found by its <label> text only (for now). One match: the step runs as a step naming that control’s selector (the row’s intent shows it), with every rule a step by selector has. No match: the row fails with the category text_match_not_found and nothing is sent. More than one: the row fails with text_match_ambiguous, its reason giving the count and up to five of the controls (kind, label, and which name matched), and nothing is sent: use a selector to name one. Two controls that share one selector (a duplicate id) count as two, and a page with more than 60 shown controls cannot prove a single match, so both fail with text_match_ambiguous. A control the browser does not draw (display: none, visibility: hidden, zero size) is never picked; controls inside embedded frames are not considered. A type by a label that names a secret field (password, PIN, card number, and the word for password in several other languages) is marked sensitive; for any other secret field, send sensitive: true. A step by label is stored without its label: the stored copy (the transcript, a recipe, a repeated request’s answer) shows text_match as …[label hidden; …], a mark with a keyed digest of the label, which only tells two steps’ labels apart within the session; send the label itself to run such a step again (the mark is refused with text_match_hidden_label). A step held for approval still names the control’s own words, as every held step does. On any other account text_match is not a field a step takes (while STEP_UNKNOWN_FIELD_REFUSAL is on it is a 400 naming steps.<i>.text_match).

Steps inside an embedded frame

A payment form, a verification check or a chat window is often an embedded frame (an <iframe>), often from another website. A step reaches inside one by naming it in frame: a type, a tap, a wait for selector_visible, an extract with body: true (the frame’s whole text) or, on a session whose browser can read an element inside a frame, an extract by selector (and by attribute), or a capture of dom_snapshot. The path is the browser’s own numbering of the page’s frames, outermost first ([0], or [0, 1] for a frame inside frame 0) — list it with GET /v1/agent-sessions/{id}/frames:

{
  "steps": [
    {
      "kind": "wait",
      "condition": "selector_visible",
      "selector": "input[name=\"cardnumber\"]",
      "frame": [1]
    },
    {
      "kind": "interact",
      "action": "type",
      "selector": "input[name=\"cardnumber\"]",
      "value": "{{credential:sec_4be1c09d2f7a4e6b8c35d1a0f9e27b64}}",
      "frame": [1]
    },
    { "kind": "interact", "action": "tap", "selector": "button[type=\"submit\"]", "frame": [1] }
  ]
}

A wait for an element inside a frame must name that frame, as above: without frame it watches the page itself, where that element never appears. When a selector_visible wait without frame runs out its time and the last list of this page’s frames (from GET /frames, a call that names a frame, or the AI’s own look) held one, its reason ends “— if the element is inside an embedded frame, name the frame in the step (see GET /v1/agent-sessions/{id}/frames)”. On a session whose browser can read an element inside a frame, the session first counts, read-only, the same selector inside each frame that list held (at most 20, at most 5 seconds in all; nothing is typed or tapped). When it matches inside one or more, the reason says where instead: “waited 5 s; #card-number is still not visible — not in the top document; it matches inside embedded frame [2, 0] (pay.example/fields)” — up to three frames, each by its path and the host and path of its address (never its query), then how many more. Name one of them in frame. diagnosis is unchanged either way.

For a call that names a frame, the session asks the browser for the page’s frames when the call begins, and:

• A path the browser does not list is not run. Its row’s reason names the path and points to GET /v1/agent-sessions/{id}/frames, and nothing is sent for it. A frame list taken while the page is still loading may be partial (truncated): a step naming a frame that list does not show is sent anyway, and the browser answers whether the frame is there.

• A wait inside a frame waits for the frame itself first. A payment frame is often inserted a moment after the page loads, so a wait for selector_visible with a frame waits, within its timeoutMs (the browser’s default of 30 seconds when none), for the browser to list that frame — asking again about once a second — and then for the selector inside it, with its full timeoutMs. Only when the frame is still not there at that time is the step a failed row, saying the frame did not appear within the time (condition_not_met, retryable). Other steps inside a frame need the frame to be there already, so wait for it first.

• After a step that may change the page — a navigate, back, tap, typing or key press — or a wait, a later step’s frame is checked again, right before it is sent, against a fresh list from the browser: a page that loaded, removed or replaced a frame renumbers the frames after it, and a path the page does not have then is not run rather than sent to whatever frame now has that number. So navigate followed by steps inside a frame can be one call, and so can a frame the page adds a moment after it loads: wait for its element on the page first (a wait for selector_visible with a selector such as iframe[name="card"] and no frame), then name it.

• Your selector is used as written (but for an id or class that is no CSS name, read as above). It is matched inside that frame, not on the page; an element that is not there fails the step, as it would on the page.

• What the browser cannot do inside a frame is a failed row, saying so: a scroll or press (they act on the page itself), a screenshot, and — on a session whose browser cannot act inside a frame — a tap or a wait, and — on one whose browser cannot read an element inside a frame — an extract by selector or by attribute (there only the whole frame’s text can be read).

• An extract by selector inside a frame succeeds only when the browser confirms it read exactly that frame; otherwise the step fails and nothing of what came back is reported as the frame’s. It changes nothing on the page. With property it reads what a field inside the frame holds now — a card field’s value, its name, autocomplete or outerHTML — returned in the result’s value and never stored. A selector the frame does not have fails the step, naming the selector and the frame:

  {
    "steps": [
      {
        "kind": "extract",
        "selector": "input[name=cardnumber]",
        "property": "value",
        "frame": [0]
      },
      {
        "kind": "extract",
        "selector": "input[name=cardnumber]",
        "property": "autocomplete",
        "frame": [0]
      }
    ]
  }

• An extract with body: true inside a frame that shows no words succeeds, and its summary says why when that is known: “extracted nothing: the embedded frame [0] shows only fields, no text: act on the fields listed under its frame row (read the page again if none are listed)” when the frame holds fields to type into or choose from (a card form: act on those fields by selector; a field’s placeholder or aria-label is not text this read returns), “… shows elements but no readable text (fields, images or a canvas), or has not drawn its content yet: act on the fields listed under its frame row, or read it again (scroll the page toward it first if it is below the fold)” when the frame shows a custom element (whose controls are drawn where a read does not reach), or images, a canvas or buttons (an input button such as a lone pay button counts) and no field or frame, or the browser reports elements but no text, and “… holds other frames: read the inner frame by its path” when the frame holds another frame and no field or custom element, or the browser reports it (name the inner one with a longer path, such as [0, 0]). Otherwise it is “… holds no readable text — it may not have loaded yet (a frame below the fold often loads only when the page is scrolled to it)”.

• An extract by selector inside a frame whose element has no text (a field: an input holds no text of its own) succeeds, and its summary is “extracted nothing: the frame has loaded, and <selector> in embedded frame [0] has no readable text (a field holds no text of its own)”. A selector that matches nothing in the frame fails, as it does on the page. A selector of the bare document skeleton alone (html, head, body, :root, *) proves nothing, since a frame that has not loaded holds those too: it gets the frame words above.

• A saved credential or session secret typed inside a frame is typed only when both the page and the frame are https:// pages on one of its websites, as the browser reports them at that moment, and only into the frame the call listed at that path.

A step inside a frame succeeds only when the browser confirms it was done in exactly that frame (see the frame rules above).

Naming a frame by its address or name. A number goes stale when the page’s scripts add or remove frames. A level of frame can instead be {"src": "…"} — the frame whose address or iframe src contains that text (1 to 512 characters) — or {"name": "…"} — the frame whose iframe name is that text (1 to 256 characters), as in [{"src": "js.example-pay.com/card"}] or [0, {"name": "card-number"}]. No control characters or line breaks. The browser finds the frame when the step runs; it is not checked against the frame list, and a wait waits within its timeoutMs for a match. Exactly one frame must match: none, or more than one, and the step fails with nothing typed or tapped inside a frame (“no frame on the page matches …” / “more than one frame on the page matches …”). Each level is matched only among the frames of the document the levels before it reached, so a frame nested inside another frame needs one level per frame, outermost first: a tap on a button in the inner frame of a widget is

{
  "kind": "interact",
  "action": "tap",
  "selector": "#continue",
  "frame": [{ "name": "linked-accounts" }, { "name": "linked-accounts-inner" }]
}

— [{"name": "linked-accounts-inner"}] alone matches nothing on the page, and a {"src": …} level that matches the outer frame’s address enters the outer frame, where the inner frame’s button is not. A selector the browser does not find inside the frame it entered fails the step as “no element inside frame … matched this selector”, with the document it searched (queried document: url=… readyState=…) — on a session whose browser acts inside frames and lists them, after waiting for it there once (not when the browser scrolled or dragged toward it first: that step is not tried again); a frame inside that one needs a level of its own. The same holds for the numbers from GET /frames: [0, 1], never [1], for frame 1 inside frame 0. A saved credential or session secret is never typed into a frame named this way; name it by its number. On a session whose browser cannot find a frame this way, the step is a failed row with nothing sent (“this browser cannot find a frame by its address or name yet — name it by its number from GET /v1/agent-sessions/{id}/frames”).

A frame that moved is refused, not used. Hosted payment fields often add and remove helper frames while the page is open, so between the list a number came from and the step, [2, 0] can come to name another frame. On a session whose browser checks this, a step that names a frame by its number is done only inside the frame your last GET /v1/agent-sessions/{id}/frames of this page showed at that number — even though the call lists the frames again when it begins — or, when you did not list this page’s frames, the frame the call’s own list showed there. A step that may have changed the page ends the page your list was of: a navigate or back, a tap, typing, a key press, an answered dialog, or a page that loaded during the step. A scroll or swipe of interact does not end it, unless a page load was seen during that step: you chose no new page, and the step after it is still held to your list. A list that GET /frames was still taking while a scroll or swipe of a running call was under way is not used, in that call or after it: list again after the call. A framed step that was refused, with nothing typed or tapped, does not end it — not even one saying a page load may have started — unless a page load was seen during that step. When the page has moved its frames since, the step fails with nothing typed or tapped inside any frame: its reason is “frame [2, 0] is no longer the one the last frame list showed, so nothing was typed or tapped inside it (the frame or page may have scrolled) — list frames again (GET /v1/agent-sessions/{id}/frames) and name it by its url” (diagnosis.category invalid_request, not retryable). The browser checks this again each time it re-enters the frame after a page scroll or a drag it already made, so look at the page before the next step. List the frames again and name the frame by a {"src": "…"} level holding a part of its url that only that frame has, which the browser finds when the step runs. The AI’s own steps fail the same way and its next look lists the frames again.

More than eight steps is refused whole with 400 (validation-failed); nothing is cut. A step field over its limit is a 400 naming the field (steps.2.selector), and so is an extract whose property is not one of the listed ones, or is sent beside attribute or body, or without a selector (steps.1.property), and an extract whose all is set and is not true or false, or is true and sent without a selector or beside body or property (steps.1.all, steps.1.selector, steps.1.body, steps.1.property), or whose index is not a whole number from 0 to 999 or is sent beside "all": true (steps.1.index), with the same rules as all otherwise, and any step that is not an extract and carries an index (steps.1.index: only an extract reads a match by its position); nothing is sent. "all": false reads one match, as if all were not there.

A step field its kind does not take is ignored. Each step takes the fields its kind has in the AgentIntent shape of the OpenAPI document (an extract takes selector, body, attribute, property, all, index and frame; a navigate only url). Any other field — limit or multiple on an extract, a selector on a navigate, all on a step that is not an extract — is dropped before the step runs (index on a step that is not an extract is the one exception: a 400, above): the step runs as if the field were not there, and results[i].intent shows it without it. So a field your step does not take does nothing; check your steps against the shape. A step sent back exactly as an answer listed it (in intents or results[i].intent) holds only its kind’s fields. A field sent as null is not set, and is accepted on any step ("selector": null on a navigate). commits is accepted on any step and ignored. A kind outside the vocabulary is a failed row rather than a 400, and an unknown key at the top of the body (beside steps) is ignored and listed in the x-driftstack-unknown-fields header.

Approving a purchase, payment or account deletion. A step the session stops at for approval comes back as confirmation_required, and nothing after it ran. To approve it, send the rest of the list — from the stopped step on — again, with approve_consequential_actions (the category and matched_text of that result). The approval counts only when the session’s latest transcript entry is that stopped call and the steps you send are its remaining steps (a typed value is compared only where both calls stored it; see the transcript below); otherwise it is ignored and the step stops for approval again. A message cannot resume a stopped list of steps: approving it in a message plans afresh. The field is spelled approve_consequential_actions here too: approvals and the other misspellings listed under Message are refused with 400 (validation-failed) naming the key, rather than ignored.

Who and what it shares with messages. A key with write (account keys only; a session’s control key is not accepted). The session must be under AI control: a session in manual mode, or one a person is driving in pair mode, answers 409 with ai_control_unavailable: true and nothing is written. It counts against its own agent_sessions:steps rate limit, not the agent_sessions:message one: a list of steps runs no AI, so steps and messages cannot use up each other’s rate. It counts against the same running-turns limit as a message, so a list of steps and a message never run on one session at once (409 with turn_in_progress). It takes an Idempotency-Key exactly as a message does; a key used for a message cannot be reused for a list of steps (409 with idempotency_status: "mismatch"). What a step types is not part of the key’s request: two lists that differ only in a typed value are the same request to it, and the second replays the first. Accept: text/event-stream streams the run as a message’s turn does (step, phase, plan, step_start and notice events, then one response).

A plain request is answered within 60 seconds, or it is cut. A request sent without Accept: text/event-stream waits for the whole list to finish. If the list takes longer than 60 seconds, our gateway ends the request with HTTP 504, and the body of that answer is not the JSON error format the API uses everywhere else, so do not parse it. The cut does not close the session and does not stop the steps. The list keeps running in the session, and until it ends a new message or list of steps on that session is refused with 409 and turn_in_progress: true.

A type step can take longer than 60 seconds on its own. Text is typed key by key, and our server waits for a type step for up to 60 seconds plus one second for each character of its text, and never past the list’s own limit of 5 minutes from the start of the call, before it gives the step up as timed out. A later step in the same list has only what is left of those 5 minutes (a step that starts with less than 15 seconds left is still given 15 seconds). A step that clears the field first (an empty value, a saved credential or a session secret) is waited for up to that same limit. After 5 minutes no further step starts, so a running list ends about 5 minutes after its call began at the latest, once the step running at that moment is over. So send long text with Accept: text/event-stream: the server opens the response at once, sends a heartbeat comment about every 15 seconds while the steps run, and then one response event with the result (see Streaming the turn). The runSteps call of the TypeScript SDK, run_steps of the Python SDK and RunSteps of the Go SDK already send that header. Or send the text as several shorter type steps, one call each (with a different Idempotency-Key each time, if you send one); each step adds to what the field holds.

After a 504. Do not send the same text again at once: a type step adds to what the field holds, so a repeat is typed after the first. First wait for the running list to end. Send a list with one read step, such as { "kind": "extract", "selector": "#email", "property": "value" }, and repeat it every 5 to 10 seconds while it answers 409 with turn_in_progress: true; each try counts against the agent_sessions:steps rate limit. When it is answered, it shows what the field holds now: type only what is missing. POST /{id}/stop asks the running list to stop instead of waiting for it: no further step starts, and the step that is running may still finish. If the request that was cut carried an Idempotency-Key, the same request sent again with the same key does not run the steps a second time: it answers 409 with idempotency_status: "in_progress" while the list is still running, and then replays the stored result of the first request (see Idempotency). The replay leaves out any value an extract step read, and a result that could not be stored is answered instead with 409 and code: "response_not_kept". Without a key the result of the cut request is not sent again; the session’s transcript has its entry.

The transcript. Each call adds one entry, with origin: "steps", listing the steps and their outcome; no entry is written for your request itself. The step record never keeps a typed value. (What you write in a message, and what the AI writes back, is kept as written, except card numbers, saved credentials a step typed, and passwords and one-time codes of four or more characters that a step types into a field we recognise as a password or code field — masked wherever they appear in the transcript, including your own messages, once the turn has finished — and secret parameters in web addresses. Other text a step typed, such as a name or a search, or the text of a sign-in or prompt dialog, can appear in the conversation. A recipe saved while its session is running keeps the values you gave it — see Recipes.) A stored type step keeps its field and never carries the typed value: it is marked sensitive: true, or value_omitted: true when the field was read as an open field (only a saved credential’s {{credential:…}} placeholder, or an empty value for a clear, is kept as a value), and that step’s own result row does not repeat it — in the transcript and in the Idempotency-Key receipt alike. The device receives every value as you wrote it, and your own response shows what it reported. A saved credential’s placeholder is stored as written. The next message’s AI sees these entries as steps you ran, not ones it planned. The history the agent works from holds at most 256 entries and 1 MiB, and each call adds one entry. When a call does not fit, the oldest entries are dropped from that history (the first task and the newest entries are kept) and the session goes on; this happens about once every 125 steps calls, and the transcript stream still returns the dropped entries. A dropped step is no longer in the history the AI or a recipe built from the session reads. The session closes with closed_reason: "transcript-limit", before the call sends anything, only when one call alone needs more room than the history has after a drop (a single call with several hundred KiB of text) or when the session has already dropped the most it can (4,096 entries or 16 MiB). session.history in each answer says how much room is left, and warning with code: "history_nearly_full" (fewer than 32 entries left, or the history close to 1 MiB) says a drop is near; you need do nothing. The “still running” answer below has no session, so it carries neither.

“Still running” instead of a cut request. A plain request (without Accept: text/event-stream) whose steps have started and have not ended about 50 seconds after it reached the server is answered 202 with a JSON body kind: "still-running" (and Retry-After: 10 when the call carried an Idempotency-Key), instead of being left for the gateway to cut at 60 seconds. The steps go on: nothing was stopped, failed or refused. The body has session_id; steps_total and steps_completed; running_step_index, the step running now (absent when none is between its start and its end); results_so_far, the rows of the steps that have ended, as a finished call lists them (first answer only; a value an extract step read is in its row here, once); elapsed_ms; ends_within_seconds, an upper bound by which the steps have ended; transcript_length_at_start, the session’s transcript_length before the call; final_result; retry_after_seconds, only when final_result is same_idempotency_key (without a key there is no retry_after_seconds and no Retry-After header: that call cannot be read again); and detail, a sentence to show. How many characters a type step has typed so far is not known until it ends, so it is not reported. The steps have ended within ends_within_seconds at the latest. To learn the outcome when final_result is same_idempotency_key, send the same request again with the same Idempotency-Key, at most once every retry_after_seconds: once the steps have ended it returns the stored result, and while they still run it answers the same 202 again, without results_so_far (each try counts against the agent_sessions:steps rate limit). transcript_length on GET /v1/agent-sessions/{id} going above transcript_length_at_start is a cheaper sign that the steps have ended, but some endings add no transcript entry (the session was closed, a person took control, an internal error), so do not wait on it alone. The key is compared on the session and the steps, not on typed text. A value an extract step reads after the 202 is not in the stored result. If that retry answers 409 with idempotency_status: "in_progress", the result is not available (for example the server restarted; within about 30 minutes after the server restarted, the same key answers 409 with code: "response_not_kept" instead): read the session transcript, and do not send the steps again under a new key. When final_result is transcript_only (no key was sent), the per-step results of the call cannot be read later; the transcript gets its one entry when the steps end normally. Do not send the same steps again to find out whether they have ended: a new call runs as soon as these have ended, and a type step adds to what the field holds. A retry is checked against the page as it is at that moment before its key is read, so a 400 on a retry says nothing about the call that is running. POST /{id}/stop cancels the steps. The event stream never gets this answer.

The 202 body of a list of steps, field by field:

Poll every retry_after_seconds, never faster. Each same-key retry counts against the agent_sessions:steps rate limit.

The SDKs: agentSessions.runSteps(id, steps, opts) in TypeScript, agent_sessions.run_steps(id, steps, ...) in Python (sync and async) and AgentSessions.RunSteps(ctx, id, steps, opts) in Go.

Dialogs

A page can open a JavaScript dialog — an alert, a confirm, a prompt, or a beforeunload prompt asking whether to leave the page — and a site can ask for an HTTP sign-in (http_auth, below). The browser shows it as the iPhone’s own sheet, and while it is open nothing else on the page can be done: every other step fails with diagnosis.category dialog_open (see Message) until the dialog is answered.

Seeing it. On a session whose browser reports dialogs, GET /v1/agent-sessions/{id}/page-state carries the open dialog in page_state.dialog: { "kind": "confirm", "message": "Empty your cart?" } (a prompt’s starting text, when it has one, in default_text). It is null once the dialog has closed, and the key is absent when the session’s browser has never reported a dialog. The AI is shown the open dialog too, on every planning step while it is open, and answers it first. The message is the page’s own text: it is returned to you and shown to the AI. The page state is not stored; a dialog’s text that the AI or you repeat in the conversation is stored with the conversation.

Answering it. A dialog step taps OK ("action": "accept") or Cancel ("action": "dismiss"); to answer a prompt, accept with text, which is typed into its field first (at most 1,000 characters):

{
  "steps": [{ "kind": "dialog", "action": "accept", "text": "Jane Smith" }]
}

Its result says what it answered — never the text you typed:

{
  "kind": "success",
  "intent": { "kind": "dialog", "action": "accept", "text": "{{prompt text not stored}}" },
  "summary": "answered the prompt dialog: OK",
  "dialog": { "handled": true, "kind": "prompt", "action": "accept" }
}

With no dialog open the step still succeeds, answers nothing, and says so: "dialog": { "handled": false, "action": "dismiss" } with the summary no dialog was open, so nothing was answered. text with "action": "dismiss" is a 400 naming the field, and so is text when the dialog the session reports open is not a prompt or a sign-in sheet (an alert, a confirm and a leave-page prompt have no field to type into).

The text typed into a prompt is never kept. A prompt can ask for a password or a code, so the text is sent to the browser and nowhere else: the response, the step stream, the stored transcript, an Idempotency-Key replay, a recipe and every log line show the step with {{prompt text not stored}} in its place. A stored step carrying that marker is never answered — run again from a recipe, or resumed after an approval, it fails without being sent ("not answered: the text typed into a prompt is never stored, …"); send the step again with its text.

OK on a dialog that asks to pay, buy or delete an account is checked like a tap. An accept of a confirm or a leave-page prompt whose message names a purchase, a payment or an account deletion — by the same words a button would be checked for (Place your order, Confirm payment, Delete my account) — stops before it is sent with a confirmation_required result, as a tap on such a button does, unless the account’s purchase_check_mode lets it through (Confirming a purchase, payment or account deletion). Send it again with approve_consequential_actions to let it through; dismiss is never held.

A site that asks for an HTTP sign-in. When a site answers with an HTTP Basic or Digest challenge (a 401), the browser shows the iPhone’s own sign-in sheet, as Safari does, and the page shows nothing until it is answered. The session reports it as an open dialog of kind http_auth, whose message is Sign in to <host> (the host that asks, nothing more): { "kind": "http_auth", "message": "Sign in to the-internet.herokuapp.com" }. dismiss taps Cancel. accept signs in, and its text is the sign-in:

• <username>:<password>, split at the first colon that is not inside a placeholder. Each part is either a saved credential’s whole placeholder or your own words: {{credential:username}}:{{credential:password}}, alice:{{credential:password}}, or alice:s3cret;
• or one saved credential’s placeholder, {{credential:<handle>}}, as the whole text, whose saved value is username:password.

A username has no colon; a password may. A placeholder is resolved only when the step is sent, and only when the host the sheet names is one of the websites the credential is saved for, as a typed credential is. A placeholder inside other text, one this session does not hold, an empty username or password, and any other text fail the step without sending anything.

When the session reports that the address asking for the sign-in is plain http ("scheme": "http" on the dialog), a sign-in with any saved part is not sent ("not signed in: this site asks for a sign-in over an unencrypted connection, so a saved password is not sent, …"); one written out in full in your own words still is. Without a scheme, the address is taken to be https. The username and password go to the browser and nowhere else: the step shows {{prompt text not stored}} in every copy, as a prompt’s text does, and its summary is answered the sign-in sheet: Sign In. Signing in this way needs a session whose browser can sign in on the sheet; on any other the accept fails without being sent ("not signed in: this session’s browser cannot sign in on the site’s sign-in sheet, …") — sign in in the live view. An address written with user:pass@ is opened without it, as Safari opens it, and the site’s sheet is answered as above.

Only on a session whose browser can answer dialogs. On any other the step fails without being sent ("not answered: this session’s browser cannot answer a dialog, so nothing was sent — answer it in the live view"); answer the dialog in the live view instead. A step whose text holds a saved credential’s or a session secret’s placeholder fails the same way: a placeholder is typed only by a type step.

A tap or typing that meets a dialog is never repeated. When a tap, typing or a key press fails with dialog_open, the input may already have reached the page (the dialog can be what it opened), so retryable is false and it is never sent again on its own; a read, a wait or a navigation never ran and is retryable: true once the dialog is answered. Answer the dialog, look at the page, then decide the next step. A tap that succeeded and opened a dialog says so with warning.kind dialog_opened (see Message): answer the dialog the same way, and do not tap again.

List the frames of a page

GET /v1/agent-sessions/{id}/frames

The embedded frames of the page the session’s browser is on, as the browser numbers them now — the paths a step’s frame names in POST /steps. The session asks the browser for them when you call it: a read, nothing anyone could see on the phone, and nothing on the page changes.

{
  "frames": [
    {
      "path": [0],
      "address": "shop.example.com/promo",
      "url": "https://shop.example.com/promo?slot=top",
      "url_masked": false,
      "url_truncated": false,
      "name": null,
      "displayed": true,
      "zero_size": false
    },
    {
      "path": [1],
      "address": "pay.example.net/embed/card",
      "url": "https://pay.example.net/embed/card?client_secret=REDACTED&sid=checkout-7",
      "url_masked": true,
      "url_truncated": false,
      "name": "card-number",
      "displayed": true,
      "zero_size": false
    },
    {
      "path": [1, 0],
      "address": null,
      "url": null,
      "url_masked": false,
      "url_truncated": false,
      "name": null,
      "displayed": false,
      "zero_size": true
    }
  ],
  "truncated": false
}

• path — the frame’s position, outermost first: what a step’s frame names. Positions follow the order the browser created the frames, which is not always the order of the <iframe> tags in the markup.
• address — host and path of the document the frame holds. Never its query or fragment; a long path segment is written …. null for a frame with no web document (an empty or inline one) or one whose address the browser reports as the page’s own, so cannot be shown as the frame’s. That includes a frame on the page’s own host and path whose query differs: address is null for it, and url is shown (below). Frames that share one address (a payment form’s hosted fields, one provider path each) keep it when the page’s own address is known and is not that address.
• url — the same document’s full address, scheme://host[:port]/path?query, so frames that share an address can be told apart. Never the fragment (the part after #) or a user:password@ part. The value of every credential-style query parameter (client_secret, token, password, access_token, api_key and the like) is replaced with REDACTED; the parameter names, their order and every other value are kept as the browser reports them. null where address is null, with one exception: a frame on the same host and path as the top page whose url differs from the page’s (another query) has address: null and its url shown, since a different query is a different document. A frame whose url is the page’s own, after masking, shows neither. At most 1000 characters.
• url_masked — true when at least one value in url reads REDACTED.
• url_truncated — true when url was longer than 1000 characters and was cut; it then ends ….
• name — the frame’s name (its window.name, which the frame’s own page sets), on one line and cut to 60 characters (a longer one ends …), or null.
• displayed, zero_size — whether it is shown, and whether it has no size.
• truncated — the browser stopped listing before the page’s last frame. A list that holds at least one frame is a real, partial list; a browser that stopped before it could list any is no list at all — that is the 409 below, never "frames": [].

Telling hosted fields apart. A payment form often embeds one frame per field from the same provider path, and only the query says which is which. Their address is the same; their url is not:

{
  "frames": [
    {
      "path": [0],
      "address": "pay.example/elements-inner-card",
      "url": "https://pay.example/elements-inner-card?componentName=cardNumber",
      "url_masked": false,
      "url_truncated": false,
      "name": null,
      "displayed": true,
      "zero_size": false
    },
    {
      "path": [1],
      "address": "pay.example/elements-inner-card",
      "url": "https://pay.example/elements-inner-card?componentName=cardExpiry",
      "url_masked": false,
      "url_truncated": false,
      "name": null,
      "displayed": true,
      "zero_size": false
    }
  ],
  "truncated": false
}

Take a piece of the query that only one frame has and name the frame by it in a step’s frame (naming a frame by its address). {"src": "…"} matches against the frame’s full address, query included, so this keeps working when the page adds or removes other frames:

{
  "steps": [
    {
      "kind": "wait",
      "condition": "selector_visible",
      "selector": "input[name=\"exp-date\"]",
      "frame": [{ "src": "componentName=cardExpiry" }]
    },
    {
      "kind": "interact",
      "action": "type",
      "selector": "input[name=\"exp-date\"]",
      "value": "12 / 29",
      "frame": [{ "src": "componentName=cardExpiry" }]
    }
  ]
}

Choose text the browser sees as written: not a REDACTED value (the browser holds the real one, so it never matches) and nothing after a #. Driftstack returns url to you and does not store it.

Frames on the page’s own path. A page can embed frames from its own host and path that differ only in the query, for example https://shop.example.com/checkout?step=card inside https://shop.example.com/checkout. Their address is null, because it would read the same as the page’s, but their url is shown:

{
  "path": [0],
  "address": null,
  "url": "https://shop.example.com/checkout?step=card",
  "url_masked": false,
  "url_truncated": false,
  "name": null,
  "displayed": true,
  "zero_size": false
}

Copy a {src} frame level from url for such a frame, as above — for this one, {"src": "step=card"}. There is no address to copy from.

Right after a page load the list may be unavailable for a few seconds: a frame still loading can stall the browser’s walk of the page before it has listed anything. The session already waits about a second and a half and asks once more on a session that went active in the last 15 seconds (or that you asked within the last 5), so the first call after activation usually gets the real list; when it still cannot, the 409 below says retryable: true — call again shortly, and read a 409 as “not yet”, never as a page with no frames.

The numbering is the page’s at that moment: a page that loads or removes a frame renumbers the frames after it, so list again after a step that changes the page. POST /steps takes its own list and checks every path against it, so a stale path is a failed row, never a step sent to another frame.

A key with read:sessions, or the session’s control key; another account’s session is a 404. It counts against the agent_sessions:live rate limit. A 409 says why there is no list: reason: "session_not_running" (the session has ended), "frames_unsupported" (its browser cannot list frames; said only once the browser can be asked) or "frames_not_listed" (the list could not be had just now — including the walk that stalled before it listed any frame, above, and a browser that cannot be reached or has not yet said what it can do after reconnecting; retryable: true) — or code: "session_not_ready" while the session’s browser is starting.

The SDKs: agentSessions.listFrames(id) in TypeScript, agent_sessions.list_frames(id) in Python (sync and async) and AgentSessions.ListFrames(ctx, id) in Go.