Drıftstack docs
Docs

Agent sessions: what a step can do

Part of the Agent sessions reference, which lists every call.

An agent session lets an AI drive a browser session from plain-language messages. The customer sends a message ("open https://example.com and capture a screenshot"); Driftstack turns it into a list of typed intents (navigate, interact, wait, capture, the behavioural scroll and behavioral_pause, the three moves for a page that resists a plain plan: back, extract and tap_at, and dialog, which answers a JavaScript dialog the page has open — see Dialogs), runs them, and returns the results in the response.

back goes one step back in the browser’s history. extract reads the rendered text of one element (selector) or of the whole page (body: true); the text comes back, bounded, in the step’s summary. With a selector, attribute reads that attribute of the element instead of its text — a link’s href, an image’s src, a data-… or aria-… value — and the summary says which it was: extracted attribute href from #terms: https://example.com/terms. An element without the attribute is said to have none (extracted nothing: the element #terms has no href attribute), which is not the same answer as a selector that matched nothing. attribute is a name of letters, digits, -, _, : and . (at most 128 characters), and it is never used with body; inside an embedded frame it is read only on a session whose browser can read an element there (see below).

What a field holds now is a property, not an attribute. An attribute is what the page’s markup said: attribute: "value" on a field someone typed into reads the markup’s starting value, usually nothing. property reads what the element holds now — { "kind": "extract", "selector": "#email", "property": "value" } — and is one of value, checked, selected, selectedIndex, outerHTML, innerHTML, textContent, name, id, type, autocomplete, placeholder, disabled, readOnly and required, with a selector only, never with attribute or body (a list of steps naming any other property, or one beside attribute or body, is a 400 naming the field). The result carries what it read in value: a string; a boolean for checked, selected, disabled, readOnly and required; a number for selectedIndex (-1 when nothing is chosen); and null when the element the selector matched has no such property — the summary says so in words, so null is never confused with an empty string. textContent of the whole page (body, html or :root) is every text node in it, including text the page never shows — a noscript fallback, script and style text — as in any browser; in a list of steps you send, its summary says so when the selector is exactly body, html, :root, html > body, html>body or html body (any letter case), and an extract with no property (the rendered text, like innerText) reads what the page shows. value_truncated is true when a string was cut to its first 100,000 characters (UTF-16 code units). The value is returned in this response and never stored: the summary never quotes it, the transcript and the Idempotency-Key receipt keep the step without it (a replay answers the step with no value), and no log line carries it. A saved credential is never read back: the field holds the real value, but what Driftstack typed from your vault or a session secret reads back, in any read, as its placeholder ({{credential:cred_…}} or {{credential:sec_…}}), and the result then carries shows_placeholder: true. Check that flag before you compare a read-back with what you expected: a field that keeps only digits does not read back the stored number, and the digits in the placeholder are the handle’s. A value you typed yourself reads back as typed — unless it is the value of a saved credential on your account, which is typed as that credential and reads back as its placeholder too. shows_placeholder is absent on every other result, including a placeholder the page wrote itself. Once the session can no longer recognise a saved credential it typed (you deleted it since), no text any extract reads is shown — the summary says it was not read back and value is null. A result that could not be checked in time is withheld. Once a saved credential or session secret has been typed in a session, every step result is checked for it before it is returned. In rare cases that check does not finish in time, and the step’s text is withheld: the result carries withheld: true, its summary is (not shown: this text could not be checked in time), and a property read’s value is null (with value_truncated: false and no shows_placeholder); a text, attribute or whole-page read has no value, only that summary. That null says nothing about the page: check withheld before reading value. A read tried again usually returns its text; a step that acted on the page did run, so do not repeat it for this. withheld is not set on a failed step whose reason is that same note, nor on a read not shown because its saved credential was deleted (above). A property is read only on a session whose browser can read one; on any other the step fails without being sent ("not run: this browser cannot read a field’s live value yet, so nothing was read") — it is never read as an attribute instead. While the session’s browser cannot be reached (right after an API restart, for example) such a read outside an embedded frame is not answered that way: it fails as a step that could not be sent, retryable (see When a step could not be sent to the session’s browser).

A read by selector that matched no element fails. An extract by selector — its text, an attribute or a property, on the page or inside a frame — whose selector matched nothing is a failed step with diagnosis.category element_not_found and a reason naming the selector and, inside a frame, the frame (no element matched `input[name=cvc]` in embedded frame [0], so nothing was read). It is never a success with nothing in it.

Every match of a selector: all: true. In a list of steps you send, an extract with all: true reads every element its selector matches, in page order, instead of the first — each element’s rendered text, or with attribute that attribute of each (a list of links’ href):

{ "kind": "extract", "selector": "li.result h3", "all": true }

The result carries them in values, one entry per element: a string, or null for an element without the attribute. At most 200 come back; when the selector matched more, values holds the first 200 and truncated is true, and so it is when an entry was cut to its first 100,000 characters (UTF-16 code units), or when the entries together reached 100,000 characters (the entry that reached it is cut there and the ones after it are left out); the summary says which (extracted text from 312 elements matching li.result h3 (the first 200), returned in this step’s values, never stored). The browser reads at most the first 1,000 matches, so a count of 1,000 is said as “1,000 or more”. The values are returned in this response and never stored, as a property read’s value is: this response’s summary says only how many were read, and the transcript, the Idempotency-Key receipt (a replay answers the step with no values, and its summary names the step, not the count), the partial_results of an error and every log line carry the step without them. A saved credential reads back as its placeholder, with shows_placeholder: true; once the session can no longer recognise a saved credential it typed, or the read could not be checked for one in time, every entry is null and the summary says why. "all": true goes with a selector only (and optionally attribute), never with body or property, and all set to anything but true, false or null is refused: an extract that breaks one of these is a 400 naming the field (steps.1.property). "all": false or null is the same as no all: it reads one match. Only an extract reads all; on any other step it is ignored, as every field a step’s kind does not take is (see Run a list of steps). A selector that matches nothing fails like any read by selector. Inside an embedded frame it is read on a session whose browser can read an element there. The AI does not use all in a message’s turn: it reads a page through its own look at it.

One match by its position: index. An extract with index: n (a whole number from 0 to 999) reads the element at that position among every element its selector matches, counting from 0 in page order — index: 0 is the first, as with no index at all — and answers as a read of one element does: its text (or its attribute) in the summary, which names the position (extracted text from match 2 of li.result h3: Gamma). A position the selector does not reach fails the step with element_not_found and a reason saying how many matched (no match at index 4: `li.result h3` matched 4 elements (index counts from 0), so nothing was read). index counts every match of the selector in the whole page, whatever its parent; :nth-of-type() in a selector counts only among an element’s siblings of the same tag (see Choosing the n-th element). index goes with a selector only (and optionally attribute), never with "all": true, body or property: an extract that breaks one of these is a 400 naming the field (steps.1.index); "all": false beside it is the same as no all. Only an extract reads index; on any other step it is a 400 naming the field (steps.1.index), and nothing is sent.

How many matched: count. A successful extract by selector with "all": true, index, attribute or property also carries count: how many elements the selector matched, as a number ("count": 312), in the page or in the embedded frame the step named. The browser counts at most the first 1,000 matches, so 1000 means 1,000 or more. A plain text read of one element (no all, index, attribute or property) and a body: true read carry no count: the browser’s answer to those does not say how many matched. A read that matched nothing fails with element_not_found, so a count is never zero. Like values, it is returned in this response and never stored: the transcript and a replay of an Idempotency-Key carry the step without it. Whether an element is visible is not returned as a field yet; a short selector_visible wait answers it.

tap_at taps a point on the screen (x, y in viewport pixels from the top-left corner): the AI may only take the point off the most recent screenshot it was shown, a point outside that screenshot is refused before it reaches the session, a session that cannot report what is at the point refuses it ("not supported on this device"), and the step’s summary names what was under the tap. The screenshot the AI looks at to choose the point is taken for that one planning step and is not stored — it is not a capture and has no captureId.

In a list of steps you send, a tap_at needs no screenshot: you chose the point, so it is sent as written, even straight after a navigate — which is what lets an AI run’s tap_at steps be replayed. The session still refuses a point off the screen or on nothing (the step fails and nothing is tapped), and when a screenshot of the page was taken in the session since it last changed, a point outside that screenshot is refused before it is sent, as for the AI. A step you send that would buy, pay or delete an account is checked by the account’s purchase_check_mode: always stops it for your approval, never does not (the per-task limit in When it stops still applies), and smart always stops it, because a step you send carries no words of yours that say you asked for it.

Read results first. Every step the turn ran, and the one it stopped at, comes back in its results, with the step itself in results[i].intent (the selector it acted on, the text it typed) and what happened: a summary for a step that succeeded, a reason for one that failed. That is the record of what was done on the page; answer is a sentence for a person. See Step results are the record.

Your message goes to the model word for word. It is also kept in the session’s history. Never write a password, a one-time code or a card number in it. Register a password or a one-time code as a session secret or save it as a saved credential; a card number can only be a session secret. Then write its placeholder ({{credential:cred_…}} or {{credential:sec_…}}) instead. The model plans with the handle, not the value.

An interact step with tap names its target by a CSS selector, and the session looks that selector up when the step runs, so the tap lands on whatever the selector names at that moment rather than on a point that may have moved. The AI is told to tap a control by its selector whenever the page offers one, and to use tap_at only for something it does not. A control with no id or name of its own is given a selector by where it sits in the page (#slots > li:nth-of-type(3)). When the session can say which element the selector resolved to, the step’s summary names it after the selector — its type and its label as the page shows it, on one line and cut to 60 characters:

{
  "kind": "success",
  "intent": { "kind": "interact", "action": "tap", "selector": "#slots > li:nth-of-type(3)" },
  "summary": "tapped #slots > li:nth-of-type(3): the button “11:00”"
}

A tap step with double: true performs a double tap, as on an iPhone: it selects a word in a text field. Only on a text field: an editable input of type text, search, email, url, tel, password or number (or no type), a textarea, or an editable area, that is not inside a link or a button. On any other element (a date, time, file, colour or range input, a drop-down or list, a field inside a link), on a field the session cannot confirm is a text field, on a step that would need approval (a control that may buy, pay or delete an account, even one an earlier approval covers), or on a session whose browser cannot double-tap yet, the step is a failed row whose reason says why, and nothing is sent. double is true or left out; false, or double on any other step, is a 400 naming steps.<i>.double. Its summary is double-tapped <selector>, with an effect_unknown warning: the browser reports that it double-tapped, not what was selected. The AI does not plan double taps yet.

Only a control is named by its label: a button, link, field, menu, option, checkbox or image. Anything else, such as a paragraph or the text inside an editor, is named only as a control, without its words. A tap_at step’s summary names what was under the tap the same way.

Choosing the n-th element

Two ways to name “the third result”, and they count differently:

  • index on an extract counts every element the selector matches in the whole page, in page order, from 0: { "kind": "extract", "selector": "li.result h3", "index": 2 } reads the third h3 that li.result h3 matches, wherever each sits.
  • :nth-of-type(n) in a selector counts from 1, and only among an element’s siblings — the children of the same parent — that have the same tag. li:nth-of-type(3) is every li that is the third li child of its parent, so on a page with two lists it matches one element in each, and a div or h2 between the lis is not counted. A class does not change what is counted: li.result:nth-of-type(3) matches an li that is the third li among its siblings and also has the class result — not the third .result. :nth-child(n) counts every sibling, whatever its tag.

To read the third result of a list, prefer index on the selector that names every result; to tap it, name its position inside its own parent (#results > li:nth-of-type(3) a), or read the list first with all: true and choose a selector from what it holds. A selector the session gave a control by where it sits (#slots > li:nth-of-type(3)) uses the sibling count.

Driving by selector

Ask for a selector, then act on it. Two calls:

// POST /v1/agent-sessions/{id}/message
{ "user_message": "What is the selector of the Pay button?" }
// → "answer": "The Pay button's selector is #pay." (for example)

// POST /v1/agent-sessions/{id}/steps
{ "steps": [{ "kind": "interact", "action": "tap", "selector": "#pay" }] }

The second call taps #pay with no AI planning (see Run a list of steps). A message works too: { "user_message": "tap #pay" }. Either way, when the session can say what the selector resolved to, the step’s summary names it (tapped #pay: the button “Pay”), and a tap recognised as a payment stops for your approval first, unless the account’s purchase_check_mode lets it through (Confirming a purchase, payment or account deletion).

You can name the element yourself. A CSS selector in your message is used exactly as you wrote it, even when the page list the AI is shown names the element another way or does not list it. That holds on the page itself: inside an embedded frame, a tap or typing is sent only for a selector the list showed under that frame, and a selector the list showed only inside a frame is never used on the page itself; either is refused without being sent. One exception: an id or a class that is no CSS name (such as one that starts with a digit, as in #1abc or .1x, or carries a dotted number, as in #section-1.2) is sent as the attribute match a browser reads, #1abc as [id="1abc"] and .1x as [class~="1x"], and a tap or a type that then finds nothing says so in its reason ((read as [id="1abc"])). It is still checked: an invalid selector fails without being sent, and every refusal still applies — a native drop-down list is never tapped (on a session whose browser has the iPhone’s list picker it is chosen from by a type step naming the option; otherwise it cannot be chosen from), a control the page does not display cannot be tapped, a saved credential is typed only on its own site, and a step that pays, buys or deletes an account waits for your approval, unless the account’s purchase_check_mode lets it through.

{ "user_message": "tap #accept-terms, then read the href of a.terms" }

The turn’s intents carry your selectors as they were sent: unchanged, but for an id or a class read as an attribute match (above):

[
  { "kind": "interact", "action": "tap", "selector": "#accept-terms" },
  { "kind": "extract", "selector": "a.terms", "attribute": "href" }
]

To find a selector, ask for it: “what is the selector of the Pay button?” or “list every button on this page with its selector”. The answer is read from the rows the AI was shown for the page that is open (#pay · button · "Pay"), so it names a selector the session can act on, quoted exactly. Those rows are given to the answer when your message asks about selectors or controls (it says selector, CSS, button, link, field or element, or names a selector such as #pay); otherwise only when they fit beside all of the page’s text, so they never cut short the text another question is answered from. An answer keeps its line breaks, so a list asked for one control per line comes back one per line (the transcript entry holds the same text on one line). It is at most 4,000 characters; a longer one is cut after its last whole line and says so on a line of its own. Limits: a control inside a shadow root is not in the page source the rows are usually read from, so it may have no row and no selector to report (name it yourself with a >>> path, which the session follows into open shadow roots — a wait for selector_visible follows one too, and waits until the element it ends on is shown); a control inside an embedded frame is listed under its frame instead; and a native drop-down list (select) has a row, and is chosen from only on a session whose browser has the iPhone’s list picker.

Choosing from a native drop-down list. On a session whose browser has the iPhone’s list picker, a type step whose selector names a <select> on the page itself, with the option’s text (or its value) as value, chooses that option the way a person does: a tap on the list, a look at the options, a tap on the option’s row. The row says chose an option in <selector>.

{ "kind": "interact", "action": "type", "selector": "#country", "value": "Canada" }

A tap on the list or one of its options, a key pressed on it, a list that takes several choices (multiple) and a list inside an embedded frame are failed rows whose reason says why, and nothing is sent. So is any choice from a list on a session whose browser has no list picker. A row also fails, saying so, when the list has no option by that name, when its options did not open, when the list did not take the option, or when the page is drawn zoomed (nothing is tapped); a failed choice that left the select showing another option says so. Any step that taps, types, presses, swipes, scrolls or pauses fails, with nothing of it sent, when an option list an earlier choice left open could not be closed or checked (invalid_request, retryable: false: it is never sent again on its own); send it again, which looks at the list once more, or navigate to the page again. A navigate, back, wait, capture, extract or dialog step still runs.

How the option is matched. An option marked disabled is never chosen; the others are compared with the step’s value in two passes. First with each option’s value, exactly: case and spaces count. Only when none has that value, with each option’s text (what is written between <option> and </option>), ignoring case and any spaces at the start and end. The whole text must match; part of it never does. For <option value="CA">Canada (CA)</option>, send "CA" or "Canada (CA)", not "Canada". When several options match in the same pass, the first in the list is chosen. When nothing matches, the row fails and nothing is chosen. When the option that matches is already the one selected, nothing is tapped and the step succeeds.

Dropdowns built from a button and a list. Many sites (react-aria, Headless UI and similar) draw their own dropdown: a button that opens a list of rows, with the real <select> kept hidden in the page for autofill (tiny, clipped, inside an aria-hidden box). A person never touches that hidden list, so a step should not either: tap the visible button, then tap the option’s row in the list that opens.

[
  { "kind": "interact", "action": "tap", "selector": "button[aria-haspopup=\"listbox\"]" },
  {
    "kind": "interact",
    "action": "tap",
    "selector": "[role=\"listbox\"] [role=\"option\"][data-key=\"NJ\"]"
  }
]

Name the button by something of its own when the page has more than one (#state-button, or the field’s [aria-label="…"]); the button is the element with aria-haspopup="listbox" or role="combobox", and the rows have role="option" (often with a data-key, or an id ending in the option’s key). The list may open at the end of the page rather than next to the button. A type step on such a hidden <select> may fail, nothing chosen, with a reason that says to tap the button and then the option (when the browser can tell the list is hidden); a step with action: "select", which is not a step this session takes, says how a list is chosen from.

A page can embed other documents in frames (an <iframe>: a payment form, a verification check, a chat window). Their contents are not part of the page itself, so a capture with dom_snapshot, or an extract with body: true, may carry a frame — the embedded document it read instead of the page, as a path of positions: its place among the page’s frames, then its place inside that frame for a nested one, outermost first ([0], [0, 1]). Positions are the browser’s own and follow the order it created the frames, which is not always the order of the <iframe> tags in the page’s markup: a frame a script inserted or moved is numbered when it was created, so [1] is not necessarily the second iframe tag. Absent, the step read the page itself; frame never appears on a screenshot or a PDF. On a session whose browser can read an element inside a frame, an extract by selector (and by attribute) may carry frame too: it read that element of the frame’s document, not the whole frame; on any other session such a step fails without running. A read with frame succeeds only when the session confirmed it read exactly that frame; a read that failed names the frame it asked for. frame also appears on an interact with type: it names the frame the step asked to type in, and the step’s selector is matched inside that frame, not the page. A type with frame succeeds only when the session confirmed it typed into exactly that frame. When the text was sent but the session did not confirm the frame, the step fails with diagnosis.category unknown, it is not tried again, and the turn ends on it: the text may be somewhere else on the page, so look at the page before typing again. A type without frame fails in the same way when the session reports it typed inside a frame. On a session whose device can act inside a frame, an interact with tap, and a wait with selector_visible, can carry frame too: the tap’s selector is matched, and the element is waited for, inside that frame. A scroll or a press never carries frame: both act on the page itself. Such a step succeeds only when the session confirmed it was done inside exactly that frame. When it did not, the step fails and is not tried again: a tap with diagnosis.category unknown, which ends the turn, and a wait with condition_not_met. A step without frame fails in the same way when the session reports it was done inside a frame. A tap inside a frame is made only when nothing on the page or in the frame covers its point; a covered one fails without tapping. After a tap or a type inside a frame, a key press that could submit a form (Enter) is judged on that frame’s form, and is not sent when that frame cannot be read. A frame drawn transformed (scaled, rotated or skewed) fails “it can’t be tapped here … — reading or waiting inside it still works”; a control off the screen inside a frame that may cover the screen fails “this frame may cover the screen, so the page could not be scrolled to the control inside it”; a frame the page could not scroll onto the screen fails “this frame is not fully on the screen and the page could not be scrolled to bring it there”, and a control the browser could not otherwise bring onto the screen inside its frame fails saying so. When the browser drags on a frame that covers the screen to bring the control into view and cannot, the step names what stopped it, and only that: a drag that did not bring the control any nearer to the screen; running out of drags or time; no safe point on the screen to start a drag from; a drag that could not be sent; a frame, control or page around it that could not be measured (this one also covers a page load or a new tab the browser could not rule out, so it ends “read the page before the next step, as the frame may have changed”); a control outside the visible area of the frame or a panel in it, or an area that could not be read; a page behind the frame that is locked or has too little room to scroll toward the control; or a frame whose content extends further that way, so a drag on it would move the page behind it. When a page load may have started while the browser places the tap — before any drag or during them — the step fails “a page load may have started while the tap inside this frame was being placed, so the control was not tapped … — read the page before the next step, as the frame may have changed”; that sentence never says a drag was made. A control inside a frame that the browser could not find again after it scrolled the page or dragged on the frame toward it fails “the control inside this frame could not be found again after the browser tried to scroll toward it, so nothing was typed or tapped inside it (the frame or page may have scrolled) — read the page before the next step”, with diagnosis.category element_not_found, and is not tried again. When the browser names no cause, or a cause this service does not know, the step fails “the browser could not act inside this frame here”. A control outside its frame’s visible area to the side that the frame could not be scrolled to fails saying so. Each of these is invalid_request, not retried, and nothing was typed or tapped inside the frame, but none of them says “nothing was done”: a drag on the frame, a scroll inside it or a page scroll the browser made before it stopped may have scrolled the frame or the page, and the sentence says so (“the frame or page may have scrolled”), so look at the page before the next step. A frame that is gone fails “is no longer on the page, so nothing was typed or tapped inside it (the frame or page may have scrolled) — look at the page or list the frames again … for its new path”: the browser can find it gone after a page scroll or a drag it already made. A tap inside a frame that pays, buys or deletes an account asks for your approval exactly as one on the page does, under the same purchase_check_mode. On any other session no other step can name a frame — a wait that names one fails without running, never as a wait on the page itself — so none can name one to tap into, though a tap_at step taps a point on the screen, and that point can be inside a frame; a tap_at never carries frame. The AI reads a frame, or acts inside one, only when the session has listed the page’s frames for it. After a type the session confirmed inside a frame, that frame, and no other, stays listed for the rest of the same plan, so clearing a field inside a frame and typing its new value run together — each later step there only once the browser reports the same frame (its address and name) at that place again; a frame the page replaced or moved meanwhile is not used, and that step fails without running. These are the rules for the AI’s own steps; steps you write yourself are checked against the browser’s own list of frames instead (see Steps inside an embedded frame).

A frame can also sit inside a shadow root (a web component’s own markup). Such a frame has no position among the page’s frames, so a level of its path is a string instead of a number: the selector of its iframe element in the document that holds it, with >>> stepping into a shadow root — ["checkout-form >>> iframe"], or [0, "x-pay >>> iframe"] for one inside the first frame. A string level is 1 to 1,024 characters, and a path is still at most 8 levels. On a session whose browser can enter such a frame, the AI is shown these frames when it looks at the page, and your own steps on POST /steps may name one by that selector path. GET /v1/agent-sessions/{id}/frames lists numbered frames only for now. A string level the browser’s own list of the page’s frames does not hold, or any string level on a session whose browser cannot enter such a frame, is not run (its row says why) and nothing is sent. A frame that is gone by the time the step runs fails with nothing typed or tapped inside it, saying it is no longer on the page (a scroll the browser made before it found the frame gone may have scrolled the frame or the page).

A session your program runs is in ai mode unless the create sends another mode (ai is the default): every message is planned and executed by the AI. Closed sessions return 409. The desktop app also runs manual and pair sessions, where a person drives the browser; the controls for those are not part of this API.

Want to tap it yourself? An ai session refuses taps and typing from the desktop app (409, “input-event requires mode=‘manual’ or ‘pair’”): you can watch it, but tapping the screen does nothing until someone switches it to Manual in the session window. A script that starts a session for a person to drive themselves creates it with "mode": "manual" (see Create).

Running a task from code? Run AI tasks from your code walks through the whole flow — create a session, send the task, read the answer, approve or answer when the agent asks, stop, close — with complete TypeScript, Python, Go and curl programs.

Scope note. The agent-session writes that start or drive a session — create, reopen, message, steps, stop, close, resume, cookie import, history, file upload and the live-video token — gate on the broad write scope; there is no agent-sessions-specific granular scope. One documented write takes a granular scope: replacing the profile’s notes needs write:profiles, because it edits the profile and runs nothing. Regular session routes accept the granular write:sessions, but agent sessions do not have a granular equivalent. Reads — list, get, the transcript stream, page state, the network log, captures, cookies and downloads — need read:sessions, which broad read and account_owner also satisfy. If you mint a narrow CI key, include the broad write scope to call the write endpoints. An OAuth token can hold only granular scopes, so OAuth apps can’t run AI sessions today; use an API key for that.