JoshuaALawrence/OpenControl

tools

• click

  Click at a point: left/right/middle, single/double/triple.

• click_element

  Click a control by its [index] (moves the real mouse, or invokes it if off-screen).

• close_window

  Request a window to close (the app may prompt to save).

• copy_selection

  Copy the current selection and return the resulting clipboard text.

• double_click

  Double-click at a point.

• drag

  Press, drag, and release for sliders, selections, and drag-and-drop.

• find_elements

  Search a window's tree for controls by name and/or control type; returns usable indices.

• find_image_on_screen

  Find a template image via normalized cross-correlation; returns matches with centers.

• find_text

  Locate on-screen text and return matching words with clickable centers.

• focus_window

  Bring a window to the foreground and give it focus.

• get_active_window

  Return the foreground window's handle, title, and app.

• get_blocklist

  Show the user-configured application blocklist (read-only). Reports which apps are redacted from captures/OCR and refused for control. The agent cannot change it.

• get_clipboard

  Read clipboard text.

• get_cursor_position

  Return the mouse position in physical screen pixels.

• get_element_at_point

  Identify the UI element under an image-space point.

• get_focused_element

  Return the control that currently holds keyboard focus.

• get_pixel_color

  Read the RGB/hex color of a pixel from the last screenshot.

• get_system_info

  Report host OS, CPU, memory, monitors, and virtual-screen size.

• get_ui_tree

  Dump an indexed accessibility tree as text (no screenshot) for a window or the whole desktop.

• hold_key

  Hold a set of keys down for a duration, then release.

• hotkey

  Alias of press_key for chord shortcuts.

• invoke_element

  Invoke a control without the mouse: invoke, toggle, expand, collapse, select, or scroll.

• key_combo_sequence

  Press a sequence of chords in order (e.g. ["alt+f", "s"]).

• kill_process

  Terminate a process by pid or name. *(Destructive; use deliberately.)*

• launch_app

  Launch an app by executable path or shell identifier (e.g. notepad, ms-settings:).

• list_apps

  List installed apps (from Start Menu shortcuts) with running state and open windows.

• list_directory

  List the entries of a directory.

• list_key_names

  List every accepted keysym-style key name.

• list_processes

  List running processes (pid + name), optionally filtered.

• list_windows

  List targetable top-level windows: handle, title, and owning app.

• mouse_button

  Fine-grained button down/up for custom press-hold-release sequences.

• move_mouse

  Move the cursor without clicking.

• move_resize_window

  Move and resize a window to an absolute rectangle.

• observe

  Primary tool. Capture a window plus its indexed accessibility tree in one call (Set-of-Marks + actions per control).

• ocr

  Read on-screen text with built-in Windows OCR; returns text plus per-word boxes in screen pixels.

• paste

  Optionally set text, then paste with Ctrl+V.

• press_key

  Press a +-separated chord using keysym names (e.g. Control_L+a, Alt_L+F4).

• read_file

  Read a UTF-8 text file (bounded).

• right_click

  Right-click to open a context menu.

• run_command

  Run an executable with arguments and return its output.

• run_powershell

  Run a PowerShell script and return stdout/stderr/exit code (bounded).

• save_screenshot

  Capture the screen (or a region) and save it to a .png on disk.

• screen_info

  List the monitor layout and virtual-desktop bounds.

• scroll

  Scroll the wheel over a point, vertically or horizontally.

• set_clipboard

  Set clipboard text.

• set_element_value

  Set an editable control's value via the ValuePattern. Faster and more reliable than typing.

• set_window_topmost

  Pin a window above all others, or unpin it.

• take_screenshot

  Capture a monitor, region, or the whole virtual desktop, with optional grid and cursor overlays.

• type_text

  Type Unicode text into the focused control, with an optional per-key delay.

• wait

  Pause for a number of seconds to let the UI settle.

• wait_for_screen_change

  Block until the screen changes or a timeout elapses. Ideal after an action whose duration is unknown.

• window_state

  Minimize, maximize, or restore a window.

• write_file

  Write a UTF-8 text file.

• zoom

  Magnify a region and return it at native resolution so small text and icons are legible.