Screenshot UI detection + img2nl diagnose adapter for VQL.
Docs: ../../docs/window-pipeline.md · ../../docs/window-uri.md · ../../TODO.md
| Command | What it does |
|---|---|
diagnose |
img2nl heuristics — send to LLM or grid-only? |
detect |
Windows, panels, titlebar, buttons with pixel bboxes |
adopt |
Detect UI → write ui.vql.json with ui_elements layer |
Detection is heuristic (color regions + contrast blobs), not ML. For OCR and labeled controls use imgl via uri2vql adopt-imgl.
pip install -e ~/github/wronai/img2nl[analyze]
pip install -e packages/img2vql
# Detect UI elements + NL description
img2vql detect /tmp/screen.png --describe --locale pl
# Detect + VQL program
img2vql adopt /tmp/screen.png --out /tmp/ui.vql.json --locale pl
# With background color grid layer
img2vql adopt /tmp/screen.png --with-grid --grid 12
# LLM routing
img2vql diagnose /tmp/screen.png --locale pl --translate-mode auto
uri2vql detect-window --image /tmp/screen.png --locale pl
uri2vql adopt-ui --image /tmp/screen.png --out ui.vql.json
uri2vql compare-window --image /tmp/screen.png --vql-program app.vql.json
uri2vql refresh-window --vql-program app.vql.json --image /tmp/screen.png
uri2vql diagnose-window --image /tmp/screen.png --vql-program app.vql.json --save
uri2vql query "vql://window/detect?image=/tmp/screen.png&locale=pl"
uri2vql query "vql://window/compare?file=app.vql.json&image=/tmp/screen.png"
from img2vql import detect_ui_elements, adopt_screenshot, describe_ui_layout
from img2vql import compare_with_program, fingerprint_for_image
det = detect_ui_elements("/tmp/screen.png")
print(describe_ui_layout(det, locale="pl"))
adopt_screenshot("/tmp/screen.png", out_program="/tmp/ui.vql.json")
adopt writes objects in layer ui_elements with metadata:
role: window, panel, titlebar, toolbar, button, icon_buttonbbox, bbox_norm, location, confidence, colorCombine with img2svg for vector sidecar or img2nl for full feature analysis.