ChatGPT/Gemini can now draw on your screen to help you navigate complex software
A new framework called SketchVLM allows vision-language models to create editable SVG overlays on images to enhance user understanding. This approach improves visual reasoning task accuracy significantly and offers better sketch quality compared to traditional methods. The framework demonstrates strong performance in both single-turn and multi-turn generation, facilitating improved human-AI collaboration.
- ▪SketchVLM enables VLMs to produce non-destructive, editable SVG overlays on images.
- ▪The framework improves visual reasoning task accuracy by up to +28.5 points.
- ▪SketchVLM enhances sketch quality by up to +48.3% over existing baselines.
Opening excerpt (first ~120 words) tap to expand
When answering questions about images, humans naturally point, label, and draw to explain their reasoning. In contrast, modern vision–language models (VLMs) such as Gemini-3-Pro and GPT-5 typically respond with only text, which can be difficult for users to verify. We present SketchVLM, a training-free, model-agnostic framework that enables VLMs to produce non-destructive, editable SVG overlays on the input image to visually explain their answers. Across six benchmarks spanning visual reasoning (maze navigation, ball-drop trajectory prediction, and object counting) and drawing (part labeling, connecting-the-dots, and drawing shapes around objects), SketchVLM improves visual reasoning task accuracy by up to +28.5 points and sketch quality by up to +48.3% over image-editing and fine-tuned…
Excerpt limited to ~120 words for fair-use compliance. The full article is at Github.
Discussion
0 commentsMore from Github
