Overcoming Challenges in Real-World Computer Vision Projects

Computer vision projects rarely go exactly as planned, and this one was no exception. The idea was simple: Build a model that could look at a photo of a laptop and identify any physical damage, such as cracked screens, missing keys, or broken hinges. It seemed like a straightforward use case for image models and large language models (LLMs), but it quickly turned into something more complicated.

Along the way, we ran into issues with hallucinations, unreliable outputs, and images that were not even laptops. To solve these, we applied an agentic framework in an atypical way — not for task automation, but to improve the model’s performance.

Where We Started: Monolithic Prompting

Our initial approach was fairly standard for a multimodal model. We used a single, large prompt to pass an image into an image-capable LLM and asked it to identify visible damage. This monolithic prompting strategy is simple to implement and works decently for clean, well-defined tasks. But real-world data rarely plays along.

We ran into three major issues early on:

Hallucinations**: The model would sometimes invent damage that did not exist or mislabel what it was seeing.
Junk image detection**: It had no reliable way to flag images that were not even laptops, like pictures of desks, walls, or people, which occasionally slipped through and received nonsensical damage reports.
Inconsistent accuracy**: The combination of these problems made the model too unreliable for operational use.

First Fix: Mixing Image Resolutions

One thing we noticed was how much image quality affected the model’s output. Users uploaded all kinds of images ranging from sharp and high-resolution to blurry. This led us to refer to research highlighting how image resolution impacts deep learning models.

We trained and tested the model using a mix of high- and low-resolution images. The idea was to make the model more resilient to the wide range of image qualities it would encounter in practice. This helped improve consistency, but the core issues of hallucination and junk image handling persisted.

The Multimodal Detour: Text-Only LLM Goes Multimodal

Encouraged by recent experiments in combining image captioning with text-only LLMs, we decided to give it a try.

Here’s how it works:

The LLM begins by generating multiple possible captions for an image.
Another model, called a multimodal embedding model, checks how well each caption fits the image. In this case, we used SigLIP to score the similarity between the image and the text.
The system keeps the top few captions based on these scores.
The LLM uses those top captions to write new ones, trying to get closer to what the image actually shows.
It repeats this process until the captions stop improving, or it hits a set limit.

While clever in theory, this approach introduced new problems for our use case:

Persistent hallucinations**: The captions themselves sometimes included imaginary damage, which the LLM then confidently reported.
Incomplete coverage**: Even with multiple captions, some issues were missed entirely.
Increased complexity, little benefit**: The added steps made the system more complicated without reliably outperforming the previous setup.

A Creative Use of Agentic Frameworks

This was the turning point. While agentic frameworks are usually used for orchestrating task flows, we wondered if breaking down the image interpretation task into smaller, specialized agents might help.

We built an agentic framework structured like this:

Orchestrator agent**: It checked the image and identified which laptop components were visible (screen, keyboard, chassis, ports).
Component agents**: Dedicated agents inspected each component for specific damage types; for example, one for cracked screens, another for missing keys.
Junk detection agent**: A separate agent flagged whether the image was even a laptop in the first place.

This modular, task-driven approach produced much more precise and explainable results. Hallucinations dropped dramatically, junk images were reliably flagged, and each agent’s task was simple and focused enough to control quality well.

The Blind Spots: Trade-offs of an Agentic Approach

As effective as this was, it was not perfect. Two main limitations showed up:

Increased latency**: Running multiple sequential agents added to the total inference time.
Coverage gaps**: Agents could only detect issues they were explicitly programmed to look for. If an image showed something unexpected that no agent was tasked with identifying, it would go unnoticed.

We needed a way to balance precision with coverage.

The Hybrid Solution: Combining Agentic and Monolithic Approaches

To bridge the gaps, we created a hybrid system:

The **agentic framework** ran first, handling precise detection of known damage types and junk images. We limited the number of agents to the most essential ones to improve latency.
Then, a **monolithic image LLM prompt** scanned the image for anything else the agents might have missed.
Finally, we **fine-tuned the model** using a curated set of images for high-priority use cases, like frequently reported damage scenarios, to further improve accuracy and reliability.

This combination gave us the precision and explainability of the agentic setup, the broad coverage of monolithic prompting, and the confidence boost of targeted fine-tuning.

What We Learned

A few things became clear by the time we wrapped up this project:

Agentic frameworks are more versatile than they get credit for**: While they are usually associated with workflow management, we found they could meaningfully boost model performance when applied in a structured, modular way.
Blending different approaches beats relying on just one**: The combination of precise, agent-based detection alongside the broad coverage of LLMs, plus a bit of fine-tuning where it mattered most, gave us far more reliable outcomes than any single method on its own.
Visual models are prone to hallucinations**: Even the more advanced setups can jump to conclusions or see things that are not there. It takes a thoughtful system design to keep those mistakes in check.
Image quality variety makes a difference**: Training and testing with both clear, high-resolution images and everyday, lower-quality ones helped the model stay resilient when faced with unpredictable, real-world photos.
You need a way to catch junk images**: A dedicated check for junk or unrelated pictures was one of the simplest changes we made, and it had an outsized impact on overall system reliability.

What started as a simple idea, using an LLM prompt to detect physical damage in laptop images, quickly turned into a much deeper experiment in combining different AI techniques to tackle unpredictable, real-world problems. Along the way, we realized that some of the most useful tools were ones not originally designed for this type of work. Agentic frameworks, often seen as workflow utilities, proved surprisingly effective when repurposed for tasks like structured damage detection and image filtering. With a bit of creativity, they helped us build a system that was not just more accurate, but easier to understand and manage in practice.