Enhancing Multi-Modal Captioning with Human Expertise
A leading multi-modal metadata search company sought a partner to refine its model for improved natural language processing. With a growing client base in media and entertainment, the Client aimed to enhance caption accuracy for video frames and stills but lacked the resources for large-scale prompt and response evaluations.
Challenges
Ensuring accurate, context-aware captions at scale required more than automation alone. The Client faced key obstacles in improving its model's reliability: fully automated captioning often produced inconsistencies, missing the nuance needed to interpret subjective inputs accurately. This resulted in errors and hallucinations in the model, impacting the quality of generated captions.
Solution
To overcome these challenges, the Client needed a solution that combined automation with human expertise, ensuring accuracy and consistency in captioning at scale.
Improving Accuracy with Expert Guidance
To enhance captioning quality and scalability, the Client partnered with Hugo. Annotators with language, media, and contextual analysis backgrounds were recruited and given tailored training to align with the client’s needs. Together, they developed structured guidelines covering:
The appropriate level of detail for key visual elements
Effective background descriptions
Best practices for identifying brand names and logos
Adapting captions for different end-use cases
Hugo’s experts then established a quality framework with defined scoring, error penalties, and "gold tasks"—exemplary captions that set a benchmark for accuracy. This ensured consistency and accelerated the model's improvement.
Human Oversight for Smarter AI
With Hugo’s human-in-the-loop (HITL) approach, the Client combined automation with expert review, leading to more precise outputs.
“Multi-modal captions require careful consideration of various challenges,” noted the Head of AI Ops. “Factors such as poor lighting, overlay text, reflections, and partially obscured words must be addressed to improve accuracy.”
Synchronizing data from multiple modalities was another challenge. In cases where a sound preceded the corresponding visual, the model alone could not establish context. Hugo’s team ensured elements were correctly linked, refining the captioning process for better accuracy.
Results: Faster Fine-Tuning and More Reliable Captions
Hugo accelerated model refinement and improved caption accuracy by integrating human feedback loops.
Key outcomes:
Fewer Errors and Hallucinations
Example: Adjusting the model when it misinterpreted facial expressions.
More Precise and Concise Captions
Example: Removing unnecessary descriptions of non-essential objects.
Stronger Context and Tone
Example: Refining captions to eliminate vague or uncertain language.
Conclusion
Impressed by the results, the client expanded its partnership with Hugo to support additional AI-driven data annotation projects, reinforcing the value of human expertise in machine learning workflows.
