PERCV: Leveraging AI to Aid the Visually Impaired

Details:

Category:AI, Product, Mobile
Client:PERCV
Time:June 2025

I had an uncle growing up who was blind and a nature lover. He would often take long walks with family members, and ask them to describe, in detail, what they saw around him. He could not stop smiling as the details got richer and more nuanced. The joy on his face during these interactions inspired me to ask: What if AI could provide this same experience to visually impaired individuals worldwide, anytime, anywhere? PERCV is my attempt to bring this joy to visually impaired individuals across the globe!

The PERCV project was born from the unique, high-stakes environment of the Bolt Hackathon. Unlike the multi-month lifecycles of long-term products, this challenge was a masterclass in rapid prototyping and extreme optimization under severe time constraints. The objective was not merely to build a functional piece of software using vibe-coding tools, but to demonstrate a profound understanding of efficiency and high-performance engineering within a specific technical domain.

Harnessing computer vision and speech to text (STT)

How we built it

PERCV was built with a focus on empowerment and independence. PERCV's design philosophy prioritizes versatility and ease of use for the visually impaired. Through every stage of the design process, I focused on keeping the app simple and accessible so that it can serve as a seamless value-add to any visually impaired individual's day to day life. As a result, you will see the PERCV has a single input interface which uses either double taps (to start and stop the service) or single taps (to continue the conversation). I built PERCV using cutting edge AI technologies like OpenAI's Whisper for automatic speech recognition (ASR) and transcription, OpenAI's GPT-4 for translation and reverse geocoding, and ElevenLab's cutting edge Turbo v2.5 text to speech (TTS) for speech synthesis and multi-language support. I also made use of Supabase Edge Functions for managing the AI workflow, Netlify for deployments, and IONOS Entri for the domain (https://percv.org).

Challenges and reflections

PERCV was an ambitious project from the start, unifying many different AI technologies and workflows. Structuring prompts so that the conversational flow felt natural and conversational was an ongoing trial-and-error challenge. Tweaking the level of detail captured from the camera was another area which took multiple iterations to get right. Balancing latency, performance, and AI-efficiency required playing with different models (eleven_turbo_v2, eleven_turbo_v3, etc.) and testing them in a variety of situations.

Due to the high cost of API calls to ElevenLabs and OpenAI Whisper, I put the main application behind a beta-code wall. However, one of the judging criteria now requires that the URL provided for judging will automatically populate any beta codes. I was able to accommodate this request, but now judges will completely bypass the landing page experience unless they visit 'percv.org' first and then input a beta code. To mitigate this, I've added screenshots of the landing page to the image gallery for my submission.

Outcomes

PERCV is fully WCAG2.2 compliant, screen-reader friendly, and receives a 100/100 accessibility score in Chrome Lighthouse, further cement its commitment to universal accessibility. Additionally, the conversation AI can switch languages based on the user's input language and / or browser settings (using navigator.languages and navigator.language elements).

The use of both TTS and computer vision makes PERCV a unique entrant into the accessibility space. The conversational nature of the AI was fine-tuned through prompt engineering, leading to evocative descriptions and a natural conversational flow.

Final Thoughts

Hitting publish on my hackathon submission was a rush. It represents one of the few moments where I’ve submitted something and felt satisfied with the process I followed to get there. PERCV is far from perfect, and I can already think of a dozen improvements I’d like to make. But what I submitted truly represents the spirit of the hackathon: going from “I don’t know how this works” to “oh wow, it works!” I don’t get to challenge myself on engineering problems as often as I’d like, so to be able to do so alongside over 130,000 other participants truly felt special. You can read my candid reflections on the hackathon on my Substack (Built to Build): https://builttobuild.substack.com/p/hackathon-reflections