Be My Eyes recently announced the integration of OpenAI's GPT-4 language model into their app, offering blind and low-vision individuals a "virtual volunteer" that can provide instant visual assistance for a variety of tasks. With the ability to not just chat intelligibly, but to inspect and understand images, this AI-powered tool offers individuals robust tools and capabilities, enhancing their virtual assistant experience in ways we never imagined were possible.

As a company that offers home employment insurance for people that employ personal assistants in the home, we understand the importance of having reliable and effective assistance. The integration of GPT-4's multimodal chat and image-to-text technology with the Be My Eyes app is a significant step towards providing better care for those who need it. With instant visual assistance available at any time, individuals can have peace of mind knowing that they have access to the help they need, when they need it.

How it Works

The video accompanying the description shows how useful GPT-4’s multimodal chat can be in the right circumstances. Be My Eyes user Lucy shows the app helping her with a bunch of things live. The AI describes the look of a dress, identifies a plant, reads a map, translates a label, directs Lucy to a certain machine treadmill at the gym and tells her which buttons to push at a vending machine

The Virtual Volunteer tool has a deeper level of understanding and conversational ability compared to other available image-to-text technology. For example, sending a picture of the inside of a refrigerator will not only correctly identify what's in it but offer recipe options and step-by-step guides. If the tool is unable to answer a question, users will have the option to be connected via the app to a sighted volunteer for assistance.

Be My Eyes will integrate the Virtual Volunteer feature into its existing app, powered by OpenAI's GPT-4 language model. The feature allows users to send images via the app to an AI-powered Volunteer, who will answer any questions about the image and provide instant visual assistance for a variety of tasks.

If you or someone you know could benefit from the Be My Eyes app, you can join the waitlist to test the iOS version today, with an Android version to follow later. Visit https://www.bemyeyes.com/download to join the waitlist.