We at BarrierBreak wanted to experiment with new ways to bring assistive technologies to the public. We decided to look into using web technologies since the power of the web (and the mobile phone) ensures instant reach to hundreds of millions of people. We have used AR (Augmented Reality) in certain apps and scenarios to provide an additional level of immersion and depth to the experience.
These tools are experimental approaches to assitive technologies aimed to help people with low vision, as well as people trying to understand low-vision issues. Some of these (like the magnifier and the color sieve) solve particular problems with certain low-vision conditions, whereas others (like the simulators) show how a person with these low-vision conditions might see the world using an augmented reality approach.
These tools are built using web technologies, which means nothing to download and install. This significantly reduces the barrier to entry to trying out such tools.
Watch the video below where we talk more about the project and how we went about implementing it from a technical perspective.
These apps are designed for use on a mobile device. They are not intended for, and in all probability, not work on a desktop.
Recommended Device(s): For now, we recommend using an Android device.
Recommended Browser: We recommend using the latest version of Chrome for Android.
On Chrome on Android, it is recommended that you enable the following flags.
Then restart the browser.
In many of these apps, you might notice the following icon.
The above icon is for enabling 'headset mode'. Doing so will enable stereoscopic video and put the app in landscape mode. This mode is specifically for use in VR headsets. Once enabled, you can mount the phone on a VR headset and get a more immersive experience.
To get out of 'headset mode', tap on the video again - this should get the app out of headset mode and back to normal display.
Nowadays, VR headsets are easily available using online stores. Currently, they can typically cost between $25-$80, but costs can go higher and lower than this too. For example, in India, VR headsets are readily available in online stores like Amazon.in and Flipkart.
Note: Make sure that the VR headset has a slot to make sure the phone's back camera can actually see to the other side. For example, Google Cardboard cannot work here since it does not have a slot to let the camera see to the other side.
You can click through various color blindness conditons and see in the video output a simulation of the way a person with that condition might see the world. You can turn the device into landscape mode too, which will give you the same options, but with a layout fit for that mode (with the video on the left, and the list of options on the right).
Point the device's back camera to an object. The app will classify and isolate the colors in the video into red-ish colors and green-ish ones. The box on the left will have the video of the parts which are red-ish in color, and the one on the right will be for green-ish colors.
In this way, a person who is not sure whether an object is red or green, will get to know for sure which of the two color categories it falls into. If the object is neither red-ish in color nor green-ish, it simply won't appear in either of the two boxes.
To magnify, extend the slider all the way to the right. The magnifier allows you to zoom all the way upto the maximum limit that the mobile phone camera allows - but then you can magnify even further using pinch-and-zoom. This way, you can combine optical and digital zoom to get a boost in magnification, and just this feature alone makes it instantly better than every other magnifier native app out there.
To enable the torch (but seeing in low-light conditions), tap on the icon to the right of the slider which looks like a torch. Tapping it again would disable it.
To freeze the camera, tap on the button which says 'Freeze Caemra'. This will give you a snapshot image of the currently focussed camera outoput, so that you can take more time to see the image and not need to keep the phone physically pointed at the object. Once you're done, and need to go back to the live camera output, click on the button which says 'Return to live camera'.
The icon which looks a bit like a book or tablet device is for 'reading mode'. For things such as text, (and especially handwriting) people with low-vision sometimes appreciate a mode where the colors could be inverted (so that most text, which is light background with dark text becomes dark background with light text - which is easier on the eyes for long reading) and the contrast could be made better. This mode does exactly that.
Click on the icon for reading mode again to exit reading mode.
These simulators all work in the same way. The portrait view shows the video output with a simulation of how a person with the respective condition would see it. Turning the device over to landscape mode would enable 'headset mode', which is stereoscopic video mode. Tapping on the video in this mode will make it go full-screen, and will be ready to be mounted on a VR headset for a more 'immersive' experience.
To exit headset mode, tap on the video again.
All these tools are highly experimental. They may not work as expected. We would love to get feedback in order to improve these tools on the email address mentioned below.