Application
Games
When Google revealed Project Gameface, the company was proud to show off a hands-free, AI-powered gaming mouse that, according to its announcement, “enables people to control a computer’s cursor using their head movement and facial gestures.” While this may not be the first AI-based gaming tool, it was certainly one of the first to put AI in the hands of players, rather than developers.
The project was inspired by Lance Carr, a quadriplegic video game streamer who utilizes a head-tracking mouse as part of his gaming setup. After his existing hardware was lost in a fire, Google stepped in to create an open source, highly configurable, low-cost alternative to expensive replacement hardware, powered by machine learning. While AI’s broader existence is proving divisive, we set out to discover whether AI, when used for good, could be the future of gaming accessibility.
It’s important to define AI, and machine learning, to understand clearly how they work in Gameface. When we use the terms “AI” and “machine learning,” we’re referring to both the same and different things.
“AI is a concept,” Laurence Moroney, AI advocacy lead at Google and one of the minds behind Gameface, tells WIRED. “Machine learning is a technique you use to implement that concept.”
Machine learning, then, fits under the umbrella of AI, along with implementations like large language models. But where familiar applications like OpenAI’s ChatGPT and StabilityAI’s Stable Diffusion are iterative, machine learning is characterized by learning and adapting without instruction, drawing inferences from readable patterns.
Moroney explains how this is applied to Gameface in a series of machine learning models. “The first was to be able to detect where a face is in an image,” he says. “The second was, once you had an image of a face, to be able to understand where obvious points (eyes, nose, ears, etc.) are.”
After this, another model can map and decipher gestures from those points, assigning them to mouse inputs.
It’s an explicitly assistive implementation of AI, as opposed to those often touted as making human input redundant. Indeed, this is how Moroney suggests AI is best applied, to broaden “our capacity to do things that weren’t previously feasible.”
This sentiment extends beyond Gameface’s potential to make gaming more accessible. AI, Moroney suggests, can have a major impact on accessibility for players, but also on the way developers create accessibility solutions.
“Anything that lets developers be orders of magnitude more effective at solving classes of problems that were previously infeasible,” he says, “can only be beneficial in the accessibility, or any other, space.”
This is something developers are already beginning to understand. Artem Koblov, creative director of Perelesoq, tells WIRED that he wants to see “more resources directed toward solving routine tasks, rather than creative invention.”
Doing so allows AI to aid in time-consuming technical processes. With the right applications, AI could create a leaner, more permissive, development cycle in which it both helps in the mechanical implementation of accessibility solutions and leaves developers more time to consider them.
“As a developer, you want to have as many tools that can help you make your job easier,” says Conor Bradley, creative director of Soft Leaf Studios. He points to gains in current implementations of AI in accessibility, including “real-time text-to-speech and speech-to-text generation, and speech and image recognition.” And he sees potential for future developments. “In time, I can see more and more games making use of these powerful AI tools to make our games more accessible.”
Koblov believes it can go even further. He’d like to see AI training on specific patterns to create a basic, adaptable accessibility framework that could be injected into games. “Such framework would adapt the visual, audio, and interactive aspects of games,” he says. “In other words, smaller developers like us wouldn’t have to conduct expensive research, develop unique solutions, and go through numerous iterations of testing on their own.”
Bradley urges caution when pulling primacy away from human input. Asked whether AI could prove an aid or a distraction to existing accessibility efforts, he said he was optimistic about its potential, but stressed that AI is not a shortcut.
“You cannot say, ‘AI, make my game accessible!’ and hey presto, you now have the most accessible game of the year,” he says. “We need players, including those from disabled and neurodiverse communities, to test our games. At the end of the day, a human will be playing your game, not a machine.”
While Koblov believes AI could be valuable for implementing and testing accessibility features, he acknowledges that thinking about AI requires “an ‘addition’ mindset,” rather than a “replacement” approach.
But conflating the generative, content-driven tools that spark fears of human redundancy with the kind of AI implementations that help accessibility is, according to Moroney, “Really dangerous.” He continues, “If we’re going to be the adults in the room when it comes to AI, we have to recognize hype and bandwagons.”
This makes clarity and transparency about AI’s capabilities all the more important, especially in its relation to accessibility. It’s not a magic wand. “AI and machine learning were doing well until the recent releases,” Moroney says. “Now they’ve fallen back in the hype cycle.”
AI can be an excellent tool for developers, but they must remain dedicated to accessibility throughout the process, whether AI is present or not. After all, as Bradley says, “At the end of the day, it still is up to the developers to want to make their games accessible by design.”
AI’s gradual progress is evident in Gameface. But another project demonstrates how AI-assisted accessibility can be implemented on a wider level. Minecraft Access is a mod that seeks to make Minecraft accessible to blind and visually impaired players. Logic, part of the team behind the mod, tells WIRED how a suite of AI tools, including ChatGPT and Google’s own Tensor Flow, are helping with the project.
“We are hoping AI can fill in … visual context for blind and low-vision players by providing information about the world as it is needed or upon request,” Logic says.
Particularly exciting is the potential for AI to not just bolster accessibility, but actively learn what a player needs. This will prove especially useful for broader applications in accessibility, given the layers of spectrums that make up disability and how personalized each player’s needs are.
We do need to rein in our expectations, however. As promising as these recent implementations have proven, and as instructive as they may be for the future, there remain significant barriers to entry. In its current stage of development, Minecraft Access requires multiple programs to function, something Logic acknowledges makes it less accessible than it could be.
“The average user is not going to want to collect a bunch of programs from different parts of the web,” Logic says.
Similarly, Ben Green, a disabled gamer, finds Gameface’s potential exciting but worries about diversity in the data. It may be able to “recognize lots of faces,” he says. “But some people with facial differences, like a ventilator in my case, or asymmetrical facial features, might be barely represented, or not at all.”
When asked about this, Miguel de Andrés-Clavera, who leads the team that developed Project Gameface, says, “We decided to create a functionality for people to customize which expressions they used to control the mouse.” This includes the ability to customize the intensity of gestures for different needs. He goes on, “With that being said, we are always looking for ways to increase the accessibility of our technology for more people. Our hope is that over time, Project Gameface will continue to improve and become even more helpful.”
Even with these caveats, it’s interesting to see how hopeful people are about AI’s role in accessibility. Once we can distinguish between unethical applications of content-driven generative AI and meaningful AI tools and implementations that can help people solve problems and benefit others, there’s plenty of cause for optimism—with the understanding that AI’s true value is tied to our ability to make it work for us.
The future of AI is ambiguous, but it holds the potential to benefit individual gamers and the industry at large. Its use requires caution, and we can expect pitfalls, but there’s every reason to believe that careful implementation of AI can contribute to a gaming landscape that encompasses a wider spectrum of players.
That’s the world Moroney wants to live in: “a world where people like Lance aren’t confined because solutions are technically infeasible, but rather one where developers have such superpowers that building solutions to allow him to connect to the world is easy.”
