Niantic have recently released information on how they have significantly expanded its use of Machine Learning (ML) throughout the company, going beyond their existing capabilities in Computer Vision and Augmented Reality (AR). Notably, Niantic has made significant strides in Wayfarer reviews. Let’s explore the critical role of ML in maintaining and enhancing Niantic’s foundational maps.
Introduction
Niantic is renowned for its leadership in Computer Vision and AR. Their research team has achieved substantial advancements in computer vision over the company’s history, with notable presentations at various conferences, including CVPR 2023. However, Niantic employs Machine Learning in a variety of ways across the company to improve their games, systems, and products. This article primarily focuses on the use of “classic” supervised ML, while also delving into the active exploration of generative models, such as LLMs, in several key areas.
Before delving into the specifics of Niantic’s ML and AI initiatives, it’s essential to highlight their ability to train and deploy large-scale ML models, which is made possible through their substantial investment in a robust data infrastructure. Niantic has dedicated significant time to establishing proper logging, telemetry, reliable metrics, and label curation to ensure the accuracy and utility of their models.
Machine Learning in Action: Mapping
Niantic’s maps are the bedrock upon which their entire operation is built. These maps must remain accurate and up-to-date. Niantic’s Wayfarer program plays a vital role in continuously updating their maps, with players identifying hidden gems in their neighbourhoods and making necessary updates to landmarks.
Given the ever-changing world, Niantic employs Machine Learning to:
- Identify low-quality wayspot nominations or edits to their maps, including blurry or inappropriate photos, inaccurate descriptions, and incorrect locations
- Flag duplicate wayspots
- Detect abusive behaviour for further review and investigation by Niantic’s team
To tackle these challenges, Niantic has developed a set of deep-learning models that can synthesize data from various sources. While the architecture of each model may differ slightly, they all utilise embedding services for different feature modalities, such as images and text, before passing this information to a fully connected layer. These models are trained using input from the Wayfarer community or Niantic’s internal Ops teams, depending on the task. Substantial effort has also gone into data cleaning, involving manual reviews of numerous examples to understand better the challenges faced by Wayfarer submitters and reviewers.
These ML models have a significant impact, reducing the number of ineligible nominations or edits. This benefits not only Niantic but also the Wayfarer community in two key ways:
- Wayfarers are relieved from reviewing ineligible nominations or edits, allowing them to focus on more interesting and creative wayspots rather than spending time on obvious issues like watermarked or inappropriate images.
- The turnaround time for Wayfarer Explorers is shortened, as models handle low-quality submissions. This means Wayfarers can review and potentially approve valid submissions more quickly, resulting in a threefold reduction in the turnaround time.
Regardless of the application, a crucial step in deploying ML models is the performance of both offline and online evaluations. For Niantic’s Maps models and games-side models, offline evaluation sets are meticulously curated to estimate model metrics, such as precision and recall. Tuning decision thresholds based on the results from offline evaluations allows them to optimise for specific metrics and estimate the impact before launching the models.
Once the models are live, Niantic collaborates closely with their experimentation platform to validate the accuracy of their offline estimates. Experimentation involves creative techniques, including geospatial or temporal testing, to understand the models’ impact. A portion of predictions is also continuously reviewed by humans to provide fresh assessments of live model performance.
In the graph above, Niantic overlaid the lift on different player behaviours to identify where they were (or were not having an impact). In this scenario, they found that their greatest lift came when no special events were running in the game. Note how the black line increases dramatically when the player behaviours (colourful histograms) are not present.
No information has been released suggesting they use a similar AI model for Route submissions, but given the nature of user-generated submissions there will be a huge volume, and so it would take too much time for a human to review them all. An AI model is likely to be able to scan for ineligible nominations in a similar way to Wayspot nominations.
There is some discussion to be had regarding false rejections. As the models are continuously learning, they may reject a nomination that to a human would be perfectly acceptable. As part of Wayfarer, you can appeal a rejection, but there isn’t a system set in place for Routes yet. Although these false rejections can be very demoralising for players, it’s a compromise to allow Wayfarer Reviewers to review legitimate nominations.
Generative AI Ventures
Niantic is at the forefront of cutting-edge technology and is actively exploring the application of Generative AI (GenAI) models. Earlier this year, they introduced Wol, a GenAI-powered mixed reality character with extensive knowledge of the Redwood forests in Northern California. Niantic has also incorporated GenAI modules into 8th Wall, making it easier for WebAR developers to integrate GenAI tools from OpenAI and Inworld.ai into their projects.
Many of the GenAI models Niantic is exploring are still in their early stages, and they are testing both externally provided solutions and their internally hosted models. Niantic’s strategy for applying these models includes:
- Enhancing internal scaling and efficiency.
- Improving gameplay features.
- Developing new experiences, such as the introduction of Wol.
Niantic invites readers to stay tuned as they roll out prototypes in these areas, both for public and internal consumption.
Looking Ahead
Niantic’s commitment to being a data-driven company places great importance on aligning with users’ needs and desires. They continue to harness the power of machine learning and explore the potential of Generative AI, ensuring that these technologies serve the community they’ve built in unique and meaningful ways. Niantic eagerly anticipates using ML and AI to further support and nurture their exceptional community.
