How can journalism leverage emerging technologies?
Overview
Research & Development (R&D) at The New York Times explores how emerging technologies can serve journalism. I led production across Spatial Journalism (AR, VR, MR) and Artificial Intelligence—supporting integrations across newsroom in the form of tools for reporters in the field or entirely new formats for publishing stories.
Our teams at NYT R&D released multiple research articles. Documenting our extended reality and headset-based work we wrote about: immersive tools for reporters in the field, location-based open-world news, design patterns for immersive storytelling, a behind-the-scenes on building for mixed reality journalism, and a case study on the 31 published handheld AR stories.
Research Article #1Exploring Mixed Reality Tools for Journalists
R&D worked directly with investigative and visual journalists to ask what mixed reality could actually help with in the field—not as a gimmick, but under deadline. The result was functional prototypes for dropping spatial notes as reporters move through a scene and streaming 3D scans to editors in real time. Reporters tested tools in the newsroom simulating breaking-news scenarios; one proof of concept captured The Times photo archive—the Morgue—in hours using a HoloLens, overlaying decades of images onto a walkable 3D model.
My Research HighlightCan mixed reality reconstructions of found media aid investigative reporters?
Verifying what actually occurred during an event is core to investigative reporting. This prototype lets reporters view found media at the locations where it was captured—overlaying videos and photos in the real world so they can compare sources side by side and piece together a chronology of what happened.
To place 2D footage in 3D space, the team used structure-from-motion computer vision to reconstruct the camera path and environment mesh, with scenes aligned using Reality Capture and viewed on a Microsoft HoloLens. Playback is controlled through WebXR hand gestures that let reporters scrub through a timeline while moving through the scene. Future work could use geolocation and camera metadata from source media to automate alignment, and machine learning to flag similarities or discrepancies between assets.
Research Article #2Designing Mixed Reality Journalism for the Open World
As mixed reality headsets learn to sense the world around you, what should news become? R&D designed a prototype for location-based journalism that rewards curiosity—standing in a place and discovering Times reporting tied to that exact spot, including stories from a 171-year archive that could contextualize the space you’re in. Icons appear as you look around; moving closer reveals more detail, with subtle wayfinding that doesn’t tell you where to look. Long-form reading moves to your phone via tethering, so the headset stays for discovery and the story can continue elsewhere.
Research Article #3Exploring Design Patterns for Building Mixed Reality Stories
Mixed reality opens limitless storytelling possibilities, but without shared formats it’s hard to produce news at scale. R&D adapted an existing Times story—on ventilation and reopening schools—for the Quest 2, using it to develop repeatable building blocks for MR journalism. The centerpiece is a tabletop stage called the fish tank that scales to any surface in your home, balancing the guided clarity of a linear story with room for hands-on exploration. Experiments included grabbable timeline controls, interactive particle simulations readers can run themselves, and voiceover to replace walls of text on a headset.
Research Article #4Building Stories for AR Headsets: Interacting in 3D
Shipping AR journalism means solving problems you never encounter on the web—hand tracking, world scale, 3D models that respond to movement. This video series goes behind the scenes with R&D engineers and producers as they build and release Times AR projects, describing the roles, workflows, and technical decisions along the way. Interacting in 3D is one installment in a multi-part series that also covers world sensing, responsive models, and designing stories that work at different scales.
Project Credits & Information
- Project type
- Research
- Partner
- The New York Times
- Agency
- —
- Role
- Production, Strategy
- Credits
- R&D Team (Nick Bartzokas, Matt Brown, A.J. Chavar, Jon Chen, Tim Clark, Chloé Desaulles, Alexandre Devaux, Lydia Jessup, Maximilian Lauter, Scott Lowenstein, Mark McKeague, Jonathan Fletcher Moore, Fabio Piparo, Anneliese Sloves, Aharon Wasserman)
- Press
- —
- Website
- NYT R&D
- Date
- 2021–2022
