SPATIAL COMPUTING CHALLENGE
"AUGMENTED REALITY TIMELINE" PROPOSAL
We intend to implement mobile phone-based Augmented Reality (AR) technology to convey event-specific or site-specific history by integrating digital and physical timelines.
Phase 1 (Minimum Viable Product): Event-Specific Timelines printed on graphic media will display surface level information on upcoming or current events (ex. Olympics, US elections). Users can open the New York Times App and view the printed timeline through their camera in order to view augmented content superimposed onto the timeline (ex. interviews, 3D models, audio clips).
Phase 2: Site-Specific Timelines embedded into storied city landmarks (ex. Triangle Shirtwaist Building, World Trade Center) will display surface level information on the history of the location. Users can open the New York Times App and view the physical timeline through their camera in order to view augmented content superimposed onto the timeline, featuring relevant current events content.
This technology hopes to enrich a user's understanding of an event or site through multimodal technology. Phase 1 provides users with mixed perspectives on upcoming or current events by providing detailed content only presentable through digital means. Phase 2 informs New Yorkers about the storied history of their environment and directly connects the events of the past with consequences we see in today's current events.
3) Implementation (Phase 1)
A. Creation of printable graphic content
Technical components: Adobe Illustrator, AprilTag Visual Fiducial System
Summary: Design a physical timeline to convey event information, including AprilTags to facilitate computer vision for AR implementation
B. Implement Computer Vision (CV) timeline detection
Technical components: AprilTag Visual Fiducial System, Swift, Xcode, RealityKit
Summary: Develop mobile environment in which CV can be utilized to recognize AprilTags on a printed timeline in order to assess the 3D positional relationship between the mobile viewer and physical timeline
C. Superimpose AR graphics onto physical timeline
Technical components: Swift, Xcode, Apple ARKit, RealityKit
Summary: Implement ARKit in order to superimpose simple graphic content onto the physical timeline, including images and text
D. Generate or research multimodal digital content
Technical components: Photography, Videography, Adobe Photoshop, Adobe Premiere Pro, Adobe After Effects, Blender
Summary: Generate digital content relevant to a particular event in the form of photos, videos, text, 3D models, animations, etc. Research existing digital media coverage on a particular event.
E. Develop "graphic content to mobile AR" pipeline
Technical components: Swift, Xcode
Summary: Develop a systematic approach for intuitively formatting and superimposing mixed media content onto the digital timeline.
4) Existing Solutions
Video content superimposed onto advertisement using AR
AR and 3D modeling as educational tools
Virtual tour using AR overlay on top of physical space
Video content superimposed onto a thumbnail image using AR
5) Technical References
ARKit Documentation: https://developer.apple.com/documentation/arkit
RealityKit Documentation: https://developer.apple.com/documentation/realitykit
Apple Computer Vision: https://developer.apple.com/documentation/vision
AprilTag Visual Fiducial System: https://april.eecs.umich.edu/software/apriltag