top of page

SPATIAL COMPUTING CHALLENGE 

"AUGMENTED REALITY TIMELINE" PROPOSAL

1) Abstract

We intend to implement mobile phone-based Augmented Reality (AR) technology to convey event-specific or site-specific history by integrating digital and physical timelines.

  • Phase 1 (Minimum Viable Product): Event-Specific Timelines printed on graphic media will display surface level information on upcoming or current events (ex. Olympics, US elections). Users can open the New York Times App and view the printed timeline through their camera in order to view augmented content superimposed onto the timeline (ex. interviews, 3D models, audio clips).

  • Phase 2: Site-Specific Timelines embedded into storied city landmarks (ex. Triangle Shirtwaist Building, World Trade Center) will display surface level information on the history of the location. Users can open the New York Times App and view the physical timeline through their camera in order to view augmented content superimposed onto the timeline, featuring relevant current events content.

2) Justification

This technology hopes to enrich a user's understanding of an event or site through multimodal technology. Phase 1 provides users with mixed perspectives on upcoming or current events by providing detailed content only presentable through digital means. Phase 2 informs New Yorkers about the storied history of their environment and directly connects the events of the past with consequences we see in today's current events.

3) Implementation (Phase 1)

A. Creation of printable graphic content

  • Technical components: Adobe Illustrator, AprilTag Visual Fiducial System

  • Summary: Design a physical timeline to convey event information, including AprilTags to facilitate computer vision for AR implementation

​

​

​

​

​

​

​

​

​

B. Implement Computer Vision (CV) timeline detection

  • ​Technical components: AprilTag Visual Fiducial System, Swift, Xcode, RealityKit

  • Summary: Develop mobile environment in which CV can be utilized to recognize AprilTags on a printed timeline in order to assess the 3D positional relationship between the mobile viewer and physical timeline

​

​

​

​

​

​

​

C. Superimpose AR graphics onto physical timeline

  • Technical components: Swift, Xcode, Apple ARKit, RealityKit

  • Summary: Implement ARKit in order to superimpose simple graphic content onto the physical timeline, including images and text

​

​

​

​

​

​

​

​

​

​

​

​

​

​​

​

​

D. Generate or research multimodal digital content

  • Technical components: Photography, Videography, Adobe Photoshop, Adobe Premiere Pro, Adobe After Effects, Blender

  • Summary: Generate digital content relevant to a particular event in the form of photos, videos, text, 3D models, animations, etc. Research existing digital media coverage on a particular event.

E. Develop "graphic content to mobile AR" pipeline

  • Technical components: Swift, Xcode

  • Summary: Develop a systematic approach for intuitively formatting and superimposing mixed media content onto the digital timeline.

Screen Shot 2019-12-03 at 5.51.58 PM.png
Screen Shot 2019-12-03 at 6.04.34 PM.png
Screen Shot 2019-12-03 at 6.21.39 PM.png

4) Existing Solutions

Video content superimposed onto advertisement using AR

AR and 3D modeling as educational tools

Virtual tour using AR overlay on top of physical space

Video content superimposed onto a thumbnail image using AR

5) Technical References

  1. ARKit Documentation: https://developer.apple.com/documentation/arkit

  2. RealityKit Documentation: https://developer.apple.com/documentation/realitykit

  3. Apple Computer Vision: https://developer.apple.com/documentation/vision

  4. AprilTag Visual Fiducial System: https://april.eecs.umich.edu/software/apriltag

bottom of page