Augmented Immersive Video-based Interactive Environments
AIVIE (2013-2014) was a project that focused on Augmented Immersive Video: a novel medium based on close integration of omni-directional/panoramic video content and computer graphics.
Within this project, my task was to integrate virtual objects in 360° videos of real-world scenes. To achieve this, a Structure-from-Motion algorithm, aimed at omni-directional video, was implemented and used to correctly align the virtual overlay with the captured environment.
An example application was to show the residents of a neighborhood what their surrounding would look like after renovation of the neighborhood. A 3D model of the new streets and town square was rendered on top of the omni-directional video recording. The result was a 360° video in which the old streets and square were replaced by the new layout, while the buildings remained untouched. Special care was taken to correctly handle occlusions: a street that was located behind a house should not be rendered on top of the building, and new objects, such as trees, cast shadows on buildings. The resulting video could be viewed in Virtual Reality, allowing the viewer to walk through the renovated streets. Additionally, it was possible to switch between different overlays (e.g., 3D model, maps, no overlay) to compare the differences between the current and the future situation.
Below are some results (please click an image to enlarge):
