Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
education:remote_collaboration_via_tele-immersion [2017/07/11 07:38]
lhkellogg
education:remote_collaboration_via_tele-immersion [2017/07/14 12:42] (current)
okreylos
Line 3: Line 3:
 ===Motivation:​=== ===Motivation:​===
  
-One of the challenges ​of working ​in an immersive visualization environment is how to work collaboratively+The collaborative nature ​of modern science, with large teams of researchers from different institutions ​working ​together to solve problems, poses grand challenges and offers great opportunities for immersive visualization. Existing means of tele-collaboration,​ such as videoconferencing or shared whiteboard software, offer little to no support for joint visual analysis of three-dimensional data. On the other hand, our decade of experience at KeckCAVES has shown that colocated visual analysis, with multiple researchers using the same immersive visualization environment ​at the same time, is a powerful research tool.
  
-In this video, we demonstrate ​the power of remote ​collaboration:​  +One major area of KeckCAVES software research and development is to replicate ​this effective style of local collaboration for groups of scientists that are not physically in the same place. Our approach is to create a pseudo-holographic videoconferencing system, where scientists can enter a shared virtual space containing their data, and see and interact with each other via life-size, pseudo-holographic 3D video avatars. 
-[[https://​www.youtube.com/​watch?​v=B4g9J-aSF-c|Pilot video: ​Collaborative Visualization of Microbialites ]]+ 
 +This video shows an early prototype ​of KeckCAVES'​ tele-collaboration ​framework, filmed by a user participating in a shared session from a standard desktop computer:  
 +[[https://​www.youtube.com/​watch?​v=B4g9J-aSF-c|Collaborative Visualization of Microbialites ​]] 
 + 
 +This video shows the experience of encountering a life-size 3D video avatar (in this case a previously recorded clip of the same user) in a large-scale immersive environment:​ [[https://​www.youtube.com/​watch?​v=VJ9Ih-49O3o|Life-size 3D Video Avatars]]
  
 ===Major activities:​=== ===Major activities:​===
  
-large software development ​effort was undertaken to support current and upcoming low-cost commodity VR display and interaction devicesspecifically the Oculus Rift head-mounted display ​and PS Move VR controller, in the Vrui VR development toolkit that is the foundation for all major goals of this project. This on-going infrastructure work will dramatically increase the potential user base for technologies and software developed under this project from several hundreds of people with access ​to CAVEs and other high-end VR environments to potentially millions ​of home users with gaming VR equipment. There are currently over 250,000 Rift development kits sold, and first-day pre-orders for the upcoming commercial version are estimated at above 100,000. The upcoming HTC Vive head-mounted display is expected ​to sell in similar numbers, and the potential market for Sony’s upcoming PSVR head-mounted display are over 30 million current owners ​of PlayStation 4 consoles.+This CI-TEAM project supported two related ​large software development ​efforts: improvement of the tele-collaboration framework itself, and further extension of the underlying ​Vrui VR operating system ​to enable deployment ​of the tele-collaboration framework ​to a wide audience ​of scientists.
  
-In parallel, work was done to support ​additional 3D cameras, specifically the second-generation Microsoft Kinect for Xbox One, as 3D capture devices for the tele-immersion component of the project. Compared to the first-generation Kinect used by previous versions of the component, the new Kinect offers significantly higher resolution and image quality, and is more widely available ​as the first-generation Kinect has been discontinued.+The first branch focused on supporting ​additional 3D cameras, specifically the second-generation Microsoft Kinect for Xbox One, as 3D capture devices for the tele-immersion component of the project. Compared to the first-generation Kinect used by previous versions of the component, the new Kinect offers significantly higher resolution and image quality, and is more widely available ​than the first-generation Kinect, which has been discontinued. During the project'​s life time, additional types of 3D cameras, such as Intel'​s RealSense series, appeared on the market, and were integrated into the framework'​s 3D video component as well. 
 + 
 +A second major goal of this first branch was to improve the network protocol underlying the tele-collaboration framework, specifically its robustness against network failures and client-side crashes. Early prototypes of the framework were prone to system-wide failures when the often fragile 3D video capture hardware crashed at one participant'​s site. 
 + 
 +The second development branch focused on adding support for current and upcoming low-cost commodity VR display and interaction devices to the underlying Vrui VR operating system. Only few scientists have access to high-end immersive visualization systems such as CAVEs, and regular (2D) desktop computers are significantly less effective for 3D data analysis than those. At the beginning of the project, before the current renaissance of head-mounted virtual reality set off by the successful Oculus Rift Kickstarter campaign, the best low-cost option for 3D data analysis were systems based on commodity 3D TVs and optical motion capture technology. Initially, this branch focused on creating tools, documentation,​ and instructional material to allow non-expert users to build their own low-cost immersive systems, and on developing software drivers for commodity interaction devices not originally aimed at immersive use, such as Nintendo'​s Wii remote, and Sony's PlayStation Move controller. 
 + 
 +This branch'​s focus shifted in response to the unforeseen emergence of high-quality commodity head-mounted displays, starting with the second development kit of the Oculus Rift, and then the development kits and commercial versions of the HTC Vive. Through a combination of reverse engineering and analysis of often experimental driver software released by the commercial entities behind these headsets, native support for multiple of them was added to the Vrui toolkit. 
 + 
 +Taken together, these efforts dramatically increased the potential user base for technologies and software developed under this project from several hundreds of researchers with access to CAVEs and other high-end VR environments to potentially millions of scientists of even home users with commodity VR equipment. It is estimated that half a million units of HTC Vive have been sold in its first year, with about half as many units of Oculus Rift shipped over the same time. The potential market for Sony's PSVR head-mounted display are over 30 million current owners of PlayStation 4 consoles, and Sony already shipped 0.75 million units in the last few months of 2016 alone.
  
 ===Specific Objectives:​=== ===Specific Objectives:​===
  
-The major activities described above required reverse-engineering of the optical tracking sub-system ​used by the second ​Oculus Rift development kit and implementation of cutting-edge computer vision algorithms, development of calibration tools and image warping techniques ​to correct ​for flaws in the low-cost optics used by commodity head-mounted displays, development of computer vision ​techniques for optical tracking ​of PS Move controllers,​ and reverse-engineering ​of the data transfer protocols ​and 3D imaging algorithms used by the second-generation Kinect 3D camera.+The major activities described above required reverse-engineering of the data transfer protocols and 3D imaging algorithms ​used by the second-generation Kinect 3D camerato create a driver that is open-source and completely unencumbered by proprietary Microsoft technology. It also required ​development of a new client-server architecture ​and network protocol ​to control shared virtual spaces robust in the face of client drop-outs. Furthermore,​ the support ​for collaborative exploration and analysis ​in several Vrui applications,​ specifically 3D Visualizer and VR ProtoShop, was improved and extended. 
 + 
 +The second activity branch required in-depth investigation of proprietary hardware to develop open-source drivers compatible with Vrui's architecture. In detail, this involved ​the analysis of USB or Bluetooth data transmission protocols to develop ​low-level tracking drivers for the Oculus Rift DK1 and DK2, development of computer vision ​algorithms to use the Oculus Rift DK2'​s ​optical tracking ​technology and to track Nintendo Wii remote and PlayStation ​Move controllers,​ and analysis ​of internal programming interfaces to use the vendor-supplied tracking driver for the HTC Vive and its controllers as input sources for the Vrui toolkit and its applications.
  
 ===Significant Results:=== ===Significant Results:===
  
-The Vrui toolkit, and therefore all Vrui-based applications created for and used by this project, now fully support the second-generation development kit for the Oculus Rift head-mounted display, and we expect that only minor further ​development ​will be required to also support the yet-unreleased ​commercial ​versionWe already acquired an HTC Vive development kit, but work to support ​it is still on-going. An optical 3D tracking system for PS Move controllers is working, but work to fuse optical tracking data with readings ​from its built-in inertial navigation system is ongoing. A low-level driver for the second-generation Microsoft Kinect camera has been completed ​and has been integrated into the tele-collaboration framework. Howevermore research ​and development into low-latency video compression is required to handle the larger bandwidth requirements ​of the higher-resolution camera.+The Vrui toolkit, and therefore all Vrui-based applications created for and used by this project, now fully support the second-generation ​Microsoft Kinect 3D camera, the entire line of Intel RealSense 3D cameras, the first and second ​development kit for the Oculus Rift VR headset, and the development ​kits and commercial ​versions of the HTC Vive VR headsetSpecific mention is due to Vrui's new HTC Vive support. It enables a non-expert user to buy a VR headset and a VR-compatible desktop PC from any of several mainstream online or brick-and-mortar vendors, set it up, and install and configure ​the Vrui toolkityielding a complete immersive visualization system with functionality roughly on par with KeckCAVES'​ CAVE, in just a few hours and for a total cost of around USD 2,500.
  
 ===Key outcomes or Other achievements:​=== ===Key outcomes or Other achievements:​===
  
 Software and technologies developed for or used by this project are now available to a larger audience (beyond the students being directly trained) using commodity hardware. As we have been releasing all software under a free and open-source license on an ongoing basis, external users have begun applying methods developed here for their own purposes. Software and technologies developed for or used by this project are now available to a larger audience (beyond the students being directly trained) using commodity hardware. As we have been releasing all software under a free and open-source license on an ongoing basis, external users have begun applying methods developed here for their own purposes.
 +
 +The progress of this development has been disseminated online, through a series of YouTube videos and blog posts, and has been followed by a large audience of non-experts. YouTube videos related to this project have been viewed a total of more than 1.1 million times, and project-related blog posts have been read a total of more than 200,000 times.
 +