The collaborative nature of modern science, with large teams of researchers from different institutions working together to solve problems, poses grand challenges and offers great opportunities for immersive visualization. Existing means of tele-collaboration, such as videoconferencing or shared whiteboard software, offer little to no support for joint visual analysis of three-dimensional data. On the other hand, our decade of experience at KeckCAVES has shown that colocated visual analysis, with multiple researchers using the same immersive visualization environment at the same time, is a powerful research tool.
One major area of KeckCAVES software research and development is to replicate this effective style of local collaboration for groups of scientists that are not physically in the same place. Our approach is to create a pseudo-holographic videoconferencing system, where scientists can enter a shared virtual space containing their data, and see and interact with each other via life-size, pseudo-holographic 3D video avatars.
This video shows an early prototype of KeckCAVES' tele-collaboration framework, filmed by a user participating in a shared session from a standard desktop computer: Collaborative Visualization of Microbialites
This video shows the experience of encountering a life-size 3D video avatar (in this case a previously recorded clip of the same user) in a large-scale immersive environment: Life-size 3D Video Avatars
This CI-TEAM project supported two related large software development efforts: improvement of the tele-collaboration framework itself, and further extension of the underlying Vrui VR operating system to enable deployment of the tele-collaboration framework to a wide audience of scientists.
The first branch focused on supporting additional 3D cameras, specifically the second-generation Microsoft Kinect for Xbox One, as 3D capture devices for the tele-immersion component of the project. Compared to the first-generation Kinect used by previous versions of the component, the new Kinect offers significantly higher resolution and image quality, and is more widely available than the first-generation Kinect, which has been discontinued. During the project's life time, additional types of 3D cameras, such as Intel's RealSense series, appeared on the market, and were integrated into the framework's 3D video component as well.
A second major goal of this first branch was to improve the network protocol underlying the tele-collaboration framework, specifically its robustness against network failures and client-side crashes. Early prototypes of the framework were prone to system-wide failures when the often fragile 3D video capture hardware crashed at one participant's site.
The second development branch focused on adding support for current and upcoming low-cost commodity VR display and interaction devices to the underlying Vrui VR operating system. Only few scientists have access to high-end immersive visualization systems such as CAVEs, and regular (2D) desktop computers are significantly less effective for 3D data analysis than those. At the beginning of the project, before the current renaissance of head-mounted virtual reality set off by the successful Oculus Rift Kickstarter campaign, the best low-cost option for 3D data analysis were systems based on commodity 3D TVs and optical motion capture technology. Initially, this branch focused on creating tools, documentation, and instructional material to allow non-expert users to build their own low-cost immersive systems, and on developing software drivers for commodity interaction devices not originally aimed at immersive use, such as Nintendo's Wii remote, and Sony's PlayStation Move controller.
This branch's focus shifted in response to the unforeseen emergence of high-quality commodity head-mounted displays, starting with the second development kit of the Oculus Rift, and then the development kits and commercial versions of the HTC Vive. Through a combination of reverse engineering and analysis of often experimental driver software released by the commercial entities behind these headsets, native support for multiple of them was added to the Vrui toolkit.
Taken together, these efforts dramatically increased the potential user base for technologies and software developed under this project from several hundreds of researchers with access to CAVEs and other high-end VR environments to potentially millions of scientists of even home users with commodity VR equipment. It is estimated that half a million units of HTC Vive have been sold in its first year, with about half as many units of Oculus Rift shipped over the same time. The potential market for Sony's PSVR head-mounted display are over 30 million current owners of PlayStation 4 consoles, and Sony already shipped 0.75 million units in the last few months of 2016 alone.
The major activities described above required reverse-engineering of the data transfer protocols and 3D imaging algorithms used by the second-generation Kinect 3D camera, to create a driver that is open-source and completely unencumbered by proprietary Microsoft technology. It also required development of a new client-server architecture and network protocol to control shared virtual spaces robust in the face of client drop-outs. Furthermore, the support for collaborative exploration and analysis in several Vrui applications, specifically 3D Visualizer and VR ProtoShop, was improved and extended.
The second activity branch required in-depth investigation of proprietary hardware to develop open-source drivers compatible with Vrui's architecture. In detail, this involved the analysis of USB or Bluetooth data transmission protocols to develop low-level tracking drivers for the Oculus Rift DK1 and DK2, development of computer vision algorithms to use the Oculus Rift DK2's optical tracking technology and to track Nintendo Wii remote and PlayStation Move controllers, and analysis of internal programming interfaces to use the vendor-supplied tracking driver for the HTC Vive and its controllers as input sources for the Vrui toolkit and its applications.
The Vrui toolkit, and therefore all Vrui-based applications created for and used by this project, now fully support the second-generation Microsoft Kinect 3D camera, the entire line of Intel RealSense 3D cameras, the first and second development kit for the Oculus Rift VR headset, and the development kits and commercial versions of the HTC Vive VR headset. Specific mention is due to Vrui's new HTC Vive support. It enables a non-expert user to buy a VR headset and a VR-compatible desktop PC from any of several mainstream online or brick-and-mortar vendors, set it up, and install and configure the Vrui toolkit, yielding a complete immersive visualization system with functionality roughly on par with KeckCAVES' CAVE, in just a few hours and for a total cost of around USD 2,500.
Software and technologies developed for or used by this project are now available to a larger audience (beyond the students being directly trained) using commodity hardware. As we have been releasing all software under a free and open-source license on an ongoing basis, external users have begun applying methods developed here for their own purposes.
The progress of this development has been disseminated online, through a series of YouTube videos and blog posts, and has been followed by a large audience of non-experts. YouTube videos related to this project have been viewed a total of more than 1.1 million times, and project-related blog posts have been read a total of more than 200,000 times.