Tele-immersion is aimed to enable users in geographically distributed sites to collaborate in real time in a shared simulated environment as if they were in the same physical room. This enterprise has engaged the skills of researchers in a variety of disciplines, including computer vision, graphics and network communications. Tele-immersion is aimed to be used in different areas, such as 3D CAD design, ergonomics, entertainment (e..g games), remote learning and training, coordination of activities (e.g. dancing, rehabilitation), 3D motion capture of body segments etc.
The tele-immersion lab at UC Berkeley is located at Hearst Memorial Mining Building #475. Our tele-immersion system has 360 degree stereo capturing capability which allows full-body 3D reconstruction of people and objects. The data is captured in real time and projected into a virtual environment as a point cloud which can be combined with virtual objects and scenes.
The current apparatus includes 48 cameras arranged in 12 stereo clusters. The images from each cluster are processed by a stereo reconstruction program running in parallel on 12 computers. The acquired data can be sent via Gigabit Internet II connection to another computer to be rendered into a 3-dimensional scene. The data can be displayed using passive stereo projection to increase the perception of depth. We also have hardware capable of capturing and playing sound from 4 microphones and 4 speakers, and 8 infrared lights that project patterns that facilitate depth detection.
We currently have several projects underway to improve the existing technology, including segmentation algorithms to recognize human beings and segment their data into individual limbs, mathematical models of human movement in order to identify actions, refinement of stereo algorithms to better discern depth information in less time, customized hardware architectures to optimize performance of existing algorithms, and ongoing experiments and refinements in the sharing of real-time video and audio data over long distances.
Intended applications of our tele-immersion system include teleconferencing, distance learning, archiving and distribution of physical performances, and training for complex and hazardous tasks where physical visual interactions are the key component.
We have collaborated with researchers in other departments, such as Dance and Psychology, as well as researchers at other schools such as the University of Illinois, Urbana-Champaign.
This project is supported by NSF grant SCI-0549231.