Fragments boost 3D TV
Technology Research News
January 2001, CBS spiced up its coverage of the Super Bowl with a special
effect that allowed the broadcaster to freeze a replay, arbitrarily change
the viewpoint and continue the replay. Researchers around the world are
looking to take this technology further by enabling viewpoint changes as
the action, including live-action, unfolds, and by letting viewers control
The formidable technical challenge in presenting real-time, free-viewpoint
three-dimensional video is the enormous amount of information contained
in the stream of video information.
Researchers from Swiss Federal Institute of Technology (ETH) in
Zürich have devised a way to process three-dimensional video in real-time
that reduces the amount of data to the manageable level of 3 megabits
The method promises to accelerate the development of three-dimensional
television, three-dimensional tele-immersion systems, and programming
that allows viewers to change viewpoints in real-time while watching televised
The researchers used the technique in their prototype videoconferencing
system Blue-c. The three-dimensional video portal uses a pair of U-shaped
booths that combine real-time image acquisition and projection. The booths'
walls switch rapidly between opaque and transparent in synchronization
with cameras and projectors in order to capture the user's image through
the walls while allowing the user to see images projected on them.
The three-dimensional video processing technique is a "complete
real-time free-viewpoint video system," said Stephan Würmlin, a researcher
at the Swiss Federal Institute of Technology from.
The system could allow images to be rendered on different types
of devices, said Würmlin. "The data representation allows for progressive
transmission and display," he said. "This enables the technology to not
only display images on projection walls but also [on] different devices,
for example handhelds, smart phones or TV screens."
Key to the method is a way to issue updates to a three-dimensional
image by inserting, deleting and updating just the changed portions of
a video frame. These changes from multiple cameras are processed and merged
into a single video stream. This way the three-dimensional geometry of
the image does not have to be recalculated for every frame, which reduces
computational load and network bandwidth consumption, said Würmlin.
The system calculates two-dimensional pixels from multiple cameras
to determine the position, orientation and color of a set of irregular
points in the three-dimensional space. The method represents these dynamic
point samples, or three-dimensional fragments, independently of each other.
Traditional three-dimensional graphics build images out of triangles,
each of which has to be connected to its neighbors. It is difficult to
properly align the triangles of a three-dimensional image produced using
multiple cameras, said Würmlin. Removing the need for connectivity information
allows for more efficient image updating and data compression, he said.
Three-dimensional free-viewpoint video systems have been developed
by researchers at Microsoft, the Max Planck Institute in Germany, the
Massachusetts Institute of Technology and Carnegie Mellon University,
said Würmlin. There is also a free-viewpoint television (FTV) project
at Nagoya University in Japan. They all have limitations that make them
impractical for real-time free-viewpoint three-dimensional video systems,
These systems tend to need a lot of bandwidth -- some an order
of magnitude more than the Swiss researchers' system, which needs an information
flow of 3 megabits per second to provide decent image quality on a projection
screen, said Würmlin.
Some of the three-dimensional video systems only work if the objects
being captured on camera are modeled beforehand, and some use dense arrays
of cameras -- as many as 100 -- in set positions, said Würmlin. "Our system
is capable of capturing arbitrary and multiple objects [and] dealing with
wide baselines -- currently two to three meters -- and arbitrary [camera]
setups," he said.
Three-dimensional video fragments make for a more efficient system
because only a subset of the cameras are used at any one time to capture
three-dimensional information. In the researchers' prototype, any given
frame uses ten of the cameras in the array to capture three-dimensional
information and two to provide for color and texture information.
The system adjusts to network and system congestion by dynamically
scaling back the number of active cameras, trading off image quality for
One challenge in using the system practically is that, though
it reduces the required network bandwidth, the three-dimensional video
stream is sensitive to network errors and needs to be made more resilient,
according to Würmlin.
The method works with any real-time three-dimensional reconstruction
method that extracts depth from images, said Würmlin. It also makes it
easier to process special effects, he said.
The technology has been adopted for the next version of the Animation
Framework Extension (AFX) of the Motion Picture Experts Group (MPEG) 4
standard for interactive and three-dimensional video, said Würmlin. The
extension defines standards for representing and transmitting animations.
"Thus, standardized streams and players are within reach," he said.
The method could be used in practical applications in one to three
years, said Würmlin.
Würmlin's research colleagues were Edouard Lamboray and Markus
Gross. The work appeared in the February, 2004 issue of Computers &
Graphics. The research was funded by ETH Zürich and Mitsubishi Electronics
Research Laboratory (MERL).
Timeline: 1-3 years
TRN Categories: Computer Vision and Image Processing; Human-Computer
Story Type: News
Related Elements: Technical paper, "3D Video fragments:
Dynamic Point Samples for Real-time Free-Viewpoint and Video," Computers
& Graphics, February, 2004.
June 16/23, 2004
Fragments boost 3D TV
Internet ups power grid
Fractals show machine
VR tool re-creates
depth to displays
Genes automate DNA
grown in place
grabs varied specks
Research News Roundup
Research Watch blog
View from the High Ground Q&A
How It Works
News | Blog
Buy an ad link