Virtual cameras in among the action

There are often 30 or more cameras covering a big soccer match, but they don't always provide the perfect view.

It might be nice to see what the referee sees (or misses…), or see what is happening from the perspective of your favourite player. This is becoming possible, thanks to virtual cameras, according to Johan De Geyter, manager of the imec Vision Technology group.

Its iVC technology (imec Virtual Camera), which will be on show at IBC (Stand 13.A21), allows an arbitrary camera position to be created, synchronic with the real camera images, in real time, and to fly it about like a crane camera. It can even be used to create 3D video.

"IMEC's multimedia group has a lot of expertise, but it usually comes down to only bits and pieces. This is because companies call in our help to solve the hardest problems. We worked for example on stereo matching for video telephone applications, feature detection to distinguish trucks from cars in tunnels. Within the virtual camera we have assembled different expertises to demonstrate that all these technology pieces are capable of quite some things," said De Geyter.

Working with Outside Broadcast, Belgium, it installed five cameras at a match to shoot the demonstration shown in its video. "The images of these cameras are being assembled, after which computer-generated high resolution images are being created, as from an arbitrary virtual camera position. For the demo set-up, we were still working with standard broadcasting cameras, but our aim is to make use of small ip-cameras, which may for instance be plugged in at fixed positions in a soccer stadium.  The installation of about 30 small cameras can show you images of the whole soccer field. For example, if you would like to have a camera image taken from in the centre spot, this will be possible without having to install a real physical camera at this spot."

In theory viewers could select their own virtual camera position. "But let’s be honest: the typical soccer game viewer already has his hands full (with potato crisps and beer) – I do not think he is willing to handle an extra joystick to change the camera position. Our technology is meant for the professional market, for producers that want to make good TV. We could indeed make different versions of one broadcast, for example a version for the Manchester United fan and one for the Real Madrid fan, both filmed from the point of view of their favourite team. "

Applications extend beyond soccer. "Imagine you can put the camera on the shoulder of the pitcher at a baseball game. At a concert you would be able to portray the whole stage without the camera crew disturbing the artist. The virtual camera technology would also be ideal for cooking shows. Today, it is apparently quite a challenge to shoot all from the right angle in cooking shows. And you have only one chance to shoot the right images, because nothing can be repeated, of course. At political interviews it seems also interesting to me to work with a free camera position. For example, when filming the US presidential candidates you need to install the cameras at quite some distance in order not to disturb them during the debate.”

It can be used to create moving cameras where you have none. It can even be used to create a 3D view, slow motion replays, do speed or offside analysis on players or objects, or to highlight the movement of an object (the flight of the ball, for example) or player (without the other players appearing in shot).

Plane sweeping

The image interpolation for foreground objects is based on plane sweeping - which casts rays from the virtual camera onto the scene and projects points at different depths on this ray into the images of the physical cameras. The projection depth that results in the best colour match over the different cameras is assumed to point to the real 3D location where a foreground object with that matching colour is located.

Since this generic plane sweeping procedure and matching process is very difficult, the iVC prototype technology explicitly exploits some of the characteristics of a football game: the dimensions of the field are known, and it is filled with relatively small and rapidly moving players and a ball on top of a mostly fixed, planar background. This application domain can easily be extended to other sports. iVC exploits these characteristics of the football game through a modified depth box constrained plane sweeping algorithm.

The analysis information gained from this algorithm also allows it to be used as telestrator (player marking, offside lines, running trail), provide game statistics (running speed) and create digital signage.

Before the virtual shots can be processed, the real cameras must go through a one-time calibration, which is fairly simple to do, although it has to be much more accurate than for telestrator applications.

While processing, iVC continuously detects and tracks the exact position of every player, the ball, and referee.

Once the virtual camera is positioned, the viewing angle is selected, and the speed of the input streams are controlled (live, slow-motion, still), iVC generates an HD video stream, based on these parameters and using the depth box constrained plane sweeping algorithm.

This technique only works for objects visible in at least two cameras, so the more cameras there are, the better (see illustration above right  - click for larger view).



By David Fox