Real-time gaze-contingent spatio-temporal filtering for gaze guidance

Michael Dorr, Sönke Ludwig, and Erhardt Barth

We present a novel display that spatio-temporally filters a movie as a function of gaze. For each pixel, filter parameters can be specified in retinal coordinates. Perry and Geisler were the first to simulate arbitrary visual fields by creating a spatial Gaussian multi-resolution pyramid; at each pixel of the  output image, two adjacent pyramid levels were interpolated to obtain a lowpass-filtered version of the input pixel.

We now extend this idea to the temporal domain. Furthermore, our display is based on a Laplacian instead of a Gaussian pyramid, so that not only a cutoff frequency can be specified for each pixel, but a full set of weights for individual spatio-temporal frequency bands.

Because of the high computational cost of upsampling all levels of the pyramid to full spatio-temporal resolution in each frame, we implemented our algorithm on the Graphics Processing Unit using the Cg shader language. We achieve real-time performance (>30 fps) on HDTV (1280x720 pixels) videos.

We aim to use our gaze-contingent display to selectively suppress and enhance spatio-temporal content as a function of current gaze. By changing the saliency distribution of a movie in real time, we intend to guide the observer's gaze such as to improve visual communication (see http://www.gazecom.eu).