The inventors have developed a new compact image pyramid representation that can be used for real-time, high-quality phase-based video magnification and in other video processing applications.
Many physical phenomena exhibit minute motions that, when recorded, require computational amplification to be perceived by the naked eye. Current phase-based video magnification techniques use complex steerable pyramid decompositions of images that are overcomplete and costly to construct, making them slow and non-trivial to implement in real time. The Inventors propose a method for real-time, phase-based video magnification that uses the Riesz pyramid, a novel image pyramid representation suitable for Eulerian phase-based video magnification. Its motion magnified videos are of comparable quality to those produce by complex steerable pyramids but can be processed in a quarter of the time, making it more suitable for real-time or online processing.
Phase-based video magnification relies on being able to manipulate the local phase in image sub-bands. Riesz pyramid breaks the input image into non-oriented sub-bands using an efficient, invertible replacement for the Laplacian pyramid and then takes an approximate Riesz transform of each band. The Riesz transform is the natural rotation-invariant, 2D transform that allows for phase analysis without having to break the image sub-bands into multiple orientations. Motion processing with the Riesz transform occurs by phase-shifting an input signal along its dominant orientation to produce motion magnification. Temporal filtering and phase denoising is conducted to sharpen the signal and improve the results.
Additionally, processing is done entirely in the spatial domain. This gives an efficient and easy way to avoid the spatial wrap-around artifacts present in the frequency domain implementation of complex steerable pyramids. Building and collapsing the Riesz pyramid is efficiently implemented because of shared computation between bands, symmetry of the filters and because the Riesz transform is approximated by two three tap finite difference filters. Phase-based video magnification using the Riesz representation allows the production of high-quality motion-magnified videos in real time.
- Representation is less over complete than complex steerable pyramid approach
- Supports real-time phase-based video magnification at a fraction of the processing time of existing methods
- Reisz pyramids can be implemented using compact, efficient linear filters in the spatial domain
- Riesz pyramid phase-shifts image features along their dominant orientation, rather than every orientation like the complex steerable pyramid, for improved video magnification