Flash + H.264 = H.264 Squared – Part III

In the last post I have explained my vision about an ultra optimized video encoding workflow with video enhancements done inside the Flash Player at run-time.

I want to conclude this serie with some additional informations on how to restore video details and film grain.

Details restoration

Flash Player has the possibility to apply predefined filters since the version 8 (es: emboss, dropshadow, etc..). Not only, it also supports the definition of custom filters using two different techniques: convolution matrices or Pixel Bender.

Pixel Bender is a rather complex topic because you need to learn a dedicated, yet simple, language to manipulate pixels during filtering iterations. Therefore I’ll concentrate on the more simple convolution method.

A convolution matrix is a mathematical object (aka “Kernel”) used to alter a pixel depending by the value of the surrounding pixels. Usually 3×3 or 5×5 matrices are used.

This is an example of a strong sharpness kernel:

What do you need to implement that filter in AS3 ? It is indeed really simple, use this code:

// "video" is your video object istance
var filter = new flash.filters.ConvolutionFilter();
filter.matrixX = 3;
filter.matrixY = 3;
filter.matrix = [-1, 0, -1, 0, 8, 0, -1, 0, -1];
filter.bias =  0;
filter.divisor = 4;
video.filters=[filter];

Film Grain restoration

One of the more interesting thing of my last experiment is the generation of a credible video noise. During strong compression, video noise (film grain) is strongly reduced. Adding a “synthetic” grain can enhance efficiently the perceived quality of the final video. Indeed this approach is also defined in one minor annex of H.264 standard but it has been rarely implemented in commercial video players.

Creating a credible video noise is not an easy task. I started with pre-rendered noise clips to overlay in trasparency over the video but this approach was not good. Then I tried to implement a noise generator in Pixel Bender but it had poor performance.  Finally I found a simple and self contained method. It uses the perlin noise generator built inside the Flash Player.

Perlin Noise is very different from a random noise (the type we need), but with a proper parametrization it can become very similar. The problem is that it is practically impossible to generate continuosly “noise frames” because the calculations for perling noise are very intensive. So I used a pre-rendering approach. During the player initialization I pre-calculate a number of “noise frame” and store them in separate bitmap objects. During the playback the frames are overlayed on the video in sequence to create the film grain effect. Take a look at the code:

// Init

var baseX:Number = 1.3;
var baseY:Number = 1.3;
var numOctaves:Number = 1;
var stitch:Boolean = true;
var fractalNoise:Boolean = true;
var channelOptions:Number = BitmapDataChannel.BLUE|BitmapDataChannel.RED|BitmapDataChannel.GREEN
var grayScale:Boolean=false;
var offsets:Array = new Array(new Point(), new Point());
var bitmapArray:Array = new Array();
var frameNumber:Number = 12;

for (var i = 0; i<frameNumber;i++) {
  var bmpData:BitmapData = new BitmapData(1280,720);  
  var bmp:Bitmap = new Bitmap(bmpData);
  bmp.alpha=0.20;
  bmp.blendMode="overlay";
  bmpData.perlinNoise(baseX, baseY, numOctaves, Math.random()*65000,stitch, fractalNoise, channelOptions, grayScale, offsets);
  bmp.visible=false;
  bitmapArray.push(bmp);
}

// Noise video sequence

setInterval(alternate,80);
var altcnt=0;

function alternate() {
  altcnt++; if (altcnt>=frameNumber){altcnt=0};
  for(var i=0;i<frameNumber;i++){bitmapArray[i].visible=false;}
  bitmapArray[altcnt].visible=true;
  bitmapArray[altcnt].alpha=(0.14+0.06*Math.random())*1.5;
}

Notice that every noise frame has an alpha value that is generate randomly around a target value. Doing that the final effect is less repetitive and less predictable.

Performance

It is not an easy task to apply a convolution matrix on a 1280×720 H.264 video and add in overlay a synthetic noise. It requires a lot of processing power and so it cannot be used in any scenario.

For example it is impossible today to use it on mobile, and on desktops you need to check carefully for the performance of the computer and disable selectively the effects to save processing power if the cpu is under pressure. I will speak about how to effectively measure video performance in a future post. In the next release of Flash Player we will be able, probably, to leverage GPU compositing and acceleration in a more deep way. Video Enhancement is a perfect way of exploiting GPU acceleration, so I’m looking forward for a GPU accelerated version of Pixel Bender (in the Flash Player, outside it is already accelerated).

In the while let me know about your experience with the enhanced video, reporting computer specs and performance. In my experience it works well on a Core2 Duo 2GHz+ on Windows (for Mac its better to upgrade to the latest accelerated version of Flash Player).