Flash + H.264 = H.264 Squared – Part III

In the last post I have explained my vision about an ultra optimized video encoding workflow with video enhancements done inside the Flash Player at run-time.

I want to conclude this serie with some additional informations on how to restore video details and film grain.

Details restoration

Flash Player has the possibility to apply predefined filters since the version 8 (es: emboss, dropshadow, etc..). Not only, it also supports the definition of custom filters using two different techniques: convolution matrices or Pixel Bender.

Pixel Bender is a rather complex topic because you need to learn a dedicated, yet simple, language to manipulate pixels during filtering iterations. Therefore I’ll concentrate on the more simple convolution method.

A convolution matrix is a mathematical object (aka “Kernel”) used to alter a pixel depending by the value of the surrounding pixels. Usually 3×3 or 5×5 matrices are used.

This is an example of a strong sharpness kernel:

What do you need to implement that filter in AS3 ? It is indeed really simple, use this code:

// "video" is your video object istance
var filter = new flash.filters.ConvolutionFilter();
filter.matrixX = 3;
filter.matrixY = 3;
filter.matrix = [-1, 0, -1, 0, 8, 0, -1, 0, -1];
filter.bias =  0;
filter.divisor = 4;

Film Grain restoration

One of the more interesting thing of my last experiment is the generation of a credible video noise. During strong compression, video noise (film grain) is strongly reduced. Adding a “synthetic” grain can enhance efficiently the perceived quality of the final video. Indeed this approach is also defined in one minor annex of H.264 standard but it has been rarely implemented in commercial video players.

Creating a credible video noise is not an easy task. I started with pre-rendered noise clips to overlay in trasparency over the video but this approach was not good. Then I tried to implement a noise generator in Pixel Bender but it had poor performance.  Finally I found a simple and self contained method. It uses the perlin noise generator built inside the Flash Player.

Perlin Noise is very different from a random noise (the type we need), but with a proper parametrization it can become very similar. The problem is that it is practically impossible to generate continuosly “noise frames” because the calculations for perling noise are very intensive. So I used a pre-rendering approach. During the player initialization I pre-calculate a number of “noise frame” and store them in separate bitmap objects. During the playback the frames are overlayed on the video in sequence to create the film grain effect. Take a look at the code:

// Init

var baseX:Number = 1.3;
var baseY:Number = 1.3;
var numOctaves:Number = 1;
var stitch:Boolean = true;
var fractalNoise:Boolean = true;
var channelOptions:Number = BitmapDataChannel.BLUE|BitmapDataChannel.RED|BitmapDataChannel.GREEN
var grayScale:Boolean=false;
var offsets:Array = new Array(new Point(), new Point());
var bitmapArray:Array = new Array();
var frameNumber:Number = 12;

for (var i = 0; i<frameNumber;i++) {
  var bmpData:BitmapData = new BitmapData(1280,720);  
  var bmp:Bitmap = new Bitmap(bmpData);
  bmpData.perlinNoise(baseX, baseY, numOctaves, Math.random()*65000,stitch, fractalNoise, channelOptions, grayScale, offsets);

// Noise video sequence

var altcnt=0;

function alternate() {
  altcnt++; if (altcnt>=frameNumber){altcnt=0};
  for(var i=0;i<frameNumber;i++){bitmapArray[i].visible=false;}

Notice that every noise frame has an alpha value that is generate randomly around a target value. Doing that the final effect is less repetitive and less predictable.


It is not an easy task to apply a convolution matrix on a 1280×720 H.264 video and add in overlay a synthetic noise. It requires a lot of processing power and so it cannot be used in any scenario.

For example it is impossible today to use it on mobile, and on desktops you need to check carefully for the performance of the computer and disable selectively the effects to save processing power if the cpu is under pressure. I will speak about how to effectively measure video performance in a future post. In the next release of Flash Player we will be able, probably, to leverage GPU compositing and acceleration in a more deep way. Video Enhancement is a perfect way of exploiting GPU acceleration, so I’m looking forward for a GPU accelerated version of Pixel Bender (in the Flash Player, outside it is already accelerated).

In the while let me know about your experience with the enhanced video, reporting computer specs and performance. In my experience it works well on a Core2 Duo 2GHz+ on Windows (for Mac its better to upgrade to the latest accelerated version of Flash Player).

18 thoughts on “Flash + H.264 = H.264 Squared – Part III

  1. What a great article is here, I have to try it later today and see how it works on my video clips. THX Fabio for your genuine sharing.

  2. Fabio

    Great set of articles. I had been hanging out for a more in depth look ever since you posted your first examples way back in 2008!

    Core2 Duo @2.5Ghz using Win7 + Chrome + FP 10,1,85,3 + Nvidia gpu + non-fullscreen = 35% CPU

  3. Fabio, tested with a MacBook Pro Intel Core 2 Duo 2,4 GHz in Firefox 4 and latest flash player I have around 70-80% use of CPU
    Talking about the video – most of the time it is important what kind of quality you get to compress – with h264 tweaking and filters you can do a lot. On the other side still impressing what you can “save” with the right settings of your video if you do not use basic compression templates and presets.
    As you mentioned GPU acceleration is the next thing we should ask the industry for to implement it in mobile devices and others.

    1. I hope to see a strong improvement on Mac with GPU acceleration. It is a pity to have the same video running with 35% CPU in Win 7 and 70% CPU in OS X. Adobe should show something new at Max about this issues.

  4. What great info! I’ve been following your methods for over a year now and wanted to say a huge thank you for all the help you have provided.

    One small question about your code above for Film Grain restoration. In the line that has the loop for setting all BitmapData visibility values to false is repeatedly setting for bitmapArray[altcnt].visible = false. Shouldn’t it set bitmapArray[i].visible = false?

    Thanks again for all your revolutionary work in streaming video!

  5. I’ve got a little doubt (sorry, maybe it’s a silly question, but I’m not very practical in actionscript so far…). What is the video format we use during the process? I suppose we take a video from a dvd, encode it into (let’s say) mpeg or hi-quality avi, denoise, resize with some tool, then import it in actionscript, apply the filters and then finally export to flv. Is it right? I’m a little bit confused, sorry…

  6. I’m sorry, but I’m a little bit confused about the processing flow. I suppose we start with a (let’s say) mpeg/avi video, denoise, resize and so on, and then import it into actionscript, apply convolutional filtering and finally export to flv. Is it correct? (Sorry for the silly question, but I’m not very practical in actionscript). Is it possible to set any parameters when exporting (i.e. flv bitrate)? Thank you for your help.

  7. Thank you for the answer. So, if I’ve understood correctly, your aim is not to create a low bitrate video with high quality appearance, but to take an encoded video at low bitrate and create a way of playing to enhance its quality. Is it correct?

  8. The mp4 video file(super-250.mp4) average bitrate is (2,896,852 bytes * 8 bits) / 84 seconds = 275,890 bits/second, but I think it is more correct to consider the length of video as 74 seconds because there is 5 seconds at start and end of video with just black background which just take tiny bits of less than 1KB for each second! so the effective average bitrate is 313,317 bits/second but it is still good low bitrate.

    Opening http://www.progettosinergia.com/flashvideo/advancedpostprocessing.html in firefox with flash player 10.1 r53 show animated white and blue dots in black areas in the video but strangely I could not take a PrintScreen to send you a screenshot because those dots get disappear as soon as I save the screenshot image in windows paint. also there is much grain that affects quality badly comparing the same video played in VLC.

    I think it is just x264 that could give such low bitrate for that quality. could you publish the command line you used to encode the original video?


  9. … impressive beyond belief, makes me realize my place in the pecking order of vid know how, ie “lamer”. Best wishes for your continuing success in pushing the envelope!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s