Market repositioning of Flash begins (updated)

I have already talked (perhaps too much) about the Future of Flash in this post. There I didn’t hide my perplexities about the Market position of Flash compared to alternative technologies. After the drop of Flash Player for Mobile there was a strong decline in confidence for Flash platform. But now the scenario is beginning to emerge sharply and I begin to understand the purpose of the Adobe strategy.

Yesterday Adobe has released a public beta of AIR 3.2 for mobile application development. This version implement the promised support for Stage3D in mobile platforms like iOS and Android. A number of demo video appeared on the web showing excellent 3D performance and a lot of renewed interest about mobile game development using  AIR:

Square Enix’s [Barts] running on Android

The time will tell, but AIR has the potentialities to become a leader platform in 2D/3D games development. A single code base is sufficient to create a game for Desktop (AIR’s captivate runtime), Browser (someone named Facebook ?) and now iOS and Android. With ConnectedTVs and STBs support to come (already showed during MAX), the dream of the Open Screen project is becoming reality, at least in the game dev area (but also intensive graphic/media applications may leverage 2D/3D accelerations).

Therefore Adobe has concentrated the resources in a promising field where Flash could easily become leader. In 2D/3D browser gaming it is just leader (500Million players on Facebook may be sufficient as business card ?) . Try by yourself searching for Stage3D demo in YouTube to see the huge amount of interest for this technology from any game developers (big and small).

The second strong commitment of the platform is for video delivery where Flash has been leader in the past 5 years and is still today. The performance of video decoding in the browser has been widely improved with a completely redesigned pipeline that now exploits mult-threading heavily. But most important, the support for accelerated H.264 streaming has been added to AIR for iOS using the standard Apple HLS (already supported by FMS 4.5 and Wowza Server).

During the spring Adobe will release the new version of Flash Access (now Adobe Access 4) that will include content protection for iOS devices (both in AIR and native application) in the form of DRM on HLS. This move has the potentiality to make Adobe re-gain the favor of majors and big content providers that would have the possibility to uniform DRMs across Android, iOS, Desktop Apps, Browsers, Google TV and some STBs.

The support for HW accelerated 2D, 3D and video playback on mobile, plus an improvement in performance for Flex applications, plus the possibility to integrate HTML5 contents with StageWebView, plus the DRM, plus native extentions, **finally**, makes AIR (for Mobile) an interesting, efficient, effective and valuable solution for cross platform application development.

(Updated 03 March 2012)

I think the platform is 99% complete now, which is very good, but I would like to see the following issues addressed ASAP to complete the features list of AIR for Mobile:

• H.264/AAC on RTMP : necessary for effecient real time video application, especially now that FP supports H.264 encoding.
• Echo cancellation : see the previous point.
• Effective and Robust support for key native features like InApp Purchase and Notification. I like Native Extentions’s idea but I’d prefer an official API for critical features like these.
• Better integration/communication between AS3 and JS in StageWebView. No more hacks please.

Make a comment if you think that there’s something else of important to add to AIR for Mobile/AIR for iOS.

What about the future of Flash ?

Long time passed since my last post on this blog. I have been very busy in an important video streaming project but this is not the only reason for my absence. I have also wanted to wait and take all the necessary time to analyze, ponder and “digest” the infamous Flash affair.
I will not hide my bitterness about the fact, but I’m also more optimitic now, after I have seen the real consequences and have had the time to elaborate on the future scenario. I’ts not all a bed of roses but I’m somewhat optimistic.

First of all, fortunately, I’m not limited to Flash technology in my consultancies. I work with .net technologies for many years and I have designed and deployed successful streaming services in HLS with both Wowza Server and FMS 4.5

You also know that I’m an encoding expert with important success cases and a deep knowledge of commercial and open source encoders like Ffmpeg, x264, Flip Factory, Telestream Vantage, Atheme KFE, Rozhet CarbonCoder, Digital Rapids to name a few.

I have created encoding pipelines and optimized existing ones for delivery platforms based on HLS, Flash HDS, MS Silverlight and ipTV and designed decoding and delivery optimizations for Flash and Silverlight.

So when I talk about my bitterness, it is not driven by the fear for the future but by the awareness of the big mistake that Adobe has done stabbing Flash in the back. I want to focus this post on the future prospectives for Flash and not on the disastrous announcement of Adobe (a masterpiece of masochism, at least from a PR point of view), however a brief summary of my thoughts on the topic is a good thing. I do two short considerations:

1. Adobe may also have had good, long term stategic reasons for dropping Flash for mobile browser, but they could choose modes and terms with much less collateral damages. Why not reduce progressively the commitments and the investments across the lifespan of FP11 to avoid harming the Flash Community ? After all, FP11 has been released for Android and QNX and it has brought important improvements in performance and stability. I know that Flash for mobile browsing has a lot of problems and those problems are due tot the excessive use of bad Flash coding that has been done over  time especially for advertising. Obviously if you have a page with 5-6 Flash banners that can kill an old desktop computer, how can be able a tablet to handle this ?
A simple solution could be to put every swf  of a page in an idle mode, with a clickable poster image that activates the swf  only when touched. Simple, clear and always better than have no Flash support in mobile browsing.

2. Adobe just does not realize that is killing the goose that lays golden eggs. Have you even thought about the fact that Flash is used every day by 2 billion people! It’s probably the most pervasive peace of sofware after MS Windows. Giants like Steve Jobs would have exploited such competitive advantage in ways that the current Adobe management are not even able to imagine. Yet it is not difficult to imagine for example a marketplace of Flash and AIR apps on the model of the MacOS AppStore (but with 20 times more potential customers). What it is worth this kind of power ? Evidently near t0 zero for Adobe.

But now the damage is done and it worth nothing to complain, and so there will be some short, medium and long term consequences. The short term consequences are paradoxically positive for experienced Flash developers. This is because new developers, creative shops and consultancy firms are focusing interest to HTML5 because of the bad medium and long term outlook for the Flash technology and because of  marketing reasons. But the demand for Flash technology is not decrasing as fast as the offer and so there is a burst in the amount of work available for skilled developers.

In a medium term I see an higher convergence between the demand and offer for Flash-based projects in general. Flash will mainain or increase it’s penetration in web gaming thanks to 3D (remember that the casual game market on Internet is completely Flash-centric today, how forget that every day 200+ million people play some Flash games in Facebook ?) and probably will remain the reference for video streaming, but in the RIA market and creative market HTML5 will definitely gain it’s momentum (in real terms, not like now where only a few important creative, video or gaming projects has migrated from Flash to HTML5).

Flash in the mobile market, as a cross platform mobile development technology, has not, in my opinion a clear outlook for the future.  The sudden drop of Flash for mobile browser and the drastic reduction of commitment for Flex has been percepited as a treachery of Adobe from the point of view of the loyal base of sustainers and developers and as a definitive change in the wind from the point of view of customers and stake holders. How to blame them ? the lack of support from its own creator is a mortal stub for a technology and the message from Adobe is clear: in the long term we’ll substitute Flash with HTML5. Not only, we will focus more on tools than technologies (Flex docet).

No place for developers in the future of Adobe ? I don’t know but the long term perspective of Flash, Flex and other Flash related technology (FMS?) has been heavely perturbated by the infamous move. Flex is now an Apache baked project but is it a guarantee of evolution and support ? Who will invest time and credibility among customers in a technology for mobile development that has not a clear commitment from its creator and controller ?

Concluding, what I intends to do as a Flash developer ? In the short term I have to do a lot of Flash related projects, so no problem. In the medium term I think to continue using Flash/AIR for Mobile development. This is a clear path for me, I can capitalize on my AS3,Flash and Flex platform skills to develop desktop, browser and mobile apps. Now the level of features for Android and iOS has become good enough to be able to develop any kind of apps without the need for adding Java and Objective C to your skill portfolio (in my opinion, the recent support for notifications, in app purchase and HLS have cleared the top three entries of the most wanted and needed features list).

And in the long-term ? I dont’ have an answer, I think I’ll simply wait and see.

PS: Very interesting article about “migrating” from Flex to JS (Thanks to Anna Karim) – https://plus.google.com/109047477151984864676/posts/CVGJKLMMehs

My presentation at MAX2011 is available on Adobe TV

Finally the recording of my presentation at MAX2011 (Encoding for performance on multiple devices) is available on Adobe TV.

You can also download the pdf version here. My using of FFmpeg for repurposing the streams of FMS has attracted quite a lot of interest and attention. I’m planning to extend the series of article dedicated to FFmpeg and also to transform it in a permanent knowledge-base on FFmpeg and related best-practices.

Flash Camp in Milan

In a few weeks (20-21 May) will start in Milan a Flash Camp dedicated to Mobile App development with the Flash Platform. The camp is hosted by whymca, a mobile developer conference that covers several mobile platforms and development tools. This year, thanks to the efforts of my friend Andrea Trento, an entire conference track will be dedicated to mobile development using the Flash Platform.

I’m one of the crew (4 Adobe Community Professionals + 1 Evangelist) that will speak at the Camp.

First of all Mihai Corlan (Adobe Evangelist) will open the track talking about Flex on Mobile and the new possibilities for cross platform development offered by the Flash platform. Andrea Trento will show how to create a cross-platform game with Flash, Luca Mezzalira will show the use of Design Patterns in mobile development, Piergiorgio Niero will focus on code optimization for mobile while I’ll speak about video encoding and delivery optimization for mobile.

It will be a great event for learning the cutting edge of technology for mobile development and for keeping in touch with the ever growing flash community.

The event will take place in Milanofiori – Assago – Milan  (more info in the whymca web site), and is completely free. Due to the limited numbers of seats I suggest you to book early your presence.

AIR 2.6 for iOS and video playback

If you work with audio and video streaming, one of the worst limitation of AIR 2.6 for iOS is that it is not possible to stream video encoded in H.264 (and audio in AAC) inside your AIR application. AIR 2.6 for iOS supports NetConnection and NetStream but can decode only Spark, VP6, MP3, NellyMoser and Speex formats. So no H.264 and no AAC (don’t ask me why).

This is a real problem. Who is using today VP6 for video streaming or MP3 for audio ? On top of that, the performance of audio/video streaming is not perfect in AIR for mobile today (even for Android) especially because you have a lot of dropped frames and no frame interpolation, so delivery a stream in VP6 and MP3 for iOS devices is a very sub optimal solution that cannot compete with the very good native streaming capabilities of iOS to which the user is accustomed.

The AIR for iOS documentation mentions that it is possible to launch the native iOS video player pointing to an .mp4 (or .m3u8) video file, but this is not handy because the video is opened outside the AIR application and especially for iPad the user experience is really bad.

Fortunately there’s a sufficiently working solution to integrate in the AIR app the flowless experience of the native player: use the StageWebView object.

The StageWebView object  is a powerful way to integrate the elements that AIR is lacking today. Do you need a list with perfect scrolling ? AAC audio streaming ? H.264 streaming ? Well you can use a StageWebView to load “HTML5” code inside the AIR app and integrate this kind of content. Let’s do a simple example:

var webView = new StageWebView();
webView.stage = this.stage;
webView.viewPort = new Rectangle( 0, 0, stage.stageWidth, stage.stageHeight);
var path:String = new File(new File("app:/html/service.htm").nativePath).url;
webView.loadURL(Path);

This code will open a fullscreen instance of the native browser (without UI) and load a locally stored .html file (notice that it is necessary in iOS to use some hack like that at line 4 to obtain a valid url to access local html). Now you can easily understand that the best is to mix AIR UI and this windowed browser to exploit HTML5 capabilities especially for media streaming. A simple <video> tag inside the html code can do the job and offer perfect H.264 progressive and streaming playback to your AIR apps both in window and at fullscreen.

Communicate with StageWebView

It is not so easy to communicate with the page loaded inside the StageWebView. The object does not provide specifics API. Fortunately exist a class (StageWebViewBridge) developed to overcome the limitations of the standard StageWebView object. With StageWebViewBridge it is possible to communicate bidirectionally with the hosted html page and so create something similar to PhoneGap with Flash AIR.

Waiting for the future AIR 2.7 (which could have yet problems in the video area if it does not implement StageVideo in iOS) this is the best solution I found to overcome the limitations of AIR for iOS.

A Flash Streaming Success Story

I work with Flash Media Server since the very first release (2003) and in the last 8 years I have designed several streaming infrastructures, developed interactive programs, optimizations, services and provided consultancy for companies that wanted to leverage the flexibility and the power of Flash streaming.

In the late 2008 I was contacted as a FMS and streaming expert by Adobe Italy to help Finelco Group replacing Windows Media with Flash technologies for audio streaming.

Finelco Group controls the largest private radio network in Italy with 3 FM national radio stations (Radio 105, RMC, Virgin Radio Italy – over 8 Million daily listeners) and recently a Digital Terrestrial TV channel (Virgin Radio TV).

The decision to migrate from Windows Media to Flash streaming was motivated by the desire of innovate the offer of streaming services. Windows Media, Real Audio, Shoutcast are system that can offer only a fraction of the flexibility of the Flash Streaming Platform while they needed the maximum flexibility to create new services and offer more interaction.

Designing a private infrastructure

Since the beginning they were convinced to set up their own infrastructure for streaming instead that leveraging CDN. I think this has been a key factor for the success of the initiative, because the audience of the streaming services (radio re-broadcasting, webradio, 24h live video and vod) is almost exclusively in Italy and with a bandwidth requirement which has not much spikes a custom infrastructure can offer the best value for money.

I have had the opportunity to work with a  team of excellent professionals like Carmine Montella, Marcello De Palo, Francesco Bruni and Salvatore Sciacco, to name a few. Salvatore has been really precious. He is an excellent system administrator and now extremely expert in FMS tuning especially on Linux.

During this multi-step project we have had the necessity to push FMS to its limit thanks to the high popularity of the services that have been implemented and almost all the sub-systems of the streaming platform have been implemented using Adobe technologies:

1. Adobe Flash Media Streaming Servers for A/V live and vod streaming.
2. Flash Media Interactive Servers with custom server side scripts for stream replication and fail over.
3. Metadata distribution using dedicated data channels in RTMP.
4. System integration with Flash Projectors or AIR to integrate RTMP messaging to legacy systems.
5. Live encoding for audio and video using FMLE.
6. Dynamic webradio using server side playlists.

More recently we have added Wowza Servers to integrate support for iOS mobile devices (using live HTTP) and generic mobile applications (using RTP).
You can find some details about the FMS infrastructure in this old blog post.

The project involved also the development of client side players for audio and video streaming, the use of dynamic streaming, advanced Flash features like smart buffering, client side playlist for pseudo-live streaming and much more.

A really successful story

I helped them to design, deploy and tune one of the largest, privately held infrastructure for audio and video streaming in Italy. Now, using this platform, Finelco is offering the re-broadcasting of the 3 national radio channels plus over 30 webradio channels, 5 web TVs dedicated to music clips and the re-broadcasting of the national DTV channel.  Add a complete series of VOD, special live events and interactive services and you have a picture of the system.

The infrastructure has up to 8 Gbit/s of  bandwidth available. Usually there are 30/40.000 concurrent users that consume over 10 Million hours of audio streaming monthly.
To make a comparison BBC iPlayer delivers 17 Million hours of audio streaming.
Concluding, I think this is definitely a new little-big Flash based success story.

What’s new in Flash Player 11

We all already know about the wonderful 3D API (Molehill) implemented in Flash Player 11. But what about other features ? Reading interviews and blogposts it is possible to derive some informations:

1. JPG-XR

Tinic Uro unveils some details about the support of the JPG-XR image format. It is a modern revision of JPG that improves quality / compression ratios and introduces some new modes useful for texture compression. It could be interesting to discover if encoding images with JPG-XR is more efficient than intra H.264 (read this post about it).

2. New iOS Packager

Not strictly related to FP 11, rumors say that a new iOS packager is on its way to offer better support to Flash Player features (stage video ?) and Molehill. Native 3D on iOS exported by Molehill is said to work in a very early stage.

3. Enhanced Mic, HID, Sound

Matthew Fabb has posted very interesting informations about Flash Player 11 derived from the analysis of the preliminar FP11. There are for example new classes to handle a wide range of human interfaces, sound object extentions and finally a new and complete Enhanced Mic class that support full echo cancellation features with many parametrizations.

4. Something more ?

Personally I hope to see any time soon VP8 encoding / deconding, H.264 encoding, Screen Capture to name a few. I don’t know if we will see such features in FP11, but up to now there are all the premises for a great flash player version.

FlashPlayer 10.2 is coming on Android

Adobe announced today, during the Mobile World Congress, that Flash Player 10.2 will be available soon for Android Devices. This is very important because FP 10.2 introduced the Stage Video object which offers a direct control over hardware acceleration in video deconding. In my opinion the worst point of weekness of FP10.1 for Android is the performance of video deconding so I’m very happy of having Stage Video ASAP.

In Flash Player 10.1 for Android the decoding of H.264 can be hardware accelerated (depending by the HW of the device) but the color conversion, the blending and the compositing of the video on the display is still demanded to a software layer. This is because the canonical Video object is part of the display list and so it injected inside the display list rendering pipeline. Stage Video is a alternative way to access video layer and it is not part of the display object. At the cost of a lower flexibility you have a direct access to harware acceleration, from bitstream decoding to video compositing.

Stage Video will be available only on Android 3.0, this means tablets like Motorola Xoom, Samsung Galaxy Tab 10.1 and so on. The need for full hardware acceleration is much more important for a tablet which has a big screen compared to a smart phone but I suppose we will see even new smart phones equipped with Android 3.0 any time soon.

Now I feel only the need for a new, efficient and accelerated iOS packager…

Flash to HTM5 fallback

After the debate around what it’s better at serving video between Flash and HTML5 I have found a lot of code in the Internet on how to serve video in HTML5 with a Flash fall back.

Independently by the fact that the debate is far from being concluded, I think that it’s a nonsense to check first for HTML5 and then for Flash instead of the contrary. Let’s follow this simple reasoning:

Today you have Flash Player in 98% of the desktops and on Android 2.2. Even excluding the fact that in a few months it will be available on RIM OS, Web OS, Windows Phone 7, Samsung and Sony connected TVs and Google TVs, the penetration of Flash Player is very very high because the computer’s desktop is still the most important place where Internet video are consumed.

The performance of Flash on these platforms is very good and I’m not speaking only about pure Fps or CPU usage statistics, but overall performance including video control, personalization, easyness of development, advertising compatibility, content protection and QoS. And now with StageVideo API the performance is very very good on Mac too.

On the other hand you have HTML5 which is a compact, standard and effective technology for video… Sorry, I correct myself: It WILL BE standard and effective because now it is a mess with every vendor implementing (or anticipating) the standard in its own way.  It works well on Mobile, especially on  iOS devices (even if the implementation does not adhere completely to the ‘draft’ standard), but on the desktop (over a Billion screens) it simply doesn’t work today, at least for video delivery.

The scenario is clear: On Desktop we have 50% of IE (sub 9) browsers who do not support HTML5 at all. 35% of Firefox browsers who support at the moment only Theora for video. Only Chrome and Safari seem to work flawlessly with HTML5, but yesterday Google has complicated everything announcing that Chrome will drop the support to H.264 in favor of VP8.  It is definitely not a good scenario…

Therefore, I wonder, why should I serve my video first to HTML5 and THEN to Flash ? that would be crazy.

Use HTML5 as a Flash fall-back

You know that I develop video encoding pipelines as well as video delivery platforms and optimizations. So for my clients I propose Flash based players (custom or OSMF based). With Flash, not only they have the widest audience possible, but even a great and consistent interactive experience. Furthermore, one very important point for top media companies is content protection, and with Flash you can protect content in several ways. This is not possible today using HTML5 (indeed it’s also not possible to stream, you have only progressive dowloading), only iOS devices support a not standard streaming and protection schema.

But the new “Mobile Wave” cannot be ignored. Everyone want to watch video on iPhone, iPad or Android. And also BlackBerry, Symbian and Windows Mobile devices need to be served in some way.

So, what I propose to my clients ?  The following Fall-Back approach:

IF The device has Flash (Desktop + Android2.2) THEN
…stream using Flash with Dynamic Streaming / Interactivity / Protection.
ELSE
……IF The device supports HTML5 (iOS, Android, etc…) THEN
………stream (on iOS) or progressive download (on Android and other webKit based browsers).
……ELSE
………serve video in a low end format for other devices in progressive downloading.

This kind of multiple fall-back schema can be achieved in several way. I like to use a compact schema, easily applicable for video that can be embedded in external pages. The example below uses the fall back feature of the HTML tag <object> and HTML5 tag <video>:

<object id="VideoPlayer" classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000" codebase="http://download.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=10,0,0,0" width="600" height="400" >
 <param name="movie" value="videoPlayer.swf?videoID=10" />
 <param name="quality" value="high" />
 <param name="allowfullscreen" value="true" />
 <param name="allowscriptaccess" value="always" />
 <param name="wmode" value="opaque">
 <object type="application/x-shockwave-flash" data="videoPlayer.swf?videoID=10" width="600" height="400" >
 <param name="movie" value="videoPlayer.swf?videoID=10" />
 <param name="quality" value="high" />
 <param name="allowfullscreen" value="true" />
 <param name="allowscriptaccess" value="always" />
 <param name="wmode" value="opaque">
 <video controls preload width="600" height="400" poster="thumbnail.jpg" >
 <source src="iPhoneVersion.m3u8" type="video/mp4" />
 <source src="AndroidVersion.mp4" type="video/mp4" />
 <a href="LowEndMobileVersion.3gp">
 <img src="thumbnail.jpg" /></a>
 </video>
 </object>
</object>

First of all I have used two nested <object> tag which is one of the techinque for embedding the Flash Player in an HTML page. The first is for IE, the second (in fall back) is for FF, Chrome and the others (Change videoPlayer.swf?videoID=10 to your video player of choice). If even this second tag fails you are probably on a mobile device or on a tablet so it’s logic to try the HTML5 <video> tag which works in IOS devices, Android and other OS with webkit based mobile browsers. The video tag allows to specify a list of possible video sources and tries to access them in sequence. Here I propose at first the Apple HTTP Live streaming technique serving the iPhoneVersion .m3u8 manifest file and only if this is not supported by the device I fall back to a progressive downloaded video in .mp4 format called AndroidVersion.mp4 (for Android, WebOS, RIM OS6, etc…). If even the <video> tag fails you are probably on a early version of Android, Symbian, BlackBerry or Windows Mobile so a low end version of video is served in progressive downloading (LowEndMobileVersion.3gp).

This approach guarantee the best performance on desktop (Flash Player, streaming with bitrate switching, advanced QoS and protection) and on iOS (Apple HTTP Live streaming with bitrate switching and protection) where this schema works for both live and ondemand video. On the rest of the devices you can still serve on demand video without protection. It’s your video platform that must selectively built the right fall-back schema to exclude the lower version of video when it must be protected or if it is a live streaming.

H.264 Encoding Strategies for all screens is on AdobeTV

The recording of my presentation at MAX2010 is finally on Adobe TV. You can find it here (Notice that there’s something wrong in the recording, so you have to skip the first 15 minutes).

Here you can find also the .pdf file.