I'll start with a simple visual effect: Radial Blur, which can give images the 200mph speed zoom look. In a naive approach you could achieve this effect by replacing every pixel with a weighted average of all neighboring pixels within the blur radius that lie on a line that goes through the center of zoom and the pixel itself. The problem with this method is that it is very slow since it would require many calculations to collect each neighbor and since the angle is different at every pixel there is not much room for other optimizations, like reusing already read pixels, which is key for every fast blur algorithm.

But using a neat trick that I remembered from my early Photoshop days allows to speed up this process a lot. Here is a demo:

The way this works is by converting the pixels from cartesian mapping (where the axes are x & y) to polar mapping (with axes angle & distance to the center) which arranges them in a way that all the neighbors that need to be averaged arrange themselves nicely in a straight line. This allows us to use a simple native Blur filter to quickly calculate the average. Depending if the blur is in x direction or y direction the final result will either be a Radial Blur (the well-known zoom effect) or a Circular Blur (which looks like a fast spinning wheel). The final step is to convert the pixels back from polar mapping to cartesian.

Open this link in a new window to see the process in a step by step demo.

By default the mapping uses the center of the image as the center of the polar coordinate system. This will result the center of the zoom to also be in the center of the image. So what if we want the zoom center to be at a different location? There are two ways of going about this - unfortunately I did not manage the faster way to work correctly yet which would be to simply use that center in the mapping algorithm. For some reason I have not found the right equations which work both ways. So for now I am using the slower method which does always work:

The trick here is to first make the image bigger by padding the borders so that the desired center of the zoom is temporarily the center of the image. So if you want the zoom center to be in the upper half of the image you add as many border pixels to the top that the chosen spot becomes the new vertical center. Then follows the cartesian-to-polar transform, the linar blur and the polar-to-cartesian mapping. The final step is to crop the image back to its original size by removing the previously added top pixels.

Another issue are artifacts that show due to the mapping. The problem is that since all the pixels that lie on a circle with a certain radius are mapped to a straight line, once the circumference of that circle gets bigger than the width of the mapped image pixel information gets lost. To fix this it is required to make the bitmap that holds the polar map as wide as the biggest possible circumference. Unfortunately that is not always possible since the temporary bitmaps could get pretty huge this way. The good news is that since we are blurring those artifacts will mostly not become too visible. Only for small blurs it improves the quality when using a bigger resolution for the cartesian image map.

You can find the source code for the RadialBlur class and the required pixel bender kernels as usual inside QuasimondoLibs

Well, here is a little alternative method: whilst working with the YUV colorspace I figured that since Y contains all the luminance information, the U + V channels must thus contain the hue and saturation. And it turns out that indeed when looking at u and v as the coordinates of a vector, its angle will represent the hue and its length will be the saturation.

So a formula that does not need any min/max and ifs to convert rgb to hsl looks like this:

// r,b and b are assumed to be in the range 0...1
luminance =  r * 0.299 + g * 0.587 + b * 0.114;
u = - r * 0.1471376975169300226 - g * 0.2888623024830699774 + b * 0.436;
v =   r * 0.615 - g * 0.514985734664764622 - b * 0.100014265335235378;
hue = atan( v, u );
saturation = Math.sqrt( u*u + v*v );

In this case hue will be between -Pi and Pi and saturation will be between 0 and 1/sqrt(2), so you might want to multiply the saturation by sqrt(2) to get it in a range between 0 and 1.

Of course this also works the other way round - hsl to rgb looks like this:

// hue is an angle in radians (-Pi...Pi)
// for saturation the range 0...1/sqrt(2) equals 0% ... 100%
// luminance is in the range 0...1
u = cos( hue ) * saturation;
v = sin( hue ) * saturation;
r =  luminance  + 1.139837398373983740  * v;
g = luminance  - 0.3946517043589703515  * u - 0.5805986066674976801 * v;
b = luminance + 2.03211091743119266 * u;

And here is a demo of a Pixel Bender kernel that allows you to adjust hue, saturation and brightness of a bitmap using this technique: HSL Adjust Demo, Source: hsladjust.zip

[Edit]As it was correctly noticed in the comments, the values that this method returns are not the HSL values that you get when you are using the classic formula. Nevertheless I think that this method has its uses, for example if you want to quickly create color schemes or if you are using it like in the demo to change the hue/saturation/Luminance of a photo.[/Edit]

[Edit]I updated the factors in the matrix one more time with values I found in the Wikipedia YUV discussion page. Those look better to me at least.[edit]

So even though I think that compared to the capabilities of the Flash Player Canvas still lacks a lot, it cannot be ignored since it is now available on most major browsers. As a Flasher I find it especially entertaining to watch how history keeps repeating - browsing Canvas community blogs is like time traveling: plasma effects, l-systems, bitmap manipulations, physics engines, vector drawing experiments, emulators - in short everything that we were excited and enthusiastic about years ago is now being rediscovered, ported or reinvented on Canvas, just like we rediscovered, ported or reinvented things that had been done in Java or C years before Flash. But hey, I actually envy these guys - there is nothing more rewarding and creativity boosting than trying to push a limited platform to the maximum.

So since we are not so different after all, here's a little contribution to world platform peace - Pibeca allows you to use pixel shaders on Canvas Bitmaps. It achieves that by using Adobe® Pixel Bender™ filters which can be written with the freely available Pixel Bender™ toolkit. Those kernels are very small programs who's only purpose in life it is to push pixels around very fast.

So here is an proof of concept which should work on any browser that supports canvas:

Under the hood a canvas bitmap and a kernel url is being sent to an invisible SWF which loads the Pixel Bender filter kernel, processes the bitmap and then sends it back to JavaScript which passes it back to a canvas. That's all. Given that I have to encode the bitmap to a string in order to get it to the other side and back the speed is pretty okay I think.

I wanted to get this prototype out quickly, so until I have prepared some docs and more examples those who are adventurous can already have a look at QuasimondoLibs and check out the Actionscript and Javascript source code for this demo. Pibeca is released under MIT License.

The motion of a particle field is controlled by the shape of the random clusters they form among each other. The orientation of the clusters control each particle's further motion which leads to all kind of interesting feedback effects and emergent patterns. The core algorithm which calculates the overall alignment and orientation of the clusters is written in Pixel Bender.

What a median filter does is to take all the pixels within a certain radius around a pixel, sort them numerically ( channel by channel) and then return the color that ends up in the middle of the sorted list. You can imagine that doing this for every pixel in an image is quite a processing effort. Fortunately people much smarter than me have found ways to speed this up considerably. The paper I had stumbled across sometimes in the past was this one by Simon Perreault and Patrick Hebert. And whilst the authors even provided the C source for this filter I did not bother to look at it since I thought that things that are fast in C are not necessarily fast in Flash and decided to roll my own version of the algorithm based on maintaining the histograms and the kernel in the form of sorted linked lists. I figured that this would speed up the process of joining and separating the histograms since I could use a kind of "zipper" technique to do that. To my astonishment the first tests were extremely slow - even though I tested the speed with a release version (testing this with the debug version is about 10 times slower).

The breakthrough came with a tip Ralph Hauwert gave me: whenever you create a linked list class in Actionscript make sure that you use the "final" keyword for it. Until then I had thought that this keyword acted more as a decorative element without real impact. Well, I'm happy that I was wrong since the speedup in this case was extraordinary.

When you compare my implementation with Eugene's note that a kernel size of 7x7 in his example is the equivalent of a radius of 3 in mine (3+3+1=7), 17x17 is a radius of 8 (8+8+1=17). As far as I can see my version is faster until a radius of 7 but after that his version surpasses mine. I have to study his code now to see how he has solved it.

Since the median filter still takes considerable rendering time I added a "asynchronous" switch which allows to render the effect in background without stalling the rest of the application. Of course this will slow down the whole rendering process.

Here is my Median example
sourcecode and zipped sources

In practice it turns out though that this value is much smaller, the reason for it I learned from Joa: memory fragmentation does have a big impact on how big the chunks of memory can be that you can allocate at a certain time. So while you might be able to create 4 bitmaps of 10 MB each you might not be able to create a single one of 40MB.

But being limited to "just" 1.5GB is unfortunately not the end of the story. It shows that in most browsers (like FireFox 3.5.2) these ca. 1.5GB are not available to every SWF individually. No, all SWFs in all open tabs or windows of Firefox are sharing that one Pool of 1.5GB. That means the more one SWF requests the less is available for the others.

So in order that you can figure out what is happening yourself I've built MemoryHog, a tiny tool which will allocate and release memory. When you open multiple instances of it in your browser of choice you can see how memory is managed between all the open SWFs. A warning though: some browsers in some systems do crash when you press the "request maximum" button. Seems like Macs are more susceptible to this than PCs.

One thing you can see very good with MemoryHog is the influence of memory fragmentation. Sometimes whilst you cannot allocate a 100MB block anymore you are still able to get more than 10 10MB blocks.

My resume for Aviary: whilst it it nice to be able to finally use bigger bitmaps in Flash player 10 when it comes to using more than a few of them at the same time we will have to think about a implementing a tile based solution that makes better use of the fragmenting memory.

I've had a look at my presentation demos and repackaged their code into a single class called ThresholdBitmap. This is a BitmapData class with a few extras that help you if you plan to extract blobs or edges from a camera stream or some other bitmap.

The general problem when you do camera based experiments is that you have unnown lighting conditions. Using a simple fixed threshold might work at your development machine but might fail completely on a user's computer that has a different camera model or a different backdrop.

In image processing there are several automated thresholding methods, that are supposed to help with this problem, by looking at an image's histogram and adjusting the threshold to a level that separates the foreground from the background in an optimum way: Moment-Preservation, Maximum Entropy, Discriminant and Otsu. Those are quite capable in finding the optimum threshold level if you have relatively uniform lighting over the whole scene.

As an (better) alternative I included an adaptive threshold method which is implemented in PixelBender and which works very well also with uneven lighting, especially if you try to detect QR codes or other markers.

Furthermore the class includes a very fast edge detection (based on the thresholded image) which gives you nice 1 pixel wide edges in most cases.

Here is a demo that allows you to play around (you'll need a webcam)

Sourceview and code can be found here

Credits: the demo uses the super useful Minimal Components by Keith Peters and the Hi.RES! Stats by Mr. Doob.

Maybe I've searched not thoroughly enough, but I didn't find any solution for this out there. So I've written a tiny class that fixes it. Here's a comparison of the difference between matrix.createGradientBox() and my new GradientMatrix.getGradientBox()

To use it you can use the same values that you would use in the native function, the only difference is that you don't have to create a new Matrix first. The syntax is var myGradientMatrix:Matrix = GradientMatrix.getGradientBox( width, height, rotation, tx, ty);

Download GradientMatrix.as here

Check out Manic Mirror at the Incubator

So I had a look at the source code and found one feature which I wanted to change: face detection is based on so-called Haar Cascades, simplified one could say that this is a very long list of zones that get checked for certain features in an image. In the orginal version this is an xml file of almost 1 MB size. Because this is so big the xml file gets compressed into a zip file where it fortunately becomes just 100K. So in the orginal version the zip file has first to be loaded and then unpacked and parsed. My consideration was that swfs get zipped anyway, so why not turn the whole xml file directly into a class?

So that's what I did. My adapted version does not need to load any external files anymore (which was my goal) and the whole swf becomes even about 50K smaller than the original version. I also did a few other optimizations like replacing Arrays with linked lists and reading a from ByteArray instead of using getPixel(). The downside is that since the Haar data is being hardcoded into the class now you will not be able to use this class to track anything else but faces (which in theory you could). But I expect that most people will not even know how to prepare a different Haar cascade recognition set - for example I'm one of those.

What's interesting is that this code will even run in Flash Player 9, since there is no PixelBender or Alchemy and neither the new Vector data type being used. Obviously by targeting Flash Player 10 one could even add a few more optimizations, but that's something for another weekend.

Example 1: Ohtsuka Masakazu's modified sample file

Example 2: Real-time webcam face tracking

Here's the modified sourcecode which includes two example files: Marilena_mod10.zip

"Moskitos":
g.clear();ls(1);o[i++]=[mouseX,mouseY,2*r(),2*r()]; for each(p in o){mt(p[0],p[1]); lt(p[0]+=2*s(p[1]*p[2]),p[1]+=2*s(p[0]*p[3])); };i%=2000

"Love in 3D":
if(i<250&&(x=y++)) with(addChild(new TextField)) text="<3",x=r()*500-250,y=r()*500-250,z=i*2-250, rotationX=rotationZ=r()*360;rotationY=i++

"Hyperdrive":
if (!o.b)addChild(o.c=new Bitmap(o.b=new BitmapData(500,500))),o.b.noise(2); z=s(i-=0.01)*15;o.c.z=-z;o.b.draw(this,o.o,o.o,"hardlight");

"Fibonacci Sequence":
if(!i)i=1,o.b=0,o.c=addChild(new TextField),o.c.autoSize="left",o.c.wordWrap=o.c.multiline=1; o.c.appendText(" "+i),o.a=o.b,o.b=i,i+=o.a

"All Invaders":
if(!i) g.beginFill(0),a=b=i=64,d=511; g.drawRect(a&=d,b&=d,i,i); g.drawRect(d-a+1-i,b,i,i); a+=r()<0.4?i:0;b+=r()<0.4?i:0;

"The Golden Days":
g.clear();g.beginFill(0); x=y=20;for(i=81;i--;g.drawCircle(40*a,40*b,d*2)) a=(i%9),b=int(i/9), d=m.sqrt((d=mouseX/40-a)*d+(d=mouseY/40-b)*d)

"Superstar":
g.beginFill(r()<0.4?0:1e8);q=r()*640;c=r()*480;a=2;u=r()*10; for(d=12;--d;(d<11?lt:mt)(q+u*a*s(i+=0.63), c+u*a*s(i+1.57)),a=7-a){}

"1967":
i+=0.06;g.clear();g.beginFill(0);for(x=80;x--;){g.drawCircle(25+70*(x%9),25+70*(y=int(x/9)),70+(s(x%9*0.7+i+s(y*0.7+i)))*35)}

As a starting point there is a small wrapper which predefines a few common functions and objects with short variable names. Above you see my first entry - which requires Flash Player 10 btw: which looks like this as a tweet

Here's a little demo.

First you might ask what the hell this method is needed for at all. There are three applications that come to my mind right away: blob tracking for multi-touch interfaces (using my flood fill method), QR-code recognition and bitmap vectorization. All of them have in common that after having detected a blob of uniform color in a bitmap you want to trace the outline of it. In order to do that you need to have a starting point of which you know that it lies on the edge. That's what the method I will show you here does.

If we were programming in C the approach would be to start at the top left corner of the bitmap and do a getPixel(x,y) from left to right and top to bottom until you find a pixel that is not 0x00000000. But since we are using Actionscript there are a few methods in the BitmapData class that will be quite faster than a loop.

The workhorse in this case is the getColorBoundsRect() method. In case you forgot what it does: this method will return you a Rectangle which encloses all pixels of a certain color inside a BitmapData object.

The starting situation: we have a transparent bitmap which contains non-transparent pixels somewhere. In step one we use getColorBoundsRect to narrow down on the minimum area that still contains any non transparent pixels:

var r1:Rectangle = bitmapData.getColorBoundsRect( 0xff000000, 0, false );

If the height of r1 equals 0 we know that the image is completely empty and can stop rightaway. If not we can continue. And here's the trick. We've got a rectangle now and we know for sure that somewhere in the topmost row of that rectangle there is a non-transparent pixel. What we cannot count on is that it is the one at the very left, aka r1.topLeft - it's absolutely possible that there is just a single pixel set in that row and it can be anywhere between r1.left and r1.right.

But here's the trick: we simply use getColorBoundsRect again - unfortunately we cannot do this using the same bitmapData. We need to extract that first row of pixels into a separate temporary bitmap:

var temp:BitmapData = new BitmapData( r1.width, 1, true, 0 );
temp.copyPixels( bitmapData, r1, new Point());

Now that we've got a 1 pixel high bitmapdata which contains at least one non-transparent pixel we can continue:

var r2:Rectangle = temp.getColorBoundsRect( 0xff000000, 0, false );

The last step is to add the offset we have found here to the previously found top left corner of first colorbounds:

var startPoint:Point = r1.topLeft.add( r2.topLeft );

All together it looks like this:

public static function getFirstNonTransparentPixel( bmd:BitmapData ):Point
{
var r1:Rectangle = bmd.getColorBoundsRect( 0xff000000, 0, false );
if ( r1.width > 0 )
{
var temp:BitmapData = new BitmapData( r1.width, 1, true, 0 );
temp.copyPixels( bmd, r1, new Point());
var r2:Rectangle = temp.getColorBoundsRect( 0xff000000, 0, false );
return r1.topLeft.add( r2.topLeft );
}
return null;
}

Or you can download the EdgeFinder.as class here

[Important Notice] I should have made some speed test before recommending this method. It turns out that getColorBoundsRect() is not as fast as I thought it was. Sakri Rosenstrom's HitTest method is twice as fast as mine. So I've taken another look at his implementation and optimized it a bit more, making a kind of hybrid between both of our methods. My test tell me that this optimzed version is about twice as fast - I have updated the code in EdgeFinder.as already

public static function getFirstNonTransparentPixel( bmd:BitmapData ):Point
{
var hit_rect:Rectangle=new Rectangle(0,0,bmd.width,1);
var p:Point = new Point();
for( hit_rect.y = 0; hit_rect.y < bmd.height; hit_rect.y++ )
{
if( bmd.hitTest( p, 0x01, hit_rect) )
{
var hit_bmd:BitmapData=new BitmapData( bmd.width, 1, true, 0 );
hit_bmd.copyPixels( bmd, hit_rect, p );
return hit_rect.topLeft.add( hit_bmd.getColorBoundsRect(0xFF000000, 0, false).topLeft );
}
}
return null;
}