After buying an XBox with a kinect sensor, noticing that there is no way to attach it to your computer when you buy the bundle, and buying a standalone kinect sensor afterwards, I looked into getting the kinect data into my flash / AIR applications.
There are a few methods around:
- AS3Kinect: http://www.as3kinect.org/ – using libfreenect, and sockets to get the 2 webcam images (depth and color) into flash.
- TUIOKinect (https://code.google.com/p/tuiokinect) + TUIO AS3 (http://bubblebird.at/tuioflash/tuio-as3-library) – track “blobs” and get them as multitouch cursors in your flash app. More info at http://silviopaganini.posterous.com/openkinect-flash-tuio-udp-flash-bridge
- OSCeleton (https://github.com/Sensebloom/OSCeleton) sends out skeleton data in the OSC format to a UDP destination
Getting the images in flash is really slow, it needs to compress (at best) each frame and send that information over a socket / UDP / Localconnection to flash. It’s quite heave, and all you’ve got to work with are images.
The blob tracking from TUIOKinect is more interesting: we only send blob tracking info to our flash app, which is a lot “lighter” to transfer. However, TUIOKinect needs to be configured (what z-depth do you want to detect) and generates quite a lot false positives (stuff you don’t want to track). You don’t know if a blob is a hand or not. The fun part of TUIOKinect is that it’s TUIO, which means that you can track a blob, and use it as a touch input.
Then we have OSCeleton. With OSCeleton, you can get real skeleton data into flash. After standing in a calibration pose, it spits out the coordinates of you skeleton joints (head, left arm, right knee, …) to a UDP connection.
I decided to combine AS3TUIO with OSCeleton, so we can translate skeleton joint information and use skeleton joins (say: left hand, right hand) as touch points. This way, you can use the standard multitouch events, only you aren’t really touching anything, but waving in front of a kinect camera.
Setting it up
First things first: getting OSCeleton running. There’s an excellent guide over at https://github.com/Sensebloom/OSCeleton – so follow those steps first.
After you’ve got OSCeleton up and running, you can get the data into your AIR application using my library.
Download the source or swc file, and add it to your project.
Download some example projects (AIR, Flash Builder).
Really like your blog, fair play! Currently trying to get a Kinect working with Adobe AIR but having trouble. Im a multimedia student and have been working with actionscript the past year. Wondering if you would be willing to offer any help?
Cheers
Nice job wouter, we posted it on http://www.kinecthacks.nl grtz kinecthacks
Very nice!.
I did a hack using this TUIOServer https://github.com/zebradog/PointViewerTUIO and the tuio-as3 http://code.google.com/p/tuio-as3/ for a flash application that the user used just one hand. My next pass was use OSCeleton server and read the skeleton data into flash, but you already did =D. I download your code, congratulation im reading and i will develop a new aplication using this. Did you think about putting this code on github as open source or commit in the tuio-as3 project?
Hi, It’s really usefull thing for me, but I’ve got huge problem:
during export to Air i get error telling Joint is a static class and you can use no property for it. Could you help me? what is wrong?
Thanks in advance.
Great post! I couldn’t find the as3osceleton.swc in the download link, is it posted somewhere?
nice work…
two questions :
1/ Why did you changed the init of GetureManager & TuioManager from one param required to two ?…
2/how could I manage a simple gesture movement between lastxposition and the really new xposition in your GestureHandsView example ?
Is there an easy way with Tuio, certainly but I don’ t get it…
If not, I’m thinking about a method with a timer and some conditions, but I’d prefer something using these libs if it exist
thanks
e.
Is a great job, work stable with skeleton data and gesture.
will you keep develop it for more feature? like depth video or video display skeleton rotation … etc
Anyway thanks for your work,you did it well.
Hi, I am sorry for posting such a dummy question… I installed everything (OSCeleton is already working with the Processing examples), I downloaded the FB example projects, and now I can’t run them… I am novice with FB. I created a project named Demo01_UserDetection, moved the be and the org directories in the src directory. The main app file is .mxml, and the one in the demo is Demo01_UserDetection.as. So: How do I get the Demo01_UserDetection.as file into my project?
Thanks a lot,
i
hi Wouter,
I posted a few messages on your excellent tutorial about the kinect and the native extension for air :)…
but i’m sure you will agree, we stay dependent of a windows’s property sdk…
I’ve just readen this good news :
http://groups.google.com/group/openni-dev/browse_thread/thread/570380583bdb02e0
the last NITE version, which does NOT need calibration !
yeah ! let’s party
will you try this on this OSCeleton solution ?
It’s just updating drivers et recompiling osceleton, right ? and it should work ? I will try this, this I can manage, further… nope…
but sure some of you will ;)
e.
Simply amazing.
Thanks for sharing.