I wrote an AIR native extension, to access this sensor information in AIR on OSX. I tried to mimic the Accelerometer API as much as possible, and make it easy to use in your AIR application.

You can check if the SuddenMotionSensor is supported on your mac, set the update interval and listen for accelerometer events:

if(SuddenMotionSensor.isSupported)
{
	suddenMotionSensor = new SuddenMotionSensor();
	suddenMotionSensor.setRequestedUpdateInterval(50);
	suddenMotionSensor.addEventListener(AccelerometerEvent.UPDATE, accelerometerUpdateHandler, false, 0, true);
}

You can then access the accelerometer info in the event handler:

private function accelerometerUpdateHandler(event:AccelerometerEvent):void
{
	trace(event.accelerationX, event.accelerationY, event.accelerationZ);
}

Download the sources & demos here, and have fun!

Wouldn’t it be fun, to combine these two projects in a demo, so you can access the kinect information through a websocket? That’s exactly what I did. You run a desktop application on your computer, which is responsible for accessing the kinect, and exposing the skeleton information over a websocket. Using a javascript client, which supports websockets, you can connect to that server, and use the skeleton information in the javascript client :-)

In this demo, I’m just rendering the skeleton points in a canvas element, using three.js.

I’ve uploaded the sources and included binary installers for the desktop application (windows 7, OSX Lion). What you’ll need to do is install & launch the desktop application, and click on the “start server” button to listen for websocket connections on the given port. Make sure you’ve got the kinect sdk installed on your computer (windows) or openni on OSX.

Using the javascript client, you connect to your ip (if you’re testing on the same ip, 127.0.0.1 should be fine), and you can start dancing in the canvas element :-).

I’ve fixed that issue, and uploaded an updated version, together with an image-sending-demo. Enjoy!

I’ve added a UDP Endpoint, so you can handle UDP traffic aswell now (check out the UDP native extension for AIR mobile, to use UDP on mobile devices). UDP is connectionless, so you can specify a timeout, when we mark a “udp client” as disconnected.

server.addEndPoint(new UDPEndPoint(1236, new NativeObjectSerializer(), 60000));

When you want to send data back over UDP, you’ll need to know the UDP listening port of the client: therefore, the client can send a “PORT” command, with the listening port as data argument:

protected function connect():void
{
	listeningSocket = new DatagramSocket();
	listeningSocket.addEventListener(DatagramSocketDataEvent.DATA, socketDataHandler);
	listeningSocket.bind(9876);
	listeningSocket.receive();
 
	sendingSocket = new DatagramSocket();
	sendingSocket.connect("127.0.0.1", int(port.text));
	currentState = "connected";
}
 
protected function sendInput():void
{
	sendObject({command: "PORT", data: listeningSocket.localPort});
	sendObject(inputField.text);
	inputField.text = "";
}
 
protected function sendObject(o:Object):void
{
	var bytes:ByteArray = new ByteArray();
	bytes.writeObject(o);
	sendingSocket.send(bytes);
}

I’ve also made some arguments optional (such as the message serializers). By default, an AMF endpoint will use a NativeObjectSerializer, websockets will use the JSONSerializer.

server.addEndPoint(new SocketEndPoint(1234, new AMFSocketClientHandlerFactory()));
server.addEndPoint(new SocketEndPoint(1235, new WebSocketClientHandlerFactory()));
server.addEndPoint(new UDPEndPoint(1236, new NativeObjectSerializer(), 60000));
server.addEndPoint(new CocoonP2PEndPoint("be.aboutme.airserver.demos.Messages"));

I’ve fixed some issues with multiple-messages in one packet. This was especially a problem with the websocket listener. The object serializer is now responsible for splitting the input into multiple messages (when necessary). By default, the JSONSerializer used for the websockets, will split messages on the newline (\n) character. Make sure you terminate each message you send from the client with this character, and you should be good to go.

As always, you can download the sources & updated demos to play with. Happy coding!

You can find the latest version here :-)

This means, from now on, you can send / receive UDP packets in your AIR mobile projects on both iOS & Android. If you find any bugs, or have suggestions, please let me know.

You can download the updated ane, and all the sources.

Enjoy!

When you’re using a mobile device as a controller for an application or a game, you’ll want fast data transfers. Classic TCP/IP traffic over sockets is a bit slow, due to the nature of TCP/IP (packets are delivered in the correct order, the receiver sends a confirmation of reception to the sender for each received packet). The alternative is UDP: you’re not sure if the packet arrives, or in what order you packets will arrive at the destination, but because of that, there is less delay between the sender and the receiver of the packet.

AIR has a builtin class to handle UDP: flash.net.DatagramSocket. However, for some reason this is not available on AIR for mobile devices. I decided to write a native extension (only for iOS for now) to offer UDP functionality on AIR for mobile devices. I tried to use the same API as the DatagramSocket for AIR for Desktop, so the principles are the same.

To send packets over UDP, you’ll create an instance of the UDPSocket class (be.aboutme.nativeExtensions.udp.UDPSocket), and use the send method with a bytearray:

var udpSocket:UDPSocket = new UDPSocket();
var bytes:ByteArray = new ByteArray();
bytes.writeUTFBytes("Hello World");
udpSocket.send(bytes, "192.168.9.1", 1234);

To receive packets, you’ll use the bind(portnr) and receive() methods of the same class, and listen to a DatagramSocketEvent.DATA event:

var udpSocket:UDPSocket = new UDPSocket();
udpSocket.addEventListener(DatagramSocketDataEvent.DATA, udpDataHandler);
udpSocket.bind(1234);
udpSocket.receive();
 
protected function udpDataHandler(event:DatagramSocketDataEvent):void
{
	trace(event.data);
}

It will transfer whatever you put in the bytearray, so you can send native actionscript objects aswell if you want:

var bytes:ByteArray = new ByteArray();
var o:Object = {};
o.command = "MESSAGE";
o.content = "Hello World!";
bytes.writeObject(o);
udpSocket.send(bytes, "192.168.9.1", 1234);

You can download the compiled ane file, together with the actionscript & objectiveC sources and 2 sample projects. The native extension id is “be.aboutme.nativeExtensions.udp.UDPSocket”. Stay tuned for the Android version The extension has been updated, and supports Android aswell now!

When air loaded the native extension, it could no longer find the necessary dylib files. Solution was:

- add a copy files phase to your build phases, which should copy your dylib file to the Resources directory of your framework
- after compilation, open up a terminal window and navigate to your build folder. Use the otool -L command to get a listing of the linked libraries:

otool -L HelloThreadsAndUSB.framework/HelloThreadsAndUSB

Output is something like this:

HelloThreadsAndUSB:
	/Library/Frameworks/HelloThreadsAndUSB.framework/Versions/A/HelloThreadsAndUSB (compatibility version 1.0.0, current version 1.0.0)
	@rpath/Adobe AIR.framework/Versions/1.0/Adobe AIR (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 15.0.0)
	libtbb.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.9.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.11)

In my case, the libtbb.dylib file could not be found when the application launched. You need to change the lookup path with the install_name_tool command:

install_name_tool -change libtbb.dylib @loader_path/Resources/libtbb.dylib HelloThreadsAndUSB.framework/HelloThreadsAndUSB

When you type in the otool command again, you should see the adjusted path:

HelloThreadsAndUSB:
	/Library/Frameworks/HelloThreadsAndUSB.framework/Versions/A/HelloThreadsAndUSB (compatibility version 1.0.0, current version 1.0.0)
	@rpath/Adobe AIR.framework/Versions/1.0/Adobe AIR (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 15.0.0)
	@loader_path/Resources/libtbb.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.9.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.11)

Now you should be able to use this framework in your air native extension :-)

(Note: I tried the other options @executable_path and @rpath, but those didn’t do the trick)

This isn’t included in the default flex mobile components / skins, so I decided to extend & skin the tabbed view navigator, so you’re able to add this functionality in your applications.

You simple use my BadgeViewNavigator instead of ViewNavigators, and set a badgeText value to display in the badge:

<views:BadgeViewNavigator label="One" width="100%" height="100%"
		firstView="be.aboutme.flex.mobile.badgeViewNavigator.views.OneView"
		badgeText="23"/>
<views:BadgeViewNavigator label="Two" width="100%" height="100%"
		firstView="be.aboutme.flex.mobile.badgeViewNavigator.views.TwoView"
		badgeText="2"/>
<views:BadgeViewNavigator label="Three" width="100%" height="100%"
		firstView="be.aboutme.flex.mobile.badgeViewNavigator.views.ThreeView"/>

You can download the sources & demo in this fxp file. Enjoy!

I’ve been playing with native extensions for air for a couple of weeks now. One of the things I wanted to do, was to get the Kinect working through a native extension. I’ve posted some sample libraries before, where the kinect date was sent to Adobe AIR through UDP sockets. However, the bandwith is quite limited, and there are noticable delays.

Using a native extension, we don’t have those limitations anymore. Another pro is that we don’t need to run a seperate program to send the data to our flash application.

I’ve got multiple skeleton tracking working, together with the video and the depth video.

kinect = new Kinect();
kinect.addEventListener(KinectEvent.SKELETON_TRACKED, skeletonTrackedHandler, false, 0, true);
kinect.startTracking();
 
kinectVideo = new KinectVideo(kinect);
kinectVideo.play();
addChild(kinectVideo);
 
kinectDepth = new KinectDepth(kinect);
kinectDepth.x = 650;
kinectDepth.play();
addChild(kinectDepth);

displaying the skeleton is pretty straightforward:

private function drawSkeletons():void
{
	skeletonsContainer.graphics.clear();
	for each(var skeleton:Skeleton in kinect.skeletons)
	{
		for each(var joint:Joint in skeleton.joints)
		{
			drawJoint(joint);
		}
		//draw some bones
		drawBone(skeleton.joints[Joint.SHOULDER_CENTER], skeleton.joints[Joint.SHOULDER_LEFT]);
		drawBone(skeleton.joints[Joint.SHOULDER_LEFT], skeleton.joints[Joint.ELBOW_LEFT]);
		drawBone(skeleton.joints[Joint.ELBOW_LEFT], skeleton.joints[Joint.WRIST_LEFT]);
		drawBone(skeleton.joints[Joint.WRIST_LEFT], skeleton.joints[Joint.HAND_LEFT]);
		drawBone(skeleton.joints[Joint.SHOULDER_CENTER], skeleton.joints[Joint.SHOULDER_RIGHT]);
		drawBone(skeleton.joints[Joint.SHOULDER_RIGHT], skeleton.joints[Joint.ELBOW_RIGHT]);
		drawBone(skeleton.joints[Joint.ELBOW_RIGHT], skeleton.joints[Joint.WRIST_RIGHT]);
		drawBone(skeleton.joints[Joint.WRIST_RIGHT], skeleton.joints[Joint.HAND_RIGHT]);
		drawBone(skeleton.joints[Joint.SHOULDER_LEFT], skeleton.joints[Joint.SPINE]);
		drawBone(skeleton.joints[Joint.SHOULDER_RIGHT], skeleton.joints[Joint.SPINE]);
		drawBone(skeleton.joints[Joint.SPINE], skeleton.joints[Joint.HIP_CENTER]);
		drawBone(skeleton.joints[Joint.HIP_CENTER], skeleton.joints[Joint.HIP_LEFT]);
		drawBone(skeleton.joints[Joint.HIP_LEFT], skeleton.joints[Joint.KNEE_LEFT]);
		drawBone(skeleton.joints[Joint.KNEE_LEFT], skeleton.joints[Joint.ANKLE_LEFT]);
		drawBone(skeleton.joints[Joint.ANKLE_LEFT], skeleton.joints[Joint.FOOT_LEFT]);
		drawBone(skeleton.joints[Joint.HIP_CENTER], skeleton.joints[Joint.HIP_RIGHT]);
		drawBone(skeleton.joints[Joint.HIP_RIGHT], skeleton.joints[Joint.KNEE_RIGHT]);
		drawBone(skeleton.joints[Joint.KNEE_RIGHT], skeleton.joints[Joint.ANKLE_RIGHT]);
		drawBone(skeleton.joints[Joint.ANKLE_RIGHT], skeleton.joints[Joint.FOOT_RIGHT]);
	}
}
 
private function drawJoint(joint:Joint):void
{
	skeletonsContainer.graphics.beginFill(0x000000);
	skeletonsContainer.graphics.drawCircle(joint.x * 640, joint.y * 480, 10 / joint.z);
	skeletonsContainer.graphics.endFill();
}
 
private function drawBone(from:Joint, to:Joint):void
{
	skeletonsContainer.graphics.lineStyle(1);
	skeletonsContainer.graphics.moveTo(from.x * 640, from.y * 480);
	skeletonsContainer.graphics.lineTo(to.x * 640, to.y * 480);
}

You can download the flash builder project and the visual studio project for the native extension. I didn’t have any C programming experience before, so there’s probably room for improvement on the C side. Currently, the extension is only available for windows using the kinect sdk. Make sure you’ve installed the microsoft kinect sdk, aswell as visual studio. You’ll also need to launch the flash builder project using my ant build script which is included in the flash builder project. You’ll want to update the path to your air sdk in the ant-debug.xml.

If you want to see it live in action, come check out our session “interREACT with the flas platform” at the FITC unconference at MAX (tuesday).

I don’t know any C++ at all, so it was quite a challenge to build a working dll and getting Adobe AIR to talk to it. Right now, all my example does is simple return the message it gets from Adobe AIR back. But it might form a starting point to build more exciting stuff on top of it.

Building and testing the native extensions is quite some command-line work, so I decided to put it in an ant script, which builds the .ANE file, extracts it and starts a AIR debug session with the extension.

You can download the visual studio project and the flash builder project. The flash builder project comes with an ant script (ant-debug.xml), which does the native extension packaging & debug launch. You may need to change the path to your Adobe AIR sdk in the ant-debug.xml file.

Enjoy!