Access the mac sudden motion sensor with an AIR native extension

AIR on mobile enables you to access the accelerometer of the mobile device. But what about the motion sensor in your macbook / macbook pro computer on the desktop? All macbooks come with a “sudden motion sensor”, which shuts down the hard disk when the laptop moves too much.

I wrote an AIR native extension, to access this sensor information in AIR on OSX. I tried to mimic the Accelerometer API as much as possible, and make it easy to use in your AIR application.

You can check if the SuddenMotionSensor is supported on your mac, set the update interval and listen for accelerometer events:

if(SuddenMotionSensor.isSupported)
{
	suddenMotionSensor = new SuddenMotionSensor();
	suddenMotionSensor.setRequestedUpdateInterval(50);
	suddenMotionSensor.addEventListener(AccelerometerEvent.UPDATE, accelerometerUpdateHandler, false, 0, true);
}

You can then access the accelerometer info in the event handler:

private function accelerometerUpdateHandler(event:AccelerometerEvent):void
{
	trace(event.accelerationX, event.accelerationY, event.accelerationZ);
}

Download the sources & demos here, and have fun!

Accessing the kinect in javascript through websockets

Good morning all! (or evening, night, … depending on when you read this post of course). As you might know, I’ve been working on AIRKinect (as3nui.com) and I’ve got a side project AIRServer aswell (which allows you to setup air as a socket server, including websocket support).

Wouldn’t it be fun, to combine these two projects in a demo, so you can access the kinect information through a websocket? That’s exactly what I did. You run a desktop application on your computer, which is responsible for accessing the kinect, and exposing the skeleton information over a websocket. Using a javascript client, which supports websockets, you can connect to that server, and use the skeleton information in the javascript client :-)

In this demo, I’m just rendering the skeleton points in a canvas element, using three.js.

I’ve uploaded the sources and included binary installers for the desktop application (windows 7, OSX Lion). What you’ll need to do is install & launch the desktop application, and click on the “start server” button to listen for websocket connections on the given port. Make sure you’ve got the kinect sdk installed on your computer (windows) or openni on OSX.

Using the javascript client, you connect to your ip (if you’re testing on the same ip, 127.0.0.1 should be fine), and you can start dancing in the canvas element :-).

AIRServer 0.5 – socket byte concatenation

I’ve just finished work on a little update of my AIRServer library (version 0.5, hooray!). Apparently, when you send large chunks of data over the socket (like sending an image to the server, over the socket), it could happen the data is split over multiple packages. This caused errors on the server side.

I’ve fixed that issue, and uploaded an updated version, together with an image-sending-demo. Enjoy!

AIRServer 0.4 – UDP Handling, Chrome 16 websockets & bugfixes

I’ve just finished some updates on my AIRServer library, which enables you to create an AIR app that listens for different inputs such as sockets, websockets and P2P traffic. This gives you the option to create a multi-user game, with different input controllers.

I’ve added a UDP Endpoint, so you can handle UDP traffic aswell now (check out the UDP native extension for AIR mobile, to use UDP on mobile devices). UDP is connectionless, so you can specify a timeout, when we mark a “udp client” as disconnected.

server.addEndPoint(new UDPEndPoint(1236, new NativeObjectSerializer(), 60000));

When you want to send data back over UDP, you’ll need to know the UDP listening port of the client: therefore, the client can send a “PORT” command, with the listening port as data argument:

protected function connect():void
{
	listeningSocket = new DatagramSocket();
	listeningSocket.addEventListener(DatagramSocketDataEvent.DATA, socketDataHandler);
	listeningSocket.bind(9876);
	listeningSocket.receive();
 
	sendingSocket = new DatagramSocket();
	sendingSocket.connect("127.0.0.1", int(port.text));
	currentState = "connected";
}
 
protected function sendInput():void
{
	sendObject({command: "PORT", data: listeningSocket.localPort});
	sendObject(inputField.text);
	inputField.text = "";
}
 
protected function sendObject(o:Object):void
{
	var bytes:ByteArray = new ByteArray();
	bytes.writeObject(o);
	sendingSocket.send(bytes);
}

I’ve also made some arguments optional (such as the message serializers). By default, an AMF endpoint will use a NativeObjectSerializer, websockets will use the JSONSerializer.

server.addEndPoint(new SocketEndPoint(1234, new AMFSocketClientHandlerFactory()));
server.addEndPoint(new SocketEndPoint(1235, new WebSocketClientHandlerFactory()));
server.addEndPoint(new UDPEndPoint(1236, new NativeObjectSerializer(), 60000));
server.addEndPoint(new CocoonP2PEndPoint("be.aboutme.airserver.demos.Messages"));

I’ve fixed some issues with multiple-messages in one packet. This was especially a problem with the websocket listener. The object serializer is now responsible for splitting the input into multiple messages (when necessary). By default, the JSONSerializer used for the websockets, will split messages on the newline (\n) character. Make sure you terminate each message you send from the client with this character, and you should be good to go.

As always, you can download the sources & updated demos to play with. Happy coding!

You can find the latest version here :-)

UDP Native Extension for AIR Mobile – now with Android Support!

I’ve continued my work on my UDP AIR native extension, to add support for Android. I’m happy to release version 0.2, which adds Android support :-)

This means, from now on, you can send / receive UDP packets in your AIR mobile projects on both iOS & Android. If you find any bugs, or have suggestions, please let me know.

You can download the updated ane, and all the sources.

Enjoy!

UDP in AIR for iOS using a native extension

update: added Android support to the extension.

When you’re using a mobile device as a controller for an application or a game, you’ll want fast data transfers. Classic TCP/IP traffic over sockets is a bit slow, due to the nature of TCP/IP (packets are delivered in the correct order, the receiver sends a confirmation of reception to the sender for each received packet). The alternative is UDP: you’re not sure if the packet arrives, or in what order you packets will arrive at the destination, but because of that, there is less delay between the sender and the receiver of the packet.

AIR has a builtin class to handle UDP: flash.net.DatagramSocket. However, for some reason this is not available on AIR for mobile devices. I decided to write a native extension (only for iOS for now) to offer UDP functionality on AIR for mobile devices. I tried to use the same API as the DatagramSocket for AIR for Desktop, so the principles are the same.

To send packets over UDP, you’ll create an instance of the UDPSocket class (be.aboutme.nativeExtensions.udp.UDPSocket), and use the send method with a bytearray:

var udpSocket:UDPSocket = new UDPSocket();
var bytes:ByteArray = new ByteArray();
bytes.writeUTFBytes("Hello World");
udpSocket.send(bytes, "192.168.9.1", 1234);

To receive packets, you’ll use the bind(portnr) and receive() methods of the same class, and listen to a DatagramSocketEvent.DATA event:

var udpSocket:UDPSocket = new UDPSocket();
udpSocket.addEventListener(DatagramSocketDataEvent.DATA, udpDataHandler);
udpSocket.bind(1234);
udpSocket.receive();
 
protected function udpDataHandler(event:DatagramSocketDataEvent):void
{
	trace(event.data);
}

It will transfer whatever you put in the bytearray, so you can send native actionscript objects aswell if you want:

var bytes:ByteArray = new ByteArray();
var o:Object = {};
o.command = "MESSAGE";
o.content = "Hello World!";
bytes.writeObject(o);
udpSocket.send(bytes, "192.168.9.1", 1234);

You can download the compiled ane file, together with the actionscript & objectiveC sources and 2 sample projects. The native extension id is “be.aboutme.nativeExtensions.udp.UDPSocket”. Stay tuned for the Android version The extension has been updated, and supports Android aswell now!

including dylib in air native extension for OSX

I’ve been struggling a bit to create a framework file to use in an air native extension on OSX. The issue was that I was using 3rd party dylib libraries (intel building blocks), which aren’t normally installed on somebody’s computer. The framework compiled and everything, but for some reason the application crashed when using the native extension.

When air loaded the native extension, it could no longer find the necessary dylib files. Solution was:

- add a copy files phase to your build phases, which should copy your dylib file to the Resources directory of your framework
- after compilation, open up a terminal window and navigate to your build folder. Use the otool -L command to get a listing of the linked libraries:

otool -L HelloThreadsAndUSB.framework/HelloThreadsAndUSB

Output is something like this:

HelloThreadsAndUSB:
	/Library/Frameworks/HelloThreadsAndUSB.framework/Versions/A/HelloThreadsAndUSB (compatibility version 1.0.0, current version 1.0.0)
	@rpath/Adobe AIR.framework/Versions/1.0/Adobe AIR (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 15.0.0)
	libtbb.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.9.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.11)

In my case, the libtbb.dylib file could not be found when the application launched. You need to change the lookup path with the install_name_tool command:

install_name_tool -change libtbb.dylib @loader_path/Resources/libtbb.dylib HelloThreadsAndUSB.framework/HelloThreadsAndUSB

When you type in the otool command again, you should see the adjusted path:

HelloThreadsAndUSB:
	/Library/Frameworks/HelloThreadsAndUSB.framework/Versions/A/HelloThreadsAndUSB (compatibility version 1.0.0, current version 1.0.0)
	@rpath/Adobe AIR.framework/Versions/1.0/Adobe AIR (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 15.0.0)
	@loader_path/Resources/libtbb.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.9.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.11)

Now you should be able to use this framework in your air native extension :-)

(Note: I tried the other options @executable_path and @rpath, but those didn’t do the trick)

websocket version 8 support in AIR Server

It’s been a while since my last update on AIRServer – an actionscript library which enables you to set up a socket server in Adobe AIR, which listens to multiple socket inputs (native AMF sockets, websockets and P2P input).

Some of my students noticed the websockets didn’t work in the latest versions of Google Chrome. This is because Google chrome uses another version of the websocket protocol. If you enjoy reading specs, you can find it at http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-10 :-)

Anyway, took me quite some work, but I finally got it working. I was able to port some of the code of the Bauglir Internet Library (http://www.webnt.eu/index.php) to actionscript. AIRServer now supports both hybi-00 and hybi-10 of the protocol.

You can download the updated library and demo’s here. Enjoy!

Update on http://blog.aboutme.be/2011/12/23/airserver-0-5-socket-byte-concatenation/

Flex Mobile Tabbed View Navigator Button Badge

I wanted to display a “badge text” in a Flex Mobile project. Badge texts are the red circles on iOS, where you can see a number of updates, unread messages, …

This isn’t included in the default flex mobile components / skins, so I decided to extend & skin the tabbed view navigator, so you’re able to add this functionality in your applications.

You simple use my BadgeViewNavigator instead of ViewNavigators, and set a badgeText value to display in the badge:

<views:BadgeViewNavigator label="One" width="100%" height="100%"
		firstView="be.aboutme.flex.mobile.badgeViewNavigator.views.OneView"
		badgeText="23"/>
<views:BadgeViewNavigator label="Two" width="100%" height="100%"
		firstView="be.aboutme.flex.mobile.badgeViewNavigator.views.TwoView"
		badgeText="2"/>
<views:BadgeViewNavigator label="Three" width="100%" height="100%"
		firstView="be.aboutme.flex.mobile.badgeViewNavigator.views.ThreeView"/>

You can download the sources & demo in this fxp file. Enjoy!

Kinect native extension for Adobe AIR

Update: This sample only supports the beta1 SDK on windows. Please check out http://www.as3nui.com, where further development is happening (newer SDK versions, support for OSX).

I’ve been playing with native extensions for air for a couple of weeks now. One of the things I wanted to do, was to get the Kinect working through a native extension. I’ve posted some sample libraries before, where the kinect date was sent to Adobe AIR through UDP sockets. However, the bandwith is quite limited, and there are noticable delays.

Using a native extension, we don’t have those limitations anymore. Another pro is that we don’t need to run a seperate program to send the data to our flash application.

I’ve got multiple skeleton tracking working, together with the video and the depth video.

kinect = new Kinect();
kinect.addEventListener(KinectEvent.SKELETON_TRACKED, skeletonTrackedHandler, false, 0, true);
kinect.startTracking();
 
kinectVideo = new KinectVideo(kinect);
kinectVideo.play();
addChild(kinectVideo);
 
kinectDepth = new KinectDepth(kinect);
kinectDepth.x = 650;
kinectDepth.play();
addChild(kinectDepth);

displaying the skeleton is pretty straightforward:

private function drawSkeletons():void
{
	skeletonsContainer.graphics.clear();
	for each(var skeleton:Skeleton in kinect.skeletons)
	{
		for each(var joint:Joint in skeleton.joints)
		{
			drawJoint(joint);
		}
		//draw some bones
		drawBone(skeleton.joints[Joint.SHOULDER_CENTER], skeleton.joints[Joint.SHOULDER_LEFT]);
		drawBone(skeleton.joints[Joint.SHOULDER_LEFT], skeleton.joints[Joint.ELBOW_LEFT]);
		drawBone(skeleton.joints[Joint.ELBOW_LEFT], skeleton.joints[Joint.WRIST_LEFT]);
		drawBone(skeleton.joints[Joint.WRIST_LEFT], skeleton.joints[Joint.HAND_LEFT]);
		drawBone(skeleton.joints[Joint.SHOULDER_CENTER], skeleton.joints[Joint.SHOULDER_RIGHT]);
		drawBone(skeleton.joints[Joint.SHOULDER_RIGHT], skeleton.joints[Joint.ELBOW_RIGHT]);
		drawBone(skeleton.joints[Joint.ELBOW_RIGHT], skeleton.joints[Joint.WRIST_RIGHT]);
		drawBone(skeleton.joints[Joint.WRIST_RIGHT], skeleton.joints[Joint.HAND_RIGHT]);
		drawBone(skeleton.joints[Joint.SHOULDER_LEFT], skeleton.joints[Joint.SPINE]);
		drawBone(skeleton.joints[Joint.SHOULDER_RIGHT], skeleton.joints[Joint.SPINE]);
		drawBone(skeleton.joints[Joint.SPINE], skeleton.joints[Joint.HIP_CENTER]);
		drawBone(skeleton.joints[Joint.HIP_CENTER], skeleton.joints[Joint.HIP_LEFT]);
		drawBone(skeleton.joints[Joint.HIP_LEFT], skeleton.joints[Joint.KNEE_LEFT]);
		drawBone(skeleton.joints[Joint.KNEE_LEFT], skeleton.joints[Joint.ANKLE_LEFT]);
		drawBone(skeleton.joints[Joint.ANKLE_LEFT], skeleton.joints[Joint.FOOT_LEFT]);
		drawBone(skeleton.joints[Joint.HIP_CENTER], skeleton.joints[Joint.HIP_RIGHT]);
		drawBone(skeleton.joints[Joint.HIP_RIGHT], skeleton.joints[Joint.KNEE_RIGHT]);
		drawBone(skeleton.joints[Joint.KNEE_RIGHT], skeleton.joints[Joint.ANKLE_RIGHT]);
		drawBone(skeleton.joints[Joint.ANKLE_RIGHT], skeleton.joints[Joint.FOOT_RIGHT]);
	}
}
 
private function drawJoint(joint:Joint):void
{
	skeletonsContainer.graphics.beginFill(0x000000);
	skeletonsContainer.graphics.drawCircle(joint.x * 640, joint.y * 480, 10 / joint.z);
	skeletonsContainer.graphics.endFill();
}
 
private function drawBone(from:Joint, to:Joint):void
{
	skeletonsContainer.graphics.lineStyle(1);
	skeletonsContainer.graphics.moveTo(from.x * 640, from.y * 480);
	skeletonsContainer.graphics.lineTo(to.x * 640, to.y * 480);
}

You can download the flash builder project and the visual studio project for the native extension. I didn’t have any C programming experience before, so there’s probably room for improvement on the C side. Currently, the extension is only available for windows using the kinect sdk. Make sure you’ve installed the microsoft kinect sdk, aswell as visual studio. You’ll also need to launch the flash builder project using my ant build script which is included in the flash builder project. You’ll want to update the path to your air sdk in the ant-debug.xml.

If you want to see it live in action, come check out our session “interREACT with the flas platform” at the FITC unconference at MAX (tuesday).