Friday, November 12, 2010

Augmented Reality with Away3D and FLARManager

Most of the examples for FLARManager out there seem to center around Papervision... which leaves a lot to be desired for those of us who follow other engines; more specifically Away3D. I've built and re-built my base scripts a few times now, but have a pretty solid mechanism for working with Augmented Reality that I figure others might like to try out.

Now, I'll mention this might seem a bit convaluted. That's because I built this to be recycled for different projects that require different approaches but at the same time required a similar core. One might have a static object that you look at, another might require some level of keyboard interactivity, but another might be mouse event based. That's why.

Outside of the regular cascade of libraries you'll need for FLARManager v0.7 (Download Page) and Away3D 3.5.2 (Download Page) (Note: Not Lite), you'll be creating a few new AS Files with me. We'll start with what I consider to be the core engine that drives everything.

Also Note: As of the date of this post, the latest version of FLARManager and Away3D are NOT compatible. So make sure you're getting the right libs.
First things first, we'll build the package and get the imports out of the way.


A3DFLAREngine.as

package classes {
  import flash.display.MovieClip;
  import flash.display.Sprite;
  import flash.events.Event;

  //FLAR MANAGER
  import com.transmote.flar.*;
  import com.transmote.flar.marker.*;
  import com.transmote.flar.utils.geom.FLARAwayGeomUtils;
  import org.libspark.flartoolkit.support.away3d.FLARCamera3D;

  // Away3D Stuffs
  import away3d.cameras.*;
  import away3d.containers.*;
  import away3d.core.base.*;
  import away3d.core.utils.*;
  import away3d.core.math.*;
  import away3d.core.render.*;
  import away3d.events.*;
  import away3d.loaders.*;
  import away3d.materials.*; 
  import away3d.primitives.Cube;
  import away3d.primitives.Plane; 


Next, we'll declare our class and setup the variables it will handle for us:
public class A3DFLAREngine extends MovieClip { 
    //flar variables 
    protected var flarManager :FLARManager;
    protected var activeMarker :FLARMarker;
    protected var camera3D :FLARCamera3D; 

    //away engine variables
    protected var view :View3D;
    private var scene :Scene3D; 

    //scene objects
    public var modelContainer :ObjectContainer3D;
    public var plane :Plane;
    public var cube :Cube;

There are 3 groups of variables here: the flarmanager's guts, a3d's glory, and the MCs to your augmented reality experience: The plane and cube. So obviously, nothing incredible going on here aside from generating 3d primitives on a marker.

Let's move along,
public function A3DFLAREngine():void {
      initFLARManager();
    }

    protected function initFLARManager():void {
      flarManager = new FLARManager("ar/myConfig.xml");
      addChild(Sprite(flarManager.flarSource));

      flarManager.addEventListener(Event.INIT, init); 
    }

So in the constructor I make a single call to the function initFLARManager(). This function then sets up my FLARManager instance and loads the config file it requires. Note that if you don't plan to extend any further beyond this class, then you could very well setup the FLARManager instance in your constructor. I choose not to, however, because at the point that I've gotten to I actually comment out the initFLARManager() call in the constructor and call it from an external swf that my augmented reality piece is loaded into.

Additionally, the FLARManager has an event dispatcher for the INIT state. So once it is ready, it will dispatch an INIT event, at which point we call:
private function init(e:Event):void {
      flarManager.removeEventListener(Event.INIT, init);

      initEngine();
      initHUD();
      initObjects();
      initListeners();
    }

The init function itself is just a collection of inits. So after some listener cleanup, we'll move on to taking care of each module independently:
private function initEngine():void {
      scene = new Scene3D();
      camera3D = new FLARCamera3D(); 
      camera3D.setParam(flarManager.cameraParams);

      view = new View3D();
      view.scene = scene;
      view.camera = camera3D;

      //view.renderer = Renderer.BASIC;
      //view.renderer = Renderer.CORRECT_Z_ORDER;
      //view.renderer = Renderer.INTERSECTING_OBJECTS;

      addChild(view);

      modelContainer = new ObjectContainer3D();
      scene.addChild(modelContainer);
    }

This is all it takes to setup an Away3D Engine. A scene, a flarcamera, and a view. All we do is setup a new instance of each, attach the scene and camera to the view, then add the view to the DisplayList via addChild. The last thing we'll do is build an empty ObjectContainer3D that will act as the root of all of our 3d primitives and Objects and add it as a child to the scene we attached to our view. Using the modelContainer in this way will let us turn on and off all of the models in our Away scene easily; as well as pushing them around in 3d space on the augmented reality marker.

Another thing of note is the 3 lines I have commented out here. This was more for informational reasons. Away3D has 3 'render modes': Basic, Correct Z Order, and Intersecting Objects. And from my experience this is a scale of quality/speed. Basic is the default renderer, which tends to have issues with triangles showing through others. When you finally publish something at the end of this walkthrough, you'll probably notice right away that it seems like the cube is cutting into the plane or the plane is somehow showing through the cube. This is due to weak z-sorting in the basic renderer. Obviously, the best way to deal with this is by changing the renderer to Correct_z_order; HOWEVER, you incur a fairly major performance decrease. Especially with Augmented Reality. I saw it suggested before, though, to stick with the Basic renderer and instead 'cheat' the z-sorting by using .screenZOffset (in units of 1000s usually) to add artificial depth to your object. That's gotten the job done nicely for me, at least, without major performance issues.

protected function initHUD():void {
      // override HUD per project
    }

The initHUD piece is optional; mainly only to use if you wanted there to be a graphic overlay, something to frame your work in. ;) Also note it is a protected function -- for me, this is the case because I would have different overlays from piece to piece. Therefore, I override this function in another Class.

protected function initObjects():void{
      plane = new Plane({material:"red#", name:"plane", width:75, height:75});
      modelContainer.addChild(plane);

      cube = new Cube({material:"blue#", name:"cube", width:20, height:20, depth:20});
      modelContainer.addChild(cube);
    }

Our glorious plane and cube. So simple, yet.. so complex. Well, okay, so not really complex.. We just make a call for a new Plane and a new Cube, specify our init Object parameters and voila. Primitives. Add each to the modelContainer (which is the root of our 3D scene) and we'll move on.

Again, note that this is a protected function. There's not a whole lot you want to do with standard primitives, unfortunately (although there is lots to do!) so this will eventually be overridden by another function to load DAEs or MD2s from 3DS Max or whatever modeling package you use. Soon, I hope to post a follow-up to this on generating MD2s from 3DS Max using QTip and porting model and animations onto the AR surface.

private function initListeners():void{
      stage.addEventListener(Event.ENTER_FRAME, loop);
   
      // begin listening for FLARMarkerEvents.
      flarManager.addEventListener(FLARMarkerEvent.MARKER_ADDED, onMarkerAdded);
      flarManager.addEventListener(FLARMarkerEvent.MARKER_REMOVED, onMarkerRemoved);
   
      stage.addEventListener(Event.RESIZE, onResize);
      onResize();
    }

Our essential list of listeners. The first listener is attached to our stage and set to run on ENTER_FRAME. This is our primary loop that will control the rendering of the 3d objects to the screen as well as keeping us updated with the tracking position of our FLARMarker.

The FlarManager instance we created long ago will be listening for our FLARMarkerEvents; specifically when a marker is seen in the camera and when it is taken away. They have aptly named functions. ;)

The last listener is for the stage, which we'll use to make sure the modelContainer and view get repositioned if the swf is resized. From what I can tell, this is still borked when previewing your movie in the flash ide. So don't expect to be able to resize your window and it work.

public function killListeners():void {
      stage.removeEventListener(Event.ENTER_FRAME, loop);
      flarManager.removeEventListener(FLARMarkerEvent.MARKER_ADDED, onMarkerAdded);
      flarManager.removeEventListener(FLARMarkerEvent.MARKER_REMOVED, onMarkerRemoved);
      stage.removeEventListener(Event.RESIZE, onResize);
    }
  
    public function resetListeners():void {
      initListeners();
    }

In the case of these 2 functions, I can't tell you that they are absolutely necessary. I actually just built them in case I needed them. So far I haven't had a reason to completely stop/freeze a simulation. But I suppose if you wanted to be able to, this would be a starting point for doing it. So leave it in or take it out, it's up to you.

protected function loop(e:Event):void { 
      if (activeMarker) {
        modelContainer.transform = FLARAwayGeomUtils.convertFLARMatrixToAwayMatrix(activeMarker.transformMatrix);
        update();
      }
   
      view.render();
    }

    protected function update():void {
      //additional override for loop during active marker state.
    }

This is the major loop that occurs at every frame. So what we're doing here is checking to see if there is an activeMarker on the screen (flagged my the FLARMarkerEvent). If there is, we convert the transformMatrix of the Marker to a transform for our modelContainer. We're also firing an update() function that, here, does nothing. But again, this was built with extending in mind, so the update could be overridden to add in functionality that should happen when a marker is active.

Lastly, we force the Away3D View to render. This will handle all of the redraw calculations that put 3d to the scene.


private function onResize(e:Event = null):void{
      view.x = stage.stageWidth / 2;
      view.y = stage.stageHeight / 2;
    }

Here is our function that handles the initial and following calls to Resize the Stage. All we're doing is making sure the view for our scene stays center screen.

private function onMarkerAdded (e:FLARMarkerEvent) :void {
      activate();
      modelContainer.visible = true;
      scene.addChild(modelContainer);
      activeMarker = e.marker;
    }

    protected function activate():void {
      //override 
    }

The first function here handles the listener attached to our Marker. It is responsible for turning on our root 3d object(modelContainer) and setting the activeMarker to the marker detected.

The activate function is an optional override for any future functionality that should occur when a marker is added to the screen.

private function onMarkerRemoved (e:FLARMarkerEvent) :void {
      deactivate();
      modelContainer.visible = false;
      scene.removeChild(modelContainer);
      activeMarker = null;
    }

    protected function deactivate():void {
      //override
    }
  }
}

And to finish things off, a final function to remove the marker and modelContainer. As before, we have a deactivate function for future functionality as an override.

You'll note that each of the marker events changes the visible property of the modelContainer as well as adding and removing it from the stage. This may be redundant and, therefore, unnecessary. I like to call it a safety option. :P

So, there it is. All of A3DFLAREngine.as complete and ready to go. All you need to do now is stick it in a folder called 'classes' outside of an .fla you create as your stage.

In your .fla you will set the document class to classes.A3DFLAREngine. I use a stage size of 640x480 and set my FPS to 60. Wouldn't it be nice if the Flash IDE would let us build swfs without .flas? ;)