Revision as of 00:47, 31 May 2010

Rationale for a common API to 3d graphics systems

Objects are sent across the thread barrier to alter the current scene graph being displayed--here's why:

In modern engines, graphics framerates should not be tied to physics framerates and networking events and decoding of said events should happen at the correct pace to keep up with the networknig adapter. Graphics, however, is tied to the DOM and therefore must be on the main thread. This forces both networking and physics to be on webworker threads if there is to be any threading.

These threads need to advertise state changes to the main graphics thread so that the scene graph may be altered at the graphics rate. This requires that the individual physics and networking threads send timestamped events to the graphics system which drive changes to it.

Since graphics can run at different rates and the updates from the network may be irregular, the graphics (main) thread needs to have a smooth interpolation scheme interpolating the current position with the timestamped updates sent by the network/physics thread(s). The interpolation scheme should use cubic interpolation using the most current update's position and orientation along with the displayed position and location of the object when that update was received to provide a smooth scheme.

Below are some example objects that may be sent cross-thread. The objects are listed in JSON format so the type information should be clear from the example.

API To Graphics System

Graphics should provide a constructor method that takes in a callback and a parent DOM element and returns a class that has a send(obj) method that takes in serializable objects from other threads that modify graphics state and an optional "destroy" method which cleans up graphics state in the DOM.

so a sequence of code to construct an graphics system, make an object, and destroy it could look like

gfx= new GLGERenderer(callbackFunction,parentElement)
gfx.send({  msg:"Create", id:"f47ac10b-58cc-4372-a567-0e02b2c3d479", time: 2181298451298491284, pos:[1,2,3], orient:[.5,0,0,.5]})
gfx.destroy();

Cross thread communication from Physics and Networking to graphics

Object Management

id's can be anything from human readable strings to uuids to integers. They just each must be unique and chosen by the user of the API

Creating a new graphics object

{
 msg:"Create"
 id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
 spaceid:"aaaaaaaa-bbbb-cccc-dada-134234ab98",//<-- optional (defaults to the empty space, 0)
 time: 2181298451298491284,//milliseconds since 1970
 pos:[1,2,3],
 vel:[.25,0,0],
 orient:[.5,0,0,.5]
 rotaxis:[0,0,1]
 rotvel:.25,
 scale:[1,1,1],//defaults to 1,1,1 if absent
 parent:"c46ac00b-58cc-4372-a567-0e02b2c3d479",//<-- optional (defaults to empty--toplevel if absent
 parentbone:"Hand"//name of the bone on the parent object that this is attached to. Assume root transform otherwise
}

Moving a graphics object

//should we define that the graphics system has some sort of interp--otherwise velocity may be useless?

Should we use "parentbone" or "attachment_point"?

FIXME: We should default these values to the last position, not to identity. Otherwise we basically have to send everything each time.

{
  msg:"Move"
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479"
  time:39852398592385,//milliseconds since 1970
  pos:[1,2,3],
  vel:[.25,0,0],
  orient:[.5,0,0,.5],//defaults to (identity) if absent
  rotaxis:[0,0,1],//defaults to 0,0,1 if absent, forcing rotvel to 0 
  rotvel:.25,//defaults to 0 if absent
  scale:[1,1,1],//defaults to 1,1,1 if absent
  interpolate:true,//set to false if the object should snap to new position
  parent:"c46ac00b-58cc-4372-a567-0e02b2c3d479"//<-- optional (defaults to previous state if absent, to clear pass empty string)
  attachment_point:"Hand"//name of the bone on the parent object that this is attached to. Defaults to previous state if absent, to clear pass empty string
}

Destroying a graphics object

{
   msg:"Destroy"
   id:"f47ac10b-58cc-4372-a567-0e02b2c3d479"
}

Managing object appearance properties

Adding/changing mesh property for an object

{
  msg:"Mesh",
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
  mesh:"http://example.com/test.dae",//the mesh should be rescaled to fit inside a unit sphere centered at 0,0,0
  billboard:false//defaults to false, if set to true the object will stay aligned with camera
}

note if the billboard property is present and not false, it should be set to one of the following 3 strings This mimics the ogre interface and we introduce a number of billboard types

point
   This approximates spherical particles and the billboards always fully face the camera. 
oriented_self
   Particles are oriented around their own direction vector, which acts as their local Y axis. As the particle changes direction, so the billboard reorients itself to face this way. Good for laser fire, fireworks and other 'streaky' particles that should look like they are traveling in their own direction. 
perpendicular_self
   Particles are perpendicular to their own direction vector, which acts as their local Z axis, and their local Y axis coplanar with their own direction vector and the common up vector (see common_up_vector). The billboard never rotates to face the camera, you might use double-side mater

For further documentation about the properties see http://www.ogre3d.org/docs/manual/manual_35.html#SEC191 and http://www.ogre3d.org/docs/manual/manual_36.html#SEC208

Updating shader property (Vertex and Fragment float4) for an object

{
  msg:"MeshShaderUniform"
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
  name:"ColorTint"
  value:[.24,.24,.25,1.0]
}

Removing mesh property for an object

{
   msg:"DestroyMesh",
   id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
}

Adding/changing light property for an object

{
  msg:"Light"
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479"
  diffuse_color:[.25,.5,1],
  specular_color: [.2,1,.5],
  power=1.0: //exponent on the light
  ambient_color: [0,0,0],
  light_range: 1.0e5
  constant_falloff: 0.5,
  linear_falloff: 0.2,
  quadratic_falloff: 0.1,
  cone_inner_radians: 0,
  cone_outer_radians: 0,
  cone_falloff: 0.5,
  type: "POINT",//options include "SPOTLIGHT" or "DIRECTIONAL"
  casts_shadow: true
}

Removing light property for an object

{
    msg:"DestroyLight"
    id:"f47ac10b-58cc-4372-a567-0e02b2c3d479"
}

Camera Management

Attaching a camera to an object

{
   msg:"Camera"
   id:"f47ac10b-58cc-4372-a567-0e02b2c3d479"

Detaching a camera from an object

{
   msg:"DestroyCamera"
   id:"f47ac10b-58cc-4372-a567-0e02b2c3d479"
}

Attach a camera to an object's texture"

{
  msg:AttachCamera",
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479", // Camera object's id
  texobjid:"9a10e9c1-31fb-43e8-9a20-6545d9a62fdb", // Id of object with a mesh
  texname:"example.png"//overwrites this texture on the texobjid object.
}

Attach a camera to a render target"

{
  msg:AttachCamera",
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479", // Camera object's id
  target:0//writes to this framebuffer--- 0 for left ,1 right for stereo, etc.
}

Skeleton Management

Streaming some joint locations

{
  msg:"AnimateBone",
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
  animation:"uniqueAnimationIdentifier",//so this movement can be associated with one animation and blended with others
                                        //if not specified this is a hard constraint (i.e. foot is glued to a wall in order to avoid penetrating it)
  weight:1.0,//the weight for prospective blending, defaults to 1.0
  time:1250120951209510295;//milliseconds since 1970
  bone:["ankle","arm"]
  pos:[[1,2,3],[2,3,4]]
  vel:[[.25,0,0],[0,0,0]]
  orient:[[.5,0,0,.5],[1,0,0,0]]
  rotaxis:[[0,0,1],[0,1,0]]
  rotvel:[.25,0],
  interpolate:true//if false then the bone should snap to the location unless smooth is set (in which case it should interpolate as quickly as possible) defaults to true
}

Scene Queries

Requesting intersection

{
  msg:"RayTrace",
  id:5
  pos:[2,3,4],
  dir:[.24,.33,.5],
  multiple:true//if false, only return first hit
  infinite:false//if false use length of dir to specify ray length
}

Intersection callback

{
  msg:"Intersections",
  id:5
  pos:[[2,3,4],[2.23,3.32,4.49]]
  normals:[[0,1,0],[.5,0,.86]]
  id:["f47ac10b-58cc-4372-a567-0e02b2c3d479","a33ff133-58dd-2272-dd6a-12aadc31d173",
}

Experimental/Brainstorming ideas for the API

I decided to reserve a section of the wiki for sort of bleeding edge ideas of cool features that would be nice to have. I could have put that in the "talk" page, but I think it makes more sense here so that it will get wider exposure. These are meant to be things that would help in drawing real scenes and building real VW systems but that we haven't figured out a good API to yet.

Attaching UI elements to graphics objects

The UI will naturally need to be in HTML since that's the best established cross platform, sandboxed UI system.

The user may specify a 3d location, orientation and scale for a UI dialog to be. The graphics system should do its best to scale and position the UI in the appropriate place, but the UI may be restricted to always face the camera and always be horizontal compared with the bottom of the screen on many system. The UI should not be displayed if it is completely invisible from the camera angle or smaller than 10 pixels.

Creating/Updating UI Element

{
  msg:"IFrame"
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
  uri: "http://example.com"
}

Destroying UI Element

{
  msg:"DestroyIFrame"
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
}

It seems like there should be a manner aside from "embedded iframes" to get art defined in the DOM into the scene graph--perhaps the canvas tag is the way to go here? But maybe that's too webGL specific and won't work for an Ogre port of this

Attaching 3d Text to an Objects

I'm just brainstorming here: it seems like WebGL has facilities to do this efficiently, but I don't have a good use case except buildnig a rendering system inside a canvas tag or something?

Perhaps the canvas tag is the way to go

{
  msg:"Text",
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
  text:"This is a test of the emergency broadcast system",
  font:"size=+1"
}

{
  msg:"DestroyText",
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
}

Particle System

Adding a particle system to an object

This mimics the ogre interface and we introduce a number of billboard types

point
The default arrangement, this approximates spherical particles and the billboards always fully face the camera.
oriented_common
Particles are oriented around a common, typically fixed direction vector (see common_direction), which acts as their local Y axis. The billboard rotates only around this axis, giving the particle some sense of direction. Good for rainstorms, starfields etc where the particles will traveling in one direction - this is slightly faster than oriented_self (see below).
oriented_self
Particles are oriented around their own direction vector, which acts as their local Y axis. As the particle changes direction, so the billboard reorients itself to face this way. Good for laser fire, fireworks and other 'streaky' particles that should look like they are traveling in their own direction.
perpendicular_common
Particles are perpendicular to a common, typically fixed direction vector (see common_direction), which acts as their local Z axis, and their local Y axis coplanar with common direction and the common up vector (see common_up_vector). The billboard never rotates to face the camera, you might use double-side material to ensure particles never culled by back-facing. Good for aureolas, rings etc where the particles will perpendicular to the ground - this is slightly faster than perpendicular_self (see below).
perpendicular_self
Particles are perpendicular to their own direction vector, which acts as their local Z axis, and their local Y axis coplanar with their own direction vector and the common up vector (see common_up_vector). The billboard never rotates to face the camera, you might use double-side mater

For further documentation about the properties see http://www.ogre3d.org/docs/manual/manual_35.html#SEC191 http://www.ogre3d.org/docs/manual/manual_36.html#SEC208 and

{  
   msg:"ParticleSystem",
   id:"f47ac10b-58cc-4372-a567-0e02b2c3d479"
   mesh:"http://example.com/billboard.dae"//the mesh should be rescaled to be a 1x1 mesh with 
   particle_size:[20,20],
   cull_each:false
   quota:10000
   billboard:"oriented_self",
   sorted:false//defaults to false--whether the particles should be sorted
   local:false//defaults to false--if true rotation of the node after the emission of the particle will rotate it
   direction: [0,0,1],///the common direction for oriented_common or perpendicular_common
   up: [0,0,1],///Only required if billboard_type is set to perpendicular_self or perpendicular_common, this vector is the common up vector used to orient all particles in the system.
   accurate_facing:false//if the facing is set to the camera facing or calculated per billboard
   iteration_interval:.125//how often the particles are updated--if set to 0, defaults to framerate
   invisibility_timeout:10//how many seconds of being outside the frustum before the system stops updating
}

{
  msg:"DestroyParticleSystem"
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479"
}

once a system is created, particles need to be emitted from it. There should be a global map of default emitters named ParticleEmitters consisting of at least "Point","Box","Cylinder","Ellipsoid","Shell","Ring" and the extra attributes are specified in http://www.ogre3d.org/docs/manual/manual_38.html

{
  msg:"ParticleEmitter",//add or upate a particle emitter
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
  name:"Flare"
  type:"Ring"
  angle 15
  emission_rate 75
  time_to_live:[2.5,3]//range between 2.5 and 3
  direction [0, 1, 0]//3d vector
  speed:[250,300]//range between 250 and 300
  colour_range:[[1 0 0],[0 0 1]]//random color
  position:[0,0,0],
  repeat_delay:[2.5,5]
  
}

{
   msg:"RemoveParticleEmitter",
   id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
   name:"Flare"
}

There may be forces applied to the emitters and there must be a global map of affectors called ParticleAffector from which the relevent affector is selected consisting of at least

LinearForce, ColourFader, Scaler, Rotator, ColourInterpolator, ColourImage, DeflectorPlane, DirectionRandomiser The detailed definitions are contanied at http://www.ogre3d.org/docs/manual/manual_40.html#SEC234

{
  msg:"ParticleAffector",//add or upate a particle emitter
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
  name:"TheForce"
  type:"LinearForce"
  force_vector: [0 -100 0]
  force_application: "add"
 
}

{
   msg:"RemoveParticleAffector",
   id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
   name:"TheForce"
}

Specifying a terrain for the world

Would need to be chunked and in some sort of widely readable format---would be nice to be able to tap into google earth for terrain--ideas for how to do this are still very very early

Deprecated API Ideas

Here we put ideas we had but decided to discard so that they don't come up again as new ideas and may be discussed here and evaluated for re-addition if someone feels strongly they should be included

Skeleton file formats

The reason these were removed is that they are too brittle (it's hard to weigh an wave and walk animation and have the steps not be half as wide) and it's difficult to keep the skeletons out of trouble (i.e. feet through the ground) so we think thta the physics system in general should send the bone positions and timestamps since it's the arbiter of what intersects what--and it can always read the skeleton file format.

Animating a skeleton based on a time based animation

{
   msg:"Ani",
   id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
   time:489192048120984102,///milliseconds since 1970 that the animation should be started from (skip frames if now is later)
   animation:"http://example.com/animation.dae",
   loop:false,
   weight:1.0 ///how strong this animation should compare with other animations that use the same bones
   fadein:2.3 //how many seconds to fade in
}

Note that the animation.dae should have annotations for loop-in point and loop-out point within the .dae so that loop can intelligently function

Stopping a skeleton based on a time based animation

{
   msg:"AniStop",
   id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
   animation:"http://example.com/animation.dae"
   fadeout:1.0//how many seconds to fade out
}

Should an object just be a sprite

We figured that a collada square file may be a more compat representation for a sprite and can contain the appropriate shader, materials, etc

Making an object a point sprite