JavascriptGraphicsAPI

From Sirikata Wiki
Jump to: navigation, search

Contents

Rationale for a common API to 3d graphics systems

Objects are sent across the thread barrier to alter the current scene graph being displayed--here's why:

In modern engines, graphics framerates should not be tied to physics framerates and networking events and decoding of said events should happen at the correct pace to keep up with the networknig adapter. Graphics, however, is tied to the DOM and therefore must be on the main thread. This forces both networking and physics to be on webworker threads if there is to be any threading.

These threads need to advertise state changes to the main graphics thread so that the scene graph may be altered at the graphics rate. This requires that the individual physics and networking threads send timestamped events to the graphics system which drive changes to it.

Since graphics can run at different rates and the updates from the network may be irregular, the graphics (main) thread needs to have a smooth interpolation scheme interpolating the current position with the timestamped updates sent by the network/physics thread(s). The interpolation scheme should use cubic interpolation using the most current update's position and orientation along with the displayed position and location of the object when that update was received to provide a smooth scheme.

Below are some example objects that may be sent cross-thread. The objects are listed in JSON format so the type information should be clear from the example.

API To Graphics System

Graphics should provide a constructor method that takes in a callback and a parent DOM element and returns a class that has a send(obj) method that takes in serializable objects from other threads that modify graphics state and an optional "destroy" method which cleans up graphics state in the DOM.

so a sequence of code to construct an graphics system, make an object, and destroy it could look like

gfx= new GLGERenderer(callbackFunction,parentElement)
gfx.send({  msg:"Create", id:"f47ac10b-58cc-4372-a567-0e02b2c3d479", time: 2181298451298491284, pos:[1,2,3], orient:[.5,0,0,.5]})
gfx.destroy();

Cross thread communication from Physics and Networking to graphics

Object Management

id's can be anything from human readable strings to uuids to integers. They just each must be unique and chosen by the user of the API

Creating a new graphics object

{
 msg:"Create"
 id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
 spaceid:"aaaaaaaa-bbbb-cccc-dada-134234ab98",//<-- optional (defaults to the empty space, 0)
 time: 2181298451298491284,//milliseconds since 1970
 pos:[1,2,3],
 vel:[.25,0,0],
 orient:[.5,0,0,.5]
 rotaxis:[0,0,1]
 rotvel:.25,
 scale:[0,0,0,1],//defaults to 0,0,0,1 if absent, but first 3 elements offset the mesh from the pivot point and last scales it
 parent:"c46ac00b-58cc-4372-a567-0e02b2c3d479",//<-- optional (defaults to empty--toplevel if absent)
 parentbone:"Hand"//name of the bone on the parent object that this is attached to. Assume root transform otherwise
}

Moving a graphics object

//should we define that the graphics system has some sort of interp--otherwise velocity may be useless?

Should we use "parentbone" or "attachment_point"?

FIXME: We should default these values to the last position, not to identity. Otherwise we basically have to send everything each time.

{
  msg:"Move"
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479"
  time:39852398592385,//milliseconds since 1970
  pos:[1,2,3],
  vel:[.25,0,0],
  orient:[.5,0,0,.5],//defaults to (identity) if absent
  rotaxis:[0,0,1],//defaults to 0,0,1 if absent, forcing rotvel to 0 
  rotvel:.25,//defaults to 0 if absent
  scale:[0,0,0,1],//defaults to 0,0,0,1 if absent: first 3 values offset mesh from pivot, last scales it
  interpolate:true,//set to false if the object should snap to new position
  parent:"c46ac00b-58cc-4372-a567-0e02b2c3d479"//<-- optional (defaults to previous state if absent, to clear pass empty string)
  attachment_point:"Hand"//name of the bone on the parent object that this is attached to. Defaults to previous state if absent, to clear pass empty string
}

Destroying a graphics object

{
   msg:"Destroy"
   id:"f47ac10b-58cc-4372-a567-0e02b2c3d479"
}


Managing object appearance properties

Adding/changing mesh property for an object

{
  msg:"Mesh",
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
  type:"collada",//string that specifies file format
  mesh:"http://example.com/test.dae",//the mesh should be rescaled to fit inside a unit sphere centered at 0,0,0
}

Querying mesh aspect ratio for an object

{
  msg:"QueryMeshAspectRatio",
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
}

Message from the graphics system to respond to mesh aspect ratio query

{
   msg:"MeshAspectRatio",
   aspect:[.025,.75,.661] // this should reflect the absolute bounding box if scale were identity [0,0,0,1.0]
}

Updating shader property (Vertex and Fragment float4) for an object

{
  msg:"MeshShaderUniform"
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
  name:["ColorTint","HowManyIterations"]
  value:[[.24,.24,.25,1.0],[1,0,0,0]]
  type:["float4", "int4"]
}

Removing mesh property for an object

{
   msg:"DestroyMesh",
   id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
}

Adding/changing light property for an object

{
  msg:"Light"
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479"
  diffuse_color:[.25,.5,1],
  specular_color: [.2,1,.5],
  power=1.0: //exponent on the light
  ambient_color: [0,0,0],
  light_range: 1.0e5
  constant_falloff: 0.5,
  linear_falloff: 0.2,
  quadratic_falloff: 0.1,
  cone_inner_radians: 0,
  cone_outer_radians: 0,
  cone_falloff: 0.5,
  type: "POINT",//options include "SPOTLIGHT" or "DIRECTIONAL"
  casts_shadow: true,
  shader:"http://www.example.com/pointLight.shader"// light shader for ray tracing (empty by default)
}

Removing light property for an object

{
    msg:"DestroyLight"
    id:"f47ac10b-58cc-4372-a567-0e02b2c3d479"
}

Camera Management

Creating camera properties on an object

{
   msg:"Camera"
   id:"f47ac10b-58cc-4372-a567-0e02b2c3d479"
}

Destroying and cleaning up a camera

{
   msg:"DestroyCamera"
   id:"f47ac10b-58cc-4372-a567-0e02b2c3d479"
}

Attach a camera to an object's texture"

{
  msg:AttachCamera",
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479", // Camera object's id
  texobjid:"9a10e9c1-31fb-43e8-9a20-6545d9a62fdb", // Id of object with a mesh
  texname:"example.png"//overwrites this texture on the texobjid object.
}

Attach a camera to a render target"

{
  msg:AttachCamera",
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479", // Camera object's id
  target:0//writes to this framebuffer--- 0 for left ,1 right for stereo, etc.
}

Detach a camera from its render target"

{
  msg:DetachCamera",
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479", // Camera object's id
}

Skeleton Management

Streaming some joint locations

{
  msg:"AnimateBone",
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
  animation:"uniqueAnimationIdentifier",//so this movement can be associated with one animation and blended with others
                                        //if not specified this is a hard constraint (i.e. foot is glued to a wall in order to avoid penetrating it)
  weight:1.0,//the weight for prospective blending, defaults to 1.0
  time:1250120951209510295;//milliseconds since 1970
  bone:["ankle","arm"]
  pos:[[1,2,3],[2,3,4]]
  vel:[[.25,0,0],[0,0,0]]
  orient:[[.5,0,0,.5],[1,0,0,0]]
  rotaxis:[[0,0,1],[0,1,0]]
  rotvel:[.25,0],
  interpolate:true//if false then the bone should snap to the location unless smooth is set (in which case it should interpolate as quickly as possible) defaults to true
}

Event handling

Mouse Events

Messages from the graphics system for standard browser events:

{
   msg:"mousemove",
   event:{...},// implementation-specific event object.
   x:100,//X coordinate of the mouse, relative to canvas
   y:102,//Y coordinate of the mouse, relative to canvas
   clientX:110,//X coordinate of the mouse, relative to client area of window
   clientY:302,//Y coordinate of the mouse, relative to client area of window
   spaceid:"loop://localhost",//Space ID of the view the mouse is hovering over.
   camerapos:[0,0,0],//Start of the mouse ray.
   dir:[1,0,0],//Direction of the mouse ray.
   shiftKey:false,//See keyup/keydown
   ctrlKey:false,//See keyup/keydown
   altKey:false,//See keyup/keydown
}
{
   msg:"mousedown",
   event:{...},
   button:0,//right mouse button = 2, left mouse button = 0
   x:100,
   y:102,
   clientX:110,
   clientY:302,
   spaceid:"loop://localhost",
   camerapos:[0,0,0],
   dir:[1,0,0],
   shiftKey:false,
   ctrlKey:false,
   altKey:false,
}
{
   msg:"mouseup",
   event:{...},
   button:0,//right mouse button = 2, left mouse button = 0
   x:100,
   y:102,
   clientX:110,
   clientY:302,
   spaceid:"loop://localhost",
   camerapos:[0,0,0],
   dir:[1,0,0],
   shiftKey:false,
   ctrlKey:false,
   altKey:false,
}
{
   msg:"wheel",
   event:{...},
   dy:40,//Basic unit of scroll==120, may be smaller on e.g. mac trackpads with wheel acceleration.
   dx:0,//Left-right scroll
   shiftKey:false,
   ctrlKey:false,
   altKey:false,
}

A click occurs after a mouseup event when the mousedown and mouseup are in close proximity. Contains the same properties as mouseup.

{
   msg:"click",
...
}

A drag occurs after each mousemove between a mousedown and a mouseup. Contains the same properties as mousemove, but also has deltas:

{
   msg:"drag",
...
   dx:5,//How many pixels did the mouse move since mousedown
   dy:10,//How many pixels did the mouse move since mousedown
}

A drop occurs after mouseup, if the mouse has moved (drag events fired). Contains the same properties as click, but also has deltas:

{
   msg:"drop",
...
   dx:5,//How many pixels did the mouse move since mousedown
   dy:10,//How many pixels did the mouse move since mousedown
}


As an alternative to the Raytrace request, we introduce pick message that will return additional 3D data. Since raytracing is an expensive operation, it is fired only on mousedown.

This message also contains a copy of fields from the mousedown event, so applications can choose to use this event instead of mousedown.

{
   msg:"pick",
...
   id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",//which object was hit? May be null.
   pos:[1,2,3],//where on the surface did it hit? May be null.
   normal:[.5,0,.86],//what's the direction of the normal is at that point? May be null.
}

To save traffic we allow messages to the graphics system that enable/disable these messages:

{
   msg:"Enable",
   type:"mousemove"//message type
}
{
   msg:"Enable",
   type:"drag"//message type
}
{
   msg:"Enable",
   type:"pick"//message type
}

More advanced messages, such as pickover, pickout, and others can be derived from the messages described above and we should keep this list as simple as possible. For example, pickover and pickout can be derived from mouseover and pick messages (when object ID under cursor changes than it's a pickover for the new object and pickout for the old one).

Moreover, we can use mouseover, mouseout to enable/disable picking to reduce inter-thread traffic. (FIXME: what are mouseover and mouseout? They don't seem to be implemented or used.)

Keyboard

For a description of keyCode and other properties, see the KeyEvent page from MDC.

glgegfx gives keyDown for each auto-repeat. For example: keyDown,keyDown,...,keyDown,keyUp.

This messages are sent from the graphics system in response to user's actions towards keyboard:

{
   msg:"keydown"//key was pressed, but not released
   altKey:true,
   metaKey:false,
   ctrlKey:true,
   shiftKey:true,
   repeat:false,//true if this keyDown is due to a keyboard repeat.
   keyCode:25//See MDC KeyEvent for list of codes
}
{
   msg:"keyup",//key was released
   altKey:true,
   metaKey:false,
   ctrlKey:true,
   shiftKey:true,
   keyCode:25//See MDC KeyEvent for list of codes
}

Requesting intersection

Request to the graphics system:

{
  msg:"Raytrace",
  id:5,//request ID
  pos:[2,3,4],//origin of the ray
  dir:[.24,.33,.5],//direction of the ray
  multiple:true//if false, only return first hit, otherwise return all intersections
  infinite:false//if false use length of dir to specify ray length
}

Response from the graphics system:

{
  msg:"Intersections",
  id:5,//request ID
  results:[{
    pos:[2,3,4],
    normals:[0,1,0],
    id:"f47ac10b-58cc-4372-a567-0e02b2c3d479"
  },{
    pos:[2.23,3.32,4.49],//positions of the points of intersections
    normal:[.5,0,.86],//normals of the surface at intersections points
    id:"a33ff133-58dd-2272-dd6a-12aadc31d173"//object IDs for each intersected surface
  }]
}

Experimental/Brainstorming ideas for the API

I decided to reserve a section of the wiki for sort of bleeding edge ideas of cool features that would be nice to have. I could have put that in the "talk" page, but I think it makes more sense here so that it will get wider exposure. These are meant to be things that would help in drawing real scenes and building real VW systems but that we haven't figured out a good API to yet.

Customizing the background

{
 msg:"Background"
 spaceid:"aaaaaaaa-bbbb-cccc-dada-134234ab98",//<-- optional (defaults to the empty space, 0)
 time: 2181298451298491284,//milliseconds since 1970
 curtextures=["http://example.com/a.jpg","http://example.com/b.jpg"]
 background_type="skydome"#skydome[1 or 2 textures], skybox [6 textures]
 prevtextures=["http://example.com/old_a.jpg","http://example.com/old_b.jpg"]#optional for fade in effects
 curweight=.75#weight of curtextures=.75 versus prevtextures=.25
}


Attaching UI elements to graphics objects

The UI will naturally need to be in HTML since that's the best established cross platform, sandboxed UI system.

The user may specify a 3d location, orientation and scale for a UI dialog to be. The graphics system should do its best to scale and position the UI in the appropriate place, but the UI may be restricted to always face the camera and always be horizontal compared with the bottom of the screen on many system. The UI should not be displayed if it is completely invisible from the camera angle or smaller than 10 pixels.

Creating/Updating UI Element

{
  msg:"IFrame"
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
  uri: "http://example.com"
}

Destroying UI Element

{
  msg:"DestroyIFrame"
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
}


It seems like there should be a manner aside from "embedded iframes" to get art defined in the DOM into the scene graph--perhaps the canvas tag is the way to go here? But maybe that's too webGL specific and won't work for an Ogre port of this

Attaching 3d Text to an Objects

I'm just brainstorming here: it seems like WebGL has facilities to do this efficiently, but I don't have a good use case except buildnig a rendering system inside a canvas tag or something?

Perhaps the canvas tag is the way to go

{
  msg:"Text",
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
  text:"This is a test of the emergency broadcast system",
  font:"size=+1"
}
{
  msg:"DestroyText",
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
}

Particle System

Adding a particle system to an object

This mimics the ogre interface and we introduce a number of billboard types

point
The default arrangement, this approximates spherical particles and the billboards always fully face the camera.
oriented_common
Particles are oriented around a common, typically fixed direction vector (see common_direction), which acts as their local Y axis. The billboard rotates only around this axis, giving the particle some sense of direction. Good for rainstorms, starfields etc where the particles will traveling in one direction - this is slightly faster than oriented_self (see below).
oriented_self
Particles are oriented around their own direction vector, which acts as their local Y axis. As the particle changes direction, so the billboard reorients itself to face this way. Good for laser fire, fireworks and other 'streaky' particles that should look like they are traveling in their own direction.
perpendicular_common
Particles are perpendicular to a common, typically fixed direction vector (see common_direction), which acts as their local Z axis, and their local Y axis coplanar with common direction and the common up vector (see common_up_vector). The billboard never rotates to face the camera, you might use double-side material to ensure particles never culled by back-facing. Good for aureolas, rings etc where the particles will perpendicular to the ground - this is slightly faster than perpendicular_self (see below).
perpendicular_self
Particles are perpendicular to their own direction vector, which acts as their local Z axis, and their local Y axis coplanar with their own direction vector and the common up vector (see common_up_vector). The billboard never rotates to face the camera, you might use double-side mater

For further documentation about the properties see http://www.ogre3d.org/docs/manual/manual_35.html#SEC191 and http://www.ogre3d.org/docs/manual/manual_36.html#SEC208


{  
   msg:"ParticleSystem",
   id:"f47ac10b-58cc-4372-a567-0e02b2c3d479"
   mesh:"http://example.com/billboard.dae"//the mesh should be rescaled to be a 1x1 mesh with 
   particle_size:[20,20],
   cull_each:false
   quota:10000
   billboard:"oriented_self",
   sorted:false//defaults to false--whether the particles should be sorted
   local:false//defaults to false--if true rotation of the node after the emission of the particle will rotate it
   direction: [0,0,1],///the common direction for oriented_common or perpendicular_common
   up: [0,0,1],///Only required if billboard_type is set to perpendicular_self or perpendicular_common, this vector is the common up vector used to orient all particles in the system.
   accurate_facing:false//if the facing is set to the camera facing or calculated per billboard
   iteration_interval:.125//how often the particles are updated--if set to 0, defaults to framerate
   invisibility_timeout:10//how many seconds of being outside the frustum before the system stops updating
}
{
  msg:"DestroyParticleSystem"
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479"
}


once a system is created, particles need to be emitted from it. There should be a global map of default emitters named ParticleEmitters consisting of at least "Point","Box","Cylinder","Ellipsoid","Shell","Ring" and the extra attributes are specified in http://www.ogre3d.org/docs/manual/manual_38.html

{
  msg:"ParticleEmitter",//add or upate a particle emitter
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
  name:"Flare"
  type:"Ring"
  angle 15
  emission_rate 75
  time_to_live:[2.5,3]//range between 2.5 and 3
  direction [0, 1, 0]//3d vector
  speed:[250,300]//range between 250 and 300
  colour_range:[[1 0 0],[0 0 1]]//random color
  position:[0,0,0],
  repeat_delay:[2.5,5]
  
}
{
   msg:"RemoveParticleEmitter",
   id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
   name:"Flare"
}

There may be forces applied to the emitters and there must be a global map of affectors called ParticleAffector from which the relevent affector is selected consisting of at least

LinearForce, ColourFader, Scaler, Rotator, ColourInterpolator, ColourImage, DeflectorPlane, DirectionRandomiser The detailed definitions are contanied at http://www.ogre3d.org/docs/manual/manual_40.html#SEC234



{
  msg:"ParticleAffector",//add or upate a particle emitter
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
  name:"TheForce"
  type:"LinearForce"
  force_vector: [0 -100 0]
  force_application: "add"
 
}
{
   msg:"RemoveParticleAffector",
   id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
   name:"TheForce"
}

Specifying a terrain for the world

Would need to be chunked and in some sort of widely readable format---would be nice to be able to tap into google earth for terrain--ideas for how to do this are still very very early

Deprecated API Ideas

Here we put ideas we had but decided to discard so that they don't come up again as new ideas and may be discussed here and evaluated for re-addition if someone feels strongly they should be included


Skeleton file formats

The reason these were removed is that they are too brittle (it's hard to weigh an wave and walk animation and have the steps not be half as wide) and it's difficult to keep the skeletons out of trouble (i.e. feet through the ground) so we think thta the physics system in general should send the bone positions and timestamps since it's the arbiter of what intersects what--and it can always read the skeleton file format.

Animating a skeleton based on a time based animation

{
   msg:"Ani",
   id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
   time:489192048120984102,///milliseconds since 1970 that the animation should be started from (skip frames if now is later)
   animation:"http://example.com/animation.dae",
   loop:false,
   weight:1.0 ///how strong this animation should compare with other animations that use the same bones
   fadein:2.3 //how many seconds to fade in
}

Note that the animation.dae should have annotations for loop-in point and loop-out point within the .dae so that loop can intelligently function

Stopping a skeleton based on a time based animation

{
   msg:"AniStop",
   id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
   animation:"http://example.com/animation.dae"
   fadeout:1.0//how many seconds to fade out
}


Should an object just be a sprite

We figured that a collada square file may be a more compat representation for a sprite and can contain the appropriate shader, materials, etc

Making an object a point sprite

This mimics the ogre interface and we introduce a number of billboard types

point
The default arrangement, this approximates spherical particles and the billboards always fully face the camera.
oriented_self
Particles are oriented around their own direction vector, which acts as their local Y axis. As the particle changes direction, so the billboard reorients itself to face this way. Good for laser fire, fireworks and other 'streaky' particles that should look like they are traveling in their own direction.
perpendicular_self
Particles are perpendicular to their own direction vector, which acts as their local Z axis, and their local Y axis coplanar with their own direction vector and the common up vector (see common_up_vector). The billboard never rotates to face the camera, you might use double-side mater

For further documentation about the properties see http://www.ogre3d.org/docs/manual/manual_35.html#SEC191 http://www.ogre3d.org/docs/manual/manual_36.html#SEC208

{
  msg:"Sprite",
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
  sprite:"http://example.com/test.jpg",//the mesh should be rescaled to fit inside a unit sphere centered at 0,0,0
  billboard:"perpendicular_self"
  sorted:false//defaults to false--whether the particles should be sorted
  up: [0,0,1],///Only required if billboard_type is set to perpendicular_self, this vector is the common up vector used to orient all particles in the system.
}

Removing point sprite property from object

{
  msg:"DestroySprite",
  id:"f47ac10b-58cc-4372-a567-0e02b2c3d479",
}