User:PatrickHorn/interaction notes

Brainstorm on interfacing with the virtual world, and how to provide as much flexibility as possible.

Some throughts

• Drag object
• Input to UI (if selected texture is a browser window)
  • Overlay UI: Attach to camera screen, become 2D
  • All UI elements must be allowed to become overlays;
  • untrusted Overlays must be allowed to be closed
• Select object
  • List of all selectable objects (Red border around all such objects)
  • Minimum r-size (if an object has no mesh or is invisible)
• Mouse cursor events
• Keep in mind tablet/ multi-input/ multi-touch support, since several consumer devices already support this!

Input devices

MOUSE/Joystick

• Want as few states as possible:

[Interaction Mode]

• CTRL "Object select mode". Upon selecting an object, a properties window appears.
• ALT "Camera rotate mode". Click and drag to rotate scene.
• CTRL+ALT "Camera rotate around object mode". Click and drag to rotate around selected object
• DEFAULT "Interface mode". Clicking interfaces only with UI elements.

[Avatar Mode]

• Mouse cursor disabled,
• Mouse fires relative events (moved up 10 pixels) rather than absolute events.

Allow UI to work without a keyboard:

1. Default clicking
2. Ctrl <=> hold down left button
3. Alt <=> hold down right button
4. Ctrl+Alt <=> hold down both OR middle click.
5. Scroll wheel: zoom?

Keyboard

[Interaction Mode]

• All keybaord keys talk to focused UI element
• Standard UI controls work:
  • Tab=next form/ Space=activate/ Enter=accept (these are all built into web browsers)
  • Escape = Close UI window (actually, maybe *hide*, not close so that you don't lose text input changes)
  • Keyboard = type into text box.
• Keyboard input should be able to function without an attached mouse

[Avatar Mode]

• arrow keys move and turn avatar.

Input system

• Each input device is separate.
• You can bind any number to controlling the actions of a ProxyObject, for example based on keyboard bindings.
• You can also link input devices together for interaction mode.

Avatar versus Interaction mode

How do we switch between these? Interaction mainly involves editing/altering the world, while Avatar is mainly about observing.

Perhaps these "modes" should not be separate at all. Should double clicking on a browser window activate it or bring it up in an overlay? Should dragging always rotate the camera?

Focus

I don't know if this is the best solution (from a usability perspective), but we can do everything by "focus". When your Avatar is focused, keyboard and mouse input (bound to the focus) controls the avatar. When a browser texture is focused, then keyboard and mouse bound to that focus goes to that browser window.

In either case, as long as these elements are within the world, it should be possible to interact with the world itself by changing focus or rotating the camera (around you or around the focused element--which one we choose depends on what feels more intuitive?)

On the other hand, if we are focused on the screen overlay, it can take control of everything, including right click and other UI actions. Exiting out of the overlay should be simple to do (Escape key, X button, Right click menu)

While I would personally prefer the idea of having a modeless UI, I have a suspicion that doing so will not be possible without sacrificing functionality in the UI/browser system. If we make the UI modal then we should try to make it as seamless as possible. Maybe display a border around the active element and make an easy way to return to your avatar.

I especially like the idea of making this as customizable as possible. If I have two mice/joysticks, I should be allowed to have one control the UI, and the other control my avatar. But if I only have a single mouse and no keyboard (or just a touchscreen like the IPhone), it should also be as easy to use as possible (with the exception of having to type in text, but that's a different problem). But either way the default should be optimized for one keyboard and one mouse.