Ad-hoc Multimodal Physical Interaction Environments

The goal of the project is to develop a system that allows easily using and converting real world objects into input and control devices to enrich and ease human machine interaction and to provide a more realistic and satisfying user experience.

The Proposal is targeting the following objectives:
· To create an environment/toolkit that allows the ad-hoc usage of physical objects for interaction purpuses and thereby transform them into interaction elements.
· To create such an environment that will be easy to setup and therfore overcome the difficulties in creating physical interfaces.
· To research and develop methods that are able to distinguise, interpret and understand the users hand motions and gestures when interacting with these physical objects.
· To integrate sound detection and recognition into the interaction with physical objects. The focus hereby is on sounds occuring naturally during the interaction and not on specially designed sounds.
· To develop an interaction alphabet, that provides basic entities for the formation of an interaction paradigma for the interaction with physical objects and to study consequences of this transformations on the interaction and the user experience
· To evaluate the concept in two different test application domains: games and office
· To provide guidelines based on empirical research findings for the implementation of such systems.

We propose the following research program:

Development of tracking facilities based on computer vision technology & sound detection
The basic idea is to use simple markers that are applied to objects so they can be tracked realiable and in realtime by use of cameras and computer vision technology. As the system doesn't need any cables and stickers can be applied to almost all rigid objects the range of possible input objects is very broad. To allow the detailed use of an object as input device the system also needs to know the objects dimensions. A 3d-model of the object either can be scanned, imported from 3d-modelling software such as e.g. AutoCAD. Also a simple modelling workbench integrated into the system should be developed within the project.

Interactive functionality mapping system
The goal of this project part is to develop an easy to understand and use mechanism that allows users to link the physical objects with interface functionalities and/or elements. We want to build a framework that allows defining relations according to several relevant criteria. Initial examples of such criteria are e.g. if one, two or three dimensions are used for the interaction, if rotation of the object (and again around which axis) is relevant, if the object has sensitive surfaces (similar to buttons), which gestures can be used with the object, if certain sound effects are related to an object, etc.

Research and Evaluation of Interaction Patterns
In a next step the developed and prototyped interaction alphabet will be systematically researched and evaluated with regard to utility, intuitiveness, learnability and practicability.

Test Application Scenarios
Within the project we want to test the approach in two distinct application domains: a games environment and an office setting. The two scenarios described below should help to clarify our vision:
• Games scenario: The user takes a broom, applyies some stickers to it, runs a short setup and then is able to use it as gun in his ego-shooter-game. The player also defined a gesture to reload his gun by tapping on an ammunition belt (his normal belt), to change the type of ammunition by tapping a special location on the "gun" etc.
• Office scenario: different objects are used to control system functionalities. For example, the placing of a small cubes control the online/offline status information of the user (when the cube is placed on the red field the user is in offline status, when placed on green field his status is online), etc.

multimodal interaction, vision based interface

- Game developers
- Experts in multimodal information integration

Game developers: We are looking for a partner that provides input into the systems requirements from the games persepective, to think about game concepts and ideas and most important to test applications and games developed with this new way of interaction

Experts in multimodal information integration: this partner should combine and integrate the data stemming from different modalities and recognition systems i.e. video analysis and sound detection.

Game Developers, Universities, Research Organistions,

No