Kinect Help Autistic Students in Classrooms; Learn English Using "Human Body" Kinect App; "ShakeID" Multi-User Device-Display Interaction Sensing b Microsoft Research

Microsoft Research has proposed a new approach to users' interaction with devices called "ShakeID," capable of associating touches with a specific user when two people interact with a display at the same time. Thus, it can deliver the appropriate content to each of them, based on the location they touched the screen. Based on the […]

Microsoft Research proposes ShakeID Multi-user device display interaction sensing

Microsoft Research has proposed a new approach to users' interaction with devices called "ShakeID," capable of associating touches with a specific user when two people interact with a display at the same time. Thus, it can deliver the appropriate content to each of them, based on the location they touched the screen.

Based on the idea that each person is holding a smartphone that can sense his/her movement, the system can be used for determining who is holding the phone and interacting with it.

"The system takes advantage of Kinect tracking data, and can even determine which hand is used to touch the phone, or can perform a log-out when the user distances from the Kinect sensor and the screen. "By comparing the motion of each phone in the scene with the motion of each user, the system can associate each phone to a specific user's hand," Microsoft Research explains.

"Next, by performing a coordinate transform from the 2D space of the display to the 3D camera space, the touches on the display are associated to users. Touches are thus associated to users and users are associated with devices they hold," MSR adds.

You can read the full proposal here.

While on Kinect, the video embedded below shows how the Kinect help autistic students in classrooms.

Now autism researchers, teachers and therapists are installing them in classrooms and clinics, reporting promising results for a fraction of the price of typical equipment. Could a teacher armed with a $300 Xbox and a $10 copy of Double Fine Happy Action Theater do as much good as months of intensive therapy?

"Nobody thought of it as a therapeutic device," said Marc Sirkin of Autism Speaks, a New York-based advocacy group. Earlier this spring, when he first got wind of computer engineering students at the University of Michigan hacking the Kinect to develop autism games, he bought a ticket on a red-eye flight to see for himself. "It turns out you don't have to look very far, you don't have to scratch very deep, to go, 'Wait a minute. There's something really cool here.' "

Microsoft's Radu Burducea stops short of calling the Kinect a therapeutic device, but says he hears every day about teachers and therapists adapting it in new and creative ways: math instruction, book criticism, counseling and physical coordination, for instance.

"We've lost control," he admitted, "and thank God that we have."

In another Kinect project, a new Kinect app called "Human Body," powered by natural user interface, helps children learn English. "It uses Drag and Drop algorithm for Kinect. The aim of this app is to select word, drag it and drop on the correct place."

"Learning English with this app is not boring, children are able to drag some word from the bottom of the screen, drag it and drop on the correct place. They are not sitting as traditional but the are moving in front of the device," writes the author on Channel9.

The features of "Human Body" Kinect application for Windows 7 features include:

  • Kinect Drag and Drop
  • NUI
  • Flying Text
  • Multidimensional

You can find more information about the application at the project page.

Here is the video demonstrating the app: