Two Microsoft interns devloped a new awesome prototype Android application that turns Android smartphone cameras into full-on spy machines. The app, called "TagSense," relies on smartphone sensors to automatically tag photographs with the identities and activities of whoever's in them. No human input is required.
The application debuted at the recent MobiSys convention in Washington.
"Although it also makes use of facial recognition technology, TagSense mainly relies on the myriad peripherals integrated into the average smartphone. Android phones' built-in accelerometers are used to determine the exact physical activity a picture subject is involved in, while light sensors in the camera are used to determine weather conditions outside in conjunction with location data obtained via GPS. The phones' microphones can then determine whether the picture subject is talking, laughing, crying, or silent. All this information is then tagged onto the photos," stated developers Chaun Qin and Xuan Bao.
But the kicker for TagSense is that it's designed to tap into information stored on other, nearby smartphones. The app will be able to interact with data stored on adjacent phones that're also using the app, enabling collaborative tagging and verification of photo subjects' identities. The creators claim that there'll be a strict opt-in requirement for use of that particular feature.
According to Qin, "phones have many different kinds of sensors that you can take advantage of. [...] They collect diverse information like motion, orientation, location, sound and light. By putting all that info together, you can sense the setting of a photograph and describe its attributes."
An excerpt from their technical paper (embedded below) gives an idea of the possibilities:
One may imagine improved image search in the Internet, or even within one's own computer--Bob may query his personal photo collection for all pictures of Alice and Eve playing together in the snow. Another application may tag videos with important event/activity markers; a user of this application may be able to move the video slider to the exact time-point where President Obama actually walks up to the podium, or starts speaking. Today, such functionalities may be available in select images and videos, where some humans have painstakingly tagged them. TagSense aims to automate this process via sensor-assisted tagging.
In the short term, apart from the app's creators having secured themselves very lucrative careers, all this'll only further the photo-tagging arms race between Facebook, Apple, Google, and Yahoo!/Flickr.
We've embedded below the full research paper for online reading, for offline, you can download the paper using the link under:
You can download the research paper here.