Qooqle is a mobile based system that frees us from the confines of search boxes by integrating digital information directly into our everyday activities. In combining mobile, cloud computing, and social search, Qooqle aims to make information more accessible and devices less visible.

Vision    Applications    Video    Contact                                                            Li Bian     Information Ecology    MIT Media Lab

From ENIAC to personal computers, from tablets to smart phones, computers have evolved a long way to become what they are today. The shrinking size of computer screens has posted an increasing challenge on Human-Computer Interaction designers. At the same time, with the compact packaging of all kinds of sensing capabilities, such as proximity sensing, gesture and speech recognition, today's mobile devices have become our gateway to channeling ourselves to the world and provide a whole new domain of search through reality mining.

In the past decade, search has evolved from ranking web pages to ranking people in virtually linked online social media. In the real world, we are already "linked" through our daily engagement with one another. As such, social search needs to be redefined and tomorrow's intelligent search engine will not only crawl people's online activities but also listen to people's daily conversations, observe their casual gestures, and interpret their emotion state in the physical world. Although multi-modal user interfaces have long existed, most applications have only focused on allowing people to give commands to computers with designated speech and gestures.

Qooqle is a mobile-based system that frees people from the confines of search boxes by integrating digital information directly into their everyday activities. The Qooqle system listens to people's daily conversations and interprets their casual gestures. It combines these unstructured, dynamically generated data from the physical world with online social media data, in order to retrieve search results in a context-aware, personalized fashion. Moreover, Qooqle allows automated, passive search without always having the users to actively enter search queries.

The current Qooqle prototype records, stores, and transfers multi-media data through Xen Virtual Machines. Its MySQL database can link raw audio or video in the physical world to text or images in the digital world. Using Google's n-gram corpus database, the Qooqle system calculates the relevance between people's physical world conversations and their online social media stream and can retrieve relevant digital information automatically based upon people's physical world activities. Furthermore, Qooqle refines and enhances information annotation or tagging by capturing the context in sync with the content, such as people's emotion when they are sending a tweet or taking a picture. Such annotation not only helps the Qooqle search engine better filter information but also allows people at the information receiver side to obtain a more holistic view of the information.

1. In the Qooqle system, mobile phones can join our meetings and stay on all the time. They listen to our conversations and perform Continuous Speech Recognition. The transcribed text is compared by topic with tweets updated by people from our social network, based on Google n-gram corpus to calculate the relevance. When relevant tweets are found, the phones interrupt us and provide us with the relevant information through text-to-speech. We can tell the phones "stop" or "continue" to help them learn our interest level on the information provided.

We can also upload and retrieve tweets on the go without ever having to stare at the screen typing. To help the Qooqle system better understand us and search, we can upload our emotional state interpreted through gestures as well as location along with the tweet. We can also hear our friends' original voice, instead of synthesized voice, while retrieving tweets that link to the original audio file on the server.


2. More accurate tagging of photos improves the image search result. To help us more accurately and conveniently tag our photos, the Qooqle system allows us to express ourselves at the moment of taking the photos. Our speech, time and location information can be uploaded together with the photos onto photo-sharing sites such as Flickr. Later, when we compose an SMS message using speech, the system helps us create a companying photo album based on keyword matching and time stamp.


3. Same as for photos, we can also instantaneously annotate our life logging videos. In addition to location-based search, Qooqle analyzes our daily interactions (such as conversations during meetings) with one another, and retrieves videos for us based on who we have interacted with most frequently in the physical world.


4. With the Qooqle system, PowerPoint Presentations can now be more interactive. To help the audience more easily follow us, topic keywords can be instantly highlighted on the slides as we speak. We can create additional information for particular topics on the go, such as searching for an image, drawing visual diagrams for slides with statistical data. The audience no longer needs to receive information passively. They can speak or type questions with their phone; and the questions are sent to the presenter unobtrusively on the slides. With their collective intelligence, the audience can also respond to one another's questions throughout the presentation. This way, the Qooqle system creates a seamlessly connected and interactive experience for everyone at the presentation.



                                                                   Li Bian                                                                                                MIT Media Lab
                                                            libian@mit.edu                                                                               Information Ecology Group
                                                            www.chakuli.com                                                                                      E14-348U
                                                            (617) 335-3781                                                                                      Cambridge, MA