Skip to content

Let your hands do the talking!

March 2, 2012

Before reading this blog, check out this video (it may take a minute to load). At first glance, it appears to involve a man wildly conducting a robotic reading of Dr. Seuss’ Sam I Am. In a way, he is, although the robot he is conducting is really his own voice! The man in the video is Sageev Oore, a participant involved in current research being done at UBC in the Department of Electrical and Computer Engineering. The instrument being used in the video is called GloveTalkII, a device that actually creates an artificial vocal tract using the movements of the fingers, hands, and arms. This not only allows the user to integrate emotional aspects and more natural sounding pitch and intonation into their synthesized speech through gesture, but expands the range of possible vocabulary words to include anything the person could think of to say. These are amazing elements that enable the user to speak using volitional movements that mimic some of the natural movements of the vocal tract. For example, different contact points on the glove are used to create stop consonants, mirroring the tract closure that occurs in natural speech. Pitch is determined by the height of the hand, an intuitive mapping of high and low pitch. Gestures are used to code the manner, place and voicing as well.

As if this device were not incredible enough, the system uses neural network processing, which means that as the user is learning how to create vowels and consonants using movements, the system is learning from the user as well! Vowel sounds are determined by user defined movements, and the consonant network is trained “based on user generated examples of phoneme sounds defined in an initial gesture vocabulary” (Fels & Hinton, 1998).

While the system is innovative and may represent the future of AAC for clients without the vocal tract requirements to produce natural speech, it is still in development. Issues of mobility, heat, bulkiness and comfort are still areas that need work, however, as future clinicians, it is exciting to keep an eye on technology that can open the world of voice and communication to our clients!

For more information, check out the Gestural Control of Digital Ventriloquized Actors page and the GloveTalkII website, where you can check out more videos of the system in action.

–Adele Courchesne

No comments yet

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: