Simple Object Control With Voice

This post uses the HoloToolkit components to configure a basic scene with voice control on an object. Then modifies the components to add a new voice command to rotate or stop rotation of the game object.

See the two prior posts (focused object voice control and selected object voice control) if needed to prepare.

A video is here.

Advertisements

Simple Voice Control Using HoloToolkit’s SelectedObjectKeywordManager

This post starts off with the same code the previous post finished with, and then converts from a focused object to a selected object for keyword management.

Here’s a video that shows.

Simple Voice Control Using HoloToolkit’s FocusedObjectKeywordManager

This post assumes you’re familiar with the Holotoolkit, so I’ve bypassed some of the steps to import the HTK into your Unity project. If you’re a new user, see this readme for the initial steps to use Holotoolkit.

The goal is to be able to use voice commands to control a game object.

Here is a simple tutorial video, using the FocusedObjectKeywordManager.

The toolkit provides both a SelectedObject and FocusedObject KeywordManager. It comes with pre-wired commands ‘Make Bigger’ and ‘Make Smaller’.

I add keyboard commands to allow easy testing from Unity.

 

 

What to do with Hololens

Though my initial impression produced some disappointment, the past few days have been spent imagining uses where Hololens can enhance. And it creates some optimism to counter the disappointment.

While Hololens isn’t really for enhancing dynamically changing spaces (when driving or walking down a city street), it can enhance a static space in extraordinary ways.

For instance, imagine you are walking around an art museum and can now see holograms with informative displays next to existing paintings or other works of art. Maybe a hologram of Ben Franklin explaining the Constitution, or George Washington talking about an ax. A user could choose a particular style of art and get a guided tour through the museum, with a clear path to follow and prompts on where to go next, for instance if they were interested in paintings only from the 1600s. This type application can likely be mostly configured (not the Ben Franklin hologram, the path and informative displays could be) already. Also a user can easily display other similar art and quickly do side by side comparisons. Of course in a pure VR experience, there is no need for a museum at all, but Hololens is about MR rather than pure VR.

I imagined a Terminator-like experience, where the Hololens could detect, identify, track and enhance objects that came into view. This isn’t available, no tracking or object recognition yet, but there is ongoing work to bring in OpenCV software to do image recognition. Many people understand the value of this feature.

Rather the current Hololens experience is like opening a portal to a new world in your existing spaces. Very Harry Potter like, in being able to see what others can’t, but only if you put on the magical glasses called Hololens. So for gaming and self entertainment, the application is clear, create a 3D game environment around you, rather than just in front of you as in 2D gaming.

Another useful application that is being discussed on the Hololens forums is data visualization. One video of a 3D dataset, shows how it’s mapped to the Hololens format and displayed, and then the hologram is expanded and the user walks inside the hologram and views it from that perspective. I can imagine a doctor and patient both wearing Hololens(es) and viewing a 3D MRI or CAT scan of the patient.

In these three examples, (museum, open new visually stunning world, data visualization) Hololens opens new ways to communicate and learn. The holograms are good enough that this communication will feel natural and likely be very effective (visuals, audibles and perhaps some sense of feel will come soon enough). There are many other uses too, particularly for education and training, for faster and more effective comprehension. But alas, none of this comes for free, and that’s the current downside, which is the significant development efforts for the more visually stunning applications. But it’s true for most new development platforms, it just takes a while to ferret out the value from the chaff as the API evolves.