Skipping Spare Information in Multimodal Inputs during Multimodal Input Fusion

by Sun, Y., Shi, Y., Chen, F. and Chung, V.

Summary:

The article is based on the assumption that multimodal user interfaces that include natural communication modalities would enable the vast usage of new technologies. A technique called Multimodal Input Fusion (MMIF) is described as the combination of inputs from different modalities and interpreting them semantically. It is described that these inputs are symbol strings that constitute of individual recognized input elements from different modalities. As a whole, these symbol strings add up to multimodal utterance as defined by the authors.

The motivation to address the spare information problem has came to surface during the pilot study of Sun et al.'s previous attempt for discovering a flexible MMIF technique. It was observed that some inputs from different modalities do not have any semantic relation with some other inputs when present in the same multimodal utterance. This suggested the presence of spare information in multimodal inputs, which made it hard on the system while making semantic interpretations. Therefore, they proposed a MMIF approach that will leave out the spare information present, and make semantic interpretations of the rest of the inputs combined. They go on and explain the structure of the proposed multimodal input parser as a decomposition of the multimodal utterance into sub-utterances through a grammer called Multimodal Combinatory Catergory Grammar. After this decomposition, a matrix is created for recombination of the sub-utterances and the results with maximum symbols have priority going into the Dialogue Management Module, which is a decision module that outputs meaningful utterance when it is found.

Discussion:

The authors' approach for skipping spare information through the proposed parsing of multimodal utterances and recombining to meaningful ones seems to be an effective way for deriving semantic information. It is absolutely needed for multimodal computer-human interaction that is natural. However, when deciding for causes of spare information it seems that the authors did not take the possible pre-perceptions of users about the system or the cognitive load imposed upon them into account. When these have not been investigated throughly it seems that with increasing modalities included in the system, spare information would increase as well. This will seriously effect the recombination and decision process that is derived from a mutrix of all sub-utterances that is recognized by the individual input modality's engine.

Citation:

by Sun, Y., Shi, Y., Chen, F. and Chung, V. Skipping Spare Information in Multimodal Inputs during Multimodal Input Fusion. In IUI '09: Proceedings of the Intelligent User Interfaces Conference, Sanibel Islands, Florida, USA, 2009.