Focused Attention: A Novel Mechanism for Dynamic Modality Weighting in Multimodal Transformers

Abstract

The paradigm that prevails in multimodal learning is that the compounding inputs are combined in onesequence and all tokens will be the same, not based on the modality or the relevance to the task. FocusedAttention proposes a new attentional mechanism with learnable focus weights to control attention scoresin the various modalities of input. The mechanism allows the models to automatically give importance to the models of information that