Knowledge-Aware Reasoning over Multimodal Semi-structured Tables

About

Existing datasets for tabular question answering typically focus exclusively on text within cells. However, real-world data is inherently multimodal, often blending images such as symbols, faces, icons, patterns, and charts with textual content in tables. With the evolution of AI models capable of multimodal reasoning, it is pertinent to assess their efficacy in handling such structured data.

This study investigates whether current AI models can perform knowledge-aware reasoning on multimodal structured data. We explore their ability to reason on tables that integrate both images and text, introducing {\sc {\sc MMTabQA}}, a new dataset designed for this purpose. Our experiments highlight substantial challenges for current AI models in effectively integrating and interpreting multiple text and image inputs, understanding visual context, and comparing visual content across images. These findings establish our dataset as a robust benchmark for advancing AI's comprehension and capabilities in analyzing multimodal structured data.

Dataset Details

we create the MMTabQA dataset with 69,740 questions over 25,026 tables by augmenting existing tables from four data sources:

• WikiSQL Dataset to benchmark model capabilities in parsing entities accurately and answering basic SQL-based questions.
• WikiTableQuestions dataset to include questions which require more complex reasoning.
• FeTaQA dataset to include long-form answer-based questions which involve multiple row/column reasoning.
• HybridQA dataset which includes extra contextual passages beyond the tables, requiring hybrid complex reasoning.

The dataset contains questions of the following types:

• Explicit Questions: which mention an entity that is replaced by an image in the table.
• Answer-Mention Questions: whose answer contains an entity that is replaced by an image in the table, but the question does not.
• Implicit Questions: where an image-replaced entity is involved in intermediate reasoning but not mentioned in the answer or the question.
• Visual Questions: Questions that involve visual aspects of entities. These are created by synthetically augmenting the explicit questions in the dataset with visual attributes of the entity they refer to.

Dataset Creation

The figure below shows the steps followed for creating the dataset:

Dataset Examples

Experimental Results

People

The MMTabQA dataset is prepared by collaboration of across multiple institutions IIIT Hyderabad, IIT Guwahati, UCLA, UPenn and UNC, Chapel Hill by the following people:

Citation

Please cite our paper as below if you use the InfoSync dataset.

@misc{mathur2024knowledgeawarereasoningmultimodalsemistructured,
	title={Knowledge-Aware Reasoning over Multimodal Semi-structured Tables}, 
	author={Suyash Vardhan Mathur and Jainit Sushil Bafna and Kunal Kartik and Harshita Khandelwal and Manish Shrivastava and Vivek Gupta and Mohit Bansal and Dan Roth},
	year={2024},
	eprint={2408.13860},
	archivePrefix={arXiv},
	primaryClass={cs.CL},
	url={https://arxiv.org/abs/2408.13860}, 
}

Acknowledgement

Research was sponsored by the Army Research Office and was accomplished under Grant Number W911NF-20-1-0080. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. This work was partially funded by ONR Contract N00014-19-1-2620. Lastly, we extend our appreciation to the reviewing team for their insightful comments.