This project is developing a methodology for automatically summarizing information graphics (non-pictorial graphics such as bar charts and line graphs that convey attributes of entities and relationships among the entities). Although some information graphics are only intended to display data, the majority of information graphics that appear in newspapers, magazines, and formal reports are intended to convey a message. This message captures the graphic's overall content and can serve as a summary of the graphic for storage, indexing, and retrieval.
The underlying hypothesis of the project is that information graphics contain communicative signals that can be utilized to identify the message that the graphic conveys. One signal is the relative effort required for different perceptual tasks, since the easiest tasks are the ones that a viewer will naturally perform and that will contribute the most to the message conveyed by the graphic. To computationalize effort, the project is building on the graph comprehension work of cognitive psychologists to develop rules for estimating perceptual effort; these rules are validated via eyetracking experiments and used to rank possible perceptual tasks that might be performed on an information graphic. A second kind of signal is provided by design choices, such as graphic type, coloring, annotations, an exploded wedge in a pie chart, etc., that draw attention to certain entities and relations in a graphic. A third kind of signal is provided by elements of the graphic's caption. Although captions are of limited utility due to their general nature or ill-formedness, the project is pursuing a shallow analysis of the caption to extract elements that serve as communicative signals.
The communicative signals provide evidence about the graphic's message. This evidence is entered into a Bayesian network that hypothesizes the graphic's message. As part of the research, a large corpus of information graphics are being collected and annotated with their identified messages. These annotated graphics are being used to compute the probability tables required by the Bayesian network. Currently the project has been limited to simple bar charts. Evaluation experiments have been conducted on simple bar charts and have demonstrated the effectiveness of our approach. Now, the project must consider line graphs and complex graphics such as grouped charts and composite graphs consisting of several interrelated graphics. The project will investigate using the graphic's messages to index them in a digital library and the retrieval of graphics based on user queries. The project will also explore the impact of information graphics on the summarization of multimodal documents consisting of both graphics and accompanying text.
This project is supported by the National Science Foundation under Grant No. IIS-0534948.
Publications:
Sandra Carberry and Stephanie Elzer.
Exploiting Evidence Analysis in Plan Recognition.
Proceedings of International Conference on User
Modeling (UM-07), 2007. (received Springer Best Paper Award)pdf version
Seniz Demir, Sandra Carberry, and Stephanie Elzer. Effectively Realizing the Inferred Message of an Information Graphic. Proceedings of Recent Advanced in Natural Language Processing (RANLP), 2007.
Sandra Carberry, Stephanie Elzer, and Seniz Demir.
Information Graphics: An Untapped Resource for Digital Libraries.
9th Annual International ACM SIGIR
Conference on Research &
Development on Information Retrieval, pp. 581-588, 2006. pdf version
Stephanie Elzer, Nancy Green, Sandra Carberry, and James Hoffman. A Model of Perceptual Task Effort for Bar Charts and its Role in Recognizing Intention. International Journal on User Modeling and User-Adapted Interaction, 16(1), pp. 1-30, 2006. (Received James Chen 2006 Award for Best Paper.) pdf version
Stephanie Elzer, Sandra Carberry, and Seniz Demir. Communicative Signals as the Key to Automated Understanding of Simple Bar Charts. (received Best Paper Award) International Conference on the Thoery and Application of Diagrams, 2006.
Stephanie Elzer, Sandra Carberry, Ingrid Zukerman, Daniel Chester, Nancy Green, and Seniz Demir. A Probabilistic Framework for Recognizing Intention in Information Graphics. Proceedings of the Nineteenth International Conference on Artificial Intelligence (IJCAI-05), 2005. pdf version
Stephanie Elzer, Sandra Carberry, Daniel Chester, Seniz Demir, Nancy Green, Ingrid Zukerman, and Keith Trnka. Exploring and Exploiting the Limitied Utility of Captions in Recognizing Intention in Information Graphics. Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL-05), pp. 223-230, 2005. pdf version
Stephanie Elzer, Nancy Green, Sandra Carberry, and James Hoffman. Incorporating Perceptual Task Effort into the Recognition of Intention in Information Graphics. Proceedings of the Third International Conference on the Theory and Application of Diagrams, 2004. pdf version
Sandra Carberry, Stephanie Elzer, Nancy Green, Kathleen McCoy, and Daniel Chester. Extending Document Summarization to Information Graphics. Proceedings of the ACL Workshop on Text Summarization, 2004 pdf version
Stephanie Elzer, Nancy Green, Sandra Carberry, and Kathleen McCoy. Extending Plan Inference Techniques to Recognize Intentions in Information Graphics. Proceedings of the 9th International Conference on User Modeling, pp. 122-132, 2003. pdf version
Stephanie Elzer, Nancy Green, and Sandra Carberry. Exploiting Cognitive Psychology Research for Recognizing Intention in Information Graphics. Proceedings of the Cognitive Society Conference, 2003. pdf version
Sandra Carberry, Stephanie Elzer, Nancy Green, Kathleen McCoy, and
Daniel
Chester. Understanding Information Graphics: A Discourse-Level
Problem,
Proceedings of SigDial, pp.
1-12, 2003. pdf
version