Our overall goal is to develop an interactive natural language system that infers the intended message underlying an information graphic (a non-pictorial graphic such as a bar chart or a line graph), provides an initial summary that includes the intended message along with notable features of the graphic, and then responds to follow-up questions from the user. Recognizing the intended message of an information graphic also has other applications. For example, as multimodal communication becomes more prevalent, we envision users engaging in interactive communication via text and graphics; an artificial agent will need to be able to recognize the intentions that the user wants to convey via his information graphics in order to respond appropriately.
Publications:
Sandra Carberry and Stephanie Elzer.
Exploiting Evidence Analysis in Plan Recognition.
Proceedings of International Conference on User
Modeling (UM-07), 2007. (received Springer Best Paper Award)pdf version
Seniz Demir, Sandra Carberry, and Stephanie Elzer. Effectively Realizing the Inferred Message of an Information Graphic. Proceedings of Recent Advanced in Natural Language Processing (RANLP), 2007.
Stephanie Elzer, Edward Schwartz, Sandra Carberry, Daniel
Chester, Seniz Demir, and Peng Wu.
A Browser Extension for Providing Visually Impaired Users
Access to the Content of Bar Charts on the Web.
Proceedings of International Conference on Web Information
Systems (WEBIST), 2007. pdf version
Stephanie Elzer, Nancy Green, Sandra Carberry, and James Hoffman. A Model of Perceptual Task Effort for Bar Charts and its Role in Recognizing Intention. International Journal on User Modeling and User-Adapted Interaction, 16(1), pp. 1-30, 2006. (Received James Chen 2006 Award for Best Paper.) pdf version
Stephanie Elzer, Sandra Carberry, and Seniz Demir. Communicaative Signals as the Key to Automated Understanding of Simple Bar Charts. International Conference on the Thoery and Application of Diagrams, 2006. (received Best Paper Award)
Stephanie Elzer, Sandra Carberry, Ingrid Zukerman, Daniel Chester, Nancy Green, and Seniz Demir. A Probabilistic Framework for Recognizing Intention in Information Graphics. Proceedings of the Nineteenth International Conference on Artificial Intelligence (IJCAI-05), 2005. pdf version
Stephanie Elzer, Sandra Carberry, Daniel Chester, Seniz Demir, Nancy Green, and Ingrid Zukerman. Exploring and Exploiting the Limitied Utility of Captions in Recognizing Intention in Information Graphics. Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL-05), pp. 223-230, 2005. pdf version
Stephanie Elzer, Nancy Green, Sandra Carberry, and James Hoffman. Incorporating Perceptual Task Effort into the Recognition of Intention in Information Graphics. Proceedings of the Third International Conference on the Theory and Application of Diagrams, 2004. pdf version
Sandra Carberry, Stephanie Elzer, Nancy Green, Kathleen McCoy, and Daniel Chester. Extending Document Summarization to Information Graphics. Proceedings of the ACL Workshop on Text Summarization, 2004 pdf version
Stephanie Elzer, Nancy Green, Sandra Carberry, and Kathleen McCoy. Extending Plan Inference Techniques to Recognize Intentions in Information Graphics. Proceedings of the 9th International Conference on User Modeling, pp. 122-132, 2003. pdf version
Stephanie Elzer, Nancy Green, and Sandra Carberry. Exploiting Cognitive Psychology Research for Recognizing Intention in Information Graphics. Proceedings of the Cognitive Society Conference, 2003. pdf version
Sandra Carberry, Stephanie Elzer, Nancy Green, Kathleen McCoy, and Daniel Chester. Understanding Information Graphics: A Discourse-Level Problem, Proceedings of SigDial, pp. 1-12, 2003. pdf version
Kathleen McCoy, Sandra Carberry, Tom Roper, and Nancy Green.
Towards
Generating Textual Summaries of Graphs. Proceedings of the
1rst
International Conference on Universal Access in Human-Computer
Interaction,
2001.