The Graphs Project

     The amount of information available electronically has increased dramatically over the past decade.  The challenge is to develop techniques for providing effective access so that all individuals can benefit from these resources and so that information is readily available when needed.    Unfortunately, many knowledge sources are provided in a single format and thus are not accessible to everyone.   For example, individuals with impaired eyesight have limited access to graphical displays, thus preventing them from fully utilizing available information resources. Although research has investigated alternative modes of presentation of graphical information for people who have visual impairments, such as converting a video image to a ``soundscape'' or by printing hard-copy that the user can decipher via touch, their focus is on rendering graphical elements in an alternative medium and they have serious limitations.  For example, it would be extremely difficult for a user to compare two related lines on a line graph via a soundscape.  The underlying hypothesis of our work is that alternative access to what the graphic  looks like is not enough --- the user should be provided with the message and knowledge that one would gain from viewing the graphic in order to enable effective and efficient use of this information resource.

Our overall goal is to develop an interactive natural language system that infers the intended message underlying an information graphic (a non-pictorial graphic such as a bar chart or a line graph), provides an initial summary that  includes the intended message along with notable features of the graphic, and then responds to follow-up questions from the user.  Recognizing the intended message of an information graphic also has other applications.   For example, as multimodal communication becomes more prevalent, we envision users engaging in interactive communication via text and graphics; an artificial agent will need to be able to recognize the intentions that the user wants to convey via his information graphics in order to respond appropriately.

Publications:

  Sandra Carberry and Stephanie Elzer.  Exploiting Evidence Analysis in Plan Recognition.  Proceedings of  International Conference on User Modeling (UM-07), 2007. (received Springer Best Paper Award)pdf version

Seniz Demir, Sandra Carberry, and Stephanie Elzer.   Effectively Realizing the Inferred Message of an Information Graphic. Proceedings of Recent Advanced in Natural Language Processing (RANLP), 2007.

  Stephanie Elzer, Edward Schwartz, Sandra Carberry, Daniel Chester, Seniz Demir, and Peng Wu.  A Browser Extension for Providing Visually Impaired Users Access to the Content of Bar Charts on the Web.  Proceedings of  International Conference on Web Information Systems (WEBIST), 2007. pdf version

  Stephanie Elzer, Nancy Green, Sandra Carberry, and James Hoffman.  A Model of Perceptual Task Effort for Bar Charts and its Role in Recognizing Intention.   International Journal on User Modeling and User-Adapted Interaction, 16(1), pp. 1-30, 2006. (Received James Chen 2006 Award for Best Paper.)     pdf version

  Stephanie Elzer, Sandra Carberry, and Seniz Demir.  Communicaative Signals as the Key to Automated Understanding of Simple Bar Charts.   International Conference on the Thoery and Application of Diagrams, 2006.  (received Best Paper Award) 

  Stephanie Elzer, Sandra Carberry, Ingrid Zukerman, Daniel Chester, Nancy Green, and Seniz Demir.  A Probabilistic Framework for Recognizing Intention in Information Graphics.   Proceedings of the Nineteenth International Conference on Artificial Intelligence (IJCAI-05), 2005.    pdf version

  Stephanie Elzer, Sandra Carberry, Daniel Chester, Seniz Demir, Nancy Green, and Ingrid Zukerman.  Exploring and Exploiting the Limitied Utility of Captions in Recognizing Intention in Information Graphics.   Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL-05), pp. 223-230, 2005.    pdf version

  Stephanie Elzer, Nancy Green, Sandra Carberry, and James Hoffman.  Incorporating Perceptual Task Effort into the Recognition of Intention in Information Graphics.   Proceedings of the Third International Conference on the Theory and Application of Diagrams, 2004.    pdf version

Sandra Carberry, Stephanie Elzer, Nancy Green, Kathleen McCoy, and Daniel Chester.   Extending Document Summarization to Information Graphics.  Proceedings of the ACL Workshop on Text Summarization, 2004     pdf version

Stephanie Elzer, Nancy Green, Sandra Carberry, and Kathleen McCoy.  Extending Plan Inference Techniques to Recognize Intentions in Information Graphics.  Proceedings of the 9th International Conference on User Modeling,  pp. 122-132, 2003.   pdf version

Stephanie Elzer, Nancy Green, and Sandra Carberry.  Exploiting Cognitive Psychology Research for Recognizing Intention in Information Graphics.  Proceedings of the Cognitive Society Conference, 2003.    pdf version

Sandra Carberry, Stephanie Elzer, Nancy Green, Kathleen McCoy, and Daniel Chester.  Understanding Information Graphics: A Discourse-Level Problem, Proceedings of SigDial, pp. 1-12, 2003.    pdf version

Kathleen McCoy, Sandra Carberry, Tom Roper, and Nancy Green.  Towards Generating Textual Summaries of Graphs.  Proceedings of the 1rst International Conference on Universal Access in Human-Computer Interaction, 2001.