bbrel
burton bradstock research labs
current research and development
clastic
classification
of text-image combinations
current research and development
the combination of different media is a defining characteristic of multimedia but prior research has concentrated on analysing and structuring multimedia data types in isolation and integrating them, if at all, in ad-hoc ways. interpreting, combining and reasoning about the semantic content of multimedia data are critical for search, summarization and personalisation systems to access personal media collections, digital libraries and the web.
clastic will bring about a paradigm shift by pioneering research into how the whole of a multimedia document conveys meaning beyond the sum of its parts. this project will, for the first time, systematically research, classify and make use of the relations between verbal and visual media types in multimedia documents.
consider how a human understands differently a document in which the text describes or explains an image, and a document in which the image illustrates the text. there are similar differences between the understanding of news broadcasts in which the speech tells the story and the understanding of films where the dialogue relates much less to what can be seen. understanding the overall meaning of these, and many other kinds of multimedia combinations, requires recognising different kinds of text-image relations in order to properly integrate verbal and visual information.
selected publications
Hughes, M., Salway, A., Jones, G., and O'Connor, N. (2007), 'Analysing Image-Text Relations for Semantic Media Adaptation and Personalisation', SMAP 2007.
Martinec and Salway (2005), ‘A System for Image-Text Relations in New (and Old) Media ’, Visual Communication 4(3), 337-371.
Salway and Martinec (2005), ‘Some Ideas for Modelling Image-Text Combinations ’, Dept. of Computing Technical Report CS-05-02, University of Surrey.