Creating and exploiting explicit links between multimedia fragments

Linkmedia is concerned with the processing of extremely large collections of multimedia material. The material we refer to are collections of documents that are created by humans and intended for humans. It is material that is typically created by media players such as TV channels, radios, newspapers, archivists (BBC, INA, BnF, …), as well as the multimedia material that goes through social-networks. It also includes material that includes images, videos and pathology reports for e-health applications, or that is in relation with e-learning which typically includes a fair amount of texts, graphics, images and videos associating in new ways teachers and students. It also includes material in relation with humanities that study societies through the multimedia material that has been produced across the centuries, from early books and paintings to the latest digitally native multimedia artifacts.

Multimedia collections are rich in contents and potential, that richness being in part within the documents themselves, in part within the relationships between the documents, in part within what humans can discover and understand from the collections before materializing its potential into new applications, new services, new societal discoveries, …  That richness, however, remains today hardly accessible due to the conjunction of several factors originating from the inherent nature of the collections, the complexity of bridging the semantic gap or the current practices and the (limited) technology. What makes the processing of multimedia collections difficult are e.g. their multimodal nature with a complex blending of texts, images, video and audio and the fact that documents are intertwined since they do not exist in isolation one from the other but rather form a collection. Furthermore, the scale of collections challenges the cost and the quality of any algorithm that runs analysis tasks.

The ambition of Linkmedia is to propose foundations, methods, techniques and tools to help humans make sense of extremely large collections of multimedia material.

Research Areas

Linkmedia follows two main directions of research that are

• extracting and representing information from the documents in collections, from the relationships between the documents and from what user build from these documents,
• facilitating the access to documents and to the information that has been elaborated from their processing. Machine learning, AI, computer vision, multimedia analytics, natural language processing, information retrieval and data mining are domains where Linkmedia does research.

Fact following team:  TEXMEX