Task-Oriented Search and Content Annotation for Media Production (EU ICT FP7)


TOSCA-MP aims to develop user-centric content annotation and search tools for professionals in networked media production and archiving (television, radio, online), addressing their specific use cases and workflow requirements. This will be achieved by scalable and distributed content processing methods performing advanced multimodal information extraction and semantic enrichment. Other key technology areas will include search methods across heterogeneous networked content repositories and novel user interfaces. An open standards based service oriented framework integrates the components of the system.

The tasks of LIIR regard advanced multimodal information extraction (with emphasis on information extraction from the texts of video transcripts) and the design of suitable retrieval models for searching heterogeneous contents.


In this project LIIR collaborates with the Joanneum Research Forschungsgesellschaft (coordinator), playence KG, Austria, Deutsche Thomson, Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung, Institut für Rundfunktechnik GmbH, Germany, Union Européenne de Radio-Télévision, Switzerland, VRT (Vlaamse Radio- en Televisieomroeporganisatie), Belgium, RAI-Radiotelevisione Italiana SpA, Fondazione Bruno Kessler, Italy.


We have witten an extensive state-of-the-art study on video search, summarization and video concept assignment. We have researched advanced methods for cross-media content recognition including segmentation of video based on text and visual features, alignment of names and faces, and animal names and visual animal patterns in video. We have built a model for improved named entity recognition in transcribed speech data. The TOSCA-MP project ranked 1st in technical and 2nd in economic impact in the MaxiCulture coordination action of the DigiCult area.

Period From 2011-10-01 to 2014-03-31.
Financed by EU FP7-287532
Supervised by Marie-Francine Moens
Staff Aparna Nurani Venkitasubramanian
Phi The Pham
Gert-Jan Poulisse
Niraj Shrestha
Contact Marie-Francine Moens

More information can be found on the project website


  1. PHAM, Phi The, TUYTELAARS, Tinne & MOENS, Marie-Francine Naming Persons in News Videos with Label Propagation. IEEE Multimedia, 18 (3), 44-55. 2011
  2. POULISSE, Gert-Jan, PATSIS, Yorgos & MOENS, Marie-Francine Unsupervised Scene Detection and Commentator Building Using Multi-modal Chains. Multimedia Tools and Applications, 70 (1), 159-175. 2014
  3. NURANI VENKITASUBRAMANIAN, Aparna & MOENS, Marie-Francine Selection of Search Facets. In Proceedings of CORIA 2013 - 10th French Information Retrieval Conference. 2013
  4. NURANI VENTIKASUBRAMANIAN, Aparna & MOENS, Marie-Francine Estimating the Breadth of Search Queries. In Proceedings of the Open research Areas in Information Retrieval Conference (OAIR 2013) (10th International Conference in the RIAO series). 2013
  5. PHAM, Phi The, DESCHACHT, Koen, TUYTELAARS, Tinne & MOENS Marie-Francine Naming Persons in Video: Using the Weak Supervision of Textual Stories. In Journal of Visual Communication and Image Representation, 24 (7), 944-955. 2013
  6. SHRESTHA, Niraj, VULIC, Ivan and MOENS, Marie-Francine An IR-Inspired Approach to Recovering Named Entity Tags in Broadcast News. In Proceedings of the 6th IRF Conference for Science and Industry (Lecture Notes in Computer Science 8201) (pp. 45-57). Springer. 2013
  7. DUSART, Thibaut, NURANI VENKITASUBRAMANIAN, Aparna & MOENS, Marie-Francine Cross-Modal Alignment for Wildlife Recognition. In Proceedings of the 2013 ACM International Workshop on Multimedia Analysis for Ecological Data. ACM. 2013
  8. SHRESTHA, N., NURANI VENKITASUBRAMANIAN, Aparna & MOENS, Marie-Francine Key Event Detection in Video using ASR and Visual Data. In Proceedings of the COLING Workshop on Vision and Language (VL'14). ACL. 2014
  9. SIMON, Anca-Roxana, GRAVIER, Guillaume, SEBILLOT, Pascale & MOENS, Marie-Francine IRISA and KUL at MediaEval 2014: Search and Hyperlinking Task. In Proceedings of MediaEval Benchmarking Initiative for Multimedia Evaluation (MediaEval 2014). 2014

Back to all projects