AMASS++

Advanced Multimedia Alignment and Structured Summarization

Aims

The massive growth of digital multimedia repositories demands for innovative methods of information access. The aim of AMASS++ is to build cross-media and cross-lingual summaries of information and to present the information in a user-friendly way. The summaries rely on automatic content recognition of the visual and textual medium, that is reinforced by their mutual and integrated processing. An important aspect of the project is aligning content across media and languages, for which the project aims to develop innovative algorithms. Another important aspect of the project is the interactive presentation of the summarized broadcasted news to be used in an exploratory search.

Partners

LIIR coordinates the AMASS++ project. The other partners are: K.U.Leuven ESAT-Visics (Prof. Luc Van Gool, Prof. Tinne Tuytelaars ), K.U.Leuven CCL (Prof. Frank Van Eynde ), and University of Hasselt EDM (Prof. Karin Coninx).

Results(Demo)

The key contributions of the AMASS++ project are:

  • Alignment of content in images and text applied to Web content and video.
  • Advanced semantic processing of natural language text and images.
  • Advanced machine translation methods for the English and Dutch languages.
  • Novel interactive tools for exploratory search of multimedia databases, using advanced visualizations.
  • Summarization of multimedia and multilingual news sources.
The main realizations by LIIR (some of which were realized in collaboration with the Visics group) regard: 1) Linking of names and faces in Web images and captions, in video news broadcasts and in soap series; 2) Location and action annotation in soap series; 3) The Latent Words Language Model which has been proven valuable for synonym replacement in texts, word sense disambiguation and the automated creation of training examples for semantic annotation of texts; 4) Cross-modal segmentation of video news broadcasts and sports videos; 5) Summarization of sports videos; 6) Cross-lingual latent Dirichlet allocation models evaluated on cross-lingual clustering, categorization and retrieval.

We have organized the First Workshop on Cross-Media Information Access and Mining (CIAM 2009). This workshop took place July 13, 2009 in Los Angeles, CA as part of the Twenty-first International Joint Conference on Artificial Intelligence.



Period From 2007-05-01 to 2011-04-30.
Financed by Instituut voor de Aanmoediging van Innovatie door Wetenschap en Technologie in Vlaanderen - IWT (SBO-060051)
Supervised by Marie-Francine Moens
Staff Wim De Smet
Koen Deschacht
Phi The Pham
Gert-Jan Poulisse
Contact Wim De Smet

More information can be found on the project website http://www.cs.kuleuven.be/~liir/projects/amass/

Publications

  1. ANGHELUTA, Roxana & MOENS, Marie-Francine. Cross-document entity tracking. In G. Amati, C. Carpineto & G. Romano (Eds.) Advances in Information Retrieval - 29th European Conference on IR Research ECIR -Lecture Notes in Computer Science 4425 (pp. 670-674). Berlin: Springer. 2007
  2. DESCHACHT, Koen, MOENS, Marie-Francine and ROBEYNS, Wouter. Cross-Media Entity Recognition in Nearly Parallel Visual and Textual Documents, in Proceedings of the 8th RIAO conference on Large-Scale Semantic Access to Content (Text, Image, Video and Sound), Pittsburgh, USA, May 30-June 1 . 2007
  3. DESCHACHT, Koen and MOENS, Marie-Francine. Text Analysis for Automatic Image Annotation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, Prague, Czech Republic, June 23th-30th, 2007 . 2007
  4. MOENS, Marie-Francine. Words and Pictures: The Power of Information Extraction (paper of keynote lecture). In Proceedings of the 29th International Conference on Information Technology Interfaces. IEEE Computer Society. 2007
  5. DESCHACHT, Koen & MOENS Marie-Francine Finding the Best Picture: Cross-Media Retrieval of Content. In C. Macdonald, I.Ounis, V. Plachouras & I. Ruthven (Eds.) Proceedings of the 30th European Conference on Information Retrieval. Lecture Notes in Computer Science 4956 (pp.539-546). Springer. 2008
  6. POULISSE, Gert-Jan Using Interconceptual Relationships to Improve SVM Classification in News Video. In Eduard HOENKAMP, Martine DE COCK & Véronique HOSTE (Eds.), Proceedings of the 8th Dutch-Belgian Information Retrieval Workshop (pp. 81-86).Maastricht University. 2008
  7. PHAM, Phi The, DESCHACHT, Koen & MOENS, Marie-Francine Document Clustering with User Feedback. In Eduard HOENKAMP, Martine DE COCK & Véronique HOSTE (Eds.), Proceedings of the 8th Dutch-Belgian Information Retrieval Workshop (pp. 73-80).Maastricht University. 2008
  8. MARTENS, Scott, BECKER Jan Hendrik, TUYTELAARS Tinne & MOENS, Marie-Francine Multimodal Data Collection in the AMASS++ Project. In Proceedings of the LREC 2008 Workshop on Multimodal Corpora. 2008
  9. BOIY, Erik, DESCHACHT, Koen & MOENS Marie-Francine Learning Visual Entities and their Visual Attributes from Text Corpora In Proceedings of the 5th International Workshop on Text-based Information Retrieval . IEEE Computer Society Press. 2008
  10. PHAM, Phi The, MOENS, Marie-Francine & TUYTELAARS, Tinne Linking names and faces: Seeing the problem in different ways. In Proceedings of ECCV Workshop on Faces in Real-Life Images: Detection, Alignment and Recognition. 2008
  11. POULISSE, Gert-Jan and MOENS, Marie-Francine "Multimodal News Story Segmentation". In Proceedings of the First International Conference on Intelligent Human Computer Interaction (IHIC 2009) (pp. 95-101). New York: Springer. 2009
  12. ARIAS, Javier, DESCHACHT, Koen & MOENS, Marie-Francine Content Extraction from Multilingual Web Pages. In Aly R., Hauff, C. Hiemstra, D., Huybers, T. & De Jong, F. (Eds.), Proceedings of the 9th Dutch-Belgium Information Retrieval Workshop. University of Twente. 2009
  13. MOENS, Marie-Francine & HIEMSTRA, Djoerd Information Extraction and Linking in a Retrieval Context. In Mohand BOUGHANEM, Catherine BERRUT, Josiane MOTHE, Chantal SOULE-DUPUY (Eds.). Advances in Information Retrieval. Proceedings of the 31st European Conference on IR Research (Lecture Notes in Computer Science 5478). Berlin: Springer. 2009
  14. DE SMET, Wim & MOENS, Marie-Francine An Aspect Based Document Representation for Event Detection. In Proceedings of the 19th Meeting of Computational Linguistics in The Netherlands. 2009
  15. POULISSE, Gert-Jan, MOENS, Marie-Francine & DEKENS, Tomas News Story Segmentation In Multiple Modalities. In Proceedings of the 7th International Workshop on Content-Based Multimedia Indexing (CBMI 2009). IEEE Computer Society. 2009
  16. DESCHACHT, Koen & MOENS Marie-Francine The Latent Words Language Model. In Proceedings of the 18th Annual Belgian-Dutch Conference on Machine Learning (Benelearn 09). 2009
  17. De BELDER, Jan, DE SMET, Wim, MOCHALES PALAU, Raquel & MOENS, Marie-Francine Google Owns YouTube? Entity Relationship Extraction with Minimal Supervision. In Proceedings of SIM 2009 Joint Conference SRL ILP MLG. 2009
  18. DESCHACHT, Koen & MOENS Marie-Francine Using the Latent Words Language Model for Semi-Supervised Semantic Role Labeling. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP 2009). ACL. 2009
  19. MOENS, M.-F., FERRARI, V., TUYTELAARS, T. & VAN GOOL (Eds.) Proceedings of IJCAI-09 Workshop on Cross-media Information Access and Mining (CIAM 2009). AAAI. 2009
  20. DE Smet, W. & MOENS, M.-F. An Aspect Based Document Representation for Event Clustering. In Proceedings of CLIN 19 . University of Groningen. 2009
  21. DE SMET, Wim & MOENS, Marie-Francine Cross-Language Linking of News Stories on the Web Using Interlingual Topic Models. In Proceedings of the CIKM Workshop on Social Web Search and Mining ( SWSM 2009). New York: ACM. 2009
  22. PHAM, Phi The, MOENS, Marie-Francine & TUYTELAARS, Tinne Cross-Media Alignment of Names and Faces, IEEE Transactions on Multimedia, 12 (1), 13-27. 2010
  23. POULISSE, Gert-Jan, MOENS, Marie-Francine, DEKENS, Tomas & DESCHACHT, Koen News Story Segmentation in Multiple Modalities, Multimedia Tools and Applications (in press). 2010
  24. DE BELDER, Jan & MOENS, Marie-Francine Sentence Compression for Dutch Using Integer Linear Programming. In Proceedings of DIR 2010 10th Dutch-Belgian Information Retrieval Workshop (pp. 16-21). Radboud Universiteit Nijmegen. 2010
  25. PHAM, Phi The, MOENS, Marie-Francine & TUYTELAARS, Tinne Cross-Media Alignment of Names and Faces. In Proceedings of DIR 2010 10th Dutch-Belgian Information Retrieval Workshop (pp. 88-89). Radboud Universiteit Nijmegen. 2010
  26. KOLOMIYETS, O. & MOENS, M.-F. Recent Advances in Temporal Processing of Text. In Proceedings CLIN 20 Meeting. Utrecht University. 2010
  27. DE BELDER, Jan & MOENS, Marie-Francine Integer Linear Programming for Dutch Sentence Compression. In A. Gelbukh (Ed.). CICLing 2010 (Lecture Notes in Computer Science 6008) (pp. 711-723). Berlin: Springer. (Best Student Paper Award) 2010
  28. KORDJAMSHIDI, Parisa, VAN OTTERLO, Martijn, MOENS, Marie-Francine Spatial Role Labeling: Task Definition and Annotation Scheme. In Proceedings LREC 2010 - The Seventh International Conference on Language Resources and Evaluation. 2010
  29. MOENS, Marie-Francine Information Extraction from Text and Beyond. In Philip M. McCARTHY & Chutima BOONTHUM (Eds.), Applied Natural Language Processing and Content Analysis: Identification, Investigation and Resolution. Hershey, PA: IGI Global. 2010
  30. PHAM, Phi The, MOENS, Marie-Francine & TUYTELAARS, Tinne Weakly Supervised Person Naming in News Video. In Proceedings RIAO 2010: 9th International Conference on Adaptivity, Personalization and Fusion of Heterogeneous Information. Paris: CID. 2010
  31. PHAM, Phi The, MOENS, Marie-Francine & TUYTELAARS, Tinne Naming Persons in News Video. In Proceedings of the International Workshop on Visual Content Identification and Search (VCIDS 2010) - IEEE International Conference on Multimedia & Expo (ICME 2010). 2010
  32. KORDJAMSHIDI, Parisa,VAN OTTERLO, Martijn & MOENS, Marie-Francine. From Language towards Formal Spatial Calculi. In Proceedings of Computational Models of Spatial Language Interpretation at Spatial Cognition 2010. 2010
  33. ENGELS, Chris, DESCHACHT, Koen, BECKER, Jan Hendrik, TUYTELAARS, Tinne, VAN GOOL, Luc & MOENS, Marie-Fracine Automatic Annotation of Unique Locations from Video and Text. In Proceedings BMVC 2010 . 2010
  34. DE SMET, Wim, TANG, Jie & MOENS, Marie-Francine Knowledge Transfer across Multilingual Corpora via Latent Topics. In Proceedings of PAKDD 2011: The 15th Pacific-Asia Conference on Knowledge Discovery and Data Mining (Lecture Notes in Computer Science 6634) (pp. 549-560). Berlin: Springer. 2011
  35. POULISSE, Gert-Jan & MOENS, Marie-Francine Unsupervised Scene Detection in Olympic Video using Multi-modal Chains. In Proceedings of the 9th International Workshop on Content-Based Multimedia Indexing (CBMI 2011). IEEE Computer Society. 2011
  36. PHAM, Phi The, TUYTELAARS, Tinne & MOENS, Marie-Francine Naming Persons in News Videos with Label Propagation. IEEE Multimedia, 18 (3), 44-55. 2011
  37. KOLOMIYETS, Oleksandr, BETHARD, Steven & MOENS, Marie-Francine Model-Portability Experiments for Textual Temporal Analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (pp. 271-276). ACL. 2011
  38. VULIC, Ivan, DE SMET Wim & MOENS, Marie-Francine Identifying Word Translations from Comparable Corpora Using Latent Topic Models. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (pp. 479-484). ACL. 2011
  39. KORDJAMSHIDI, Parisa, VAN OTTERLO, Martijn & MOENS, Marie-Francine Spatial role labeling: Towards extraction of spatial relations from natural language. ACM Transactions on Speech and Language Processing (in press). 2011
  40. DESCHACHT, Koen, DE BELDER, J. & MOENS, Marie-Francine The Latent Words Language Model. Computer Speech and Language, 26 (5), 384-409. 2012
  41. VULIC, Ivan, DE SMET, Wim & MOENS, Marie-Francine Cross-Language Information Retrieval Models Based on Latent Topic Models Trained with Document-Aligned Comparable Corpora. Information Retrieval. DOI 10.1007/s10791-012-9200-5 2012
  42. DE SMET, Wim & MOENS, Marie-Francine Representations for Multi-Document Event Clustering. Data Mining and Knowledge Discovery, 26(3): 533-558. 2013
  43. HAESEN, Mieke, MESKENS, Jan, PHAM, Phi T., POULISSE, Gert-Jan, BECKER,Jan H., CONINX, Karin, LUYTEN, Kris, TUYTELAARS, Tinne & MOENS, Marie-Franciine Finding a Needle in a Haystack: An Interactive Video Archive Explorer for Professional Video Searchers. Multimedia Media Tools and Applications, 63 (2), 331-356. 2013


Back to all projects