Machine Understanding for interactive StorytElling (EU ICT FP7 FET)


The MUSE project introduces a new way of exploring and understanding textual information by “bringing text to life” through 3D interactive storytelling. Taking as input natural language text like children’s stories or medical patient education materials, MUSE processes natural language, translate it into formal knowledge that represents the actions, actors, plots and surrounding world by means of advanced machine learning techniques, and then render these as virtual 3D worlds in which the user can explore through interaction, re-enactment and guided game play.

MUSE’s long-term goal is to enable the innovative text-to-virtual-world translation system to be used in many applications and a variety of domains. Current information communicated to citizens is often very complex making written text difficult to understand. Comparable to the invention of symbolic writing systems several millennia ago, MUSE contributes to a novel symbolic system of communicating natural language utterances.

LIIR is involved in the coordination of the project and in the advanced natural language processing research.


LIIR coordinates the MUSE project. KU Leuven co-promotors are Prof. Steven Bethard (partially affiliated with the University of Boulder, USA) and Prof. Johan De Tavernier. The other partners are: Jožef Stefan Institute (Prof. Nada Lavrac), Teesside University (Prof. Marc Cavazza), Leiden University (Prof. Paul van den Broek) and Haute Autorité de Santé (Dr. Gersende Georg).

Results (Demo)

In the frame of the EU projects MUSE and TERENCE we have built an integrated system for semantic role labeling, noun phrase coreference resolution, event classification and temporal relation extraction (see Story Annotation Service). We have developed a novel, scalable framework for jointly learning a model for knowledge extraction from natural language text the validity of which has been proven for extraction of relations, their composing entities and attributes. We have built a novel model for mapping text and its semantic annotations to the knowledge representation used to construct the virtual world. In the framework of the MUSE project we have organized the SemEval 2015 task of SpaceEval and we are involved in the organization of the Workshop on Replicability and Reusability in Natural Language Processing: From Data to Software Sharing at the 2015 International Joint Conference on Artificial Intelligence, Buenos Aires, Argentina on July 27, 2015. The MUSE technologies will be demonstrated at ICT 2015 Innovate, Connect, Transform in Lisbon, Portugal on October 20-22 2015.

Period From 2012-09-01 to 2015-08-31.
Financed by EU FP7-296703 (FET-open call).
Supervised by Marie-Francine Moens
Staff Parisa Kordjamshidi
Steven Bethard
Quynh Do Thi Ngoc
Juan Carlos Gomez
Oleksandr Kolomiyets
Denis Lukovnikov
Contact Marie-Francine Moens

More information can be found on the project website


  1. KOLOMIYETS, Oleksandr & MOENS, Marie-Francine MotionML: Motion Markup Language: Shallow Approach for Annotating Motions in Text. In Proceedings of Corpus Linguistics. 2013
  2. KOLOMIYETS, Oleksandr, KORDJAMSHIDI, Parisa, BETHARD, Steven & MOENS, Maie-Francine SemEval-2013 Task 3: Spatial Role Labeling. In Proceedings of the Second Joint Conference on Lexical and Computational Semantics, Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (pp. 255-262). ACL. 2013
  3. KOLOMIYETS, Oleksandr & MOENS, Moens Marie-Francine KUL: A Data-Driven Approach to Temporal Parsing of Documents. In Proceedings of the Second Joint Conference on Lexical and Computational Semantics, Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (pp. 83-87). ACL. 2013
  4. KOLOMIYETS, O. & MOENS, M.-F. Towards Animated Visualization of Actors and Actions in a Learning Environment. In Proceedings of the 3rd International Workshop on Evidence Based and User centred Technology Enhanced Learning (EbuTEL 2013) (Advances in Intelligent and Soft-Computing) (in press). Springer. 2014
  5. DE MULDER, Wim, DO THI, Ngoc Quynh, VAN DEN BROEK, Paul & MOENS, Marie-Francine Machine Understanding for Interactive Storytelling. In Proceedings of the 8th International Conference on Knowledge, Information and Creativity Support Systems (KICSS'2013) (pp. 73-80). 2013
  6. KORDJAMSHIDI, Parisa & MOENS, Marie-Francine Designing Constructive Machine Learning Models based on Generalized Linear Learning Techniques. In Proceedings of the NIPS Workshop on Constructive Machine Learning. 2013
  7. KORDJAMSHIDI, Parisa & MOENS, Marie-Francine Structured Learning for Mapping Natural Language to Spatial Calculi Models. Abstract published in Proceedings of the NIPS Workshop: Knowledge Extraction from Text (KET). 2013
  8. KORDJAMSHIDI, Parisa & MOENS, Marie-Francine Global Machine Learning for Ontology Population. Journal of Web Semantics, special issue on Semantic Search, 1-29. 2015
  9. NGOC THI DO, Q., BETHARD, S. & MOENS, M.-F. Text Mining for Open Domain Semi-Supervised Semantic Role Labeling. In Proceedings of the First International Workshop on Interactions between Data Mining and Natural Language Processing (pp. 33-48). 2014
  10. DE MULDER, Wim, BETHARD, Steven & MOENS, Marie-Francine A Survey on the Application of Recurrent Neural Networks to Statistical Language Modeling. Computer Speech & Language, 30 (1), 61-98. 2015
  11. CERNY, Martin & MOENS, Marie-Francine aMUSE: Translating Text to Point and Click Games. In Proceedings of the 2015 AI Games Conference. 2015
  12. MOENS, Marie-Francine, DO THI, Ngoc Quynh, KORDJAMSHIDI, Parisa & LUDWIG, Oswaldo Visualising the Content of a Text in a Virtual World. In BUSCH C. & SIECK J. (Eds.), Kultur und Informatik: Cross Media (pp. 33-37). Verlag Werner Hülsbusch. 2015
  13. KORDJAMSHIDI, P., VAN OTTERLO, Martijn & MOENS, Marie-Francine Spatial Role Labeling Annotation Scheme. In Pustejovsky J. & Ide N. (Eds.), Handbook of Linguistic Annotation. New York: Springer (in press). 2015
  14. PUSTEJOVSKY, James, KORDJAMSHIDI, Parisa, MOENS, Marie-Francine, LEVINE, Aaron, DWORMAN, Seth & YOCUM, Zachary SemEval-2015 Task 8: SpaceEval. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015) (pp. 884-894) ACL. 2015
  15. LUDWIG, Oswaldo, DO THI, Ngoc Quynh & MOENS, Marie-Francine (2015). Mapping from Written Stories to Virtual Reality. In Proceedings of the 25th Meeting of Computational Linguistics in the Netherlands (p. 40). 2015
  16. HENS, Kim, DO THI, Ngoc Quynh & MOENS, Marie-Francine Automatic Conversion of an Episodes Script to a 3D Movie. In Proceedings of the 25th Meeting of Computational Linguistics in the Netherlands (p. 80). 2015
  17. DO THI, Quynh Ngoc , BETHARD, Steven & MOENS, Marie-Francine Domain Adaptation in Semantic Role Labeling Using a Neural Language Model and Linguistic Resources. IEEE/ACM Transactions on Audio, Speech and Language Processing (in press). 2015
  18. PEROVSEK, Matik, PODPECAN, Vid, KRANJC, J., ERJAVEC, T., Pollak, S., DO THI, Ngoc Quyhn, LIU, Xiao, SMITH, Cameron, CAVAZZA, Marc, LAVRAC, Nada (2015) Text mining platform for NLP workflow design, replication and reuse. In Proceedings of IJCAI Workshop on Replicability and Reusability in Natural Language Processing: From Data to Software Sharing. 2015
  19. DO THI, Quynh Ngoc, BETHARD, Steven & MOENS, Marie-Francine Adapting Coreference Resolution for Narrative Processing. In Proceedings of EMNLP 2015: Conference on Empirical Methods in Natural Language Processing. ACL. 2015
  20. DO THI, Quyhn Ngoc, BETHARD, Steven & MOENS, Marie-Francine Visualizing the Content of a Children’s Story in a Virtual World: Lessons Learned. In Proceedings of the EMNLP Workshop on Uphill Battles in Language Processing: Scaling Early Achievements to Robust Methods. 2016
  21. DO THI, Quyhn Ngoc, BETHARD, Steven & MOENS, Marie-Francine Facing the Most Difficult Case of Semantic Role Labeling: A Collaboration of Word Embeddings and Co-training. In Proceedings of the 26th International Conference on Computational Linguistics (COLING 2016). ACL. 2016
  22. LUDWIG, Oswaldo, DO THI, Quynh Ngoc, SMITH, Cameron, CAVAZZA, Marc & MOENS, Marie-Francine Learning to Extract Action Descriptions from Narrative Text. IEEE Transactions on Computational Intelligence and AI in Games. 2017

Back to all projects