The data ecosystems around us contain many different media types ranging from audio, image and video to time series, 3D data and textual (meta-)data available especially on (social) media platforms. Multimedia AI deals with the detection, retrieval, analysis, and recommendation of media data in unimodal or multimodal (when combining two or more different media data types) settings.

We carry out award-winning multimedia AI research in computer vision (CV), natural language processing (NLP), multimodal retrieval (MMR), and time series analysis & pattern analysis, which we apply inter alia in social media analysis, property market analysis, human gait analysis, digital heritage and digital humanities.

We support real estate companies with the satellite image-based assessment of land plot location quality and the age prediction of buildings based on real estate ad images. In cooperation with paleographers, digitised medieval manuscripts serve as the basis for the retrieval and identification of writers. To support archaeologists, we use 3D scans of rock surfaces as a basis for segmenting human-carved prehistoric figures out of the rocks.

We make use of NLP methods to analyse social media for tasks such as fake news detection or sexism detection (10). We employ multiple modalities for MMR in order to extract information from social media or to allow a more precise assessment of real estate. To support physiotherapists in the diagnosis of human gait deficits, we analyse time-based gait measurements in 2D and 3D to detect and classify characteristic patterns.