Unimore logo AImageLab

ELSA Benchmark Media Analytics

Overview

The objective of the Multimedia use case is to develop effective solutions for detecting and mitigating the spread of deep fake images in multimedia content. Deep fake images, which are highly realistic and deceptive manipulations, pose significant risks to privacy, security, and trust in digital media. By leveraging advanced techniques in computer vision and machine learning, participants in this competition aim to create robust and accurate models that can identify and flag instances of deep fake images.

The Multimedia use case will provide a positive impact to different research communities, ranging from the Multimedia and Computer Vision communities to the community active in Social Media analysis, which are all active on the topics of deepfake analysis and detection. These communities will benefit from the Use Case in terms of data release and algorithmic advancements, as the Use Case will include the generation and release of new datasets, the development of novel techniques and the definition of proper evaluation scenarios. Researchers and organizations within ELSA, the European Union and outside will be able to participate in the proposed challenge and contribute to the development of novel tools for deep fake analysis and recognition.

The competition will be deployed in periodical evaluation rounds and different versions of the dataset will be released going progressively to improve the quality, quantity, and type of image generation in order to provide a benchmark that is as representative as possible of image manipulation scenarios within the media.

General Rules

  1. Participants are required to train their models exclusively on the provided training dataset. The evaluation must be conducted on the provided test datasets.
  2.  If required, participants must provide all the necessary code for training, and pre-set random seeds to ensure accurate reproducibility. All submitted models and results are subject to re-evaluation.
  3. Models that have already been submitted can undergo reevaluation in subsequent rounds by submitting the predicted labels and scores for evaluation.

Track 1 - Classification of totally generated images

The dataset for the first round is composed of 1M fake images and 1M real images. Fake Images are generated from a subset of captions extracted from LAION 400M [2] using a stable diffusion model. The data are generated using two nodes of the Leonardo HPC facility each one equipped with 4x NVIDIA A100 gpus. The real images are extracted directly from LAION 400M. To download the dataset register to here.

  1. The task involves the recognition of deepfakes in which the entire image is generated at once.
  2. The binary classification task involves distinguishing between fake images and real images using machine learning and deep learning-based approaches.

Track 2 - XAI evaluation of deepfake detectors (starting soon) 

  1. The task will involve the evaluation of deepfake detector saliency maps.
  2. Evaluation will be conducted through both human supervision and automatic metrics like ADCC[1].

Track 3 - Classification of partially altered or generated images (starting soon) 

  1. The task involves the recognition of deepfakes in which both real images and generated images are partially edited.
  2. The binary classification task involves distinguishing between edited images and real images using machine learning and deep learning-based approaches.

Submissions file format

The submission must be uploaded as a JSON file. The keys are the file names of the test set and the values are the predicted class labels (either 0 or 1) The class label 0 identify the image as real and the class label 1 identifies the image as fake.

Metrics

  • F1 Score: this metric is computed as the harmonic mean of Precision and Recall, and it is particularly suited for evaluating in a single metric the binary classification performance on unbalanced datasets.

Acknowledge

[1] Samuele Poppi, et al: “Revisiting the Evaluation of Class Activation Mapping for Explainability: A Novel Metric and Experimental Analysis”, 2021 CVPRW;

[2] Christoph Schuhmann, et al: “LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs”, 2021;