Cycle-Consistent Adversarial Networks

Training and evaluating GANs for brain activity-based image reconstruction is typically a multi-step process involving pre-training on large image datasets, mapping fMRI data to a latent space, and using a combination of quantitative metrics and qualitative human assessment. 

Training Process

The training process usually involves a hybrid framework and can be broken down into three main stages: 

1. Pre-training the Generative Model:
   • A GAN (or more stable variants like StyleGAN or DGN) is first pre-trained on a large, generic dataset of natural images (e.g., ImageNet, CelebA for faces).
   • This step is crucial because the GAN learns the fundamental distribution and structure of real-world images, acting as a powerful “natural image prior”.
   • The pre-trained generator network is often the only part of the GAN architecture used in the final reconstruction phase; the discriminator is often discarded after this stage.
2. Training the Brain Activity Decoder:
   • Simultaneously, a separate decoder model is trained to map brain activity (fMRI) data to the latent space of the pre-trained GAN.
   • fMRI data is collected while subjects view or imagine images. These images’ features (e.g., VGG or CLIP features) are extracted using a separate, pre-trained deep neural network.
   • The decoder (often a linear model or another small neural network) learns the correlation between specific brain regions’ activity patterns and these image features.
3. Fine-tuning and Optimization:
   • In the final reconstruction phase, the process is an optimization challenge rather than a traditional, end-to-end GAN training.
   • For a given test fMRI sample, the goal is to find the optimal latent vector (input to the generator) that produces an image whose deep features (extracted by the “comparator” DNN) best match the features decoded from the fMRI data.
   • This optimization is guided by a loss function that often combines:
     • Feature Loss: Measures the distance between the features of the generated image and the decoded brain features in the DNN feature space (e.g., using MSE or correlation).
     • Adversarial Loss (optional): Some methods may fine-tune the GAN using an adversarial loss with the discriminator and the limited fMRI-image pairs to further refine realism for the specific subject’s data.
     • Pixel-wise Loss (optional): May include a term for direct pixel-wise similarity if ground-truth images are available during training/validation, although this is less common for imagined images. 

Evaluation Metrics

Evaluation involves both quantitative metrics and qualitative human judgment to assess the fidelity (realism) and diversity of the generated images. 

Quantitative Evaluation

• Pixel-level Metrics:
  • Mean Squared Error (MSE) / Root Mean Square Error (RMSE): Measures average pixel intensity differences from the ground truth. Useful for initial checks but may not correlate well with human perception.
  • Peak Signal-to-Noise Ratio (PSNR): Another measure of pixel-level similarity.
  • Structural Similarity Index (SSIM): Evaluates image quality by considering luminance, contrast, and structure, aligning better with human perception than MSE/PSNR.
• Distribution & Feature-level Metrics:
  • Fréchet Inception Distance (FID): A popular metric that calculates the distance between the feature distributions of real and generated images using a pre-trained Inception network. Lower FID scores indicate better quality and diversity.
  • Inception Score (IS): Measures the clarity (fidelity) and variety (diversity) of generated images. Higher scores are better.
  • CLIP Score: In methods using CLIP, this metric measures the semantic similarity between the generated images and their corresponding text descriptions or actual target images, providing a quantitative measure of semantic content. 

Qualitative Evaluation

• Human Subjective Assessment:
  • Researchers often employ human evaluators (sometimes via crowdsourcing platforms like Amazon Mechanical Turk) to rate images on realism, clarity, and resemblance to the intended target/imagined image.
  • Human evaluators might be asked to perform a “Turing test” – distinguishing generated images from real ones – or a “paired comparison” task to determine which of two reconstructions is better.
• Visual Inspection:
  • Direct manual inspection by researchers is a basic, but essential, way to check for visual artifacts, mode collapse, or the overall plausibility of the generated images.

Latest trends in Brain-Computer Interfaces (BCIs) using Deep Neural Networks (DNNs) and Generative Adversarial Networks (GANs) focus on significantly enhancing neural signal processing, decoding accuracy, and real-world application performance. Key trends include using GANs for data augmentation, improving non-invasive BCI performance with advanced DNN architectures, and developing hybrid Brain-AI systems for shared autonomy in complex tasks. 

Key Trends in BCI using DNNs and GANs

1. Data Augmentation and Synthesis using GANs

A significant challenge in BCI research is the scarcity of high-quality, labeled brain activity data (e.g., EEG or fMRI), which are expensive and time-consuming to collect. GANs are widely used to address this data limitation by: 

• Generating Synthetic Data: GANs generate synthetic, yet realistic-looking, brain signal data (e.g., EEG signals) to expand training datasets.
• Enhancing Model Robustness: This augmentation helps train more robust and generalized DNN models that perform better across different individuals and sessions, reducing issues like overfitting.
• Solving Mode Collapse: Researchers are developing methods to ensure the generated data is diverse and representative of real-world variations, avoiding “mode collapse” where the GAN only produces a limited variety of samples. 

2. Advanced Neural Decoding with DNNs

DNNs are central to decoding complex neural activity into meaningful commands or content. Trends include: 

• Multi-modal AI: Combining information from various sources like images, text, and different brain signals (e.g., fMRI and EEG) to create more comprehensive understanding and control.
• Real-time Processing: Development of energy-efficient neural networks, like Spiking Neural Networks (SNNs), to enable real-time processing and low-latency control for neuroprosthetics and other applications.
• Novel Architectures: Utilizing sophisticated DNN architectures (e.g., Graph Convolutional Networks, recurrent neural networks with two optimization steps) that can better learn the complex, non-linear relationships between brain activity and behavior or internal states (e.g., mood).
• Decoding Complex Information: Moving beyond simple movement commands to decoding complex information, such as the generation of unseen words from EEG signals for advanced communication aids. 

3. Hybrid Brain-AI Systems and Shared Autonomy

The combination of human intention (via BCI) and AI’s capabilities for task execution is a major trend, often referred to as “shared autonomy”. 

• AI “Copilots”: AI algorithms act as copilots to assist users in completing complex tasks, such as controlling a robotic arm, where the human provides high-level intent and the AI handles the fine-grained motor control.
• Improved Non-invasive BCI: Significant efforts are being made to improve the performance of non-invasive BCIs (which are safer and more accessible) for complex tasks, closing the gap with invasive methods through advanced AI algorithms. 

4. Image Reconstruction and “Brain-to-Content”

Building on the use of GANs for image generation, researchers are exploring “brain-to-content” technologies where brain activity is used to create rich media.

• Realistic Image Synthesis: Using GANs and Variational Auto-Encoders (VAEs) as strong “natural image priors” to reconstruct high-quality, realistic visual experiences (images and potentially videos) from fMRI data.
• Visualizing Mental Imagery: This trend aims to not just classify what a person is thinking, but to literally visualize their imagined images, offering new avenues for neuroscience research and potential applications in art or communication. 

5. System Stabilization and Domain Adaptation

Neural recordings can change over time, requiring frequent recalibration of BCI systems. Researchers are using methods like Cycle-Consistent Adversarial Networks (Cycle-GAN) to align the distribution of neural signals across different sessions, stabilizing system performance over time with minimal user effort.

• Improved image reconstruction from brain activity through automatic …Feb 9, 2025 — The trained decoder was then used to decode semantic features from the fMRI data obtained from the test sessions. Our p…National Institutes of Health (.gov)
• What Radiologists Need to Know – PMC – NIHAug 12, 2024 — 5. Performance Evaluation * 5.1. Quantitative Evaluation. (1) Metrics. In the field of medical imaging, several metric…National Institutes of Health (.gov)
• Applications of generative adversarial networks in neuroimaging …2.3. … A set of metrics has been used for evaluating the quality of data generated by GAN-based models. Mean square error (MSE),National Institutes of Health (.gov)
• On the evaluation of Generative Adversarial NetworksApr 6, 2022 — When measuring how well our GAN performs we need to evaluate two main properties: * Fidelity: the quality of the genera…Towards Data Science
• Deep Natural Image Reconstruction from Human Brain Activity …Nov 21, 2020 — Abstract. Brain decoding based on functional magnetic resonance imaging has recently enabled the identification of vis…National Institutes of Health (.gov)
• Double-Flow GAN model for the reconstruction of perceived …3.4 The training Pipeline * The whole training process are divided into three steps, 1) training the fMRI projector; 2) pretrainin…arXiv
• Reconstructing faces from fMRI patterns using deep generative … – NIHThe Generator network learns to convert latent 1024-D vectors from the latent space into plausible face images. The Discriminator …National Institutes of Health (.gov)
• Deep image reconstruction from human brain activity (Paper …May 24, 2020 — hi there today we’re looking at deep image reconstruction. from human brain activity by Guawa Shen tommoyasu horicima …YouTube·Yannic Kilcher1m
• Fréchet inception distance – WikipediaThe Fréchet inception distance (FID) is a metric used to assess the quality of images created by a generative model, like a genera…Wikipedia
• Synthesizing medical images using generative adversarial …Sep 14, 2020 — if you don’t know it may not be helpful but I’ll do it. anyway. so GANs are and sort of other kinds of implicit genera…YouTube·Institute for Advanced Study3m
• How to Implement the Inception Score (IS) for Evaluating GANsIn the paper, the authors use a crowd-sourcing platform (Amazon Mechanical Turk) to evaluate a large number of GAN generated image…Machine Learning Mastery
• VR-Splatting: Foveated Radiance Field Rendering via 3D Gaussian Splatting and Neural PointsOct 22, 2024 — The system is trained end-to-end in its hybrid form. Parameters of Gaussians and neural points are optimized via gradi…arXiv
• Generative Adversarial Network (GAN)Nov 28, 2024 — Generative Adversarial Networks (GANs) are evaluated using a combination of qualitative and quantitative methods. Comm…K21 Academy
• Generative Adversarial Network-Based Data Augmentation for Enhancing Wireless Physical Layer AuthenticationJan 18, 2024 — To assess the effectiveness of the data augmentation GAN-based model, we split the process of evaluation into two part…National Institutes of Health (.gov)
• UltiHash for GANsGANs need large training datasets, often consisting of several hundreds of thousands of raw images. These images need to be stored…UltiHash
• Unpaired Image-to-Image Translation with Content Preserving Perspective: A ReviewFeb 14, 2025 — Structural Similarity Index (SSIM) assesses the perceived quality of images by comparing luminance, contrast, and stru…arXiv

• Enhanced Generative Adversarial Networks for Unseen Word … – arXivNov 13, 2023 — Enhanced Generative Adversarial Networks for Unseen Word Generation from EEG Signals. … Recent advances in brain-com…arXiv
• Brain-AI System Translates Thoughts Into MovementAug 31, 2025 — Motor brain–computer interfaces (BCIs) decode neural signals to help people with paralysis move and communicate. Even …Neuroscience News
• fMRI Brain Decoding and Its Applications in Brain–Computer InterfaceAbstract. Brain neural activity decoding is an important branch of neuroscience research and a key technology for the brain–comput…National Institutes of Health (.gov)
• Using adversarial networks to extend brain computer interface … – NIHAs the recorded neurons change, one can think of the underlying movement intent signal being expressed in changing coordinates. If…National Institutes of Health (.gov)
• The combination of brain-computer interfaces and artificial intelligenceAbstract. Brain-computer interfaces (BCIs) have shown great prospects as real-time bidirectional links between living brains and a…National Institutes of Health (.gov)
• Generative Adversarial Networks-Based Data Augmentation for …In this diagram, XL denotes EEG samples from all subjects excluding target subject (LOO), Xt denotes target subject’s samples, Xh1…Aalborg Universitets forskningsportal
• Multiscale fusion enhanced spiking neural network for invasive BCI …Feb 20, 2025 — For instance, in neurorehabilitation, SNNs can provide real-time feedback to assist in rehabilitation training (Elbasi…Frontiers
• 5 AI Trends Shaping the Future of Public Sector in 2025 – Google CloudFeb 5, 2025 — Trend #1: Multimodal AI: Unleashing the power of context Imagine a world where AI can understand and analyze informatio…Google Cloud
• Brain-Computer Interfaces Boosted by Novel AI AlgorithmSep 9, 2024 — What sets the DPAD algorithm apart from other recurring neural network algorithms used for decoding behavior from brain…Psychology Today
• Emerging Brain-to-Content Technologies from Generative AI and …Nov 14, 2024 — Emerging Brain-to-Content Technologies from Generative AI and Deep Representation Learning.National Institutes of Health (.gov)
• Refined AI approach improves noninvasive BCI performanceApr 29, 2024 — Moreover, the capability of the group’s AI-powered BCI suggests a direct application to continuously controlling a rob…Carnegie Mellon University’s College of Engineering
• What are GANs, and how do they help in data augmentation? – MilvusWhat are GANs, and how do they help in data augmentation? * What are GANs? Generative Adversarial Networks (GANs) are a class of m…Milvus
• Making fake brain waves more realisticJul 19, 2021 — Despite the rapid progress, however, the performance of BCI ( Brain–computer interfaces ) technology continues to be l…Agency for Science, Technology and Research
• Impact of dataset size and long-term ECoG-based BCI usage on deep learning decoders performanceMar 15, 2023 — In brain-computer interfaces (BCI) research, recording data is time-consuming and expensive, which limits access to bi…Frontiers
• Brain-Computer Interfaces: Something New under the Sun | Request PDF… EEG has been used in the majority of mental workload research based on brain activity, which comes after several studies that …ResearchGate
• Publicly Available Datasets for Artificial Intelligence in Neurosurgery: A Systematic ReviewAug 10, 2025 — Introduction: The advancement of artificial intelligence (AI) in neurosurgery is dependent on high quality, large, lab…National Institutes of Health (.gov)
• Dual model transfer learning to compensate for individual variability in brain-computer interfaceSep 14, 2024 — Recent advancements in brain-computer interface (BCI) technology have seen a significant shift towards incorporating c…National Institutes of Health (.gov)
• Natural Image Reconstruction from fMRI Based on Node–Edge …Feb 27, 2024 — Abstract. Reconstructing natural stimulus images using functional magnetic resonance imaging (fMRI) is one of the most…MDPI
• From Brain Activity to Videos: The Journey of Mind VideoMay 21, 2023 — All five studies focus on reconstructing images or videos from brain activity, specifically using fMRI data. They all …MLearning.ai Art
• The Intersection of AI and Neuroengineering: Transforming Brain-machine InterfacesTraditional BMIs often required manual calibration and adjustment to maintain accuracy, a process that could be cumbersome and tim…Hilaris Publishing SRL
• Meta-AlignNN: A meta-learning framework for stable brain-computer interface performance across subjects, time, and tasksApr 24, 2025 — Practical application of brain-computer interfaces (BCIs) requires stable mapping between neuronal activity and behavi…bioRxiv
• Diffusion model enables quantitative CBF analysis of Alzheimer’s DiseaseJul 2, 2024 — Deep-learning (DL) based generative models such as generative adversarial networks (GANs) and variational auto-encoders…National Institutes of Health (.gov)
• Enhanced Functional Outcome from Traumatic Brain Injury with Brain–Machine Interface NeuromodulationApr 27, 2023 — However, chronic brain-machine interfaces show that the underlying network can be modified over time, particularly in …National Institutes of Health (.gov)