UNASSIGNED: The status of BRCA1/2 genes plays a crucial role in the treatment decision-making process for multiple cancer types. However, due to high costs and limited resources, a demand for BRCA1/2 genetic testing among patients is currently unmet. Notably, not all patients with BRCA1/2 mutations achieve favorable outcomes with poly (ADP-ribose) polymerase inhibitors (PARPi), indicating the necessity for risk stratification. In this study, we aimed to develop and validate a multimodal model for predicting BRCA1/2 gene status and prognosis with PARPi treatment.
UNASSIGNED: We included 1695 slides from 1417 patients with ovarian, breast, prostate, and pancreatic cancers across three independent cohorts. Using a self-attention mechanism, we constructed a multi-instance attention model (MIAM) to detect BRCA1/2 gene status from hematoxylin and eosin (H&E) pathological images. We further combined tissue features from the MIAM model, cell features, and clinical factors (the MIAM-C model) to predict BRCA1/2 mutations and progression-free survival (PFS) with PARPi therapy. Model performance was evaluated using area under the curve (AUC) and Kaplan-Meier analysis. Morphological features contributing to MIAM-C were analyzed for interpretability.
UNASSIGNED: Across the four cancer types, MIAM-C outperformed the deep learning-based MIAM in identifying the BRCA1/2 genotype. Interpretability analysis revealed that high-attention regions included high-grade tumors and lymphocytic infiltration, which correlated with BRCA1/2 mutations. Notably, high lymphocyte ratios appeared characteristic of BRCA1/2 mutations. Furthermore, MIAM-C predicted PARPi therapy response (log-rank p < 0.05) and served as an independent prognostic factor for patients with BRCA1/2-mutant ovarian cancer (p < 0.05, hazard ratio:0.4, 95% confidence interval: 0.16-0.99).
UNASSIGNED: The MIAM-C model accurately detected BRCA1/2 gene status and effectively stratified prognosis for patients with BRCA1/2 mutations.

UNASSIGNED: Previous studies based on resting-state functional magnetic resonance imaging(rs-fMRI) and voxel-based morphometry (VBM) have demonstrated significant abnormalities in brain structure and resting-state functional brain activity in patients with early-onset schizophrenia (EOS), compared with healthy controls (HCs), and these alterations were closely related to the pathogenesis of EOS. However, previous studies suffer from the limitations of small sample sizes and high heterogeneity of results. Therefore, the present study aimed to effectively integrate previous studies to identify common and specific brain functional and structural abnormalities in patients with EOS.
UNASSIGNED: The PubMed, Web of Science, Embase, Chinese National Knowledge Infrastructure (CNKI), and WanFang databases were systematically searched to identify publications on abnormalities in resting-state regional functional brain activity and gray matter volume (GMV) in patients with EOS. Then, we utilized the Seed-based d Mapping with Permutation of Subject Images (SDM-PSI) software to conduct a whole-brain voxel meta-analysis of VBM and rs-fMRI studies, respectively, and followed by multimodal overlapping on this basis to comprehensively identify brain structural and functional abnormalities in patients with EOS.
UNASSIGNED: A total of 27 original studies (28 datasets) were included in the present meta-analysis, including 12 studies (13 datasets) related to resting-state functional brain activity (496 EOS patients, 395 HCs) and 15 studies (15 datasets) related to GMV (458 EOS patients, 531 HCs). Overall, in the functional meta-analysis, patients with EOS showed significantly increased resting-state functional brain activity in the left middle frontal gyrus (extending to the triangular part of the left inferior frontal gyrus) and the right caudate nucleus. On the other hand, in the structural meta-analysis, patients with EOS showed significantly decreased GMV in the right superior temporal gyrus (extending to the right rolandic operculum), the right middle temporal gyrus, and the temporal pole (superior temporal gyrus).
UNASSIGNED: This meta-analysis revealed that some regions in the EOS exhibited significant structural or functional abnormalities, such as the temporal gyri, prefrontal cortex, and striatum. These findings may help deepen our understanding of the underlying pathophysiological mechanisms of EOS and provide potential biomarkers for the diagnosis or treatment of EOS.

Machine learning (ML) applications in medical artificial intelligence (AI) systems have shifted from traditional and statistical methods to increasing application of deep learning models. This survey navigates the current landscape of multimodal ML, focusing on its profound impact on medical image analysis and clinical decision support systems. Emphasizing challenges and innovations in addressing multimodal representation, fusion, translation, alignment, and co-learning, the paper explores the transformative potential of multimodal models for clinical predictions. It also highlights the need for principled assessments and practical implementation of such models, bringing attention to the dynamics between decision support systems and healthcare providers and personnel. Despite advancements, challenges such as data biases and the scarcity of \"big data\" in many biomedical domains persist. We conclude with a discussion on principled innovation and collaborative efforts to further the mission of seamless integration of multimodal ML models into biomedical practice.

Camellia oleifera is a crop of high economic value, yet it is particularly susceptible to various diseases and pests that significantly reduce its yield and quality. Consequently, the precise segmentation and classification of diseased Camellia leaves are vital for managing pests and diseases effectively. Deep learning exhibits significant advantages in the segmentation of plant diseases and pests, particularly in complex image processing and automated feature extraction. However, when employing single-modal models to segment Camellia oleifera diseases, three critical challenges arise: (A) lesions may closely resemble the colors of the complex background; (B) small sections of diseased leaves overlap; (C) the presence of multiple diseases on a single leaf. These factors considerably hinder segmentation accuracy. A novel multimodal model, CNN-Transformer Dual U-shaped Network (CTDUNet), based on a CNN-Transformer architecture, has been proposed to integrate image and text information. This model first utilizes text data to address the shortcomings of single-modal image features, enhancing its ability to distinguish lesions from environmental characteristics, even under conditions where they closely resemble one another. Additionally, we introduce Coordinate Space Attention (CSA), which focuses on the positional relationships between targets, thereby improving the segmentation of overlapping leaf edges. Furthermore, cross-attention (CA) is employed to align image and text features effectively, preserving local information and enhancing the perception and differentiation of various diseases. The CTDUNet model was evaluated on a self-made multimodal dataset compared against several models, including DeeplabV3+, UNet, PSPNet, Segformer, HrNet, and Language meets Vision Transformer (LViT). The experimental results demonstrate that CTDUNet achieved an mean Intersection over Union (mIoU) of 86.14%, surpassing both multimodal models and the best single-modal model by 3.91% and 5.84%, respectively. Additionally, CTDUNet exhibits high balance in the multi-class segmentation of Camellia oleifera diseases and pests. These results indicate the successful application of fused image and text multimodal information in the segmentation of Camellia disease, achieving outstanding performance.

The brain-computer interface (BCI) is one of the most powerful tools in neuroscience and generally includes a recording system, a processor system, and a stimulation system. Optogenetics has the advantages of bidirectional regulation, high spatiotemporal resolution, and cell-specific regulation, which expands the application scenarios of BCIs. In recent years, optogenetic BCIs have become widely used in the lab with the development of materials and software. The systems were designed to be more integrated, lightweight, biocompatible, and power efficient, as were the wireless transmission and chip-level embedded BCIs. The software is also constantly improving, with better real-time performance and accuracy and lower power consumption. On the other hand, as a cutting-edge technology spanning multidisciplinary fields including molecular biology, neuroscience, material engineering, and information processing, optogenetic BCIs have great application potential in neural decoding, enhancing brain function, and treating neural diseases. Here, we review the development and application of optogenetic BCIs. In the future, combined with other functional imaging techniques such as near-infrared spectroscopy (fNIRS) and functional magnetic resonance imaging (fMRI), optogenetic BCIs can modulate the function of specific circuits, facilitate neurological rehabilitation, assist perception, establish a brain-to-brain interface, and be applied in wider application scenarios.

Medical image registration has become pivotal in recent years with the integration of various imaging modalities like X-ray, ultrasound, MRI, and CT scans, enabling comprehensive analysis and diagnosis of biological structures. This paper provides a comprehensive review of registration techniques for medical images, with an in-depth focus on 2D-2D image registration methods. While 3D registration is briefly touched upon, the primary emphasis remains on 2D techniques and their applications. This review covers registration techniques for diverse modalities, including unimodal, multimodal, interpatient, and intra-patient. The paper explores the challenges encountered in medical image registration, including geometric distortion, differences in image properties, outliers, and optimization convergence, and discusses their impact on registration accuracy and reliability. Strategies for addressing these challenges are highlighted, emphasizing the need for continual innovation and refinement of techniques to enhance the accuracy and reliability of medical image registration systems. The paper concludes by emphasizing the importance of accurate medical image registration in improving diagnosis.

This study aims to explore the efficacy of a hybrid deep learning and radiomics approach, supplemented with patient metadata, in the noninvasive dermoscopic imaging-based diagnosis of skin lesions. We analyzed dermoscopic images from the International Skin Imaging Collaboration (ISIC) dataset, spanning 2016-2020, encompassing a variety of skin lesions. Our approach integrates deep learning with a comprehensive radiomics analysis, utilizing a vast array of quantitative image features to precisely quantify skin lesion patterns. The dataset includes cases of three, four, and eight different skin lesion types. Our methodology was benchmarked against seven classification methods from the ISIC 2020 challenge and prior research using a binary decision framework. The proposed hybrid model demonstrated superior performance in distinguishing benign from malignant lesions, achieving area under the receiver operating characteristic curve (AUROC) scores of 99%, 95%, and 96%, and multiclass decoding AUROCs of 98.5%, 94.9%, and 96.4%, with sensitivities of 97.6%, 93.9%, and 96.0% and specificities of 98.4%, 96.7%, and 96.9% in the internal ISIC 2018 challenge, as well as in the external Jinan and Longhua datasets, respectively. Our findings suggest that the integration of radiomics and deep learning, utilizing dermoscopic images, effectively captures the heterogeneity and pattern expression of skin lesions.

In the delivery room, fetal well-being is evaluated through laboratory tests, biosignals like cardiotocography, and imaging techniques such as fetal echocardiography. We have developed a multimodal machine learning model that integrates medical records, biosignals, and imaging data to predict fetal acidosis, using a dataset from a tertiary hospital\'s delivery room (n=2,266). To achieve this, features were extracted from unstructured data sources, including biosignals and imaging, and then merged with structured data from medical records. The concatenated vectors formed the basis for training a classifier to predict post-delivery fetal acidosis. Our model achieved an Area Under the Receiver Operating Characteristic curve (AUROC) of 0.752 on the test dataset, demonstrating the potential of multimodal models in predicting various fetal outcomes.

For better collaboration among radiologists, the interpretation workload should be evaluated, considering the difference in difficulty for interpreting each case. However, objective evaluation of difficulty is challenging. This study proposes a multimodal classifier of structural and textual data to predict difficulty based on order information and patient data without using images. The classifier showed performance with a specificity of 0.9 and an accuracy of 0.7.

Single-cell multimodal datasets have measured various characteristics of individual cells, enabling a deep understanding of cellular and molecular mechanisms. However, multimodal data generation remains costly and challenging, and missing modalities happen frequently. Recently, machine learning approaches have been developed for data imputation but typically require fully matched multimodalities to learn common latent embeddings that potentially lack modality specificity. To address these issues, we developed an open-source machine learning model, Joint Variational Autoencoders for multimodal Imputation and Embedding (JAMIE). JAMIE takes single-cell multimodal data that can have partially matched samples across modalities. Variational autoencoders learn the latent embeddings of each modality. Then, embeddings from matched samples across modalities are aggregated to identify joint cross-modal latent embeddings before reconstruction. To perform cross-modal imputation, the latent embeddings of one modality can be used with the decoder of the other modality. For interpretability, Shapley values are used to prioritize input features for cross-modal imputation and known sample labels. We applied JAMIE to both simulation data and emerging single-cell multimodal data including gene expression, chromatin accessibility, and electrophysiology in human and mouse brains. JAMIE significantly outperforms existing state-of-the-art methods in general and prioritized multimodal features for imputation, providing potentially novel mechanistic insights at cellular resolution.