UNASSIGNED: The status of BRCA1/2 genes plays a crucial role in the treatment decision-making process for multiple cancer types. However, due to high costs and limited resources, a demand for BRCA1/2 genetic testing among patients is currently unmet. Notably, not all patients with BRCA1/2 mutations achieve favorable outcomes with poly (ADP-ribose) polymerase inhibitors (PARPi), indicating the necessity for risk stratification. In this study, we aimed to develop and validate a multimodal model for predicting BRCA1/2 gene status and prognosis with PARPi treatment.
UNASSIGNED: We included 1695 slides from 1417 patients with ovarian, breast, prostate, and pancreatic cancers across three independent cohorts. Using a self-attention mechanism, we constructed a multi-instance attention model (MIAM) to detect BRCA1/2 gene status from hematoxylin and eosin (H&E) pathological images. We further combined tissue features from the MIAM model, cell features, and clinical factors (the MIAM-C model) to predict BRCA1/2 mutations and progression-free survival (PFS) with PARPi therapy. Model performance was evaluated using area under the curve (AUC) and Kaplan-Meier analysis. Morphological features contributing to MIAM-C were analyzed for interpretability.
UNASSIGNED: Across the four cancer types, MIAM-C outperformed the deep learning-based MIAM in identifying the BRCA1/2 genotype. Interpretability analysis revealed that high-attention regions included high-grade tumors and lymphocytic infiltration, which correlated with BRCA1/2 mutations. Notably, high lymphocyte ratios appeared characteristic of BRCA1/2 mutations. Furthermore, MIAM-C predicted PARPi therapy response (log-rank p < 0.05) and served as an independent prognostic factor for patients with BRCA1/2-mutant ovarian cancer (p < 0.05, hazard ratio:0.4, 95% confidence interval: 0.16-0.99).
UNASSIGNED: The MIAM-C model accurately detected BRCA1/2 gene status and effectively stratified prognosis for patients with BRCA1/2 mutations.

UNASSIGNED: Previous studies based on resting-state functional magnetic resonance imaging(rs-fMRI) and voxel-based morphometry (VBM) have demonstrated significant abnormalities in brain structure and resting-state functional brain activity in patients with early-onset schizophrenia (EOS), compared with healthy controls (HCs), and these alterations were closely related to the pathogenesis of EOS. However, previous studies suffer from the limitations of small sample sizes and high heterogeneity of results. Therefore, the present study aimed to effectively integrate previous studies to identify common and specific brain functional and structural abnormalities in patients with EOS.
UNASSIGNED: The PubMed, Web of Science, Embase, Chinese National Knowledge Infrastructure (CNKI), and WanFang databases were systematically searched to identify publications on abnormalities in resting-state regional functional brain activity and gray matter volume (GMV) in patients with EOS. Then, we utilized the Seed-based d Mapping with Permutation of Subject Images (SDM-PSI) software to conduct a whole-brain voxel meta-analysis of VBM and rs-fMRI studies, respectively, and followed by multimodal overlapping on this basis to comprehensively identify brain structural and functional abnormalities in patients with EOS.
UNASSIGNED: A total of 27 original studies (28 datasets) were included in the present meta-analysis, including 12 studies (13 datasets) related to resting-state functional brain activity (496 EOS patients, 395 HCs) and 15 studies (15 datasets) related to GMV (458 EOS patients, 531 HCs). Overall, in the functional meta-analysis, patients with EOS showed significantly increased resting-state functional brain activity in the left middle frontal gyrus (extending to the triangular part of the left inferior frontal gyrus) and the right caudate nucleus. On the other hand, in the structural meta-analysis, patients with EOS showed significantly decreased GMV in the right superior temporal gyrus (extending to the right rolandic operculum), the right middle temporal gyrus, and the temporal pole (superior temporal gyrus).
UNASSIGNED: This meta-analysis revealed that some regions in the EOS exhibited significant structural or functional abnormalities, such as the temporal gyri, prefrontal cortex, and striatum. These findings may help deepen our understanding of the underlying pathophysiological mechanisms of EOS and provide potential biomarkers for the diagnosis or treatment of EOS.

Camellia oleifera is a crop of high economic value, yet it is particularly susceptible to various diseases and pests that significantly reduce its yield and quality. Consequently, the precise segmentation and classification of diseased Camellia leaves are vital for managing pests and diseases effectively. Deep learning exhibits significant advantages in the segmentation of plant diseases and pests, particularly in complex image processing and automated feature extraction. However, when employing single-modal models to segment Camellia oleifera diseases, three critical challenges arise: (A) lesions may closely resemble the colors of the complex background; (B) small sections of diseased leaves overlap; (C) the presence of multiple diseases on a single leaf. These factors considerably hinder segmentation accuracy. A novel multimodal model, CNN-Transformer Dual U-shaped Network (CTDUNet), based on a CNN-Transformer architecture, has been proposed to integrate image and text information. This model first utilizes text data to address the shortcomings of single-modal image features, enhancing its ability to distinguish lesions from environmental characteristics, even under conditions where they closely resemble one another. Additionally, we introduce Coordinate Space Attention (CSA), which focuses on the positional relationships between targets, thereby improving the segmentation of overlapping leaf edges. Furthermore, cross-attention (CA) is employed to align image and text features effectively, preserving local information and enhancing the perception and differentiation of various diseases. The CTDUNet model was evaluated on a self-made multimodal dataset compared against several models, including DeeplabV3+, UNet, PSPNet, Segformer, HrNet, and Language meets Vision Transformer (LViT). The experimental results demonstrate that CTDUNet achieved an mean Intersection over Union (mIoU) of 86.14%, surpassing both multimodal models and the best single-modal model by 3.91% and 5.84%, respectively. Additionally, CTDUNet exhibits high balance in the multi-class segmentation of Camellia oleifera diseases and pests. These results indicate the successful application of fused image and text multimodal information in the segmentation of Camellia disease, achieving outstanding performance.

The brain-computer interface (BCI) is one of the most powerful tools in neuroscience and generally includes a recording system, a processor system, and a stimulation system. Optogenetics has the advantages of bidirectional regulation, high spatiotemporal resolution, and cell-specific regulation, which expands the application scenarios of BCIs. In recent years, optogenetic BCIs have become widely used in the lab with the development of materials and software. The systems were designed to be more integrated, lightweight, biocompatible, and power efficient, as were the wireless transmission and chip-level embedded BCIs. The software is also constantly improving, with better real-time performance and accuracy and lower power consumption. On the other hand, as a cutting-edge technology spanning multidisciplinary fields including molecular biology, neuroscience, material engineering, and information processing, optogenetic BCIs have great application potential in neural decoding, enhancing brain function, and treating neural diseases. Here, we review the development and application of optogenetic BCIs. In the future, combined with other functional imaging techniques such as near-infrared spectroscopy (fNIRS) and functional magnetic resonance imaging (fMRI), optogenetic BCIs can modulate the function of specific circuits, facilitate neurological rehabilitation, assist perception, establish a brain-to-brain interface, and be applied in wider application scenarios.

This study aims to explore the efficacy of a hybrid deep learning and radiomics approach, supplemented with patient metadata, in the noninvasive dermoscopic imaging-based diagnosis of skin lesions. We analyzed dermoscopic images from the International Skin Imaging Collaboration (ISIC) dataset, spanning 2016-2020, encompassing a variety of skin lesions. Our approach integrates deep learning with a comprehensive radiomics analysis, utilizing a vast array of quantitative image features to precisely quantify skin lesion patterns. The dataset includes cases of three, four, and eight different skin lesion types. Our methodology was benchmarked against seven classification methods from the ISIC 2020 challenge and prior research using a binary decision framework. The proposed hybrid model demonstrated superior performance in distinguishing benign from malignant lesions, achieving area under the receiver operating characteristic curve (AUROC) scores of 99%, 95%, and 96%, and multiclass decoding AUROCs of 98.5%, 94.9%, and 96.4%, with sensitivities of 97.6%, 93.9%, and 96.0% and specificities of 98.4%, 96.7%, and 96.9% in the internal ISIC 2018 challenge, as well as in the external Jinan and Longhua datasets, respectively. Our findings suggest that the integration of radiomics and deep learning, utilizing dermoscopic images, effectively captures the heterogeneity and pattern expression of skin lesions.

Deep learning is playing an increasingly important role in accurate prediction of molecular properties. Prior to being processed by a deep learning model, a molecule is typically represented in the form of a text or a graph. While some methods attempt to integrate these two forms of molecular representations, the misalignment of graph and text embeddings presents a significant challenge to fuse two modalities. To solve this problem, we propose a method that aligns and fuses graph and text features in the embedding space by using contrastive loss and cross attentions. Additionally, we enhance the molecular representation by incorporating multi-granularity information of molecules on the levels of atoms, functional groups, and molecules. Extensive experiments show that our model outperforms state-of-the-art models in downstream tasks of molecular property prediction, achieving superior performance with less pretraining data. The source codes and data are available at https://github.com/zzr624663649/multimodal_molecular_property.

The floor constitutes one of the largest areas within a building with which users interact most frequently in daily activities. Employing floor sensors is vital for smart-building digital twins, wherein triboelectric nanogenerators demonstrate wide application potential due to their good performance and self-powering characteristics. However, their sensing stability, reliability, and multimodality require further enhancement to meet the rapidly evolving demands. Thus, this work introduces a multimodal intelligent flooring system, implementing a 4 × 4 floor array for multimodal information detection (including position, pressure, material, user identity, and activity) and human-machine interactions. The floor unit incorporates a hybrid structure of triboelectric pressure sensors and a top-surface material sensor, facilitating linear and enhanced sensitivity across a wide pressure range (0-800 N), alongside the material recognition capability. The floor array is implemented by an advanced output-ratio method with minimalist output channels, which is insensitive to environmental factors such as humidity and temperature. In addition to multimodal sensing, energy harvesting is co-designed with the pressure sensors for scavenging waste energy to power smart-building sensor nodes. This developed flooring system enables multimodal sensing, energy harvesting, and smart-sport interactions in smart buildings, greatly expanding the floor sensing scenarios and applications.

UNASSIGNED: Lung cancer is a malignant tumor, for which pulmonary nodules are considered to be significant indicators. Early recognition and timely treatment of pulmonary nodules can contribute to improving the survival rate of patients with cancer. Positron emission tomography-computed tomography (PET/CT) is a noninvasive, fusion imaging technique that can obtain both functional and structural information of lung regions. However, studies of pulmonary nodules based on computer-aided diagnosis have primarily focused on the nodule level due to a reliance on the annotation of nodules, which is superficial and unable to contribute to the actual clinical diagnosis. The aim of this study was thus to develop a fully automated classification framework for a more comprehensive assessment of pulmonary nodules in PET/CT imaging data.
UNASSIGNED: We developed a two-stage multimodal learning framework for the diagnosis of pulmonary nodules in PET/CT imaging. In this framework, Stage I focuses on pulmonary parenchyma segmentation using a pretrained U-Net and PET/CT registration. Stage II aims to extract, integrate, and recognize image-level and feature-level features by employing the three-dimensional (3D) Inception-residual net (ResNet) convolutional block attention module architecture and a dense-voting fusion mechanism.
UNASSIGNED: In the experiments, the proposed model\'s performance was comprehensively validated using a set of real clinical data, achieving mean scores of 89.98%, 89.21%, 84.75%, 93.38%, 86.83%, and 0.9227 for accuracy, precision, recall, specificity, F1 score, and area under curve values, respectively.
UNASSIGNED: This paper presents a two-stage multimodal learning approach for the automatic diagnosis of pulmonary nodules. The findings reveal that the main reason for limiting model performance is the nonsolitary property of nodules in pulmonary nodule diagnosis, providing direction for future research.

The primary aim of this study is to assess the viability of employing multimodal radiomics techniques for distinguishing between cervical spinal cord injury and spinal cord concussion in cervical magnetic resonance imaging. This is a multicenter study involving 288 patients from a major medical center as the training group, and 75 patients from two other medical centers as the testing group. Data regarding the presence of spinal cord injury symptoms and their recovery status within 72 h were documented. These patients underwent sagittal T1-weighted and T2-weighted imaging using cervical magnetic resonance imaging. Radiomics techniques are used to help diagnose whether these patients have cervical spinal cord injury or spinal cord concussion. 1197 radiomics features were extracted for each modality of each patient. The accuracy of T1 modal in testing group is 0.773, AUC is 0.799. The accuracy of T2 modal in testing group is 0.707, AUC is 0.813. The accuracy of T1 + T2 modal in testing group is 0.800, AUC is 0.840. Our research indicates that multimodal radiomics techniques utilizing cervical magnetic resonance imaging can effectively diagnose the presence of cervical spinal cord injury or spinal cord concussion.

OBJECTIVE: Deep learning can enhance the performance of multimodal image analysis, which is known for its noninvasive attributes and complementary efficacy, in predicting axillary lymph node (ALN) metastasis. Therefore, we established a multimodal deep learning model incorporating ultrasound (US) and magnetic resonance imaging (MRI) images to predict ALN metastasis in patients with breast cancer.
METHODS: A retrospective cohort of patients with histologically confirmed breast cancer from two hospitals composed of the primary cohort (n = 465) and the external validation cohort (n = 123). All patients had undergone both preoperative US and MRI scans. After data preprocessing, three convolutional neural network models were used to analyze the US and MRI images, respectively. After integrating the US and MRI deep learning prediction results (DLUS and DLMRI, respectively), a multimodal deep learning (DLMRI+US+Clinical parameter) model was constructed. The predictive ability of the proposed model was compared to that of the DLUS, DLMRI, combined bimodal (DLMRI+US), and clinical parameter models. Evaluation was performed using the area under the receiver operating characteristic curves (AUCs) and decision curves.
RESULTS: A total of 588 patients with breast cancer participated in this study. The DLMRI+US+Clinical parameter model outperformed the alternative models, achieving the highest AUCs of 0.819 (95% confidence interval [CI] 0.734-0.903) and 0.809 (95% CI 0.723-0.895) on the internal and external validation sets, respectively. The decision curve analysis confirmed its clinical usefulness.
CONCLUSIONS: The DLMRI+US＋Clinical parameter model demonstrates the feasibility and reliability of its performance for ALN metastasis prediction in patients with breast cancer.