OBJECTIVE: Artificial Intelligence (AI) models in radiation therapy are being developed with increasing pace. Despite this, the radiation therapy community has not widely adopted these models in clinical practice. A cohesive guideline on how to develop, report and clinically validate AI algorithms might help bridge this gap.
METHODS: A Delphi process with all co-authors was followed to determine which topics should be addressed in this comprehensive guideline. Separate sections of the guideline, including Statements, were written by subgroups of the authors and discussed with the whole group at several meetings. Statements were formulated and scored as highly recommended or recommended.
RESULTS: The following topics were found most relevant: Decision making, image analysis, volume segmentation, treatment planning, patient specific quality assurance of treatment delivery, adaptive treatment, outcome prediction, training, validation and testing of AI model parameters, model availability for others to verify, model quality assurance/updates and upgrades, ethics. Key references were given together with an outlook on current hurdles and possibilities to overcome these. 19 Statements were formulated.
CONCLUSIONS: A cohesive guideline has been written which addresses main topics regarding AI in radiation therapy. It will help to guide development, as well as transparent and consistent reporting and validation of new AI tools and facilitate adoption.

OBJECTIVE: To evaluate the reliability of consensus-based segmentation in terms of reproducibility of radiomic features.
METHODS: In this retrospective study, three tumor data sets were investigated: breast cancer (n = 30), renal cell carcinoma (n = 30), and pituitary macroadenoma (n = 30). MRI was utilized for breast and pituitary data sets, while CT was used for renal data set. 12 readers participated in the segmentation process. Consensus segmentation was created by making corrections on a previous region or volume of interest. Four experiments were designed to evaluate the reproducibility of radiomic features. Reliability was assessed with intraclass correlation coefficient (ICC) with two cut-off values: 0.75 and 0.9.
RESULTS: Considering the lower bound of the 95% confidence interval and the ICC threshold of 0.90, at least 61% of the radiomic features were not reproducible in the inter-consensus analysis. In the susceptibility experiment, at least half (54%) became non-reproducible when the first reader is replaced with a different reader. In the intra-consensus analysis, at least about one-third (32%) were non-reproducible when the same second reader segmented the image over the same first reader two weeks later. Compared to inter-reader analysis based on independent single readers, the inter-consensus analysis did not statistically significantly improve the rates of reproducible features in all data sets and analyses.
CONCLUSIONS: Despite the positive connotation of the word \"consensus\", it is essential to REMIND that consensus-based segmentation has significant reproducibility issues. Therefore, the usage of consensus-based segmentation alone should be avoided unless a reliability analysis is performed, even if it is not practical in clinical settings.

UNASSIGNED: Contouring Collaborative for Consensus in Radiation Oncology (C3RO) is a crowdsourced challenge engaging radiation oncologists across various expertise levels in segmentation. An obstacle to artificial intelligence (AI) development is the paucity of multiexpert datasets; consequently, we sought to characterize whether aggregate segmentations generated from multiple nonexperts could meet or exceed recognized expert agreement.
UNASSIGNED: Participants who contoured ≥ 1 region of interest (ROI) for the breast, sarcoma, head and neck (H&N), gynecologic (GYN), or gastrointestinal (GI) cases were identified as a nonexpert or recognized expert. Cohort-specific ROIs were combined into single simultaneous truth and performance level estimation (STAPLE) consensus segmentations. STAPLE nonexpert ROIs were evaluated against STAPLE expert contours using Dice similarity coefficient (DSC). The expert interobserver DSC ( IODSC expert ) was calculated as an acceptability threshold between STAPLE nonexpert and STAPLE expert . To determine the number of nonexperts required to match the IODSC expert for each ROI, a single consensus contour was generated using variable numbers of nonexperts and then compared to the IODSC expert .
UNASSIGNED: For all cases, the DSC values for STAPLE nonexpert versus STAPLE expert were higher than comparator expert IODSC expert for most ROIs. The minimum number of nonexpert segmentations needed for a consensus ROI to achieve IODSC expert acceptability criteria ranged between 2 and 4 for breast, 3 and 5 for sarcoma, 3 and 5 for H&N, 3 and 5 for GYN, and 3 for GI.
UNASSIGNED: Multiple nonexpert-generated consensus ROIs met or exceeded expert-derived acceptability thresholds. Five nonexperts could potentially generate consensus segmentations for most ROIs with performance approximating experts, suggesting nonexpert segmentations as feasible cost-effective AI inputs.

Cryo-electron microscopy (cryoEM) is becoming popular as a tool to solve biomolecular structures with the recent availability of direct electron detectors allowing automated acquisition of high resolution data. The Bsoft software package, developed over 20 years for analyzing electron micrographs, offers a full workflow for validated single particle analysis with extensive functionality, enabling customization for specific cases. With the increasing use of cryoEM and its automation, proper validation of the results is a bigger concern. The three major validation approaches, independent data sets, resolution-limited processing, and coherence testing, can be incorporated into any Bsoft workflow. Here, the main workflow is divided into four phases: (i) micrograph preprocessing, (ii) particle picking, (iii) particle alignment and reconstruction, and (iv) interpretation. Each of these phases represents a conceptual unit that can be automated, followed by a check point to assess the results. The aim in the first three phases is to reconstruct one or more validated maps at the best resolution possible. Map interpretation then involves identification of components, segmentation, quantification, and modeling. The algorithms in Bsoft are well established, with future plans focused on ease of use, automation and institutionalizing validation.

BACKGROUND: Clustering methods are increasingly employed to segment brain regions into functional subdivisions using resting-state functional magnetic resonance imaging (rs-fMRI). However, these methods are highly sensitive to the (i) precise algorithms employed, (ii) their initializations, and (iii) metrics used for uncovering the optimal number of clusters from the data.
METHODS: To address these issues, we develop a novel consensus clustering evidence accumulation (CC-EAC) framework, which effectively combines multiple clustering methods for segmenting brain regions using rs-fMRI data. Using extensive computer simulations, we examine the performance of widely used clustering algorithms including K-means, hierarchical, and spectral clustering as well as their combinations. We also examine the accuracy and validity of five objective criteria for determining the optimal number of clusters: mutual information, variation of information, modified silhouette, Rand index, and probabilistic Rand index.
RESULTS: A CC-EAC framework with a combination of base K-means clustering (KC) and hierarchical clustering (HC) with probabilistic Rand index as the criterion for choosing the optimal number of clusters, accurately uncovered the correct number of clusters from simulated datasets. In experimental rs-fMRI data, these methods reliably detected functional subdivisions of the supplementary motor area, insula, intraparietal sulcus, angular gyrus, and striatum.
METHODS: Unlike conventional approaches, CC-EAC can accurately determine the optimal number of stable clusters in rs-fMRI data, and is robust to initialization and choice of free parameters.
CONCLUSIONS: A novel CC-EAC framework is proposed for segmenting brain regions, by effectively combining multiple clustering methods and identifying optimal stable functional clusters in rs-fMRI data.