We aimed to compare the performance and interobserver agreement of radiologists manually segmenting images or those assisted by automatic segmentation. We further aimed to reduce interobserver variability and improve the consistency of radiomics features. This retrospective study included 327 patients diagnosed with prostate cancer from September 2016 to June 2018; images from 228 patients were used for automatic segmentation construction, and images from the remaining 99 were used for testing. First, four radiologists with varying experience levels retrospectively segmented 99 axial prostate images manually using T2-weighted fat-suppressed magnetic resonance imaging. Automatic segmentation was performed after 2 weeks. The Pyradiomics software package v3.1.0 was used to extract the texture features. The Dice coefficient and intraclass correlation coefficient (ICC) were used to evaluate segmentation performance and the interobserver consistency of prostate radiomics. The Wilcoxon rank sum test was used to compare the paired samples, with the significance level set at p < 0.05. The Dice coefficient was used to accurately measure the spatial overlap of manually delineated images. In all the 99 prostate segmentation result columns, the manual and automatic segmentation results of the senior group were significantly better than those of the junior group (p < 0.05). Automatic segmentation was more consistent than manual segmentation (p < 0.05), and the average ICC reached >0.85. The automatic segmentation annotation performance of junior radiologists was similar to that of senior radiologists performing manual segmentation. The ICC of radiomics features increased to excellent consistency (0.925 [0.888~0.950]). Automatic segmentation annotation provided better results than manual segmentation by radiologists. Our findings indicate that automatic segmentation annotation helps reduce variability in the perception and interpretation between radiologists with different experience levels and ensures the stability of radiomics features.

BACKGROUND: Early detection and accurate pathological assessment are critical to improving prognosis of pancreatic cancer. EUS has been widely used in diagnosing pancreatic lesions and can obtain histological diagnosis by endoscopic ultrasound-guided fine needle aspiration (EUS-FNA). However, comprehensive assessment of the interobserver agreement (IOA) among cytopathologists evaluating EUS-FNA specimens is still limited. Therefore, this study evaluated IOA among cytopathologists for EUS-FNA specimens of solid pancreatic lesions, especially in false-negative cases of cytological diagnosis and analyzed the factors that influence cytological diagnosis of EUS-FNA so as to improve the diagnostic efficiency of EUS-FNA.
METHODS: We retrieved EUS-FNA samples of pancreatic solid lesions from 2017 to 2021 and collected their clinical/cytological data. Two cytopathologists independently reviewed these cases using a quoted, novel standardized cytology scoring tool. Ultimately, we calculated IOA among cytopathologists and performed a binary logistic regression analysis to evaluate factors influencing the cytological diagnosis of EUS-FNA.
RESULTS: 161 patients were included, and 60 cases with a clinical diagnosis of pancreatic cancer but a cytological diagnosis of benign and atypical constituted the false-negative group. IOAs for cytological diagnosis of overall patients and the false-negative group were in perfect/moderate agreement with Kendall\'s W values of 0.896 and 0.462, respectively. The number of diagnostic cells in the scoring tool had the highest level of agreement (κ = 0.721) for overall patients. There was at best moderate agreement on other quantity and quality parameters for both all cases and false-negative group. Logistic regression analysis showed the number of diagnostic cells (OR = 6.110, p < 0.05) and amount of blood (OR = 0.320, p < 0.05) could influence cytological diagnosis.
CONCLUSIONS: The false-negative rate of our study as high as 37.26% (60/161) is mainly related to strict standards of cytopathologists, and their ability to standardize pancreatic cytology is still improving. Suboptimal agreement among cytopathologists for cytological diagnosis and the number of diagnostic cells may be associated with the occurrence of false-negative diagnosis. Further regression analysis confirmed that the number of diagnostic cells and obscuring blood were important factors in cytological diagnosis. Therefore, refinement of cytological diagnostic criteria, standardization of specimen quality evaluation, and training of cytopathologists may improve the agreement of cytopathologists, thus improving the repeatability of cytological diagnosis and reducing the occurrence of false-negative events.

The reproducibility of imaging models for predicting microvascular invasion (MVI) in patients with hepatocellular carcinoma (HCC) remains questionable due to inconsistent interpretation of image signs. Our aim was to screen for high-consensus MRI features to develop a repeatable model for predicting MVI.
We included 219 patients with HCC who underwent surgical resection, and patients were divided into a training cohort (n = 145) and a validation cohort (n = 74). Morphological characteristics, signal features on hepatobiliary phases, and dynamic enhancement patterns were qualitatively interobserver evaluated. Interobserver agreement was assessed using Cohen\'s κ for selecting features with high interobserver agreement. Risk factors that were significant in stepwise multivariate analysis and that could be measured with good interobserver agreement were used to construct a predictive model, which was assessed in the validation cohort. The diagnostic performance of the model was evaluated based on area under the receiver operating characteristic curve (AUC).
Multivariate analysis identified nonsmooth tumor margin, absence of radiologic capsule, and intratumoral artery as independent risk factors of MVI. These MRI-based features showed good or nearly perfect interobserver agreement between radiologists (κ > 0.6). The predictive model predicted MVI well in the training (AUC 0.734) and validation cohorts (AUC 0.759) and fitted well to calibration curves.
MRI features included nonsmooth tumor margin, absence of radiologic capsule, and intratumoral artery that can be assessed with high interobserver agreement can predict MVI in HCC patients. The predictive model described here may be useful to radiologists, regardless of experience level.

This study aimed to develop a comprehensive classification system for fractures of the lateral process of the talus (LPTF) based on CT, and to evaluate its prognostic value, reliability and reproducibility. We retrospectively reviewed 42 patients involving LPTF with an average follow-up of 35.9 months for clinical and radiographic evaluations. In order to develop a comprehensive classification, a panel of experienced orthopedic surgeons discussed the cases. All fractures were classified according to Hawkins, McCrory-Bladin and new proposed classifications by 6 observers. The analysis of interobserver and intraobserver agreements was measured using kappa statistics. The new classification included 2 types based on presence of concomitant injuries or not, with type I consisting of 3 subtypes and type II of 5 subtypes. Average AOFAS score was 91.5 in the type Ia of new classification, 86 in type Ib, 90.5 in type Ic, 89 in type IIa, 76.7 in type IIb, 76.6 in type IIc, 91.3 in type IId, and 83.5 in type IIe. Interobserver and intraobserver reliability of the new classification system were almost perfect (κ = 0.776 and 0.837, respectively), showing a higher interobserver and intraobserver reliability compared to the Hawkins classification (κ 0.572 and 0.649, respectively) as well as McCrory-Bladin classification (κ = 0.582 and 0.685, respectively). The new classification system is a comprehensive one that takes into account concomitant injuries and shows good prognostic value with clinical outcomes. It is more reliable and reproducible and could be a useful tool for decision-making on treatment options for LPTF.

OBJECTIVE: Distal clavicle fracture classification directly affects the treatment decisions. It is unclear whether the classification systems implemented differ depending on surgeons\' backgrounds. This study aimed to compare the interobserver agreement of four classification systems used for lateral clavicle fractures by shoulder specialists and general trauma surgeons.
METHODS: Radiographs of 20 lateral clavicle fractures representing a full spectrum of adult fracture patterns were analyzed by eight experienced shoulder specialists and eight general trauma surgeons from 10 different hospitals. All cases were graded according to the Orthopedic Trauma Association (OTA), Neer, Jäger/Breitner, and Gongji classification systems. To measure observer agreement, Fleiss\' kappa coefficient (κ) was applied and assessed.
RESULTS: When only X-ray films were presented, both groups achieved fair agreement. However, when the 3D-CT scan images were provided, improved interobserver agreement was found in the specialist group when the OTA, Jäger/Breitner, and Gongji classification systems were used. In the generalist groups, improved agreement was found when using the Gongji classification system. In terms of interobserver reliability, the OTA, Neer, and Jäger/Breitner classification systems showed better agreement among shoulder specialists, while a slightly lower level of agreement was found using the Gongji classification system. For the OTA classification system, interobserver agreement had a mean kappa value of 0.418, ranging from 0.446 (specialist group) to 0.402 (generalist group). For the Neer classification system, interobserver agreement had a mean kappa value of 0.368, ranging from 0.402 (specialist group) to 0.390 (generalist group). For the Jäger/Breitner classification system, the inter-observer agreement had a mean kappa value of 0.380, ranging from 0.413 (specialist group) to 0.404 (generalist group). For the Gongji classification system, interobserver agreement had a mean kappa value of 0.455, ranging from 0.480 (specialist group) to 0.485 (generalist group).
CONCLUSIONS: Generally speaking, 3D-CT scans provide a richer experience that can lead to better results in most classification systems of lateral clavicle fractures, highlighting the value of digitization and specialization in diagnosis and treatment. Competitive interobserver agreement was exhibited in the generalist group using the Gongji classification system, suggesting that the Gongji classification is suitable for general trauma surgeons who are not highly experienced in the shoulder field.

OBJECTIVE: To compare the expression of programmed cell death ligand 1 (PD-L1) in different paraffin blocks from the same triple-negative breast cancers (TNBC) specimen and between matched primary tumors and lymph node metastases (LNMets). We also aim to determine the interobserver agreement between pathologists trained on PD-L1 (SP142) assay in assessing TNBC.
METHODS: 426 histologically confirmed TNBC cases, in which 85 have LNMets, were included in this study. A PD-L1 (SP142) assay was used to identify PD-L1 expression on tumor infiltrating immune cells (IC) and also on tumor cells (TC) in primary tumors and LNMets of TNBC by two trained pathologists. PD-L1 scoring and assessment were based on criteria in IMpassion 130 trial criteria. Concordance of PD-L1 expression in TNBC were analyzed using Kappa-test and assessed by the Kappa value.
RESULTS: Prevalence of positive PD-L1 expression (PD-L1 +) on tumor-infiltrating immune cells (PD-L1 IC+) (IC≥1%) in LNMets (49.4%) was higher than in the matched primary tumors (38.9%). Concordance of PD-L1 expression on IC between the two paraffin blocks from the same primary tumor specimen was substantial (P < 0.000, Kappa = 0.627) and was identified in 83.1% (108/130) of the selected cases. For TNBC cases with matched primary and LNMets blocks, the concordance of PD-L1IC scoring between the two blocks was moderate (P < 0.000, Kappa = 0.434). Interobserver agreement of PD-L1 assessment was 78.2% (P < 0.000, Kappa = 0.567) in primary tumors and 61.4% (P < 0.000, Kappa = 0.253) in the matched LNets.
CONCLUSIONS: Substantial intratumor concordance of PD-L1 scoring of the primary tumors in TNBC patients was determined, implying that immunohistochemically detection using one representative block of the primary tumor should be enough to assign the expression status of PD-L1 in clinical practice. The prevalence of PD-L1 + in lymph node metastases (LNMets) was higher than in the matched primary tumors, implying that PD-L1 detection in LNMets may provide additional PD-L1 expression information, especially in TNBC cases with PD-L1- in the matched primary breast tumors. Interobserver agreement of PD-L1 scoring in primary tumors was moderate while only fair in LNMets, implying that the additional training for PD-L1 assessment of TNBC LNMets specimens is recommended to enhance interobserver agreement.
METHODS: The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

To validate a vascular severity score as an appropriate output for artificial intelligence (AI) Software as a Medical Device (SaMD) for retinopathy of prematurity (ROP) through comparison with ordinal disease severity labels for stage and plus disease assigned by the International Classification of Retinopathy of Prematurity, Third Edition (ICROP3), committee.
Validation study of an AI-based ROP vascular severity score.
A total of 34 ROP experts from the ICROP3 committee.
Two separate datasets of 30 fundus photographs each for stage (0-5) and plus disease (plus, preplus, neither) were labeled by members of the ICROP3 committee using an open-source platform. Averaging these results produced a continuous label for plus (1-9) and stage (1-3) for each image. Experts were also asked to compare each image to each other in terms of relative severity for plus disease. Each image was also labeled with a vascular severity score from the Imaging and Informatics in ROP deep learning system, which was compared with each grader\'s diagnostic labels for correlation, as well as the ophthalmoscopic diagnosis of stage.
Weighted kappa and Pearson correlation coefficients (CCs) were calculated between each pair of grader classification labels for stage and plus disease. The Elo algorithm was also used to convert pairwise comparisons for each expert into an ordered set of images from least to most severe.
The mean weighted kappa and CC for all interobserver pairs for plus disease image comparison were 0.67 and 0.88, respectively. The vascular severity score was found to be highly correlated with both the average plus disease classification (CC = 0.90, P < 0.001) and the ophthalmoscopic diagnosis of stage (P < 0.001 by analysis of variance) among all experts.
The ROP vascular severity score correlates well with the International Classification of Retinopathy of Prematurity committee member\'s labels for plus disease and stage, which had significant intergrader variability. Generation of a consensus for a validated scoring system for ROP SaMD can facilitate global innovation and regulatory authorization of these technologies.

OBJECTIVE: The purpose of this study was to quantitatively assess the longitudinal acquisition repeatability of MRI radiomics features in a three-dimensional (3D) T1-weighted (T1W) TSE sequence via a well-controlled prospective phantom study.
METHODS: Thirty consecutive daily datasets of an ACR-MRI phantom were acquired on two 1.5T MRI simulators using a 3D T1W TSE sequence. Images were blindly segmented by two observers. Post-acquisition processing was minimized but an intensity discretization (fixed bin size of 25). One hundred and one radiomics features (shape n = 12; first order n = 16; texture n = 73) were extracted. Longitudinal repeatability of each feature was evaluated by Pearson correlation and coefficient of variance (CV68% ). Interobserver feature value agreement was also quantified using intraclass correlation coefficient (ICC) and Bland-Altman analysis. A most repeatable radiomics feature set on both scanners was determined by feature coefficient of variance (CV68% <5%), ICC (>0.75), and the ratio of the interobserver difference to the interobserver mean δ<5%.
RESULTS: No trend of radiomics feature value changed with time. Longitudinal feature repeatability CV68% ranged 0.01-38.60% (mean/median: 12.5%/9.9%), and 0.01-40.47%, (8.49%/7.34%) on the scanners A and B. Shape features exhibited significantly better repeatability than first-order and texture features (all P < 0.01). Significant longitudinal repeatability difference was observed in texture features (P < 0.001) between the two scanners, but not in shape and first-order features (P > 0.30). First-order and texture features had smaller interobserver-dependent variation than acquisition-dependent variation. They also showed good interobserver agreement on both scanners (A:ICC = 0.80 ± 0.23; B:ICC = 0.80 ± 0.22), independent of acquisition repeatability. The repeatable radiomics features in common on both scanners, including 12 shape features, 0 first-order features, and 3 texture features, were determined as the most repeatable MRI radiomics feature set.
CONCLUSIONS: Radiomics features exhibited heterogeneous longitudinal repeatability, while the shape features were the most repeatable, in this phantom study with a 3D T1W TSE acquisition. The most repeatable radiomics feature set derived in this study should be helpful for the selection of reliable radiomics features in the future clinical use.

BACKGROUND: Hepatocellular carcinoma is the most common primary liver malignancy. From the results of previous studies, Liver Imaging Reporting and Data System (LI-RADS) on contrast-enhanced ultrasound (CEUS) has shown satisfactory diagnostic value. However, a unified conclusion on the interobserver stability of this innovative ultrasound imaging has not been determined. The present meta-analysis examined the interobserver agreement of CEUS LI-RADS to provide some reference for subsequent related research.
OBJECTIVE: To evaluate the interobserver agreement of LI-RADS on CEUS and analyze the sources of heterogeneity between studies.
METHODS: Relevant papers on the subject of interobserver agreement on CEUS LI-RADS published before March 1, 2020 in China and other countries were analyzed. The studies were filtered, and the diagnostic criteria were evaluated. The selected references were analyzed using the \"meta\" and \"metafor\" packages of R software version 3.6.2.
RESULTS: Eight studies were ultimately included in the present analysis. Meta-analysis results revealed that the summary Kappa value of included studies was 0.76 [95% confidence interval, 0.67-0.83], which shows substantial agreement. Higgins I 2 statistics also confirmed the substantial heterogeneity (I 2 = 91.30%, 95% confidence interval, 85.3%-94.9%, P < 0.01). Meta-regression identified the variables, including the method of patient enrollment, method of consistency testing, and patient race, which explained the substantial study heterogeneity.
CONCLUSIONS: CEUS LI-RADS demonstrated overall substantial interobserver agreement, but heterogeneous results between studies were also obvious. Further clinical investigations should consider a modified recommendation about the experimental design.

BACKGROUND: The evaluation of PD-L1 expression in nonsmall cell lung carcinoma (NSCLC) is becoming increasingly important given the effectiveness of PD-L1 inhibitors. Although cytologic specimens have been shown to be compatible with surgical specimens to evaluate PD-L1 immunohistochemistry (IHC), evidence of the reproducibility of PD-L1 in cytologic specimens is lacking. The aim of this study is to evaluate interobserver agreement in PD-L1 IHC in cytologic specimens.
METHODS: PD-L1 IHC was performed on 86 NSCLC cytology specimens using Dako PD-L1 IHC 22C3 pharmDx. The digitally scanned whole slide images (WSI) were read by five pathologists. Each case was given a Tumor Proportion Score (TPS) and the results were compared between the observers. The interobserver concordance was assessed using 1% and 50% as cutoffs.
RESULTS: TPSs were highly correlated among observers (Spearman correlation coefficient, 0.86-0.94). Using greater than 1% as a cutoff, interobserver agreement measured by Fleiss Kappa was 0.74 for all pathologists and Cohen\'s Kappa coefficient ranged from 0.49 to 0.83, consistent with moderate to substantial agreement. With a cutoff of greater than 50%, Fleiss Kappa was 0.79 for all pathologists and the kappa values ranged from 0.63 to 0.90, consistent with substantial to almost perfect agreement. Several pitfalls were identified by reviewing discordant cases, including staining in macrophages, stromal cells, and intratumoral heterogeneity.
CONCLUSIONS: Our data suggest that TPS of PD-L1 IHC on cytology specimens is reproducible, with a better agreement when using 50% as the cutoff value. However, special attention is required when the TPS is near the 1% cutoff.