Robust statistics

鲁棒统计
  • 文章类型: Journal Article
    本文提出了一种在项目反应理论(IRT)模型中评估差异项目功能(DIF)的方法。该方法不需要预先指定锚项目,这是它的主要美德。它的开发主要分为两个步骤:首先,展示如何将DIF重新表述为基于IRT的缩放中的离群值检测问题,然后使用鲁棒统计方法解决后者。该建议是IRT缩放参数的递减M估计器,该估计器被调整为以所需的渐近I型错误率使用DIF标记项目。理论结果描述了在没有DIF的情况下估计器的效率及其在存在DIF的情况下的鲁棒性。仿真研究表明,该方法与目前可用的DIF检测方法相比具有优势,和一个真实的数据例子说明了它在一个研究环境中的应用,其中锚项目的预规范是不可行的。本文的重点是两个独立群体的双参数Logistic模型,扩展到结论中考虑的其他设置。
    This paper proposes a method for assessing differential item functioning (DIF) in item response theory (IRT) models. The method does not require pre-specification of anchor items, which is its main virtue. It is developed in two main steps: first by showing how DIF can be re-formulated as a problem of outlier detection in IRT-based scaling and then tackling the latter using methods from robust statistics. The proposal is a redescending M-estimator of IRT scaling parameters that is tuned to flag items with DIF at the desired asymptotic type I error rate. Theoretical results describe the efficiency of the estimator in the absence of DIF and its robustness in the presence of DIF. Simulation studies show that the proposed method compares favorably to currently available approaches for DIF detection, and a real data example illustrates its application in a research context where pre-specification of anchor items is infeasible. The focus of the paper is the two-parameter logistic model in two independent groups, with extensions to other settings considered in the conclusion.
    导出

    更多引用

    收藏

    翻译标题摘要

    我要上传

    求助全文

  • 文章类型: Journal Article
    纳入性别和性别(S/G)相关因素通常被认为是朝着更个性化的躯体诊断和治疗迈进的必要步骤。精神病学,和神经系统疾病。直到现在,整合S/G相关因素的大多数尝试已减少到确定女性和男性在行为/生物学变量方面的平均差异。本评论通过强调三个主要局限性来质疑这种传统方法:1)使用经典参数方法来比较均值的问题;2)与均值准确表示组内数据和组间差异的能力有关的挑战;3)均值比较强加了结果二值化和二元理论框架,阻碍了精准医学的发展。还讨论了没有这些限制的替代方法。我们希望这些论点将有助于反思如何对S/G因素进行研究并加以改进。
    The incorporation of sex and gender (S/G) related factors is commonly acknowledged as a necessary step to advance towards more personalized diagnoses and treatments for somatic, psychiatric, and neurological diseases. Until now, most attempts to integrate S/G-related factors have been reduced to identifying average differences between females and males in behavioral/ biological variables. The present commentary questions this traditional approach by highlighting three main sets of limitations: 1) Issues stemming from the use of classic parametric methods to compare means; 2) challenges related to the ability of means to accurately represent the data within groups and differences between groups; 3) mean comparisons impose a results\' binarization and a binary theoretical framework that precludes advancing towards precision medicine. Alternative methods free of these limitations are also discussed. We hope these arguments will contribute to reflecting on how research on S/G factors is conducted and could be improved.
    导出

    更多引用

    收藏

    翻译标题摘要

    我要上传

    求助全文

  • 文章类型: Journal Article
    在这项工作中,我们介绍了一条城市河流的水质评价,圣路易斯河,位于圣路易斯省,阿根廷。圣路易斯河流经两个发展中的城市;因此,城市人为活动影响其水质。对河流进行了空间和时间采样,评估每个水样的十个物理化学变量。这些数据用于计算简化的水质指数,以估计河流水质并推断可能的污染源。使用开源软件R对数据进行统计分析,4.1.0版本。主成分分析,聚类分析,相关矩阵,并进行了热图分析。结果表明,发生人为活动的地区的水质下降。进行了稳健的推理统计分析,采用多变量方差分析(MANOVA)的替代方法,MANOVA.广泛的功能。与水质下降相关的最统计相关的物理化学变量用于开发多元线性回归模型来估计有机质,减少连续监测河流所需的变量,因此,降低成本。鉴于该地区关于这一特定河流类别的特征和恢复的信息有限,所开发的模型至关重要,因为它可以快速检测人类的变化,并有助于河流的环境管理。该模型还用于估计位于其他类似河流中的地点的有机物,取得满意的结果。
    In this work, we present the water quality assessment of an urban river, the San Luis River, located in San Luis Province, Argentina. The San Luis River flows through two developing cities; hence, urban anthropic activities affect its water quality. The river was sampled spatially and temporally, evaluating ten physicochemical variables on each water sample. These data were used to calculate a Simplified Index of Water Quality in order to estimate river water quality and infer possible contamination sources. Data were statistically analyzed with the opensource software R, 4.1.0 version. Principal component analysis, cluster analysis, correlation matrices, and heatmap analysis were performed. Results indicated that water quality decreases in areas where anthropogenic activities take place. Robust inferential statistical analysis was performed, employing an alternative of multivariate analysis of variance (MANOVA), MANOVA.wide function. The most statistically relevant physicochemical variables associated with water quality decrease were used to develop a multiple linear regression model to estimate organic matter, reducing the variables necessary for continuous monitoring of the river and, hence, reducing costs. Given the limited information available in the region about the characteristics and recovery of this specific river category, the model developed is of vital importance since it can quickly detect anthropic alterations and contribute to the environmental management of the rivers. This model was also used to estimate organic matter at sites located in other similar rivers, obtaining satisfactory results.
    导出

    更多引用

    收藏

    翻译标题摘要

    我要上传

    求助全文

  • 文章类型: Journal Article
    数据增强是一种在深度学习中自动扩展训练数据的有效技术。受大脑启发的方法是从人脑的功能和结构中汲取灵感,并将这些机制和原理应用于人工智能和计算机科学。当训练数据和测试数据之间存在较大的样式差异时,常见的数据增强方法无法有效提升深度模型的泛化性能。为了解决这个问题,我们改进了具有不确定性的域移位(DSU)建模,并提出了一种新的大脑启发的计算机视觉图像数据增强方法,该方法由两个关键组件组成,即,使用鲁棒统计量并控制DSU(RCDSU)和特征数据增强(FeatureDA)的方差系数。RCDSU计算特征统计量(均值和标准差)与稳健的统计量,以削弱异常值的影响,使统计数据接近真实值,提高深度学习模型的鲁棒性。通过控制变异系数,RCDSU在语义保留的情况下使特征统计移位,并增加移位范围。FeatureDA类似地控制方差系数以生成语义不变的增强特征并增加增强特征的覆盖范围。建议RCDSU和FeatureDA在特征空间中执行样式传递和内容传递,并分别提高了模型在风格和内容层面的泛化能力。在照片上,艺术绘画,卡通,和草图(PACS)多样式分类任务,RCDSU加FeatureDA实现了具有竞争力的准确性。将高斯噪声添加到PACS数据集后,RCDSU加FeatureDA对异常值具有很强的鲁棒性。FeatureDA在CIFAR-100图像分类任务上取得了出色的结果。RCDSU加FeatureDA可以作为一种新颖的大脑启发语义数据增强方法,具有隐式机器人自动化功能,适用于训练和测试数据之间风格差异较大的数据集。
    Data augmentation is an effective technique for automatically expanding training data in deep learning. Brain-inspired methods are approaches that draw inspiration from the functionality and structure of the human brain and apply these mechanisms and principles to artificial intelligence and computer science. When there is a large style difference between training data and testing data, common data augmentation methods cannot effectively enhance the generalization performance of the deep model. To solve this problem, we improve modeling Domain Shifts with Uncertainty (DSU) and propose a new brain-inspired computer vision image data augmentation method which consists of two key components, namely, using Robust statistics and controlling the Coefficient of variance for DSU (RCDSU) and Feature Data Augmentation (FeatureDA). RCDSU calculates feature statistics (mean and standard deviation) with robust statistics to weaken the influence of outliers, making the statistics close to the real values and improving the robustness of deep learning models. By controlling the coefficient of variance, RCDSU makes the feature statistics shift with semantic preservation and increases shift range. FeatureDA controls the coefficient of variance similarly to generate the augmented features with semantics unchanged and increase the coverage of augmented features. RCDSU and FeatureDA are proposed to perform style transfer and content transfer in the feature space, and improve the generalization ability of the model at the style and content level respectively. On Photo, Art Painting, Cartoon, and Sketch (PACS) multi-style classification task, RCDSU plus FeatureDA achieves competitive accuracy. After adding Gaussian noise to PACS dataset, RCDSU plus FeatureDA shows strong robustness against outliers. FeatureDA achieves excellent results on CIFAR-100 image classification task. RCDSU plus FeatureDA can be applied as a novel brain-inspired semantic data augmentation method with implicit robot automation which is suitable for datasets with large style differences between training and testing data.
    导出

    更多引用

    收藏

    翻译标题摘要

    我要上传

       PDF(Pubmed)

  • 文章类型: Journal Article
    不同的算法,例如Savitzky-Golay滤波器和Whittaker平滑器,已经被提出来提高实验色谱的质量。这些方法避免了妨碍数据分析的过多噪声,并且因此允许分析物的准确检测和定量。这些算法需要微调其超参数以调节其粗糙度和灵活性。传统上,这种微调是手动完成的,直到获得一个信号,去除噪声,同时保存有价值的峰值信息。更客观和自动化的方法是可用的,但这些通常是方法特定的和/或需要先前的知识。在这项工作中,L和V曲线,k折交叉验证,自相关函数和残差方差估计方法被引入作为替代的自动化和普遍适用的参数调整方法。这些方法不需要任何先前的信息,并且与多种去噪方法兼容。此外,对于k折交叉验证,自相关函数和残差方差估计,提出了一种基于中位数估计器的新颖实现来处理色谱的特定形状,通常由交替的平坦基线和尖锐峰组成。这些调谐方法结合四种去噪方法进行了研究;Savitsky-Golay滤波器,惠特克更光滑,稀疏度辅助信号平滑器和使用稀疏度方法的基线估计和去噪。已经证明,中值估计器方法显着提高了相关平滑调谐器组合的去噪和信息保存性能,对于模拟数据集,最高可达4倍,对于实验色谱图,最高可达10倍。此外,依赖残差方差估计的参数整定方法,k-fold交叉验证和自相关函数在不同的模拟数据集和实验色谱图上导致类似的小的均方根误差。稀疏度辅助信号平滑器和基线估计以及使用稀疏度方法去噪,这两者都依赖于稀疏性的使用,系统地优于其他两种方法,因此最适合色谱图。
    Different algorithms, such as the Savitzky-Golay filter and Whittaker smoother, have been proposed to improve the quality of experimental chromatograms. These approaches avoid excessive noise from hampering data analysis and as such allow an accurate detection and quantification of analytes. These algorithms require fine-tuning of their hyperparameters to regulate their roughness and flexibility. Traditionally, this fine-tuning is done manually until a signal is obtained that removes the noise while conserving valuable peak information. More objective and automated approaches are available, but these are usually method specific and/or require previous knowledge. In this work, the L-and V-curve, k-fold cross-validation, autocorrelation function and residual variance estimation approach are introduced as alternative automated and generally applicable parameter tuning methods. These methods do not require any previous information and are compatible with a multitude of denoising methods. Additionally, for k-fold cross-validation, autocorrelation function and residual variance estimation, a novel implementation based on median estimators is proposed to handle the specific shape of chromatograms, typically composed of alternating flat baselines and sharp peaks. These tuning methods are investigated in combination with four denoising methods; the Savitsky-Golay filter, Whittaker smoother, sparsity assisted signal smoother and baseline estimation and denoising using sparsity approach. It is demonstrated that the median estimators approach significantly improves the denoising and information conservation performance of relevant smoother-tuner combinations up to a factor 4 for simulated datasets and even up to a factor 10 for an experimental chromatogram. Moreover, the parameter tuning methods relying on residual variance estimation, k-fold cross-validation and autocorrelation function lead to similar small root-mean squared errors on the different simulated datasets and experimental chromatograms. The sparsity assisted signal smoother and baseline estimation and denoising using sparsity approach, which both rely on the use of sparsity, systematically outperform the two other methods and are hence most appropriate for chromatograms.
    导出

    更多引用

    收藏

    翻译标题摘要

    我要上传

    求助全文

  • 文章类型: Journal Article
    统计学中的稳健性一词是指通常对模型假设的偏差不敏感的方法。换句话说,即使数据不完全符合统计模型,稳健的方法也能够保持其准确性。在分析概率分布的混合时,稳健的统计分析特别有效。因此,这些方法使得能够将X射线串行晶体学数据离散化为两个概率分布:包括真实数据点(例如背景强度)的组和包括离群值(例如X射线探测器上的布拉格峰或坏像素)的另一组。强大的统计分析的这些特征对于同步加速器和X射线自由电子激光(XFEL)源产生的连续晶体学(SX)数据集的不断增加的数量是有益的。在SX数据分析中的某些应用中使用稳健统计的关键优势是,由于其对输入参数不敏感,因此需要进行最少的参数调整。在本文中,介绍了一个基于鲁棒统计概念的称为鲁棒高斯拟合库(RGFlib)的软件包。针对两个SX数据分析任务,基于鲁棒统计和RGFlib的概念提出了两种方法:(i)鲁棒的寻峰算法和(ii)自动鲁棒的方法来检测X射线像素检测器上的不良像素。
    The term robustness in statistics refers to methods that are generally insensitive to deviations from model assumptions. In other words, robust methods are able to preserve their accuracy even when the data do not perfectly fit the statistical models. Robust statistical analyses are particularly effective when analysing mixtures of probability distributions. Therefore, these methods enable the discretization of X-ray serial crystallography data into two probability distributions: a group comprising true data points (for example the background intensities) and another group comprising outliers (for example Bragg peaks or bad pixels on an X-ray detector). These characteristics of robust statistical analysis are beneficial for the ever-increasing volume of serial crystallography (SX) data sets produced at synchrotron and X-ray free-electron laser (XFEL) sources. The key advantage of the use of robust statistics for some applications in SX data analysis is that it requires minimal parameter tuning because of its insensitivity to the input parameters. In this paper, a software package called Robust Gaussian Fitting library (RGFlib) is introduced that is based on the concept of robust statistics. Two methods are presented based on the concept of robust statistics and RGFlib for two SX data-analysis tasks: (i) a robust peak-finding algorithm and (ii) an automated robust method to detect bad pixels on X-ray pixel detectors.
    导出

    更多引用

    收藏

    翻译标题摘要

    我要上传

       PDF(Pubmed)

  • 文章类型: Journal Article
    用于识别项目不匹配或差异项目功能(DIF)的可行方法是缩放结构和声音测量的核心。许多方法依赖于在某个模型完美拟合数据的假设下的极限分布的推导。即使在经典测试理论中,也存在诸如项目函数的单调性和总体独立性之类的典型DIF假设,但是在使用项目响应理论或其他潜在变量模型来评估项目拟合度时,这些假设得到了更明确的说明。这里提出的工作为DIF检测提供了一种稳健的方法,该方法不假设完美的模型数据拟合,而是使用Tukey的污染分布概念。该方法使用强大的异常检测来标记无法建立足够的模型数据拟合的项目。
    Viable methods for the identification of item misfit or Differential Item Functioning (DIF) are central to scale construction and sound measurement. Many approaches rely on the derivation of a limiting distribution under the assumption that a certain model fits the data perfectly. Typical DIF assumptions such as the monotonicity and population independence of item functions are present even in classical test theory but are more explicitly stated when using item response theory or other latent variable models for the assessment of item fit. The work presented here provides a robust approach for DIF detection that does not assume perfect model data fit, but rather uses Tukey\'s concept of contaminated distributions. The approach uses robust outlier detection to flag items for which adequate model data fit cannot be established.
    导出

    更多引用

    收藏

    翻译标题摘要

    我要上传

       PDF(Pubmed)

  • 文章类型: Journal Article
    人类文化的空间是广阔的,但是某些文化结构比其他文化结构更符合认知和社会约束。这导致了我们物种在几千年的文化进化中探索的可能性的“景观”。然而,这个健身景观是什么,限制和引导文化进化,可以回答这些问题的机器学习算法通常是为大规模数据集开发的。应用到稀疏,不一致,历史记录中发现的不完整数据受到的关注较少,标准建议可能导致对边缘化的偏见,研究不足,或者少数民族文化。我们展示了如何适应最小概率流算法和逆伊辛模型,受物理学启发的机器学习主力,挑战。一系列自然扩展-包括对缺失数据的动态估计,和交叉验证与正则化-使基础约束的可靠重建。我们在宗教史数据库的精选子集上展示了我们的方法:来自人类历史上407个宗教团体的记录,从青铜时代到现在。这揭示了一个复杂的,崎岖,景观,两者都很锋利,国家认可的宗教倾向于集中的明确定义的高峰,和传播福音派宗教的文化洪泛区,非国家精神实践,可以找到神秘的宗教。
    The space of possible human cultures is vast, but some cultural configurations are more consistent with cognitive and social constraints than others. This leads to a \"landscape\" of possibilities that our species has explored over millennia of cultural evolution. However, what does this fitness landscape, which constrains and guides cultural evolution, look like? The machine-learning algorithms that can answer these questions are typically developed for large-scale datasets. Applications to the sparse, inconsistent, and incomplete data found in the historical record have received less attention, and standard recommendations can lead to bias against marginalized, under-studied, or minority cultures. We show how to adapt the minimum probability flow algorithm and the Inverse Ising model, a physics-inspired workhorse of machine learning, to the challenge. A series of natural extensions-including dynamical estimation of missing data, and cross-validation with regularization-enables reliable reconstruction of the underlying constraints. We demonstrate our methods on a curated subset of the Database of Religious History: records from 407 religious groups throughout human history, ranging from the Bronze Age to the present day. This reveals a complex, rugged, landscape, with both sharp, well-defined peaks where state-endorsed religions tend to concentrate, and diffuse cultural floodplains where evangelical religions, non-state spiritual practices, and mystery religions can be found.
    导出

    更多引用

    收藏

    翻译标题摘要

    我要上传

    求助全文

  • 文章类型: Journal Article
    目的:我们评估了DetectDeviatingCells(DDC)算法的错误检测性能,它在连续变量中的观察(逐例)和变量(单元)级别标记数据异常。我们将其性能与模拟数据集中的其他方法进行了比较。
    方法:我们模拟了2-20岁假设个体的身高和体重数据。我们根据预定的错误模式改变了高度值的比例。我们应用了DDC算法和其他错误检测方法(描述性统计,地块,固定阈值规则,经典和鲁棒的马氏距离),我们比较了误差检测性能和灵敏度,特异性,似然比,预测值和ROC曲线。
    结果:在我们选择的阈值下,所有方法的错误检测特异性在所有情况下都非常好,多变量和稳健方法的灵敏度更高.DDC算法的性能与其他稳健的多变量方法相似。对ROC曲线的分析表明,所有方法在粗差(例如错误的测量单位)方面都具有可比的性能,但是DDC算法在更复杂的错误模式上优于其他算法(例如仍然合理的转录错误,虽然极端)。
    结论:DDC算法有可能改善观测数据的错误检测过程。
    We evaluated the error detection performance of the DetectDeviatingCells (DDC) algorithm which flags data anomalies at observation (casewise) and variable (cellwise) level in continuous variables. We compared its performance to other approaches in a simulated dataset.
    We simulated height and weight data for hypothetical individuals aged 2-20 years. We changed a proportion of height values according to predetermined error patterns. We applied the DDC algorithm and other error-detection approaches (descriptive statistics, plots, fixed-threshold rules, classic, and robust Mahalanobis distance) and we compared error detection performance with sensitivity, specificity, likelihood ratios, predictive values, and receiver operating characteristic (ROC) curves.
    At our chosen thresholds error detection specificity was excellent across all scenarios for all methods and sensitivity was higher for multivariable and robust methods. The DDC algorithm performance was similar to other robust multivariable methods. Analysis of ROC curves suggested that all methods had comparable performance for gross errors (e.g., wrong measurement unit), but the DDC algorithm outperformed the others for more complex error patterns (e.g., transcription errors that are still plausible, although extreme).
    The DDC algorithm has the potential to improve error detection processes for observational data.
    导出

    更多引用

    收藏

    翻译标题摘要

    我要上传

       PDF(Pubmed)

  • 文章类型: Journal Article
    未经证实:产后抑郁症可以采取多种形式。不同的症状模式可能会对我们如何筛查产生不同的影响,诊断,治疗产后抑郁症.我们试图利用最近开发的鲁棒估计算法来自动识别抑郁症状的差异模式,并随后表征表现出不同模式的个体。
    UNASSIGNED:抑郁症状数据(N=548)来自美国两个城市的患有神经精神疾病的女性,参与了围产期压力的纵向观察研究。数据收集自1994年至2012年的埃默里大学和2012年至2017年的阿肯色大学医学科学。我们使用稳健的期望最大化算法对贝克抑郁量表(BDI)项目进行了探索性因子分析,而不是传统的期望最大化算法。这种最近开发的方法能够自动检测差异症状模式。我们描述了症状模式的差异,并对症状模式与人口统计和精神病史的关联进行了未经调整和调整的分析。
    UNASSIGNED:该算法确定53(9.7%)参与者与其他参与者相比具有不同的报告症状模式。该组在所有项目中都有更严重的症状,尤其是与自我伤害和自我判断有关的项目,并且其症状与潜在心理结构的关系有所不同。社交焦虑症病史(OR:4.0;95%CI[1.9,8.1])和儿童创伤史(每增加5分,OR:1.2;95%CI[1.1,1.3])在调整其他协变量后与这种差异模式显着相关。
    未经评估:社交焦虑障碍和儿童创伤与严重产后抑郁症状的不同模式有关,这可能需要定制的筛查策略,诊断,以及解决这些合并症的治疗。
    UNASSIGNED:没有可申报的资金来源。
    UNASSIGNED: Postpartum depression can take many forms. Different symptom patterns could have divergent implications for how we screen, diagnose, and treat postpartum depression. We sought to utilise a recently developed robust estimation algorithm to automatically identify differential patterns in depressive symptoms and subsequently characterise the individuals who exhibit different patterns.
    UNASSIGNED: Depressive symptom data (N = 548) were collected from women with neuropsychiatric illnesses at two U.S. urban sites participating in a longitudinal observational study of stress across the perinatal period. Data were collected from Emory University between 1994 and 2012 and from the University of Arkansas for Medical Sciences between 2012 and 2017. We conducted an exploratory factor analysis of Beck Depression Inventory (BDI) items using a robust expectation-maximization algorithm, rather than a conventional expectation-maximization algorithm. This recently developed method enabled automatic detection of differential symptom patterns. We described differences in symptom patterns and conducted unadjusted and adjusted analyses of associations of symptom patterns with demographics and psychiatric histories.
    UNASSIGNED: 53 (9.7%) participants were identified by the algorithm as having a different pattern of reported symptoms compared to other participants. This group had more severe symptoms across all items-especially items related to thoughts of self-harm and self-judgement-and differed in how their symptoms related to underlying psychological constructs. History of social anxiety disorder (OR: 4.0; 95% CI [1.9, 8.1]) and history of childhood trauma (for each 5-point increase, OR: 1.2; 95% CI [1.1, 1.3]) were significantly associated with this differential pattern after adjustment for other covariates.
    UNASSIGNED: Social anxiety disorder and childhood trauma are associated with differential patterns of severe postpartum depressive symptoms, which might warrant tailored strategies for screening, diagnosis, and treatment to address these comorbid conditions.
    UNASSIGNED: There are no funding sources to declare.
    导出

    更多引用

    收藏

    翻译标题摘要

    我要上传

       PDF(Pubmed)

公众号