Single-case experimental design (SCED) data can be analyzed following different approaches. One of the first historically proposed options is randomizations tests, benefiting from the inclusion of randomization in the design: a desirable methodological feature. Randomization tests have become more feasible with the availability of computational resources, and such tests have been proposed for all major types of SCEDs: multiple-baseline, reversal/withdrawal, alternating treatments, and changing criterion designs. The focus of the current text is on the last of these, given that they have not been the subject of any previous simulation study. Specifically, we estimate type I error rates and statistical power for two different randomization procedures applicable to changing criterion designs: the phase change moment randomization and the blocked alternating criterion randomization. We include different series lengths, number of phases, levels of autocorrelation, and random variability. The results suggest that type I error rates are generally controlled and that sufficient power can be achieved with as few as 28-30 measurements for independent data, although more measurements are needed in case of positive autocorrelation. The presence of a reversal to a previous criterion level is beneficial. R code is provided for carrying out randomization tests following the two randomization procedures.

This article surveys the use of nonparametric permutation tests for analyzing experimental data. The permutation approach, which involves randomizing or permuting features of the observed data, is a flexible way to draw statistical inferences in common experimental settings. It is particularly valuable when few independent observations are available, a frequent occurrence in controlled experiments in economics and other social sciences. The permutation method constitutes a comprehensive approach to statistical inference. In two-treatment testing, permutation concepts underlie popular rank-based tests, like the Wilcoxon and Mann-Whitney tests. But permutation reasoning is not limited to ordinal contexts. Analogous tests can be constructed from the permutation of measured observations-as opposed to rank-transformed observations-and we argue that these tests should often be preferred. Permutation tests can also be used with multiple treatments, with ordered hypothesized effects, and with complex data-structures, such as hypothesis testing in the presence of nuisance variables. Drawing examples from the experimental economics literature, we illustrate how permutation testing solves common challenges. Our aim is to help experimenters move beyond the handful of overused tests in play today and to instead see permutation testing as a flexible framework for statistical inference.
UNASSIGNED: The online version contains supplementary material available at 10.1007/s10683-023-09799-6.

We propose a method to construct simultaneous confidence intervals for a parameter vector from inverting a series of randomization tests (RT). The randomization tests are facilitated by an efficient multivariate Robbins-Monro procedure that takes the correlation information of all components into account. The estimation method does not require any distributional assumption of the population other than the existence of the second moments. The resulting simultaneous confidence intervals are not necessarily symmetric about the point estimate of the parameter vector but possess the property of equal tails in all dimensions. In particular, we present the constructing the mean vector of one population and the difference between two mean vectors of two populations. Extensive simulation is conducted to show numerical comparison with four methods. We illustrate the application of the proposed method to test bioequivalence with multiple endpoints on some real data.

Analyses of longitudinal data with non-linear mixed-effects models (NLMEM) are typically associated with high power, but sometimes at the cost of inflated type I error. Approaches to overcome this problem were published recently, such as model-averaging across drug models (MAD), individual model-averaging (IMA), and combined Likelihood Ratio Test (cLRT). This work aimed to assess seven NLMEM approaches in the same framework: treatment effect assessment in balanced two-armed designs using real natural history data with or without the addition of simulated treatment effect. The approaches are MAD, IMA, cLRT, standard model selection (STDs), structural similarity selection (SSs), randomized cLRT (rcLRT), and model-averaging across placebo and drug models (MAPD). The assessment included type I error, using Alzheimer\'s Disease Assessment Scale-cognitive (ADAS-cog) scores from 817 untreated patients and power and accuracy in the treatment effect estimates after the addition of simulated treatment effects. The model selection and averaging among a set of pre-selected candidate models were driven by the Akaike information criteria (AIC). The type I error rate was controlled only for IMA and rcLRT; the inflation observed otherwise was explained by the placebo model misspecification and selection bias. Both IMA and rcLRT had reasonable power and accuracy except under a low typical treatment effect.

Relative growth rate (RGR) has a long history of use in biology. In its logged form, RGR = ln[(M + ΔM)/M], where M is size of the organism at the commencement of the study, and ΔM is new growth over time interval Δt. It illustrates the general problem of comparing non-independent (confounded) variables, e.g. (X + Y) vs. X. Thus, RGR depends on what starting M(X) is used even within the same growth phase. Equally, RGR lacks independence from its derived components, net assimilation rate (NAR) and leaf mass ratio (LMR), as RGR = NAR × LMR, so that they cannot legitimately be compared by standard regression or correlation analysis.
The mathematical properties of RGR exemplify the general problem of \'spurious\' correlations that compare expressions derived from various combinations of the same component terms X and Y. This is particularly acute when X >> Y, the variance of X or Y is large, or there is little range overlap of X and Y values among datasets being compared. Relationships (direction, curvilinearity) between such confounded variables are essentially predetermined and so should not be reported as if they are a finding of the study. Standardizing by M rather than time does not solve the problem. We propose the inherent growth rate (IGR), lnΔM/lnM, as a simple, robust alternative to RGR that is independent of M within the same growth phase.
Although the preferred alternative is to avoid the practice altogether, we discuss cases where comparing expressions with components in common may still have utility. These may provide insights if (1) the regression slope between pairs yields a new variable of biological interest, (2) the statistical significance of the relationship remains supported using suitable methods, such as our specially devised randomization test, or (3) multiple datasets are compared and found to be statistically different. Distinguishing true biological relationships from spurious ones, which arise from comparing non-independent expressions, is essential when dealing with derived variables associated with plant growth analyses.

Researchers conducting small-scale cluster randomized controlled trials (RCTs) during the pilot testing of an intervention often look for evidence of promise to justify an efficacy trial. We developed a method to test for intervention effects that is adaptive (i.e., responsive to data exploration), requires few assumptions, and is statistically valid (i.e., controls the type I error rate), by adapting masked visual analysis techniques to cluster RCTs. We illustrate the creation of masked graphs and their analysis using data from a pilot study in which 15 high school programs were randomly assigned to either business as usual or an intervention developed to promote psychological and academic well-being in 9th grade students in accelerated coursework. We conclude that in small-scale cluster RCTs there can be benefits of testing for effects without a priori specification of a statistical model or test statistic.

The dysbiosis of the gut microbiome associated with ulcerative colitis (UC) has been extensively studied in recent years. However, the question of whether UC influences the spatial heterogeneity of the human gut mucosal microbiome has not been addressed. Spatial heterogeneity (specifically, the inter-individual heterogeneity in microbial species abundances) is one of the most important characterizations at both population and community scales, and can be assessed and interpreted by Taylor\'s power law (TPL) and its community-scale extensions (TPLEs). Due to the high mobility of microbes, it is difficult to investigate their spatial heterogeneity explicitly; however, TPLE offers an effective approach to implicitly analyze the microbial communities. Here, we investigated the influence of UC on the spatial heterogeneity of the gut microbiome with intestinal mucosal microbiome samples collected from 28 UC patients and healthy controls. Specifically, we applied Type-I TPLE for measuring community spatial heterogeneity and Type-III TPLE for measuring mixed-species population heterogeneity to evaluate the heterogeneity changes of the mucosal microbiome induced by UC at both the community and species scales. We further used permutation test to determine the possible differences between UC patients and healthy controls in heterogeneity scaling parameters. Results showed that UC did not significantly influence gut mucosal microbiome heterogeneity at either the community or mixed-species levels. These findings demonstrated significant resilience of the human gut microbiome and confirmed a prediction of TPLE: that the inter-subject heterogeneity scaling parameter of the gut microbiome is an intrinsic property to humans, invariant with UC disease.

Permutation tests are useful in stepped-wedge trials to provide robust statistical tests of intervention-effect estimates. However, the Stata command permute does not produce valid tests in this setting because individual observations are not exchangeable. We introduce the swpermute command that permutes clusters to sequences to maintain exchangeability. The command provides additional functionality to aid users in performing analyses of stepped-wedge trials. In particular, we include the option \"withinperiod\" that performs the specified analysis separately in each period of the study with the resulting period-specific intervention-effect estimates combined as a weighted average. We also include functionality to test non-zero null hypotheses to aid the construction of confidence intervals. Examples of the application of swpermute are given using data from a trial testing the impact of a new tuberculosis diagnostic test on bacterial confirmation of a tuberculosis diagnosis.

Between-group comparison based on the restricted mean survival time (RMST) is getting attention as an alternative to the conventional logrank/hazard ratio approach for time-to-event outcomes in randomized controlled trials (RCTs). The validity of the commonly used nonparametric inference procedure for RMST has been well supported by large sample theories. However, we sometimes encounter cases with a small sample size in practice, where we cannot rely on the large sample properties. Generally, the permutation approach can be useful to handle these situations in RCTs. However, a numerical issue arises when implementing permutation tests for difference or ratio of RMST from two groups. In this article, we discuss the numerical issue and consider six permutation methods for comparing survival time distributions between two groups using RMST in RCTs setting. We conducted extensive numerical studies and assessed type I error rates of these methods. Our numerical studies demonstrated that the inflation of the type I error rate of the asymptotic methods is not negligible when sample size is small, and that all of the six permutation methods are workable solutions. Although some permutation methods became a little conservative, no remarkable inflation of the type I error rates were observed. We recommend using permutation tests instead of the asymptotic tests, especially when the sample size is less than 50 per arm.

Two common barriers to applying statistical tests to single-case experiments are that single-case data often violate the assumptions of parametric tests and that random assignment is inconsistent with the logic of single-case design. However, in the case of randomization tests applied to single-case experiments with rapidly alternating conditions, neither the statistical assumptions nor the logic of the designs are violated. To examine the utility of randomization tests for single-case data, we collected a sample of published articles including alternating treatments or multielement designs with random or semi-random condition sequences. We extracted data from graphs and used randomization tests to estimate the probability of obtaining results at least as extreme as the results in the experiment by chance alone (i.e., p-value). We compared the distribution of p-values from experimental comparisons that did and did not indicate a functional relation based on visual analysis and evaluated agreement between visual and statistical analysis at several levels of α. Results showed different means, shapes, and spreads for the p-value distributions and substantial agreement between visual and statistical analysis when α = .05, with lower agreement when α was adjusted to preserve family-wise error at .05. Questions remain, however, on the appropriate application and interpretation of randomization tests for single-case designs.