Computing intervals using Cover2D.exe
Appendix 1 gives output from the provided software (see Software section at end of Methods) for the example given in Fig. 1b (see Example 1 in the user's notes). The problem is to find 95% confidence intervals for both the Hedges & Olkin and Steiger & Fouladi methods, each based on either d or g (i.e., 4 separate intervals), when d = 1.0 and ν = 16. The program reads the instructions from the input file and writes an output text file that can be opened with a text editor or spreadsheet (see Appendix 1). The output contains the mean difference, lower and upper limits, lower and upper segments (the distance between the non-centrality parameter and the respective upper and lower confidence limits), and the full width of the interval for both the standardized biased scaling and the standardized unbiased scaling for both the Hedges & Olkin and Steiger & Fouladi methods. The d-scaled confidence intervals from Fig. 1b are given under "Standardized Biased Scaling" for Hedges & Olkin and Steiger & Fouladi confidence intervals.
Supplement 1 gives detailed example calculations for each of the four methods used in this paper.
Predicting coverages for confidence intervals
It is obvious from Fig. 1 that the Hedges & Olkin and Steiger & Fouladi methods differ, and it is important to understand how they differ in order to predict and compare the coverages of the two methods. When the d is 0.0, both methods produce symmetrical confidence intervals, and in that characteristic, they are similar to central t confidence intervals which can be expressed as a mean value plus and minus a calculated half width.
When the d is nonzero, such as 1.0, all three distributions in Fig. 1a are positively skewed, and the distances from the non-centrality parameter to the confidence limits differ on the right and left sides. I will call these unequal distances segments. With the Hedges & Olkin method, the wider segment is always on the same side as the direction of skewness; that is, a positively skewed distribution will have a wider right segment and a negatively skewed distribution will have a wider left segment. With the Steiger & Fouladi method, the reverse is true: a positively skewed distribution will have a wider left segment and a negatively skewed distribution will have a wider right segment. We can see in the example for d = 1.0 in Fig. 1b that the wider segment for the Hedges & Olkin method is obviously in the direction of skewness to the right, and is less obviously in the opposite direction for the Steiger & Fouladi method (i.e., 1.0 – .001 = .999 on the left and 1.972 – 1.0 = .972 on the right, see segment widths in Appendix 1). The confidence interval for a d of –1.0 would simply be the mirror image of this graph in the negative direction.
The difference in the direction of the wider segment for Hedges & Olkin and Steiger & Fouladi methods has implications for the coverages of the intervals. With effect size 0 (Fig. 1a), the symmetrical Hedges & Olkin interval is wider than the symmetrical Steiger & Fouladi interval. This implies greater coverage for the Hedges & Olkin interval with δ at or near 0. As the effect size increases to a value such as 1.0 in Fig. 1b, the Hedges & Olkin interval is wider in the direction of skewness toward the tails, where there is a smaller probability density. The Steiger & Fouladi interval is wider toward the center of the distribution, where there is a larger probability density. Thus, a small increase in the wider segment of a Steiger & Fouladi interval will add more coverage probability than an equal increase in the wider segment of a Hedges & Olkin interval. Steiger & Fouladi intervals with moderate to large effect sizes can be narrower (more precise) and yet have higher coverage than Hedges & Olkin intervals.
Overview and method of predictions
Before proceeding to the technical details, let us examine the output of the provided software to calculate the expected coverages for all four confidence intervals using d or g and using the Hedges & Olkin or Steiger & Fouladi methods. The procedure is given in Example 3 in the user's notes and the output for 16 degrees of freedom rounded to three decimal digits is displayed in Table 1. Ignoring the intermediate calculations and examining the expected coverages in the far right column, we see that coverage for Hedges & Olkin using d (Hd) is .930, Hedges & Olkin using g (Hg) is .944, Steiger & Fouladi using d (Sd) is .950, and Steiger & Fouladi using g (Sg) is .957.
The expected coverage in Table 1 is the probability of sampling a d value between two boundary d values. These boundaries for d are called "leftd" and "rightd" in the text output in Table 1. These d values are converted to t values called 𝜆̂𝑑,𝑙𝑒𝑓𝑡 and 𝜆̂𝑑,𝑟𝑖𝑔ℎ𝑡 using Eq. 9b. These t boundaries are selected to represent the most extreme possible d value in that direction (left or right) that could have a confidence interval of the given type that still includes δ. That is to say, all d values to the left of leftd will have confidence intervals of that type that do not include δ, and all d values to the right of rightd will also have confidence intervals of that type that do not include δ. The probability of a 𝜆̂𝑑 value to the left of 𝜆̂𝑑,𝑙𝑒𝑓𝑡 (called 𝛼̂𝑙𝑒𝑓𝑡) or to the right of 𝜆̂𝑑,𝑟𝑖𝑔ℎ𝑡 (𝛼̂𝑟𝑖𝑔ℎ𝑡) can be determined directly from the cumulative distribution function of the corresponding noncentral t distribution with non-centrality parameter 𝜆𝛿=𝛿𝐴‾‾√  and ν as given in Eqs. 10 and 11. The sum of these two tail probabilities ("ltail" and "rtail" in Table 1) is the total probability of sampling a d that does not include δ (𝛼̂ = 𝛼̂𝑙𝑒𝑓𝑡 + 𝛼̂𝑟𝑖𝑔ℎ𝑡). The complement, 1 − 𝛼̂, is the expected coverage of the confidence interval as given in the far right column in Table 1.
For Hd and Sd, the d boundaries (leftd and rightd in Table 1) are determined by a process using Hd and Sd confidence intervals directly. For Hg and Sg, the d boundaries (leftd and rightd) are determined by a two-step process: (1) use Hg and Sg confidence intervals to determine g boundaries, then (2) convert g boundaries to d boundaries by dividing the g boundaries by J(ν). For example, rightd for method Hg is computed as rightd = rightg/J(ν) = 1.972/.95225 = 2.071. The leftg and rightg values have been calculated but are not needed for methods Hd and Sd.
The process for discovering these d boundaries involves a computer search for a confidence interval of a given type that has one of the standardized limits equal to the standardized population effect size, δ. The standardized limits of a confidence interval are determined with respect to a corresponding t-scaled non-centrality parameter, and the non-centrality parameter to be used depends on whether the confidence interval is a d interval (𝜆̂𝑑, Eq. 9b) or a g interval (𝜆̂𝑔, Eq. 9c). The left d boundary for determining the expected coverage of Hd or Sd confidence intervals is a d whose Hd or Sd standardized confidence interval has an upper limit of δ. The left d boundary for determining coverage of Hg or Sg confidence intervals is a d whose associated Hg or Sg standardized confidence interval has an upper limit of δ. Note that we do not rescale δ for use with a g interval. In either case, there can be no d value to the left of the selected boundary that has a confidence interval of the same type that includes δ. For the Hg and Sg confidence intervals, this involves converting the d to a g, searching for the g boundary that has an upper limit of δ, and then converting that g boundary back to a d boundary. Any d to the left of that d boundary would have a corresponding g that is to the left of its g boundary. The d boundary generated by a Hg or Sg confidence interval method will not be the same as the d boundary generated by a Hd or Sd confidence interval method.
Computation of d boundaries is done by a computer search algorithm, but the equivalent hand calculation could be done with much effort by trial and error. Suppose we wish to calculate the lower d boundary, leftd, for the Sd method. A standardized confidence interval of the Sd type around this d boundary should have an upper limit δ (i.e., the population standardized effect size), so that is our target d. We select an initial educated guess as to the d boundary, dguess (it must be less than δ if the right limit will be δ), then search for a Sd confidence interval for dguess and examine the upper limit. If it is greater than δ, reduce the size of dguess and try again. If it is less than δ, increase the size of dguess and try again. Stop when the value of dguess generates a confidence interval suitably close to δ. The computer does this to eight decimal digits. Note that similar searches are required for repeatedly computing Steiger & Fouladi confidence intervals on the way to determining that boundary, so the searching is onerous indeed. There are shortcuts to this end, discussed later, but the computer always calculates the value from this basic principle.
Pseudo-algorithm used by computer program
The procedure used by my computer program for determining the expected coverage of a d or g confidence interval for either method follows. Steps beginning "Search for..." are done using specialized search functions written for that purpose.
1. Calculate 𝜆𝛿=𝛿𝐴‾‾√ and ν (Eqs. 9a to 11).
2. If finding the boundaries for Hd or Sd use Step 2a. If finding the boundaries for Hg or Sg use Step 2b. For each instance of the word "[Method]," substitute "Hedges & Olkin Method" if finding the coverage of Hd or Hg confidence intervals or substitute "Steiger & Fouladi Method" if finding the coverage of Sd or Sg confidence intervals.
2a. Hd or Sd procedure
i. Search for the left t boundary, 𝜆̂𝑑,𝑙𝑒𝑓𝑡, which is the non-centrality parameter of the [Method] confidence interval to the left of λδ = 𝛿𝐴‾‾√ that includes λδ as the right-hand confidence limit.
ii. Search for the right t boundary, 𝜆̂𝑑,𝑟𝑖𝑔ℎ𝑡, which is the non-centrality parameter of the [Method] confidence interval to the right of λδ = 𝛿𝐴‾‾√ that includes λδ as the left-hand confidence limit.
2b. Hg or Sg procedure
i. Search for the left t boundary, 𝜆̂𝑔,𝑙𝑒𝑓𝑡, which is the non-centrality parameter of the [Method] confidence interval to the left of λδ = 𝛿𝐴‾‾√ that includes λδ as the right-hand confidence limit. Note that we do not rescale λδ by the bias correction factor because the population value is not biased.
ii. Search for the right t boundary, 𝜆̂𝑔,𝑟𝑖𝑔ℎ𝑡, which is the non-centrality parameter of the [Method] confidence interval to the right of λδ = 𝛿𝐴‾‾√ that includes λδ as the left-hand confidence limit.
iii. Calculate 𝜆̂𝑑,𝑙𝑒𝑓𝑡 = 𝜆̂𝑔,𝑙𝑒𝑓𝑡/J(ν) and 𝜆̂𝑑,𝑟𝑖𝑔ℎ𝑡 = 𝜆̂𝑔,𝑟𝑖𝑔ℎ𝑡/J(ν).
3. Using the noncentral t distribution for λδ and ν determine the cumulative probability, pleft, of 𝜆̂𝑑,𝑙𝑒𝑓𝑡 and assign 𝛼̂𝑙𝑒𝑓𝑡 = pleft. Do the same for the other tail to compute pright based on 𝜆̂𝑑,𝑟𝑖𝑔ℎ𝑡 and assign 𝛼̂𝑟𝑖𝑔ℎ𝑡 = 1 – pright.
4. Calculate 𝛼̂ = 𝛼̂𝑙𝑒𝑓𝑡 + 𝛼̂𝑟𝑖𝑔ℎ𝑡.
5. Calculate expected coverage = 1 – 𝛼̂.
Supplement 1 gives examples of how to calculate expected coverages for both the Hedges & Olkin and Steiger and Fouladi methods used in this paper.
Comments and shortcuts for predicting coverage
These d boundaries are not confidence intervals, although they can be interpreted in a fashion similar to confidence intervals. That is, the tail probabilities represent d values that have confidence intervals that exclude δ, and the probabilities between the boundaries represent d values that have confidence intervals that include δ. However, 𝛼̂ is a dependent variable in this process rather that an independent variable.