# STATISTICA Numerical Precision

## Tests of numerical precision and accuracy of statistical algorithms of the main computational engines of *STATISTICA* (by StatSoft, Inc.)

The following selection of 49 datasets and analysis designs included in these validation benchmarks represent:

- Standard tests of numerical accuracy for floating point mathematical operations (such as the small relative variance test, etc.);
- Published benchmark datasets developed for the purpose of testing statistics programs and used in published reviews of statistics and math packages (including all benchmark datasets proposed in
*"Benchmark datasets for Evaluating Microcomputer Statistical Programs,"*by Elliott, Reisch, and Campbell); and - A comprehensive selection of sample datasets for complex and demanding numerical problems (and some unusual datasets) recommended to us by leading experts in the respective areas of statistics¹ and/or published in statistics textbooks and special monographs, including representative samples of computational problems from the
*"Analysis of Messy Data"*, by Milliken and Johnson (1984), and*"Applied Linear Statistical Models"*, by Neter, Wasserman, and Kutner (1985), as well as books by Box et al., Cox, Lindman, Searle, and other authors.

The accuracy criterion for all benchmarks presented below are either the respective published sources or, where applicable, the internal consistency of the results.

**To the best of our knowledge, STATISTICA is the only statistics package available on the market which has successfully passed every test included in this set of benchmarks (and some tests reported here cannot be passed by any program other than STATISTICA).**

* We are grateful to Dr. Lynn Brecht (UCLA), Dr. John Castellan (Indiana University), Dr. Elazar Pedhazur (New York University), Dr. Dallas Johnson (Kansas State University), Dr. Geoffrey Keppel (University of California, Berkeley), Dr. Michael Kutner (Emory University), Dr. George Milliken (Kansas State University), Dr. Paul Switzer (Stanford University), Dr. William Wasserman (Syracuse University), Dr. Thomas Wickens (UCLA), and Dr. Arthur Woodward (UCLA) for their advice, and for recommending to us some of the datasets used in these validation benchmarks, and to Drs. A. Woodward and L. Brecht for allowing us to use datasets from the technical documentation for *Ganova*. We are also grateful to all those researchers and practitioners who generously provided us with their raw datasets and allowed us to use them in the validation benchmarks. We would appreciate readers' suggestions concerning any additional benchmarks which could be included in this set.

## Examples

###### Example 1: The "small relative variance test" of numerical precision

In the following sample dataset, variable *var2* (the second column) which features a small relative variance is a linear function of *var3* (the third column); thus, the correlation coefficient between any variable (e.g., variable *var1*) and *var2* should be identical to the correlation between that variable and *var3*.

Here are the two correlation coefficients (*var1***var2 *and *var1***var3*) calculated by *STATISTICA* and displayed to fifteen decimal places.

To our knowledge, *STATISTICA* is the only statistical or data mining application program available on the market that will correctly compute these correlations (or correlations from other datasets featuring very small relative variances).

###### Example 2: A medium size multi-factor unbalanced ANOVA design

The following design is a 5 x 5 x 5 x 3 (between-group) x 3 x 3 x 3 (repeated measures) design (with unequal N). Thus there are 375 groups and 27 dependent variables. The between-group design matrix for the highest order interaction has 128 degrees of freedom. Shown below are the univariate and multivariate results

###### Example 3: A medium size multi-factor unbalanced ANOVA design (very large and very small values)

Example 3.1. In the first part of this test, the dataset used in the previous example was transformed by multiplying each dependent variable in the original dataset by 100,000; then, the analysis of variance reported in the previous example was performed on the transformed data. Shown below are the univariate and multivariate results.

Example 3.2. In the second part of this test, the dataset used in Example 2 was transformed by dividing each dependent variable in the original set by 100,000; the analysis of variance reported in Example 2 was then performed on the transformed data. Shown below are the univariate and multivariate results.

###### Example 4: A large multi-factor unbalanced ANOVA design

The following design is a 20 x 10 x 2 x 2 (between-group) x 3 (repeated measures) design with unequal N. Thus, there are 800 groups and 3 dependent variables. The between-group design matrix for the highest order interaction has 171 degrees of freedom. Shown below are the univariate and multivariate results.

###### Example 5: Precision of ANOVA routines (small within-cell variances relative to the between-group variance)

Here is a test of the precision of computations in ANOVA: A data file was created with 10 cases and 5 groups (2 cases per group), and 12 dependent variables. The groups in the grouping variable *IV *were coded 1 through 5. The dependent variables *DVi *(*i *=1 to 12) were then computed as *DVi = IV + casenumber/10**i *(each successive dependent variable was computed as a constant plus the case number divided by 10 to the power of *i*). This results in small within-cell variances relative to the between-group variance.

To our knowledge, *STATISTICA* is the only program available on the market that will correctly compute the within Mean Square Error (MSE) component for all dependent variables in this design.

###### Examples 6-7: Logistic regression, maximum likelihood

### Example 6.

Cox (1970, p. 86) reports data describing the failure (variable *Failure) *of objects as a function of time (*Time*). Cox fitted the data by the logistic model. Shown below are the maximum likelihood estimates and their standard errors produced via *STATISTICA*: Nonlinear Estimation (see also Brown et al., 1983, p. 317).

### Example 7.

A dataset reported in Neter, Wasserman, and Kutner (1985, p. 365) describes the results of a study of coupon redemption.

The coupons differed in their value, that is, with regard to the price reduction offered. The dependent variable of interest is how many coupons of each type were redeemed. Shown below are the maximum likelihood parameter estimates for the logistic regression model computed by *STATISTICA*: Nonlinear Estimation (weighted least squares estimates are reported in Neter et al., p. 365).

###### Example 8: Exponential regression, ordinary least squares

This example is based on a dataset reported in Neter, Wasserman, and Kutner (1985, p. 469). The data contain information on the number of days that each of 15 severely injured patients were hospitalized (variable *Days*) and an index of the prognosis for long-term recovery for each patient (variable *Prognos*). Shown below are the parameter estimates produced by *STATISTICA*: Nonlinear Estimation for the exponential regression model: *Prognos=g0 * exp(g1*Days) *[*g0 *and *g1 *are parameters], the loss function is least squares (see also Neter et al., p. 478, Table 14.3).

###### Example 9: User-defined (exponential) regression, ordinary least squares

The dataset for this example is again based on Neter, Wasserman, & Kutner (1985, p. 484). To study the efficiency of two new manufacturing plants, a ratio was computed of the per-unit-production cost expected in a modern facility after learning has occurred, over the actual per-unit-production cost for selected weeks over a 90-week span. Neter et al. fit the following model to these data: *y = b0 + b1 * xg + b3 * exp(b2*x), *where *xg *is an indicator variable to denote the two plants, *x* denotes the number of weeks, *y* is the efficiency index, and *b0*, *b1*, *b2*, and *b3* are parameters. This formula can be typed "as is" into the user-defined model specification editor. Shown below are the results computed by *STATISTICA*: Nonlinear Estimation (using the Rosenbrock and quasi-Newton Estimation method with start values at 0.01 for all parameters; Neter et. al. report the results on p. 484-485).

###### Example 10: Discontinuity (breakpoint) in regression function

This example is also based on a dataset reported in Neter, Wasserman, & Kutner (1985, p. 348). Specifically, the dataset pertains to a production process in which the per-unit cost is related to the lot size. Supposedly, for lots greater than 500, the relationship between the variables changes; Neter et al. (1985) fit a linear model that allowed for different slopes for lots of sizes less than or equal to 500, and lots greater than 500. Specifically, Neter et al. fit the following model: *y = b0 + b1*x + b2*(x-500)*(x>500) *(*b0, b1, *and *b2 *are parameters). In this model, the logical expression (*x>500*) serves as a multiplier: If the expression is true, it will evaluate to *1*, if it is false, it will evaluate to *0*. Therefore, this equation actually represents two models: *y = b0 + b1*x *for *x<=500, *and *y = b0 + b1*x + b2*(x-500) *for*x>500*. The model can again be typed in to the user-model specification editor "as is"; shown below are the parameter estimates computed by *STATISTICA*: Nonlinear estimation (see Neter et al., p. 348).

###### Example 11: Weighted Least Squares

Weighted least squares or any other (user-specified) loss function can be specified in *STATISTICA*: Nonlinear Estimation. An example of weighted least squares is presented in Neter et al. (1985, p. 169). The example dataset contains information concerning the cost for preparing a bid and the size of the bid. Neter et al. fit a linear regression model (*Bid cost = b0 + b1 * Bid size*), using the residuals weighted by the inverse of the squared *Bid size* values in the loss function [*Loss = ((Predicted-Observed) **2)*(1/Bid size**2)*]. Here are the results computed by *STATISTICA*: Nonlinear Estimation (see Neter et al., p. 169-170).

###### Example 12: Robustness against collinearity problems (a linear model test of accuracy of nonlinear estimation)

The so-called *Longley *data (Longley, 1967) is a well-known dataset for testing linear-least-squares regression programs for their ability to handle regression problems with redundant predictor variables (this dataset is also referenced below for *STATISTICA*: Multiple Regression, Example 27). In this example, it will be used to test the accuracy of the general nonlinear estimation module of *STATISTICA*. In the user-model specification editor of *STATISTICA*: Nonlinear Estimation, we can specify the linear regression model, and request least squares parameter estimates. The parameter estimates computed by *STATISTICA*: Nonlinear Estimation (via *quasi-Newton *iterations) and their (asymptotic) standard errors (computed via finite difference approximation) are shown below (for comparison, see also Elliott, Reisch, & Campbell, 1989, p. 296). Note that *STATISTICA*: Multiple Regression will reproduce the parameter estimates with all 12 digits of precision.

###### Example 13: Unbalanced ANOVA designs (Type I and III Sums of Squares)

Milliken and Johnson (1984, p. 129) discuss in some detail the analysis of a 2 x 3 unbalanced (due to unequal N) between-group design. Shown below are the summary ANOVA tables for that design; both the results for Type I Sequential Sums of Squares (see Milliken & Johnson, p. 142) and Type III Sums of Squares (see Milliken & Johnson, p. 132) are shown below.

###### Example 14: A 2-way nested design

Lindman (1974, p. 167) discusses a two-way nested design where factor *A* has three levels, and factor *B* has six levels, with two levels each nested in each level of factor *A*. Here is the results summary computed by *STATISTICA*: ANOVA/MANOVA (see Lindman, p. 172).

###### Example 15: A 3-way nested design with customized error term

Milliken & Johnson (1984, p. 418) present an example of a 3-way nested design. In this experiment, male and female subjects were randomly assigned to one of 9 environmental chambers; the 9 environmental chambers, in turn, were assigned to 3 levels of a temperature factor. Thus, in this design *Chamber *is nested in *Temperature*, and *Gender* is nested in *Chambers*. To produce the table of sums of squares as presented in Milliken & Johnson (1984, p. 419), the *Gender *by *Chambers *interaction was pooled into the error term before computing the table of all effects.

###### Example 16: A nested design with a random effect

*STATISTICA*: ANOVA/MANOVA will automatically handle random effects. Lindman (1974, p. 173) shows an example of a nested design, where the nested factor is random. Factor *A* has four levels, factor *B* has 3 levels, and factor *C* (subjects) is a random effect with 9 levels. Shown below is the summary table for this design (see Lindman, 1974, p. 178).

###### Example 17: Weighted means analysis of a nested design with unequal N (and missing cells)

The next example was taken from Searle (1987, p. 62). The data presented there describe a two-way nested classification of student opinions concerning computers. There were two classes -- *English *and *Geology* (factor *Course*) -- with different numbers of sections (taught by different teachers): *English *had two sections, *Geology* had 3 sections. To test the main effect for *Course*, Searle constructs a weighted means comparison. Shown below is the result of that comparison as computed by *STATISTICA*: ANOVA/MANOVA (see also Searle, 1987, p. 71).

###### Example 18: A split-plot design with customized error term

Milliken and Johnson (1984, p. 297) present an example of a split-plot design. The design pertains to the effectiveness of 4 different fertility regimes on two varieties of wheat. Each of the four fertilizer levels was randomly assigned to one whole plot within each of one of two blocks. Shown below are the results (see Milliken & Johnson, 1984, p. 299).

###### Example 19: Strip-plot designs

Milliken and Johnson (1984, p. 320) discuss an experiment on the relationship between two irrigation methods and three levels of nitrogen on the yield of wheat. Again, the analysis requires the specification of custom error terms. All sums of squares are automatically computed by *STATISTICA*: ANOVA/MANOVA for the *Table of all Effects*. Note that there is a typographical error in the table presented in Milliken and Johnson (p. 320); specifically the sum of squares for factor *Irrigation* is 570.4 (and not 507.4).

###### Example 20: Split-plot designs with unequal numbers of subplots

Milliken and Johnson (1984, p. 385) discuss an example of such a design. Five patients suffering from depression were randomly assigned to one of two treatment conditions (*Treatment*: *Placebo *vs. *Drug*). They were then examined after one week and after five weeks; the dependent variable was the patients' depression score during those examinations. Two patients did not return for the second examination, creating an unequal number of subplots in the design. In *STATISTICA*: ANOVA/MANOVA the results were produced via analysis of covariance, with covariates coding the effect for subjects within-treatment conditions. Here are the Type III sums of squares for the effects of interest (for a discussion of the choice of error terms see Milliken & Johnson, p. 394).

###### Example 21: Youden square designs

An example of a 4 x 4 Youden square with three factors *A*, *B*, and *C *is presented in Lindman (1974, p. 209). Factor *A* is "rotated" in its position with respect to factor *B*. Here is the table of all effects computed by *STATISTICA*: ANOVA/MANOVA (see also the results table in Lindman, page 209).

###### Example 22: A 4 x 11 nested design with unequal numbers of levels (missing cells)

Milliken and Johnson (1984, p. 415) present an example dataset, comparing 11 insecticides produced by four different companies. One company makes three insecticides, another makes four, and the remainder make two each. Shown below are the results computed by *STATISTICA*: ANOVA/MANOVA (note that these results are slightly different than those reported in Milliken and Johnson on page 422; the analysis reported there is not consistent, and a typographical error must have found its way into the presentation; e.g., compare the mean reported on page 417 for the last group with the data from page 415).

###### Example 23: A 2 (between) x 3 x 3 (repeated measures) design with missing cells

This example is based on a (fictitious) dataset reported in Winer (1962, p. 324). The design has two repeated measures factors, each with 3 levels. Shown below is the summary univariate ANOVA table as computed by *STATISTICA*: ANOVA/MANOVA (see also Winer, p. 328); the multivariate tests for the *Noise x Time* interaction are also shown.

###### Example 24: Longley dataset (linear regression)

The so-called *Longley** *data (Longley, 1967) is a well-known dataset for testing multiple regression programs for their ability to handle regression problems with redundant predictor variables. Shown below are the (partial) results computed by *STATISTICA*: Multiple Regression (see Longley, 1967; Elliott, Reisch, & Campbell, 1989, p. 296).

Note that there is a typographical error in the table presented in Elliott et al., 1989 (Table 4.3.1, p. 296); specifically, the *B* coefficient for *TIME* is 1829.151464__61__4 (as reported in *STATISTICA*) and not 1829.151464__16__ (6 and 1 are reversed).

To our knowledge, *STATISTICA* is the only statistics program available on the market that will correctly compute and report regression coefficients for the Longley dataset with this level of precision (Excel will correctly report the first 8 significant digits, Lotus will correctly report all 12 digits).

###### Example 25: Polynomial regression

Elliott, Reisch, and Campbell (1989, p. 295) present a data file to test polynomial regression. Shown below are the (partial) results computed by *STATISTICA*: Multiple Regression for the sixth degree polynomial fit (see Elliott, Reisch, and Campbell, 1989, p. 297). Note that this test is even more "demanding" than the previous one and an extremely low setting of the minimum tolerance parameter is required to obtain the parameter estimates.

###### Example 26: Kaplan-Meier product limit estimates

Lee (1992, p. 25) discusses a dataset first presented by King et. al. (1979). Shown below is part of the product-limit analysis for the *low-fat* group of rats as computed by *STATISTICA*: Survival Analysis (see also Lee, 1992, p. 74-75).

###### Example 27: Comparing multiple samples of censored survival times

Lee (1992, p. 127) presents a dataset of initial remission times for leukemia patients as a function of three treatments. Shown below is the summary of the comparison computed by *STATISTICA*: Survival Analysis (see also Lee, p. 127).

###### Example 28: Proportional hazard regression for censored data

Crowley and Hu (1977) present an analysis of the well-known Stanford heart transplant data. Shown below are the (partial) results of the (Cox) proportional hazard regression analysis computed by *STATISTICA*: Survival Analysis (see also Brown, Engelman, Jennrich, 1990, p. 773).

###### Example 29: Exponential regression model for censored data

Lawless (1982, p. 287) discusses an example censored dataset pertaining to lung cancer survival and fits to it an exponential regression model with six covariates (plus a constant). Shown below are the parameter estimates and their asymptotic standard errors computed by *STATISTICA*: Survival Analysis.

###### Example 30: Stepwise discriminant function analysis and canonical analysis

The "classic" *Iris** *dataset* **(*Fisher, 1936) is widely referenced to discuss discriminant function analysis. Shown below is the summary of the stepwise discriminant function analysis for those data, and the summary of the canonical analysis with all variables in the model (see also Jennrich 1977, pp. 92-94; Brown et al., 1990, p. 341-342).

###### Example 31: Log-linear model (a 5-way frequency table)

Bishop, Fienberg, & Holland (1978, p. 103) present a complex 5-way frequency table describing the three-year survival of cancer patients in different locations. Shown below are the tests of all models of full order (see also Brown et al., 1983, p. 180; note that *delta=0.5** *was added to each cell in the frequency table).

###### Example 32: Experimental Design: A 2**(7-4) fractional factorial design

Box, Hunter, and Hunter (1978, p. 391) present an example data set for a 2-level fractional factorial design; specifically the design is a *2**(7-4)* fractional factorial. Shown below are the effect estimates as computed by *STATISTICA*: Experimental Design (see also Box, Hunter, & Hunter, p. 392).

###### Example 33: Experimental Design: A second-order central composite (response surface) design

Box, Hunter, and Hunter (1978, p. 519) present an example data set for a 2-factor second-order central composite (response surface) design with two blocks. Shown below are the parameter estimates computed by *STATISTICA*: Experimental Design.

###### Example 34: Experimental Design: A Taguchi robust design experiment (*L18*, S/N: *Smaller-the-Better*)

Phadke (1989, p. 82-83) discusses in detail the analysis of a robust design experiment pertaining to the manufacture of silicon wafers. Shown below is the summary ANOVA table computed by *STATISTICA*: Experimental Design for the *Surface Defect* data (a smaller-the-better problem; see also Phadke, p. 88, Table 4.6); note that as described in Phadke (p. 88), factor *Cleaning* was pooled into the error term.

###### Examples 35-49: Analysis of Benchmark datasets

A standard set of benchmark datasets for the most common analyses was originally proposed by Elliott, Reisch, & Campbell (1989) and has since then been used in published reviews of statistical packages. Shown below are the results for all proposed benchmark analyses (and extensions of some of those tests designed to make them more demanding) as computed by *STATISTICA*.

###### Example 35: Descriptive statistics with small relative variances

Here are the results computed for the example dataset proposed by Elliott et al. (p. 290). To demonstrate the precision of *STATISTICA* we have extended the test to extremely small relative variances (100000000001 to 100000000009).

###### Example 36: Independent group t-test

Here are the results computed for the t-test benchmark dataset proposed by Elliott et al. (p. 290).

###### Example 37: Paired t-test

Here are the results computed for the paired t-test benchmark dataset proposed by Elliott et al. (p. 290).

###### Example 38: One-way ANOVA (test 1)

Here are the results of the one-way ANOVA benchmark (Example 1) proposed by Elliott et al. (p. 291).

###### Example 39: One-way ANOVA (test 2)

Here are the results of the one-way ANOVA benchmark (Example 2) proposed by Elliott et al. (p. 291)

###### Example 40: One-way repeated measures ANOVA

Here are the results for the one-way repeated measures ANOVA benchmark proposed by Elliott et al. (p. 292).

###### Example 41: Two-way ANOVA (balanced)

Here are the results for the two-way balanced ANOVA benchmark proposed by Elliott et al. (p. 292).

###### Example 42: Two-way ANOVA (unbalanced)

Here are the results for the unbalanced ANOVA benchmark data proposed by Elliott et al. (p. 293). We show here only the results for the Type III analysis (as "recommended" by Elliott et al., Table 3.7.2); note that Type I and II analyses can also be performed with *STATISTICA*: ANOVA/MANOVA.

###### Example 43: Simple linear regression

Here are the results (computed via *STATISTICA*: Multiple Regression) for the data proposed by Elliott et al. (p. 294; note that the result reported in Elliott as *r-square* is in fact the result for the simple Pearson correlation coefficient *r*).

###### Example 44: Multiple linear regression (Example 1)

Here are the results for the data proposed by Elliott et al. (p. 295; note that the result reported in Elliott as *r-square** *is in fact the result for the multiple correlation coefficient *r*).

###### Example 45: Multiple linear regression (Example 2)

The next multiple regression benchmark proposed by Elliott et al., 1989 (p. 295, Example 2) is based on the well-known *Longley *dataset (with redundant predictor variables, Longley, 1967). The results of this test are reported in **Example 27**, above. As mentioned before (see our Example 27), there is a typographical error in the table presented in Elliott et al., 1989 (Table 4.3.1, p. 296). Specifically, the *B *coefficient for *TIME *is 1829.151464__61__4 (as reported in *STATISITCA*) and not 1829.151464__16 __(6 and 1 are reversed).

To our knowledge, *STATISTICA* is the only statistics program available on the market that will correctly compute and report regression coefficients for the Longley dataset with this level of precision (Excel will correctly report the first 8 significant digits, Lotus will correctly report all 12 digits).

###### Example 46: Multiple linear regression (Example 3)

Here again are the (partial) results for the polynomial regression problem reported in Elliott et al. (1989, p. 297). Note that this test is even more "demanding" than the previous one and an extremely low setting of the minimum tolerance parameter is required to obtain the parameter estimates for the sixth order polynomial.

###### Example 47: A 2 x 2 contingency table and Fisher exact test

Here are the results for the 2x2 contingency table presented in Elliott et al. (p. 295).

###### Example 48: An R x C contingency table (Example 1)

Here are the (partial) results for the 2x4 contingency table presented in Elliott et al. (p. 298). Note that the expected frequency for group *Male/Hair_Red** *is incorrectly reported in Elliott et al. as 3.125 (and thus the expected frequencies in the third column do not add up to the marginal frequency); the correct expected frequency for this cell is 8.33333.

###### Example 49: An R x C contingency table (Example 2)

Here are the (partial) results for the 2x4 contingency table presented in Elliott et al. (p. 298). Note that the expected frequency for group *Negative/Days_0** *is incorrectly reported in Elliott et al. as 14628 (and thus the expected frequencies in the first column do not add up to the marginal frequency); the correct expected frequency for this cell is 14628.5.

## References

Bishop, Y. M. M., Fienberg, S. E., & Holland, P. W. (1975). *Discrete multivariate analysis*. Cambridge, MA: MIT Press.

Box, G. E. P., Hunter, W. G., & Hunter, S. J. (1978). *Statistics for experimenters: An introduction to design, data analysis, and model building.* New York: Wiley.

Brecht, L., & Woodward, A. (1985). *Ganova* (Technical Documentation). Unpublished Manuscript.

Brown, M. B., Engelman, L., & Jennrich, R. I. (1990). *BMDP, Statistical software manual. *Los Angeles: University of California Press.

Brown, M. B., Engelman, L., Frane, J. W., Hill, M. A., Jennrich, R. I., & Toporek, J. D. (1983). *BMDP, Statistical software manual. *Los Angeles: University of California Press.

Cox, D. R. (1970). *The analysis of binary data*. New York: Halsted Press.

Crowley, J., & Hu, M. (1977). Covariance analysis of heart transplant survival data. *Journal of the American Statistical Association,**72*, 27-36.

Elliott, A. C., Reisch, J. S., & Campbell, N. P. (1989).* Benchmark datasets for evaluating microcomputer statistical programs. Collegiate Microcomputer*,*11*, 289-299.

Enslein, K., Ralston, A., & Wilf, H. S. (1977). *Statistical methods for digital computers.* New York: Wiley.

Fienberg, S. E. (1977). *The analysis of cross-classified categorical data*. Cambridge, MA: MIT Press.

Finn, J. D. (1974). *A general model for multivariate analysis. *New York: Holt, Rinehart & Winston.

Finn, J. D. (1977). Multivariate analysis of variance and covariance. In K. Enslein, A. Ralston, and H. S. Wilf (Eds.). *Statistical methods for digital computers.* Vol. III, New York: Wiley.

King, M., Bailey, D.M., Gibson, D.G., Pitha, J.V., & McCay, P. B. (1979). Incidence and growth of mammary tumors induced by 7,12-dimethylbenz (alpha) antheacene as related to the dietary content of fat and antioxidant. *Journal of the National Cancer Institute, 63, *656-664.

Lawless, J. F. (1982). *Statistical models and methods for lifetime data.* New York: Wiley.

Lee, E. T. (1992). *Statistical methods for survival data analysis *(2nd edition). New York: Wiley.

Lindman, H. R. (1974). *Analysis of variance in complex experimental designs*. San Francisco: W. H. Freeman & Co.

Longley, J. W. (1967). An appraisal of least squares programs for the electronic computer from the point of view of the user. *JASA*,*62*, 819-831.

Milliken, G. A. & Johnson, D. E. (1984). *Analysis of messy data. Vol. I: Designed experiments.* New York: Van Nostrand Reinhold, Co.

Neter, J., Wasserman, W., & Kutner, M. H. (1985). *Applied linear statistical models: Regression, analysis of variance, and experimental designs.* Homewood, Ill.: Irwin.

Phadke, M. S. (1989). *Quality engineering using robust design. *Englewood Cliffs, NJ: Prentice Hall.

Searle, S. R. (1987). *Linear models for unbalanced data. *New York: Wiley.

Winer, B. J. (1962). *Statistical principles in experimental design*. New York: McGraw-Hill. (2nd edition, McGraw-Hill, 1971).

Woodward, J. A., Bonett, D. G., & Brecht, M. L. (1990). *Introduction to linear models and experimental design.* New York: Harcourt, Brace, Jovanovich.

This material was developed by StatSoft, Inc. StatSoft, Inc. does not copyright the selection of the benchmark materials used in this text and explicitly encourages the use of those tests by others to the benefit of all statistics software. The proper citation for this selection of validation benchmarks is:*Validation Benchmarks for Statistical Algorithms (version 1B).* (1992-2009). Tulsa, OK: StatSoft. We would appreciate your comments or questions.