Design Your Experiments Part V:
A Formal Design Method
by Kevin T. Kilty
Why use a formal design
method?
There are several reasons...
- When a process might involve
many different parameters the formality of experimental design will
help insure that I gain information about all relevant parameters.
- It provides a means of helping
me improve the quality of information I gain from an experiment.
- It suggests how much replication
I require and how to obtain it.
- It lets me know what types
of interactions among factors might be aliased in the event that I have
not enough replication.
Definitions
Over the next few installments
I may refer to these concepts without stopping to explain them...
- Factors. Factors
might be continuous values or category. A continuous factors has a real
number measure indicating its magnitude, but a category has merely a
label.
- Parameters. Coefficients
in a mathematical model of a process which indicate the gain of a factor
or a product of factors.
- Treatment. A specific
combination of factors at particular levels.
- Experimental unit.
The smallest unit to which we can apply a treatment.
- Observational unit.
That unit of an experiment which we measure.
- Replication. Redundant
application of treatments to experimental units. This allows us to estimate
experimental error. Replicating observational units merely minimizes
the impact of observational error.
- Randomization. Performance
of experimental runs randomly in an effort to eliminate systematic errors.
- Local control of error.
Control of everything except the factors involved in the treatment in
some way or other. Examples include pairing of data and randomizing
assignment of treatments to experimental unit.
Examples of these concepts:
Cleaning capability of a solvent.
My objective is to measure
the surface cleaning capability of a solvent such as isopropyl alcohol
(IPA) in water. There is only one factor in this, which is the concentration
of IPA in the water. I plan to use a high concentration and a low one;
so, I am testing the factor at two levels. There is a problem with the
experiment in that a surface may have distinct cleaning properties not
related to concentration of IPA. This is a source of noise in the experiment
I can handle with local control of error. The method of control I plan
to use is to divide each test surface into two samples. I perform cleaning
using one level of IPA on one sample of the pair and use the other level
of IPA on the other sample. I will pair the results for analysis. The
result of my experiment is N different blocks each with 2 experimental
units. Replication in the experiment I get by having N experimental units
for each of the two IPA concentrations. I randomly allocate IPA concentration
to sample within each block, and randomly order my testing. Pairing and
randomization provide local control of error.
Factorial Design
I'll describe a randomized
complete block design using what is known as a contrasts table to represent
it. First, I decide upon the factors that my experiment will test. At
the present time I'll denote these by capital letters such as A, B, C,
and so forth. I'll use subscripted symbols xai, xbi,
and so forth to indicate the value of a factor on the ith replication
of an experiment. Factors could represent a continuous quantity like temperature
or pressure, or a category like the manufacturer of a product. A factorial
design uses every possible combination of treatments.
The contrasts table is a way
of representing this. Typically, when I run an experiment to screen many
factors to find those having the most influence, I apply factors at two
levels, a high level and a low one. The high level I indicate with a +1
and the low level with a -1. The exact value of the level is unimportant
at this time, but as an example, suppose the factor is temperature. Then
the +1 level could be 75C and the -1 level could be 45C. In other words
I have scaled the actual units of a factor in such a way that 30C corresponds
to +2 change in factor level. I need this information later to return
to true units.
The form of the contrasts table
at this stage involves only the factors (I=a constant intercept) and particular
levels involved in the experiment. The contrasts table helps insure that
I include all factor combinations.
Contrasts Table
Factor Levels
I A B C...
-----------------------
1 -1 -1 -1
1 +1 -1 -1
1 -1 +1 -1
1 +1 +1 -1
1 -1 -1 +1
1 +1 -1 +1
1 -1 +1 +1
1 +1 +1 +1
-----------------------
At this point I see that I'll
have 8 distinct treatments, so I will need materials to produce at least
8 experimental units. I'll allocate treatments randomly to the experimental
units. In a completely randomized design all treatment combinations
I allocate in such a way that each experimental unit has the same chance
of receiving any treatment. If I have organized my experiment into blocks
to help control error locally, such as a paired experiment, then I need
to make up sufficient experimental units per block for me to apply all treatments
per block. This is a randomized complete block design.
A mathematical model
Consider an experiment involving
only 2 factor levels. The most general model I can produce from it is
one that includes terms involving each factor raised to the first power.
I cannot go beyond first power because I am testing each factor only at
two levels and only a line through two points is a unique construction.
I can, however, include products of factors. So a possible model from
my experiment is...
Yi = C0 + C1xai + C2xbi + C4xaixbi + ni
where,
The Y's are an outcome, or objective,
of the experiment. The C=92s are coefficients which I can determine through
the experiment outcome. They describe the gain that each factor supplies
in controlling the outcome of the experiment. The factor n is an error or
noise component. If this model is a true model, then the n's are nothing
more than random variables. That is, they are realizations which follow
some noise density. If the model is not a true one, then in addition to
random noise the n's contain wrong model noise. Wrong model
noise may look so much like random noise that I cannot tell the difference.
However, I can reduce or even eliminate the wrong model noise by choosing
a better model.
Concept of interaction
The model I introduced above
has linear terms in each factor, so that I can calculate the effect of
each factor in isolation. The factors in a factorial design experiment
are like basis vectors in a space. The parameter associated with each
factor is orthogonal to the parameter for each other factor. You can show
this by using a dot product if you wish.
There are also terms in the
products of factors. These allow me to represent interaction of one factor
with another. By interaction I mean that the effect of one factor or the
other is not solely dependent on the level of that factor, but depends
also on the level of another factor. The contrasts table below shows all
of the factors and their various possible interactions. Depending on how
many interactions I plan to calculate, a full factorial design like this
might have hidden (implicit) replication in the sense that it contains
more information than I need at a minimum to calculate all interactions.
The minimum requirement is that I need one experiment per interaction
or factor I plan to calculate. If my mathematical model has X unknown
coefficients, then I need at least X experiments to calculate them. If
I hope to account for experimental noise, then I need more experiments
beyond this minimum, and I'll obtain them from replicating experiments
at some, or all, treatments.
Once I have used data to estimate
parameters, some degree of freedom in the data is gone. One degree of
freedom per parameter is the rule. However, I have an estimate of n for
each experimental result. This means the estimates of n from one observation
to the next are never fully independent of one another.
Contrasts Table
Factor Levels and Interactions
I A B C AB AC BC ABC
-------------------------------------------------
1 -1 -1 -1 +1 +1 +1 -1
1 +1 -1 -1 -1 -1 +1 +1
1 -1 +1 -1 -1 +1 -1 +1
1 +1 +1 -1 +1 -1 -1 -1
1 -1 -1 +1 +1 -1 -1 +1
1 +1 -1 +1 -1 +1 -1 -1
1 -1 +1 +1 -1 -1 +1 -1
1 +1 +1 +1 +1 +1 +1 +1
-------------------------------------------------
The use of this model and experiment
design predates the modern computer, so there are manual methods of estimating
the effects of factors and the value of coefficients by hand quickly.
It is more expedient to use a spreadsheet like Excel to perform a regression
analysis on the experimental data. However, let me explain the manual
method.
In the table below I have included
only the first four columns of the contrasts table, but I have added an
extra column which I built from the headings of the contrasts table by
noting which factor or combinations of factors have plus signs. I have
added a further column for the observed effect (Y) for each treatment
from the outcome of each experiment.
To find the coefficient appropriate
to factor A, I simply use the sign in the 'A' column as a multiplier for
the corresponding experimental result. Then I add all of the effects and
divide by the number of rows having the same sign for each factor. In
other words, I calculate the average response at the high level minus
the average response at the low level.
Contrasts Table
Factor Levels
I A B C Y
-----------------------...-------------
1 -1 -1 -1 I 20
1 +1 -1 -1 A 18
1 -1 +1 -1 B 3
1 +1 +1 -1 AB -5
1 -1 -1 +1 C 6
1 +1 -1 +1 AC 15
1 -1 +1 +1 BC 23
1 +1 +1 +1 ABC -2
-----------------------...-------------
So, as an example I can find the
coefficient for The 'A' factor, the coefficient in front of xa
in the model, as...
Effect of A = (A+AB+AC+ABC)/4 - (I+B+C+BC)/4 = (18-5+15-2)/4-(20+3+6+23)/4=-6.5
I can find all of the other effects
similarly. Remember that the coefficient found through this method is scaled
to a total difference of 2 in the response. To convert to actual units I
need to return to the original scaling I did to set up levels, and recall
that a difference of 2 for my temperature response is actually 30C.
Using a probability plot
It is rare to compute a model
by hand nowdays. Generally I use standard regression software, running
as a stand alone program, or using Excel, to compute the regression coefficients.
The regression software will also give me statistics regarding how well
the data fit the model. However, an alternative means of evaluating the
success of the experiment is available by making a normal probability
plot. Let me explain this now, and save the use of a spreadsheet for later.
Suppose I have a model of three
factors and all possible interactions, including the interaction term
for all three factors. I have 7 effects values. I can rank these from
smallest to largest and graph each against a percentile rank calculated
as (i-0.5)/k, where i is the rank of the factor and k is the total number
of factors. I can also calculate the standard normal deviate for these
ranks using a tables book. I then plot the magnitude of the effects, which
I have just calculated by hand, against the normal deviate. This produces
what people call a normal probability plot. If all effects
land along a single line, then they are consistent with the hypothesis
that all of the true coefficients in the model (the effects) are actually
zero, and they are only obtaining non-zero values through noise inspired
fluctuation. If one or several of the effects land far from the straight
line, I mean unusually far to the right or left, then this implies effects
which are over represented compared to normal fluctuations and must have
a significance to them.
The normal probability plot
is more qualitative than it is quantitative. The analysis of variance,
using as it does the F ratio test, is much better for quantitative analysis.
Center runs when the factor
is continuous.
My experiments to this point
use just two factor levels. When factors are continuous then it makes
sense for me to place a third factor level at the mid-point between the
two extreme values, or even take additional data between any two factor
levels. What I gain by doing this is:
- I might want to see whether
there is non-linear behavior of a factor.
- In the case where I plan
to use my experiment to optimize something, the optimum is likely to
fall between two values and I ought to perform a reality check to see
if an intermediate value actually does improve the process.
- Center runs provide me with
additional replication. This helps to estimate experimental noise.
Non-linear factors involve new
terms in the model that have forms like...
Ckxai2
In the next installment I'll
show an example of an experiment planned from the beginning, and the analysis
of experimental results using Excel. 
|
Sponsored by: