This tutorial demonstrates the use of Design-Expert® software for two-level factorial designs. These designs will help you screen many factors to discover the vital few, and perhaps how they interact. If you are in a hurry, skip the “Note” sections—these are sidebars for those who want to spend more time and explore things.

Note

**Fundamental features of the program**: Before going any
further with this tutorial, go back and do the one on a General One-Factor
experiment. Features demonstrated there will not be detailed here.

The data you will now analyze comes from Douglas Montgomery’s textbook, *Design
and Analysis of Experiments*, published by John Wiley and Sons, New York. A
waferboard manufacturer must immediately reduce the concentration of formaldehyde
used as a processing aid for a filtration operation, or they will be shut down by
regulatory officials. To systematically explore their options, process engineers
set up a full-factorial two-level design on the key factors, including
concentration at its current level and an acceptably low one.

At each combination of these process settings, the experimenters recorded the filtration rate. The goal is to maximize the filtration rate and also try to find conditions that allow a reduction in the concentration of formaldehyde, Factor C. This case study exercises many of the two-level design features offered by Design-Expert. It should get you well down the road to being a power user. Let’s get going!

Note

**What to do if a factor like temperature is hard to change**:
Ideally the run order of your experiment will be completely randomized, which
is what Design-Expert will lay out for you by default. If you really cannot
accomplish this due to one or more factors being too hard to change that
quickly, choose the Split-Plot design. However, keep in mind that you will pay
a price in reduced statistical power for the factors that become restricted in
randomization. Before embarking on a split plot, take the “Feature Tour” to get
an orientation on how Design-Expert designs such an experiment and what to
watch out for in the selection of effects, etc.

Start the program and click **New Design**.

You now see four branches to the left of your screen. Stay with the Factorial
choice, which comes up by default. You’ll be using the default selection:
**Randomized Regular Two-Factorial**.

Note

Design-Expert’s design builder offers
full and fractional two-level factorials for 2 to 21 factors in powers of two
(4, 8, 16…) for up to 512 runs. The choices appear in color on your screen.
White squares symbolize full factorials requiring 2^{k} runs for k (the
number of factors) from 2 to 9. The other choices are colored like a stoplight:
green for go, yellow for proceed with caution, and red for stop, which represent
varying degrees of resolution: ≥ V, IV, and III, respectively. For a quick
overview of these color codes, press the screen tips button (or select
**Tips**, **Screen Tips**) and click topic 1: “**What type of information do
you want?**”

Let’s get on with the case at hand – a full-factorial design. Click the white
square labeled **2**^{4} in column 4 (number of factors) in the Runs row
labeled 16.

Click the **Next** button. You can also just double-click on a design to select
it and continue. You can now enter the names, units of measure, and levels for
your experimental factors. Use the arrow keys, tab key, or mouse to move from one
space to the next. Enter for each factor (A, B, C and D) the **Name**, **Units**,
**Low** and **High** levels shown on the screen shot below.

Note

**How to enter alphanumeric levels**: Factors can be of two
distinct types – “Numeric” or “Categoric.” Numeric data characterizes a
continuous scale such as temperature or pressure. Categoric data, such as
catalyst type or automobile model, occurs in distinct levels. Design-Expert
permits characters (for example, words like “Low” or “High”) for the levels of
categorical factors. You change the type of factor by clicking on cells in the
Type column and choosing “Categoric” from the drop down list, or by typing “C”
(or “N” for numeric). Give this a try – back and forth! Leave the default as
“Numeric” for all factors in this case.

Now click **Next** to bring up the Responses dialog box. With the list arrow you
can enter up to 999 responses (more than that can be added later if you like).
In this case we only need to enter a single response name (**Filtration Rate**)
and units (**gallons/hour**) as shown below.

It is good to now assess the power of your experiment design. In this case,
management does not care if averages differ by less than **10** gallons per hour
(there’s no value in improvements smaller than this). Engineering records
provide the standard deviation of **5** (the process variability). Enter these
values as shown below. Design-Expert then computes the signal to noise ratio
(10/5=2).

Press **Next** to view the positive outcome – power that exceeds 80 percent
probability of seeing the desired difference.

Click **Finish** to accept these inputs and generate the design layout window.

You’ve now completed the first phase of DOE – the design. Notice that this is one of four main features (branches) offered by Design-Expert, the others being Analysis, Optimization and Post Analysis (prediction, confirmation, etc.).

Note

Click the Notes node to take a look at what’s written there by default. Add your own comments if you like.

Exit the notes page by clicking the Design node. (Notice the node appears as Design (Actual) – meaning your factors are displayed in actual levels, as opposed to coded form).

You’ve put in some work at this point so it is a good time to save it. The quickest way of doing this is to press the standard save icon .

At this stage you normally would print the run sheet, perform the experiments,
and record the responses. The software automatically lists the runs in randomized
order, protecting against any lurking factors such as time, temperature,
humidity, or the like. For this tutorial we will just load the data by clicking
**Help, Tutorial Data -> Filtration Rate**.

Double-click the **Std** column header (on the gray square labeled Std) to sort
as shown below.

Your data should now match the screen shot shown below except for a *different
random run order*. (When doing your own experiments, always do them in random
order. Otherwise, lurking factors that change with time will bias your results.)

Note

**How to adjust column widths**: Note that the Column Header
for Factor 3 is truncated (ie. Concentrat… percent). To automatically re-size
the column, move the cursor to the right border of the column header until it
turns into a double-headed arrow. Double-click and the column will be
resized to fit the Column Header.

Now that you’ve got responses recorded, it’s another opportune time to save the updated file by clicking the Save icon .

Note

**How to change number formats**: The response data came in
under a general format. In some cases you will get cleaner outputs if you change
this to a fixed format. Click the **Response** column heading (top of the
response column), and look at the **Design Properties** pane on the left of
your screen. Click the **Format** box and you’ll see the down arrow appear next
to **General**. Click the down arrow and select **0.0**

Using the Design Properties pane, you can also change input factors’ format, names, or levels. Try this by clicking any other column headings.

Design-Expert provides two methods of displaying the levels of the factors in a design:

- Actual levels of the factors.
- Coded as -1 for low levels and +1 for high levels.

The default design layout is actual factor levels in run order.

Note

To view the design in coded values, click **Display Options** on the
menu bar and select **Process Factors - Coded**. Your screen should now look
like the one shown below.

Notice that the Design node now displays “coded” in parentheses – **Design
(Coded)**. This can be helpful to see at a glance whether anyone changed any
factor levels from their design points.

Now convert the factors back to their original values by clicking on **Display
Options** from the menu bar and selecting **Process Factors - Actual**.

Design-Expert provides various ways for you to get an overall sense of your data before moving on to an in-depth analysis. For example, you can quickly sort columns by double-clicking on them.

To see this, move your mouse to the top of column **Factor 1** (A: Temperature)
and double-click to sort ascending. Double-click again and it will be descending.
Do it once more (the arrow should be pointing down) to see how going from low to
high temperature affects Filtration.

You will now see more clearly the impact of temperature on the response. Better
yet, you can make a plot of the response versus factor A by selecting the **Graph
Columns** node that branches from the design ‘root’ at the upper left of your
screen. You should now see a scatter plot with (by default) factor A:Temperature
on the X-axis and the response of Filtration Rate on the Y-axis.

Observe by looking at the graph how temperature makes a big impact on the response. This leads to the high correlation reported on the legend.

Another indicator of the strong connection of temperature to filtration rate is the red color in the correlation grid at the intersection of these two variables. Note that you can also see the correlation number just above the grid next to a colored scale indicating correlation.

To see the impact of the next factor, B, click the next square to right as shown above. Notice now that pressure has little correlation with filtration rate—this relationship turns out to be insignificant (correlation is low at 0.08)

Note

**How to use the Color By feature**:
Go back to the scatter plot of **A** versus **Filtration Rate**. By default the
points are colored by standard order. Click the **Color By** drop-down list and
select **C: Concentration** as shown below.

Do you see how two colors stratify at each level of temperature – but oppositely – red at the top for the rates plotted at the left (temperature 24) versus blue coming out higher for filtration rate at the right? Consider what this may indicate about how concentration interacts with temperature to produce an effect on filtration rate. However, let’s not get ahead of ourselves – this is only a preliminary to more thorough analyses using much more sophisticated graphical and statistical tools.

You may wonder why the number “2” appears besides a few points on this plot. This notation indicates the presence of multiple points at the same location. Click on one of these points more than once to identify the individual runs (look at the legend to the left of the graph).

Coming up soon, you will use powerful analysis features in Design-Expert to find out what’s really going on in this wafer-board production process.

To begin analyzing the design, click the **Filtration Rate** response node on the
left side of your screen. This brings up the analytical tool bar across the top
of the screen. To do the statistical analysis, simply click the tabs
progressively from left to right.

The **Transform** tab is initially highlighted, as shown below. It displays a
list of mathematical functions that you may apply to your response.

Note

For helpful advice on when to transform your response (or not!), press the Tips button.

Near the bottom of your Transform screen, Design-Expert notes that the response
range is more than two-fold (“Ratio of max to min is 2.4186”). This number falls
below the ratio of 3 where “…power transforms have little effect.” Therefore,
you can leave the transformation at its default: **None**. You will also see a
diagnostics plot later on (Box-Cox) that will alert you when a transform may
help.

Click the **Effects** tab. The program displays the absolute value of all effects
(plotted as squares) on a half-normal probability plot. Color-coding provides
details whether the effects are positive or negative.

Note the message on your screen: “Select significant terms – see Tips.” You must choose which effects to include in the model. If you proceed without doing so at this point, you will get a warning message stating “You have not selected any factors for the model.” The program will allow you to proceed, but with only the mean as the model (no effects), or you can opt to be sent back to the Effects view (a much better choice!).

You can select effects by simply clicking on the square points. Start with the largest effect at the right side of the plot, as shown below.

By default the red “error line” will be placed such that it represents the smallest 50% of the effects. It is intended to be a visual guide to assist with selecting effects. The unimportant effects should line up on a line near zero. If the line doesn’t match the small effects, you can click and drag it to line up with group of smallest (on the left) effects.

Keep selecting individual effects from right to left until you’ve selected all the effects that are off the line. In this case, the last effect to select is factor C. There is a big gap between C and the smaller effects that line up near zero. This gap is often a good indication to stop selecting effects.

Note

Press the handy screen button to learn more about using the half-normal plot to select effects.

You can also select multiple effects at once by dragging a box around them.

To really see the magnitude of the chosen effects, display them on an ordered
bar chart. Do this via the **Effects Tool**, **Pareto**. Notice the vertical
axis shows the t-value of the absolute effects. This dimensionless statistic
scales the effects in terms of standard deviations. In this case, it makes no
difference to the appearance of the chart, but when you encounter botched factor
levels, missing data, and the like, the t-value scale provides a more accurate
measure of relative effects. Click the next biggest bar and notice it is
identified as **ABD** as shown below (lower right).

Notice that the **ABD** bar falls below the bottom limit, so click the bar again
to deselect it.

Note

To see quantitative detail on the chosen model effects and those
remaining for estimation of error, click **Numeric**. This screen also enables
another method of model selection via the “Autoselect…” button. For now, we’ll
stick with the model chosen on the half-normal plot. Autoselect is more useful
for response surface designs and will be detailed in those tutorials.

It is now time to look at the statistics in detail with the analysis of variance
(ANOVA) table. Click the **ANOVA** tab to see the selected effects and their
coefficients. By default, Design-Expert provides annotations in blue text.

Note

Annotations can be toggled off via the View menu.

Check the probability (“p-value”) for the Model. By default, Design-Expert
considers values of ≤ 0.05 to be significant. This can be changed via
**Edit > Preferences > Math Preferences > Math Analysis**.

Inspect the p-values for the model terms A, C, D, AC, and AD: All pass the 0.05 test with room to spare.

Note

**Context-sensitive Help**: Definitions for numbers on
the ANOVA table can be obtained by right clicking and choosing Help. Try this
for the Mean Square Residual statistic, as shown below.

Observe the other panes of the ANOVA output for further statistics such as
R-squared and the like. Refer to the annotations and also access Help for
details. Take a look at the **Coefficients** tab to see estimates for the model
coefficients and their associated statistics. Lastly, click the **Coded
Equation** and **Actual Equation** tabs for the predictive equations both in
coded and uncoded form.

Rather than belabor the numbers, let’s move on and ultimately let the effect graphs tell the story. However, first we must do some diagnostics to validate the model.

Click the **Diagnostics** tab to generate a normal probability plot of the
residuals.

By default, residuals are studentized – essentially a conversion to standard deviation scale. Also, they are done externally, that is, with each result taken out before calculating its residual. Statisticians refer to this approach as a “case deletion diagnostic.” If something goes wrong in your experiment or measurement and it generates a true outlier for a given run, the discrepant value will be removed before assessing it for influencing the model fit. This improves the detection of any abnormalities.

Note

**Raw residuals**: For standard two-level factorial designs like
this, plotting the raw residuals (in original units of measure) will be just
as effective. On the **Diagnostics Toolbar**, pull down the options
as shown and choose **Residuals** (raw) to satisfy yourself that this is true
(the pattern will not change appreciably).

We advise that you return to studentized scale in the end, because as a general rule this is the most robust approach for diagnosing residuals.

Ideally the normal plot of residuals is a straight line, indicating no abnormalities. The data doesn’t have to match up perfectly with the line. A good rule of thumb is called the “fat pencil” test. If you can put a fat pencil over the line and cover up all the data points, the data is sufficiently normal. In this case the plot looks OK, so move on.

Select the **Resid. vs. Pred.** (residuals versus predicted) tab, shown below.

The size of the residual should be independent of its predicted value. In other words, the vertical spread of the studentized residuals should be approximately the same across all levels of the predicted values. In this case, the plot looks OK.

Next select **Resid. vs. Run** to see a very useful plot that’s often referred
to as “Outlier t” because it shows how many standard deviations (t-values) a
given run falls off relative to what one would expect from all the others.

Design-Expert provides upper and lower red lines that are similar to 95% confidence control limits on a run chart. In this case none of the points stands out. Because this graph is plotted in randomized run order, the ordering of the points on your screen will be different than shown here. Normally, though, you should look for patterns, not just outliers. An obvious example would be a steady decrease in residuals from start to finish, in other words, a downward trend. That would be cause for concern about the stability of your system and merit investigation. However, by running the experiment in random order, you build in protection against trends in response biasing the results.

Note

**What to do if a point falls out of the limits**: If there
were an outlier, you could click on it to get the coordinates displayed to the
left of the graph. The program remembers the point. It will remain highlighted
on other plots. This is especially helpful in the residual analysis because you
can track any suspect point. This feature also works in the interpretation
graphs. Give it a try! Click anywhere else on the graph to turn the point off.

Even more helpful may be the option to highlight a point as shown below.

This flags the run back in your design layout and elsewhere within Design-Expert so you can keep track of it. Check it out!

Skip ahead to the **Box Cox** plot. This was developed to calculate the best
power law transformation. (Refer to Montgomery’s Design and Analysis of
Experiments textbook for details.) The text on the left side of the screen gives
the recommended transformation: in this case, “None.” That’s all you really need
to know!

Note

For those of you who want to delve into the details, note that the Box-Cox screen is color coded to help with interpretation. The blue line shows the current transformation. In this case it points to a value of 1 for “Lambda,” which symbolizes the power applied to your response values. A lambda of 1 indicates no transformation. The green line indicates the best lambda value, while the red lines indicate the 95% confidence interval surrounding it. If this 95% confidence interval includes 1, then no transformation is recommended. It boils down to this: If the blue line falls within the red lines, you are in the optimal zone, so no change is necessary in your response transformation.

P.S. The Box Cox plot will not help if the appropriate transformation is either the logit or the arcsine square root transformation. See the program’s Help system write-up on “Response Transformations” for further details.

Go back and select the **Report** tab. Here you see the numerical values for
diagnostic statistics reported case-by-case. Discrepant values will be flagged.
In this case nothing is detected as being abnormal.

Assuming that the residual analyses do not reveal any problems (no problems are evident in our example), it’s now time to look at the significant factor effects.

On the analytical tool bar at the top of the screen, choose the **Model Graphs**
tab. The AC interaction plot comes up by default. (If your graph displays the x-axis
in coded units, return to actual units by choosing **Display Options**,
**Process Factors - Actual**.)

The “I-Beam” symbols on this plot (and other effect plots) depict the 95% least significant difference (LSD) interval for the plotted points.

Those points that have non-overlapping intervals (i.e. the LSD bars don’t intersect or overlap from left to right through an imaginary horizontal line) are significantly different.

Note

**How to do pairwise comparisons**: An easy way to verify
separation is to do a pairwise comparison. Click on any model prediction (for
example the triangle in the middle of the red LSD bars on the left), and you
will be shown the pairwise comparisons.

A horizontal line is drawn through the predicted mean of the highlighted point. Any vertical bars that overlap with this horizontal line indicate predicted means that are not significantly different from the selected point (for example, the red triangle at the right of the graph indicating the prediction for A+, C+). The legend will also tabulate which means are significantly different. Note that even though the displayed pairwise tests are two-sided, only half of the interval is displayed for easier interpretation.

Note also that the spread of the points on the right side of the graph (where Temperature is high) is smaller than the spread between the points at the left side of the graph (where Temperature is low.) In other words, the effect of formaldehyde concentration (C) is less significant at the high level of temperature (A). Therefore, the experimenters can go to high temperature and reduce the concentration of harmful formaldehyde, while maintaining or even increasing filtration rate. This combination is represented by the black square symbol at the upper right of the interaction plot.

The **Factors Tool** opens along with the default plot, pinned to the
right side of the screen. Move the tool as needed by clicking the top blue
border and dragging it. Click and drag the tool back to the right side of the
screen until you see a blue shadow, then release to lock it back in place. This
tool controls which factor(s) are plotted on the graph. At the bottom of the
Factors Tool is a pull-down list from which you can also select the factors to
plot. Only the terms that are in the model are included in this list.

Note

**What happens if you pick a main effect that’s involved in
interaction**: Click the Term list down-arrow and select **A**. Notice that the
Graphs Tool shifts from Interaction to One Factor.

More importantly, pay heed to the warning at the top of the plot of A (Temperature). It states “Warning! Factor involved in multiple interactions.” You should never try to interpret main effects plots of factors involved in interactions because they provide misleading information.

Let’s do something more productive at this stage: Go back to the **Factors
Tool** and select from the **Term** list the other significant interaction,
**AD**.

Notice on the Factors Tool that factors not already assigned to an axis (X1 or X2) display a vertical slider bar. This allows you to choose specific settings. The bars default to the midpoint levels of these non-axis factors.

Note

**Using the factor sliders**: You can change levels of fixed
factors by dragging the bars, or by typing the desired level in the numeric
space near the bottom of the Factors Tool. Check this out by grabbing the slide
bar for factor **C: Concentration** and moving it to the left. Notice how the
interaction graph changes.

It now becomes clear that a very high filtration rate can be achieved by going to the high stir rate (the red line for factor D). Click this high point to get all the details on its response and factor values. This is the optimum outcome.

To reset the graph to its default concentration, you could type “3” in the box at the bottom of the Factors Tool (below Stir rate). Factor C must be clicked and highlighted for this to take effect. You can also get the original settings back by pressing the Default button. Give it a try, but remember that going to low concentration of formaldehyde was a primary objective for this process-troubleshooting experiment, so be sure to slide C back to the left again.

You can view the AD interaction with the axes reversed by right-clicking the box
next to **D: Stir** Rate and changing it to **X1 axis**.

It makes no difference statistically, but it may make more sense this way.

Note

One last thing: You can edit at least some text on many of your graphs by right-clicking your mouse. For example, on the interaction graph you can right-click the X1-axis label.

Then choose Edit Text. The program then provides an entry field. Try it!

Now from the **Graphs Toolbar** select **Cube** to see the predicted response as
a function of the three factors that created significant effects: A, C, and D.

This plot shows how three factors combine to affect the response. All values shown are predicted values, thus allowing plots to be made even with missing actual data. Because the factors of interest here are A, C, and D, the program picked them by default. (You can change axes by right-clicking any factor on the Factors Tool.)

Filtration rate is maximum at settings A+, D+, C- (upper front right corner with predicted response over 100), which also corresponds to the reduced formaldehyde concentration. Fantastic!!

An interaction represents a non-linear response of the second order. It may be helpful to look at contour and 3D views of the interaction to get a feel for the non-linearity.

First select **Contour** to get a contour graph. The axes should come up as A
(Temperature) and D (Stir Rate). If not, simply right click over the Factor Tool
and make the appropriate changes.

You may be surprised to see the variety of colors – graduated from cool blue for lower response levels to warm yellow for higher values. To match the above graph, be sure you have set the concentration (C) to the low level.

Note

Design-Expert contour plots are highly interactive. For example, you can click on a contour to highlight it. Then you can drag it to a new location. Furthermore, by right-clicking anywhere within the graph you can bring up options to add flags, add contours, or change graph preferences.

Now, to create an impressive-looking graph that makes it easy to see how things
shape up from this experiment, select **3D Surface**.

Move your mouse cursor over the graph. When it turns into a hand left-click and rotate the view however you like. Double click to reset the graph to its original coordinates.

Before moving on to the last stage, take a look at the first interaction by
going to the **Factors Toolbar** and from the **Term** list selecting **AC**.
Move the slide bar on **D: Stir Rate** to the right (maximum level) to increase
the response.

We have nearly exhausted all the useful tools offered by Design-Expert for such a basic design of experiments, but there’s one more tool to try.

The last feature that we will explore appears at the lower left of the screen
under the main branch of **Post Analysis**: Click the **Confirmation** node to make
response predictions for any set of conditions for the process factors. When you
first enter this screen, the **Factors Toolbar** palette defaults to the
center points of each factor. Levels are easily adjusted with the slider bars or
more precisely by using the **Sheet** view. In this case, the analysis suggests
that you should slide the factors as follows:

- A (Temperature) right to its high level (+)
- B (Pressure) leave at default level of center point
- C (Concentration) left to its low level (-)
- D (Stir Rate) right to its high level (+)

Design-Expert uses the model derived from experimental results to predict the mean and interval (PI) for the number (n) of confirmation runs at the conditions you dialed up, which provide the highest filtration rate with the least amount of formaldehyde. This is mission accomplished!

Note

To provide greater assurance of confirmation it makes sense to increase the
sample size (n). The returns in terms of improving the prediction interval
(PI) diminish as n increases. You can see this for yourself by trying
different values of n in the **Confirmation** tab. Watch what happens to the
PI—it approaches a limiting value. [*] Running six or so confirmation runs
may be a reasonable compromise.

As shown below, you can enter your confirmation runs. Design-Expert then calculates the mean for the actual results. If this falls within the PI, you can press ahead to the next phase of your experimental program. Otherwise you must be wary.

[*] | Statistical detail: For an infinite number of confirmation runs the PI converges to the confidence interval (CI) which is a function of the number (N) of runs in the original experiment. See the CI in the Point Prediction node. There you also find the tolerance interval (TI) calculated to contain a proportion (P) of the sampled population with a confidence (1-α). By default, the interval contains 99% of the sampled population with 95% confidence. Note that the 95% confidence interval on a mean will have the narrowest spread, with the 95% prediction interval for a single observation (given in the Confirmation node) being wider and the 95% TI which contains 99% of the population being the widest. The most rigorous interval, TI, is often required for setting manufacturing specifications, but most experimenters will settle for the PI as a way to ‘manage expectations.’ Thus the Confirmation tool will be your best friend for moving forward. |

You’ve now viewed all the important outputs for analysis of factorials. We
suggest you do a **File**, **Save** at this point to preserve your work.
Design-Expert will save the model you created so the outputs can be quickly
reproduced if necessary. Remember that if you want to write some comments on the
file for future identification, you can click the Notes folder node at the top
left of the tree structure at the left of your screen, then type in the
description. It will be there to see when you re-open the file in the future.

Now all that remains is to prepare and print the final reports. If you haven’t already done so, just click the appropriate icon(s) and/or buttons to bring the information back up on your screen, and click the print icon (or use the File, Print command).

You can also copy graphs to other applications: Use ctrl+c or press the copy icon as shown below.

For ANOVA or other reports be sure to do a **Select All** first, or highlight the
text you wish to copy.

Note

**Options for exporting results**: Try right clicking over
various displays within Design-Expert. It often provides shortcuts to
other programs that facilitate your reporting. For example, see the screenshot
below taken from the design layout. (Tip: To highlight all of the headers and
data, click the Select button.)

If you’ve made it this far and explored all the sidebars, then you know that the best way to learn about features is to be adventurous and try stuff. But don’t be too stubborn on learning things the hard way: Press screen tips for advice and go to the main menu Help and search there as well.

This completes the basic tutorial on factorial design. Move on to the tutorials
on advanced topics and features if you like, or exit from Design-Expert by
choosing **File**, **Exit** from the menu. If you have not stored your data, or
you made changes since the last save, a warning message will appear. Exit only
when you are sure that your data and results have been stored.