Disclaimer: I’m not a statistician. Nor do I want you to think that I am. I am a marketing guy (with a few years of biochemistry lab experience) learning the basics of statistics, design of experiments (DOE) in particular. This series of blog posts is meant to be a light-hearted chronicle of my travels in the land of DOE, not be a textbook for statistics. So please, take it as it is meant to be taken. Thanks!
As I learn about design of experiments, it’s natural to start with simple concepts; such as an experiment where one of the inputs is changed to see if the output changes. That seems simple enough.
For example, let’s say you know from historical data that if 100 children brush their teeth with a certain toothpaste for six months, 10 will have cavities. What happens when you change the toothpaste? Does the number with cavities go up, down, or stay the same? That is a simple comparative experiment.
“Well then,” I say, “if you change the toothpaste and 6 months later 9 children have cavities, then that’s an improvement.”
Not so fast, I’m told. I’ve already forgotten about that thing called variability that I defined in my last post. Great.
In that first example, where 10 kids got cavities. That result comes from that particular sample of 100 kids. A different sample of 100 kids may produce an outcome of 9, other times it’s 11. There is some variability in there. It’s not 10 every time.
[Note: You can and should remove as much variability as you can. Make sure the children brush their teeth twice a day. Make sure it’s for exactly 2 minutes each time. But there is still going to be some variation in the experiment. Some kids are just more prone to cavities than others.]
How do you know when your observed differences are due to the changes to the inputs, and not from the variation?
It’s called the F-Test.
I’ve seen it written as:
s = standard deviation
s2 = variance
n = sample size
y = response
ӯ (“y bar”) = average response
In essence, this is the amount of variance for individual observations in the new experiment (multiplied by the number of observations) divided by the total variation in the experiment.
Now that, by itself, does not mean much to me (see disclaimer above!). But I was told to think of it as the ratio of signal to noise. The top part of that equation is the amount of signal you are getting from the new condition; it’s the amount of change you are seeing from the established mean with the change you made (new toothpaste). The bottom part is the total variation you see in all your data. So, the way I’m interpreting this F-Test is (again, see disclaimer above): measuring the amount of change you see versus the amount of change that is naturally there.
If that ratio is equal to 1, more than likely there is no difference between the two. In our example, changing the toothpaste probably makes no difference in the number of cavities.
As the F-value goes up, then we start to see differences that can likely be credited to the new toothpaste. The higher the value of the F-test, the less likely it is that we are seeing that difference by chance and the more likely it is due to the change in the input (toothpaste).
Question: Why is this thing called an F-Test?
Answer: It is named after Sir Ronald Fisher. He was a geneticist who developed this test while working on some agricultural experiments. “Gee Greg, what kind of experiments?”. He was looking at how different kinds of manure effected the growth of potatoes. Yup. How “Peculiar Poop Promotes Potato Plants”. At least that would have been my title for the research.
We recently wrapped up the 2019 Analytics Solutions Conference in Minneapolis, MN. This was the first time we organized a US conference on analytics with our Norwegian partners, Camo Analytics. It could not have gone better! A BIG congratulations on a job well done to Shari Kraber who spearheaded the organization of this conference.
The goal of the conference was to help attendees transform their business, from R&D to production, using data-driven tools. If I may toot our own horn here, we did that. There was a lot of information to digest at this meeting, but the camaraderie of attendees made it easy. We overheard many conversations in the halls between attendees about what they just learned or discussing potential solutions to business problems they were facing.
The conference started with a short course day. Two all day seminars were conducted in parallel, one titled “Practical DOE: ‘Tricks of the Trade’” and the other “Realizing Industry 4.0 through Industrial Analytics”. We saw a lot of people experienced with design of experiments (DOE) go to the Industry 4.0 course and a lot of the analytics pros headed to learn about DOE. What a great way to start!
Day one wrapped up with an opening reception for all attendees and speakers, fostering networking and building relationships.
The conference days were filled with session talks that wrapped around keynote presentations (a fantastic trio of speakers, find out more at the link below). The result was a great mix of academic-style overviews with down-to-earth, real-world case studies. It struck a perfect balance.
If you couldn’t make it to the conference, details on each presentation (including pdf’s of the presentations), are posted on the ASC speakers page.
To add some fun to the mix, the evening’s social event was a dinner cruise on the Mississippi River for all attendees. It was quite the treat, especially to those of us who live here! We rarely take the opportunity to enjoy activities like this in our own backyard. We traveled thru a lock, spotted a bald eagle in the trees, and enjoyed the casual atmosphere.
The next Stat-Ease hosted conference will be our 8th European DOE Users Conference. This year it will be held in Groningen, Netherlands from June 17-19. We hope to see you there!
If you would like to be kept in the loop about the conference, sign up for our mailing list. We will be sending out information regarding a call for speakers and registration later in the fall. Sign up now, before you forget!
See you in Groningen!
Graduate students are frequently expected to use design of experiments (DOE) in their thesis project, often without much DOE background or support. This results in some classic mistakes.
The number one thing to remember is this: Using previous student’s theses as a basis for yours, means that you may be repeating their mistakes and propagating poor practices! Don’t be afraid to forge a new path and showcase your talent for using state-of-the-art statistical designs and best practices.
If you read my previous post, you will remember that design of experiments (DOE) is a systematic method used to find cause and effect. That systematic method includes a lot of (frightening music here!) statistics.
[I’ll be honest here. I was a biology major in college. I was forced to take a statistics course or two. I didn’t really understand why I had to take it. I also didn’t understand what was being taught. I know a lot of others who didn’t understand it as well. But it’s now starting to come into focus.]
Before getting into the concepts of DOE, we must get into the basic concepts of statistics (as they relate to DOE).
Basic Statistical Concepts:
In an experiment or process, you have inputs you control, the output you measure, and uncontrollable factors that influence the process (things like humidity). These uncontrollable factors (along with other things like sampling differences and measurement error) are what lead to variation in your results.
We all pretty much know what this is right? Add up all your scores, divide by the number of scores, and you have the average score.
Also known as a bell curve due to its shape. The peak of the curve is the average, and then it tails off to the left and right.
Variance is a measure of the variability in a system (see above). Let’s say you have a bunch of data points for an experiment. You can find the average of those points (above). For each data point subtract that average (so you see how far away each piece of data is away from the average). Then square that. Why? That way you get rid of the negative numbers; we only want positive numbers. Why? Because the next step is to add them all up, and you want a sum of all the differences without negative numbers getting in the way. Now divide that number by the number of data points you started with. You are essentially taking an average of the squares of the differences from the mean.
That is your variance. Summarized by the following equation:
In this equation:
Yi is a data point
Ȳ is the average of all the data points
n is the number of data points
Take the square root of the variance. The variance is the average of the squares of the differences from the mean. Now you are taking the square root of that number to get back to the original units. One item I just found out: even though standard deviations are in the original units, you can’t add and subtract them. You have to keep it as variance (s2), do your math, then convert back.
Hi there. I’m Greg. I’m starting a trip. This is an educational journey through the concept of design of experiments (DOE). I’m doing this to better understand the company I work for (Stat-Ease), the product we create (Design-Expert® software), and the people we sell it to (industrial experimenters). I will be learning as much as I can on this topic, then I’ll write about it. So, hopefully, you can learn along with me. If you have any comments or questions, please feel free to comment at the bottom.
So, off we go. First things first.
What exactly is design of experiments (DOE)?
When I first decided to do this, I went to Wikipedia to see what they said about DOE. No help there.
“The design of experiments (DOE, DOX, or experimental design) is the design of any task that aims to describe or explain the variation of information under conditions that are hypothesized to reflect the variation.” –Wikipedia
The what now?
That’s not what I would call a clearly conveyed message. After some more research, I have compiled this ‘definition’ of DOE:
Design of experiments (DOE), at its core, is a systematic method used to find cause-and-effect relationships. So, as you are running a process, DOE determines how changes in the inputs to that process change the output.
Obviously, that works for me since I wrote it. But does it work for you?
So, conceptually I’m off and running. But why do we need ‘designed experiments’? After all, isn’t all experimentation about combining some inputs, measuring the outputs, and looking at what happened?
The key words above are ‘systematic method’. Turns out, if we stick to statistical concepts we can get a lot more out of our experiments. That is what I’m here for. Understanding these ‘concepts’ within this ‘systematic method’ and how this is advantageous.
Well, off I go on my journey!