If you haven’t read my first two posts on Missing Data, it might be worth taking a look before you read this. You can find the first post here, and the second, here.

I had some misgivings about imputation before I learnt about methods to quantify imputation uncertainty.

My misgivings centred around the fact that with imputation we are sort of making the data up (in a statistically rigorous fashion, of course!). But even so, how happy could we be with our analysis after imputing?

It turns out we can use a method that gives us insight into how much variability is down to the fact that we have imputed missing data.

This can help us to understand how confident we can be in our statistical analysis, given that it is based in part on missing data.

One popular method that gives us a measure of imputation uncertainty is Multiple Imputation.

How do we do Multiple Imputation?

• We repeat this, to create M imputed data sets.

• We can analyse these data sets, to come up with estimates of parameters we are interested in.

• We can then combine these estimators. There are also formulas that we can apply to calculate within imputation variance, across imputation variance, and overall variance.

• These can give us an idea of how much of the variability in our estimates is down to the imputation process.

Multiple Imputation isn’t the only method that can help us with Imputation Uncertainty. You can read more about them in some of the references below.

Further reading

You can find out more about Imputation Uncertainty in Chapter 5 of the below book. Multiple imputation is discussed in Chapters 5 and 10.

Little, R. J. A. and Rubin, D. B. (2020). Statistical analysis with missing data. Wiley Series in Probability and Statistics. Wiley, Hoboken, NJ, third edition.

This paper below contains a nice summary of Multiple Imputation and goes on to discuss the issue of variable selection. In other words, it considers what to do if your different imputed data sets imply that different variables are valuable and should be kept in a statistical model, while others should be discarded.

Wood, A. M., White, I. R., and Royston, P. (2008). How should variable selection be performed with multiply imputed data? Statistics in Medicine, 27(17):3227-3246.

And here is a long report I wrote as part of my studies at STOR-i on the broader topic of Missing Data: click to read. It discusses Imputation Uncertainty, and other issues, in more depth.

How methods to deal with missing data will depend on the Missingness Mechanism that informs the missingness pattern. I describe Missingness Mechanisms in my previous post, which you can find here. If you haven’t read that, you may want to take a look before continuing with this post.

There are four techniques I’m going to describe: Complete Case Analysis, Unconditional Mean Imputation, Regression Imputation and Stochastic Regression Imputation.

I am going to illustrate these methods with an example. Imagine we are interested in racehorse performance and we conduct a survey of racehorse heights and weights. Unfortunately, some of the weight data has gone missing.

I wrote about Missing Data for a second, longer research report at STOR-i (I mentioned the shorter first report in this post). I created the images below from a simulation study that I carried out for this report (where I followed the racehorse example described above). I will post the link to the full report at the end of this blog.

Complete Case Analysis

One of the simplest ways of dealing with missing data is Complete Case Analysis. This means that we delete any responses that have any missing data in them. This method is okay if we have data that is Missing Completely at Random or if we do not have a lot of missing data. But, if we do not have these — nice — conditions then we can introduce bias into the dataset.

For example, if tall horses do not like being weighed (perhaps the weigh bridge is claustrophobic), then if you delete all cases where weights are missing you will probably be looking at a sample with lower heights and weights than is representative of the population.

So, if you don’t use Complete Case Analysis there are still some fairly simple options open to you. These next three methods involve forms of something known as imputation.

Imputation? What is that?

The next method is the first I introduce where you impute data — this means you replace missing values with estimated values.

This may seem strange at first — isn’t this just making up values? How could this make your analysis better?

Unconditional Mean Imputation

The first method I am going to describe is the simplest: Unconditional Mean Imputation.  You take the mean of your observed data and impute any missing values with it. So, if we have missing racehorse weights, we fill in missing weight values with the average of those that we do observe.

This, you can probably imagine, can be problematic, too. If we look at the below plot, where I’ve imputed data that is simulated from the above situation — where taller horses are more reluctant to be weighed — we can see that the imputation will underestimate the true mean and the variation in the data.

Regression Imputation

Regression Imputation is a little better as in the situation we are considering it should help to eliminate some of the bias that Unconditional Mean Imputation brings.

This works by drawing a regression line, based on the complete observations that you have, and using that to predict where a piece of missing data will fall on the regression line.

The method works better in terms of bias, but it still underestimates the variation in the data — as we can see in the figure.

To deal with that we have our final method, Stochastic Regression Imputation.

Stochastic Regression Imputation

This methods works in a similar way to Regression Imputation — but it adds a random element so, instead of points being interpolated directly onto the regression line, they are scattered about it in a random fashion.

As we can see in the plot, this shows a much more realistic representation of the variation in the data.

You may be thinking that introducing values yourself to replace missing ones will cause its own problems.

Read my next post to find out how to account for the uncertainty that you are introducing through imputing missing data.

Want to know more?

You can read more about Complete Case Analysis (and some variations/adaptations of it) in Chapter 3 of the below book. The Single Imputation Methods I’ve described are discussed in Chapter 4.

Little, R. J. A. and Rubin, D. B. (2020). Statistical analysis with missing data. Wiley Series in Probability and Statistics. Wiley, Hoboken, NJ, third edition.

Another single imputation method that I do not discuss here (but I do in my report) is Hot Deck. You can read about it in this paper:

Andridge, R. R. and Little, R. J. (2010). A review of hot deck imputation for survey non-response. International Statistical Review, 78(1):40-64.

I used the R Package to do my analysis. You can read about it in this paper:

van Buuren, S. and Groothuis-Oudshoorn, K. (2011). mice: Multivariate imputation by chained equations in R. Journal of Statistical Software, 45(3):1-67.

Thank you for reading! Click here to go to the next post on Missing Data.

Click here to see my first post on Missing Data: on the different types of missingness.

You can see my full report for STOR-i on Missing Data, .

What causes the data to be missing is known as the Missingness Mechanism. There are three main types: Missing Completely at Random (MCAR), Missing at Random (MAR), and Missing Not at Random (MNAR).

You can think of these as traffic lights: MCAR is green (easy to deal with), MAR is amber (a bit problematic but there are some decent methods out there) and MNAR is red (a total pig to deal with).

Missing Completely at Random

As the name suggests, Missing Completely at Random data means that the missingness presenting in your data is in a totally random pattern. There isn’t anything in the data driving it that you need to be further concerned with. This is nice, because you can get away with some simplistic methods to deal with it.

I describe some of those methods in my next post on missing data.

Missing at Random

Missing at Random data is where what drives the missingness is something in the data we are collecting, but that what drives it is something we have observed. The preceding sentence starts to give me a headache if I think about it too much, so I prefer to think of it in terms of an example.

Imagine a university does a survey of previous students, to find out where they are working, what their income bracket is, etc.

Let’s say that alumni that work in a particular sector are less likely to disclose their income. But, they do disclose what sector it is that they work in. That data would be Missing at Random.

Missing Not at Random

But, what if students are less likely to respond to that income question the more they earn? Then we have Missing Not at Random data. The missingness depends on something we do not observe.  

This is very difficult to deal with and often causes bias in our analysis. To make it even more difficult, we cannot test whether the missingness mechanism is Missing at Random or Missing Not at Random.

Want to know more?

You can read more about missingness mechanisms in Chapter 1 of the book below — this is a really good book on missing data in general.

Little, R. J. A. and Rubin, D. B. (2020). Statistical analysis with missing data. Wiley Series in Probability and Statistics. Wiley, Hoboken, NJ, third edition.

I’ve also included a link to the paper that introduced the idea of considering missingness mechanisms.

Thank you for reading. Click here to see my next post in this series. This will discuss some simple methods to deal with missing data.

Or you can skip to my final post on missing data: this will discuss a method that allows you to quantify the uncertainty that you are introducing into your analysis by using some of the methods discussed in my second post.

I wrote a 20 page report on Missing Data as part of my studies at STOR-i. It discusses the ideas above in more depth. If you want to take a look, .

Many people appear to regard the subject with suspicion, feeling that statistics are more often used to bamboozle than to enlighten.

When, in my 20s, I started studying Applied Statistics in the evenings I got a lot of puzzled looks from friends and colleagues. What on earth did I want to do that for? What was the point?

To me this seemed odd.

As a journalist you are always trying to find out what is going on, and what it might mean. In my History degree I looked to try to find out what went on in the past and what it might mean.

These questions are very similar to those that statisticians ask. Only, statisticians have an additional tool to help them — and it’s a big one: maths.

Mathematical frameworks help us understand what conclusions we can draw from data, and how confident we can be in them. They give us tools to deal intelligently with what we do not — or cannot — know.

And this is great. Because of course, in the real world, we never get all the information we need. We are always having to piece together a picture based on what we can see, and we need steer on what we cannot see and how important that might be.

For example: I might have a disagreement with my dog, Markus, about his biscuits. He might insist that Brand A’s dog biscuits are bigger than Brand B’s, and therefore he should be bought Brand A’s.

As he’s a scientifically minded dog, he would allow me to take a random sample of each to weigh.

What if, based on the sample, Brand A’s biscuits are slightly larger than Brand B’s? Are Brand A’s biscuits bigger, or could this be a fluke?

Well, there are statistical tests to decide whether there is actually enough evidence to accept Markus’s claim.

There are also well defined frameworks to assess the probability that you will wrongly accept the claim by chance (that is, you select a random sample that happens to be of unusually large biscuits from Brand A, when in fact Brand A’s biscuits are not bigger than Brand B’s).

Of course, advanced statistical methods deal with much more difficult and nuanced situations than my dog biscuit example.

Statistics has not advanced to the point where we can guarantee that we are right about everything all the time*. But that’s not a reason to dismiss it.

Statistical methods, if used properly, bring us a vast amount of insight into problems. Why wouldn’t you want to know about that?

*On the subject of things I’ve not always been right about: I was going to put something in this post about Benjamin Disraeli’s famous “Lies, damned lies and statistics” quote, only to discover that we don’t actually know who coined that one.[1]

[1]