# The only statistical distribution you need to know

If you pick two people at random and measure their heights, what can you say about the average height of the population?

When dealing with small sample sizes (less than, say, 20) it’s generally not possible to assume that the sample variance is a good approximation of the population variance. There simply isn’t enough data, and you cannot use the normal distribution to infer meaningful confidence intervals.

Statistics is like electrical work: it’s very easy to do a job that looks correct but could kill or maim someone. For example we have all, at least once, taken a small sample and used the normal distribution to get confidence intervals. Here is what we would get for the example above, assuming that the measured heights were 184 cm and 178 cm:

> x <- c(184, 178)
> xbar <- mean(x)
> s <- sd(x)
> qnorm(c(0.025, 0.975), xbar, s)
[1] 172.6846 189.3154

The confidence interval we just obtained ($172 \le \mu \le 189$) looks perfectly reasonable, but it’s wrong. What’s worse, there’s nothing obviously wrong with it. If $s$ was the true population standard deviation it might even be correct; but that’s extremely unlikely.

This is why Student’s t-distribution was invented (or discovered?), to deal with small sample sizes. If we draw $N$ random samples from a normally distributed population, compute the sample mean $\bar{x}$ and the sample standard deviation $s$, then the so-called t-value $\frac{\bar{x}-\mu}{s/\sqrt{N}}$ will be distributed according to the t-distribution with $N-1$ degrees of freedom.

With our example above, $N=2$, $\bar{x} = \frac{184+178}{2} = 181$, and $s=\sqrt{(3^2 + 3^2)/(N-1)}=4.24$. If we want a 95% confidence interval for the true mean $\mu$ we use the 2.5% and the 97.5% quantiles of the t-distribution with 1 degree of freedom:

> N <- 2
> x <- c(184, 178)
> xbar <- mean(x)
> s <- sd(x)
> qt(c(0.025, 0.975), N-1) * s / sqrt(N) + xbar
[1] 142.8814 219.1186

That last result gives you the correct 95% confidence interval for the true mean: $142 \le \mu \le 219$. Compare this with the interval obtained earlier: it is much wider, and therefore the uncertainty on the true mean is much higher than what the normal distribution would have let you believe.

This is, however, pretty good news. As long as your sample size is larger than 1, you can always infer correct (though perhaps useless) confidence intervals for the true mean, provided the population is normally distributed.

Recently I got angry was discussing statistics in a phone conference. The question was how many experimental test sites were necessary to estimate the CO2 reduction potential of Neurobat’s technology. I pointed out that no matter how many test sites we had, the t-distribution would always let us infer confidence intervals about the true CO2 reduction potential. But the statistics experts on the other side of the line disagreed, arguing that anything less than 10 was not statistically significant.

I tried the analogy above and showed that $N=2$ was enough to derive a confidence interval for the average height of the population.

“No, we don’t think so”, was the answer.

That’s when I decided to shut up and write a blog post.

We’ll run a little computer experiment. Suppose again that the height of the swiss population is drawn from a normal distribution of mean 180 cm and standard deviation 20 cm (I am totally making this up). We’re going to draw one sample at a time from this distribution and see the evolution of the sample mean and the confidence interval.

The true mean $\mu$ lies with 95% confidence within $\left[t_{2.5\%, N-1}\times(\frac{s}{\sqrt{N}}+\bar{x}), t_{97.5\%, N-1}\times(\frac{s}{\sqrt{N}}+\bar{x})\right]$. Let’s form this confidence interval as we progressively sample elements from the parent population:

N <- 1000
mean.height <- 180
sd.height <- 20
height.sample <- rnorm(N, mean.height, sd.height)

mean.sample <- numeric(N)
sd.sample <- numeric(N)

for (i in 1:N) {
mean.sample[i] <- mean(height.sample[1:i])
sd.sample[i] <- sd(height.sample[1:i])
}

low.interval <- sd.sample / sqrt(1:N) * qt(0.025, 1:N - 1) + mean.sample
upper.interval <- sd.sample / sqrt(1:N) * qt(0.975, 1:N - 1) + mean.sample

And here is the result:

It’s obvious that for small sample sizes, the confidence interval quickly becomes smaller than the parent standard deviation. We can see this by plotting the log of the confidence interval over time:

The confidence interval becomes smaller than the population standard deviation (20) after only 15 samples. However, even for extremely small sample sizes (less than, say, 5) the confidence interval is roughly 40 cm. The true mean can therefore be estimated within $\pm 20$, or with about 11% error, after only 5 samples. And that is thanks to the t-distribution.

In: Uncategorized

# Review: RESTful Web Services

I began reading “Restful web services” while researching technical solutions for Neurobat Online, the web service version of our intelligent heating controller. Prior to this, most (all?) web service projects I had been involved in were based on SOAP.

REST is a heavily overloaded term in our industry, and can mean different things to different people. The author avoids that controversy by coining the term “Resource-Oriented Architecture”, and shows different examples of web services that can be built using this approach: a social bookmarking service inspired by Delicious, and a mapping service. Both examples are RESTful but the author does an excellent job at showing why being RESTful is not enough. To fully leverage the existing web architecture (including the full HTTP protocol) you need, he argues, to do more than merely being RESTful, and he shows how.

The author never says so explicitly, but after reading this book I found that RESTful web services have at least two significant advantages over what the author coyly calls “Big Web Service” (aka SOAP):

1. Testability: do not underestimate the advantage of exposing a service that your testing team can test with cURL instead of having to setup a tool such as SOAPUI.

2. Discoverability: everything is a resource, and all resources will respond to a limited number of HTTP verbs. You don’t have to worry whether adding a bookmark is done through addBookmark() or appendBookmark(); if your service is RESTful, you know that you need to send a POST to some URI.

The biggest takeaway from this book for me was to realise that it’s possible to design an application where “everything is a resource”, and that all resources respond to the same set of methods. Think of it for a moment. Will this not change the way you design non-web services too? Are not all your objects resources? Imagine, for a minute, if all your classes were restricted to expose not more than 5 public methods, and if these methods had the same names. It may sound crazy, but it’s quite possible that you’d end up with a cleaner design built out of many small classes with small interfaces. Is this not an easy way to clean code?

What makes this book great and not merely good is that the author doesn’t simply explain what RESTful web services are about. He is also clearly opinionated about it; however he is never patronising, never condescending. He takes very occasional jabs at systems he calls “Big Web Services” but never belittles them. His message comes across as entirely believable and convincing. By example after example he shows how popular services can be designed as collections of resources, and it is up to the intelligence of the reader to judge whether which kind of design is better.

Posted on December 22, 2013 at 6:11 am by David Lindelöf · Permalink · One Comment
In: Book reviews

# Pair-programming girls did just as well as boys

For the past three years I’ve taught a freshman-level programming course at the Swiss Federal Institute of Technology in Lausanne. Students are asked to form groups of 2 and to work on a semester project, consisting in the development of a simple library of numeric routines (e.g. square root function, integrals, etc). I then submit their code to a suite of unit tests (including the Valgrind memory checker) and assign them a grade linearly proportional to the number of unit tests that pass. The same grade is assigned to both members of the pair.

Most students will pair with a fellow student of the same sex. In the spring 2014 session, 43 pairs out of 52 were of the same sex. This year’s class was large enough to consider carrying out statistically significant studies on the students’ grades. More specifically, I wanted to examine whether pairs of girls obtained significantly different results from pairs of boys.

Here I show the boxplots of the grades assigned to the 52 pairs, depending on whether it was two females, mixed sex, or two males. The median grade for females is 5.5 out of 6, while the median grade for males is 5 out of 6.

The Welch two sample t-test (used to determine whether two samples are drawn from populations with the same mean) yields a p-value of 0.32. The 95% confidence interval for the difference in means between all-females and all-males is between -0.27 and 0.80. In other words, there is no statistically significant difference between the grades obtained by two-female pairs of students and two-male ones.

And what about the pairs of mixed sex? The boxplot suggests that their results are lower, and I can think of a hypothesis to explain that. But with a sample size of only 9 it is hard to draw any conclusion.

Posted on December 16, 2013 at 10:03 am by David Lindelöf · Permalink · One Comment
In: Uncategorized

# Statistically significant energy savings: how many buildings are enough?

From Neurobat’s website it is now possible to download a brochure with the 2012-2013 test results. It summarises the findings we published at the CISBAT 2013 conference, describing the energy savings that we have achieved on four experimental test sites. Of these four, one is an administrative office. Another included the (domestic) hot water in its energy metering. Therefore, only two of them are single-family houses whose energy savings concern the space heating alone. The energy savings we measured on these two sites are 23% and 35%.

It is natural to ask oneself what the average energy savings on a typical house might be. It’s possible to give an estimate, provided that we make several assumptions. We’re going to assume that the energy savings that Neurobat can achieve on a single-family house in Switzerland is a random variable drawn from a normal distribution. Therefore, our best estimates for the mean and standard deviation of that parent distribution are:

$$\mu = \frac{23 + 35}{2} = 29$$

and

$$s = \sqrt{\frac{(23-29)^2 + (35-29)^2}{1}} = 8.49$$

The sample mean estimated from n samples of a normal parent distribution is distributed according to a t-distribution with $n-1$ degrees of freedom. The 95% confidence interval for the true (parent) mean can therefore be found by looking up the 0.975 and 0.025 quantiles of the t-distribution with $n-1$ degrees of freedom. In our case, $n = 2$ and the 95% confidence interval of the true mean is therefore:

$$\left[\mu – t_{n-1, 0.975} \frac{s}{\sqrt{n}}, \mu + t_{n-1, 0.975} \frac{s}{\sqrt{n}}\right] = \left[ -47, 105 \right],$$

where $\mu = 29$, $s$ is the sample standard deviation calculated above and $n = 2$. Not the most helpful estimate ever.

We are currently repeating the experiment for this 2013-2014 heating season. A natural question that has come up is “How many buildings do we need to have a usable confidence interval for the average energy savings?”

As always, reformulating the question in precise terms is half of the battle. We want a narrow confidence interval around the mean energy savings. We can make it as narrow as we want by increasing n, or by relaxing our confidence requirement. Suppose then that we want a 95% confidence interval not wider than 10%; i.e., we want to state that Neurobat achieves $X\pm5\%$ energy savings with 95% confidence.

By a bit of arithmetic, what we are looking for is the number $n$ such that

$$n \ge 4t^2_{n-1, 0.975}\frac{s^2}{w^2},$$

where $s$ is our sample standard deviation and $w$ is the desired width of our confidence interval. There is no closed-form solution for this equation (except for large $n$, where the t-distribution can be approximated with a normal distribution), so finding $n$ is an iterative process. In R, the right-hand side of this formula can be computed with:

4 * qt(.975,n-1)^2 * s^2 / w^2

Evaluate this expression with increasing values of n until is becomes smaller than n.

Assuming that $s = 8.49$ as above, we obtain:

$$n = 14.$$

And that’s it. Again, assuming that the energy savings are drawn from a normal distribution whose parent standard deviation is about 8.49, we will need experimental results on 14 buildings to give a 95% confidence interval not larger than 10% on the expected energy savings. For $n = 10$, the width of the confidence interval increases to 12% and for $n = 5$, it is 21%.

The next time you hear someone claim suspiciously precise energy savings with their miracle device, you have my permission to ask them what their confidence interval is, how they calculated it, and what underlying assumptions they are making.

In: Uncategorized

# Welcome back, Climate Charts & Graphs

I was happy to learn a few days ago that the Climate Charts & Graphs blog is being reactivated by its author. I used to subscribe to it back in the Google Reader days. In the current climate change conversation we need more blogs like CCG, where arguments can be conclusively settled with (preferably graphical) evidence.

So welcome back, and there’s no reason to apologise!

In: Uncategorized

# Review: The Thoughtworks Anthology: Essays on Software Technology and Innovation

I read this first volume after reading its successor. Compared with the latter, I found the first volume to be slightly disappointing.

Like its successor, it’s a series of essays from Thoughtworks employees, including Martin Fowler. Whereas the second volume had some detailed, practical advice, I found this one to be much more vague and generic. It sounds almost as if it was written during the early years of the agile movement (which it maybe was), giving advice and recommendations that seem common sense today.

Martin Fowler’s article on Domain Specific Languages, although interesting, is of limited value now that his book on the subject has been published. Rebecca Parson’s article on programming languages sounds like yet-another-look-at-how-many-languages-I-know kind of article. Neal Ford’s article on Polyglot Programming recommends we build solutions with more than one language; well, people have been calling Fortran routines from C, or testing Java code with Scala, for several years now.

The only exception I want to make is Jeff Bay’s Object Calisthenics article. He proposes 9 rules to deliberately follow during your next project, and claims that following those rules will yield a superior design. This is one article I definitely want to apply, and which has practical value. Some of the rules sound extreme, such as “Don’t use any classes with more than two instance variables.”. But it’s definitely worth a look.
<br/><br/>
View all my reviews

In: Uncategorized

My rating: 4 of 5 stars

You may have written hundreds, maybe thousands of programs, but if you are like most programmers then everything that happens after the compilation is kind of mysterious. Why does the compiler have to create object files? What are they? What is this so-called linker who combines those files into a library, or an executable? What’s its purpose? John Levine’s book answers those questions, and more.

Item 53 in 97 Things Every Programmer Should Know: Collective Wisdom from the Experts is “The Linker Is not a Magical Program”, and this book goes a long way towards taking that magic away. It carefully explains step by step what happens from the moment the code is compiled until it actually runs on the machine; and what’s more important, it makes it very clear why things are as they are today.

I was recommended this book in a reply to a Stackoverflow question, and I am not disappointed. The book goes occasionally perhaps a little bit too much into technical details, which I felt could be safely skipped. Perhaps a case study, i.e. going through every single step towards running a complete program, would have been useful, instead of exposing how different systems solve the different steps one by one.

Until I read this book I simply did not understand how a program actually ran on my computer. A few details are still a bit fuzzy, but now I feel much better equipped for dealing with obscure linker errors or custom linker scripts. Highly recommended for any programmer who wants to get to the bottom of things.

View all my reviews

In: Book reviews

# Review: Applying UML and Patterns: An Introduction to Object-Oriented Analysis and Design and Iterative Development

Easily one of the best books on object-oriented design I’ve ever read.

Through two case studies (a point-of-sale terminal application and a Monopoly game) the author goes through the entire process of eliciting use cases, domain modelling, design modelling, and implementation. The UML notation is introduced and used along the way, as are several patterns—not only the classic GoF patterns, but also some extremely useful design guidelines.

This book should be read by any senior developer who’s currently involved in the early stages of a software project. Even if you do not follow its recommendations to the letter, it will certainly improve the chances of success. For example, since reading it I make a nuisance of myself by insisting on having detailed, written use cases—not simply bullet points on some powerpoint.

This book belongs on any developer’s bookshelf, right next to the classics.
<br/><br/>
View all my reviews

In: Uncategorized

# When climate change hits us, at least we’ll know where

Switzerland is a small country in the middle of Europe, wedged between France, Germany, Austria and Italy (see below). In a previous post I have documented the above-average rise of air temperatures in this country for the past 50 years, leading to a significant increase of extreme weather events.

Being a small country, our climate change mitigation efforts are more symbolic than effective. The swiss federal government recognizes the inevitability of rising temperatures and decreasing precipitations, and a national climate change adaptation strategy is to be published by the end of 2013.

Switzerland is prone to natural disasters, be they flash floods, landslides, avalanches, or storms. A nation-wide project is under way to map out the exposure of inhabited land to natural dangers, a project that is about to be completed by the individual cantons (the rough equivalent of a state in America). Danger zones can now be avoided for new buildings; for those already built in such zones, it’s now possible to decide where to build protective installations.

For example, many waterways converge in the canton of Geneva where I live. It is also here that the lake of Geneva flows out into the Rhone river. The area is prone to floods, flash floods and storms. The exposure of inhabited land to water-related dangers is now documented by the local government and can be accessed by anyone through a web portal maintained by the canton.

The figure below gives an example of the kind of information one can obtain through this portal. I have asked to highlight the inhabited land exposed to a risk of flooding. Red areas are the most exposed, followed by blue and orange. Two major waterways, the Rhone and the Arve river, converge in the city itself and their banks are clearly the most exposed areas in the whole canton. Other areas at risk are clearly delineated.

Maps such as these (for all kinds of natural disasters) are being prepared by all cantons of Switzerland and are deemed an essential tool against the increasing number of natural disasters. For example, weather-related events caused about 3 billion swiss francs (about 3.2 billion USD) in damages in August 2005. But such maps had correctly predicted which habitations would be the most severely hit; several lives have been saved by evacuating buildings at risk of landslides, for example. The figure below shows the yearly and cumulative cost of natural disasters in Switzerland; there is no increasing trend, in spite of an increase in extreme weather events and in population density. This stability is believed to partially attributable to these nation-wide programs.

The most recent project by the national government is the nation-wide OWARNA (Optimierung von Warnung und Alarmierung vor Naturgefahren) platform to develop better forecasting methods and better alerting procedures. This will further help the local authorities defend themselves against natural disasters.

Adapted from my second assignment in the Climate Literacy class on Coursera.

In: Climate change

# Climate change in Switzerland: skipping the waiting line

I live next to Geneva in Switzerland. If you come to Geneva, be sure to include in your visit the ramp leading up from the Parc des Bastions and running parallel to the edge of the old town. There you will find the longest bench in Europe, the last meters of which sit in the cool shadow of a chestnut tree, called the Geneva Official Chestnut. Since 1818, an official of the Geneva government records the date when the first leaves sprout on this tree; that date has become the de facto definition of the arrival of spring in Geneva.

Until about 1900, the chestnut budded between mid-March and mid-April, with a fairly stable average at the end of March. But since 1900, the budding date has literally gone downhill.

The plot below is extracted from a report by the FOEN (Federal Office for the Environment) published in 2013. It shows the evolution of the official chestnut’s budding date, along with a 20-year moving average. The chestnut now buds on average in February; the 2003 budding happened in December 2002 (!); and in 2006 the tree budded twice, once in March and once in October. This happened again in 2011, when it budded in February and in November.

Temperatures across the global may have risen by 0.8 degrees on average in a century, but “average” means that some places have gotten hotter and others have gotten colder. The landmass in the nothern hemisphere is a place that has gotten warmer; and Switzerland has warmed more than the average landmass in the northern hemisphere. According to the FOEN report, the average warming in this country since 1864 has been 0.12 degrees per decade; since 1961 it has accelerated to 0.38 degrees per decade. For the past 50 years, summer temperatures have increased by 2.5 degrees and winter temperatures by 1.5 degrees. Switzerland is the perfect, if unwilling, laboratory for observing in accelerated time the environmental effects of global warming.

Perhaps the clearest indicator of Switzerland’s warming is the number of heating days (i.e., days with an average temperature of 12 degrees or less) and cooling days (i.e., days with an average temperature of 18.3 degrees or more). These are plotted below, the heating days on the left and the cooling days on the right. The decreasing trend in the former, and the increasing trend of the latter, are evident.

We lose our glaciers at a rate of 2–3% per year; the number and intensity of heatwaves increases; so do the number of so-called tropical nights (nights during which the temper- atures exceeds 20 degrees); there is less and less snow each year; the snow line climbs by about 10 meters each year. Extreme weather events are more frequent; this month (June), the city of Montreux (on the shores of Lake Geneva) braced itself against the possibility of snow. And I’m typing this piece after three days of heatwave, and just two hours after peach-sized hailstones fell in a furious storm on Geneva, paralyzing the city (see below). For Swiss people, global warming is not a remote concept whose consequences may only be felt in a century or so. Its effects are being felt almost everyday.