Statistically significant energy savings: how many buildings are enough?

From Neurobat’s website it is now possible to download a brochure with the 2012-2013 test results. It summarises the findings we published at the CISBAT 2013 conference, describing the energy savings that we have achieved on four experimental test sites. Of these four, one is an administrative office. Another included the (domestic) hot water in its energy metering. Therefore, only two of them are single-family houses whose energy savings concern the space heating alone. The energy savings we measured on these two sites are 23% and 35%.

It is natural to ask oneself what the average energy savings on a typical house might be. It’s possible to give an estimate, provided that we make several assumptions. We’re going to assume that the energy savings that Neurobat can achieve on a single-family house in Switzerland is a random variable drawn from a normal distribution. Therefore, our best estimates for the mean and standard deviation of that parent distribution are:

$$\mu = \frac{23 + 35}{2} = 29$$


$$s = \sqrt{\frac{(23-29)^2 + (35-29)^2}{1}} = 8.49$$

The sample mean estimated from n samples of a normal parent distribution is distributed according to a t-distribution with $n-1$ degrees of freedom. The 95% confidence interval for the true (parent) mean can therefore be found by looking up the 0.975 and 0.025 quantiles of the t-distribution with $n-1$ degrees of freedom. In our case, $n = 2$ and the 95% confidence interval of the true mean is therefore:

$$\left[\mu - t_{n-1, 0.975} \frac{s}{\sqrt{n}}, \mu + t_{n-1, 0.975} \frac{s}{\sqrt{n}}\right] = \left[ -47, 105 \right],$$

where $\mu = 29$, $s$ is the sample standard deviation calculated above and $n = 2$. Not the most helpful estimate ever.

We are currently repeating the experiment for this 2013-2014 heating season. A natural question that has come up is “How many buildings do we need to have a usable confidence interval for the average energy savings?”

As always, reformulating the question in precise terms is half of the battle. We want a narrow confidence interval around the mean energy savings. We can make it as narrow as we want by increasing n, or by relaxing our confidence requirement. Suppose then that we want a 95% confidence interval not wider than 10%; i.e., we want to state that Neurobat achieves $X\pm5\%$ energy savings with 95% confidence.

By a bit of arithmetic, what we are looking for is the number $n$ such that

$$n \ge 4t^2_{n-1, 0.975}\frac{s^2}{w^2},$$

where $s$ is our sample standard deviation and $w$ is the desired width of our confidence interval. There is no closed-form solution for this equation (except for large $n$, where the t-distribution can be approximated with a normal distribution), so finding $n$ is an iterative process. In R, the right-hand side of this formula can be computed with:

 4 * qt(.975,n-1)^2 * s^2 / w^2 

Evaluate this expression with increasing values of n until is becomes smaller than n.

Assuming that $s = 8.49$ as above, we obtain:

$$n = 14.$$

And that’s it. Again, assuming that the energy savings are drawn from a normal distribution whose parent standard deviation is about 8.49, we will need experimental results on 14 buildings to give a 95% confidence interval not larger than 10% on the expected energy savings. For $n = 10$, the width of the confidence interval increases to 12% and for $n = 5$, it is 21%.

The next time you hear someone claim suspiciously precise energy savings with their miracle device, you have my permission to ask them what their confidence interval is, how they calculated it, and what underlying assumptions they are making.

Posted on December 9, 2013 at 10:00 am by David Lindelöf · Permalink · Leave a comment
In: Uncategorized

Welcome back, Climate Charts & Graphs

I was happy to learn a few days ago that the Climate Charts & Graphs blog is being reactivated by its author. I used to subscribe to it back in the Google Reader days. In the current climate change conversation we need more blogs like CCG, where arguments can be conclusively settled with (preferably graphical) evidence.

So welcome back, and there’s no reason to apologise!

Posted on December 4, 2013 at 1:10 pm by David Lindelöf · Permalink · Leave a comment
In: Uncategorized

Review: The Thoughtworks Anthology: Essays on Software Technology and Innovation

The Thoughtworks Anthology: Essays on Software Technology and Innovation
The Thoughtworks Anthology: Essays on Software Technology and Innovation by ThoughtWorks Inc.<br/>
My rating: 3 of 5 stars

I read this first volume after reading its successor. Compared with the latter, I found the first volume to be slightly disappointing.

Like its successor, it’s a series of essays from Thoughtworks employees, including Martin Fowler. Whereas the second volume had some detailed, practical advice, I found this one to be much more vague and generic. It sounds almost as if it was written during the early years of the agile movement (which it maybe was), giving advice and recommendations that seem common sense today.

Martin Fowler’s article on Domain Specific Languages, although interesting, is of limited value now that his book on the subject has been published. Rebecca Parson’s article on programming languages sounds like yet-another-look-at-how-many-languages-I-know kind of article. Neal Ford’s article on Polyglot Programming recommends we build solutions with more than one language; well, people have been calling Fortran routines from C, or testing Java code with Scala, for several years now.

The only exception I want to make is Jeff Bay’s Object Calisthenics article. He proposes 9 rules to deliberately follow during your next project, and claims that following those rules will yield a superior design. This is one article I definitely want to apply, and which has practical value. Some of the rules sound extreme, such as “Don’t use any classes with more than two instance variables.”. But it’s definitely worth a look.
View all my reviews

Posted on November 14, 2013 at 7:58 am by David Lindelöf · Permalink · Leave a comment
In: Uncategorized

Review: Linkers and Loaders

Linkers and Loaders Linkers and Loaders by John R. Levine

My rating: 4 of 5 stars

You may have written hundreds, maybe thousands of programs, but if you are like most programmers then everything that happens after the compilation is kind of mysterious. Why does the compiler have to create object files? What are they? What is this so-called linker who combines those files into a library, or an executable? What’s its purpose? John Levine’s book answers those questions, and more.

Item 53 in 97 Things Every Programmer Should Know: Collective Wisdom from the Experts is “The Linker Is not a Magical Program”, and this book goes a long way towards taking that magic away. It carefully explains step by step what happens from the moment the code is compiled until it actually runs on the machine; and what’s more important, it makes it very clear why things are as they are today.

I was recommended this book in a reply to a Stackoverflow question, and I am not disappointed. The book goes occasionally perhaps a little bit too much into technical details, which I felt could be safely skipped. Perhaps a case study, i.e. going through every single step towards running a complete program, would have been useful, instead of exposing how different systems solve the different steps one by one.

Until I read this book I simply did not understand how a program actually ran on my computer. A few details are still a bit fuzzy, but now I feel much better equipped for dealing with obscure linker errors or custom linker scripts. Highly recommended for any programmer who wants to get to the bottom of things.

View all my reviews

Posted on November 13, 2013 at 4:23 am by David Lindelöf · Permalink · Leave a comment
In: Book reviews

Review: Applying UML and Patterns: An Introduction to Object-Oriented Analysis and Design and Iterative Development

Applying UML and Patterns: An Introduction to Object-Oriented Analysis and Design and Iterative Development
Applying UML and Patterns: An Introduction to Object-Oriented Analysis and Design and Iterative Development by Craig Larman<br/>
My rating: 5 of 5 stars

Easily one of the best books on object-oriented design I’ve ever read.

Through two case studies (a point-of-sale terminal application and a Monopoly game) the author goes through the entire process of eliciting use cases, domain modelling, design modelling, and implementation. The UML notation is introduced and used along the way, as are several patterns—not only the classic GoF patterns, but also some extremely useful design guidelines.

This book should be read by any senior developer who’s currently involved in the early stages of a software project. Even if you do not follow its recommendations to the letter, it will certainly improve the chances of success. For example, since reading it I make a nuisance of myself by insisting on having detailed, written use cases—not simply bullet points on some powerpoint.

This book belongs on any developer’s bookshelf, right next to the classics.
View all my reviews

Posted on November 1, 2013 at 4:25 am by David Lindelöf · Permalink · Leave a comment
In: Uncategorized

When climate change hits us, at least we’ll know where

Switzerland is a small country in the middle of Europe, wedged between France, Germany, Austria and Italy (see below). In a previous post I have documented the above-average rise of air temperatures in this country for the past 50 years, leading to a significant increase of extreme weather events.


Being a small country, our climate change mitigation efforts are more symbolic than effective. The swiss federal government recognizes the inevitability of rising temperatures and decreasing precipitations, and a national climate change adaptation strategy is to be published by the end of 2013.

Switzerland is prone to natural disasters, be they flash floods, landslides, avalanches, or storms. A nation-wide project is under way to map out the exposure of inhabited land to natural dangers, a project that is about to be completed by the individual cantons (the rough equivalent of a state in America). Danger zones can now be avoided for new buildings; for those already built in such zones, it’s now possible to decide where to build protective installations.

For example, many waterways converge in the canton of Geneva where I live. It is also here that the lake of Geneva flows out into the Rhone river. The area is prone to floods, flash floods and storms. The exposure of inhabited land to water-related dangers is now documented by the local government and can be accessed by anyone through a web portal maintained by the canton.

The figure below gives an example of the kind of information one can obtain through this portal. I have asked to highlight the inhabited land exposed to a risk of flooding. Red areas are the most exposed, followed by blue and orange. Two major waterways, the Rhone and the Arve river, converge in the city itself and their banks are clearly the most exposed areas in the whole canton. Other areas at risk are clearly delineated.


Maps such as these (for all kinds of natural disasters) are being prepared by all cantons of Switzerland and are deemed an essential tool against the increasing number of natural disasters. For example, weather-related events caused about 3 billion swiss francs (about 3.2 billion USD) in damages in August 2005. But such maps had correctly predicted which habitations would be the most severely hit; several lives have been saved by evacuating buildings at risk of landslides, for example. The figure below shows the yearly and cumulative cost of natural disasters in Switzerland; there is no increasing trend, in spite of an increase in extreme weather events and in population density. This stability is believed to partially attributable to these nation-wide programs.


The most recent project by the national government is the nation-wide OWARNA (Optimierung von Warnung und Alarmierung vor Naturgefahren) platform to develop better forecasting methods and better alerting procedures. This will further help the local authorities defend themselves against natural disasters.

Adapted from my second assignment in the Climate Literacy class on Coursera.

Posted on July 29, 2013 at 10:30 am by David Lindelöf · Permalink · Leave a comment
In: Climate change

Climate change in Switzerland: skipping the waiting line

I live next to Geneva in Switzerland. If you come to Geneva, be sure to include in your visit the ramp leading up from the Parc des Bastions and running parallel to the edge of the old town. There you will find the longest bench in Europe, the last meters of which sit in the cool shadow of a chestnut tree, called the Geneva Official Chestnut. Since 1818, an official of the Geneva government records the date when the first leaves sprout on this tree; that date has become the de facto definition of the arrival of spring in Geneva.

Geneva Official Chestnut

Until about 1900, the chestnut budded between mid-March and mid-April, with a fairly stable average at the end of March. But since 1900, the budding date has literally gone downhill.

The plot below is extracted from a report by the FOEN (Federal Office for the Environment) published in 2013. It shows the evolution of the official chestnut’s budding date, along with a 20-year moving average. The chestnut now buds on average in February; the 2003 budding happened in December 2002 (!); and in 2006 the tree budded twice, once in March and once in October. This happened again in 2011, when it budded in February and in November.

Official chestnut budding date

Temperatures across the global may have risen by 0.8 degrees on average in a century, but “average” means that some places have gotten hotter and others have gotten colder. The landmass in the nothern hemisphere is a place that has gotten warmer; and Switzerland has warmed more than the average landmass in the northern hemisphere. According to the FOEN report, the average warming in this country since 1864 has been 0.12 degrees per decade; since 1961 it has accelerated to 0.38 degrees per decade. For the past 50 years, summer temperatures have increased by 2.5 degrees and winter temperatures by 1.5 degrees. Switzerland is the perfect, if unwilling, laboratory for observing in accelerated time the environmental effects of global warming.

Perhaps the clearest indicator of Switzerland’s warming is the number of heating days (i.e., days with an average temperature of 12 degrees or less) and cooling days (i.e., days with an average temperature of 18.3 degrees or more). These are plotted below, the heating days on the left and the cooling days on the right. The decreasing trend in the former, and the increasing trend of the latter, are evident.

Heating- and cooling-days in Switzerland

We lose our glaciers at a rate of 2–3% per year; the number and intensity of heatwaves increases; so do the number of so-called tropical nights (nights during which the temper- atures exceeds 20 degrees); there is less and less snow each year; the snow line climbs by about 10 meters each year. Extreme weather events are more frequent; this month (June), the city of Montreux (on the shores of Lake Geneva) braced itself against the possibility of snow. And I’m typing this piece after three days of heatwave, and just two hours after peach-sized hailstones fell in a furious storm on Geneva, paralyzing the city (see below). For Swiss people, global warming is not a remote concept whose consequences may only be felt in a century or so. Its effects are being felt almost everyday.

Posted on June 24, 2013 at 10:30 am by David Lindelöf · Permalink · Leave a comment
In: Uncategorized

Great moments in the history of CO2 mitigation



Posted on June 17, 2013 at 10:30 am by David Lindelöf · Permalink · Leave a comment
In: Uncategorized

Foreigners make better programmers

We have recently been recruiting a couple of new programmers at Neurobat. This time we submitted the candidates to an online programming test, which consists of a couple of (relatively) simple programming assignments to be completed in a given time.

Out of about 100 applications, 34 were given this test. I have collected the data from these programming tests to see whether any personal factors correlated with the applicant’s performance. The results suggest that the candidate’s country of origin has the biggest influence on the final score.

Out of these 34 candidates, 10 were from France, 5 from Switzerland, and the others from other countries. 6 came from outside of Europe.

Here I show the boxplots for the normalized test scores, where I plotted separately the swiss and the french candidates, and lumped everybody else in a third category. The boxplots are sorted by increasing median.


There is a clear trend suggesting that the further away a candidate comes from, the higher their test scores. However, I must stress that with such low statistics the difference is not statistically significant. An analysis of variance test on the test scores against a simple 2-valued factor (swiss vs non-swiss) gives an F-value for one degree of freedom of 2.076, i.e. a p-value of 0.163. Similarly, a Wilcoxon rank sum test gives a p-value of 0.1698.

Nevertheless, the trend seems to be there, and there is some anecdotal evidence to support it. I can think of at least three hypotheses to explain it:

  1. Skilled swiss-born programmers have no difficulty finding jobs in larger, better-paid companies and have little incentive to apply to small startups such as Neurobat.
  2. Skilled swiss-born programmers tend to leave the country after graduation, whereas only the best and brightest foreign-born programmers are able to get the necessary work permits and/or scholarships to come to this country.
  3. The swiss educational system, especially in the field of computer science, sucks (for lack of a better word).

Comments? Questions? Remarks? Feel free to post your observations below.

Posted on May 6, 2013 at 11:00 am by David Lindelöf · Permalink · 6 Comments
In: Uncategorized

Phileas Fogg on Agile Project Management

I just finished reading Jules Verne’s Around The World In 80 Days, which I had never read before. It’s a great read and I highly recommend it to children and adults alike. But half-way through the book I realized there’s more to this book than a nice tale of adventure.

This book is a complete manual about managing a fixed-time, fixed-price project.

The story starts off with a bet that Phileas Fogg makes with the members of his club, namely, that it is possible to circumnavigate the globe in 80 days or less. Once the bet is placed he takes off immediately, with 20,000 pounds of cash for his travel expenses.

Travelling around the world in 80 days with a limited (albeit sizeable) budget fills, of course, anyone’s definition of a fixed-time, fixed-time project. What lessons can be learned from this adventure for today’s project managers?

Start your project immediately

Mr Fogg and his valet, Passepartout, waste no time in preparing for their trip. Once the project approved they head immediately to the train station. No feasibility study, no pre-study, no preliminary analysis, no analysis paralysis. Mr Fogg is convinced that the project is feasible and starts immediately. The lesson here is that there is value in speed. The earlier you can start on your project, the quicker you will gain feedback and learn.

Expect the unexpected

When his friends object that an 80-day schedule makes no allowance for unexpected delays or accidents, Mr Fogg calmly replies that it is “all included.” He knows very well that the 80-day schedule includes some buffer time which, it is hoped, will not be exceeded. Indeed—though I will not give away the surprise ending—the whole trip, in spite of all the delays and adventures, ends up taking less than the stated 80 days.

Whenever Mr Fogg’s party is met with surprises (such as the railway across India being not even finished), he never, ever betrays any surprise or emotion. It is as if he knew from the outset which delays his project should meet. Of course he didn’t, but he knows that his schedule accounts for them.

Don’t get emotionally involved

Mr Fogg is a very mysterious character in this book. Nothing is told about his origin, his past, or his relatives. He is described as the quintessential englishman: utterly imperturbable yet fiercely resolute in his decisions.

Even assuming that Mr Fogg included some extra buffer time in his estimate, it is hard to believe that he could possibly take into account all the mishaps his party encountered: Aouda’s rescue; Passepartout’s disappearance; Passepartout’s capture by indians and subsequent rescue; Mr Fogg arrest by Mr Fix. Yet at no time does he show the slightest sign of annoyance, hesitation or worry that he might miss his schedule. Is it because he genuinely had factored even such accidents in his estimates? Or is it because he doesn’t allow himself to become too emotionally attached to his project? I’d rather believe the latter.

Track your progress carefully

Mr Fogg keeps a kind of diary throughout his trip where he carefully records exactly how much ahead or behind he is on time. He knows exactly when he is supposed to be where on his trip. In fact, if he had a whiteboard he would probably have plotted what is known as a velocity chart, showing exactly how well he is sticking to his schedule.

He presumably does something similar with his finances, since he starts out for his journey with a fixed amount of cash and seems always to know how much he can afford to spend.

Open up communication

Mr Fogg’s party narrowly miss the steamer from Hong Kong to Yokohama, which would have spelled disaster for the journey. But Mr Fogg wastes not an instant and scours the docks of Hong Kong, looking for a boat which could take him to Japan.

However, salvation comes not from a boat that can do the Hong Kong-Yokohama trip, but from a tugboat which takes his party to Shanghai. Why Shaghai? Because Mr Fogg, instead of insisting on going to Yokohama, always explained the reasons for his going there: that he needs to catch the transpacific steamer that will take him to the USA. And the crew of the tugboat knew that this steamer does not depart originally from Yokohama, but from Shanghai. Therefore, instead to travelling to Yokohama, Mr Fogg is better served by heading to Shanghai and boarding the steamer there.

Mr Fogg scored points here by clearly explaining his needs and his situation, instead of insisting on a solution of his own making. We too, when faced with customer demands, have the responsibility to understand exactly the context in which such requests are made.

Accept mistakes and take advantage from them

I’m not going to spoil the ending for you; I’ll simply say that Mr Fogg miscalculates the total time taken on his trip, something which he has a very hard time accepting. But once the reality dawns on him, he wastes not an instant and seizes the opportunity given him, and turns disaster into victory. And Jules Verne gives the best explanation I’ve ever read for the need of an international date line, which was established 7 years after the publication of his book.

Forget now everything I just wrote. Even if you are not involved in project management, Around The World In 80 Days is a great book, a true classic, which I heartily recommend. You won’t regret reading it, but you will regret never having read it.

Posted on April 29, 2013 at 11:00 am by David Lindelöf · Permalink · Leave a comment
In: Uncategorized