Software engineering best practices in academia
As you might know, my primary background stems from the field of academia and research, but over the past years my interests have focused increasingly on software engineering.
With the benefit of hindsight, it’s clear to me today that if I had known what I know today about software, I would without doubt have been a much, much more productive researcher and graduate student. It’s simply not possible today to carry out research without programming. And research itself, to be considered valuable, requires exactly the same qualitities demanded from modern software engineering: repeatability, versioning, and safe explorations.
I’m convinced today that researchers would benefit if practicing software engineers would give them some feedback on how they solve these problems. And I’ve often pondered whether I should begin writing on software engineering topics that I think could be relevant for scientists and/or engineers, particularly in the academic field. It could even form the basis for a series of blog posts.
I’d rather ask you, dear reader, for advice on this. Would you like me to begin a series of posts on software engineering topics relevant to scientists and engineers in academia? And if yes, which particular subjects would you like to see me discuss?
I’m really, really looking forward to reading your comments on this matter.
Weird certificate verification error
I spent most of the day today debugging a very mysterious error we encountered when trying to programmatically call a web service over SSL from Java.
Here is the source code with which we managed to reliable reproduce the error:
import javax.net.SocketFactory;
import javax.net.ssl.SSLSocketFactory;
import java.io.*;
import java.net.Socket;
public class SimpleSSLTest {
public static void main(String[] args) throws IOException {
try {
int port = 443;
String hostname = "somehost.com";
SocketFactory socketFactory = SSLSocketFactory.getDefault();
Socket socket = socketFactory.createSocket(hostname, port);
InputStream in = socket.getInputStream();
OutputStream out = socket.getOutputStream();
PrintWriter pout = new PrintWriter(new BufferedWriter(new OutputStreamWriter(out)));
pout.println("GET " + "/" + " HTTP/1.0");
pout.println();
pout.flush();
BufferedReader bin = new BufferedReader(new InputStreamReader(in));
String inputLine;
while ((inputLine = bin.readLine()) != null) {
System.out.println(inputLine);
}
in.close();
out.close();
} catch (IOException e) { throw e; }
}
The website, somehost.com, used a SSL certificate signed by our own
internal certificate authority. That authority’s certificate was
stored in a cacerts Java keystore. We run this code from the command
line thus:
$ java -Djavax.net.ssl.trustStore=cacerts -cp target/classes/ SimpleSSLTest
When we run this, the application bombs with an exception, the root cause of which reads as follows:
Caused by: java.security.cert.CertPathValidatorException: CA key usage check failed: keyCertSign bit is not set
at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:153)
at sun.security.provider.certpath.PKIXCertPathValidator.doValidate(PKIXCertPathValidator.java:325)
at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:187)
at java.security.cert.CertPathValidator.validate(CertPathValidator.java:267)
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:261)
... 22 more
We’ve tried to wrap our heads around this problem the whole day and could make neither head nor tail about it, especially as we didn’t get this error at all when targeting another host, using another certificate but signed by the same certificate authority.
As a last resort, I thought of checking exactly which version of Java we were using. Turned out we were using OpenJDK, the version that replaced Sun’s version in Ubuntu 10.4. Running the same code with Sun’s Java SDK solved the problem, but we can’t confidently state that we understand what was wrong. Perhaps a bug in OpenJDK’s implementation of JSSE. Who knows.
If you’ve run into the same problem, feel free to leave a comment. I’d be interested to hear if (and how) you’ve solved it.
In: Programming · Tagged with: ssl
MATLAB’s inane idea of time
MATLAB seems to have a very peculiar notion on how to represent dates and times. Yesterday I spent a wonderful couple of hours debugging some code that’s supposed to compute the sun’s position, most of which could have been avoided if the MATLAB designers had followed a simple convention used by, I believe, most computing platforms.
In MATLAB, dates and times are represented internally by a so-called serial date number, defined as the number of time units counted since a given reference date. If you are like me you will, I suppose, assume that this reference date is the standard UNIX epoch, i.e. midnight, January 1st, 1970. Well you’re only about two millenia off—the reference date in MATLAB is the hypothetical (and non-existent) date of midnight, January 1st, 0000. Never mind that there never was a year 0000—the calendar goes straight from 1 BC to 1 AD.
And if you really are like me you will of course assume that the unit of time in which this serial date number is counted is seconds, or at least milliseconds. Wrong again—MATLAB choosed days as its fundamental unit of time. And of course, Octave was forced to follow MATLAB’s choice:
octave:4> format long
octave:5> now
ans = 734313.962094548
Besides making it much more difficult to make MATLAB interoperate with, say, Java libraries, there are several problems with this approach (documented in Octave’s help file, haven’t checked in MATLAB):
- The Julian calendar is ignored, so anything before 1582 will be wrong;
- Leap seconds are ignored. In other words, MATLAB ignores days that happened to be 86401 seconds long (yes, there are).
When working with timeseries data, in particular climate data, I
always try to count time from the UNIX epoch—ideally as the number
of seconds from the epoch, the way date(1) works when called with
the +%s format argument:
18:08:49@netbook$ date +%s
1277309333
18:08:53@netbook$ date +%s
1277309336
In Java, System.currentTimeMillis() will return the number of
milliseconds since the epoch:
scala> System.currentTimeMillis
res0: Long = 1277395240485
In R, converting a DateTime object to numeric yields the number of
seconds:
> as.numeric(Sys.time())
[1] 1277310008
In short, every computing platform I’ve touched in the recent weeks represents time starting from the standard UNIX/POSIX epoch, and always do so in a unit related to seconds. In other words, there is no justification for MATLAB’s decision to represent time since year 0000, and even less for doing so in number of days. I don’t mean to bash MATLAB (well… a bit, maybe). I just regret that anytime I need MATLAB to interoperate with some other code, I need to include a factor of 86400 and shift everything by 1970 years.
In: Programming · Tagged with: matlab
Trying out new themes
I’m currently experimenting with new themes for this blog. I was not quite satisfied with the width that Typebased would leave me for giving code samples. So do expect some visual instabilities on this blog for the days to come. :-)
Installing ESP-r on Ubuntu 9.10
ESP-r, is an integrated modelling tool for the simulation of the thermal, visual and acoustic performance of buildings and the assessment of the energy use and gaseous emissions associated with the environmental control systems and constructional materials, in the words of its official website. In other words, it’s a computer program for modeling a building’s thermal and energy performance. It’s especially popular in Europe, particularly among academia.
Recently I wanted to install it on my laptop running Ubuntu 9.10 (Karmic Koala). The standalone installers provided on the main downloads website didn’t quite work, complaining about the lack of the libg2c library. Well of course it’s not available. It’s been obsoleted and is now deprecated.
Your best choice when installing ESP-r on Ubuntu is, quite frankly, to rebuild it from source. And it’s not complicated either. Here’s how I did it.
Check the project out from SVN:
$ svn co https://espr.svn.cvsdude.com/esp-r/trunk espr
$ cd espr/src
Ensure you have the required development libraries installed. In particular you will need libxml2-dev if you build with XML support and libX11-dev if you build the X version (which I recommend). Note that you really need the -dev packages, these contain header files required when compiling an application against those libraries.
Choose now an installation directory. Being the only user of my machine, I installed under ~/programs/esru, but you might want to install under /opt/esru. Now you should be able to build:
$ ./Install -d ~/programs/esru
ESP-r installation script.
Please consult the README file before commencing
installation. This script will rebuild the ESP-r
modules on your system. You can abort this process
at any time by pressing c.
Please answer the following questions. Default answers
are in []. To accept the default, press return.
Your computer identifies itself as Linux.
Is this information correct (y/n)? [y]
ESP-r can be built with the Sun Fortran 90, GNU or
intel compilers.
Compiler:
(1) Sun fortran 90 (cc and f90)
(2) GNU fortran (gcc 3.X and g77)
(3) Intel fortran (icc, icpc and ifort)
2
Install with experimental XML output support? This may
significantly increase simulation run-time. (y/n) [n]
y
XML output enabled for bps
Graphics library: [2]
(1) GTK graphics library
(2) X11 graphics library
(3) no graphics library (text-only application)
2
ESP-r can optionally retain debugging symbols and
object files for use with a debugging program such
as GDB.
Retain debugging symbols? (y/n) [n]
Install ESP-r database files? (y/n) [y]
Install training files? (y/n) [y]
Proceed with installation of esp-r modules (y/n) [y]?
Installing ESP-r system. This may take some time.
...
Once the build is complete (took me an hour on a Asus 1005HA netbook), you should be able to run ESP-r:
$ path/to/install/espr/bin/prj
Enjoy!
Alternatives to Java for home automation devices
I think there’s no escaping this simple fact: the Java Virtual Machine (JVM) is definitely here to stay, and all the evidence shows that it has tremendous potential for being used in home automation devices.
I still remember from a previous life when we hunted for a JVM implementation that would run with a minimal footprint in, if I remember correctly, a 16 MB flash drive with about the same amount of RAM. It was compliant with Java 1.3 at that time (around 2003), and I remember we were having issues with its total lack of logging libraries.
Things have changed a lot since then, but the JVM is still a superb platform for future home automation devices. It’s just such an ubiquitous and comfortable environment to work with; the code you compile and unit-test on your laptop will be the same one being later executed on whatever water heating controller you’re gonna use. And any platform that’s more or less compliant with the official Java specs will come preloaded with plenty of libraries for your everyday needs.
Still, I’m not writing this to encourage you to learn or even perfect yourselves at Java. Java is actually a rather weak and inexpressive language. Instead, remember that JVM-the-platform is not quite the same thing as Java-the-programming-language. You compile Java code into bytecode that then gets executed by a JVM, but who said you needed to compile Java code? There are, indeed, about 200 different languages that can be compiled to Java bytecode, many of which are arguably far better, more powerful, and more expressive languages than Java.
One such language is Scala, which I’ve been studying for a while now, and I’m impressed by the ease with which this language allows you to create Domain Specific Languages (DSLs), i.e. highly expressive pseudo-languages that apply to a very specialized field or domain. I would not be surprised to hear, five years from now, that about 10% of all bytecode being run in the world were in fact compiled from Scala.
If you’re into writing embedded code for home automation devices, I strongly encourage you to learn at least one JVM language in addition to Java. I think you would be surprised how easier and faster it can be to write controller code in a programming language that lets you think at the right level of abstraction—which Java never let me do.
Why I’m disabling MathML for now
In a previous post I described how I tweaked my WordPress installation to support the display of MathML markup, for displaying mathematical equations.
One of the steps involved changing the content-type from application/html to application/xhtml+xml. That step was necessary, or else Firefox would simply not render the MathML markup properly.
Unfortunately, application/xhtml+xml is simply not supported on a host of other browsers, including Internet Explorer. Which means that this blog became unreadable overnight to anyone coming to it with anything else than Firefox.
This is why I’m disabling direct MathML support on this blog. If you’re interested you can view the original blog post on my blog’s old server.
There are, however, alternative (and arguably simpler) ways to display mathematics on the web, such as MathJax, or jsMath (a Javascipt library used on the Maths Q&A site MathOverflow
Equivalent conductance due to air infiltrations
Here is how you compute the thermal coupling between a room’s indoor temperature and the outdoor temperature due to infiltrations.
The equation governing the exchange of heat is as follows:
Cair × dT/dt = g × Δ T
where Cair is the indoor air thermal mass [J/K], T is the indoor air temperature [K], g is the equivalent conductance [W/K] and Δ T is the temperature difference between indoors and outdoors.
What we need to compute is how much heat is exchanged per unit of time when air seeps into the indoor room. The air that infiltrates will draw (or more rarely, yield) heat from the indoor air until it reaches thermal equilibrium with T. So we obtain:
V’ × ρ × Cp air × Δ T = J
where V’ is the infiltration rate [m3 / s], ρ is the air density (about 1.2 kg/m3), Cp air is the air thermal capacity (about 1008 J/kgK) and J is the rate of heat exchange in W.
Now we immediately recognize that actually, J / Δ T = g. But we can also simplify further by noting that air infiltration rates are more commonly given in room volumes renewed by some unit of time (commonly the hour). Note first that:
Cair = Vair × ρ × Cp air
and therefore:
V’ × Cair / Vair = g
or more simply, if Nren is the room volume renewal rate per hour:
Nren / 3600 × Cair = g
And therefore the main equation becomes simply:
dT/dt = Nren / 3600 × Δ T
I find this to be an amazingly simple equation, especially as it applies to other situations similar to air infiltrations. For example, you can use it to compute the exchange of heat among the elements of a heater through which hot water circulates.
In: Energy · Tagged with: air infiltration, thermal conductance
Event rate of arrival analysis with R
Here is a very common problem: suppose you’re give a series of event timestamps. The events can be anything—website logins, persons entering a building, anything that recurs regularly in time but whose rate of arrival is not known in advance. Here is, for example, such a file which I had to analyze:
05.02.2010 09:00:18 05.02.2010 09:00:18 05.02.2010 09:00:21 05.02.2010 09:00:23 05.02.2010 09:00:24 05.02.2010 09:00:29 05.02.2010 09:00:29 05.02.2010 09:00:30 05.02.2010 09:00:35
and so on for several thousand lines. Your task is to anlyze this data and to derive “interesting” statistics from it. How do you do that?
My initial reaction was, hey let’s try to derive the rate of arrival per second over time. Histograms are one way of doing this, except that histograms are known for the dramatically different results they can yield for different choices of bin width and position. So instead of histograms, I tried doing this with so-called density plots.
That, it turns out, was a terrible idea. I think I must have spent a day and a half figuring out how to use R’s density function and its arguments, especially the bandwidth parameter. There are two problems with density: 1) it yields a density, which means that you have to scale it if you want to obtain a rate of events; 2) it’s an interpolation of a sum of kernels, which has the unfortunate side-effect of yielding a curve whose integral is not necessarily unity.
In the morning of the second day, I realized I had been solving the wrong problem. I’m not really interested in knowing the rate of arrival. What I really need to know is how many items is my system handling simultaneously. Think about that for a second. If your data represent the visitors to your website, you really don’t want to know how many visitors come per second if you don’t know how much time the server needs to serve each one. In other words, if you want to make sure the server never melts down, you need to know how many users are served concurrently by the server, and to know that you also need to know how much time is needed for each request.
Or again, if you’re designing a building or a space to which people are supposed to come, be served somehow, and then leave, you really don’t need to know how many people will come per hour; you need to know how many people will be in the building at the same time.
There is something called Little’s Law which states this a bit more formally. Assuming the system can serve the requests (or people, or jobs) without any pileup, then N=t× Λ, where N is the number of requests being served concurrently, t is the time spent on each request, and Λ is the request rate. Now it should be obvious that if you know t, the data will give you N from which you can derive Λ (if you want).
Here’s how I did it in R. Suppose the data comes in a zipped data.zip file, with timestamps formatted as above. Then:
library(lattice) # always, always, always use this library
# Get raw data as a vector of DateTime objects
data <- as.POSIXct(scan(unzip("data.zip"),
what=character(0),
sep="\n"),
format="%d.%m.%Y %T")
# Turn it into a dataframe which will be easier to use
data.lt <- as.POSIXlt(data)
data.df <- data.frame(time=data,
sec=jitter(data.lt$sec, amount=.5),
min=data.lt$min,
hour=data.lt$hour)
data.df$timeofday <- with(data.df, sec+60*min+3600*hour)
rm(data, data.lt)
Note that for this example we’ll assume all events happened on the same day.
Now here’s the idea. We’re going to build a counter that counts +1 for each event and -1 when that event has been served. In R, we can do that with the cumsum function. For example, suppose we have a series of ten events spaced apart according to a Poisson distribution with mean 4:
> x <- cumsum(rpois(10,4))
> x
[1] 6 9 14 20 26 33 37 38 38 44
Suppose each event takes 6 seconds to serve, and build a structure holding x, the coordinates of the original events and of their completion times, and y, the running counter of the number of events being served:
> temp <- list(x=c(x, x+kLatency),y=c(rep(1,length(x)),
rep(-1, length(x))))
> reorder <- order(temp$x)
> temp$x <- temp$x[reorder]
> temp$y <- cumsum(temp$y[reorder])
If you now plot the temp structure here is what you would get:
With all this in place, we have now everything we need to go ahead. The little script above can go into its own function or it can be defined as xyplot‘s panel argument:
kLatency = 4 # seconds
xyplot(1~timeofday,
data.df,
main = paste("Concurrent requests assuming", kLatency, "seconds latency"),
xlab = "Time of day",
ylab = "# concurrent requests",
panel = function(x, darg, ...) {
temp <- list(x = c(x, x + kLatency),
y = c(rep(1,length(x)),
rep(-1, length(x))))
reorder <- order(temp$x)
temp$x <- temp$x[reorder]
temp$y <- cumsum(temp$y[reorder])
panel.lines(temp, type="s")
},
scales = list(x = list(at = seq(0, 86400, 7200),
labels=c(0,"","",
6,"","",
12,"","",
18,"","",24)),
y = list(limits = c(-1, 20)))
)
Unfortunately I cannot show you the results here, but this analysis showed me immediately when and how often the webserver would be under its most heavy load, and directly informed our infrastructure needs.
Book review: Agile Project Management with Scrum
I began reading Ken Schwaber’s ‘Agile Project Management with Scrum’ for two reasons: 1) it’s a book about Scrum, and 2) it’s from Ken Schwaber, one of the fathers of Scrum. Having now read it, I think these are the only reasons I don’t entirely regret reading it.
The book is a series of case studies, bases on real-world experiences Schwaber has had managing projects. Each case study shows how a particular aspect of Scrum was applied, adapted, or tweaked on a real project. Schwaber devotes each chapter to a distinct aspect of Scrum, e.g. the ScrumMaster’s role, the project backlog, the sprint planning, etc.
As such, the book is clearly intended for experienced ScrumMasters, which I am not. But I think an experienced ScrumMasters reading this book will find it lacking in depth. There is almost too much material in this book, covered too shallowly. Each chapter in this book might easily provide enough material for a separate book, or a workshop, or a detailed whitepaper. Even with my limited knowledge and experience of Scrum I felt frustrated by the lack of detail Schwaber gave in the book. For instance, he never shows us a real-world example of a product backlog. Instead of showing us a snapshot of a real sprint backlog taped to a team’s room, the book shows us a nicely formatted table with very obviously watered-down entries.
In short, I think an experienced ScrumMaster will find this book lacking in detail, whereas the Scrum student will find it hard to relate to the case studies. As such I find it hard to recommend this book, and I think anyone interested in Scrum should rather consult Schwaber’s earlier book, ‘Agile Software Development with Scrum’, or Henrik Kniberg’s excellent ‘Scrum and XP from the Trenches’, also available for free from here.