Estimation

Quick! How long does it take to execute about 200 lines of java code (no I/O) on your machine? Why does it take 70 milliseconds?

This is the kind of problem I faced yesterday (although, I didn’t actually measure how many line of java code there was). So I add several breakpoints logging current time to console. In IntelliJ you can click on gutter with Shift pressed (doesn’t work in KDE4 at the time of writing) and get window like this one.

After running code with breakpoints output shows that there is from 15 to 30 milliseconds delay between each of breakpoints, which add up to some 70ms. Is it all? I thought so, but it’s not. After suggestion from a colleague to try “profiling” without writing to console straightaway, I added code like this one (not real code):

    public static final long TIMES[] = new long[20];

    public static void takeTimestamp(int i) {
        TIMES[i] = System.currentTimeMillis();
    }

    public static void main(String[] args) {
        takeTimestamp(0);
        // ... some code
        takeTimestamp(1);

I ran it without debugger attached and see that output is a bit strange because all timestamps are the same. I do make stupid mistakes sometimes, but with code like the above it seems very unlikely. I change timestamps to nanoseconds and see quite surprising result. There are no mistakes, it's just that code I’m “profiling” executes in less than 1 millisecond.

Lessons learned:

• don’t “profile” with debugger attached.
• “for interval timing, always use System.nanoTime in preference to System.currentTimeMillis” (see EJ 2nd edition). I could’ve saved some time if I followed this advice. I knew I should do it, but didn’t remember why and, seeing that several other developers use currentTimeMillis() in preference of nanoTime(), decided to use it too. Really bad idea. The reason is that it’s platform specific and on XP machine it may have “a granularity of 15 ms” (see this post).
• most importantly you should be able to “Estimate to Avoid Surprises” (see The Pragmatic Programmer Tip 18).

Update: I wasn't benchmarking but rather measuring how long it takes for specific code path to be executed in live environment under certain conditions. Should you want to do benchmarking you probably should look at something like framework from this article.