Agile Estimation

You a member of an agile development team planning out your next sprint.  You have estimate your velocity at 33.  You currently have a load of 32.  There is a remaining story that has been estimated as having 2 story points (using a Fibonacci sequence).  Would it be a mistake to try to fit it?

If you find yourself asking this, you are doing it wrong.

One premise of most agile techniques is that we are really bad at estimating.  Story points do not try to correct that fact, they simply work around it.  Unless you have the gift of psychic clairvoyance, there is no point in attempting high precision estimates because any such estimate will be wrong.  Hence the use sizes that increase either exponentially or through a Fibonacci sequence.  Assuming you were reasonably accurate, that story you estimated at 13 points might be as little as 12 points.  Or it might be closer to 15.

Remember high school chemistry when you learned about significant digits?  Story points are so low precision they don’t even have one significant digit.  And in a calculation involving low precision measurements, claiming a higher precision result is misleading at best.  Its downright fraudulant at worst.

So back to our above scenario, claiming your story point load is “32″ is wrong. You don’t have enough precision in your measurements to say that.  In reality, your load without that extra story is better expressed as “around 30″.  And with the extra story, it is also “around 30″.  If your current load is dominated by a couple 13 point stories, those are what will determine whether or not you meet your goal.  If it is dominated by many small 1, 2, or 3 point stories, you are misleading yourself if you argue you can predict exactly how many you are going to finish.

Is my point that you should give up on estimating?  Of course not.  Just don’t obsess yourself with getting all your numbers to line up.  Commit to enough that you feel comfortable with, and then give yourself plenty of stretch goals. That way you can meet your commitment if your estimates were too low, and you will have enough to do the entire sprint if your estimates were too high.  Because all you really know is that it is unlikely your estimates are spot on.

Programming in a keyboard-less world

Just the other day, my brand new Transformer Prime tablet arrived.  Aside from being a high quality tablet (quad core processor, one of the very first to offer Android 4.0), it is well known for having a docking station accessory, complete with a keyboard, that essentially transforms it from a tablet to a 10 inch net-book.  My phone, a HTC G2, also has a fold out keyboard, as did its predecessor, my old G1 phone.  So I think its safe to say, I am a fan of physical keyboards.  Sure, voice recognition can be good for some things, and Swype produces a nice on screen keyboard, but if I want to type anything of substance, I’m much more comfortable typing it on a nice hard QWERTY keyboard with actual buttons I can press.

Which makes the fact that I’m writing this post a bit ironic.

There was an interesting podcast last week from the IEEE about the keyboard going the way of the typewriter.  Of course I was rather dismayed by the thought.  Its not just that oncreen touch keyboards will replace them, but that new input devices, such a stylus with handwriting recognition or a microphone with voice recognition, will be the computer input of the future.

I would argue we are nowhere close to that with today’s technology, at least from where I stand.  Being that my Transformer keyboard hasn’t arrived, I originally tried to “write” this blog post with my tablet’s “voice keyboard”, and I couldn’t get through the first paragraph without getting frustrated and giving up.  I haven’t really tried using a stylus recently, but handwriting such as mine is typically so bad even I have trouble reading it, so I won’t begrudge any computer program which can’t read it (and before you try to say that’s just because I’m so used to keyboards I’ve lost the ability to write neatly, my grade school teachers would be quick to point out I never had that skill, even before I learned to type).  On the other hand, I can type at a reasonably fast pace with pretty good accuracy, so there is no debate on which method is more proficient for me.

But one argument made on the podcast was that kids today will likely grow up so used to voice recognition and handwriting recognition that they may view keyboards as obsolete.  That they may offer a technically superior method of writing fast will not matter to them.  After all, one could easily argue that command line interfaces can be much more productive that GUIs for many tasks, but outside of hard core hackers, the world has largely moved away from them.  Even software developers have largely embraced tools such as Eclipse as an alternative to hacking on the command line.

And I can’t deny that there are some areas which keyboards are not very good at.  For instance, look at writing math problems.  Math is typically full of Greek letters, superscript/subscripts, and other things which are just plain hard to type.  Sure, there are usually obscure keyboard shortcuts for them, and specialized software for it (such as Mathematica), but no real general purpose solution.  When I was trying to take notes for the Stanford Machine Learning class on Evernote last year, I can’t tell you how much time I wasted trying to come up with notations for random symbols that kept on coming up.

And of course more creative endeavors such as building “mind maps”, that is just hard to do without a more free flow input format.  That’s why many still argue that pen and paper is a superior note taking device.  Keyboards are great for writing lines of text using a small set of well known characters, but are rather limited beyond that.

So as keyboard-less input becomes more and more mainstream, how will that affect computer programming?  Today, programming is a perfect example of lines of text optimized for keyboard input.  Using voice recognition to write a Java program?  How insane would that be?  ”For, begin paren, double, no capital ’D’ double input colon input-capital-V-vals, end paren, open bracket” instead of just typing:

for (Double input: inputVals) {

Case sensitivity, the frequency of special characters and common symbols, terse variable names, camelCase, none of that will work with voice recognition input.  Computer programming is clearly not a place where you want creative, free form input, but you want to heavily restrict it to what are legal values.

Or is it?

Will computer languages evolve to utilize the advantages of newer input methods.  Will they start to incorporate more free-form writing rather than just plain text?  Will it even be possible to come up with languages like that?  Or will future freshman computer science students have to spend hours learning ancient typing techniques that have become obsolete outside of writing programs?

I suppose time will tell.

Clojure Conj Keynote and Day 3

Now that I have (somewhat) recovered from the 2011 Clojure Conj, I can finish up my reflections.

Starting with Rich’s keynote (which was on Day 2, but I didn’t have time to respond to then), as I said earlier it was more of a technical talk than the philosophical keynotes he gave at both StrangeLoop and last year’s Conj.  This may have disappointed some people (his philosophical rants have become quite popular it seems), but it was still a good talk.  He mainly discussed places for Clojure to move forward, not in so much as a timeline of new features but more of a nice to have list.  The two points that stuck with me were making a leaner version of Clojure for restricted build environments (such as Android), and adding extensional types to values.  The example he gave was date formats; different people may want to have different representations of a data object.  But as long as the date that is read in is annotated in some way its type (to show that it is a long representing the ms since the Unix epoch, for example), the reader could convert it when it is read to whatever your preferred type is (smells a bit like implicits in Scala).  This could make Clojure values a much more attractive general purpose transfer format like XML (except without the suck XML tends to drag in).  In terms of syntax, I may be missing something, but couldn’t that be handled today with metadata?  There would have to be some agreed upon conventions (and the reader would need to be able to recognize it), but I wouldn’t think a new syntactical form would be needed.  I would also worry that this could result in Clojure values becoming too big, though this would of course be optional.

On to Saturday’s talks.  It opened up with Nathan Marz’s Cascalog talk.  At StrangeLoop I attended his Cascalog workshop, so I already am a bit familiar with it, but it looks like a great library if you are working with Hadoop and HBase tables.  That was followed by an interesting talk by an overly caffeinated Daniel Spiewak on immutable persistent data structures.  He is a member of the Scala community (which Neal Ford reminded everyone is not the enemy), so that is what the talk focused on, but for the most part it was applicable to both languages.  It might say something about the Clojure community that we find talks on data structures interesting regardless of the language its in, though.  And on the subject of talks that cross language boundaries, Craig Andera’s talk on debugging performance problems involved almost no Clojure at all, though that’s a good thing since it turned out Clojure itself wasn’t responsible for the performance problems he saw in this case study (those were the database (of course) and logging).  When that talk makes it on video, I would recommend it for anyone interested in performance analysis, which hopefully is every software engineer out there.

There were a few other good ones throughout the day.  Fogus gave a talk on macros, and there were a few interesting lightning talks.  Out of those, the one that stuck with me the most was Chris Granger’s demo of Korma, a library for working with SQL (relational databases, how old school is that?).

But the conference ended with a bang with Sam Aaron showing off Overtone, described as  ”a Clojure front-end to the state-of-the-art realtime sound synthesis engine SuperCollider”.  But honestly, you need to see (or rather hear) it for yourself to understand it.  Writing software to literally make music is an intriguing (if not beyond my talent level, when I last played with Overtone I spent most of my time trying to figure out why it wasn’t making any noise, until I realized my sound was off) idea.

It was a great conference, and I learned a lot and took away a lot of ideas to play with.

Clojure Conj, day 2

Ok, so after 3 days of agile training followed by two full days of Clojure Conj, I am a bit ready for a rest.  Too much coffee and convention food can make my stomach feel a tad off.

Anyway, yesterday started off ominously enough, with a Veterans Day parade shutting down a couple of streets, including the ones around the hotel, making getting to the hotel a bit difficult for those of us local who drive in each day.  But I did make it in just in time to hear a very interesting (and perhaps a bit controversial) talk by Neil Ford on taking enterprise mindshare.  He argued that Clojure can indeed plant itself inside enterprises, but that raises the question, should it?  Should the goal be to make roads in large companies, or be a tool to compete with them (I think it was Paul Graham who said Lisp can be a startup’s secret weapon).

There were a few other interesting talks, though quite a few seemed to be on works in progress (in fairness, that can be expected from a young language).  David Nolen spoke on predicate dispatch, a feature I can’t wait to make it in the language.  And Daniel Gomez gave an intriguing talk on running Clojure on Android.  There are still a few hurdles to cross, though the fact that Scala has made it on Android with very little overhead does give me hope.  And the fact that I now have a Clojure REPL on my phone might result in me having one more thing to waste time on.

A few other notable talks included Chas Emerick’s talk on Bayesian networks and Clojure, something I am looking forward to playing with.  That’s the problem with these conferences, I come up with long lists of things I want to work on.  In fact you can to that list ClojureScript and Logs as data from yesterday’s talks as well.

The day finished up with Rich’s keynote.  It was more of a technical talk, not a rant-style talk like his Simple talk at Strange Loop or the Hammock talk last year, but still interesting.  I’ll go into it in more depth later since today’s talks are about to start.

The party was nice as well, though I wasn’t feeling well so I didn’t stay long.  I still got to have some interesting talks with people (and free beer of course).  Still no bagpipes though.  I suppose my sneezing will have to be the loud annoying sound this year.

Clojure Conj Day 1 Wins and Losses

So a quick recap of the good and bad during the first day of the 2011 Clojure Conj.

Lets start with the good:

• A fantastic talk by Arnoldo Muller-Molina on using Clojure in some very interesting bioinformatics problems.
• Two very interesting talks on logic programming, one by Ambrose on Clojure’s core.logic, another less formal talk by William Byrd and Dan Friedman (two of the authors of the Reasoned Schemer, which I really need to read) on minikanren, which included writing a program to write 50 functions that return 6 (though I think it would have been more useful had it found functions that returned 42…).
• I finally got my print copy of Clojure In Action (which I ordered as part of Manning’s early access program when the book was still in Beta… just over 18 months ago).
• A great talk by the precocious Anthony Grimes on Clojail.
• A lot of people I talked to are using Clojure not just for hobby development, but in their day jobs as well.
• The Sheraton seems very capable of handling the size of the crowd.

But of course there were a few negatives as well.

• Ragweed is in season and I really should have started taking allergy pills a few days ago.  And by the sneezes I heard behind me, I wasn’t the only one.
• Parking in the lot is a tad bit expensive for those of us not staying at the hotel (though of course I didn’t have to travel, so I can’t complain about cost too much).
• Lunch was, well, less than inspired.  I mean come on, make your own sandwiches?  I can make those at home and bring it with me…
• No bagpipes (yet…).

All in all, a very good first day.

Farewell IBM

After almost 7 years, today was my last day at IBM.  And starting Monday, I will start my new job as a Senior Software Developer at Wireless Generation, an educational software company opening a new office down here in Durham (this transition is my excuse for not blogging much recently).  I enjoyed working at IBM (well, most of the time), and wish the best of luck to my former co-workers and IBM in general, but I am also very excited to start this new chapter in my career.

Big Data at Strange Loop

Ok, time to finally review the talks I attended at last month’s Strange Loop conference in St Louis.  The last two weeks were a tad bit busy (a bit more on that later), so this post was delayed a bit.  But lets start with the sessions on big data, starting with the first keynote of the conference.  Data was a common theme at the conference and was one of the conference’s dedicated tracks.  That shouldn’t be much of a surprise to anyone following the software industry these days, as the need to analyze huge amounts of data is becoming more essential for businesses.  So Erik Meijer (architect at Microsoft) kicked things off with his talk, “Category Theory, Monads, and Duality in (Big) Data.”  I can’t find a link to the slides of the talk, but here is the paper it is based on.

Even though the title contained references to category theory and monads, you didn’t need a PhD in Mathematics to get what Erik Meijer was getting at.  And that was a very good thing, since his talk was very useful.  The essence of it was a comparison between traditional table-based SQL databases and the new breed of so-called NoSQL object databases, specifically that they are not as different as we tend to think.  In fact, he proposes replacing the ambiguous term NoSQL with CoSQL, to show how the two have a mathematical duality between them.  Basically in table based databases, you have entities (each of which can stand on their own) using foreign-primary key relationships to point to their parents.  Meanwhile in object based CoSQL databases, you have parent entities pointing to their children, who really have no context outside of their parents.

It was a really interesting talk, and not just because it had a lot of abstract math in it (I guess that may be an odd phrase for most of the world to hear).  He finished with a plea for developers to make their design decisions not on emotion or on what appears hot today, but on which design better modeled their data.  Both have advantages.  While object based CoSQL databases are more open and composable and tend to horizontally scalable, the rigidity of table based SQL databases offer plenty of advantages as well, when the problem domain calls for it.

Back from Strange Loop

I just returned from the Strange Loop conference earlier tonight.  I don’t really have time to get into too much detail of what I saw (I do have to get back on Eastern Standard Time), but let me say it was a great conference.  Some of the standouts were Rich Hickey’s keynote (if you’ve never heard him speak before, watch either Are We There Yet or Hammock Driven Development from last years Clojure Conj while waiting for the Strange Loop recording to be made available), Nathan Marz’s talk on Storm (I’m really going to have to start searching for a problem to use this for), the language panel (even though it was accidentally cut short), Erik Meijer’s keynote on Big Data, and Blake Mizerany’s talk on Doozer.  And the part of Sussman’s speech I was able to understand before my brain started to short out was great too.  Unfortunately there were a lot of talks I missed which I will have to wait for the recordings to come out.  That might have been the biggest problem with the conference (well, other than the elevator fiasco), there arguably were too many tracks with competing talks at the same time (most blocks had 7 talks to choose between).  Of course there were a lot of talks scheduled and only a limited amount of time to give them in, but perhaps it would have been possible to fit in more time slots by varying the times a bit more.  While some talks definitely required the 50 minute slot and in fact often went over, others had less information to present and could had made do with a shorter, maybe 15 minutes to half hour block “lighting” talks.  Then a couple of extra time slots could have been made by splitting a few hours up.  But all in all, it was a great conference (again, except for the elevators).  I’ll post my notes and reactions to the speeches I saw in the days to come…

Also as a note, the Clojure Conj has posted their schedule, and it looks like a great lineup as well (including several speakers I just saw this week).  Its local, so it will be easier to attend, assuming I can manage to take some more time off…

Odd Circle-like-things Day 0

So today was, lets say Day 0 of Strange Loop.  Aside from some hiccups on the plane to St Louis regarding a very important part of the plane (we were delayed for half an hour while they fixed the flushing mechanism in the lavatory), things have gone well.  The actual conference begins tomorrow, but today were the optional workshops and, of course, was the day most people arrived for the conference proper.  Let me preface this post by stating that the rooftop bar is very nice.  Where else can you drink Scotch while discussing the uptake of Clojure within business software companies with some of the smartest people in the industry?  But as a warning, that might impact the quality of this post…

Anyway, I had signed up for only one workshop, Nathan Marz’s Cascalog workshop.  Cascalog is a Clojure based library for data processing with Hadoop.  Think SQL on steroids.  I have interests in both Clojure and Hadoop, and yet don’t have too much experience with databases, so it was an obvious session for me to sign up for.  It was a three hour workshop, but at the start Nathan warned he was used to giving similar workshops for around 9 hours so this one might be a little rushed.  I had initially laughed, thinking 3 hours sounded like a long time, but it definitely was rushed.  Yet we learned quite a lot.  Cascalog definitely is a powerful library which I will certainly play around with later.  I was very impressed with the quality of the queries that could be made with what seemed very intuitive Clojure code, not the convoluted nonsense many SQL queries end up as.

Tomorrow comes the hard part though, choosing between several very different, yet very interesting talks that are scheduled at the same time.  In fairness to the conference planners, with the number of talks that look interesting it would have been mathematically impossible for them to schedule everything so I could attend everything that I wanted.  But still, as Alan Dipert of Relevance said, the thing I am least looking forward to is having to choose between talks.  I just hope enough will be recorded that I will be able to watch the ones I miss later.

ClojureScript announced

Rich Hickey and the Clojure/core team just announced (ok, they announced it last week, but I was on vacation then, so I actually have an excuse to be blogging late) a new project, ClojureScript, a Clojure to JavaScript compiler.  With Clojure being a great language to develop and JavaScript being nearly ubiquitous for web programming (though not always that great to program in), it does certainly look interesting.  I was a little disappointed as when I heard rumors of it at the TriClojure meetup earlier this month, I was hoping it would be an Android library (that wasn’t slow and didn’t add a few megs to the app), but I will have to play with it some.  I haven’t done much with JavaScript in several years (for which I am thankful for), but this should give me an excuse to get back into it.

The “Clojure”/”Google Closure” thing is going to get a bit confusing though.