Fear and Loathing in Software Development
May 6th, 2007A couple weeks ago I came across a video on dzone called How To Design A Good API and Why it Matters, by a very smart man named Joshua Bloch. I am always reading and watching anything that I can find programming related, especially when the material comes from the most distinguished among our industry. This video was overall very good, and I would recommend it to everyone interested in software design, but a note came out of the talk that I wanted to comment on.
Basically, Joshua asserts that a public API is permanent and as such you have one chance to get it right. He gives an example that if you write some code to solve a problem, pretty soon your buddy is going to want to solve that same problem and he will use your solution, and then he is going to pass your code onto ten friends, and so forth until your API is entrenched. Additionally, he asserts that once a person has learned an API they won’t want to relearn it as it changes. I don’t necessarily disagree with the either of those points, and as he says “This is scary”.
It reminds me of a project that I was involved in at my previous position at the Anchorage School District. I was charged with improving the ability of our systems to sync student enrollments and departures with our Active Directory. You see, we had a 20-or-so year old student information system that the school registrars used to enter student data, which should automatically be imported into our AD such that the students would have computer accounts to log in to the schools’ labs. Seems simple enough, but for various reasons the project ultimately failed. One of the major reasons was that I didn’t prototype properly, but that is beside the point. The main reason that this project failed was because of fear.
You see, we did not have a test environment for programming against our Active Directory; it was deemed too expensive to setup a parallel forest that could be over-written and rebuilt from our production environment. Try as I might at the time, the simple fact was that I would be developing a large application against a live data source that 55,000 students and teachers would be depending on day in, day out. If that wasn’t enough, the administrator in charge of maintaining the AD at the time was someone who I had very little confidence in: If I busted this thing, I gave it about a 35-50% chance that the administrator could bring everything back to working order. He was a nice enough guy, just not very good at his job. All of this added up to overwhelming amount of fear. Paralyzing would be an understatement. Every line of code had to be double and triple checked by me. I spent extra care on unit testing what pieces I could, but the fact was that the system was completely devoid of true functional testing. I was unable to test actually pulling data from the student system and writing data to the active directory.
This article isn’t about that project though, suffice to show that fear is the mind-killer. Directly following the failure of that project I wrote a similar system of reduced scale to populate Open Directory servers in much the same way. The difference was that I could hack and destroy to my hearts content, because I had a stupid-simple way to fix what I broke, and no one was depending on the systems I was developing against. The result: it rolled out to 100 elementary schools just fine. There were some hiccups, but not nearly on the scale of the previous project.
Maybe it is just me. Maybe it’s because I equate my programming style to that of a sketch or comic book artist. The point is that if you work through prototype and iteration, gradually morphing your initial failed tries into the final working API, having pressure to get it right the first time is killer.
So what can we do to alleviate that pressure when it comes to API design? Lets look at some of the points Joshua used to back up the notion that you have one try to get it right.
First of all, lets look at his assertion that programmers won’t want to relearn an API. I believe this is a side-effect of the cultural environment that a programmer works within. Perhaps this is just the Java community, with which he is so intimate; I certainly don’t see this hesitance to adapt in, for example, the Ruby community. Take for example REXML and Hpricot. Before Hpricot was released by my personal hero, _why, most everyone who was working with parsing XML in Ruby was using REXML. REXML works fine, has a reasonably simple API to interact with, but isn’t that fast. And then comes along Hpricot, which is super-fast comparatively, can work on malformed XML (is XML that is malformed still XML?), but has a different API. Some would say that the API for Hpricot is easier and more powerful (I certainly do), but the point is that I had to relearn the method in which I interact with XML. Did I want to? I sure as fuck did, and on a further note I’ve used Hpricot in just about every project I’ve worked on since it was initially released. This can further be evidenced by various people around the net, and several personal acquaintances.
If your new API is compelling, rather than change just for change’s sake, people will relearn it. Free of bureaucratic constaints, I think about which API would be best to accomplish the task at hand, rather than how I can fit an API to a problem. I don’t think “I need to use Hpricot to solve this problem”, but rather “I need to parse XML here, Hpricot is the best in this situation, I’ll use it”. Given that attitude, and exposure to new APIs, a programmer will adapt and evolve along with the code he is relying upon.
Lets discuss the entrenchment of an API that Joshua initially talked about. He is absolutely right: the more popular an API becomes the less likely it will be that you can change it. This is why we have versioning, it really is a non-issue given that. I don’t care that come Rails 2.0 my thousands of invocations of image_path that look like:
imagepath( ’somefilewithoutan_extension’ )
will no longer work, that code is bound to the version that allows invocations like that. Sure, a changing API means I pay tolls to keep my code that consumes the API up the date, but that is part of doing business. A well created test suite, and working to isolate coupling between an API and your code will reduce this toll. As a teaser into an entirely different discussion, work of this nature is so incredibly easy in a dynamic language such as Ruby. I can almost effortless produce an adapter than behaves as the old API while consuming the new API, and then gradually morph that adapter to expose the new features and changes as my application needs them.
Anyways, this rant has been going on long enough. I would like the make clear the theme I am trying to convey. Fear is the death of agility. If you have to second guess yourself, or feel restricted in how you can approach building a solution, the solution will suffer. If you are afraid at the consequences your software will have, you will fail unless you contain an inhuman capacity for mental anguish. Sketch, re-conceive, evolve all of your code. Make it better, even if others are relying upon it. Communicate with them how things are changing and work with them to reconcile those changes with their applications. Revolution is hard, evolution is easy, but use both to grow your code and make it more beautiful.
