Thursday, December 11, 2008

Conferences in Emerging Tech Countries

I speak at lots of conferences in the US and Europe, and they pretty much have the same feel (modulo different languages, both programmer and spoken). But I also speak at a few conferences in emerging technology markets, like my recent conference appearance for JAX-Asia, a Java-centric conference that happens over the course of three days in Singapore, Kuala Lumpur, and Jakarta. Attendees at these conferences are very much like attendees at conferences elsewhere: slightly geeked out, interested in learning about new stuff, and inquisitive. They also exhibit a lot of enthusiasm: there aren't as many conferences there as here, so going to a conference is a bigger deal.

I have two observations from my recent trip. I first observed (with the help of my colleague, Erik Doernenburg) that all the conferences in Asia have lots of sponsorships. Generally, attendees pay very little (if anything) to attend the conference; most of the cost is offset by vendor sponsorships deals. But along with the booth in the exhibit area, vendors get one or more keynotes. Now, I don't want to completely denigrate vendor keynotes. I've seen a couple that had actual content, not just blatant marketing. But, for the most part, the mode in Asia seems to be outrageous over-the-top marketing, making them not so much keynotes as "marketing-notes". Because most of the attendees aren't yet jaded and cynical, they stick around for the marketing-note, succumbing to the danger that their marketing bullshit detector is not as well honed as ones in the US and Europe. They might actually believe what this vendor is saying about SOA -- what a frightening concept! To their credit, the conference also books in regular sessions that offset some of the damage caused by toxic marketing-notes. One of my talks at JAX-Asia was my Evolutionary SOA talk, which is pretty much the polar opposite of what most vendors say about SOA. One of the reasons I like speaking at out of the way conferences like JAX-Asia concerns around this very issue. I'm very interested in providing my (and ThoughtWorks) perspective on technology and methodology to offset the message of people just trying to sell something. I genuinely want to advance the state of software development, which has no geographic boundaries.

The second observation from JAX-Asia is the span of sophistication of the attendees. Just like any conference, you get a fair number of wide-eyed innocents, who just discovered Ant last week and want more cool stuff like that. To those attendees, preaching about dynamic languages and other cutting edge stuff is pointless: their primary interest in the conference is shoring up their current development practice, not spelunking into the future of technology. But it's a mistake to assume that all the attendees are beginners. Like any conference, a wide variety of skill sets and experience show up. For example, for JAX-Asia, I did a talk on JRuby, and another comparing Groovy & JRuby. JRuby was one thing: at least a few of the attendees had heard of Ruby. But the comparison talk was brutal on both me and my audience. No one had even heard of Groovy, so discussing the nuances of the differences between a technology represented by a word they had heard ("Ruby"), with another technology for which no one was familiar ("Groovy") was tough. Yet, generalizations are dangerous. After my talks in Jakarta, I was chatting with one of the attendees, who turns out to be the president of the Ruby users group in Jakarta, which has 50 members! To me, this shows two things. One, never assume that the crowd to which you are speaking in homogeneous. Even if the majority are novices, chances are good that there are some experienced developers there as well. And, two, Ruby has managed to penetrate pretty far and wide. A 50 person Ruby users group in Jakarta encourages me greatly because it shows that the foothold that dynamic languages have in the US and Europe is spreading to other parts of the world.

Monday, December 01, 2008

Irrational Artifact Attachment

The lowly whiteboard is one of my favorite tools for design work on projects: you can stand in front of it as a group, you can easily play "what-if" games with emergent designs, and you can argue until everyone agrees (or at least until everyone is equally unhappy). Once you've got it done, a quick snap with a digital camera and you've got a project artifact, ready to post on a wiki or similar until supplantation by actual code. Once you have real code, you are better off allowing the design to continue to emerge from it rather than trying to keep the two in sync. Alternatively, you can use a reverse engineering tool to produce a prettier version of the original diagram from the code.

I prefer this low-tech approach to the more formalized version using tools (from drawing tools all the way up to really formal tools like Rational Rose) because of the proportional relationship between a person's irrational attachment to some artifact to how long it took to produce. If you create a beautiful UML diagram using some tool like Visio that takes 2 hours, you have an irrational attachment to that artifact that's roughly proportional to the amount of time invested. That means that you'll be more attached to a 4 hour diagram than a 2 hour one. By "irrational attachment", I mean that it's harder to listen to reason as to why it's wrong because you know how much time it took to create it (and therefore the required effort to update it).

This applies to pretty much any software artifact. I've seen this effect for design documents and diagrams, architecture, requirements, use cases, and database schemas. This suggests that really elaborate tools favor one-shot designs, where the pressure is on to get it right on the first pass. Of course, you can change it, but how much willpower exists to do that? One of the benefits to the low-ritual approach to much of agile software development revolves around creating "just in time" artifacts, with as little ceremony as possible (this helps explain the dedication to lots of agilists to index cards and sticky notes). Using low-tech tools frees you to throw away what's not right, freeing you to experiment and allow the true nature of the artifact emerge through revision, collaboration, and discussion.

Monday, November 17, 2008

Generative Ambient Music: Bloom

One of the techniques I describe in The Productive Programmer is focus, distancing yourself from the all too common distractions in modern office environments. I suggest that you can wear headphones (or earbuds) while coding as an indicator that others shouldn't bother you. Some developers can code to music (in fact, some developers have a hard time not coding to music), but others fine it distracting. If you are in either group, you should try ambient music. The goal of ambient music is to create non-offensive background music. I find Muzak-style background music incredibly annoying and distracting, partially because they always choose sappy music (or taking good music and sappifying it) and partially because of a personality quirk described here. Ambient music strives to create a sonic background that not only doesn't distract but qualifies as good music. You can search for ambient music as a category on Amazon. Ambient music frequently uses techniques borrowed from Minimalist music, where you create a melody (which is too stong a term for some minimalist music) that repeats with minor variations.

Brian Eno is one of the pioneers of ambient music. He and Robert Fripp recorded the track The Heavenly Music Corporation on the album No Pussyfooting in 1972 by combining some experimental tape loop techniques from Eno, combined with Fripps guitar loops (called Frippertronics). No Pussyfooting is still available, and quite enjoyable. But the interesting part about it for this post concerns how it was created. Once Eno and Fripp had all the equipment set up, they started the tape loops and recorders for the background track and retired to Eno's front room and drank tea while the album recorded itself. No Pussyfooting has some early examples of generated ambient music (they were certainly not the first to experiment with generated music, but the first in a main-stream context). Eno and Fripp created several albums together. Eno has created a large catalog of ambient music (including the classic Music for Airports, recorded by several ensembles including the incredible Bang on a Can All-stars), and Fripp expanded the ideas behind Frippertronics using electronics for a stunning collections of recordings called Soundscapes.

Which brings me finally to Bloom. Bloom is an iPhone application created by Brian Eno and Peter Chilvers. When you launch Bloom, it starts a gentle drone sound as a background, and gives you the option to create ambient music by tapping the screen. When you tap, a note plays based on the location of your tap. You can tap single notes or chords. After about 5 seconds, the note repeats and continues while it gently decays. Using Bloom, you can create your own ambient music. I can play with Bloom for hours. It is one of the best iPhone applications I've found: a trully innovative application that takes great advantage of the iPhone interface.

The other option when you start Bloom is for it to generate music for you randomly. That brings us back around to focus. Using Bloom, you can generate unique, non-distracting music for an entire day. Rather than buy a collection of ambient recordings, you can generate your own ambient music. Bloom lets you set several "moods", changing the tonal range and drone sounds to add just enough variety in the music to keep it just interesting enough to not distract. What used to take tons of studio equipment now runs on the iPhone. This makes a great way to help enhance your focus in noisy environments. Allow Bloom to create music for you that doesn't distract yet effectively drowns out all the other distracting sounds in your cube-ridden environment. Bloom is a great example that the iPhone isn't just another mobile device: it's a new platform for software development.

Friday, November 07, 2008

Comments == Code Smell

I am sometimes asked about my position on code comments, and, like most things, I have strong opinions about it. Two kinds of comments exist:

  • JavaDoc-style comments (which encompasses JavaDoc, XMLDoc, RDoc, etc), which are designed to produce developer documentation at a high level (class and method names and what they do)

  • In-line comments, generally scattered around the code to indicate a note from developer to developer

Both kinds of comments represent different smells, each with different odors depending on the target.

What makes comments so smelly in general? In some ways, they represent a violation of the DRY (Don't Repeat Yourself) principle, espoused by the Pragmatic Programmers. You've written the code, now you have to write about the code. In a perfect world, you'd never hove to write comments for this purpose: the code will be expressive enough that someone who reads it will understand it. Two things help achieve this result: expressive, readable languages and the composed method pattern.

The language makes a big difference as to the readability of the code. If you write in assembly language, you're pretty much forced to write comments; no one on earth can read the code directly. As languages have matured, you can get much closer to the ideal of self-documenting code (which was explicitly attempted in Literate Programming). Especially in the modern wave of non-ceremonious languages (like Ruby, Groovy, Scala, etc), you can craft extremely readable code. To this end, ThoughtWorks projects generally avoid the more magical features of languages (like the implicit global variables in Ruby, for example) because it hurts readability. One of the side effects of dynamic typing is outstanding method names. The method name is the only vector of information about what the method does (you can't crutch on return or parameter types), so methods tend to be named much better on the dynamic language projects upon which I've worked.

The second key to readable code is not to have too much of it, especially at the method level. In Smalltalk Best Practice Patterns, Kent Beck defines composed method (which I write about extensively in The Productive Programmer). Composed method encapsulates the idea that all your methods should do one and only one thing, making them as small as possible. The side effect of this discipline leads to public methods that mainly consist of calls to a large number of very small private methods, each of which do only one thing. Composed method has lots of beneficial side effects on your code: small methods are easier to test, you end up with really low cyclomatic complexity for your methods, you discover and harvest reusable code chunks more easily, and your code is readable. This last one brings us back to the topic of comments. If you use composed method, you'll find much less need to have comments to delineate sections of code within methods (actual method calls do that), and you'll find that you use better method names.

Now let's talk about how this applies to the two types of comments. First, where are comments indeed useful (and less smelly)? If you are writing an API, you need some level of generated documentation so that people can use your API. JavaDoc style comments do this job well because they are generated from code and have a fighting chance of staying in sync with the actual code. However, tests make much better documentation than comments. Comments always lie (maybe not now, but on a long enough timeline, all comments will become outdated). Tests can't lie or they fail. When I'm looking at work in progress on projects, I always go to the tests first. The comments may or may not be there, but the tests define what's now done and working.

We were on a project where the client insisted on Javadoc comments for every public class and method. We started the project adding those comments, but eventually stopped. When doing agile development, you don't want anything that hampers refactoring. Having comments in place caused a dilemma: do I refactor the method and change the comment (the most work), refactor the method and leave the old comment (with the theory being that I'll change it again later, and would rather just have to update the comment once), or not refactor? No good options here. Having pervasive comments discourages refactoring because it adds significant extra friction. On this project, we abandoned commenting as we went along. The last week of the project we literally did nothing but go back and add comments to the code, which worked well because the code base had settled down by that point. But notice what's lurking in wait: the client wanted all the comments there to make it easier to maintain in the future. But who's to say that whoever maintains that code will keep the comments up to date? You have the same DRY violation as before. If they maintain the tests, they have to change as code changes because they are executable.

The new wave of Behavior Driven Development tools (like JBehave, RSpec, and easyb make this "executable specification" style of comment + test feasible. I expect to see the usage of these tools skyrocket because they give you documentation that has a fighting chance of staying up to date.

Inline comments are almost always a smell. The only legitimate use of inline comments is when you have some very complex algorithm that you need to have some thoughts about beside the code. Otherwise, the presence of inline comments indicates that you've written code that needs explanation, meaning that it cries out for refactoring. I frequently troll code bases upon which I'm working to look for inline comments so that I can refactor the code to eliminate the need for them.

Comments are a great example of something that seems like a Good Thing, but turn out to cause more harm than good. Fortunately, we've figured out how to achieve the same benefits that comments allegedly provide with tests, particularly BDD-style tests

Monday, November 03, 2008

Voices That Matter: Crafting Software Conference

Voices that Matter conference logo Just a quick note to let my readers know that I'm speaking at the Voices That Matter: Crafting Software Conference coming up in San Francisco in early December. I'll be speaking on Real-world Refactoring, and I'll be doing a pre-conference tutorial called 10 Ways to Improve Your Code. This is based on material from The Productive Programmer and my 90 minute conference talk of the same name, but I'm expanding it to 3 hours. That shouldn't be hard because there's a lot of rich material there, and 90 minutes always made it hard for cramming in that much information Now, each topic can spread its wings more.

Of course, I don't mention every conference for which I'm speaking in my blog (otherwise, it'll be nothing but conference announcements). I bring this one up because you can get a discount: use CSNPKRA as the priority code when you register and, lo and behold, a discount! Don't ever say you never got anything of value from this blog!

Wednesday, October 22, 2008

Code, Forrest, Code!

The movie Forrest Gump has been re-playing a lot lately, and, as often happens, I see parallels in the software industry (I've learned not to bring these up to my wife so that I don't get the look). Of course, Forrest was mentally challenged, yet he had a wildly improbable life. The real point of the movie is the exploration of destiny vs. free will, as exemplified by the feather that floats freely, is captured and carried to new places in the Curious George book, and then floats free again. But that's not the point of this blog entry.

How did the mentally challenged Forrest achieve so much? Total concentration. If you notice in the scene where he first learns to play ping pong, the guy showing him says "The most important part of this game is to never take your eye off the ball". As soon as he says that, Forrest squints his eyes and starts concentrating hard on the ball, and starts hitting it. Of course, in the movie, he eventually becomes a celebrated table tennis player, going to China, meeting the president, etc. The events in the movie are highly improbable of course, but the writer had to come up with some talent that Forrest had that would enable all his amazing feats. And that was flow. I talk about flow in The Productive Programmer (in the Focus chapter) as defined by Michael Csikszentmihalyi. Flow is a state intimately familiar to all developers. It's that state of total concentration, where time disappears and you get your best work done. In fact, Csikszentmihalyi argues convincingly in his book that flow is the secret to a happy life: finding things that are challenging enough to engage your full concentration but not enough to be persistently frustrating. The ability to get to this state explains why so many developers love their career. Software development offers lots of opportunities to get to this state, offering puzzles that are always just within reach.

But flow doesn't come for free. It's a psychological state, meaning you can't just force yourself into it. My friend Brian Goetz has the best comparison I've heard: flow is like sleep. You can't force yourself to go to sleep, but you can set up the environment to make the state easier to achieve. That's one of the reasons people have so many rituals around bedtime. Flow is the same way. You can't force it, but you can set up the correct environment. Brian explained this to me in the context of interruptions. When he works at home and his family interrupts him, he's grumpy because they've broken his flow. But they don't understand: it's only a 5 minute disruption. But his killer analogy? What if someone awoke you 4 or 5 times a night for just 5 minutes at a time? You'd have a terrible night's sleep.

This matters a lot because perhaps the most hostile environment for flow is cube land. Putting developers (or any knowledge workers, for that matter) in a sea of cubes forces you to constantly deal with an environment that's hostile to getting to the state where you are most productive. Lots of developers have figured out how to survive (just like big city dwellers learn to sleep through noisy environments). One of my favorite lines from The Blues Brothers movie is when Jake comes "home" with Elwood, directly adjacent to an elevated train. Jake asks "How often do those trains run" to which Elwood answers "So often you won't even notice it!". How often do meetings happen in your office? So often you won't even notice that you only have about 2 hours a day to actually get work done. This is by the way one of the secrets to the effectiveness of pair programming. It is possible for 2 developers to get in flow at the same time. Because more than one person is working, external forces are less likely to disturb them, allowing them to stay in flow for long periods of time. This explains why, if you are new to pair programming, you are hammered after 8 hours of pairing: you've been in that deep concentration state for longer than what you are accustomed.

I talk about strategies for getting to and staying in flow in The Productive Programmer. That's the secret to real productivity when coding. Be like Forrest Gump: learn how to achieve that deep, deep concentration state in your environment (or, if you can't change your environment, change your environment). Code, Forrest, Code!

Monday, October 06, 2008

Library Oriented Programming

Way back in 1968, Edsger Dijkstra almost caused a riot at the ACM conference. His audacious crime? "Use of GOTO considered harmful." No one there could imagine writing code without GOTO's. Yet, of course, within a few years, GOTO-oriented programming faded away, replaced by structured, modular, and object-oriented programming. Structured programming has become such the mainstream that we don't even refer to that feature of the language anymore: we never say that C is a "Non-GOTO" language.

Fast forward to today. In the same way that C isn't a "Non-GOTO" language, I contend that Java is no longer an object-oriented language. Sure, the object abstraction is still there, but it has sunk so far down into the infrastructure we don't even think about it anymore. This blog really drove it home for me: Java is a library oriented language. Think about it: when was the last time you downloaded a single class from the Internet to use in an application? Libraries and frameworks are the code reuse mechanisms in Java. Ultimately, the class is just too fine grained to really achieve reuse, but libraries can.

This explains tools like Maven and Ivy. While Java is a library oriented language, no good facilities exist within the language to manage libraries. Some work has been done on the "module" specification, but word from insiders say that is has essentially collapsed into two warring factions: the OSGI vs. Maven camps. Consequently, it looks doubtful that a module facility will appear anytime soon for Java. That's OK: we'll keep doing what we've always done while waiting for ratification of infrastructure -- use open source tools that solve this problem.

Ultimately, we need to embrace the idea of Java as a Library-Oriented Language. That means treat libraries (and dependencies between libraries) as first class citizens in your projects. Developers tend to try to avoid nasty coupling and circular references with classes, yet too often tolerate it with their JAR files. Looking at Java in this light forces you to think about how you create project directory structure (Maven helps with this, as well as tools like AppFuse, which creates consistent directory structures for different kinds of web applications). Convention over configuration was one of the "breath of fresh air" approaches from Ruby on Rails. It makes building tools and plugins for Rails easy because you can always rely on a standard directory structure for projects. Of course, it's too late now to create the universal standard Java project structure, but you can do this within your own company. Once you have a consistent structure, it makes writing little utilities that can do things to all your projects much easier.

Consistency is the key to automation.

A consistent directory structure for all projects makes it easy to plug into continuous integration, write test helper classes, manage dependencies across projects, and a slew of other benefits. Start treating Java like a library oriented language, think hard about dependencies, versions, and where things live. And, finally, create consistency across projects so that you can take advantage of economy of scale.

Wednesday, September 24, 2008

Remote Usability

broken remote control Ever since I read Don Norman's The Design of Everyday Things, I've been hyper-sensitive to things that have really poor usability. His classic example is doors: how often to you walk up to a door that has handles on both sides, tricking you into pulling instead of pushing? Doors like this require an instruction manual: "Push" or "Pull". But the problem is that, if it's a glass door, you'll read the backwards word through the glass and still do the wrong thing. It isn't you, it's the door. If something as simple as a door requires a manual, it's really poorly implemented.

Here's a recent case of something simple whose usability is so profoundly broken it needed a manual. I spoke at TechEd this year. They are one of the conferences that absolutely require that you use their machine for your presentation. On the stage, they had a remote control to use for your slides. It looks like the image to the left. Normally, the button the right moves to the next slide and the left button moves you one slide back. Yet this stupid remote reverses that. Of course, you can turn it upside down, but it clearly had a right-side up. To fix it? Someone affixed labels to it to fix the utterly broken usability.

As developers, we need constant diligence for usability. While it probably made perfect sense to some remote control creator because of the electronics to put the buttons on the wrong sides, but it ruins the usability. If a 2 button remote can't get this right, what hope do we have when building a rich client?

Tuesday, September 16, 2008

SOA (Sidekick Oriented Architecture)

Warning: smiling broadly or laughing out loud while reading this will expose your age to those around you.

When Microsoft released DOS 3.3, they added a much requested feature: background printing. That was the state of the art of cool back then: at the time, DOS was a single-process operating system, and users complained that they couldn't do anything else while waiting for something to print. Thus, Microsoft added the print spooler. But that opened a can of worms, because other software companies (like Borland) reversed engineered what it did and built applications around it. The way it worked was by chaining to a low-level interrupt in the underlying plumbing of the operating system. Borland exploited this to create a massively useful little utility called SideKick. SideKick was awesome. While working in an application, you could hit a hot key and pop up SideKick, make a calendar entry, browse the file system, and lots of other stuff. Then, with a hot key, you were back in your application. This was old-school multi-tasking: incredibly primitive, yes, but oh so useful. This category of tool was categorized as a TSR's (Terminate and Stay Resident) application.

Of course, you can imagine what happened next: everyone and their brother built TSR's. And if you weren't careful with your interrupt chaining, the whole house of cards came crashing down. This became an annoying issue in the dying days of DOS. Windows rescued us because it had an event driven architecture, and you could run multiple applications at the same time. Of course, Windows was really just running on the shaky DOS ground, but it allowed developers to take advantage of the event driven system automatically and stop fighting with other applications over interrupts.

The problem with the TSR approach was the fragile high coupling it required for the limited resource of operating system interrupts. We know now (and then) that highly coupled systems are fragile. Yet we seem to never learn that lesson.

Look at the state of most "enterprise" applications today. You think you have 5 different applications that share the same database. In reality, you have one massive application, partitioned at the application layer but still highly coupled once you get to the database. It's hard to make changes to the shared database without breaking (or at least looking hard) at the existing highly coupled applications. In hindsight, I think we're going to regret the way we've tended to use databases in large corporate environments. Part of the problem is the myth of a single source of knowledge throughout the enterprise. I've talked to a bunch of CIO's who say "I don't care what you do in the applications you build, but my database schemas are precious". This is a short-sighted view of how to build flexible architectures. Much of the work done now in SOA is the attempt to untangle this mess. How did we solve the problem with TSR's? Event driven operating systems. And how are we going to fix enterprise architectures? The same way! Just as Windows (built on the foundation of DOS) gave us a sane way to do multi-tasking, event driven architectures built on the foundation of our ridiculously coupled databases will make "enterprise architecture" more sane. Eventually, we'll make fundamental changes (just like Windows did when it added true premptive multi-tasking) to make things even better.

This is one of several places that Ruby on Rails nailed the right answer dead on. One of the obvious-in-hindsight decisions Rails encouraged/enforced was the "1 application, 1 database" style of application. If you need 2 applications to share knowledge, you do it with ActiveResource and messaging between applications. Once you assume that style of application, so much gets easier. You can stop worrying about the nasty high coupling between application's data access and start thinking about what individual applications need to do.

It's going to take a long time to fix this, though. While TSR's and DOS was a bit of a mess, it was a pretty small scale mess compared to what enterprise architecture looks like now. Given the fetishization of databases and the flawed assumptions that people apply to how to use them, it's going to take the rest of our lives to untangle this mess. There are movements afoot to rethink this within the database community (exemplified by books like Refactoring Databases by Scott Ambler and Pramod Sadalage. Just like with TSR's, we need to rethink Sidekick Oriented Architecture.

Saturday, September 06, 2008

Thanks for, Building

When things are automated, you just forget about them. And that's a good thing: you'd rather spend your time thinking about important things, like "is my design correct" or "should I be refactoring now". Developers who haven't embraced continuous integration yet spend way too much time doing something (builds) that are better left to automation. At the No Fluff, Just Stuff shows, one of the Sunday poll questions for the crowd is "How many are using continuous integration?" I'm always depressed to see that it's generally less than half the crowd. I can't think of any single practice that makes projects better than CI. Moving from building your project as a bunch of snowflakes (if you build using an IDE, what you build will be subtley different than what your neighbor builds, thus you aren't building the same thing: each build is like a snowflake) to a single, canonical build allows you to stop worrying about build problems. If it fails on the CI machine, it's broken, regardless of what happens on your machine. It frees you from the "midnight before the big demonstration, everyone's standing around Fred's machine because his is the only build that works" anti-pattern.

Having all that noise automated just lets you forget about it. As I was using the restroom in our Chicago office, I saw the perfect metaphor for this. Most of the newer buildings you enter now have auto-flush toilets, so that you don't have to touch any of the surfaces. It's become so common, in fact, that you forget that step in places that don't offer automatic flushing. In our office in Chicago, in the public restrooms, you see the "Thanks for flushing" sign on the wall. What a perfect metaphor! When it happens automatically, you can forget about it. When it doesn't, you have to remind people that it's important for the hygeine and smell of the environment to do the right thing.

If you haven't already, install auto-flushers for your project, in the form of continuous integration, and start thinking about more important things.

Friday, August 29, 2008

The Music Project

I was talking to my friend Brian Goetz recently, and he reminded me of a blog entry he posted a while back. He's digitizing his entire music collection, and he's done all the research. This is appealing to me because I recently switched over to zero reliance on the shiny plastic disks once I've ripped them. All my music consumption is now electronic, mostly via iPod (the car was the last frontier, and now that's switched over as well). I have a large-ish CD collection (~1500 CDs), so translating them to electronic form is no small feat. And I only have to do it once for the rest of my life.

Based on my conversations with Brian, I realized the following:
  • Ripping and encoding are completely separate activities
  • The most important thing is to get a reliable, error corrected rip
  • You don't want to get stuck with a proprietary format (as much as I love Apple, I don't want my music collection tied up in one of their formats forever)
  • Lossless is the way to go, so that you don't loose any data in this process
  • Cross-encoding allows you to rip your music in a "core" format and selectively support proprietary formats and reduced file sizes (and lossiness)
His original entry is about his Windows and Linux based infrastructure. I replicated his same setup (with minor differences) on the Mac.

A Home

First, you need a home for all these files. I recently got a network attached storage device to hold our growing collection of digital photos. I'm my family's tech support, and if anything ever happens to those photos, I'm a dead man! After doing a lot of research on quality, Mac compatibility (my household is 100% Mac now), I ended up with the NETGEAR RND4250 ReadyNAS, which comes with 500 GB of storage (2 mirrored 500 GB drives), with a capacity of 2 TB. I ended up bumping it up to a TB when I realized how big the combination of music + pictures would be, and the Netgear handled it beautifully.


I didn't realize this, but most rippers don't take advantage of the error correction bits on the CD, and they are wimps: they give up way too easily. The trick is to find a ripper that is relentless and tries its hardest to get all the bits off the CD. The ripper/encoder I ended up using is Max, which supports Leopard in it's latest (allegedly) unstable form, and previous Mac OS X versions in it's previous (stable) release. I caveat "unstable" because I used it a lot and it was rock solid for me on Leopard. One of the nice things about Max is its support for different rippers: fast & carefree, or paranoid. The latter is the one I want, based on the cdparanoia project. You can configure this ripper to never give up until it gets a clean read of the disk. Several of my CDs trundled for 6-10 hours before Max finally reported success, including a couple that I'd given up for dead.


OK, so now I have a good rip, I need to encode it (this post makes it sound like separate steps, but Max handles both for you). As I stated earlier, I don't want to get trapped by a specific format. I ended up (like Brian) choosing FLAC. FLAC is a open standard encoding for music that offers lossless compression, which is what I wanted. The FLAC spec also allows for more aggressive compression without loss of data, depending on your patience. It's designed to take longer to encode but have no impact on playback time. I choose the most aggressive because I have time, hardware (a MacPro with 4 processors), and I want to conserve space if possible. But, iTunes (which is how I play and sync my music) doesn't support FLAC. Max to the rescue: it will let you do parallel encoding. I set Max up to encode the ripped music files to both lossless FLAC and lossless MP4 (Apple's format). The only downside is that it won't allow you to choose different directories for the encoding. The FLAC files I'm placing on a RAID mirrored network-attached storage drive (remember, I never want to do this again!). So, I ended up writing a little Rake file to handle automatically moving the files from one place to another. I rip them all to the RAID drive, then let the script move them (preserving directories) to the other. The script is here, if anyone wants it (no warranty expressed or implied -- you'll have to change all the directories, and if you use this to erase your hard drive I'll shed a tear for you, but might just laugh).
task :copy do
count = 0
skipped = 0
FileList["**/*.m4a"].each do |f|
artist, album = recording_info_based_on f
if File.exist? "#{DEST}/#{artist}/#{album}/#{File.basename(f)}"
puts "\tsomething is amiss; I'm skipping: #{f}"
skipped += 1
FileUtils.mkdir "#{DEST}/#{artist}" unless File.exist? "#{DEST}/#{artist}"
FileUtils.mkdir "#{DEST}/#{artist}/#{album}" unless File.exist? "#{DEST}/#{artist}/#{album}"
puts "#{artist} - #{album} - #{File.basename(f)}"
count += 1
FileUtils.cp f,"#{DEST}/#{artist}/#{album}"
puts "copied #{count} files\nskipped #{skipped} files"

def recording_info_based_on filename
File.expand_path(filename) =~ /.*\/(.*)\/(.*)\/.*/
return $1, $2

I also made a rake task to report any that ended up missing from the original FLAC directories to the AAC files (just in case something went amiss during a copy process, or I screwed up and deleted something by mistake). I want to make sure that the convenience Apple-format files match the canonical source (the FLAC) files. So, this is the "missing" rake task:

task :report_missings do
count = 0
FileList["**/*.flac"].each do |f|
artist, album = recording_info_based_on f
dest_file_name = File.basename(f).sub /flac/, "m4a"
unless File.exist? "#{DEST}/#{artist}/#{album}/#{dest_file_name}"
puts "missing #{f.sub /\.flac/, ''}"
count += 1
puts "found #{count} missing files"


It took me about 2 months of ripping while I'm around my computer, running 2 computers (my laptop and desktop) in parallel. In the end, though, I ended up with 453 GB of music files, the FLAC ones safely tucked away on a mirrored drive and the M4A ones on my desktop, ready to be synced to my iPod (or a subset of them, anyway). Now, when I get a new CD, I rip it using Max to the NAS and either copy the files by hand (if it's just one CD) or use the Rake file to move lots en-masse. Storage is now dirt cheap, and I've leveraged almost a terabyte of it keeping the music files in 2 formats. I also recently bought a portable 500 GB drive so that I can keep all my music with me on the road. It's a copy of the desktop M4A files, but it's easy just to mirror the Music directory from the desktop to the portable drive.

I achieved my goal: an open archival format that I hope will be around for a very long time, and a convenience version for the way I happen to consume them today. And the shiny disks? I put them all in binders, so that if I ever need one of them (or it's sleeve), I can rummage around in the (mostly) alphabetical CD volumes. I didn't put a huge amount of effort creating an expandable storage that makes it easy to keep them in strict order because that would take lots of effort and it isn't something I expect to have to do often. If it turns out I got back to them all the time, I'll invest the time then.

Wednesday, August 20, 2008

97 Things Every Software Architect Should Know

A while back, Richard Monson-Haelfel was working on a presentation called "10 Things Every Software Architect Should Know", which was a great idea for a talk. To solicit ideas, he posted to several mailing lists where architect-types lurk about, and he got flooded with responses. I was one of the early contributors because I had been thinking about some of this stuff anyway at the time he posted the call. Richard liked the entries so much, he decided to put up a wiki through O'Reilly to publish all these little snippets of advice (or axioms). O'Reilly liked the results so much that they are considering making it a book, sort of like Beautiful Code, but with much shorter (and many more) axioms. I'm not sure where the magic number of 97 came from, but that seems like a reasonable number. The whole "About" story appears on the site, here.

Well, it's out of the shadows now. O'Reilly has moved the site to NearTime (a sparkly hosted wiki site) and it's now public here. Go visit: there is some incredibly good advice here, in little bite-sized chunks. I greatly enjoyed reading the site and soaking up some of the great advice, all derived from real-world slings and arrows. Kudos to Richard for seeing this through.

Tuesday, August 12, 2008

In Praise of Technical Reviewers

It came to my attention recently that I had made a bad assumption about the Productive Programmer book. My understanding (and apparently this is common) is that the technical reviewers of the book would get an entry on the title page of the book. Apparently, that's not the case. That's why I didn't put them in the acknowledgements: I assumed they had already been recognized. But they haven't, so I'm going to rectify it.

First, I've added a special paragraph to the acknowledgements in the 2nd printing of the book, thanking the hard-working technical reviewers. This is a little unusual (generally, nothing changes between printings, but I felt badly about this). The other thing I'm going to do is thank them here. This is the new paragraph in the 2nd printing:

A special thanks goes out to the technical reviewers for this book. Without their hard work and dedication, this book would suffer lots of silly mistakes and confusing explanations. Thanks to Greg Ostravich (who has reviewed every book of mine for the last few years and gotten no recognition, unfortunately), Venkat Subramaniam, David Bock, Nathaniel Schutta, and Matthew McCullough.

Greg gets a special thanks. He's reviewed everything I've written over the past few years, and circumstances keep preventing him from being acknowledged. In the 2006 No Fluff, Just Stuff Anthology (which he reviewed), I was under the same mistaken assumption that the reviewers got a shout out. In the 2007 No Fluff, Just Stuff Anthology (which he reviewed), I specifically wrote a thanks to him and the other reviewers. But, alas, the book came in too long, and several pieces got cut, along with my original introduction to the book (it was replaced by Ted Neward's). Unfortunately, the shout out got axed with the introduction. And, now, clearly demonstrating hope over experience, Greg volunteered to review the Productive Programmer, and the same thing happened. So while I'm thanking the other Productive Programmer technical reviewers, I'm both thanking and begging forgiveness from Greg. Good job, buddy, and unacknowledged for too long.

Wednesday, July 23, 2008

Rich Web Experience 2008 x 2

Rich Web Experience East Rich Web Experience EastLast year, one of the most refreshing conferences where I spoke was the Rich Web Experience. Too often, conferences are mono-cultures, and mono-cultures are generally boring. RWE is fundamentally different because it's what I call a condiment conference. The focus is on how to make applications better, not how to make the plumbing better. Nothing wrong with plumbing, mind you, it's just refreshing to look at faucets for a change.

This conference has an eclectic mix of developers. Hallway conversations lack the implicit assumptions you can generally make at main course conferences. For example, all Java developers have an implicit context. At The Rich Web Experience, you have to throw away your base assumptions, both in sessions and conversations. Just like travel broadens you because you meet people with different contexts and experiences, attending the Rich Web Experience does the same for technologists. Instead of the usual low-level animosity that each technology tribe exhibits for the non-tribe members, everyone focuses on common ground. This year, there are 2: one near the east coast and another on the west.

Wednesday, July 16, 2008

Finally Treeware

pp book cover on tableOK, it's finally here. I got my copy of The Productive Programmer in the mail last week, and today Amazon updated it's status to "In stock". For all those who have been patiently awaiting this book, it's finally here. Finally! The first glimmer of this book started in 2005(!), so it's taken upwards of 3 years to convert what David Bock and I were talking about into actual tree-ware. Because it took so long to produce, I suggest that, if you are going to buy one, you should go ahead and buy 2, in case you lose one. And, of course, nothing says loving to your spouse like an O'Reilly book. A great gift for anniversaries, birthdays, bar mitzvah's, etc.

Even if you don't buy the book (and can ignore my bitter tears) and you still have an interest in developer productivity, you can head over to the Productive Programmer wiki and contribute.

Thursday, July 10, 2008

Sony eReader Update: It's all Bad and Ugly

Back in January of 2007, I wrote about my impressions of the Good, the Bad, and the Ugly of the Sony eReader electronic book reader. I thought I’d update that, given my time with the device and the eBook service. And the news isn’t good.

One of the major annoyances of the eReader is the inability to reflow PDFs. That’s a complex problem, so they get a pass on the actual reflowing part. The solution they offered was the ability to view PDFs in either of two formats: either the entire page or just the width of the text on the page. The latter worked reasonably well for most PDFs when viewed in the landscape mode of the eReader. But here’s the fatal bug: when you switch pages in the PDF view, it restores the “width of text” view to the “width of page” view, but doesn’t update the mode on the reader. Thus, you must hit the button to actually get it back to what it thinks is officially “width of page” view, then hit it again to actually get back to “width of text” mode. Page refreshes on the eReader are very slow, which isn’t a big deal when reading a book because you only have to do it occasionally. But forcing the multiple page switches just to restore it to the mode it says it’s on is deadly. It effectively made reading PDFs on the device unpalatable. I reported this back in my original review, and predicted that it would be fixed in a software update (leaving aside for the moment how the device could have ever shipped with such an obvious bug).

It never happened. The new version of the reader came out with no software update in sight for this killer bug. I don’t know if they’ve fixed it in the new version because, frankly, I wouldn’t take one if they gave it to me. They did offer to sell me a new one when my original died. I turned it on one day in December and the screen was a garbled mess. I contacted their support who told me that, since mine was out of warranty, my best course of action was to purchase another, new version. That’s never going to happen.

In fact, the whole experience has soured me on Sony. They used to compete in mind share at least with Apple for delivering innovative products, with a modicum of understanding things like design and aesthetics. Apparently, they’ve abandoned that. I still own a single Sony product, the PSP Portable, which embodies many good features and design touches. But seeing how they treat their customers for an admittedly small, trivial market is enough for me to cast all their products in doubt. They clearly don’t have any concept of quality assurance (given the original bug) or responsibility (never fixing it). It’s a shame too because reading actual eBooks on the reader wasn’t bad. I probably read about 40 books on it, and liked it a lot. But, given that you can’t read PDFs on it, and their eBook format is proprietary (another annoying characteristic about Sony — memory sticks anyone?), I won’t buy another one.

I’ve looked at the Kindle but haven’t taken the plunge yet. I want a killer user experience, and it doesn’t look like it. I wish Apple (or someone who understands design like Apple) would release an eReader so I could read the Neal Stephenson Baroque Trilogy without herniating myself!

Tuesday, July 01, 2008

Expert Panel at Agile Experience

Last weekend, I spoke at the Agile Experience in Reston. It was a great conference, lots of interesting topics, and a different crowd than most technical conferences. Half the attendees were managers, and everyone was enthused about Agile development. The experience level with Agile was diverse too, so it was fun to get out-of-the-blue questions. My first talk was Real-World Agile, and I start that talk by soliciting the agenda from the crowd (I open up a text editor and make the group tell me what they want to talk about). That was great here because it let me understand the actual pain points for the attendees, and it makes for a more interesting presentation for me.

On Friday night, we had a wide-ranging expert panel discussion which ranged from methodology to soft skills to cultural fit. And it was recorded. Come hear me stereotype all Americans as assholes!

Friday, June 27, 2008

Communication Channels & Cell Phones

I noticed the other day that lots of states now ban talking on cell phones while driving, yet they don’t outlaw talking to a person sitting on the seat next to you. Why is that? I have a suspicion: talking to someone on a cell phone takes a lot more concentration. Many studies place accident rates for drunk drivers and cell-phone talkers evenly. The implication is that talking on your cell phone robs you of as much concentration as drinking!

But my experience with software development bears this out. I’ve worked on several geographically diverse projects, where the communication channel is reduced to phone calls in conference rooms. This is a terrible communication medium because you can’t see the other people, can’t see their facial expressions and body language, and don’t understand the subtleties of their vocal inflections. Humans are social animals (even developers who don’t interact well with other humanoids). We don’t realize how much we rely on those non-verbal communication channels until we notice how much concentration it takes to talk on a cell phone in a car.

Alistair Cockburn has a great diagram in his book Agile Software Development.

Temperature of Communication Channels

He has modeled communication effectiveness to temperature dispersal from a heating source. The further away from the source, the colder you are. This is a great metaphor because it takes into account the subtle communication channels upon which we depend but don’t realize are there until they’re not.

The importance of this for software development shouldn’t be ignored. If you engage in distributed development, insist on warmer communications, like video conferencing. Even Skype video, while crude sometimes, beats the phone-call-in-crowded-conference-room anti-pattern. On one of my projects, we had serious cross-ocean communication gaps because we’d never met in person. The results aren’t surprising: personality clashes, misunderstanding about technical problem resolution that cost us time, and general stand-offishness between the teams. Wisely, my project manager invoked the XP “Move people around” principle, and we shuffled developers from the US to India and vice-versa, for a couple of iterations each. This was on a fixed-bid project, so we incurred the cost for the travel ourselves. And it was the best thing that ever happened to the project. Once you’ve met people and spent time with them, it makes even conference call communication better, because you can pick up on subtle clues in people’s voices that indicate a facial expression you’ve seen for yourself. One of the developers on our US team was widely loathed by the developers in India (so much so that, when I went there, they had drawn an “X” over his face in the team photo we had on the wall!). But, when he went there himself, he became fast friends with many of the developers. Some personalities must be experienced in person before they are tolerable, especially to different cultures.

So, the next time you are driving and talking on your cell phone, do 2 things: keep an eye out for the police in your rear view mirror, and notice how much more mental energy it takes to talk on the phone rather than to a person sitting with you. This to me is the proof that Cockburn nailed it.

Tuesday, June 17, 2008

Seoul Observations

I have a cool job. The opportunity to travel to interesting places is one of the coolest parts. Recently, I had the chance to go for the first time to Seoul, Korea, speaking at the Entrue Developers Conference.The conference was very well organized, but it was a conference like any other. The interesting things for me were the observations I made while I was there.

First up, phones and Internet. I proudly carry an iPhone (one of the first purchasers, don't you know), and smugly rub it in the face of all the poor phone 1.0 owners. In Korea, their cell network is far more advanced than ours. So advanced, in fact, that they aren't even backwards compatible with our phones. None of my other co-worker's phones worked either, except the one that had a special phone for Korea. The other notable technology difference was the speed of the Internet connections. The Internet in my hotel room was one of the fastest I've ever used! Stuff that takes 45 minutes to download here takes 7 minutes in Seoul. Unfortunately, I didn't actually measure the speed, but it was blindingly fast. And that was just my hotel room connection!

Second, business in Seoul is much more formal than here. Before travelling there, I asked my co-worker from China meeting me there about the dress code. I try to remember to do that when I travel so that I minimize the ugly Americaness that leaks through anyway. He said "Oh, don't worry -- it's business casual. Slacks and button up shirt is fine." Well, I get there, and I'm the only one there without a suit. One of the nice things our host did for us was an invitation for a lunch with about 200 corporate CIO's in Seoul. My co-worker (the same one who advised me) leaned over and said "See that guy over there -- he isn't wearing a suit, so you aren't the only one." Thanks! In general, business is much more formal. Developers wear suits and ties to work, which is increasingly rare here.

Third, I was asked to give a presentation at one of the local companies, to their enterprise architecture group (SOA is big in Korea). The only snag: they don't speak english, and I don't speak Korean. The local guy there who invited me to speak at the conference volunteered to translate for me. This was a first: I would discuss the contents of a slide or example, then pause while he translated the entire thing to Korean. He was extremely knowlegable about technology, so I have high confidence in his translation. It was just weird: speak, pause, speak, pause...for almost 2 hours!

Fourth, they have different technology priorities there. While a lot of big companies in the US have embraced the run-away accidental complexity that is SOA, they have taken it to a manaical level. We sat through a presentation of an SOA Goverance Framework. They have every detail nailed down to the tiniest degree. It looked great on paper, but it was absolutely unworkable in the real world. Can you really tell the business that they must slow down their pace of innovation because the SOA goverance framework keeps all IT at a snail's pace? Developers spend more time filling out forms and consulting the thousands of pages of documentation on how to govern their SOA than they do actually writing code. I asked during the presentation how this impacts the business and got blank stares: business? What does the business have to do with our enterprise architecture? It was the most elaborate, meticulous, detailed, unworkable framework I've ever seen.

My fifth observation is the dichotomy of their IT priorities. On the one hand, they've embraced SOA (particulary the bad parts, in my opinion) whole-heartedly. The important priorities for their enterprise development seems to be to lock down all variability (and therefore flexibility, evolvability, and adaptability) from their enterprises. They have state of the art SOA fetish there, building meta-SOA frameworks. Yet, you ask them about some of the things considered interesting in the states (like the rise of dynamic languages), it's not even on their radar. It's not that they have looked at it and disregarded it. They haven't even heard of most dynamic languages. Ruby? Groovy? Python? The only one they've even heard of is Perl, and they have a poisionous disdain for it. They have of course heard of JavaScript, but it's a toy language for visual effects in browsers. They treat it like an extension of CSS.

Of course, the best part of traveling is the people: I met so many friendly, gregarious, funny, open people there. We had several great meals, lubricated either by fine wine or Soju, the Korean version of Sake. It's wonderful stuff: it tastes like mild Sake, but it packs almost the punch of vodka. We talked about lots of stuff, both technical and non-technical. The last speaker's dinner had geeks from Korea, the US, China, Japan, Singapore, Malyasia, and lots of other places. Travel does indeed broaden you, often in unexpected ways.

Monday, June 09, 2008

So Many Conferences, So Little Time...

Lots of people know me from the conferences where I speak. If you only see me at Java conferences, you probably have the impression I'm primarily a Java guy. But I have a great love for lots of types of technology. Languages and platforms are just tools, and you can build effective software in all the major platforms. Which is why I set out with an informal goal this year: speak at the major technology conferences for each of the three technologies I follow the most closely: Java, Ruby, and .NET. And I succeeded! As far as I know, I'm the only person on earth speaking in rapid succession at JavaOne, RailsConf, and TechEd (in that order).

This isn't just a vain pursuit. One of the reasons I love working at ThoughtWorks is the experiment of doing agile software development at the enterprise level. To do that, you need an understanding of the major technology stacks in play. Yes, I love Ruby and Ruby on Rails right now, but there are times when the appropriate choice for the client is a Java and/or .NET solution. In fact, I think that being able to compose solutions with languages on platforms is an important trend. I want to understand technology divorced from hype and religion, leaving just the efficacy of the platform in place. Our challenge is understanding how to best leverage the platform to meet the business needs for that application and that customer. And that's plenty challenge enough.

Monday, June 02, 2008

Trippin' with TripIt

Let's say you were traveling to Chicago back in 1990, and you wondered about the weather during your trip. How would you find out Chicago's weather? Well, you could turn on the Weather channel and wait for Chicago to come up. Or, some national news will have it, but you have to catch it just at the right time. Or, if you really needed to know, you could go to a library and look at a newspaper. There were also call-up services you could call and get the information. It was such a hassle, though, that you would probably not bother, especially if you already had a feel for what the weather is like for a particular time of year. The Internet changed that. Now, it literally takes me less than 15 seconds when I'm at my computer to get a 10-day forecast for Chicago (or Singapore or Jakarta). The Internet makes things so easy that you'll actually do them. While getting the weather wasn't impossible, it was such a hassle that it was discouraging.

Fast forward to now. A while back, a friend invited me to Dopplr, a social networking site for travelers. It allows you to see when the other people in your social network are nearby. Great idea, and useful service, but the duplication of data was bothersome for me. My calendar is already incredibly complex, and keeping another calendar up to date didn't appeal to me. I decided to wait and see if you could point it to an iCal instance or something, which would ease that pain.

Then, TripIt came along. TripIt is a new web application that helps manage your travel. Here is the scenario. Let's say you are going on a trip to Des Moines. You book your flight on Delta, a room at the Marriott, and a rental car from Avis. Each of them sends you a confirmation email. In the past, I had an editor template set up to enter all that information (confirmation numbers, addresses, etc.) so that they would be consistent, and I harvested the disparate information myself. That's where TripIt's genius comes in. They parse standard confirmation emails from airlines, hotels, car rental companies, aggregators (like Orbitz), and even travel agents and build a standardized itinerary for you. When you get the confirmation email, you just forward it to and it automagically does the rest. It builds a nice itinerary, with airline checkin links, a link to SeatGuru with the type of plane already selected, a map of your destination city -- basically, everything you need to know. That's nice and painless.

But it gets even better than that. TripIt maintains an iCal instance for you, to which you can subscribe with all your travel details. So, here's my new workflow. I have forwarding rules set up for the airlines and other travel services I frequent to automatically forward the confirmation to TripIt as soon as it hits my inbox. I book travel, and that's it. Five minutes later, it shows up on my calendar, organized beautifully. And remember Dopplr? It will also allow you to see when friends are close and let your invited friends see your itineraries (on a case-by-case basis -- you can share specific trips with friends). It does what Dopplr does without Dopplr. TripIt took the good idea of Dopplr and supercharged it.

TripIt is an awesome combination of travel service, social networking site, and personal travel information manager rolled into one seamless whole. It is one of those services that you instantly wonder how you ever lived without. In the past, when I knew I was going to be in the same city as a friend, we would email exchange our itineraries. Now, we just share them on TripIt. It's so easy, it's trivial to do, like getting the weather, so it's much more likely to happen. For some social networking sites, you have to experience it before you understand the attraction (like Twitter). TripIt has no such grokking curve: I instantly knew that it was going to be an important improvement to my workflow, with social networking goodies thrown in for free.

Friday, May 23, 2008

ThoughtWorks Anthology

ThoughtWorks Anthology CoverThis isn't a book review because it's considered bad taste to review books in which you participate. But, The ThoughtWorks Anthology is an interesting project, and I thought I could provide some perspective on it's genesis. TWA offers a glimpse of why ThoughtWorks is such a cool place to work: lots of people passionate about technology, who care about changing the IT industry.

As you may already know, I compiled two other anthologies, the 2006 and 2007 No Fluff, Just Stuff Anthologies (editing is too strong a word, I was mostly just wrangling articles from writers and organizing stuff). Those were easy because we had a hard deadline: it doesn't make sense to release the 2006 anthology in 2008. Shortly after the first No Fluff, Just Stuff anthology was published, several people at ThoughtWorks (including Martin Fowler and Rebecca Parsons, the ThoughtWorks CTO) and I started talking about the possibility of a ThoughtWorks anthology. I don't mean to suggest that it was my idea; they were just talking to me about logistics. It was decided that Rebecca would edit it, and solicitations for articles went out. Lots of ThoughtWorkers expressed an interest (including me), and writing began.

But three problems manifested: first, no implied deadline existed, because this wasn't a dated book. Two, everyone working on and contributing to the book had day jobs, actually doing the stuff they were writing about in lots of cases. In addition to their day jobs, many were involved in open source projects: ThoughtWorkers contribute a lot to open source projects. And, third, ThoughtWorkers are notoriously ADD. Rebecca had the unenviable and well nigh impossible job of cat herding a bunch of ThoughtWorkers under the above constraints towards creating the anthology. And, of course, Rebecca is insanely busy as well. But, obviously, she made it happen. It took a while, but it's out now. It covers the gamut of things that interest ThoughtWorks, from project management to pretty hard core technical stuff, with some forward looking tidbits thrown in. The ThoughtWorks Anthology has essays by Roy Singham (the Roy of Roy's Social Experiment), Martin Fowler, Rebecca Parsons, me, Jeff Bay, Michael Robinson, Tiffany Lentz, Stelios Pantazopoulos, Ian Robinson, Erik Doernenburg, Julian Simpson, Dave Farley, Kristan Vingrys, and James Bull. The article titles give a glimpse of the range of information in this book: Solving the Business Software "Last Mile", One Lair and Twenty Ruby DSLs, The Lush Landscape of Languages, Polyglot Programming, Object Calisthenics, What Is an Iteration Manager Anyway?, Project Vital Signs, Consumer-Driven Contracts: A Service Evolution Pattern, Domain Annotations, Refactoring Ant Build Files, Single-Click Software Release, Agile vs. Waterfall Testing for Enterprise Web Apps, and Pragmatic Performance Testing.

Anthologies are currently all the rage. No only the No Fluff, Just Stuff ones, but O'Reilly's Beautiful Code (and a rumored upcoming one on architecture), and, of course, this one. Anthologies are nice because you can read them in little bursts. I think this one has both interesting stuff and does ThoughtWorks proud as a company. Kudos to all the writers and Rebecca for pulling it off. And don't ask her about a sequel, at least not yet: she needs time to heal.

Thursday, May 15, 2008

RIA == Platform Play

The shortness of the collective memory of the development world depresses me sometimes. Joel Spolsky has a great blog post from 2004 entitled How Microsoft Lost the API War. In it, he describes the real Microsoft crown jewel that lead to their domination of the personal computer: the Win32 API. If you were writing software in the mid-90's, you were writing it to the Win32 API. You might be using Visual BASIC, Delphi, PowerBuilder, Visual Objects (all 2 of it's users), FoxPro, dBASE for Windows (all 4 of it's users), etc. But you were writing desktop applications that ran on Windows. Then, Netscape came along and showed everyone that you could write software to open standards. Even though Microsoft eventually trampled Netscape, the damage was done: now you could write software that ran in browsers. Of course, Java had the whole "write once, run anywhere" mantra going, but building desktop applications in Java sucked then and it sucks now. Only a few companies have ever done it well (like JetBrains). Netscape's enduring legacy was that writing to standards (like HTML, CSS, eventually DHTML and JavaScript) was, while painful, possible. You can create rich applications where no one knows or cares in which language it's written.

Fast forward to today's development world. We're used to the pain required to write good web applications. We've got Ajax frameworks to smooth over the painful parts of JavaScript, people actually understand CSS, and you can build compelling applications on the web. Not an office suite, mind you, but pretty functional applications.

Which brings us around to the current hotness, Rich Internet Applications (RIA). Ever wonder why Adobe and Microsoft are slugging it out for that space? And Sun is running along behind with JavaFX saying "Wait, we want to fight too!". It's the new platform play. We've dealt with the pain of writing good looking web applications so long that when someone comes along and shows pretty pixels, we swoon. Yes, you can create beautiful applications using Silverlight and Flex. And Sun showed some awesome demos of JavaFX at JavaOne. But, if you write an application in one of those tools, you've bought a platform. You are no longer in a standards space. You can't take a Silverlight application and port it to Flex without a rewrite. Same goes with JavaFX. Whoever wins the RIA war has the new dominant web platform, just like Win32 back in the day. Sounds like a good reason for big companies to pour resources into the effort.

Of course, as my friend Mike Nygard pointed out during several expert panels, you pretty much buy into a platform play every time you start writing software. Yes, JEE is a standard across many application servers, but how many people are really careful to isolate the seductive goodies that come with your application server from the real JEE stuff? My guess is not many. So, you've kind of implicitly bought into a platform anyway. That's doubly true for .NET: yes, Mono exists, but more of a proof of concept than anything else.

Maybe this is OK. But I remember the reason to avoid single vendor platforms. One of the compelling reasons to move to Java when the JEE specs came out was the possibility of moving to another vendor, even with some strenuous effort. I can't tell you have many consulting clients I encountered in the mid-90's who had bought into a particular platform (say, Microsoft Transaction Server or ColdFusion) and were very productive with it...until they hit a real bug in the platform itself. I consulted at one company who had an 18 month project written to a proprietary platform when they encountered a show-stopping bug. They contacted the vendor who said "Yeah, that's a known bug. We're going to get a fix out for that in the next release, which should be in about 12 months". Even though it took them a long time, they scrapped what they written and started over, in Java. And you can bet they were careful about isolating the application server stuff from the specification.

Building to a single platform isn't necessarily a bad thing, as long as you go in with your eyes open. It's a trade-off: perhaps the pretty dancing pixels in Silverlight are worth the chance that you'll find something it doesn't support and end up with an application that doesn't quite do what your users want. But, beware of Dietlzer's Law: users want what they want, and are grumpy if you tell them "no". And, if you go with one of the RIA platforms, get ready to marginalize everyone without a PC. Some of these platforms have some support on mobile devices (and surely more to com, in a massive wave), but it's not going to be as good as plain old HTML for a long time.

If you are going to choose an RIA platform, please use it appropriately. Don't build forms in it: HTML is plenty good for data entry forms. I saw a demo recently of a text box that rippled when you touched it with a mouse or typed in it. Please. RIA is extraordinarily good for displaying information that is well nigh impossible in HTML and JavaScript. Use it for those kinds of visualizations. Most of the RIA platforms now don't support web metaphors like bookmarks, or addressable URL's. There's no reason to completely discard the parts of HTML that work well for which it is designed. Use HTML for general interactions, and use RIA for sophisticated visualizations. And know going in that you've bought a platform that you're going to have to live with for a long time.

A slightly different version of this rant, more Silverlight focused, appears here. I was at a party at a friends house that had lots of .NET folks there, during the Microsoft MVP Summit. I was getting some impending dead-line work done and Eric was recording a series of podcasts, the tone of which can be summed up by "Silverlight will cure cancer, solve world hunger, and paint your kitchen for you". After he was done, I was quietly fuming and asked him if he wanted a contrarian view. That's what's on the podcast. You've been warned!

Tuesday, May 13, 2008


One of the interesting things that's happening in the Ruby community is the sprouting of regional conferences. Because Ruby is still nascent in the Enterprise space, it's hard to conduct conferences (especially regional ones like No Fluff, Just Stuff) because individuals probably have to pay for themselves (rather than having a corporation pay for them). This is a clear case of developers seeing the next thing they want to do, but companies lagging behind them. But Ruby (and Rails) has some pretty sophisticated stuff, which lends itself well to a conference. Thus, the rise of the regional conferences, many of them basically operating as a non-profit, trying just to cover expenses with the conference fees. And that describes RubyNation on August 1st & 2nd, where I'm delivering the opening keynote. It speaks highly of the Ruby community that the conference organizers are willing to put in a vast amount of effort (and believe me, putting on a conference is a huge undertaking) just for the love of the technology. That's a rare thing indeed, and shows what a wonderful place in general that the Ruby community is. I'm happy to speak at these regional conferences because I like the passion for technology on display. It's fun being around people who really care about technology, and love to debate it, discuss it, and generally wallow around in it (which is one of the reasons that I love ThoughtWorks so much). So, even if you have to get on a plane, come to Ruby Nation and wallow around with me. And, it that's no enough incentive, come because Stuart Halloway is making a relatively rare appearance as the closing keynoter. Should be a great couple of days.

Monday, May 12, 2008

Agile IT! Experience

I'm speaking at the upcoming Agile IT! Experience, in Reston from June 26 through June 28. Most conferences focus on one attendee demographic: developers, business analysts, managers, CIO's (which entails much golfing and little conferencing). But projects must encompass a wide variety of roles, all of whom must work together. The upcoming Agile IT! conference breaks that mold. It caters to all members of the project ecosystem. That means that managers will rub elbows with developers, business analysts can chat with testers, and use the synergy of their experience to both understand how the other half lives and pick up valuable insights from other people that will make their work better.

I really like these kind of mixed conferences. Too often, conferences are too homogeneous. Getting a group of people together that have different perspectives always broadens the perspectives of everyone. This new conference blends several normally separate worlds, enriching all of them. I'm looking forward to IT!

Tuesday, April 29, 2008

Book Review: Programming Groovy

Groovy Recipes CoverIn the movie 2001, A Space Odyssey, the director Stanley Kubrick spends a vast amount of time with lava lamp-style special effects. The last part of the movie is an interminable journey through the same scene with different colors. Kubrick would never make that movie now: at least a third would end up on the cutting room floor. But he was entranced with the shiny new technology and indulged it to distraction.

The same thing tends to happen when a new technology comes out. If you are an early adopter, you'll slog through the lava-lamp effects to get to the meat of the new technology. Groovy has followed this route. Because I've been following Groovy for a while, most of the previous Groovy books have been in two categories:
  • wow, look at this new thing! Isn't it cool that you can do all this stuff, which contains useful information but also a fair amount of "look, she's walking upside down for 10 minutes" kind of stuff

  • recipes for getting real work done (where Groovy Recipes falls, so I think it's very well named. I reviewed it just before this one.)
I don't mean for this list to sound pejorative: every new technology comes out with these books first: it's an inevitable part of the technology life-cycle for new languages. But eventually someone comes out with a book that no longer treats the new language like a shiny new thing, but as a serious contender in the language universe. And that's Programming Groovy by Venkat Subramaniam (disclaimer: I know Venkat, but you shouldn't hold that against him). It doesn't treat Groovy like a novelty act. It also doesn't provide recipes. Instead, it treats Groovy as a real programming language. I'll have to admit, this is the Groovy book for which I've been waiting.

Groovy is friendlier syntax for programming Java (I called it the real JDK here). Books about the practical aspects of Groovy are very important because, as a language, it resides in a unique place: it's a low impedance way to program the Java platform, and it displaces Java for lots of common tasks. But it is also a powerful language in its own right. Programming Groovy has 4 chapters on meta-programming Groovy, and another long chapter on building domain specific languages. That's meaty stuff. It never condescends or makes excuses for Groovy but treats it like a real language. It includes lots of material that's hard to find online (like how ExpandoMetaclass really works). This is going to be both a classic in the Groovy literature and an exemplar for describing the real power of new languages.

Sunday, April 27, 2008

Book Review: Groovy Recipes

Groovy Recipes CoverWhile I've been "away", hammering away at my own book project (see this blog entry for that history), several important books have arrived that I haven't had a chance to talk about. I'll cover them in chronological order. The first is Groovy Recipes by Scott Davis. This is the Scott half of a book project that he and Venkat Subramaniam started a while ago as a single book (disclaimer: I know both authors quite well), and they quickly realized that they were writing separate books, so they made that a reality (I'll talk about Venkat's book in my next post). Groovy Recipes does what the title says: gives you recipes for how to get stuff done in Groovy. But that's only part of the value of this book. It also teaches how to become an idiomatic Groovy developer. And that's incredibly important. The classic book on C, the K&R book The C Programming Language, did 2 things for C. First and foremost, it taught developers about the c programming language. But the second more subtle thing it did was to teach developers how to be idiomatic C programmers. I can remember reading the book and marveling at the conciseness of the code, which had as much to do with the way the language was used as the language itself.

Anytime you learn a new language, you have 2 battles: first, learn the syntax (which is the easiest part -- it's just details of how familiar concepts are expressed in the new syntax). The second battle is the more important one: how to become an idiomatic programmer in that language. Developers new to a language tend to write new code just like code from their former language, using new syntax. Only when they've had time to steep in the better, more elegant ways of expressing yourself in a new language do they truly become proficient. That's what Groovy Recipes does for Groovy developers. It shows not just the syntax, but how to idiomatically use that syntax to become proficient with Groovy. Groovy is a much more powerful language than Java. While you can take a Java source file and rename it with a groovy extension and have it still work, you're writing Groovy code like a Java developer. After you've seen and used Groovy for a while, you start writing code like a Groovy developer. The Groovy Recipes book is two things: recipes for using Groovy to solve problems. But, more importantly, it teaches idiomatic Groovy programming, which is the long-term benefit of the book.

Friday, April 25, 2008

The Productive Programmer - History & Future

Productive Programmer CoverNo, I haven't fallen off the face of the earth. I haven't been blogging lately because I've been heads-down, finishing (finally!) The Productive Programmer. Understanding why it took so long requires some recent history.

Back in 2005, after having fully recovered from writing the Art of Java Web Development in 2003, I decided to look for a fresh book project. I started wondering if the world could use another book on Regular Expressions, other than the seminal work on the subject by O'Reilly. I asked a few people and the universal opinion was a resounding "No!": the current book is so good that it's pointless to try to write another. I was talking about it to David Bock, friend and No Fluff, Just Stuff colleague when I visited NovaJUG, the Java User's Group he facilitates in the DC area. He agreed with everyone else, and we started chatting about what we thought would make a good book. We had both recently observed developers struggling to make an IDE do something that was trivial on the command line, but the developers had spent their entire careers in graphical operating systems and IDE's, and couldn't bash their way out of a paper bag. We decided that it would be cool to write a book about command line tricks and techniques for developers, and that we would write said book. The original idea was to write a gigantic recipe book for programmer productivity. We contacted a publisher (who suggested the title The Productive Programmer), and we started gathering recipes and writing about them.

Then, 2 things happened. David left his job of many years and founded a consultancy, Code Sherpas, and instantly became 300% allocated. At about the same time, his wife Lorna became pregnant with triplets! Clearly, David now has his hands more than full. At about the same time, I realized that a book consisting solely of recipes would be the most boring book every written. But, by that time, I had also noticed some patterns in the productivity stuff in which I was now immersed. In fact, I remember the exact moment I realized that this book should show patterns (or principles) of productivity. I was in India, working on a ThoughtWorks project by day, and writing the book by night. I had some long conversations with some of the folks with which I was working (Hi, Mujir!) about this, and finally settled on the five principles of productivity (which later became 4). Instead of being a book of fish, I now had a fishing manual.

Of course, I've been busy as well, so the work on the book was intermittent at best. Plus the fact that the publisher was really expecting a recipe book, and it was no longer that. So, the publisher and I started going back and forth, wrestling over the format and content to try to mold it into something upon which we could agree. By this time, I had kind of taken over the book because David was so slammed. Eventually, I realized that no amount of massaging was going to create the book the publisher thought they wanted, so the publisher and I decided to part friends. I now had distilled the 5 principles down to 4 (I realized at some point that indirection is really just an aspect of canonicality) and written about them. While I was writing, I was also doing Productive Programmer talks at No Fluff, Just Stuff, honing the material that was perpetually going to be out RSN (Real Soon Now). I now had about 100 pages of refined material. But I also realized that the principles really only covered one aspect of developer productivity, the mechanical side. I also realized that I had been talking about the practical side of productivity in my older talk Clean Up Your Code and its modern sequel 10 Ways to Improve Your Code. That was the last piece. I contacted my now publisher O'Reilly with 1/2 of a productivity book and a clean vision for what the second half should be. So, from December until March this year, I wrote the 2nd half. And, fortunately, David agreed to write the forward of the book, which gave a pleasing resolution to the original vision we both had.

For all those long suffering attendees of my Productive Programmer talks at No Fluff, Just Stuff for the last 2 years: the book is ready. It's in copy editing now, and should be out soon. RSN. For real this time.

This is a thanks to all the people who've attended the talk over the last couple of years. Your questions and responses made the book better, and now you can hold one in your hands. And, now that I'm back from intensive writing, ready to reengage the blogsphere with a vengeance.

Monday, February 18, 2008

Rental Car IT

I travel a lot. Mostly, when I travel and need a car, I get one through one of the major vendor's premiere customer program, meaning that I get to walk right to the car and drive away without much formality. Which is a good thing, because every once in a while, something is amiss and I have to go to the counter. And that entails dealing with rental car IT.

The amount of information you need to rent a car seems pretty small. Even if it's not, it's very well defined. You'd think that they would have the process honed; it is, after all, the primary thrust of their business. But watching the sheer number of keystrokes that the counter attendee must perform to put me in a car is astounding. The same is true when you bring it back. It's like they are typing the great American novel. What could possibly require so typing? They know who I am. They have all my credit card, insurance, and driver's license information (enough that they'll usually just let me get in a car and drive away). This isn't confined to a single rental car company either. I've dealt with a bunch of them and they all have an enormous amount of data entry.

Unlike a lot of problem domains in which I've specifically worked, I don't know for sure what's going on for rental car companies. But here's what I suspect: their IT infrastructure is strangling them. Ironically, many of the companies that embraced IT very early on (i.e., in the mainframe era) are suffering a lot now. Much of the interest in SOA derives from the desire to get out of the deadly embrace of fossilized, astoundingly brittle IT infrastructure.

Closer to home, I've recently been doing some consulting at a large company, helping them define their evolution towards a better tomorrow. I was talking to one of their senior IT staff, and I mentioned that it seems like they don't have a really good idea what their users need software for, and that maybe we should go talk to them. He looked at me like I had a new eye sprouting on my forehead, and said to me "We've tried talking to them -- they don't know what they want, so we have to define it for them. You can't be so naive to think that we can actually talk to the users." At this large enterprise, software is its own ecosystem. The users of the software almost never come up in meetings. Instead, it's a gigantic plumbing exercise. What's ironic is that this particular company is primarily run on information. Yet, they are so far down the rabbit hole of plumbing and complexity, they have managed to create mountains of rubbish. They are in the deadly embrace of accidental complexity, not essential complexity. My coworker and I identified their IT needs for the entire enterprise as small to medium, if this were a green field. But, of course, they have information trapped in mainframes, in several different formats of databases, in packaged applications, and scattered hither and fro, like lots of enterprises.

By way of trying to get them to rethink some of their decisions, we tried (and were marginally successful) in getting them to define some core principles, like simplicity and flexibility. What was funny about that exercise were the looks we got from the no nonsense King of IT: "Of course, we want things to be simple and flexible -- why are you bothering to tell us this?" Yet, in the next sentence, they are talking about spending 3 million dollars on a packaged application to help them with one small part of their business, rather than building it themselves (which we all thought would be cheaper but take longer). That's $3,000,000. But, of course, the packaged application talks directly to their databases, meaning that we can no longer freely make changes to the database without breaking the package, meaning that we can't evolve the database, meaning that we've lost both simplicity and flexibility. Over and over, they complain when we talk about rethinking their priorities, then turn around and make the same decisions that got them where they are now. Frustrating!

At another company I did consulting for years ago, they had a special mainframe application that handled one part of their business. I don't know how much it cost, but it had to be a lot because their vendor came to them one day and said "Look, we really just can't take any more money -- we're bloated and frankly we can't figure out how to spend all you've given us already." Instead of taking this opportunity to rethink how this part of their business works, the found another mainframe application that did the same basic thing, and spent 43 million dollars changing around the life support for this thing. Fundamentally, the problem they were solving wasn't that tough. I know; we were working on a piece of it and got a good feel for the whole thing. They were so drowning in the accidental complexity of the way they had always done it that they couldn't see the actual problem anymore. I walked into a conference room at one point, and saw a diagram that literally covered the entire wall with boxes, lines, and 8-point font describing the enterprise architecture (no, sorry, Enterprise Architecture). When one of the client developers walked in, I said "Wow, it's a good thing you guys dismantling this mess" to which he replied "No, that's the new one we're going towards". Decisions like this mean I have to spend my time in long lines in places like rental car counters.

Going back to first principles is hard, and many companies think it's too late. I've been involved on projects where getting back to a simple solution takes years. But it's worth doing. Otherwise, the company will drown in self-inflicted complexity, when IT becomes a governor on the business rather than a strategic advantage. When that happens, some other company with nimble IT will eat their lunch. Software can be either an asset or a liability.

Monday, February 11, 2008

Missing Family Member John Glasgow

picture of john glasgow My wife Candy's cousin John Glasgow is missing. John was last seen leaving his Little Rock, AR home Monday morning around 5:30 am, January 28, 2008. A cell phone ping later that day indicated that he was in the vicinity of Petit Jean Mountain. His car was found unlocked the next day at the Mather lodge on Petit Jean with valuables still inside. He is believed to have been wearing a green Marmot down jacket and khaki pants at the time of his disappearance.

His family and friends are asking for help in locating John. We are hoping that someone out there will see John's picture and read his story and come forward with information that can help us find him. Distressingly, official search efforts were called off 2 weeks ago on Friday, even though the family has mounted their own search. Please go to to see a picture of John and print off a flyer. If you could put it in your car window or some other visible place, it would help us a lot. It is possible that he could have traveled out of the area where he went missing, so we are trying to get the word out on a national level, to cover all possible scenarios.

Sunday, February 10, 2008

The Real JDK 2.0

I always wondered what in the world the Sun marketing guys were thinking. They kept changing the real name of the JDK. Starting with JDK 1.2, they decided that they wanted to call it "Java 2", and the "Java 2 Platform". But developers still downloaded JDK 1.2. OK, that makes some sense, I guess. Then, when JDK 1.3 came out, it was still "Java 2 Platform". If there was any logic, it would now be the "Java 3 Platform". Well, OK, so now it's Java 2 forever. Then, when JDK 1.5 appeared, it became Java 5. Now they are clearly just making this stuff as they go. Now, the current thinking is that it's just the "Java Platform", and "Java Enterprise Edition" (to avoid the charming "JPEE" acronym). What in the world is going to have to happen for the real version number to trip up to 2.0. And why the reluctance? Is there something magical about version 2.0? Why didn't the marketing and technical guys just get together when JDK 1.2 came out and call it "Java 2" then? It's not like they're going to run out of numbers -- there are literally an infinite number of them!

Without realizing it, the technical guys didn't think (and apparently still don't) that Java was good enough for a real Version 2. They've been waiting...and now it's here. Groovy is the REAL Java 2. Henceforth, we can think of the Java language (not the platform) as forever 1.x. Groovy is what the real second version of Java should have been. The technical guys must have felt it their guts all along.

I've heard some rumors about wacky version numbers for Groovy. It leapt from 1.1 to 1.5. Stop it! When you make changes to it, increment the bloody first number! Unless you want someone in a decade writing a smarmy piece about how your technical marvel just got supplanted, and that it was a version 1.x technology all along.