Of course, the governor on software development insinuates itself into all technical development, from chip design to manufacturing, to higher-level embedded applications, to software development tools – not just to enterprise apps, consumer apps and web sites. This is a HUGE issue your bring up.

Perhaps it gets down to that term “creative process” because usually we have a “cat chasing its tail” syndrome of developer trying to anticipate what the user wants/needs, which of course the user doesn’t really know either, so we get stuck in step-by-step cycles of mildly creative ping-pong, which often loses momentum and ends up in a quagmire. No commercial tool or technique I have seen directly addresses this syndrome. A tool and/or technique that breaks the barriers in that process has been a long-time coming.

Most likely the answer(s) remain buried in the pure research that happens at places like MIT, Stanford, Carnegie-Mellon and some of the commercial vendors labs (so HP, where I work, IBM, MSFT, etc.). If there are answers down there the technology transfer has not yet made it to the development marketplace. Probably the last three major movements in software development, from the most recent, let’s say SOA, then Java/OOP, and before that visual development (VB, Delphi and such) were all primarily focused on coding. The groundwork at OMG around MDA/development holds a piece to changing that “user requirements definition” puzzle. But largely we remain stuck in that user/developer process, whether tactically together or via “throw it over the wall and see what sticks” method. Crowdsourcing has to be part of the new app dev tool to address this old conundrum.

Some start-up some day will change all of this, someone will break the code of how to engage users/developers into a massive and breakthrough fast creative landslide of requirements definition, and further how to turn that into code. Hope I get invited to family and friends of that start-up. Until then, the developer community will continue to slog through understanding and predicting what users want. Perhaps most ironically, the deep research on social computing and user preference (which is built on generations of technology) holds the key to the lock on user requirements. Developers need to find a way to get inside of users’ minds, and unlock the creativity therein. The genius developer understands that the really genius resides in the users.