Ray Valdes

Yesterday, Google announced a new open-source programming language called Go, a systems implementation language that is a blend of Python and C++.

Thus far, the reaction from developers has been mixed. Most geeks are not so much agog, as perplexed: Why a new language? Why now?

For many, their initial reflex is to question or even scoff. This reaction comes primarily from those developers who have invested years in learning Java or C#, and more recently have strayed into Javascript, Ruby or Python. They look at the “Hello world” example in Go, read the summary description, and say “No thanks, I’ll stick with ___“.

However, there are a few, those with the strong strain of the curiosity gene, who’ve dived into the docs and downloaded the code, who hold a different view. They see the rationale behind Google’s move, but are mostly too busy to vent their views in the blogosphere.

They see a significant advance in language design, one that adroitly addresses a complex mix of requirements and tradeoffs, in order to reach the objective of a high-performance high-productivity development tool for concurrent systems programming. I have dipped into the system and do see the logic. At the same time, I think Go faces a huge adoption challenge. Any language, now matter how well-crafted, faces huge odds.

Before making pronouncements as to future outcomes, it needs to be made clear that this is an experimental language at its earliest stage of development. It is the result of a small project that began less than two years ago. Go does not represent a “long march” strategic initiative for Google (in the way that Google Wave does). Failure of this language won’t have much negative impact for Google.

Anyone introducing a new language must face a vast number of competitors for developer mindshare: niche languages that percolate under the surface of the mainstream, a mainstream that includes not just Java and C# but Perl, Python, PHP and Ruby. These are all well-established, visible alternatives, compared to what lies under the mainstream: languages like Squeak, Haskell, ML, Erlang, Lua, Alef, Limbo, Scratch, and, going further back, Dylan, Telescript, Liana, Oberon, Bliss,Modula, Kaleida, and many others. Some of these niche languages have a loyal following, but won’t become mainstream any more than Lisp, APL, or Smalltalk will grow beyond their present tiny communities.

So if it were anyone else, it would be easy to forecast the eventual outcome of Go as an interesting footnote in the history of programming.

However, there are two aspects that should give pause: the pedigree of the authors (Thompson, Pike and Griesemer), and the corporate entity behind them (Google). Thompson is one of original authors of Unix, and Pike is a long-time collaborator of his at Bell Labs, working on seminal (but niche) operating systems and languages like Plan 9 and Limbo. Griesemer was an author of the Java hotspot compiler and the superfast V8 Javascript compiler. Pike has been working on inventing languages for literally decades. The concurrent programming features in Go build on the foundational computer-science work of Tony Hoare and his CSP language from the 1970s.

Google is not the ivory tower of Bell Labs. Instead, Google is a vast, real-world, proving ground for the most intensive software development projects. Google has intensive needs at two ends of the spectrum:

• the requirement for high agility and programmer productivity, of the kind that results from a modern dynamic language
• the requirement for high-performance and scalability, including the ability to exploit multi-core architectures and concurrent processing

Historically, a common approach to software development at Google is to create a prototype in a high-productivity language like Python, one where a developer can iterate rapidly through the compile-run-debug cycle until the appropriate mix of functions and features is arrived at. Once the design is considered complete, the development team then undertakes the laborious step of re-casting their design in a high-performance, highly scalable system: in most cases C++ (in some cases, as in Google Wave, it is Java).

The grand experiment that Pike and colleagues have embarked upon is to see whether these two very divergent needs can be satisfied by a carefully designed and well crafted language.

The syntax has been designed for rapid compilation and linking. In a TechTalk at Google a couple of weeks ago, Pike demonstrated compiling and linking a math library of perhaps a couple of thousand lines of code in 200 milliseconds, and recompiling the entire Go system (about 120,000 lines of code) in about 8 seconds — all on a laptop with single-core. This is an order-of-magnitude faster than conventional systems.

The Go language is larger than C, but smaller than C++, and feels like a blend of Python, C, with echoes of Squeak, Alef and Limbo (earlier languages that Pike has been involved with).

This appears to be entirely a Google project. It is driven by the internal motors and motivation of Pike and colleagues, as well as by Google’s top-level requirement for agility-plus-performance. The BSD-style license is intended to foster diffusion of this infrastructure-level software, but I think Google cares more about exploiting this tool internally — as opposed to projecting it externally, as some kind of “lock-in” weapon in a protracted platform war.

If Google views this as something to market and project onto the outside world, I think they will struggle to get adoption — primarily because of the mountains of legacy code that are out there. Many Web developers embarking on a project hold their nose and program in PHP because of the large corpus of open-source components and solutions that are available in that language compared to others. Likewise those writing back-end systems in Java, C# or C++ think: Why reinvent the wheel when you can download it from SourceForge?

Even within Google’s own internal context, the Go language will have a challenge getting incorporated into the fabric of company software assets.

I think it will be 3 years before Go has a big impact within Google (assuming it does not wither on the vine). And there will likely be another 3 years after that, before any significant impact in the enterprise sector. Adoption will be driven from by third-parties building solutions, and by enterprises adopting Google App Engine (assuming GAE gets updated to support Go).

In the meantime, the Go source code makes for interesting reading on a cold winter night. It harkens back to the day when programmers read entire programs, similar to young writers learning their craft by reading long novels written by masters. The work of computer-science masters (Thompson, Kernighan, Ritchie, Dijstra, Knuth, Wirth, and so on) constitutes genuine literature, in contrast to more recent work which may be functional but is not so readable. I still have my printout of V7 of the Unix operating system, on the shelf next to A Portrait of the Artist As A Young Man. The design of Go is intended to encourage readability, unlike modern programs that result from developers using Intellisense and auto-completion to churn out endless streams of verbose text that may be syntactically correct but hard to savor.

What are your thoughts on language? In the dead of night, are all cats gray, and all languages nothing more than Turing-machine equivalents?

At the Adobe Max conference in LA today, Adobe announced a side-door way to get Flash applications onto the iPhone platform. This Flash-on-iPhone question has been a vexing issue for years, ever since the iPhone was first introduced, because both Apple and Adobe have strong loyal user communities with a high degree of overlap. At every gathering I have attended over the past two years, whether for iPhone developers or Flash designers, the question comes up: When will Flash run on the iPhone?

The faultline in the common ground shared by these two vendor’s ecosystems has remained a visible and ugly gap in the market landscape. I’m only an outside observer, but it appears that there is a backstory not without intrigue, twists and turns. A year ago, Adobe said that it had Flash running on the iPhone in the lab, and developers should ask Apple for the official details of when and why. Apple’s response was typically unspecific, but contained allusions to performance, battery life and memory issues.

Although I am not privy to details, it appears that the Flash-on-iPhone question goes beyond narrow-scope technical issues of the Flash VM and the iPhone operating environment, and goes back to broader interactions between these companies dating back to the 1990s. It spills over to issues of personalities and conflicting business agendas. For example, I gather some of this bad blood dates back to the days of the Next Machine (remember that bygone era?). If stories are to be believed, Adobe put a lot of pressure on Next to use Adobe’s implementation of Postscript for the on-screen rendering. Instead, Steve Jobs and his team pushed back hard with their own implementation of Display Postscript, which they felt was faster and better integrated into the environment. Today, Mac users rely on the Apple “Preview” application on OS/X to view PDF files, and I can say that it is certainly a lot faster and easier than the experience of firing up Adobe Acrobat on Windows. (However, I won’t venture into a guessing game on which element in the mix is to blame for latency: the hardware, operating system, or PDF reader software.)

In the past, I have observed senior managers at these two companies talk about the other company with dark frowns and firmly clenched teeth. After various shuffles in management team at both companies, there was hope that bygones would be bygones (also that perhaps people at Apple would remember that Flash originated at Macromedia, not at Adobe, and therefore should not be tarred with the same brush as the PDF engine). But nothing, until today.

Adobe’s announcement of Flash-on-iPhone is a pleasant surprise to those of us who had resigned themselves, like children of divorced parents, never to have a unified holiday gathering again. However, what Adobe has done is not get a full, welcomed, seat at the dining room table, but has instead snuck in through the side door, lurking in the kitchen with the catering staff. By that I mean that Flash is a second-class participant in the iPhone platform. SWF files that contain dynamic ActionScript cannot be loaded on the iPhone on-demand or embedded in Web pages. Instead, Flash programs must be pre-compiled by developers and packaged into binaries (boo!), which can then be distributed and monetized through the AppStore (Yay!). By the way, if you are interested in seeing some source code that works with iPhone, check out http://bit.ly/flash2iphone

In conjunction with this save-as-iPhone-app feature in the beta version of the upcoming Flash Pro CS5 tool, Adobe also announced Flash Player 10.1 for smartphones, which, when released in first half of 2010, will have improvements in memory consumption, battery use, hardware acceleration support, and input modes.

Adobe’s hope seems to be that as more Flash-related content surfaces on the iPhone platform, to the delight (presumably) of users, then Apple will relent and let the banished partner in from the kitchen to the dining room table. So this is only a first step. However, there is a genuine risk of adverse consequences. Just like an ex-spouse sneaking into the kitchen may not be viewed favorably by the host of the gathering, even if no rules are broken (which none are in this case, because Adobe complies with Apple’s rules of the house, and there are already a few Flash-origin apps in the AppStore today), it is possible that Apple will respond negatively to this move by Adobe. In an extreme hypothetical scenario, developers could find their apps yanked from the AppStore by a punitive Apple. Thus far, Apple has provided no reassurance.

Beyond any sudden sanctions from Apple, Flash developers risk disappointment in other ways: the Flash-on-iPhone approach has certain constraints that will prevent some apps from running unmodified (for example, no embedded HTML content, no dynamic SWFs, no dynamically loaded SWFs, no RTMPE, no PixelBender filters, no microphone or video camera access). Also, an existing Flash app might be assume certain operating environment and performance characteristics (for example, low latency network access or copious amounts of RAM) that are degraded on the iPhone platform. Lastly, a Flash application may have a look and feel that, despite being carefully crafted, might in some cases violate the voluminous Apple interface design guidelines, and would need some rework in order to be approved for AppStore distribution.

So this announcement is far from a game changer, but it does mean a step forward for developers — as long as those hatchets from past grudge matches don’t resurface.

This melodrama might make for interesting reading, but it should not detract from the impressive announcements Adobe made today, about a plethora of mobile devices, tools, and services — FlashPlayer 10.1, OpenScreen, Omniture Developer Sandbox, Cold Fusion 9, and so on.

According to various reports in the financial press today, there is a $100M investment in Twitter at a valuation of $1B. Over the past few months, an entertaining parlor game in certain circles has been the Twitter Valuation Game: how to value a social site that has no revenues and no business model. The beauty of this game is that anyone can play. It does not matter if you lack an MBA or other financial background, because everyone is on an equal footing in this virgin territory (at least for those who were too young to remember the dotcom bubble).

The reported valuation of $1B for Twitter is breath-taking. However, there is a certain logic to this, and it is not just froth. The starting point for any discussion of Twitter is its explosive growth rate and sheer number of users. There was a twelve month period in which Twitter grew from 3 million users to 30 million. Now it is over 40 million with a still-robust growth curve. Given the pesky lack of business model, one must acknowledge that this is still the “wishful thinking” phase. Having said that, however, the growth rate, user engagement, and the skills and background of Twitter management mean that a bet on its eventual monetization is probably not too risky. The question is how large a bet.

The high-water mark for social network valuation was Microsoft’s investment in Facebook in October 2007, which valued that company at $15B or about $250 per user. Five months later, AOL acquired second-tier site Bebo for the much lower figure of $21.50 per user. This lower figure is consistent with Twitter’s reported valuation this week, at $25 per user.

One can argue that, even at $21.50 per user, AOL overpaid for Bebo. Hindsight shows that AOL has not been able to effectively manage the potential of this property, which now has less than 1/20th the user population of Facebook. The latter has grown exponentially, which the former has dwindled, from a point in time where they were in the same ballpark.

The Facebook valuation set by Microsoft in 2007 is clearly an outlier, and Facebook’s perceived valuation has settled to the same planet as everyone else, orbiting at $30 to $35 per user. Facebook deserves a certain price premium for being cash-flow positive, for maintaining a robust growth trajectory, and for the sheer magnitude of the population number (within striking distance of 1/3 of a billion active users).

At this point in the eye-glazing discussion of per-user valuation, the impatient observer might again raise the pesky issue of Twitter’s lack of revenues. No matter what one thinks of Bebo, at least it had millions in revenue. Twitter fans will respond with the fact that it took Google several years of explosive growth before it discovered its dynamic ad-based business model.

The price premium that Bebo enjoyed, during the social-network mania of early 2008, could be termed the “I don’t know what the company I’m buying does but it sounds cool and everyone else is doing it” premium.

Twitter perhaps enjoys a bit of this premium, also some other premiums:

1. The “common parlance premium“: Twitter has become a verb (”I twittered this yesterday after I googled the info”). By contrast, a much smaller number of people say “I facebooked you, why didn’t you respond?”. Almost no one says “I yahooed yesterday” nor have I ever heard anyone say “I microsofted”.

2. The “celebrity premium“: Ashton, Kanye, Oprah, Sarah, Barack all Twitter. That is worth something to some observers, and perhaps to a star-struck investor.

3. The “real-time Web premium“: tapping into the big meme among geeks and digerati. Still somewhat niche-y, but likely has some effect.

Even when one discounts all those premiums as fashion-driven motivations that will fade, the fact remains that Twitters growth curve is remarkable and will continue, at least for the near term. (By the way, I gave a presentation last week in Madrid to a small auditorium within a large corporation, an estabished, conservative, company outside of the technology sector, and it turned out that about 60% of the audience had Twitter accounts.)

Bottom line is that a residual, premium-free valuation of $700 million is not illogical, and could go north of $1B if one layers the various price premiums noted above. What seems clear is that it would have taken a number much larger than $1B to get the Twitter founders to agree to an outright acquisition. Now that they are getting an infusion of cash, the outright-acquisition number and associated timeframe recede into the horizon but are still in the realm of possibility. We are, fortunately, still left with the remnants of the parlor-game question (”will Twitter ever be acquired, by whom and for how much?”), a distraction that will continue, at least until the day Twitter has measurable revenues, an event that will separate the MBAs from the boys.

Got some interesting press calls today about the FriendFeed acquisition. Obviously, everyone knows what Facebook is. Your grandma. My aunt in the old country. They all have Facebook accounts. Twitter is white-hot in some circles, perhaps not your grandma or my aunt. But daily fodder for those that know or care who Ashton Kutcher is (@aplusk with 3 million followers), Oprah (@OPrah with 2 million), or Snoop Dogg (@snoopdogg with 388k).

While Twitter is well-known to people in media, some reporters calling me for comment were not exactly sure what FriendFeed was about. I view FriendFeed as a site for social-media power users, and something of an acquired taste. I have observed debates among the social-media cognoscenti who discuss FriendFeed like wine connoisseurs would discuss an interesting vintage. Some think it’s the ultimate in social sites, while others find the user interface opaque, the discussion insular, and its attention requirements tedious. For many social-media mavens, FriendFeed was the latest destination in a personal circuit that began with Facebook 4 or 5 years ago, and Twitter two years ago, and a regular hangout for the past year. I got my FriendFeed account in October 2007 but must confess I only log on about once every few months. I guess my palate never acquired the taste for this flavor of social media aggregation.

Although FriendFeed has not experienced the explosive growth that Twitter enjoyed, its technology is arguably an order of magnitude better. I recall the days when Twitter crashed every single day for weeks on end. FriendFeed has not inflicted on its users the reliability, security, or performance problems that those on Twitter had to suffer. Nevertheless, Twitter users did not leave, because of the site’s simplicity, speed, and wide choice of client software (due to early introduction of an API for third-party developers) that allowed you to choose your user experience.

According to papers purloined from Twitter by a hacker last month and published on the Web, the team at Twitter was entertaining some serious offers from Google, Microsoft and Facebook. But the Twitter principals had an ambitious goal: to become the first social site to reach one billion users. This lofty goal probably resulted in a lofty price tag for any would-be acquirers. By contrast, FriendFeed’s growth curve is flat (see http://bit.ly/FF-vs-Twitter ) and presumably the price more down to earth.

Facebook is clearly not acquiring FriendFeed for its users. Given the power-user appeal, I suspect that 95% of FriendFeed users have Facebook accounts (and Twitter accounts). Facebook is also not acquiring FriendFeed for its raw technology (although there may be some interesting, patentable intellectual property at a level above the raw code). Facebook is written in PHP, and unlikely to make a wholesale change to a new platform. For the ex-Googlers who started FriendFeed, PHP is a lightweight scripting language to be looked upon with disdain (in preference to C++, Java and Python).

In my view, Facebook is acquiring FriendFeed for its talent pool (world-class developers who could learn PHP in a day if they wanted to), and more importantly for its sense of mission: to open up “walled garden” social sites and shift the world to the distributed social web, a web of interoperable sites that share data in real-time.

Over the past year, Facebook already has implemented real-time Twitter-like features, as well as FriendFeed-like interoperability and social architecture (the “like” social gesture, for example). This acquisition moves Facebook further in that direction and allows it to leapfrog Twitter in some respects.

One issue that has yet to play out is the reaction of FriendFeed community. Many of them are refugees from Facebook or Twitter, and today are feeling at best irritated and at worst apoplectic that their quiet corner has been discovered and owned.

Facebook faces a challenge if it seeks to integrate the FriendFeed site with its own. It is like trying to combine the dashboard of a Chevy Tahoe with that of a Ferrari. I think Facebook will leave this quiet corner alone, and instead will move the engineering talent quietly over to the Facebook R&D department to continue implementing the FriendFeed vision on the Facebook platform.

Everyone’s game (Twitter, Google, Microsoft, MySpace, SixApart) has to step up a level. More pairing off will occur. Twitter’s price just went up, perhaps along with their anxiety level and disposition to sell.

Perhaps prodded by a news story in today’s New York Times, Google announced, via a blog post a few hours ago, an operating system called Chrome OS.

Key points in the announcement:

• the OS is designed for Web apps and cloud computing
• it combines the Chrome browser plus a lightweight windowing environment on top of a Linux kernel
• goals are lightweight, fast to boot, fast to allow users to access the Web
• targeted to netbook category of devices initially (via hardware partners), but eventually for desktop machines
• will be released as open source
• relies on a security architecture designed for the Internet era
• it’s not Android
• discussions underway with hardware partners for shipment after mid-2010

If two dots allow one to draw a trend-line, the cloud-centric OS built on a browser can be considered a trend. The two large dots that define this trend-line are Chrome OS plus a Microsoft Research project called Gazelle, a browser-based operating system with an approach similar to Chrome OS. More about Gazelle in a moment.

Based on the limited information that Google has provided thus far, it appears that Chrome OS leverages Google’s core competencies, which include ability to design user experiences that are simple, effective and fast, backed by “rocket-science” software technology under hood. Google’s approach is to first narrow the scope of the problem in order to put more substance and depth into the remainder.

There is some substantive information upon which to make inferences if one looks at the open-source version of the Chrome browser, available at Chromium.org.

The current Chrome browser consists of 1.7 million lines of C++ code, and already incorporates many OS-like aspects (multi-processing, robust isolation). About 60% of the Chrome browser source code is related to rendering HTML, and much of the rest (about 700,000 lines of code) implements OS-like aspects such as multiple process models, interprocess communication (IPC) secure sandbox isolation mechanism, and so on. This system could rest on a foundation of a Linux kernel. The major new piece is the unspecified windowing environment, which presumably would mostly be a pass-through mechanism. Speculating wildly here, I would say the combination of all of the above would result in a system of about 5 to 7 million lines of code, including a stripped-down Linux kernel. This is about the same size as Windows NT 3.1, and about one-tenth the code size of Windows Vista. This is smaller than Google Android, which is variously estimated at between 8M to 11M lines of code. The size estimates perhaps answer the question “Why not use Android?”. That is, Google is betting on a lean and fast browser-based OS rather than one that is built for comfort across a wide range of scenarios.

One might also ask: “Why a browser-based OS? Why is this worthwhile in a time when users can simply procure the combination of Linux, Firefox and OpenOffice on a modest laptop?”. A paper on Chrome’s Multi-Process Architecture provides an answer:

“The current state of web browsers is like that of the single-user, co-operatively multi-tasked operating systems of past. As a misbehaving application in such an operating system could take down the entire system, so can a misbehaving web page in a modern web browser….Modern operating systems are more robust because they put applications into separate processes that are walled off from one another…. We use separate processes for browser tabs to protect the overall application from bugs and glitches in the rendering engine. We also restrict access from each rendering engine process to others and to the rest of the system. In some ways, this brings to web browsing the benefits that memory protection and access control brought to operating systems.”

This text is accompanied by the following diagram:

The multi-process approach is not unique to the Chrome browser. Recent versions of Internet Explorer have something similar. What is different is that this multi-process approach is extended to include the entire machine environment. A crisp explanation comes from Helen Wang, a member of Microsoft Research Labs working on Gazelle, a lightweight, browser-centric OS prototype:

“Everyone accepts that applications need to run on operating systems. However, this has not been the case for Web applications; they depend on browsers to render pages and handle computing resources. Yet browsers have never been constructed to be operating systems. Principals are allowed to coexist within the same process or protection domain, and resource management is largely non-existent.”

Microsoft writer Janie Chang elaborates:

“In the Gazelle model, the browser-based OS, typically called the browser kernel, protects principals from one another and from the host machine by exclusively managing access to computer resources, enforcing policies, handling inter-principal communications, and providing consistent, systematic access to computing devices.”

When Google Wave was introduced in May, I noted how Microsoft Research had seen prototypes of Office that supported real-time collaborative editing in 2003 but never moved forward on this. Now perhaps, this ironic pattern is repeating, assuming that Chrome OS actually sees the light of day.

If Google delivers on its plan, it seems that Chrome OS will be the first cloud-oriented OS to ship. This will be a consumer-oriented offering initially, similar to Google’s past practice in other categories (Web maps, Web email, and the Chrome browser). It will be years (three to five) before it has any impact on the enterprise sector.

For Google to succeed with this initiative, Chrome OS must deliver a user experience that is perceptibly better from the outset. Google was able to achieve this with Google Maps, which reinvented online mapping in a way that users immediately noticed a better user experience. Google has arguably also achieved this objective with Gmail and with the Chrome browser. The question is whether they can achieve a similar goal with an OS — a clearly envisioned target that they will reach for, but one that may prove difficult for them to grasp and hold.

What do you think?

Apologies for the provocative headline, but I just couldn’t resist. Actually, there is no single secret sauce, but rather multiple key ingredients of varying degrees of obscurity.

Google Wave is a fascinating melange of audacity and hope. Google has blended multiple, powerful, sauces into this melange, some secret and other less so. One in particular that was quite obscure to me is operational transformation (OT) theory, which is the algorithmic framework that enables multiple people to edit a single document in real-time across a wide-area network with unpredictable latency. More about this in a moment.

As I stated in last week’s blog entry, written minutes after the keynote ended at the Google I/O conference, Wave has surprisingly broad scope, cutting across formerly separate application categories like email, blogging, wikis and instant messaging. My initial reaction was colored by instinctive reflex of cynicism, and basically amounted to: Yes, it’s very cool and innovative, but what has Google done for the enterprise lately?

After a healthy debate with Gartner colleagues, spanning a range of views pro and con, I reviewed the Wave video and the documentation, and felt greater excitement than I did during the keynote (where I was one of the few sitting down during the standing ovation). I won’t use this post to make one of those forecasts, such as “Google Wave will kill X”, where X can be any number of well-known vendors or products. That kind of statement is overly glib, because we are just a few days into a scenario that will take 5 years or more to play out, with many twists and turns along the way.

Instead, I want to first draw your attention to some of the enabling technologies that represent the key ingredients behind Wave, and then talk about OT.

Key enabling technologies include:

• GWT Ajax library, which not only powers complex interactions on a browser, but also provides a reasonably tuned rendering for small form-factor mobile devices, such as Android and iPhone
• XMPP protocol that provides a foundation for the Wave Federation Protocol
• the real-time keystroke-by-keystroke communication pioneered several years ago in Google Suggest, and now in production in high-scalability deployment
• the AppEngine cloud computing platform on which are hosted the Robot extensions to Wave
• the Big Table data persistence mechanism that powers Google’s implementation of a Wave server (something which is not required by the spec, but which facilitates developer productivity and production scalability)

Note that these are only enabling technologies, and Wave is a large collection of software built using these enabling technologies by a hundred-person team over the past two years. That system is currently in closed beta, so the best we can do is circle around it and measure where we can.

All of the above items have been discussed over the past months or years in the industry circles, so none of these qualifies as a “secret sauce”. Imho, if there is a secret sauce, it is Google’s extensions to operational transformation theory (OT).

OT is an area of computer science that spans decades, but is nevertheless obscure. The name is derived from the on-the-fly transformation of operations (in the case of a word processing document, the operations would be things like inserting or deleting text) to enable multiple independent reads and writes without locking. One phrase comes up again and again in the literature, is that this is an “optimistic” approach — not in the everyday sense of audacity and hope, but in a technical sense where each parallel repository of content takes its own path to an eventually consistent state.

So this Saturday evening, instead of going out to the movies, I decided to entertain my curiosity by looking further into OT. Here are some tidbits I found while scanning an area that is new to me. It is possible I have missed major landmarks and garbled some of the concepts in this rich and fascinating area. Like a tourist in an unfamiliar realm, I am enjoying the sights even though their significance might not be fully apparent.

Here is a simple illustrative sketch of OP, from well-known researcher Dr Chengzhen Sun, that hopefully will become more clear by the end of this blog post:

Seminal work that preceded OT was done by Leslie Lamport, one of the giants in computer science, in 1978 in a classic paper called “Time, Clocks and the Ordering of Events in a Distributed System”. This paper presented a fundamental algorithm for distributed computing based on distributed state machines, time stamps, and vector clocks. Lamport later said that his algorithm was based on his understanding of Special Relativity, which “teaches us that there is no invariant total ordering of events in space-time; different observers can disagree about which of two events happened first. There is only a partial order in which an event e1 precedes an event e2 iff e1 can causally affect e2.”

In an interesting side note, Lamport works for Microsoft Research these days.

Here is a simplified version of Lamport’s algorithm for distributed mutual exclusion, from a paper by RR Hoogerwoord in 2002.

Lamport’s work is generally applicable across many aspects of computer science, but is referenced by researchers in the field known as group editing. A pioneering work in collaborative authoring systems was by C.A. Ellis and S.E. Gibbs in 1989 on “Concurrency Control in Groupware Systems”. . There is a whole community of researchers that have been working in this field, but Dr. Clarence Ellis is one of pioneers. In an interesting side note, Ellis was the first African-American to get a PhD in Computer Science in 1969 (see his photo from that era).

Later on, in 1998, there was an important paper by Chengzheng Sun and Clarence Ellis on “Operational transformation in real-time group editors” This paper was followed by further work by Chengzheng Sun at Griffith University in Australia. (Is it a coincidence that the Google Wave team is based there?).

One of Dr Sun’s projects is CoWord, an attempt to retrofit Microsoft Word with real-time collaborative editing features (like those demonstrated by Google): CoWord seeks

“to apply the state-of-the-art collaborative editing technologies to widely used single-user commercial word processors in a transparent way, i.e., without modifying the source code of single-user applications. MS Word has been chosen as the first target of this application. A collaborative Word (named as CoWord) is being built in our lab, which not only retains existing MS Word features, but also includes new features that enable multiple users to perform and undo MS Word editing operations on the same MS Word document concurrently and consistently.”

Interestingly, Sun presented his work in a lecture to Microsoft Research in 2003. (Also worth noting is that Sun recently moved to Nanyang Technological University at Singapore.)

The above are only a couple of representative pieces in a swath of research in this field. My understanding is that Google has built on this work and come up with their own innovations, which I know about only as a distant observer. One of those innovations, I understand, has to do with maintaining a representation of a Wave as a tree structure in the browser, and keeping this structure in synch with the server version and that on other machines. If that is the case, I ran across a patent issued in 2005 that seems to be directly relevant: “Browser to browser, DOM-based, peer-to-peer communication with delta synchronization”:

“The present invention solves the complexity of integration of Web browsing, Web authoring and Instant Messaging, and offers the uniqueness of browser-based rich media manipulation and synchronization of Web contents among participating peers. The advantages are exemplified in the combination of the three…”

I’ll stop here to let the patent lawyers fire up their engines.

One last, possibly unrelated landmark, in my Saturday night tour of OT, is a variant approach which seeks the same goal as OT, to maintain consistency of shared data as it is manipulated by distributed authors in real time. The authors have come up with an approach that does not have to use vector clocks and is supposedly simpler and more efficient than classic OT. Because this approach is “With Out Operational Transform“, the authors chose the charming name of WOOT.

Updated (6/5/09): Corrected a typo (pointed out by Shantanu in comments).

At the Google I/O conference in San Francisco, Google just finished a demo of Wave, an early preview of a product that they hope will change how people collaborate on the Web.

Lots of people, including my colleagues at Gartner, likely will have an opinion on Google Wave, because of its breadth. As the saying goes, it is both a floor wax and a dessert topping. Google says “a wave is equal parts conversation and document”, but that is too terse. Wave’s functionality is surprisingly broad, and spans email, instant messaging, discussion forums, wiki, web publishing — and even eforms and a platform for social gaming. If you are not familiar with Wave, Tim O’Reilly has published a good overview.

My main takeaway is that Wave is innovative and has great potential, but obviously in the earliest stages of its life cycle (pre-Beta). Therefore it is premature to talk about things like whether it will kill Lotus Notes or Exchange. Enterprises have accumulated archeological layers of software infrastructure through years of investment, and that technology is solidified by deep-rooted behavior patterns. Applications have worn deep grooves into the craniums of many users, and it will be hard to shift them off those familiar tracks.

In no particular order, here are some factual tidbits and spare-change observations coming out of today’s session:

• A key aspect of Wave is the distributed nature of processing, a highly federated model that likely requires fiendishly clever algorithms for multi-way concurrent interaction, synchronization, and conflict resolution. Other systems such as Lotus Notes and Microsoft Groovy view these distributed computing algorithms as a key part of the system value, and it appears that Google goes further than these approaches, at a finer-grain and with an extensibility mechanism. The message from Google is that Wave is equal parts product, platform and protocol.
• Even though open-source is a key part of Google’s strategy, the current server-side implementation relies on BigTable, which would makes proliferation in enterprises a challenge. It is possible that Google might come up with a reference implementation on a LAMP stack, but that is something that is further in the future.
• Wave validates Google Web Toolkit as a choice for developers considering Ajax toolkits. GWT is one of many offerings in a crowded space that includes jQuery, Dojo, Prototype, Yahoo UI. GWT was conspicuous in that no major Google properties (Maps, Mail) used it even though Google was the company that put Ajax on the map. GMail and Maps were built out of hand-coded. Even though Wave originated with the same technical team that built maps, they chose GWT for the Wave project back in 2007, which could have been considered a risky bet at the time.
• Wave has the benefit of learning from experience with email and spam, and incorporates mechnaisms that would enable strong identity and authentication compared to what is in SMTP.
• Wave sets a new target for browser deployments, in that presents an incentive for users to upgrade to an HTM5-capable browser, for a user experience that includes real-time interaction on a keystroke-by-keystroke baseis.
• There are various extensibility mechanisms and avenues for third-party developers to add value, such as robots (server-side modules that function as virtual participants in converations) and gadgets (client-side views). These have a reasonable possibility of gaining traction among developers, similar to the many uses of Google Maps.

A debate that was taking place last month in blogs and in cafes around San Francisco’s South Park neighborhood has spilled over into the mainstream, with an article by Miguel Helft in today’s New York Times.

The article is about two differing perspectives on the design process, one exemplified by Doug Bowman, a visual designer that recently left Google with a loud exit, and the other by Marissa Meyer, the long-standing VP of User Experience at Google. The issues relate to a dichotomy that I have been presenting on lately, which I term the dichotomy between intuition and evidence.

Most Web sites and applications are still built with intuition-driven design. This is design that results from the subjective perception, thought, and imagination of a single individual or small team. If the people involved have talent, the result might be a breakthrough design such as the Apple iPhone, Google Maps, and, going further back, Lotus 1-2-3, MacPaint, and HyperCard. Intuition-driven designs don’t necessarily have to be breakthrough landmarks, however. Many designs are understated, cohesive and compelling, such as Flickr, Twitter, and my recent discovery, the Scribble paint program for Mac OS/X.

Despite some inspiring landmark examples, I find that, in too many cases. intuition fails to meet the needs of both the user and the organization. The results depend on who is doing the designing. There is a spectrum of dismal results, some inflicted by designers, others by developers.

On the one hand, the classic examples of designer-created excesses are found in the dotcom era, such as early Razorfish designs or the Boo.com site before it went out of business: small fonts that are unreadable, striking colors that overwhelm content, pixel-precision layouts that only work on the designer’s browser and large-screen display. Illustrating the other side of the spectrum, of dismal designs that come from developers, are many Web sites I ran into during the early days of Ajax or Javascript, pages that showcase a cool visual effect that unfortunately derails the user from ordering a product – not to mention that it only works on one browser, and has a memory leak that results in a crash if left running for any significant time.

Looking at the enterprise setting, it appears that development staff have big backlogs and not much time, so their reach is not as long. Unfortunately, the grasp is often not there either. The user experience of the average corporate application is too often tone-deaf, opaque, and oblivious to user needs. When was the last time you encountered an internal corporate app with a user experience that was less than painful? For many corporate applications, even a mediocre user experience would be an improvement over the current dismal state of affairs.

Countering intuition-driven design is a process with a systematic reliance on objectively observed user behavior. I have been using the term “evidence-based” design to refer to this approach, which others call data-driven or analytics-driven — all synonyms for the same approach. With this approach, design becomes less of an art and more of a science backed by engineering-style methods: “measure first, then fix”. Google is famously driven by the empirical approach, to the extent that they will run tests to determine which of 51 shades of blue to use for a given page. This extreme approach led Douglas Bowman, to resign, and to document his frustration as a creative professional trying to function in a setting dominated by empiricists. While his criticisms are valid if you are at the Google end of the spectrum, most organizations and development teams are far from the Google-style obsession with reality. Most enterprise development teams would benefit from a strong dose of empiricism, and a strong connection with user-centered reality. The results may not be pretty, but they are more likely to be effective.

Which side of this dichotomy do you find yourself on? Is it working for you, or will you push the pendulum in the other direction?

On the one-year anniversary of Google AppEngine (GAE), Google has released, in preview form, support for Java. This adds to the Python support already in place.

Last year, when the Python version GAE was first announced, it was open to the first 10,000 developers starting at 9 pm Pacific time. I signed up about 5 minutes past and was one of the initial contingent. However, I must admit that, as an old C programmer from way back, I was initially resistant to learning both Python and other, unfamiliar, aspects of the GAE platform, such as the Django framework and BigTable data persistence. Over decades of software development, I had found curly brackets to be a reassuringly familiar element when moving from C to C++, Java, C#, and most recently, Objective-C. Over the years, I had occasional fun experimenting with different styles of indentation allowed by the free-wheeling lexical rules of C and its descendants.

Python, by contrast, has no brackets, and enforces whitespace and indentation in an almost-pedantic manner. Nevertheless, to me, the GAE platform was intriguing enough to outweigh stylistic allergies.

For developers, the GAE value proposition is that you can write and test code on a local machine, and upload it to Google’s servers. Then, from that point on, it is “fire and forget”, i.e., your program can be used by one person or by a million users, and may store a single data record or a million records — all without further intervention on your part (other than to write a check to Google at the end of the month for usage). No need for sysadmin or DBA. No impedance mismatch between objects in memory and persisted data on disk, etc. (Note that I am only describing the vision, and that the reality is that there are significant operational and architectural constraints, mentioned further below).

Despite my initial misgivings, over the past year I have found Python to be an enjoyable programming language to work in, and the AppEngine platform to offer a similar, minimalist, attraction. No big honking IDE is necessary. Just write code with a text editor and upload. Also, I found Python used in unexpected places, such as the Maya 3D modeling program, or on the Symbian mobile platform. Although I will never spend enough time coding Python to become a pythonista, and my code is more C-like than it is pythonic, it has nevertheless been an enjoyable pastime.

My use of the AppEngine platform has been for hobbyist-style exploration rather than any kind of serious development. After all, writing code is not part of the analyst job description. Therefore, I have not run into the operational issues that more serious developers have encountered: the 10-second timeout on requests, the 1000 record limit on query results, 1 megabyte limit on responses, 1000-item limit for data offsets, and so on. Although many of these operational constraints have been relaxed, others remain, along with architectural constraints. (I have a research note underway, coming soon, that details these constraints and provides recommendations.)

Many developers have been standing outside the GAE platform, reluctant to step inside because of lack of familiarity with Python. Numbers have been clamoring for Java support in GAE (even while others seek Ruby or PHP).

Google has added Java support, in my opinion, for several reasons. First, Google has a strong organizational commitment to Java (used in GWT and Android, including the Dalvik JVM). Second, Java is a language that has a long history of operating in sand-boxed environments where security is a concern, an essential part of cloud computing. And lastly, Java is not .NET.

There are perhaps 4 million Java developers out in the world, and some large number work in enterprise settings. It is logical to expect a percentage of these to come knocking on the doors of GAE. They will find that many of the architectural constraints in the Python version of GAE have carried over to the Java version (despite a few pleasant surprises, such as support for Groovy). If you are expecting to take a large existing program that depends on hand-tuned SQL, uses multi-threading and EJB, and migrate this over to GAE, you will be disappointed. On the other hand, if you start small and plan to grow a system over time, Java on GAE might just be, excuse the metaphor, your cup of tea.

Today, Apple demonstrated a host of improvements to the iPhone software platform, in the form of a beta version of iPhone OS V3.0 and accompanying SDK, which Apple says has one-thousand added APIs.

Many of the features shown at the launch event were long-promised, long-awaited and much needed: cut-and-paste, push notification to apps, Bluetooth access for apps, horizontal type, etc. In the past, developers had to work around these limitations (lack of cut-and-paste) in various ways, and now can dispense with these workarounds and enjoy a straight path to building solutions.

I have been on a plane today and haven’t had a chance to look over all the details, but a couple of nice surprises stand out, in particular the in-game or within-app payment mechanism that supports a broader range of business models (gamers can purchase new levels without leaving the game, apps can deliver subscription-based content, etc). There were some interesting segments in the launch demo: for example, Oracle showing the push-notification feature added to their enterprise mobile apps. For me, the highlight of the session was the consumer health scenario presented by Lifescan (a Johnson & Johnson company) showing a young diabetic patient with a glucose meter connected to iPhone. Others may have found this section boring in contrast to the 3D game shown a few minutes earlier, but I found it to be a comprehensive scenario with undeniable utility and value.

The bottom line is that the thousand APIs added to the SDK will hopefully bring to a close the era of one-trick novelty apps such as the 50 different flatulence noisemakers. We can bury these in the same landfill as the pet rock.

The iPhone 3.0 OS is, of course, software only. It is logical to expect that Apple will introduce a new device that moves the hardware portion of the platform forward within the next few months, to deliver other long-awaited and much-needed enhancements. An obvious priority is a higher-resolution camera with video.

The mobile platform race is getting pretty heated, with major players and broad initiatives: Nokia/Ovi, Microsoft, Blackberry, Android, and recently Palm. Too many platforms, too many choices for developers. The space will consolidate, but not for a while. We are still in a period of growth as the global transition to high-bandwidth, high-resolution, high-touch mobile computing continues. Apple remains in the leading position, at least in terms of mind-share, as the mobile platform leader. Others need to formulate effective counter-strategies and execute these plans quickly.