There are many ways to go about usability testing and many new tools are appearing to aid with performing more effective tests.  Examples include UsabilityTesting.com where crowd-sourcing and video capture is employed and www.optimalworkshop.com where a set of tools are offered to gain insight into customer patterns.  But you can’t just go after these tools, skills are required which either means gaining them internally or hiring consultants.  A great blog post on the challenges of usability testing we published recently by Extractable on avoiding pitfalls.  If you are working on a Testing COE, consider the need for skills in usability as something that should be on the map and the path to skills or relationships you will use. 

Today SOASTA announced 3 new products in the mobile space that show a lot of promise.  With a Cloud delivered testing solution the company is able to get to market and evolve the product more readily than traditional software delivery which will be important in an area that sees constant evolution in platforms and devices.  The ability to deliver full automation without jailbreaking or tethering a device coupled with the high maintainability we have seen in SOASTA’s testing product will give users another solid option.  As devices shift to business critical application platforms the ability to have consistent test automation will be a key to success.

During AADI we held a roundtable on ALM Practices and had a fairly quick conversation on this topic.  The nut of the conversation is that no-one in IT uses the term ALM.  Either SDLC (software development life-cycle) or more specific terms: requirements management, project management, defect management, etc. are used.  There is a slowly emerging convergence of APM and PPM tools with ALM and a convergence between ALM and Release Management in DevOps that at some point may deliver a real “ALM” suite or solution set  but the reality at this point is we probably should modify our terminology to say ADLM.  I would love to get your input either here or tweet me @metamurph

Jim and I are also taking the time together to work on completing the Magic Quadrant for ALM.  This has been a challenging process over the last 6 months given the ever growing size of the market and the evolving nature of ALM.  One of the key things we have noted is that in general most tools aiming at enterprise use have gained strong integration facilities recognizing that they won’t be the single stack provider.  There are single provider solutions and a few vendors still oriented toward these solutions but in general most organizations won’t have all ALM tasks managed in one single tool.  We have noted this in two notes recently: Selection Criteria for Success in Choosing ALM Products and Application Life Cycle Management Matters Where Diversity Persists.  In the near term, it would appear that Cloud delivery of solutions will only better enable users to choose governance tools that fit the various roles in an organization and this is a good thing.

1. Scenarios under test: customer facing, App Store, location, motion
2. Breadth of devices: phones, tablets, carriers
3. Development technology: MEAP, native, cross platform
4. Load vs. Functionality
5. Skill set – devices will require more technical testers and more usability experts

There are a growing number of options based on these needs including crowdsourced testing with Utest, outsourced with companies like InfoStretch, tools that extend existing frameworks and stand-alone device testing solutions such as (DeviceAnywhere, eggPlant,  Jamo Solutions, Perfecto Mobile) but this market will continue to evolve, we expect acquisitions and general disruptive behavior for the next 3-5 years.

The case for testing is never one dimensional it must be built to the specific needs of the organization e.g. risk, regulation, agility, cost, can all be focus factors as well as understanding the types of issues being produced currently.  But seeing the cost to your ecommerce efforts of slow page response is quickly eye opening.

This however points out how tools may or may not fit for your company.  Open source tools evolve “naturally” without the rigor of planning and there are few open source planning facilities, and the tools themselves are often a patchwork that has to be carefully assembled by the user. 

As companies mature in their use of Agile and it moves from dark corners to mainstream development, open source will still be an important driver of innovation but I expect that commercial tools will adapt: more frequent product releases, published defect repositories, greater community involvement to shape directions and ways for the community to extend and often freemium type offerings.  These behaviors or the ability to effectively utilize them will separate traditional structured ALM from Agile ALM.