Thomas Murphy » Testing

Discovering Usability

Thomas Murphy — Tue, 24 Jan 2012 00:03:47 +0000

An element of software quality often overlooked is the issue of usability. With internal applications you train users on the process to follow and this may make up for a world of ugly. The web has changed our perceptions and devices and Apps will continue to drive home the need to deliver applications that are intuitive and efficient. However, usability isn’t just something we ignore, it is something for which most companies are ill equipped to perform. Part of this is skill and part of this is tools.

There are many ways to go about usability testing and many new tools are appearing to aid with performing more effective tests. Examples include UsabilityTesting.com where crowd-sourcing and video capture is employed and www.optimalworkshop.com where a set of tools are offered to gain insight into customer patterns. But you can’t just go after these tools, skills are required which either means gaining them internally or hiring consultants. A great blog post on the challenges of usability testing we published recently by Extractable on avoiding pitfalls. If you are working on a Testing COE, consider the need for skills in usability as something that should be on the map and the path to skills or relationships you will use.

The Year of Mobile Testing

Thomas Murphy — Mon, 23 Jan 2012 22:02:00 +0000

The explosion of devices and apps is creating a strain on testing organizations. A broad number of tools have hit the market and we expect to see strong growth in the number of options during the next three years. A plethora of challenges exist: gestures, geolocation, motion, and how to realistically conduct load tests to name a few. Many will rely on manual tests and a prayer.

Today SOASTA announced 3 new products in the mobile space that show a lot of promise. With a Cloud delivered testing solution the company is able to get to market and evolve the product more readily than traditional software delivery which will be important in an area that sees constant evolution in platforms and devices. The ability to deliver full automation without jailbreaking or tethering a device coupled with the high maintainability we have seen in SOASTA’s testing product will give users another solid option. As devices shift to business critical application platforms the ability to have consistent test automation will be a key to success.

Mobile Testing Tools – It is a Tactical Decision

Thomas Murphy — Fri, 24 Jun 2011 00:26:05 +0000

The rapid movement to build and deploy applications to devices is creating a challenge for testing teams. Solutions from the leading testing vendors are missing, or provided by partner extensions, and the pace of technology evolution will keep them chasing for several more years. Organizations should recognize that testing choices (tools, services, etc) will be tactical during this time period. During this time consider the following:

Scenarios under test: customer facing, App Store, location, motion
Breadth of devices: phones, tablets, carriers
Development technology: MEAP, native, cross platform
Load vs. Functionality
Skill set – devices will require more technical testers and more usability experts

There are a growing number of options based on these needs including crowdsourced testing with Utest, outsourced with companies like InfoStretch, tools that extend existing frameworks and stand-alone device testing solutions such as (DeviceAnywhere, eggPlant, Jamo Solutions, Perfecto Mobile) but this market will continue to evolve, we expect acquisitions and general disruptive behavior for the next 3-5 years.

An Economic Foundation to Performance Testing

Thomas Murphy — Thu, 07 Apr 2011 05:14:04 +0000

Companies often wonder about the economic value of testing. For functional testing this generally boils down to defect containment and reducing the cost of fixing defects by finding them earlier but putting performance in economic terms has been more of a challenge. However we continue to see high numbers of sites with performance failures and often during critical times especially in the e-commerce arena. Recently Strangeloop (a site optimization company) built an interesting infographic pointing to the growing performance problem and putting it in economic terms http://bit.ly/e4WCxT

The case for testing is never one dimensional it must be built to the specific needs of the organization e.g. risk, regulation, agility, cost, can all be focus factors as well as understanding the types of issues being produced currently. But seeing the cost to your ecommerce efforts of slow page response is quickly eye opening.

The Apple Testing Problem

Thomas Murphy — Thu, 24 Feb 2011 04:15:00 +0000

As we worked through the MQ for Integrated Software Quality Suites one of the key issues we are seeing is that tools are lagging behind the pace of innovation and are often lacking support for platforms and technologies that are becoming popular leaving organizations to perform a lot of manual testing. Key in these areas are lack of support for Apple and its Safari browser (best option is TestPlant) and generally poor support for devices (one good option is InfoStretch). With the continued fast pace of change in client technologies we expect this disparity to continue and most organizations will continue to see a need to have testing tools from multiple vendors.

Magic Quadrant for Integrated Software Quality Suites

Thomas Murphy — Tue, 01 Feb 2011 02:02:19 +0000

We published the updated MQ for Integrated Software Quality Suites today (see http://bit.ly/hd4reJ ) after several months of work with vendors, users, and community review. This document focuses on vendors who provide (or are working to provide) a complete set of tools for primarily driving the software testing process including test case management, defect management, functional automation, and load/stress testing. The testing tool market continues to be dynamic with many new entrants each year. Most of these are new solutions for a specific area: functional automation, device testing, bug tracking but there are also new players entering the market each year going after more comprehensive solutions.

Software testing continues to be hard, costly, and embattled to explain why it is both of these. It is hard to measure the ROI of testing. However, we will next turn our attention to the ALM market which plays a key element into Quality because “Quality can’t be tested in” some of the players in that document will match this but just as their is growth in quality and testing tools, the ALM market is exploding and our biggest challenge will be focusing in on a reasonable subset of these products. It is great that options are expanding, this means better pricing, improving functionality, and support for a wider variety of technologies. It also means that a decision isn’t as simple as picking from a grid. So if you read the research and have questions about how to apply it to your needs, give me a call.

New Year 2011 AD Reading List

Thomas Murphy — Fri, 07 Jan 2011 05:51:40 +0000

A few relatively new books have been filling my reading time lately and they offer great content on how to improve the development process and team especially if you are making a shift towards Agile:

Continuous Delivery – this is a great new book on what it really takes to make Continuous Integration work and beyond. If you want to go faster, you have to automate and remove friction from the cycles and this book helps explain how.

The RSpec Book – Solid text that introduces the concepts of Behavior Driven Development using RSpec and Cucumber and drives a refocus of unit testing as a design method – a method designed to create well crafted code and to drive productive interaction between users, developers, and testers in a virtuous cycle of acceptance criteria and unit tests.

Agile Testing – There are many Agile books out there but they are about developer practices, project management and sometimes requirements. Testers have been left hanging in the wind. This book helps define terminology, practices and the shifts a team will go through in the transition from waterfall to agile.

Factors Behind Improving Testing Efficiency

Thomas Murphy — Wed, 29 Dec 2010 06:28:27 +0000

During his GTAC 2010 talk, Robert Binder covers a lot of interesting material. He ranges through white box and black box tests, the value of automation and movement toward model-based testing paradigms. One of the most interesting elements comes near the end of the talk, around 59 minutes in when we presents a chart on the factors behind testability and who controls those factors (see slide 38 – I suggest downloading to view as Google Docs PPT rendering is poor). If testing productivity is low, it is generally an issue outside the control of the tester as their overall performance is controlled by the work of the architects and developers: size and complexity of the system, structure of the code, testability is driven in before the tester gets his or her hands on the system. If you want to increase testing success therefore, don’t just look at the testers but pay close attention to the guys up-stream.

2011 Expect Open Source Testing Progress

Thomas Murphy — Wed, 22 Dec 2010 01:04:43 +0000

We are seeing greater uptake of open source testing frameworks. This is happening in software companies (as underpinnings of commercial products), at end user companies (to enhance overall toolset), and in services companies (who are using open source as a foundation for value ad products and services). A key driver for the increased use is the inability of commercial tools to meet the needs of all users and keep up with the rapid pace of technology innovation. Browser not supported? Write the extension yourself. AJAX widget sets, devices, cloud platforms and applications all are areas where open source provides a foundation that can be exploited and extended to meet needs faster than commercial products. On their own the products are still much more basic than commercial tools (see Hype Cycle for Open-source Software, 2010) but they provide a solid foundation that will impact that tools market during the next 5 years.

Technology Running Faster than Testing

Thomas Murphy — Tue, 21 Dec 2010 02:33:15 +0000

As the number of mobile devices raced past the number of traditional computers the challenges for testing organizations continue to pile up. In many instance tools (especially those of market leaders) is lagging behind market needs. Support for Chrome and Safari, the ability to drive load tests (or functional tests) from a device (rather than an emulator), and a lack of ability to work with the variable connection speeds will leave users scrambling for alternate tools, working with less robust open source products, and piecing together products from multiple vendors to create a “complete solution” for at least the next year. The shift to cloud, composite, and mobile will drive m&a activity but currently presents a murky market.