Why you should use crowdsourced testing

Back in May, Creative Bloq ran an article about how web experts manage cross-platform testing. The developers and designers consulted employed a variety of methods for ensuring their software functioned correctly across multiple desktop and mobile platforms, and the combination of approaches used depended largely on the organisations in which the experts worked.

However, an important point of consensus emerged between virtually everyone who contributed to the piece: that testing software on real devices is the only way to guarantee functionality.

While many of the experts advocated using emulators, they emphasised that these need to be used in conjunction with actual Smartphones or tablets. But it was also acknowledged that developers are often significantly limited in terms of access to devices on which to test. For development teams without adequate in-house resources, performing testing at an Open Device Lab was recommended.

However, while Open Device Labs are a great innovation that provides much needed resources to developers working on a tight budget, they too offer limited device coverage and time in which to test.

Some try to resolve the problem of device coverage by prioritising platforms – that is, testing on a small number of the devices, browsers and operating systems most popular with their users. However, this is becoming less and less of an option, particularly for anyone developing customer-facing sites and apps – recent research has shown that in Q3 of 2014, UK consumers accessed the web via 5,000 unique mobile devices, a 23 per cent increase on the previous quarter.

This means that developers testing on only a narrow range of devices may begin to find that a growing number of their users are encountering bugs on their software.

Crowdsourced software testing, however, can enable developers to have software tested across hundreds of unique real devices. A relatively recent innovation in how software testing is approached, the crowdsourcing model, used by companies such as bugfinders.com, pulls together thousands of professional testers from all over the world and deploys them to test software on the devices, browsers and operating systems they use every day.

The extensive size of most crowdsourced testing communities means it is not uncommon for sites and apps to be tested on as many as 300 unique devices, allowing developers to ensure their software works for the vast majority of their user base. And because testing is performed by testers working remotely, developers and their teams don’t need to spend long hours testing software themselves.

So instead of, say, a team of four developers spending four hours at an Open Device Lab testing on eight unique devices, that same team could get on with development work on other projects while their software was tested on 20 times that number by a crowdsourced team.

The crowdsourcing model can also complete testing within short timeframes. In part, this is down to the fact that hundreds of testers can be deployed to test a piece of software simultaneously.

But it also results from the international nature of crowdsourced testing communities, which are not subject to the time constraints of office hours. Because testers exist in different time zones, they are able to work overnight and on weekends, meaning a fast turnaround for projects. A large ecommerce site, for example, could be tested on 200 devices in perhaps 48 hours, and as such could be ready for fixes on a Monday morning, having been handed to a testing partner on Friday afternoon.

Developers can also benefit from crowdsourcing in terms of the effectiveness of its results. Because crowdsourced testers are paid for the bugs they find, rather than for time spent testing, a strong impetus is created to find critical bugs that could severely disrupt user experience. Consequently, the results tend to be much better than when time-pressured developers perform their own testing on limited resources.

Crowdsourced software testing, then, has the capacity to render emulators redundant, and can save developers time while providing coverage on far more devices than can be found in the average Open Device Lab. In doing so, it enables developers to follow the advice of web experts to always test software on real devices.

Words: Martin Mudge

Martin Mudge is the founder and technical director at BugFinders. Working in testing for over 15 years, he has headed up teams and corporates including Orange, Sophos and Nationwide.

Like this? Read these!

• Web design training: The top online resources
• Underrated web design tools
• Photoshop web design: tips to make you more efficient