The Definitive Guide to Crowdsourced Testing (for Testers and Companies)

Through this article we will take a look at every aspect of Crowdsourced Testing and QA outsourcing. This guide is useful for companies looking to Crowdsource their testing work and testers who want to make career in Crowdsourced Software testing to bring in some extra income.

Somehow or other we all have thought of adding extra income to our daily work income. And something extra, no matter what, always brings smile and is always welcomed.

Getting known for your skills and experience is a feeling of proud for any working professional. Learning and getting acquainted with new testing tools is adding experience, knowledge and income for testers but every tester knows that their power of out of box thinking is what they are actually paid off.

The Definitive Guide to Crowdsourced Testing

Crowdsourced Software Testing is one such form of testing where you are paid off for your testing capabilities.

Crowdsourced Testing, also commonly known as Crowd Testing (referred as CT henceforth) or Crowd Sourcing is a unique way of involving crowd i.e. the real users/testers, into software testing under real world conditions.

The name “Crowd” itself signifies its meaning. Crowd here refers to not just any individual but they are a group of individuals across the globe or selected few on the basis of their profile, professional background, qualification, devices on which they are able to test and other parameters.

What is Crowdsourced or Crowd testing?

what is Crowdsourced Testing

In the recent years, CT has gained much significance as it not just only the way to get the feedback, real insight about the product’s quality or to find bugs/defects but also the testers are getting paid for their efforts. The flexibility provided to the testers in terms of pay, time, work location, diverse range of testing options for testing diverse range of applications and many more have convinced many testers to select Crowdsourcing Testing as their second job.

Nowadays, easy access to internet through web or mobile devices, awareness of social media has played an important role in helping individuals to use their technical knowledge, thoughts, permutation of scenarios and experience as a part of CT to achieve their goals which is quality improvement.

In many cases, “Crowd sourced tester is proved to be more realistic tester than an experienced tester”.

People’s first impression about Crowd Sourcing and its actual meaning:

It is often misinterpreted with concept that CT is all about engaging a crowd of people to test diverse range of applications on the basis of their skill set and the tester are paid for each valid bugs/defects they report.

Well, definitely this is one phase of Crowdsourced Software Testing but it’s not just that, other phases include:

  • Sourcing relevant and skilled testers from every disciplines and levels, appropriate for the given task within the organization.
  • Building a crowd of end users or testers on the basis of analysis of their performance over time. These testers can provide feedback on diverse phases of product development.
  • Partnering with different organizations and universities leveraging domain knowledge and are presentable to the end users.

Crowd Sourcing vs. Outsourced Testing/QA Outsourcing:

Crowdsourced Testing vs QA outsourcing

Image source.

“More heads are better than one”. Task results turns into superior quality when there are combined ideas, skills and participation of people from varied background. This defines the term ‘Crowd sourcing’.

Crowd Sourcing is the collective effort from crowd of people working at different work places; basically the process is online, leading to significant results. Testers involved in Crowd Sourcing are paid only when their work is done or say when the bug found is validated.

The most famous example of Crowd Sourcing is “Wikipedia”. Wikipedia, the most comprehensive encyclopedia is the result of the information created by writers and editors from the crowd.

On the other hand, QA Outsourcing is the collective effort of the testers working in same workplace under same conditions. In these cases, unlike Crowd Software Testing scenario, testers always get paid irrespective of their output i.e. number of bugs found or validated.

The most famous and common example of Outsourcing is BPO (Business Process Outsourcing). BPO includes call centre, internet outsourcing, human resource outsourcing, etc.

Difference between ‘Crowd Sourcing’ and ‘Outsourcing’:

Let’s look at the below diagram which explains the major differences which are applicable for Crowdsourced testing and outsource testing QA:

Difference Crowd Sourcing and Outsourcing

Image Source: Lionbridge.com

Crowdsource Testing: How does it work?

Crowdsourcing Testing is a means of software testing where ‘Talent’ is recognized as ‘Service’. In any Crowd Software Testing company, the whole testing process is managed by a qualified project manager.

The whole Crowd Testing Process: 

The work of the client is to enlist all requirements, including device configuration required for testing project as well as necessary skill set of the testers suitable for that particular project. The testers complete their profile providing all necessary information along with their experience, skill set, availability of devices, background, qualifications, etc. Sometimes, few companies take a trial test or assessment of the testers to determine if they are suitable for the particular project on the basis of their skill.

After assigning project to the testers who are suitable for that particular project, they are provided with instructions, detailed test plans, scripts etc. Testers record their observation while execution or say, performing given task, finding defects, etc and in turn they are rewarded with incentives on the basis of the quality of the output delivered. Some Crowd source companies have also rating systems which determine the quality of job delivered. Discussion of relevant issues and concerns is also done through forums.

5 Phases Process:

So, to summarize, Crowd Sourcing is a 5 phase process. This is explained in below figure:

5 Phases Process in Crowdsourced Testing

Image Source and reference

Benefits and Limitation:

Now let’s look at the Benefits and Limitations of this type of testing:

Advantages:

There are multiple Advantages of this type of testing. Few of them are listed below:

  • Cost effective – Adapting method of CT allows tasks to be shared with suitable crowd and in turn they are paid for the quality delivered. If output delivery is nil then payment is also zero. This method does not give fix salary to their employees regardless of their output as is done in many organizations.
  • Cheap – As the crowd itself enroll to get testing task under CT technique, so the expenses involved in hiring engineers, specialists are less. Thus this method is cheaper.
  • Localization – Numerous ideas from numerous people are involved in this technique. And their diverse thoughts and variation in language helps as an essential in testing localization based projects.
  • Easy availability of resources – Different software’s have different specifications and requirements and it’s not always possible for any organization to have all combinations of the requirement at the same time. But this limitation can be overcome by crowd source testing technique as crowd having diverse knowledge and resources are available at the same time.
  • Fast – As large number of people are involved leading to diverse availability of resources, for a particular application there are multiple people available at the same time. This leads to fast execution and better result.

Limitations:

------------

With multiple benefits there are few limitations too of Crowd Sourced QA testing. Enlisted below are few:

  • Confidentiality – As task output is posted online and can be seen by many others, many a times critical data of any project can be shared. It may not affect many projects but confidentiality can be a major factor for few projects.
  • Larger or low/minor bugs – As crowd is paid on the number of defects reported, sometimes there may be large number of low/minor or trivial defects/bugs while critical/high bugs are skipped.
  • Track of testing effort – It is difficult for Crowd Source Testing companies to manage testing efforts and progress of testers residing in different locations and different time zones.
  • Communications – Communication plays an important role in any online job. This is sometimes a major problem especially in cases where prompt response is required. As it largely depends on availability of testers as per their time zones.

In many cases, Crowd Software Testing act as a means to ensure all the testing areas are covered. CT does not follow any rule. Crowd is allowed to test the software with ‘test in all’ approach, without following any procedure, as is done in any testing organization with limited test engineers. However, it’s not a replacement of traditional test method but it acts as an add-on to test the software before production release. CT can be beneficial in early phases of testing too.

Where is Crowd Sourcing more beneficial?

Enlisted below are few cases where Crowd Source Testing is most useful:

  • Load Test application servers – As multiple testers will hit the application at same time, intentionally or unintentionally, this form of testing can be proved as very useful in determining the behavior of application when multiple users hit at the same time.
  • Under geographical weather conditions – This parameter is applicable for mobile devices. As crowd from multiple locations having diverse weather conditions are involved in this testing, they can be helpful in determining the performance of mobile devices under bad weather conditions or conditions like impact by mountains or building basement, etc.
  • Identifying browser specific/ device specific issues – Crowd involved in testing a particular application provides multiple combinations of devices and browsers and thus certain browser/device specific issues can be determined which is normally escaped on following traditional testing process.
  • Network issues – This testing can also be helpful in determining the behaviour of application under varied network conditions like: cellular mode, flight mode, Wi-Fi, etc.
  • Testing with rarely available equipments – Sometimes, few application require testing on devices with particular combination which are rarely found or say are very expensive. But with help of this testing, this rare condition can be easily manageable and problem can be solved easily without many efforts.
  • Usability issues Usability testing focus those areas of any application which works correctly in functionality but is not much of use, or may be any other condition, from user’s perspective. It is best suited here as it involves multiple live users.

Crowdsourcing and Beta Testing

“Crowdsourced Testing is often referred to as Beta Testing, and its true also.

Beta testing is the second form of software testing where final application is tested for the last time for any left out issues and concerns before production release. Through Beta testing, real users are able to use the application for the first time, other than the development team members, before the final release of product. It can be taken as the last chance to introduce any new improvements and fix left out issues.

Beta testing was an extra expenditure earlier because involving real users before product release was extra to the pocket but with tech savvy crowd and through this testing technique, this challenge is easily fulfilled.

Nowadays, Crowd Sourced Software Testing has become the first option among the developers to run their latest application.

Let us understand this with the help of few live examples where Beta testing of live application is applicable through this testing:

  • Android Play Store – Multiple applications in their Beta version are available in Google Play, app store to let the user have an initial tour of the application. With the comments and reviews it can be ensured that the product is ready for their official and production release.
  • Wikipedia – This app beta version was first disclosed to the crowd through app store. The crowd was invited to test beta version of Wikipedia and report their observations and defects through email. Thus Crowd Sourcing was involved here.
  • Microsoft Office – Microsoft released their beta version of Office 2010 before releasing the final product. And as a result of this testing, Office was downloaded and tested by around 9 million people who provided their valuable comments and observation which ultimately resulted in many improvements in final product.

This type of Testing is more suitable to user centric applications. So, many companies release beta version of their applications to improve their final product.

How to Choose the Best Crowd Sourcing Companies?

Now, an important question arises here – How to choose a best Crowd Sourcing company Company?

Crowd testing company is not just a platform where client meets their testers. Before choosing the right CT company, it is mandatory that the client company is capable enough to perform extensive project related evaluations. Enlisted below are few criteria which should be considered while choosing the best company:

  • Experienced CT companies can be proved as best choice because they require less maintenance and governance because of their relevant experience during operational execution from past.
  • Company’s global outstretch, experience in skill delivery, security and minimum functionality
  • Mid-scale CT companies can offer only small projects like mobile applications, website testing whereas large scale companies can offer testing of enterprise applications and provide testing services along with multiple big and small projects.
  • It is important that a crowd sourced testing platform companies is financially stable as this will increase the number of long term projects without the concern factor, service, which can be hampered due to financial instability.
  • Those CT companies should be avoided which only serves as a mediator between client company and testers as they could not provide quality of test output neither they can keep a track of amount of work done by the testers.
  • Terms and Conditions, Price structure, Security, Confidentiality, Safety, Quality, capabilities, Delivery methodologies, Compensation structure, Risk management are the few major points that should be definitely analyzed before choosing CT company.

Below is a list of few good companies:

  • uTest.com
  • usertesting.com
  • mob4hire.com
  • 99tests.com
  • applause.com
  • crowdsourcedtesting.com
  • mycrowd.com
  • mogotest.com
  • usabilityhub.com
  • passbrains.com
  • oDesk.com
  • feedbackarmy.com
  • pay4bugs.com
  • bugfinders.com
  • lionbridge.com

If you know any other good Crowdsourced testing company that you want to list here – please contact me.

How to Avoid Failures in Crowd source testing

Sometimes “Crowd sourcing goes wrong” and expectations comes out as unexpected

Such failures can be avoided by keeping few points in mind:

  • Be specific in expectations from the testers and provide detailed instructions of what is to be done.
  • Develop standardized forms
  • Avoid miscommunication or communication gap with crowd.
  • FAQs should be self explanatory to avoid flood of questions email.
  • Recruit right testers and put a check on the work quality delivered to meet client’s requirement and trust.
  • Make the project task more challenging and interactive. Reward for the efforts and quality delivered to maintain tester’s trust and interest.

Crowd Sourcing as a Career Option:

We have seen various aspects of CT throughout this article. Many testers think to build a second career in Crow QA Testing.

Suggested below are few checks that should be kept in mind while choosing Crowdsource testing as a career:

  • Opt for right Company which gives you more options and flexibility.
  • Testing assignments are not always available or say testing assignment cannot always be matching your skills. So, enroll in multiple CT companies to determine which best suits your conditions.
  • Money should not be the only factor. Testers should be passionate enough regarding their work to prove their worthiness for the work.
  • Revisit your skills learned in past as well as learn new skills to increase the number of chances to become a crowd tester.
  • Don’t take multiple projects at the same time as this may end up randomizing the work with no proper output. Opt for only as much where you can go an extra mile to deliver best results.
  • Participate in CT discussions, read articles, attend conferences, etc. This will strengthen your career as well as builds your confidence.

Conclusion

Thus we can conclude, Crowdsourced Testing is the most reliable, cost effective and fast way to determine product’s quality, making it bug free. It’s a unique way of involving hired consultants, skilled users and professionals for testing diverse applications under real world conditions.

Testing applications with user’s perspective and with real crowd under real world conditions is best way to build successful software.

Crowd Testing can be proved as very effective in finding UI based and configuration based defects before Production release of any application. So, it acts as complementary testing prior to production release of any product. Thus it works as an addition to testing team of any organization but should not be thought as a replacement of testing team.

Let us know if you need more information on any topic that is not covered here. Feel free to suggest good Crowdsourcing sites if you know.

Recommended reading

16 comments ↓

#1 Meelik

There is also Testlio – crowdsourced community of expert testers.

#2 ravi s

Superb guide.
can you post more guides on how to start freelancing career?

#3 Testadrome

Great article, very interesting indeed.

Testadrome is also a young crowdtesting platform based in Barcelona, check it out!

#4 neha

hello, can you provide the details that how can i make my career in freelancing. i want to work as a freelancer but don’t know the process that how to start and take the project and all. please guide me.. thanks in advance

#5 pawan sharma

Neha,

Make an account on elance or any freelance site. and just update the information of skills and the experiences. And then do bid on the testing jobs or any other and then if client reply on your bid communicate with them..

Best
Pawan

#6 neha

thanks pawan…. is this process same for all freelancing sites?

#7 Dheeraj

Hi All,
Basically which kind of tester(Experienced or fresher) would crowd sourcing companies hire for their projects.

#8 asdfghj

I joined to Testadrome, but i didn’t liked site. Really I don’t like that sites that don’t have payout by bugs. I hate all that sites with points system to pay you after of a big consume of time testing shit… However I joined to utest and the site have a great platform system. Really is nice when a tester is respected

#9 Gaurav Khurana

Nice article covering different things.. i would like you to include below things

1) Accountability is less i think though you have shown it 100% in the diagram,though you included a point saying some critical defects can be skipped because there is no test plan .. so all may be targeting a specific area.

2) also 1 point can be included like some applications suppose banking or telecom domain cannot be easily outsourced because those testers have to know the business before and its not possible to test new features. So its suitable only for few kind of applications(Web etc)

thanks for sharing this article

#10 Akshay

Interesting topic. This would be a g8 help for freshers who want to make their carrier in Software testing and also they can earn money. This will be really a second job. Thanks for sharing this information.

#11 Maqsood Syed

This concept is new and very interesting :)

#12 vikky

there is also crowdanalytix.com

#13 Wanoj

Great tutorial..! Thanks

#14 MollyMolly

10G-force.com is a perfect example of one of the biggest software testing companies on the market. A biggest benchmarking platform out there.

#15 Fer

You lost me at “bug free”! :)

#16 Bhav

Can you please name few crowdsourced testing sites that pay per bug instead of points?

Leave a Comment