Top 6 BEST Python Testing Frameworks [Updated 2020 List]

This Tutorial Explains How Python can be Used for Test Programming and Lists the Features and Comparison of the Top Python Testing Frameworks:

With the widespread application of Artificial Intelligence, Python has become a popular programming language.

This tutorial will cover how Python can be used for test programming along with some Python-based testing frameworks.

Let’s start!!

Python Testing Frameworks

What Is Python?

According to the traditional definition, Python is an interpreted, high-level, general programming language that helps programmers to write manageable and logical code for small as well as large-scale projects.

Some of the benefits of Pythons are:

  • No compilation causes the fast execution of the Edit-Test-Debug cycle.
  • Easy debugging
  • Extensive support library
  • Easy to learn data-structure
  • High productivity
  • Team collaboration

Working In Python

Working in Python

  • The interpreter reads the python code from the source file and examines it for a syntax error.
  • If the code is error-free then the interpreter converts the code to it’s equivalent ‘Byte code’.
  • This byte code is then transmitted to the Python Virtual Machine (PVM) where the Byte code is again compiled for error if any.

What Is Python Testing?

  • Automated testing is a well-known context in the world of testing. It is where the test plans are being executed using script instead of a human.
  • Python comes with the tools and libraries that support automated testing for your system.
  • Python Test cases are comparatively easy to write. With the increased use of Python, Python-based test automation frameworks are also becoming popular.
=>> Contact us to suggest a listing here.

List Of Python Testing Frameworks

Enlisted below are some Python Testing frameworks that you should know.

  1. Robot
  2. PyTest
  3. Unittest
  4. DocTest
  5. Nose2
  6. Testify

Comparison of Python Testing Tools

Let’s quickly summarize these frameworks in a short comparison table:

 LicensePart ofCategoryCategory
Special feature

Robot

Robot Logo

Free software
(ASF License}
Python generic test libraries.Acceptance testingKeyword-driven testing approach.
PyTest

PyTest Logo
Free software (MIT License)Stand alone, allows compact test suites.Unit TestingSpecial and simple class fixture for making testing easier.
unittest

UnitTest Logo
Free software (MIT License)Part of Python standard library.Unit TestingFast test collection and flexible test execution.
DocTest

Doctest Logo
Free software (MIT License)Part of Python standard library.Unit TestingPython Interactive Shell for the command prompt and inclusive application.
Nose2

Nose Logo
Free software
(BSD License)
Carries unittest features with additional feature and plugins.unittest extensionA large number of plugins.
Testify

Testify Logo
Free software
(ASF License)
Carries unittest and nose features with additional feature and plugins.unittest extensionTest discovery enhancement.

(Abbreviations: MIT = Massachusetts Institute of Technology (1980), BSD = Berkeley Software Distribution (1988), ASF = Apache Software Foundation(2004))

Let’s start!!

#1) Robot

  • The most popular Robot Framework is an open-source Automation Testing framework based on Python.
  • This framework is entirely developed in Python and is used for Acceptance Testing and Test-driven development. Keyword style is being used to write test cases in Robot framework.
  • The Robot is capable of running Java and .Net and also supports automation testing on cross-platform like Windows, Mac OS and Linux for desktop applications, mobile applications, web applications, etc.
  • Along with Acceptance Testing, Robot is also used for Robotic Process Automation (RPA).
  • Pip (Package Installer for Python) is highly recommended for Robot installation.
  • The use of tabular data syntax, keyword-driven testing, rich libraries & toolset, and parallel testing are some of the strong features of Robot that make it popular among testers.

Example:

*** Settings ***
Library           SeleniumLibrary

*** Variables ***
${SERVER}         localhost:7272
${BROWSER}        Firefox
${DELAY}        0
${VALID USER}     demo
${VALID PASSWORD}    mode
${LOGIN URL}      http://${SERVER}/
${WELCOME URL}    http://${SERVER}/welcome.html
${ERROR URL}      http://${SERVER}/error.html

*** Keywords ***
Open Browser To Login Page
    Open Browser    ${LOGIN URL}    ${BROWSER}
    Maximize Browser Window
    Set Selenium Speed    ${DELAY}
Login Page Should Be Open
    Title Should Be    Login Page

Go To Login Page
    Go To    ${LOGIN URL}
    Login Page Should Be Open

Input Username
    [Arguments]    ${username}
    Input Text    username_field    ${username}

Input Password
    [Arguments]    ${password}
    Input Text    password_field    ${password}

Submit Credentials
    Click Button    login_button

Welcome Page Should Be Open
    Location Should Be    ${WELCOME URL}
    Title Should Be    Welcome Page

Here is a sample of Failed Test Execution.

Robot - Failed tests

Here is a sample of Successful Test Execution.

Robot - Successful tests

Packages/Methods:

Package NameWorkingPackage Import
run()To run tests.from robot import run
run_cli()To run tests with command line argument.from robot import run_cli
rebot()To process test output.from robot import rebot

Link to API: Robot Framework User Guide
Download Link: Robot


#2) PyTest

  • PyTest is an open-source Python-based testing framework that is generally all-purpose but especially for Functional and API testing.
  • Pip (Package Installer for Python) is required for PyTest installation.
  • It supports simple or complex text code to test API, databases, and UIs.
  • Simple syntax is helpful for easy test execution.
  • Rich plugins and is able to run tests in parallel.
  • Can run any specific subset of tests.

Example:

import pytest                                //Import unittest module//
def test_file1_method():               //Function inside class//
      x=5
      y=6
      assert x+1 == y,"test failed"

To run the test use the py.test command.

Screenshot for Reference:

PyTest

[image source]

Packages/Methods:

FunctionParametersWorking
pytest.approx()expected,
rel=None,
abs=None,
nan_ok=False
Assert that two numbers or two
sets of numbers are approximately
equal to some differences.
pytest.fail()msg (str)
pytrace(bool)
If the executing test fails explicitly the message is shown.
pytest.skip()allow_module_level(bool)Skip the executing test with the message shown.
pytest.exit()msg (str)
returncode (int)
Exit testing process.
pytest.main()args=None
plugins=None
Return exit code once in-process test execution is done.
pytest.raises()expected_exception: Expectation[, match]Assert that a code block call raises expected_exception or to raise a failure exception
pytest.warns()expected_warning: Expectation[, match]Asserting warning with the functions

If you want to access a test written in a specific file we use the below command.

py.test <filename>

Pytest Fixture: Pytest Fixture is used to run code before executing the test method to avoid code repetition. This is basically used to initialize database connection.

You can define PyTest fixture as shown below.

@pytest.fixture

Assertion: Assertion is the condition that returns true or false. Test execution stops when the assertion fails.

Given below is an Example:

def test_string_equal():
assert double(55) == 62
assert 25 == 62
+  where 25 = double(55)

Link to API: Pytest API
Download Link: Pytest


#3) Unittest

  • Unittest is the very first Python-based automated unit test framework that was designed to work with the Python standard library.
  • Supports the reuse of test suits and test organization.
  • It was inspired by JUnit and supports test automation including test collections, test independence, setup code for tests, etc.
  • It is also being called as PyUnit.
  • Unittest2 is a backport of additional new features added to the Unittest.

Standard workflow of Unittest:

  • Import the Unittest module in the program code.
  • You can define your own class.
  • Create functions inside the Class that you have defined.
  • Place unittest.main() which is the main method at the bottom of the code to run the test case.

Example:

import unittest                                 //Import unittest module//
def add(x,y):
   return x + y

class Test(unittest.TestCase):          //Define your class with testcase//

   def addition(self):
      self.assertEquals(add(4,5),9)<strong>//Function inside class//

if __name__ == '__main__':
   unittest.main()<strong>//Insert main() method//

Screenshot for Reference:

unittest

[image source]

Packages/Methods:

MethodWorking
setUp()Called before test method execution to prepare test installation.
tearDown()Called after test method execution even if the test throws an exception.
setUpClass()Called after tests in an individual class.
tearDownClass()Called after tests in an individual class.
run()Run the test with results.
debug()Run the test without result.
addTest()Add the test method in the test suite.
Discover()Finds all the test modules in subdirectories from the specific directory.
assertEqual(a,b)To test equality of two object.
asserTrue/assertFalse(condition)To test Boolean condition.

(Note: unittest.mock() is a library for Python testing that allows replacing system parts with mock objects. The core mock class helps to create a test suite easily.)

Link to API: Unittest API
Download Link: Unittest


#4) DocTest

  • Doctest is a module that is included in Python’s standard distribution and is used for White-box Unit Testing.
  • It searches for interactive python sessions to check if they are working exactly as required.
  • It makes use of selective Python capabilities such as docstrings, The Python interactive shell and Python introspection (determining properties of objects at runtime).
  • Core Functions:
    • Updating docstring
    • Performing Regression Testing
  • The functions testfile() and testmod() are used to provide basic interface.

Example:

def test(n): 
import math
    if not n >= 0:
        raise ValueError("n must be >= 0") //number should be 0 or greater than 0
    if math.floor(n) != n:
               raise ValueError("n must be exact integer")   
                                                                     //Error when number is not an integer
  if n+1 == n:  
        raise OverflowError("n too large") //Error when number is too large
    r = 1
    f = 2
    while f <= n:                                      //Calculate factorial
        r *= f
        f += 1
    return r

if __name__ == "__main__":
     import doctest                        //Import doctest
    doctest.testmod()                    //Calling the testmod method

Screenshot for Reference:

doctest

[image source]

Packages/Functions:

FunctionParameters
doctest.testfile()filename (mendatory)
[, module_relative]
[, name][, package]
[, globs][, verbose]
[, report][, optionflags]
[, extraglobs][, raise_on_error]
[, parser][, encoding]
doctest.testmod()m][, name][, globs]
[, verbose][, report]
[, optionflags]
[, extraglobs]
[, raise_on_error]
[, exclude_empty]
doctest.DocFileSuite()*paths, [module_relative][, package][, setUp][, tearDown][, globs][, optionflags][, parser][, encoding]
doctest.DocTestSuite()[module][, globs][, extraglobs][, test_finder][, setUp][, tearDown][, checker]

Note: For checking interactive examples in the text file we can use the testfile() function;

doctest.testfile (“example.txt”)

You can directly run the test from the command line with;

python factorial.py

Link to API: DocTest API
Download Link: Doctest


#5) Nose2

  • Nose2 is the successor of Nose and it is a Python-based Unit Testing framework that can run Doctests and UnitTests.
  • Nose2 is based on unittest hence it is referred to as extend unittest or unittest with the plugin that was designed to make testing simple and easier.
  • Nose uses collective tests from unittest.testcase and supports multiple functions for writing tests and exceptions.
  • Nose supports package fixtures, classes, modules, and complex initialization to be defined at a single time instead of writing frequently.

Example:

from mynum import *
import nose

def test_add_integers():
    assert add(5, 3) == 8

def test_add_floats():
    assert add(1.5, 2.5) == 4

def test_add_strings():
    nose.tools.assert_raises(AssertionError, add, 'paul', 'carol') 
// To throw one of the expected exception to pass

if __name__ == '__main__':   
    nose.run()

Screenshot for Reference:

Nose 2

Packages/Methods:

MethodParametersWorking
nose.tools.ok_(expr, msg = None)Shortcut to assert.
nose.tools.ok_(a,b, msg = None)Shortcut to ‘assert a==b,
“%r != %r” % (a, b)
nose.tools.make_decorator(func)To replicate metadata for the given function.
nose.tools.raises(*exception)To throw one of the expected exceptions to pass.
nose.tools.timed(limit)To specify the time limit within which the test should get a pass.
nose.tools.with_setup(setup=None, teardown=None)To add setup method to a test function.
nose.tools.intest(func)Method or function can be referred as test.
nose.tools.nottest(func)Method or function cannot be referred as test.

Link to API: Plugins for Nose2
Download Link: Nose2


#6) Testify

  • Testify was designed to replace unittest and nose. Testify has more advanced features over unittest.
  • Testify is popular as a Java implementation of semantic testing (Easy to learn and implementing software testing specification).
  • Performing Automated unit, Integration and System Testing is easier to Testify.

Features

  • Simple syntax to fixture method.
  • Improvised test discovery.
  • Class-level setup and teardown fixture method.
  • Extensible plugin system.
  • Easy to handle testing utilities.

Example:

from testify import *

class AdditionTestCase(TestCase):

    @class_setup
    def init_the_variable(self):
        self.variable = 0

    @setup
    def increment_the_variable(self):
        self.variable += 1

    def test_the_variable(self):
        assert_equal(self.variable, 1)

    @suite('disabled', reason='ticket #123, not equal to 2 places')
    def test_broken(self):
        # raises 'AssertionError: 1 !~= 1.01'
        assert_almost_equal(1, 1.01, threshold=2)

    @teardown
    def decrement_the_variable(self):
        self.variable -= 1

    @class_teardown
    def get_rid_of_the_variable(self):
        self.variable = None

if __name__ == "__main__":
run()

Screenshot for Reference:

Testify

[image source]

Packages/Methods:

Package NameWorkingPackage import
assertProvides comprehensive testing tools for the system testing.import "github.com/stretchr/testify/assert"
mockUseful to test your objects and calls.import "github.com/stretchr/testify/mock"
requireWorks as same as to assert but stops test execution when tests fail.import "github.com/stretchr/testify/require"
suiteIt provides logic for creating testing suite structure and methods.import "github.com/stretchr/testify/suite"

Link to API: Package files of Testify
Download Link: Testify


Additional Python Testing Framework

So far we have reviewed the most popular Python Testing Framework. There are few more names on this list which might become popular in the future.

#7) Behave

  • Behave is referred to as BDD (Behavior Driven Development) test framework that is also used for Black box testing. Behave uses the natural language for writing tests and works with Unicode Strings.
  • Behave directory contains feature files that have a plain text format looks like natural language and Python step implementations.

Link to API: Behave User Guide
Download Link: Behave


#8) Lettuce

  • Lettuce is useful for Behavior Driven Development Testing. It makes the testing process easy and scalable.
  • Lettuce includes steps such as:
    • Describing behavior
    • Steps definition in Python.
    • Running the code
    • Modifying code to pass the test.
    • Running the modified code.
  • These steps are being followed for 3 – 4 times to make the software error-free and thereby enhance its quality.

Link to API: Lettuce Documentation
Download Link: Lettuce


Frequently Asked Questions And Answers

Let’s have a look at some of the most common FAQs on this topic-

Q #1) Why is Python used for automation?

Answer: As ‘Python comes with the tools and libraries that support automated testing for your system’, there are several other reasons why Python is used for testing.

  • Python is object-oriented and functional that lets programmers conclude whether the function and classes are suitable as per the requirements.
  • Python offers a rich library of useful packages for testing after installing ‘Pip’.
  • Stateless functions and simple syntax are helpful to create readable tests.
  • Python plays the role of the bridge between the test case and the test code.
  • Python supports dynamic duck typing.
  • Offers well configured IDE and good support to the BDD framework.
  • Rich command line support is helpful to perform a manual check.
  • Simple and good structure, modularity, rich toolset, and packages can be useful for scale development.

Q #2) How to structure a Python test?

Answer: By the time you create a test in Python, you should consider two things as stated below.

  • Which module/part of the system you want to test?
  • Which type of testing you are opting in (whether unit testing or integration testing)?

The overall structure of the Python Test is as simple as others where we decide the components of tests such as – inputs, test code to be executed, output and comparison of output with expected results.

Q #3) Which automation tool is written in Python?

Answer: Buildout is an automation tool that is written in and extended with Python and is used for automating software assembly. Buildout can be applicable to all the software phases right from development to deployment.

This tool is based on 3 core principles:

  • Repeatability: It states project configuration developed in the same environment should produce the same result regardless of their history.
  • Componentization: Software service should include self-monitoring tools and should configure the monitoring system while product deployment.
  • Automation: Software deployment should be highly automated and time-saving.

Q #4) Can Python be used with Selenium?

Answer: Yes. Python language is used with Selenium to perform testing. Python API is helpful to connect with the browser through Selenium. Python Selenium combination can be used to write functional/acceptance tests using Selenium WebDriver.

Q #5) Is Selenium with Python good?

Answer: There are several reasons why Selenium and Python are considered as a good combination:

  • Selenium has the strongest toolset to support quick test automation.
  • Selenium offers Dedicated test functions to perform web application testing that helps to examine real application behavior.
  • Whereas, Python is a high-level, object-based and user-friendly scripting language with a simple keyword structure.

Now, when it comes to using Selenium with Python it has several benefits as stated below.

  • Easy to code and read.
  • Python API is extremely useful to connect you to the browser through Selenium.
  • Selenium sends standard command of Python to various browsers regardless of its design variations.
  • Python is comparatively simple and compact than the other programming languages.
  • Python comes with a big community to support those who are completely new to use Selenium with Python to perform automation testing.
  • It is free and open programming language all the time.
  • Selenium WebDriver is another strong reason for using Selenium with Python. Selenium WebDriver has strong binding support for Python’s easy user interface.

Q #6) What are the measures to choose the best Python testing framework?

Answer: For choosing the best Python testing framework, the below points should be taken into consideration:

  • If the quality and structure of the scripts, is fulfilling your purposes. The programming script should be easy to understand/maintain and free of defects.
  • The programming structure of Python plays an important role in choosing the testing framework which consists – Attributes, statements, functions, operators, modules and standard library files.
  • How easily you can generate tests and at what extent they can be reused?
  • The method adopted for test/test module execution (Module running techniques).

Q #7) How to choose the best Python Testing framework?

Answer: Understanding the advantages and limitations of each framework is a better way to choose the best Python Testing framework. Let us explore –

Robot Framework:

Advantages:

  • Keyword-driven test approach helps to create readable test cases in an easier way.
  • Multiple APIs
  • Easy test data syntax
  • Supports parallel testing via Selenium Grid.

Limitations:

  • Creating customized HTML reports is quite tricky with Robot.
  • Less support to parallel testing.
  • It requires Python 2.7.14 and above.

Pytest:

Advantages:

  • Supports compact test suite.
  • No need for the debugger or any explicit test log.
  • Multiple fixtures
  • Extensible plugins
  • Easy and simple test creation.
  • Possible to create test cases with fewer bugs.

Limitations:

  • Not compatible with other frameworks.

Unittest:

Advantages:

  • No need for any additional module.
  • Easy to learn for testers at beginner’s level.
  • Simple and easy test execution.
  • Rapid test report generation.

Limitations

  • snake_case naming of Python and camelCase naming of JUnit cause a bit of confusion.
  • Unclear intent of the test code.
  • Requires a huge amount of boilerplate code.

Doctest:

Advantages:

  • A good option for performing small tests.
  • Test documentation within the method also provides additional information about how the method works.

Limitations

  • It only compares the printed output. Any variation in the output will cause a test failure.

Nose 2:

Advantages:

  • Nose 2 supports more testing configuration than unittest.
  • It includes a substantial set of active plugins.
  • Different API from unittest that provides more information about the error.

Limitations:

  • While installing third-party plugins you must install setup tool/distribute package, as Nose2 supports Python 3 but not third-party plugins.

Testify:

Advantages:

  • Easy to understand and use.
  • Unit, Integration and System tests can be easily created.
  • Manageable and reusable test components.
  • Adding new features to Testifyis easy.

Limitations:

  • Initially Testify was developed to replace unittest and Nose but the process of transiting it to pytest is on, so it is recommended for the users to avoid using Testify for few upcoming projects.

Behave Framework:

Advantages:

  • Easy execution of all types of test cases.
  • Detailed reasoning & thinking
  • Clarity of QA/Dev output.

Limitations:

  • It only supports black box testing.

Lettuce Framework:

Advantages:

  • Simple language to create multiple test scenarios.
  • Helpful for behavior-driven test cases for black-box testing.

Limitations:

  • It strongly needs strong coordination among developers, testers & stakeholders.

You can choose the best suitable Python testing framework by considering the above advantages and limitations that will help to develop the criteria suitable for your business needs.

Q #8) Which framework is best for Python Automation?

Answer: While considering the advantages and limitations, we can consider the testing type as one of the measures for choosing best testing framework:

  • Functional Testing: Robot, PyTest, Unittest
  • Behavior-Driven Testing: Behave, Lettuce

Robot is the best framework for those who are new to Python testing and wish to get a solid start.

Conclusion

Subunit, Trial, Test resources, Sancho, Testtools are some more names added in the list of Python Testing Framework. However, there are only a few tools that have been popularized so far as Python testing is a comparatively new concept that is introduced in the testing world.

Companies are working on making these tools better so that they are easy to understand and perform testing. With the rich and precise class fixtures, plugins and packages these tools can become well-versed and preferable for performing Python Testing.

Meanwhile, frameworks mentioned above from unittest to Testify are providing much necessary support and service to achieve the intended system performance.

=>> Contact us to suggest a listing here.