Structural Testing Tutorial – What Is Structural Testing

This comprehensive Structural Testing Tutorial explains what is Structural Testing, its types, what is Control Flow Testing and Control Flow Graph, Coverage Levels, etc.:

A quick Google search of some of the most expensive software bugs left my mind reeling – $500 billion. Yes, that’s how costly a bug can get. Reading anything related to lost lives in the transport and healthcare industries due to a software bug can be horrifying as well.

While errors in code are not always that extreme where they involve loss of copious amounts of money and lives, the only key takeaway here we are trying to convey is that one cannot overlook testing.

When testing is done frequently throughout the SDLC, it allows us to catch bugs that would need much more time to fix after the shipping of the product.

What is of importance is the software testing types that we choose. There are several of these, including functional, structural, and change-based testing.

This tutorial also explains Structural Testing Types. Learn how to do Mutation Testing, Slice Based Testing, Data Flow Testing in detail with examples and explanations.

Why Is Software Testing Important

In addition to saving money, and avoiding disasters like the cases mentioned above, there are several other reasons to justify the importance of testing.

Enlisted below are some reasons:

#1) To ensure that the stipulated requirements are met before beginning to build a project. Stakeholders (for example, developers and clients) must agree on all aspects of the solution/product/software that are required to build a project.

Testing involves verifying whether the software requirements are met. The developer or the company involved in building the solution also gain a good reputation for designing such a high-quality solution that is run by eclat code.

#2) It verifies that the code function is working as intended. Testing also involves verifying the software’s functionality and in case of any malfunction, it should be fixed during the early phases of SDLC (Software Development Life Cycle).

#3) It checks for performance: For example, to identify the total time elapsed while running code. If we use several For Loops in our code, it will take a long time to get the intended output and can even timeout sometimes.

#4) It helps to achieve a better user experience. Users will not enjoy using software that is malfunctioning, buggy, or ‘too slow’. Users will likely get impatient and drop off using the software. Testing gives us a better shot at ensuring that users can easily use our products.

#5) It checks for scalability. A developer should aim at building software that can be scaled.

#6) It checks for vulnerabilities in the code. Testing allows us with the opportunity to look out for security vulnerabilities, For example, code that may compromise PII (Personally Identifiable Information) which is a high priority for the GDPR.

In this article, we are going to focus on one type of testing i.e. Structural Testing. As the name suggests it has to do with the structure of the code. This is different than what we had mentioned earlier that testing helps to determine aspects like code performance, functionality, and security.

Structural Testing Vs Other Testing Types

There are many types of software testing. However, the ISTQB (International Software Testing Qualifications Board), defines 4 major software testing types, namely

• Functional
• Non-functional
• Structural
• Change-based

The differences can be explained as below:

Functional testing: This involves verifying the functionality of the software against the stipulated requirements. Test data is used as input. We also check that the output given is as expected.

Non-functional testing: This involves a testing process to analyze how well the software works, for example, the number of users it can handle simultaneously.

Structural testing: This type of testing is based on the code’s structure. For example, if a code is meant to calculate the average of even numbers in an array, then structure-based testing would be interested in the ‘steps that lead to the average being calculated’, rather than whether the final output is a correct numerical value.

Suppose we have to check whether we have defined the code that differentiates even numbers from odd numbers. We may have a conditional statement here, like, if an array element is divisible by two without a remainder, if (arr[i] % 2 === 0) then the number can be said as an even number.

Structural testing is carried out by the same people who write the code as they understand it best.

Change-based testing: This involves testing the effects of making changes to the code and then ensuring that the made changes have been implemented. It also ensures that the changes to the code do not break it.

What Structural Testing Is Not

We have mentioned earlier that structure-based testing refers to the structure of the code. Note that, we deal with the actual code here. We do not check against the requirements or even test inputs against the expected outputs. We are not concerned with functionality, user experience, or even performance at this point.

What Is Structural Testing

Structure-based testing, therefore, can be defined as a type of software testing that tests the code’s structure and intended flows. For example, verifying the actual code for aspects like the correct implementation of conditional statements, and whether every statement in the code is correctly executed. It is also known as structure-based testing.

To carry out this type of testing, we need to thoroughly understand the code. This is why this testing is usually done by the developers who wrote the code as they understand it best.

How To Carry Out Structural Testing

To test different aspects of the code, we need to first understand the control flows.

Control Flow Testing

This is deriving tests from the code’s control flows (the order in which statements, functions, and different aspects of the code are implemented).

Control Flow Testing Process:

Control Flow Graph

The control flow process begins by creating a visual representation of different sections of the code that helps us to define the paths that can be followed during execution.

These visual representations are known as Control Flow Graphs (CFGs) and have several components like nodes, edges, paths, junctions, and decision points. The graph can be created manually or automatically, where software is used to extract the graph from the source code.

Let’s look at these components below:

#1) Process block

This part is used to represent a section of code that is executed sequentially. This means that it is executed in the same way every time, and there are no decisions or ‘branching out’ that needs to be done. It is made up of nodes with one entry and exit path.

Example of a process block:

[image source]

The process block is not an essential part of the control flow and as a result, needs to be tested only once.

#2) Decision points

These are some key components in the code’s control flow. Within these nodes, decisions are made. This is usually done via comparison and the control flow changes, depending on the decision. This part of the CFG is made up of one node with at least 2 outputs.

The decision made here could be conditional statements like, if-else statements (which have two possible outputs) and case statements (which can have more than two outputs).

[image source]

In the above diagram, there is a decision point (from the conditional ‘age=18’) which is followed by ‘yes’ or ‘no’ options.

#3) Junction points

From the above diagram, we can easily identify junction points as to where the decision points join. Junction points can have many entry paths but can have only one exit path.

Control flow graphs best practices:

There are a few things to note when constructing control flow graphs:

• Try as much as possible to keep the CFG simple. We can do this by combining parts that may be deemed as ‘less significant’, for example, process blocks.
• Ensure that at decision points only one decision is made. In more complex CFGs, there are ‘consequences’ that comes after the decision is made. In our above example, we could also add that if an individual is 18 years or older, then they are eligible and need to pay for a ticket. If they are not, then the entry is free. The ‘else’ decision needs to ‘skip’ a few nodes, and all those steps need to be shown in our CFG.

Once we have defined our CFG, it is now time to move to the next step in the control flow testing process i.e. to define the extent to which we are going to test the code.

Defining how much to test:

How much of the source code should be tested? Should we test each possible path? Trying to cover all paths in our tests is practically impossible. We need to find a middle ground to determine how much testing we can do.

If we say that we aim at testing 50% of our code, then this could mean that we will define all executable code statements and aim at testing at least half of them. However, the question that arises here is ‘do we then need to define all possible executable paths?’

This again may be practically impossible. A better approach might be aiming at testing 50% of the paths that we can identify at each section of the code.

There are different levels of coverages, namely statement, branch, and path coverage. We will briefly look at them later.

Creating test cases:

The next step is creating the test cases that we will use. The test cases in structure-based testing are based on the following factors:

• The executable statements.
• The ‘decisions’ that need to be made.
• The possible paths that can be followed.
• The conditions that need to be met (these can be multiple or boolean).

The above factors give us an idea of the types of test cases that we need to create. We can also use a structural test generation tool. If our code is in the C programming language, we can use PathCrawler to generate test code. Another tool that we can use is fMBT.

Executing the test cases:

Here, we get to run the tests. We can enter input or data to check how the code executes it, and then verify whether we get the expected results. For example, enter an array in a function call to observe that the results that we get after looping through it, or to check whether the decision points are making the correct decisions.

Analyzing the results:

In this part, all we do is to check whether we get the correct results after execution. For example, if we enter an array where all the values are above 18, then we should have all the decision points resulting in ‘eligible’.

Control Flow Assumptions

It is important to note that to carry out control flow testing, there are a few assumptions that are made. These include:

• The only bugs present are those that can affect control flow.
• All variables, functions, and elements are accurately defined.

Coverage Levels In Control Flows

As we have mentioned earlier, there are different levels of coverage in control flow testing.

Let’s look at them briefly.

#1) Statement Coverage

In structural testing, executable code statements play a vital role when it comes to deciding the methods of designing the tests.

We aim at achieving 100% coverage, which means that every executable statement has been tested at least once. The higher the coverage, the less there is the likelihood of missing the bugs and errors.

It is required to use test cases here. The data we go for needs is to ensure that every executable statement in a block of code gets executed at least once.

#2) Branch Coverage

This coverage level involves testing the points in the CFG branches (where decisions are made). The outcomes are boolean. Even if a switch statement is used and there are multiple outcomes, in essence, each case block is a comparison of a pair of values.

Just like with statement coverage, we should aim at 100% branch coverage. To achieve this, we need to test each outcome at each decision level at least once. Since we are dealing with boolean outcomes, then we should aim at running at least 2 tests per section of code.

#3) Path Coverage

This level of coverage is more thorough when compared to decision and statement coverage. The aim here is to ‘discover’ all possible paths and test them at least once. This can be extremely time-consuming. It can, however, help discover bugs or errors in our code, or even aspects that we need to define, for example, user input.

Structural Testing Types

[image source]

Mutation Testing

Mutation testing is a fault-based testing technique in which various variations of a software application are tested against the test dataset.

>> Refer to this tutorial for an in-depth look at Mutation testing.

Slice Based Testing

Slice Based Testing (SBT) can be defined as a software testing technique that is based on slices – executable parts of the program or groups of statements which affect some values at particular points of interest in the program, for example, parts where variables are defined or the output of a group of statements.

How To Do Slicing

Slicing example in SBT: Code to print out even and odd numbers (Python)

num_list = range(1,12)
even_nums = []
odd_nums = []
for var in num_list:
      if var%2==0:
          even_nums.append(var)
          print(f"Even numbers: {even_nums}")
      elif var%3==0:
          odd_nums.append(var)
          print(f"Odd numbers: {odd_nums}")

There are two ways to look at a slice: By following the path of a variable of interest or the part of the code that affects the output.

In our example, if we look at the odd numbers output, we can trace the part of the code that leads us to this output.

In the slicing criteria given by Mark Weiser (who introduced SBT), a slice is defined using this formula: S(v, n), where, v refers to the variable in question(for example, where a variable is defined), and n is the statement of interest (for example, where output is given), and S stands for the slice.

In the above example, to get the slice, we start from our output on line 10, which becomes our n. Our variable is var.

So our slicing criteria is:

 S(v,n) = S(var,10)

Our concern is the statements that lead us to the output.

These are:
10,9,8,4,3,1

So, our slice in this code is:

num_list = range(1,12)
odd_nums = []

for var in num_list:
     elif var%3==0:
        odd_nums.append(var)
        print(f"Odd numbers: {odd_nums}")

Slice Based Testing Types

There are two types of SBT: Static and Dynamic

#1) Dynamic Slice Based Testing

The SBT example explained above where we looked at the statements that affect the printing of the odd numbers is dynamic SBT. Our concern is very specific. We get to focus only on what directly affects the particular output.

We execute the code and use test data to ensure that it works as it is supposed to. We could increase the range to range(1,50), for example, to see whether it still generates only odd numbers. Dynamic SBT is also known as validation testing.

#2) Static Slice Based Testing

Unlike Dynamic SBT, static testing’s focus is on a particular variable. If we think about our output in the above example as var, we can trace the slice that affects it as 10,9,8,7,6,5,4,3,2,1

It is basically the entire code block! Here we verify that the code is correct in terms of syntax and requirements, and we do not execute it. Static SBT is also known as verification testing.

It is important to note that dynamic SBT is ‘smaller’ when compared to its static counterpart. It is also more specific.

Slice Based Testing Best Practices/Guidelines

Slicing criteria should be determined by:

• Statements where values are defined or assigned value, as well as reassigned value.
• Statements where values are being received from outside the program, for example, via user input.
• Statements that print output/return output.
• The program’s last statement, for example, a function call which may define values, or provide values to arguments

Advantages of slice-based testing include:

• Since in SBT we only work with specific areas of interest, it makes it easier to effectively generate test suites.
• The path is defined by dependencies within the code, which is better than using path coverage.
• With SBT, it’s easier to find errors in the source code.

Disadvantages of slice-based testing include:

• If we use dynamic testing when testing a large codebase, we will need a lot of computational resources.
• If we use static testing, we might miss out on errors.

Data Flow Testing

Data flow testing can be defined as a software testing technique that is based on data values and their usage in a program. It verifies that data values have been properly used and that they generate the correct results. Data flow testing helps to trace the dependencies between data values on a particular execution path.

Data Flow Anomalies

Data flow anomalies are simply errors in a software program. They are classified into types 1, 2, and 3 respectively.

Let’s delve into them below:

Type 1: A variable is defined and a value is assigned to it twice.

Example code: Python

lst_1 = [1,2,3,4]
lst_1 = [5,6,7,8]

for var in lst_1:
    print(var)

Lst_1 is defined, and two different values are assigned to it. The first value is simply ignored. Type 1 anomalies do not cause the program to fail.

Type 2: The value of a variable is used or referenced before it is defined.

Example code: Python

for var in lst_1:
     print(var)

The loop above has no values to iterate over. Type 2 anomalies cause the program to fail.

Type 3: A data value is generated, but is it never used.

Example code: Python

lst_1 = [1,2,3,4]
lst_2 = [5,6,7,8]

for var in lst_1:
    print(var)

The variable lst_2 has not been referenced. Type 3 anomalies may not cause program failure.

Data Flow Testing Process

To define the dependencies between data values, we need to define the different paths that can be followed in a program. To effectively do this, we need to borrow from another structural testing type known as control-flow testing.

Step #1) Draw a control flow graph

We need to draw a control flow graph, which is a graphical representation of the paths that we could follow in our program.

Example code: Python

cost = 20
y = int(input("How many visitor seats did you reserve? "))
x = int(input("How many member seats did you reserve? "))
if y>x:
    bill = cost -1
else:
    bill = cost
print(bill)

In the above code example, a member should get a discount if they invite a visitor.

Control Flow Graph (CFG):

Step #2) Explore the definition and usage of variables and data values.

A variable in a program is either defined or used. In CFG, we have variables at each node. Each node is named according to the variable type it houses. If a variable is defined at a particular node, it creates a defining node. If a variable is used at a node, it creates a usage node.

If we consider the variable cost in CFG, these are the defining and usage nodes:

Step #3) Define definition-usage paths.

There are two types of definition-usage paths: du paths and dc paths. du paths are definition paths that begin with a definition node and end with a usage node. This is the case for the path in reference to the variable cost above.

An example of a dc path, a decision clear path, is the path with regards to the bill variable as below:

dc path has more than one definition node even though it still ends at a usage node.

Step #4) Create the test suite.

This is adding input. Note that we need to have a different test suite for each variable. The test suite will help us identify data flow anomalies.

Data Flow Testing Types

There are two types – Static and Dynamic.

Static means that we go through the code and CFG to identify data anomalies, without executing it. Dynamic means that we actually identify the specific paths and then create test suites to test it in a bid to ‘catch’ anomalies that we may have missed during static testing.

Advantages and disadvantages of data flow testing:

• Data flow testing is ideal for identifying data flow anomalies, which makes it a very effective structural testing method.
• Its downside is that there is a need to be well versed in the language used to write the code to use data flow testing. It is also time-consuming.

Advantages And Disadvantages Of Structural Testing

Let us now find the reasons why structural testing is a great approach, and explore some of its downsides as well.

Advantages:

• Allows for thorough code testing, resulting in minimal errors. Structure-based testing gives room for software to be thoroughly tested. The different levels of coverage – statement by statement, every decision point, and path aims at achieving 100% coverage which greatly reduces the chances of errors going undetected.
• The ability to automate. There are several tools that we can use to automate testing. This will help us achieve maximum code coverage and within a shorter time when compared to doing the testing manually.
• It results in higher quality code. The developers have a chance to study the code’s structure and implementation and fix any errors, as well as improve on these aspects. It allows us to keep the great structure in mind as we write subsequent parts of code or implement remaining features.
• It can be done through each phase of the SDLC – Structural testing can be done at each phase of the SDLC without waiting for development to be completed 100%. This makes it easy to identify errors early phase and thus saving much time when compared to testing after development is complete.
• It helps get rid of dead code. This can be seen as ‘extra’ or unnecessary code, for example, code that will calculate a result but never uses it in any of the following calculations.
• Efficiency – Since the developers writing the code are the same ones who test it, there is no need to involve other people like QAs.

Disadvantages:

• The developers who perform structure-based testing need to have a thorough understanding of the language. Other developers and QAs who are not well versed in the language cannot help with testing.
• It can become quite expensive in terms of time and money. A lot of time and resources are required to do testing efficiently.
• It causes delays in features delivery. This is because developers are pulled from building software to do testing.
• Scaling is an issue, especially where large applications are involved. A large application equals an excessively high number of routes to cover. Achieving 100% coverage becomes impossible.
• There may be missed cases and routes, for example, in a case where features are not fully developed or are yet to be developed. This means that it needs to be combined with other testing types like, requirements testing (where we check against the specified features that needed to be built).

Structural Testing Best Practices

Some of the factors that require attention while carrying out structure-based testing are as follows:

• Clearly label and name the tests. If someone else needs to run the tests, they need to be able to locate them easily.
• Before enhancing code, i.e. by refactoring it, and optimizing it for use in different environments, ensure that its structure and flow are ideal.
• Run tests separately. In this way, it is easy to identify bugs and fix them. On the other hand, we are less likely to miss bugs or paths as a result of overlaps in code sections, blocks, or paths.
• Generate tests before making changes. The tests are required to run as expected. This way, if something breaks, then it is easy to trace and fix the problem.
• Keep the tests for each section or block of code separate. This way, if there are changes down the line, we need not change a lot of tests.
• Fix bugs before moving on with testing. If we identify any bugs, we are better off fixing them before proceeding to test the next section or block of code.
• Never skip structural testing with the assumption that a QA will ‘still do testing anyway’. Even if the bugs may seem insignificant at first, cumulatively, they can result in buggy code that can never achieve its intended purpose.

FAQs For Structure-based Testing

Here we will explore the frequently asked questions when it comes to structure-based testing.

Q #1) What is the difference between functional testing and structural testing?

Answer: Functional testing is a type of software testing based on stipulated requirements in the SRS (Software Requirements Specifications). It is usually done in a bid to find disparities between specs in the SRS and how the code works. Structural testing is based on the code’s internal structure and its implementation. A thorough understanding of the code is required.

Q #2) What are the types of structural testing?

Answer: The types include:

• Data flow testing
• Mutation testing
• Control flow testing
• Slice-based testing

Q #3) What is a structural testing example?

Answer: Here is an example showing statement coverage:

const addNums = (num) => {
	    let sum = num.reduce ((a,b) => a+b);
                  if (sum > 0) {
                    alert(sum);
                  }
                  else {
                    alert(‘please enter positive numbers’);
                 }
            };
            addNums();

The amount of coverage that we get depends on the test data that we provide as input (whether it meets the sum>0 conditions).

Q #4) What is the difference between data flow testing and control flow testing?

Answer: Both data flow testing and control flow testing use control flow graphs. The only difference is that in control flow testing, we focus on the paths generated from the code, while in data flow testing, we focus on the data values, their definition, and usage within the paths identified within a program.

Q #5) What is data flow testing used for?

Answer: Data flow testing is ideal for identifying anomalies in the usage of data values within paths in a control flow graph, for example, one variable that has been assigned value twice, a variable that has been defined and not used, or a variable that has been used or referenced and not defined.

Q #6) What is the difference between slicing and dicing in software testing?

Answer: Slicing means focusing on particular statements of interest in a program and ignoring the rest. Dicing is when we identify a slice that is having wrong input and then further slicing it to trace correct behavior.

Q #7) What is the difference between mutation testing and code coverage?

Answer: In Mutation testing, we consider the number of killed mutants as a percentage of the total mutants. Code coverage is simply the amount of code that has been tested in a program.

Conclusion

In this tutorial we looked at structural testing in depth – what it is, what it is not, how to go about it, coverage types, advantages, and disadvantages, best practices, and even some FAQs regarding this software testing type.

There is still so much more that we can learn about structure-based testing. In future tutorials, we will explore code coverage (statement, decision, branch, and path), structural testing types (mutation, data-flow, and slice-based), and even the tools that we can use to automate these testing processes.

It is important to note that there is no software testing type or approach that is 100% efficient. It is always advisable to combine different testing types and approaches.

For example, structural testing is greatly complemented by requirements testing, as there may be features that may not have been developed at the time when structure-based testing was being carried out.

Structural testing techniques are based on the errors that human programmers make when writing code. The assumption is that the programmer is an expert and knows what he or she is coding, but errs from time to time.

The different structural testing types that we have looked at – mutation testing, slice-based testing, and data flow testing could be traced back to errors like using the wrong operator (mutation testing) or referencing a variable before using it (data flow testing).