This complete guide to Benchmark Testing explains what is it, why do we need it, the different phases involved, advantages and challenges faced in Benchmark Testing:
Benchmark Testing is a set of standards, metrics, or a reference point, against which, the performance quality of a product or a service is assessed or evaluated.
Example:
Yo-Yo Test in cricket: Yo-yo test in cricket is an aerobic fitness endurance test. Indian cricket team has to undergo the Yo-yo fitness test as per the BCCI norms.
The benchmark score to pass the test is set as 19.5, depending upon various speeds and endurance levels of the sport. The cricketers have to reach the benchmark of 19.5 to qualify for the Indian Cricket team. Thus a benchmark serves as a basis for evaluating performance metrics.
What You Will Learn:
Benchmark Testing
Load testing of a module or an entire end to end software system to determine its performance is termed as Benchmark Testing. It determines a repeatable set of experimental results that aid in baselining the functionalities for current as well as future software releases.
Benchmark testing compares the performance of a software or hardware system (commonly known as SUT, System Under Test). A web-based application can be said as SUT.
Benchmark Testing is creating a standard for the software delivered. The standard is set across companies or organizations. Benchmark testing allows the standard of work or workability that is delivered to be compared across companies.
Example: Internet Speed
Nowadays multiple software applications or websites are available to determine the performance of your internet speed. These applications have benchmarked the internet speed depending upon various factors like country, download or upload speed etc.
The internet speed for any broadband connection is evaluated as good or bad depending upon this benchmarked internet speed.
Importance Of Benchmark Testing
The importance of benchmark testing in the Software Development Life Cycle (SDLC) is explained in the points below. Benchmark software testing technique aids the team of skilled and proficient testers in numerous ways.
- The performance characteristics of an application are tested. The performance should be consistent, as per the standards defined by the organization.
- The effects of the performance characteristics are tested after the changes are done to the system.
- The response of a ‘Database Manager’ under varying conditions can be monitored with the help of benchmark testing.
- The response time, concurrent users, and website’s consistent availability can be checked. It ensures that the website follows the organizational standards and top practices.
- The performance of the application is as per the defined SLA’s (service level agreement).
- To test the rate of transactions as more users are added.
- Deadlock handling scenarios can be tested so that deadlock situations can be avoided.
- A system’s‘ utility performance’ can be tested. Loading of data with various methods.
- Impact, behavior, and characteristics of an application after a new release.
- Benchmark Tests done are repeatable – they have the same conditions under which the same tests are run. The results yielded from these tests are compared legitimately.
- As performance testing is done it helps in improving the performance as well as the functionality of the application.
A simple performance test can be done for your PC as shown below:
- On your laptop or PC press? Win + R to open the Run dialog box.
- Enter ‘dxdiag’ in the Run dialog box and press the ‘Enter’ key or ‘OK’ button.
- On the System Tab, the ‘Processor’ entry can be checked.
Components Of Benchmark Testing
Specifying Workload Conditions: The type and frequency of the requests need to be determined.
Enlisted below are the points to be considered while specifying the workload conditions:
- Hardware: Database nodes, elastic nodes, coordinating nodes, cluster.
- Network Configuration and security.
- Operating system Version.
- Patch levels
- Software: JVM and component applications.
- Servers
- Libraries and software packages etc.
Metrics Specification: The elements that are going to be tested is determined.
Example: Download Speed, Application code, SQL queries (determining which is the fastest: Left Join or Correlated Query).
Measurement Specification: The way to measure the specified metric or elements for determining the expected and appropriate results.
Pre-requisites
To set the software for benchmark testing, some crucial settings of the software, environmental conditions, and vital software requirements need to be completed. This ensures a smooth performance of benchmark testing.
The pre-requisites of Benchmark Testing can be specified as:
- All software components are functioning as expected.
- Operating System and supporting drivers are updated as per the requirements and are in good working condition.
- Cache files and temporary files are cleared from the system and no unnecessary residue files are remaining.
- Processes and applications running in the background are closed.
- Software architecture, design, test data, test criteria, database structures, file structures, etc should perform accurately and its performance should be well under control.
- Hardware and software components should be synced duly and seamlessly without any errors.
- No unnecessary bugs should occur and the software should not break in between, it should perform accurately with the same consistency.
- Real-world, environmental configurations need to be set.
- Must have updated operating systems as per the requirements.
- Exactly the same environmental conditions should be provided for each and every test run.
Phases Of Benchmark Testing
Firewall Testing
#1) Planning Phase
Planning Phase – (What to Benchmark and When to Benchmark)
It is the initial and most important phase. Devoted time and attention are given to this phase to ensure that planning becomes error-free and the rest of the phases are effective as well as efficient. The concerned stakeholders are closely involved in this phase.
- The standards and requirements are identified and then prioritized.
- Benchmark criteria are decided.
Let’s take the example of setting up a Firewall for an organization or a company.
Example:
In the planning phase, the standards or rules will be set for benchmarking a firewall as follows:
- New and established incoming traffic is accepted on a public network interface on Port 80 and 443 (HTTP and HTTPS web traffic)
- Incoming traffic from IP addresses of non-technical staff will be dropped to port 22.
- Rejecting incoming traffic on the public network from unknown IP addresses.
Accept traffic: Allowing the traffic through a port.
Drop traffic: Blocking the traffic and sending no reply.
Reject traffic: Blocking the traffic and sending an “unreachable” error reply.
#2) Application Phase
The dataset collected during the Planning phase is analyzed in the Application phase.
- Root cause analysis (RCA) is done to avoid error and thereby improve quality.
- Goals are set for the test process.
Example:
In the Application Phase, the Root Cause Analysis will be done for Firewall Testing.
- Error: Non-technical staff’s incoming traffic is dropped but the outside network is able to establish a connection with the open service on your network.
- Root Cause Analysis: The firewall has a loosely and poorly configured rule-set. It keeps out the only subset of the non-technical staff from accessing the server. The server remains open for the other outside traffic.
The application phase thus assists in avoiding such mistakes and thereby helps in improving the security level of the firewall.
#3) Integration Phase
This phase is the connector between the earlier two phases of planning analysis and the final phase i.e. action phase.
- The outcomes or results from the earlier two phases are shared with the concerned persons (Project Managers, Leads, stakeholders, etc.).
- Goals are set for the test process.
Example:
In the Integration phase, the port setting will get approved by the concerned people and an action plan will be decided.
- Port settings are done accurately as per the standard rule-set.
- The rule-set gets approved by the concerned people.
- The action plan is decided to monitor and protect network traffic.
#4) Action Phase
Action Phase: (Keep the Process Continuous): This phase ensures that all the improved steps, standards, and rule sets have been taken into consideration and implemented successfully.
- The action plan is developed for implementation.
- Actions determined in the previous processes are implemented and monitored.
- Mechanisms are developed to periodically review the actions implemented so that the performance remains good and the benefits are retained.
Example:
In the Action Phase, the outcomes from the earlier phases are implemented.
- The network traffic is closely monitored.
- Intrusion attacks and other threats to the network are handled.
- Updates and patches are periodically provided to handle new threats.
Advantages Of Benchmark Testing
- As per the new users, the initial data must be examined and updated.
- Ensures that all the software components are working precisely as per the expectations.
- A meticulously built application that can sustain and face all the real-world rigors.
- Software developers and testers can confidently launch their applications. They themselves are very confident about the applications released.
- The effectiveness and performance of the released product are well up to the mark.
Challenges Faced
- Not able to determine the actual risk involved concerning the load and performance issue. As the actual risk (high) is not clearly determined, the level of testing done may get lower.
- As the risk predicted is not accurate the budget finalized by the stakeholders is not sufficient. The stakeholders or budget approvers do not recognize the value of benchmark testing as it’s non-functional testing. Though all projects have some level of risk involved, however, more problems may arise as risk is not understood clearly and hence not mitigated correctly.
- Benchmark Testing requires time and money. But usually, during the planning phase of testing (not the benchmark testing planning phase), less time and comparatively low budget are allocated for benchmark testing. This happens as there are less awareness, less knowledge, and a lack of appetite regarding benchmark testing.
- Suitable tools need to be selected for benchmark testing. The factors involved in selecting the right tools are the skills and experience of the testers involved, licensing costs, and corporate standards. Frequently open source tools that can lead to higher project risks are used, as essential tools are not used.
Challenges faced during benchmark testing are largely tactical and require a lot of patience, time, and budget. Moreover, it needs more involvement and understanding from the stakeholders or decision-makers to successfully benchmark testing any deliverable.
Areas Of Implementation
#1) Browser Compatibility:
The factors include load time, startup time, frames-per-second for live streaming of videos, javascript runs, the time taken for the browser to start drawing the page on the screen, and the number of bytes downloaded (the faster the bytes are loaded, the faster everything is displayed on screen) and browser requests.
Fluctuations in the results (tests are done multiple times and hence multiple results are compared for multiple browsers) for all the factors mentioned above are calculated and depending on these factors the fastest browser is determined.
#2) Broken Links:
Link, when clicked on a webpage, leads to an error or an empty webpage. This creates an unprofessional impression on the website viewers and also leads to low ranking during search engine results. These links are reported and thereby aid in re-directing or excluding the broken links.
#3) HTML Compliance:
This is important to ensure the interoperability of the website. When a website is launched, it should adhere to some of the coding practices regarding HTML or XHTML usage, Cascading Style Sheets (CSS), layout definitions, etc.
HTML 5 includes the syntactic features for multimedia and graphical content. The main objective is to improve the language that supports the latest multimedia & other new features and thereby is easily readable by both humans as well as computer devices.
#4) SQL:
Factors for Benchmarking:
- SQL queries (algorithmic complexity, Reduce I/O, deciding whether a correlated sub-query or Left join is faster).
- SQL server (Batch Requests/sec, SQL compilations/sec, SQL recompilations/sec, max workers, idle workers, deadlocks).
#5) CPU Benchmark:
Benchmarking clock speed of the CPU, per cycle registry calls, instructions executed, and disk architecture.
#6) Hardware Configuration (Domain networks and standalone PCs):
Processor, co-processor, scalable parallel processor, motherboard, chipset, memory, CPU cooler, CPU socket, computer system cooling, etc.
#7) Application:
The benchmarks set for the application depend on factors such as robustness, efficiency, security, changeability, transferability, technical size, functional size, etc.
#8) Networks:
Any network (Ethernet, dial-up modems, ADSL, cable modems, LAN or WAN, or any wireless network i.e. Wi-Fi) has a benchmark set for it.
The factors that are considered for benchmarking networks are set as per the KPI’s (Key Performance Indicators) defined for voice and data. The KPI’s include accessibility, retainability, coverage, quality, application throughput, latency, session events, etc
#9) Firewalls:
The firewalls are benchmarked depending on the following factors:
Anti-spoofing filter (blocking specific IP addresses), deny or allow traffic, log traffic for analysis, intrusion detection, latest attack signatures, downloaded content digital signature are verified before download, email, and links in emails, verifying the URLs and filtering them appropriately, accurate authorizations are, etc.
Conclusion
The performance of any deliverable can be standardized using Benchmark testing. The performance quality of the software or hardware system i.e. SUT (System Under Test) can be compared to the benchmarked deliverables (hardware or software) and improvements or changes can be done accordingly.
Benchmark Testing helps an organization to provide specific metrics to measure the quality of its deliverable which adds great value to its product and thus helps in being one of the best in the corporate competition.