What is the Difference Between a High-quality and Low-quality Penetration Test

TL;DR:

• Penetration testing is often misunderstood and confused with vulnerability scanning.
• Low-quality pentests are characterized by low cost, lack of thorough process, poor reporting, and minimal remediation guidance.
• High-quality pentests are more expensive but offer thorough testing, detailed reporting, customer service, and effective remediation assistance.
• A high-quality pentest considers the context of the target, involves thinking like an attacker, provides detailed reports, and offers custom remediation guidance.
• Software Secured's Penetration Testing as a Service (PTaaS) and Pentest 360 are recommended for high-quality pentests.

Evaluating your security is as important as implementing security. There are various types of security testing such as Vulnerability Scanning, Bug Bounty Programs, and Risk Assessment. Our focus in this post is on one of these types - Penetration Testing.

If you search for penetration testing (pentesting) services on the internet, you’ll find exhaustive results. With so many options, how will you decide what’s best for you? How will you decide what’s good enough? That’s what we’ll address in this. We’ll first see how the term penetration testing is misrepresented and then look into the characteristics of a low and high-quality pentest.

Penetration Testing is a Misrepresented Term

Cybersecurity is a huge domain and has several sections under it. With this exhaustive list of sections, concepts, and terminologies, it’s common for one to misunderstand terms. Penetration testing is one such term in the security testing category. Understanding what is the difference between a high-quality and low-quality pentest is crucial in making informed decisions.

Many people do not know the difference between a vulnerability scan and a penetration test, amongst other security testing concepts. Some people use the terms vulnerability scan and penetration test interchangeably. But that’s far from the truth.

A vulnerability scan is a test performed using automated tools to scan the target to identify vulnerabilities. A penetration test is a simulation of an attack trying to break the target’s security to evaluate its security. Vulnerability scans use a database of known vulnerabilities and check if the target contains these vulnerabilities. These tools are only as good as the database. Therefore these tools can’t detect zero-day vulnerabilities. Additionally, a penetration test considers the context, includes an attack plan, and focuses on exploiting a discovered vulnerability. Many companies use the term “penetration testing” incorrectly, just because a human “fact checks” a vulnerability scan, does not mean this is a penetration test.

If that makes the difference clear, we’re not done yet. Another difference one should know is the difference between the qualities of a penetration test. Because penetration testing is so widely used, it can be hard to tell if you are getting a low-quality or a high-quality pentest. So let’s look into how you can differentiate between them.

Identifying Characteristics of a Low-Quality Pentest

Cost Discrepancies

We all love a good deal. But when it comes to a pentest, the quality of the pentest is directly proportional to the cost. So, if the deal is too good to be true, it’s likely that the quality is low. “Why?”, you ask. Because a high-quality pentest requires resources for thorough testing and that doesn’t come cheap.

The price of a pentest is also proportional to the test size. For example, let’s say you have a massive application and they give you a quote for $1200, likely, they’re just using automated tools or not being thorough enough, and someone is creating beautified reports for you. Doesn’t look like a high-quality “pentest”, does it? The bottom line is that the cost of a pentest is one of the factors to determine its quality and the cost of a low-quality pentest is on the lower side.

Thorough Process Implementation

As mentioned earlier, pentesting is far from merely running automated scans. The process of a “pentest” is a crucial factor. The process of a low-quality pentest is usually along the lines of running an automated tool. A slightly better version is to run the tool, and then have a person (“pentester”) verify the results from the automated tool. This is a tiny bit better but still not good enough. There’s a huge difference between the verification of a scan and a pentest. You will get fewer false positives and they can claim they “manually pentested” the application but they did not.

Even with advanced and sophisticated tools, there’s still a lack of that human element. A human will see the whole picture and have instincts when something doesn’t look right. Tools cover the breadth but a pentester looks at it like a maze and focuses on the depth as well. Understanding the target, context, and logical workflows plays a crucial role in pentest, and tools lack in this aspect.

Detailed Reporting Standards

The pentest report is equally important as the test itself. Low-quality pentest will not deliver a detailed, descriptive, and customized report for the findings. You might receive a report autogenerated by a tool, a report written by another 3rd party tool, not the pentesting company or even a human. And to make it look like a high-quality pentest, minor issues are sensationalized. This selling of artificial value reduces the overall value of the pentest report and impacts the outcome of the test.

Effective Remediation Techniques

Cheaper options will point you to the references generated by the 3rd party tool and/or tell you to do your research. They’re not typically inclined to sit down and explain their findings on how to properly fix them. Applying a fix that doesn’t properly fit your system might not completely fix the issue or it might create even more security weaknesses.

Now that we know some indicators of a low-quality pentest and what to avoid, let’s look into the characteristics of a high-quality pentest.

Recognizing Characteristics of a High-Quality Pentest

Consideration of Costs

Several factors determine the cost of a pentest such as the size of the target, scope, methodology, experience, endpoints, test duration, and providing recommendations for fixing found security weaknesses. Each of these comes at a price.

You would want the test to cover both breadth and depth while testing so that you can get the most out of the test. This takes time, resources, and effort. So it’s natural for the cost of a high-quality pentest to go up. But bear in mind, that high-quality pentests come at high costs but that doesn’t mean all high-cost pentests are of high quality.

Emphasis on Customer Service

Pentesting is not a one-sided activity. The pentesting side and the product owner side have to come together before and after the pentest. Before the pentest begins, expectations have to be set, there is an exchange of information. After the pentest, report sharing and helping the teams fix the issues. All of this might need both the aforementioned parties to come together on calls or meetings and communicate several times throughout the process.

High-quality pentests focus on customer success and customer service equally high as the pentest itself. These pentesting teams make communication easy and efficient because they understand its impact on the overall success of the task at hand. The turnaround time for meetings, calls, emails, etc., is good. They try to understand your problem and consider it while pentesting your product.

Thorough Process Implementation

A high-quality pentest involves thinking and acting like an attacker. There might be different ways to break into a system and might also require someone to think outside the box especially if the target has some security mechanisms in place.

A penetration tester has to gather information about the target, identify potential weak points, think about the tactics, and custom attack cases that target specific goals. Threat modelling is another crucial part of security testing because you consider the context of the target to identify and understand threats.

Without understanding the target, its context, and how it works, the pentest might not be rigorous. And a high-quality pentest should cover all these aspects. You might have to do some sessions with the pentesters and show demos, but all of this will reflect in the outcome. Another factor of a high-quality pentest process is retesting after the team has fixed the security issues to confirm the mitigation.

Detailed Reporting Standards

High-quality pentests result in descriptive and informative reports. It’s not just about what a pentester found but also how they found and exploited a security weakness. This again involves the context keeping the target system/application in mind. Different applications/industries have different PII to be determined for each report, and some things should be included and others omitted based on context.

High-quality pentest reports should have in-depth reporting, such as our portal features: how many vulnerabilities were found, comments, mitigations, and guidelines for your vulnerabilities. A high-quality pentest report should also include the severity of the security issue found. The baseline is having CVSS. Such information helps in prioritizing the mitigation.

SLAs are based on severity and recommendation. This might change from one organization to organization depending on the client's data and context. General SLAs are as follows:

• CRITICAL (5 business days)
• HIGH (asap but within 30 days)
• MEDIUM (90-180 days)
• LOW (around 180 days)

The number of vulnerabilities discovered during a penetration test can serve as a key indicator of its quality and thoroughness. Experienced penetration testers typically uncover a significant number of vulnerabilities in most systems they assess. Industry data suggests that high-quality penetration tests often identify more than 10 vulnerabilities in a given target environment. This benchmark implies that penetration tests revealing fewer than 10 vulnerabilities may be of lower quality or less comprehensive in their approach. Software Secured identifies an average of 26 vulnerabilities per penetration test, with 20% being critical and high severity. However, it's important to note that the raw number of vulnerabilities should not be the sole measure of a test's effectiveness. The severity, potential impact, and relevance of the identified vulnerabilities also play crucial roles in determining the overall quality and value of a penetration test.

Effective Remediation Techniques

High-quality pentest providers will be more inclined to sit down with you and thoroughly explain findings in a language that everyone can understand, especially the ones that will have to work on fixing the issue. Providing the context of the attack, the tactic, and the thought process of an attacker will not just help in fixing the issue but will also help in designing and building more secure systems and applications in the future.

These are some major factors that one should look for in a high-quality pentest.

Analyzing Pentest Quotes: Impact of Scope, Complexity, and Size on Quality and Cost

When evaluating pentest quotes from multiple providers, it's crucial to consider the scope, complexity, and size of the engagement to ensure a fair comparison. These factors directly impact the depth and breadth of the assessment, influencing the overall cost and value of the service. Be wary of significantly low prices, as they may indicate inadequate expertise or a limited approach, such as relying solely on automated vulnerability scans rather than comprehensive manual testing. To make an informed decision, obtain quotes from several reputable providers, allowing you to compare pricing structures and service offerings. This approach helps identify the best fit for your specific security needs and budget constraints, ensuring you receive a high-quality penetration test that aligns with your organization's requirements.

Drawing Conclusions

Pentesting is one of the commonly misrepresented terms. And there’s even more obscurity in the understanding of a high-quality and a low-quality pentest. We addressed what a pentest is and how it is different from its commonly misrepresented term, vulnerability scan. Then we understood how to differentiate a high vs low-quality pentest based on cost, process, reporting, and remediation. Exploring what is the difference between a high-quality and low-quality pentest is crucial for making informed decisions.

Here’s a quick summary of the factors of a high-quality pentest:

• It is more costly due to resources, time, and effort.
• Considers the context of the target and attackers’ thought processes in attacking the system.
• Reporting is detailed along with helping with prioritizing the fix.
• Help with custom remediation along with an insight into the attack process.

In addition to high quality, there are also some aspects that the company should take care of such as regular audits and reminders to resolve SLA.

If you’re looking for a high-quality pentest, you might want to look at Software Secured’s Penetration Testing as a Service (PTaaS) and Pentest 360.

ed’s Penetration Testing as a Service (PTaaS) and Pentest 360.