3 major flaws of the black-box approach to security testing

March 9, 2022 by

Ted Harrington

Imagine a castle with a king who wants to know if he could be assassinated. He orders a loyal noble to send some knights to try to break into the castle. He gives no information to those knights about the castle defenses. After all, the king thinks that what he needs is for them to pretend to be his enemies, and his enemies don’t have any of that information.

This is the black-box approach to security testing, the methodology that people frequently request. Unfortunately, they’re usually unaware of its many drawbacks. In black-box, you don’t tell your security evaluators anything about how the system works. The goal of the methodology is to limit information in an attempt to replicate real-world conditions, but it is flawed. 

Download Ted's free ebook, "How to secure your software faster and better."

Get Your Copy

A few weeks later, the king is murdered. His enemies found a secret tunnel that the knights didn’t know about and used it to get to the king. The king knew about this secret tunnel; it was his escape route in the event of a siege. But he never told the knights about it. By intentionally limiting information, the king lost his life. 

Yes, this metaphor is a bit whimsical, but it makes clear why a black-box approach can be counterproductive. By understanding the methodology’s three primary flaws, you and your security team can be more effective in protecting your company’s assets.

Black-box flaw #1: You waste time and money

In our example, the knights spent time figuring out how deep the moat was, how many alligators were in it, and where it would be easiest to cross. But the king already knew all of these things, which means every minute the knights spent trying to figure them out didn’t help the king. 

In this way, black-box security is inefficient. By limiting the information supplied to your assessor, it requires them to invest resources — your time and your money — in obtaining information you could supply in minutes. You literally pay them to figure out the things you already know. Worse yet, it rarely results in the same level of knowledge that would be delivered if you just told them. 

The black-box approach risks your security assessor only providing you information you already know while missing the subtle vulnerabilities that you actually need them to identify. 

Black-box flaw #2: You don’t test your system; you test your partner

The king determined that these particular knights didn’t break in within the amount of time they were allowed, but that didn’t mean that other knights — let alone his enemies — couldn’t. What had he actually evaluated? The knights, not his defenses.

The sneakiest drawback to black-box testing is that you aren’t testing the system; you’re testing your security partner. You determine whether this security expert can compromise this system within this amount of effort. That’s pretty much it. 

Instead, you want to vet the skills of your security partner before you start testing; it should not be the purpose of your testing. If you need to validate the skills of your assessors, there are better and less expensive ways to do that. Security is about finding and fixing flaws. However, black-box methodology is about limiting information, which limits value.

Black-box flaw #3: You get low-value results

Finally, black-box testing provides low-value results. 

If vulnerabilities aren’t found, it does not mean they don’t exist; it simply means that the testing didn’t find them yet. You have no way of knowing if there are other issues or even how close to discovering an issue they may have been. Security is about understanding and minimizing risk. A black-box methodology isn’t well suited to deliver that.

Also, with black-box testing, you don’t get helpful remediations. Your partners don’t know how the system works, so they can’t recommend how to fix any issues they find. You might be able to figure out the solution on your own. However, it puts the onus back on you to do the problem-solving. As a result, you lose the many years of experience that your security partner has with solving problems just like yours. 

Ultimately, you get less value out of what you’re paying for than if you fully informed your security partner going into testing. 

11 courses, 8+ hours of training

Learn cybersecurity from Ted Harrington, the #1 best-selling author of "Hackable: How to Do Application Security Right."

See Courses

An informed security partner is more effective

Imagine instead that the king walks the knights around the castle, pointing out the features of the walls, moats and turrets. As a result, the knights intimately understand the castle defenses and can probe accordingly for weaknesses. They’ll find more vulnerabilities without wasting time and effort on things the king already knows. Better yet, because they understand how all the defenses work together, they’ll be able to advise the king on how to fix any issues they find.

In the same way, informing your security partner about how your system works ensures that you’ll avoid the flaws of black-box testing: wasted resources; testing the partner, not the system and low-value results. 

Instead of black-box, get white-box. White-box is about sharing information. Sharing information maximizes the value you get as a result. White-box testing is faster, more efficient and delivers more valuable results.

Good security is about collaboration, and collaboration delivers much more value than withholding information ever can. By understanding your system, your partner can help you secure it to the best of their ability.