Incredibuild Team
reading time:
There are plenty of vendor claims about SBOM accuracy. There’s less hard evidence.
So we ran a direct test, one we use ourselves when benchmarking Build Guard’s accuracy. Two tools. One build. A well-documented open-source project with a verifiable ground truth. Here’s what happened.
When we were developing our SBOM tool for C++, we needed a reliable way to measure our own accuracy. OpenCV became our primary benchmark project for three reasons.
Syft is one of the most widely used open-source SBOM generators. It’s well-maintained, has broad language support, and is the kind of tool many teams reach for first. We’re not singling it out as uniquely flawed. It’s representative of the source-scanning approach.
Against our OpenCV build, Syft reported 92 components. That sounds comprehensive. Here’s what was actually in the list:
FFmpeg: reported six times
TWhat happened is exactly what you’d expect from a source scanner: Syft scanned every file in the OpenCV repository. OpenCV has Java bindings for Android, CI configuration, and various non-C++ language artifacts. Syft found them all and reported them as software components. This is because from a file-scanning perspective, they are files in the repository.
This is not a bug. Syft is working as designed. The fundamental limitation is that scanning the repository is not the same as scanning what was compiled.This is what creates a lot of false positives and unnecessary noise.
FFmpeg appearing six times deserves its own explanation, because it illustrates the accuracy gap precisely.
FFmpeg is not compiled into OpenCV. It’s an optional, platform-specific, runtime-loaded dependency. On Linux, OpenCV uses whatever FFmpeg the OS has installed. On Windows, it loads a pre-built DLL at runtime, and only if it’s present. If FFmpeg isn’t installed, OpenCV falls back to alternative implementations entirely. FFmpeg never enters the compiled codebase.
In our Windows build, FFmpeg wasn’t installed. It wasn’t compiled. It wasn’t linked. It’s not in the binary.
This is what a false positive looks like in practice. FFmpeg carries real CVEs. When your SBOM reports it as a component of your software, your security team will investigate. They’ll find nothing. They’ll lose time. And they’ll start treating SBOM alerts as noise, which is the most dangerous outcome of SBOM inaccuracy.
Build Guard doesn’t scan the repository. It hooks into the build process and records every file the compiler accessed and every library the linker resolved during the actual compilation.
BuildGuard produced 87 components. At first glance, this looked similar in scale to Syft. But when we examined the results more closely, we found that BuildGuard successfully identified all 12 third-party C++ dependencies, with version and license information included wherever it could be derived from the build artifacts.
The remaining 73 components were internal OpenCV code dependencies that were compiled during build time, and these can be easily filtered out with the tool.
Most importantly, every component reported was real: there were zero false positives. In addition, the output provided a full provenance chain, showing exactly how each component entered the build.
Needless to say this is a big difference.
OpenCV is public. You can run this test without any commercial tooling. Clone the repository. Build it in your environment. Run your current SBOM tool. Then compare the output against the documented third party dependency list.
The exercise is more revealing than most teams expect and it will tell you a lot about how much you can trust the SBOM you’re currently relying on.
If you want to include BuildGuard in that comparison, reach out to the Incredibuild team. We’ll help you set it up and walk through the results together.see.
Table of Contents
Incredibuild empowers your teams to be productive and focus on innovating.
