Security Testing

Security testing is important to ensuring the product can live up to the security requirements in ANNEX I. Security testing and reviews are required according to ANNEX I, Part II, (8). Test results assist in demonstrating compliance during conformity assessment.

This page lays out a broad outline on security testing.

Software Composition Analysis (SCA)

Most modern software is build on open source components (dependencies). To fulfill the requirement, that products shall be made available on the market without known exploitable vulnerabilities (ANNEX I, Part I 2.a), requires awareness of vulnerabilities in open source components. SCA can detect if dependencies are up-to-date, contain known vulnerabilities and in license compliance. It can not determine whether the vulnerability is exploitable through your product. SCA tools can work with package lock files or SBOMs. You can think of SCA as npm audit, but working across different package managers and build systems.

We recommend scanning dependencies with an SCA tool such as OWASP dependency-check or Syft+Grype on a regular basis. It can be automated in CI/CD by adding it as a check when merging into release branch.

Static application security testing (SAST)

SAST tools can detect vulnerabilities in source code through static analysis. Traditional SAST uses a combination of pattern, rules and analysis of abstract syntax tree (AST). Many modern SAST tools also utilize LLMs, to detect issues that are hard to write patterns for, and to better explain findings.

SAST can be executed from CLI, build into CI/CD and integrated into the IDE with plugins. IDE integration provides (near) real-time feedback as the code is being written.

SAST is good at detecting potential issues early in the software development life cycle (SDLC), where they are cheaper to fix. It is better at detecting some categories of vulnerabilities than others. Also, SAST commonly report many false positives which can contribute to developer fatigue.

For a list of tools, see Source Code Analysis Tools.

Dynamic Application Security Testing (DAST)

DAST tools can detect potential vulnerabilities in running applications. This category of security testing tools are commonly used for web applications. DAST is a good supplement to SAST. They allow for more frequent testing than manual penetration testing.

We recommend running a SAST tool on release to staging/pre-production.

For a list of tools, see Vulnerability Scanning Tools.

Vulnerability scanners

Vulnerability scanners are automated tools aimed at detecting vulnerabilities on (machine) host level. Periodic scanning of servers and devices can help detect vulnerabilities and expose attack surface visibility before exploitation.

If you develop products based on embedded Linux, it can be an idea to run such a scanner against the device on regular interval (monthly, quarterly or yearly).

Some well-known vulnerability scanners are Tenable Nessus and OPENVAS by Greenbone.

Secret scanning

Secrets such as: default passwords, API tokens, encryption keys etc. often end up source repositories and build artifacts (including binaries and container images). A category of tools commonly known as secret scanners can help avoid accidental disclosure of secrets. These tools can also assist in fulfilling the requirement of secure by default configuration (ANNEX I, Part I 2.b), by detecting default password, certificates etc. It is important to note that these scanners are based on patterns and rules, and therefore provides no guarantee that a secret will be detected. Such tools should be used in combination with rigors manual review.

Penetration testing

Penetration testing (pentest for short) is the most realistic form of security testing, as it aims to attack the product the same way a real threat actor might.

A pentest is conducted by a skilled professional. Often by a consultant from a specialized pentest company. Even if two pentesters follow same methodology, the result will depend on the skill and creativity of the individual penters. A pentest can only show the presence of vulnerabilities, not the absence.

Conducting a pentest is expensive compared to more automated forms of security testing. It is therefore not feasible for most SMEs to conduct a pentest on their products very often.

A pentest can be used to increase assurance that a product lives up to security requirements. It can be done before being put on market, or major modification to design of existing product. Whether a pentest is necessary depends on the risk profile of product (category and risk assessment). It is best used in addition to other kinds of security testing, as a pentest only makes sense late in the development life cycle. Please beware that it is important that enough time is given to the pentester and to developers fixing potential issues, before launching the product.

Word of caution

Running any form of security testing against a live production OT environment comes with a significant risk. The unusual traffic patterns the testing creates can crash devices, thereby affecting availability of the production line. Or worse, resulting in misbehavior violating safety, thereby resulting in risk of damage to personal and/or material.

Vendors and products

Many security testing tools have capabilities that span several of the categories discussed on this page. A none exhaustive list of products/vendors of security testing tools are:

We do not have affiliation with any of these vendors. Nor do we recommend a particular vendor over another. Do your own research.