The use of Open Source Software has greatly changed the management of software assets especially over the past ten years. Software developers are using open source and commercial third party components, programming routines that have already been perfected by someone else, so they can focus on the hard stuff requiring their innovative skills. In fact, open source components are used in as much as 50 percent of code in all software, everything from desktop applications, SaaS applications all the way the code powering Internet of Things (IoT) devices. Because software vulnerabilities are the most popular attack vector for hackers, open source and commercial, third-party vulnerabilities are particular sources of risk for these applications.
This change in how software is built has lead to a vulnerability and compliance problem for asset managers. This is because the vast majority of software companies lack strong processes and automation in place for tracking and managing these third party components.
Third Party Component Vulnerabilities
All software has bugs and the pace of discovering these bugs is ever increasing. More and more malicious software is taking advantage of known vulnerabilities in third-party software in order to infect or spread an infection. One of the dirty secrets of the software development world is the amount of third-party software components with known vulnerabilities that are present in shipping products.
Recent attacks in the SaaS and IoT spaces have depended on known vulnerabilities in both open source and third-party components. For example, the Equifax breach was related to exercising a vulnerability in the open source Struts2 software component. In the IoT space, the recent Bashlite IoT malware originally used the “Shellshock” vulnerability in the bash shell in order to infect IoT systems. Other attacks target commercial components with known embedded username/passwords in white labeled security camera systems.
Research shows that the while the typical product is made of at least 50 percent third-party software components, it is extremely uncommon to have a list of these components available for security or legal review. The most common “Bill of Materials” for a software product is a blank sheet. Additionally, it is very common to encounter very old vulnerabilities in currently shipping products such as the Heartbleed vulnerability in OpenSSL. The cause of the lack of knowledge of dependencies, alongside a lack of knowledge leading to shipping components with known vulnerabilities, can be traced to a few behaviours in the software development community and their management.
For many years, open source license compliance has taken a back seat to shipping new features or versions of products. This lack of legal compliance, while an important defect in its own right, has led to a situation where companies are shipping hundreds to thousands of third-party software components with little to no knowledge, and no process for periodic review of remediation of discovered problems. This is especially true in the embedded and IoT space, which has the additional complexity of the requirement to ship an embedded operating system in the device. In many cases this operating system is open source, and often with usage terms dictated by an open source license such as the General Public License (GPL). The GPL requires the development team to disclose usage, and disclose the source code for components and products linked against the GPL-licensed components. This, unfortunately, rarely happens in practice.
Comparing Software Practices to Aerospace
With every new attack, the comparisons between the software industry and the aerospace industry are brought up again and again. If we built aircraft the way we build software systems, no one would ever want to take a flight. The recent ransomware attack that took down much of the UK’s National Health Service (NHS) computer systems, shows how software defects related to known unpatched vulnerabilities are becoming a health and safety issue. This change of perception of the risks is causing the software industry to look into treating the design and upkeep of systems in similar ways that the aerospace industry looks to design and maintain the physical aircraft. One of the core tenets of the aerospace industry is inventory and bill of material tracking. This allows them to understand the capabilities and tolerances of the products they build, and at the same time, allow them to trace failures back to root causes. If a bolt breaks and causes a larger failure, the chain of control of that part can be traced back to its order and vendor and even to the specification during the design process.
The software industry, for the most part, does not have this built into its design and maintenance process. Until the Heartbleed vulnerability, it was very rare to have a bill of materials for a software product. Post-Heartbleed, it is more common to have a list of components, but on average the percentage of known components has gone down due to the massive increase of components that are being used.
For many developers, this process is seen as “grunt” work or maintenance, and is not seen as a desirable set of tasks to perform. This attitude among our more senior developers causes a problem where the people who could add the most to the security process, are often routing themselves out of this process.
Additionally, there is often a discounting of the need to keep a list of licenses for compliance purposes by some elements of the developer team. The world of third-party software is ruled by licenses and copyrights, and senior team members must become well versed on the ins and outs of them in order to create a software product that is compliant and whose origins can be traced.
Everyone in the engineering process needs to mindful of security, but those especially who hold themselves up as the engines of productivity and innovation must lead the charge.
One of the roles of the asset manager is to require and review the disclosure lists that accompany any third party software or hard product. If no list is delivered with an asset, it is strongly suggested that the asset manager request one to be provided. Many commercial contracts regarding these assets have language requiring this disclosure, and many of the open source licenses in place also have this disclosure as an obligation on the shipping company.
Understand the Disclosures You Receive
One of the first things a development team must do is to start building the most basic bill of materials. Many teams start by recording all new top-level component requests or usage. This is a good way to build a bill of materials for the newer components, but does not provide much insight into the choices made previously. At some point, a concerted effort must be made to create the inventory of the third-party software that is currently in use. Some companies choose to track only the top-level components, but ignore the lower level dependencies. This level of analysis is easier for legal, engineering and security teams to handle, but is typically only 10-30 percent of the true bill of materials. This can lead to missing dependencies on components with vulnerabilities that are subcomponents of other “top-level” components. Often, targeted analysis looking for well-known vulnerabilities or components like OpenSSL or Struts2 can be a good way to increase the knowledge of the bill of materials, but will leave many components in an undiscovered state. By requiring these disclosures, reviewing them periodically for known issues, and requiring updates and patches for vulnerabilities, the asset manager can encourage their vendors to deliver the most secure applications. Additionally, they can use this information to discover potentially vulnerable applications that are still in use.
Understanding the Scope of How Components Are Introduced
The number of third-party components used in a typical software product grows every year, and an average enterprise application or IoT device contains more than 500 components. While many of these components are selected by a developer, many are brought in automatically by a repository manager like Maven or Yocto. These top-level components and their dependencies are the most common items tracked and managed by the teams, but other components are used as well. The next most common components are monolithic source trees copied into the codebase. This is typically how a library like OpenSSL or zlib is downloaded and introduced into a codebase. These are often found through visual inspection, or by grepping or searching for license files, readme files, known filenames or known embedded strings. Other forms of component use include source code coming from forums or tutorial sites. Components at this level should eventually be tracked as well. Components in both source and binary forms can be reviewed, and those with binary forms should be examined to confirm if the source code used to produce them is available.
The Software Supply Chain
Often, time code coming from outside sources is accepted into the codebase with little question or review – especially if it is coming from a commercial vendor. It is important for developers to have similar high expectations for the code they bring in from others, as the code they write themselves. If they are not receiving a Bill of Materials with the code, this should be treated like a software defect. There are a couple of ways companies are building this awareness into their procurement process. The first is to insert contract language that details the expectations around disclosure of third-party components, as well as the process for receiving notice of patches or upgrades related to third-party vulnerabilities. Additionally, development teams are using targeted analysis for source and binary materials looking for undisclosed content. Typically, two or three undisclosed components, especially if they have either reported vulnerabilities or license issues, are enough to open a dialogue with the outside software vendor. This conversation regarding the undisclosed components is used to stress the importance of receiving the full bill of materials, and also shows that the company is watching out for its interests. Vendors who are not able to deliver the expected level of disclosure should not be used for future work. The philosophy of “it was safe when we shipped it” is no longer sufficient, especially if the code is ending up in devices with full-time network connections and little monitoring. The sooner a vendor can raise the alarm, the sooner a tested upgrade can be pushed to the devices and installed base.
Pulling It All Together
By changing attitudes about who is responsible for security around third-party component usage, educating teams about what can be done to discover, manage and remediate issues involving these components, as well as holding vendors and supplies to the same level of expectations, the modern asset manager can encourage security in an environment where open source and third party software is ever present.