Synopsis: This report explores the state of security across all of The Apache Software Foundation projects for the calendar year 2021. We review key metrics, specific vulnerabilities, and the most common ways users of ASF projects were affected by security issues.


Released: January 2022


Author: Mark Cox, Vice President Security, The Apache Software Foundation

Background

The security committee of The Apache Software Foundation (ASF) oversees and coordinates the handling of vulnerabilities across all of the 350+ Apache projects.  Established in 2002 and composed of all volunteers, we have a consistent process for how issues are handled, and this process includes how our projects must disclose security issues.


Anyone finding security issues in any Apache project can report them to security@apache.org where they are recorded and passed on to the relevant dedicated security teams or private project management committees (PMC) to handle.  The security committee monitors all the issues reported across all the projects and keeps track of the issues throughout the vulnerability lifecycle.  


The security committee is responsible for ensuring that issues are dealt with properly and actively reminds projects of their outstanding issues and responsibilities.  As a board committee, we have the ability to take action including blocking their future releases or, worst case, archiving a project if such projects are unresponsive to handling their security issues.  This, along with the Apache License v2,0, are key parts of the ASF’s general oversight function around official releases, allowing the ASF to protect individual developers and giving users confidence to deploy and rely on ASF software.  


The oversight into all security reports, along with tools we have developed, gives us the ability to easily create metrics on the issues.  Our last report covered the metrics for 2020.

Statistics for 2021

In 2021 our security email addresses received in total ~18,500 emails. After spam filtering and thread grouping there were 1272 (2020: 946, 2019: 620) non-spam threads.  Unfortunately security reports do sometimes look like spam, especially if they include lots of attachments or large videos, and so the security team are careful to review all messages to ensure real reports are not missed for too long.


Diagram 1: Breakdown of ASF security email threads for calendar year 2021


Diagram 1 gives the breakdown of those 1272 threads.  359 threads (28%) were people confused by the Apache License.  As many projects use the Apache License, not just those under the ASF umbrella, people can get confused when they see the Apache License and they don't understand what it is. This is most common for example on mobile phones where the licenses are displayed in the settings menu, usually due to the inclusion of software by Google released under the Apache License.  We no longer reply to these emails. This is up from the 257 received in 2020.


The next 337 of the 1272 (26%) are email threads with people asking non-security (usually support-type) questions.


The next 135 of those reports were researchers reporting issues in an Apache web site.  These are almost always false positives; where a researcher reports us having directory listings enabled, source code visible, public “.git” directories, and so on.  These reports are generally the unfiltered output of some publicly available scanning tool, and often where the reporter asks us for some sort of monetary reward (bounty) for their report.


That left 441 (2020: 376, 2019: 320) reports of new vulnerabilities in 2021, which spanned 99 of the top level projects.  These 441 reports are a mix of external reporters and internal. For example, where a project has found an issue themselves and followed the ASF process to assign it a CVE (Common Vulnerabilities and Exposures) name and address it, we’d still count it here.  We don’t keep metrics that would give the breakdown of internal vs external reports.


The next step is that the appropriate project triages the report to see if it's really an issue or not.  Invalid reports and reports of things that are not actually vulnerabilities get rejected back to the reporter.  Of the remaining issues that are accepted they are assigned appropriate CVE names and eventually fixes are released.


As of January 1st 2022, 50 of those 441 reports were still under triage and investigation. This is where a project was working on an issue and had not rejected the issue or assigned it a CVE as of the snapshot taken on January 1st 2022.  This number was higher than what we’d normally expect and was due to the large influx of reports that came at the end of December 2021.


The remaining 391 (2020: 341, 2019: 301) reports led to us assigning 183 (2020: 151, 2019: 122) CVE names.  Some vulnerability reports may include multiple issues, some reports are across multiple projects, and some reports are duplicates where the same issue is found by different reporters, so there isn't an exact one-to-one mapping of accepted reports to CVE names.  The Apache Security committee handles CVE name allocation and is a MITRE Candidate Naming Authority (CNA), so all requests for CVE names in any ASF project are routed through us, even if the reporter is unaware and contacts MITRE directly or goes public with an issue before contacting us.

Noteworthy events

During 2021 there were a few events worth discussing; either because they were severe and high risk, they had readily available exploits, or there was media attention. These included:

  • January: A cross-site scripting (XSS) flaw was found in the default error page of Apache Velocity (CVE-2020-13959) which affected a number of public visible websites. Despite a fix being available it then took several months to produce a new release to include the fix, causing the reporter to publicise it early. As a consequence, the security team did a deeper dive through all the outstanding open issues with the affected PMCs to ensure they were all being handled.

  • January: A report was published which showed how malware is still exploiting Apache ActiveMQ instances that have not been patched for over 5 years (CVE-2016-3088)

  • June: The Airflow PMC published a blog about how they handle security issues, how users are sometimes slow to deploy updates (CVE-2020-17526), and how flaws in dependencies can affect Airflow.

  • July: A third-party blog explained how threat actors are exploiting mis-configured Apache Hadoop YARN services

  • August: A researcher discovered a number of issues in HTTP/2 implementations.  The Apache HTTP Server was affected by a moderate vulnerability (CVE-2021-33193)

  • September: A keynote presentation at ApacheCon 2021 discussed the security committee, the US Executive Order on Improving the Nation’s Cybersecurity, and third party security projects such as those under the OpenSSF.

  • September: A flaw in Apache OpenOffice could allow a malicious document to run arbitrary code if opened (CVE-2021-33035)

  • October: A critical issue was found in the Apache HTTP Server. The default configuration protected against this vulnerability, but in custom configurations without those protections, and with CGI support enabled, this could lead to remote code execution (CVE-2021-41773). The issue was fixed in an update 5 days after the issue was reported to the security team, however the fix was quickly found to be insufficient and a further update to fully address it was released 3 days after that (CVE-2021-42013). A MetaSploit exploit exists for this issue.

  • October: The Internet Bug Bounty from HackerOne extended their program to include Apache Airflow, the Apache HTTP Server, and Apache Commons.  Unlike many other programs, this program relies on vulnerability finders following the standard ASF notification process, and allows finders to claim a reward for eligible issues after the fix is available and the issue is public.

  • December: A vulnerability in Log4J 2 (CVE-2021-44228, “Log4Shell”), a popular and common Java logging library, allowed remote attackers to achieve remote code execution in a default and likely installation.  The issue was widely exploited, starting the day before a release with a fix was published.  There is a MetaSploit exploit module for this issue. After the fixed release a few subsequent Log4J vulnerabilities were also fixed, but none had the same impact or default conditions.  

  • December: The ASF is invited to a forum in 2022 around open source security. White House Extends Invitation to Improve Open-Source Security.  We produced a position paper in advance of the meeting.

Timescales

Our security teams and project management teams are all volunteers and so we do not give any formal SLA on the handling of issues.  However we can break down our aims and goals for each part of the process:


Triage: Our aim is to handle incoming mails to the security@apache.org alias within three working days.  We do not measure or report on this because we assess the severity of each incoming issue and apply the limited resources we have appropriately.  The alias is staffed by a very small number of volunteers taken from the different project PMCs.  After the security team forwards a report to a PMC, the PMC will reply to the reporter.  Sometimes reporters send reports attaching large PDF files or even movies of exploitation that don’t make it to us due to size restrictions on incoming email, so please ensure any follow ups are a simple plain text email.


Investigation: Once a report is sent to the private list of the projects management committee, the process of triage and investigation varies in time depending on the project, availability of resources, and number of issues to be assessed.  As security issues are dealt with in private, we send reports to a private list made up only of the PMC. Therefore these reports do not reach every project committer, so there is a smaller set of people in each project able to investigate and respond.  As a general guideline we try to ensure projects have triaged issues within 90 days of the report.  The ASF security team follow-up on any untriaged issues over 90 days old.


Fix: Once a security issue is triaged and accepted, the timeline for the fixing of issues depends on the schedules of the projects themselves.  Issues of lower severity are most often held to pre-planned releases.  


Announcement: Our process allows projects up to a few days between a fix release being pushed and the announcement of the vulnerability.  All vulnerabilities and mitigating software releases are announced via the announce@apache.org list.  We now aim to have them appear in the public CVE project list within a day of that announcement, and even quicker for critical issues.

Conclusion

The Apache Software Foundation projects are highly diverse and independent.  They have different languages, communities, management, and security models.  However one of the things every project has in common is a consistent process for how reported security issues are handled. 


The ASF Security Committee works closely with the project teams, communities, and reporters to ensure that issues get handled quickly and correctly.  This responsible oversight is a principle of The Apache Way and helps ensure Apache software is stable and can be trusted.


This report gave metrics for calendar year 2021 showing from the 18,500 emails received we triaged over 390 vulnerability reports relating to ASF projects, leading to fixing 183 (CVE) issues.  The number of non-spam threads dealt with was up 34% from 2020 with the number of actual vulnerability reports up 17% and assigned CVE up 21%.


While the ASF often gets updates for critical issues out quickly, reports show that users are being exploited by old issues in ASF software that have failed to be updated for years, and vendors (and, thus, their users) still make use of end of life versions which have known unfixed vulnerabilities. This will continue to be a big problem and we are committed to engaging on this industry-wide problem to figure out what we can do to help.


If you have vulnerability information you would like to share please contact us or for comments on this report see the public security-discuss mailing list.