by Dirk-Willem van Gulik
December 2016, v1.09

Background

The important role of open source software in key infrastructures was brought to collective attention by two major security vulnerabilities in the core of the internet infrastructure. Heartbleed and Shellshock of 2014 caused significant concern. It made a lot of people realise how important the collective efforts around these open source infrastructures are. And how much key internet infrastructure relies on open source communities. Such as the Apache community.

Two of those people were Julia Reda and Max Andersson; Members of the European Parliament. As a result they proposed (and directed Europe to fund) a pilot project:  the "Free and Open Source Software Audit (FOSSA)" within a larger workstream that was about "€1 million to demonstrate security and freedom are not opposites".

One part of the money was about developing a methodology; the other about actually auditing some widely used open source software. After soliciting votes from the public - two projects "won": KeePass and the Apache Web Server.

Audit Process

The European Commission (easiest thought of as executive part of Europe) commissioned Spanish Aerospace and Defence company Everis to carry out the review on the Apache HTTPD server (and associated APR).  Their first draft had a considerable number of false positives and a fair bit of focus on some of the more arcane build tools (e.g. our libtool that is used on OS/2 where there is no gnu-libtool). At  Apache vulnerabilty scans are most valuable if we see analysis and at least a theory as to why something is vulnerable -- so we then worked with Everis to improve the report. Their final report on Apache HTTPD and APR has since gone live along with the other audits reports and results.
As none of the vulnerabilities found were particularly severe, we did not need to go through a responsible disclosure path; but could post the issues publicly to the developer mailing list.

Feedback on FOSSA

As part of this work, we were also asked for feedback - especially important now that Julia Reda and Max Andersson have managed to secure a recent vote in the the European Parliament for additional budget.
So in the remainder of this post I'll try to outline some of the conflicting forces around a security issue report v.s. a report of a vulnerability.

Security Reports

Infrastructure software needs constant maintenance to accommodate the evolving platforms; and to back port or propagate improvements and new learnings throughout the code. It is not a static piece of code with 'security holes' waiting to be found. `Fixing' a hole without `lifting the helicopter' is not net-positive by definition; in fact it can be negative. For example if a 'fix' makes the code more complex, if it reduces the number of people that understand it, or if it has an adverse effect on systems that use a different CPU architecture, build environment or operating system.
So in general terms, the main metric is whether security overall gets better - and indirectly about optimising efficient use of the available (existing and extra), but always limited, capacity and capabilities of the resources. At any given time there is both a known 1) backlog of deficiencies and known loose ends and 2) a reservoir of unknown issues. Tackling the first will generally make things more secure. Whereas searching in the latter space only makes things more secure if one finds issues that are severe enough to warrant the time spent on the unknown versus the time not spent on the known deficiencies.
To illustrate this with examples; a report from a somewhat outdated automated vulnerability tool often reduces overall security. Time that could be spent on fixing real issues and cleanups is instead spent on dealing with the false positives and minor stuff. The opposite is also true: bringing a verified security issue to us with a modest bit of analysis as to how such is exploitable, is virtually always a straight win. This obviously is even more true for a very severe issue (where it is immediately clear how it is exploitable). 
But it is also true for the case where someone bestows time on us on a small deficiency (e.g. initially found by a tool) - provided they spend significant time and engineering on handing us the 'fix' on a well tested silver platter. And it is even more useful if a class of issues is tackled throughout; with things like updated test cases.
Throughout this it is very important to consider the threat model and what or whom the bad actors are that you are protecting against. This includes questions like: Is it when the server runs in production? Or also during build? What is the attack surface?. This is particularly important when using (modern!) automated scanning tools (even after you laboriously winnow down the 1000's of false positives for the 1 nugget).
The reason for this is that it is common for constructs such as:
  ....
  results = (results_t *) mallocOrDie(sizeof(results_t));
  results->sum = 0;
  for(int i = 0; i < ptr->array_len; i++) {
    results->sum += ptr->array[i];
  ....
to be automatically flagged by (old-fashioned) tools. This is because there is seemingly no error trapping on mallocOrDie() and because there is no bound checking on ptr->array[i]. So in those cases you need to carefully analyse how this code is used; and what assumptions there are in the API; how exposed it is and so on (e.g is len public or private to the API). 
The last thing you want (when the situation is more complex) is to add a whole load of sentinels to the above code. That would make the code harder to maintain, harder to test and introduce things like the risk of a dangling else going unnoticed. As then you've just reduced security by tackling a non-existent issue. It would have been better to focus, for example, on making sure that mallocOrDie() always bombs out reliably when it fails to allocate.

People and Community versus tools

So specifically this means people, rather than tools, spending a lot of time analysing issues are the thing that is most valuable to Open Source communities.
By the time open source infrastructure code sees use in the market that is significant enough for the likes of FOSSA to consider it 'infrastructure and important' by some metric, it is likely that it is reasonably robust and secure.  As it is open source, it has some standing and is probably used by sizeable organisations that care about security or are regulated. Therefore, it has probably seen a fair bit of (automated and manual) security testing. 
In fact, once an open source project has become part of the landscape every security vendor worth their salt will probably test their tools on it - and try to use it as a wonderful (because you are public) example they can talk about in their sales pitches (that is, if they find something).
It also means that the issues that remain tend to be hard; and are more likely to require structural improvements (e.g. hardening an API) and large scale, systematic changes. Which result in totally disproportional amounts of time to be spent on updating test cases, testing and manual validation. As otherwise it would probably already have been done before. To some extent this also applies to automated tooling; we see that modern/complex tools that are hard to run; require a lot of manual work to update their rule bases for false positives or require sizeable investments (such as certain types of fuzzing, code coverage tools, automated condition testing/swaps) are used less often (but thus tend to sometimes yield promising new strains of issues).

Secondly there is the process of impact and the cost of dealing with the report and changes.  Often the report will find a lot of 'low' issues and perhaps one or two serious ones. For the latter it is absolutely warranted to 'light up' the security response of an open source project; and have people rush into action to do triage, fix and follow up with responsible disclosure.

Given that the code is already open source, the same cannot be said for the 'low' issues. Generally anyone (bad actors and good actors) can find these too. So in a lot of cases it is better to work with the community to file these as bug reports; or even better - as simple issues usually have simple non controversial fixes, submit the fixes and associated test cases as contributions. (It is often less work for the finder of the bug to submit a technical patch & test case than to fully write up a nicely formatted PDF report)

Bug Bounties - a Panacea ?

One 'solution' which is getting a lot of media attention is that of bug bounties; where the romantic concept of a lone open source volunteer coder code the internet is replaced by a lone bounty hunter - valiantly searching for holes & getting paid if they shoot first. 
If we review that solution against the needs of large, stable, communities that deal with relatively mature and stable infrastructure code (as opposed to commercial project or new code that is still evolving) we have seen a number of counter-indications stack up:
  • Fees are not high enough for the expert volunteers one would need to be enticed by the fee alone `in bulk'.

    Take the recent Azure-Linux update reporting or the Yahoo issue as examples. 5 to 10k is unlikely to come even close to the actual out of cost of a few weeks to a few months of engineering time at that quality level (or compensating the years invested in training) that was required to find, analyse and report that issue.

  • The same applies for the higher `competition' fees - topping out at 30-100k. In those cases only the first to report gets it. So your actual payment-per-issue found is lower on average; with some 4 to 8 top global teams at this level and with 2 to 4 high-value target events per year - that works out at well below 8k/teammember per year on average.
That in itself has a number of ramifications:
  • The very best people will only engage in this as a hobby and (hence) for personal credit and pride; OR when they work for a vulnerability company that wants the PR and marketing.

BUT that means that it is personal credit & marketing that is the real driving value, not the money itself. So what then happens if we introduce money into this (already credit and marketing driven) situation? 

  • Very large numbers of people without sufficient skill may be tempted --- but then one has to worry about the impact on the open source community: is dealing with reports at that level a better time spend for volunteers than having insiders look for things ? Will time spent on these fixes distract from the important things ?

    Should we ask people to pre-filter; or ask people managing bug hunting programmes to pre-vet or otherwise carry an administrative burden ? (Keep in mind that there are third party bug-hunting programmes for Apache code that the Apache Software Foundation has no control over).

Secondly - we know (from various dissertations and experience) that introducing money into a volunteer arrangement has an impact on group dynamics and how volunteers feel rewarded; or what work they seek to get rewarded for. 
With that - it may be so that:
  • It is likely that `grunt' and `boring' work in the security area will suffer --- `let that be done by paid folks';
  • It fundamentally shifts the non-monetary (and monetary - but not relevant as too low) reward from writing secure/good code and caring/maintaining --- to the negative - finding a flaw in (someone else) code. So feel-good, job-well-done and other feedback cycles now bypass primary production processes (that of writing good code), or at the very least, make that feedback loop involve a bug bounty party.
Finally - in complex/mature code - the class of vulnerabilities that we probably want to get fixed tend to be very costly to fix/find - and any avenue you go down has a high risk of not finding a security issue but a design/quality issue. 
Bug bounty finders, unlike the coding volunteers are NOT incentivised to report/fix these.  
On top of this, they are more likely to go for the higher reward/lower risk kind of niggle stuff. Stuff that, without digging deeper, is likely to cause higher layers of the code to get convoluted and messy. As these groups have no incentive to reduce complexity or fix deeper issues (in fact, if one were cynical - they have every reason to stay clear of such - as it means ripe hunting grounds during periods of drought).
So at some level Bug bounties are about the trade-off between rewarding, paying, a single person versus saddling a community of motivated volunteers with the fallout - not so much of genuine reports; but of everything else.

So ultimately - it is about the risks of what Economists call "Externalisation"; making a cost affects a party who did not choose to incur that cost - or denying that party a choice how to spend their resources most effectively.

Summary and suggestions for the next FOSSA Audits

In summary:
  1. Submitting the results of automated validation (even with some human vetting) is generally a negative contribution to security. 
  2. Submitting a specific detailed vulnerability that includes some sort of analysis as how this could be exploitable is generally a win. 
  3. Broad classes of issues which (perhaps rightly!) give you hits all over the code base are generally only worth the time spent on them if there are additional resources willing to work on the structural fixes, write the test cases and test them on the myriad of platforms and settings -- and if a lot of the analysis and planning for this work has been done prior to submitting the issue (to generally a public mailing list).

    From this it also follows that narrow and specific (and hence more "new" and "unique") is generally more likely to increase overall security; while making public the results of something broad and shallow is at best not going to decrease security.

  4. Lighting up the security apparatus of an open source project is not 'free'. People are volunteers. So consider splitting your issues into: ones that need a responsible disclosure path; and ones that can go straight to the public lists. Keep in mind that, as the code is open source, you generally can err towards the open path a bit - other (bad) actors can run the same tools and processes as you.
  5. Consider raising the bar; rather than report a potential vulnerability - analyse it; have the resources to (help) solve it and support the community with expensive things; such as the human manpower for subsequent regression testing, documentation, unit tests or searching the code for similar issues. 
  6. Security is a process; over very long periods of time. So consider if you can consistently spend resources over long periods on things which are hard to do for (isolated) volunteers. And if it is something like comprehensive fuzzing, code-coverage, condition/exchange testing -  then consider the fact that it is only valuable if it is; a) done over long periods of time and b) comes with a large block of human manpower that do things like analyses of the results and updates of test cases.
  7. Anything that increases complexity is a risk; and may have long term negative consequences. As it may lead to code which is harder to read, harder to maintain or where the pool of people that can maintain it becomes disproportionally smaller. A broad sweeping change that increases complexity may need to be backed by a significant (5.10+ years) commitment of maintenance in order to be safe to implement; especially if the security improvement it brings is modest.
  8. Carefully consider threat model and actors when you are classing things a security hole - especially around APIs.
  9. Carefully consider what type of resources you want to mobilise in the wider community; and what incentivises the people and processes that are most likely to improve the overall security and safety. And take the overall, longterm, health and social patterns of the receiving community into account when there such forces for good are "external".  It is all to easy to in essence to in effect cause a "Denial of Service" style effect; no mater how well intentioned.
  10. World-class expertise is rare; and by extension - the experts are often isolated. Bringing them together for long periods of time in relatively neutral settings gives synergy which is hard to get otherwise. Consider using a JRC or ENISA setting as a base for long term committed efforts. An effort that is perhaps more about strengthening and improving large scale (IT) infrastructures and (consumer) safety - rather than security.
  11. Bug bounties are not the only option. Some open source communities have benefited from "grants" or "stipend"; where a specific issue got tackled or addressed. In some cases, such as in for example Google its Summer of Code - it is focused on relatively young people; and helps train them up; in other cases it gives established experts room for a (few) year(s) to really bottom out some long standing issue.

With respect to the final point - security engineering (and its associated areas; such as privacy, trust and so on) is a "hard" thing to hire; the market generally lacks capacity and capability. Also in Europe. 

While open source its access to `lots of eyeball's does help; it does not magically give us access to a lot of the right eyeballs.

Yet increasing both Capacity and Capability in society does help. And that is a long process that starts early.

# # #