Eric Brewer, Google Fellow, Google
State of Open: The UK in 2022
Phase One: “The Open Source Journey”
The past decade has witnessed explosive growth in open source software (OSS), fostering innovation and efficiency in diverse sectors. However, the “as is” nature of OSS conflicts with the top-down expectations of government projects, necessitating a solution. The proposed remedy is “curation”, wherein intermediary providers enhance raw OSS to meet specific requirements, manage vulnerabilities, and ensure accountability. Curation involves explicit promises, legal responsibility, and diligent tracking of dependencies. Examples like Red Hat and Google Assured OSS illustrate this concept, emphasising the need for curated solutions in government projects to bridge the expectations gap and foster trust in the powerful potential of open source for public sector innovation.
Curation: The Path to Trustworthy Open Source – Thought Leadership
Eric Brewer, Google Fellow, Google
Although Open Source Software has been around for decades, the last decade in particular has seen explosive growth across all sectors and nearly all nations. Open source enables developers to build “on the shoulders of giants” and thus achieve rapid innovation. There are now millions of easy-to-reuse packages in many different languages that enable this innovation. As a consequence Open Source is now used widely by governments and in much of the critical infrastructure of many nations. GOV.UK, the UK government’s platform for hosting government websites, was built using Open Source and its code has been publicly available since 2012. Overall this is a great outcome: citizens and taxpayers benefit from more innovative, more efficient public services.
At the same time, Open Source delivers software “as is” — it literally comes with a licence that says the creators are not responsible for any defects, nor are they liable for any damages.. Many consumers of Open Source do not really understand “as is” and often expect a higher level of service and accountability. But this misunderstanding falls entirely on the consumer.
Conversely, most government projects have “top down” requirements and expectations that are important to creating trustworthy solutions. These requirements are in some sense in conflict with the “as is” nature of open source.
The solution to this fundamental incompatibility is “curation” — the use of an intermediary provider or contractor that provides Open Source solutions that are NOT “as is” and in fact meet the top-down expectations, whatever they may be (and those expectations vary by sector and nation). The curator is building on top of raw “as is” Open Source Software: finding and fixing vulnerabilities, managing technical debt, and building new capabilities. The Open Source software remains the engine of innovation, and the curator’s key role is to bridge the expectation gap.
Curation costs money and it should.
It is hard work to bridge the gap, and it takes both engineers to do the work, plus non-trivial operation expenses to regularly build and test software. In addition, when a vulnerability is uncovered, such as the recent og4j incident, there is a huge amount of work to do to bring the curated solutions back into compliance.
A good curator should be making explicit promises about their solutions, and should be legally accountable for those promises. Similarly, a good curator should not only fix problems in the solution, but track the many dependencies used by their solution; 90% of vulnerabilities in a solution are actually in its dependencies (again log4j is an example).
Curation comes in many forms. Existing examples include Red Hat’s supported libraries for Linux, the new Google Assured OSS product, and corporate versions of Open Source Software projects that come with strong support, such as DataStax version of Cassandra, or Databricks or Cloudera’s version of Spark. Curation will also be layered, with upper layers using curated packages from lower layers (and paying for it). The goal, in progress, is a healthy collection of curators that sometimes work together and sometimes compete.
Overall, both governments and Open Source Software communities need forms of curation.
If we are to unlock the power of Open Source to drive public sector innovation, curation is the key to bridging the expectations gap between governments and Open Source communities and to establishing a level of trust commensurate with the degree of trust we already place in it as a global society.