Fourth Quarter, 2019 update

It’s been a year since we began our project to develop a framework for evaluating the health of open source software used in academic research settings by measuring different aspects or factors of OSS projects, which will help answer questions such as how easy it is for people to contribute to OSS projects and how easy it is to use and deploy the software. After initial research into software evaluation frameworks and a number of meetings and workshops with experts, we have chosen the 20 projects that we will use to evaluate our framework.

All of the projects listed below are used in academic libraries and research labs. The OSS experts we have been collaborating with this past year also contribute to many of these projects, which will make it easier to get feedback about the quality and feasaibility of the factors and continue improving the framework.

  • Archi
  • Archivematica
  • Bioconductor
  • Blacklight
  • Dataverse
  • Districtbuilder
  • DSpace
  • Fedora Commons
  • JabRef
  • Jupyter notebook
  • LOCKSS Lots of Copies Keep Stuff Safe
  • Mirador
  • Omeka
  • Open Journal Systems
  • Parsl
  • R Markdown
  • Scikit-learn
  • Stencila
  • Zotero


Next steps


Our next steps include finalizing the factors in the framework and identifying potential methods for gathering data from these projects for each factor. In some cases, information about the projects can be mined from their GitHub repositories, and we’ve been working closely with the CHAOSS Project team, whose Augur software suite can collect and visualize GitHub data.

Other information might be collected by surveying and interviewing administrators, contributors, funders and end users of the projects, integrating measurement tools in project-contribution interfaces, and various other approaches. We will continue working with our collaborators, including the CHAOSS team, the team behind LYRASIS’s It Takes a Village Guidebook, and the Software Sustainability Institute to develop methods for answering questions such as how diverse and inclusive different project communities are.

In future stages of this project we will evaluate this framework on the 20 projects and work with contributors of the projects to understand how the framework might be improved. Does the framework help groups in academic research settings decide which OSS projects to adopt or continue investing in?

About this project


Our work has been ongoing since November 2018, funded by grants from the Institute of Museum and Library Services (IMLS) and the Alfred P. Sloan Foundation. This past June, we hosted a full-day workshop at Harvard to conceptualize and prioritize meaningful factors that contribute to the health of open source software projects. The workshop included approximately 30 experts from diverse fields and backgrounds and included structured discussions and assessments to identify the factors that will be useful to people adopting or maintaining particular software for academic institutions and libraries.

In the morning, speakers gave talks on a variety of subjects related to open source software and sustainability. Speakers included Mercè Crosas, Sean Goggins, Matt Germonprez, Laura Arp, Megan Forbes, Caroline Jay, Matt Turk, Arfon Smith, Dan Katz, Chase Harrison, Julian Gautier, and Derek Murphy.

In the afternoon, attendees separated into groups to discuss different ways of measuring an OSS project, including evaluating its community, the technical details and user experience of the software, how well it communicates with different stakeholders, its governance, and funding. A group also discussed how we might encourage ethical use of any such framework and prevent or minimize unethical use.

The workshop agenda includes links to presentation slides and collaborative meeting notes.

Then in September, the IQSS team met Sean Goggins and Matt Germonprez from CHAOSS for a full-day meeting to discuss approaches to integrating measurement strategies between our emerging work and the existing CHAOSS work. A key distinction between the focus areas of the two projects is that CHAOSS focuses on open source software in general, including packages with diverse constituencies (e.g. corporate sponsors), while our work focuses on research software, primarily situated in academic, library, and research settings.

We discussed how we could build on the similarities between our projects:

  • In some cases, CHAOSS’s Augur software suite might already be collecting information from OSS project GitHub repositories that our project can use to measure factors of OSS health.
  • Other factors might be measurable using data that can be mined from software repositories and from the software itself, but would require the development of additional tools and techniques.
  • Finally, some metrics might need to be gathered using designed measurement approaches such as surveys and active data collection techniques.

For more information about the project, contact the IQSS team.