Reference environments for reproducible research

What are we doing when we publish a (peer-reviewed, computational) paper?

One way to look at it

• We are making assertions about the relationship between some data, some code (software objects) and some results

• We are claiming that others should be able to follow what we have done, and come to similar results

• Even if... they differ in the conclusions drawn from those results

But

There are a lot of reasons why this might not be easy

Here are some classics

The consequences are..

Computational research is hard work to reproduce1,2

Many publications still don't provide code or data3

Those that do may not contain enough detail to reproduce results4

Some responses

There are a lot of tools to enable 'reproducible research'

And there are two conceptions of 'reproducible research'

Both are important..

 and ideally we build a snapshot from the recipe

A simple stepto improve reproducibility

• For every research output, we can produce a minimal software stack required to reproduce some or all of the results
• We call this a reference environment

Reference environments generally include a basic operating system, libraries, tools and data

From a single set of installation/provisioning scripts, we can produce them in different formats

Why?

Because people use different tools and different environments and this is OK

'Single-sourcing' environment configuration means we can deliver reproducible research to the broadest possible audience

We don't have to be limited by 'who uses Docker', or 'who has access to cloud infrastructure'

But (again)

What about the challenges we discussed at the beginning?

• Dependency problems
• Configuration issues
• Environment differences
• Access to data
• Licensing
• Architecture
• Scale

These are all genuine challenges

But they all have mitigating strategies

And they are barriers to all reproducibility of research, not to reference environments alone