At some level, this becomes impossible. Consider the case of the famous Pentium floating point bug: you not only need to conserve your models, your data, your parameters, your packages, all external packages, the host system or language (say, R) as well as the OS ... plus potentially the hardware it all ran on. Now consider that some results may be simulation based and required a particular cluster of machines...
That's just a bit much for being practical.
With that said, I think more pragmatic solutions of versioning your code (and maybe also your data) in revisions control, storing versions of all relevant software and making it possible to reproduce the results by running a single top-level script may be a "good enough" compromise.
Your mileage may vary. This also differs across disciplines or industry. But remember the old saw about the impossibility of foolproof systems: you merely create smarter fools.
"Reproducible research" as reproducible analysis
Reproducible research is a term used in some research domains to refer specifically to conducting analyses such that
- code transforms raw data and meta-data into processed data,
- code runs analyses on the data, and
- code incorporates analyses into a report.
When such data and code are shared, this allows other researchers to:
- perform analyses not reported by the original researchers
- check the correctness of the analyses performed by the original researchers
This usage can be seen in discussions of technologies like Sweave. E.g., Friedrich Leisch writes in the context of Sweave that "the report can be automatically updated if data or analysis change, which allows for truly reproducible research."
It can also be seen in the CRAN Task View on Reproducible Research which states that "the goal of reproducible research is to tie specific instructions to data analysis and experimental data so that scholarship can be recreated, better understood and verified."
Broader usage of the term "reproducibility"
Reproducibility is a fundamental aim of science. It's not new.
Research reports include method and results sections that should outline how the data was generated, processed, and analysed. A general rule is that the details provided should be sufficient to enable an appropriately competent researcher to take the information provided and replicate the study.
Reproducibility is also closely related to the concepts of replicability and generalisation.
Thus, the term "reproducible research", taken literally, as applied to technologies like Sweave, is a misnomer, given that it suggests a relevance broader than it covers.
Also, when presenting technologies like Sweave to researchers who have not used such technologies, such researchers are often surprised when I call the process "reproducible research".
A better term than "reproducible research"
Given that "reproducible research" as used within Sweave-like contexts only pertains to one aspect of reproducible research, perhaps an alternative term should be adopted.
Possible alternatives include:
- Reproducible analysis:
- Reproducible data analysis
- Reproducible statistical analysis
- Reproducible reporting
All of the above terms are a more accurate reflection of what Sweave-like analyses entail. Reproducible analysis is short and sweet. Adding "data" or "statistical" further clarifies things, but also makes the term both longer and narrower. Furthermore, "statistical" has a narrow and a broad meaning, and certainly within the narrow meaning, much of data processing is not statistical. Thus, the breadth implied by the term "reproducible analysis" has its advantages.
It's not just about reproducibility
The other additional issue with the term "reproducible research" is the aim of Sweave-like technologies is not just "reproducibility". There are several interrelated aims:
- Reproducibility
- Can the analyses easily be re-run to transform raw data into final report with the same results?
- Correctness
- Is the data analysis consistent with the intentions of the researcher?
- Are the intentions of the researcher correct?
- Openness
- Transparency, accountability
- Can others check and verify the accuracy of analyses performed?
- Extensibility, modfifiability
- Can others modify, extend, reuse, and mash, the data, analyses, or both to create new research works?
There is an argument that reproducible analysis should promote correct analyses, because there is a written record of analyses that can be checked. Furthermore if data and code is shared, it creates accountability which motivates researchers to check their analyses, and enables other researchers to note corrections.
Reproducible analysis also fits in closely with concepts around open research. Of course, a researcher can use Sweave-like technologies just for themselves. Open research principles encourage sharing the data and analysis code to enable greater reuse and accountability.
This is not really a critique of the use of the word "reproducible". Rather, it just highlights that using Sweave-like technologies is necessary but not sufficient to achieving open scientific research aims.
Best Answer
Frank Harrell has been beating the drum on reproducible research and reports for many, many years. You could start at this wiki page which lists plenty of other resources, including published research and also covers Charles Geyer's page.