Biologists rely on animal models to answer important questions that can’t be addressed with cells in a dish. Often, these animals are deliberately inbred; a less diverse population of animals means that data obtained from experiments with these animals will be less noisy and easier to interpret, so fewer animals are needed for meaningful results. Other “non-model” organisms like hamsters and gerbils are usually outbred. In theory, this means that the animal population being used has greater genetic variation, but in practice, it simply means that siblings are not intentionally mated to create truly inbred strains. In a report published in G3: Genes|Genomes|Genetics, Brekke et al. set out to determine the amount of genetic variation in separate colonies of supposedly outbred Mongolian gerbils (Meriones unguicalatus).

The authors used high-throughput sequencing to identify single-nucleotide polymorphisms (SNPs) in three separate colonies of gerbils. They found that each population had unique SNPs, suggesting genetic divergence between the populations. This is particularly noteworthy because all three colonies were started using gerbils purchased from the same source—so, ostensibly, they should be genetically indistinguishable. The authors point out that the realities of maintaining an animal colony make it impossible for such similarity to be maintained long term; animals are moved, new colonies are started, and logistical and ethical realities all mean that divergence due to genetic drift and founder effects is the rule, not the exception.

That isolated colonies can harbor dramatic genetic differences may contribute to the so-called replication crisis. If two independent researchers believe they are using genetically identical populations—but they’re not truly the same—their results can be inconsistent or unrepeatable. For example, previous reports have stated that genetic diversity in gerbil populations is relatively low, but these reports are biased by divergence in the “outbred” population being studied. While isolated colonies of laboratory animals are susceptible to becoming less genetically diverse, Brekke et al.’s findings suggest that diversity is maintained across many different populations. Effective breeding using different colonies could capture a range of diversity more representative of wild populations.

Although animal models are often described as either “inbred” or “outbred,” perhaps merely labeling a model as “outbred” is misleading. The authors recommend using sequencing to characterize variation within laboratory animal populations before designing studies, allowing the results to be properly interpreted.

CITATION

In response to concerns that have been raised about reproducibility in biomedical research, the Federation of American Societies for Experimental Biology (FASEB) invited experts, delegates from its member societies (including GSA), and representatives from the National Institutes of Health (NIH), National Science Foundation (NSF), and other stakeholders to discuss general factors that may impede the ability to reproduce experimental results.

It should be emphasized that the challenges in reproducing research results are not due to research misconduct nor to poor research design; rather, they generally result from the lack of uniform definitions, insufficient reporting of specific experimental details, and gaps in our understanding of biological systems. Indeed, variations in results often provide key insights into previously unrecognized phenomena and enhanced understanding of biology.

The dialog convened by FASEB is captured in Enhancing Research Reproducibility, which provides recommendations to promote the reproducibility and transparency of biomedical and biological research. The 15 recommendations in the report, which address actions for researchers, institutions, professional societies, journals, and federal agencies are listed below.

Recommendations:

FASEB identified three overarching recommendations that it believes will help enhance the reproducibility and transparency of research overall:

1. Scientists, policy makers, and journalists should use precisely defined terms and definitions when discussing research rigor and transparency to promote uniform understanding.
2. The inability to reproduce research findings among laboratories may result from a lack of sufficient detail in the reporting of critical materials or methods. FASEB
   recommends that professional societies, commercial reagent vendors, scientific journals, and funding agencies engage in a dialog to develop expectations for describing reagents, laboratory tools, and protocols in publications and grant applications.
3. Rigorous and transparent research is dependent upon adherence to good research practices by all research team members, including but not limited to principal investigators/faculty members, staff scientists, postdoctoral scholars, graduate and undergraduate students, core facility staff, and institutional leadership.

The FASEB report also includes focused discussion on some of the research areas that have provided some of the most concern: research using mouse and other animal models—and research using antibodies. Recommendations in these areas are below:

4. In 2010, two independent groups—the National Centre for the Replacement, Refinement, and Reduction of Animals in Research (NC3Rs) and the Institute for Laboratory Animal Research (ILAR) published guidelines to improve the reporting of research using animals, maximizing the utility of published studies and minimizing the need for additional studies. NC3Rs’s Animal Research: Reporting of In Vivo Experiments (ARRIVE) Guidelines and ILAR’s Guide for the Care and Use of Laboratory Animals  and Guidance for the Description of Animal Research in Scientific Publications serve as useful references. FASEB recommends that institutional animal facilities, Institutional Animal Care and Use Committees (IACUCs), and professional societies work together to promote awareness and use—as appropriate—of the ARRIVE and ILAR guidelines in animal research.
5. As their research roles expand or change, research team members may require additional training to understand, implement, and report animal husbandry practices and genetic backgrounds of animal subjects. Access to statisticians or additional statistical training may also be necessary to ensure that animal sample sizes are sufficient for the required analyses. FASEB recommends that training resources be made available to the research community on a regular, on demand basis.
6. The organization, daily operation, environment, and staffing of an animal facility can affect the outcomes of experiments. Therefore, animal facilities and staff should be considered extensions of an investigator’s laboratory and research team. FASEB recommends that investigators include animal facility staff in discussions of relevant aspects of experimental design. FASEB also recommends development of checklists to facilitate review of animal care variables and to denote study-specific variations.
7. Animal models provide critical insights into human biology and health. When communicating to the public about preclinical studies, researchers should clearly articulate the rationale for the choice of an animal model as well as its value and limitations in recapitulating human disease and its treatment.
8. Investigators and animal facility staff ensure humane treatment and care of animal subjects by adhering to established guidelines and federal regulations. To minimize environmental effects on experimental outcomes and continue to improve animal care standards, FASEB encourages institutions, veterinarians, and researchers to identify, understand, and promote the adoption of evidence-based husbandry practices. Reporting of animal care practices and study-specific variations will enhance reproducibility.
9. FASEB recommends that researchers, professional societies, and journals use standard nomenclature when reporting animal husbandry practices, breeding practices, and genetic backgrounds of study subjects in grant applications and publications.
10. FASEB strongly recommends that researchers, journals, and funding agencies work together to develop and adopt a standard format for citing antibodies in grant applications and publications. At minimum, this standard format should include the complete product name, catalog number, antibody type (monoclonal, polyclonal, or recombinant), vendor, target, lot number, and dilution/concentration.
11. There is a growing concern that all lab personnel may not fully appreciate the underlying science or limitations of commercially available antibodies or antibody based kits. Although vendor-supplied technical information may help investigators select reagents such as antibodies, this information is insufficient for validation. Therefore, FASEB recommends that stakeholders—including researchers, funding agencies, product vendors, and journals—convene an impartial advisory body to determine information needed for high quality technical bulletins. This might include immunogenic sequence, epitope sequence, cross-species reactivity, and methodologies for which the antibody is validated.
12. To achieve uniform reporting of research findings, FASEB recommends that investigators, funding agencies, and journals adopt best practices for experiments
    using antibodies. Best practices may vary by field and technique.

Finally, FASEB recommended several strategies that it believes will help increase awareness about these issues among scientists:

13. Rigor and reproducibility are critical for the advancement of science. To improve awareness within the scientific community regarding concerns about the translatability of basic research discoveries to clinical applications, strategies to enhance transparent reporting of research findings, and the new NIH policy that will go into effect on January 25, 2016, FASEB recommends that professional societies emphasize proactive efforts demonstrating the research community’s commitment to rigorous and transparent research. This could take several forms, including editorials in scientific journals, press releases, and development of uniform talking points. These efforts should acknowledge the distinction between strategies to enhance scientific rigor and transparency versus identification of scientific misconduct.
14. To facilitate investigator access to resources intended to enhance scientific rigor and transparency, FASEB recommends the establishment of a publicly accessible clearinghouse website. This website would aggregate existing training resources and best practices to enhance scientific rigor and transparency, such as those developed by NIH, professional societies, and research consortia. A resource hosted by a third-party such as FASEB would allow distribution of uniform information regardless of discipline or funding agency.
15. While the goal of FASEB’s rigor and reproducibility roundtables was to develop strategies to assist individual investigators in addressing the NIH policy that will go into effect on January 25, 2016, several opportunities for journals and society publications committees were also identified. Many journals have taken significant actions to address concerns about rigor and reproducibility. FASEB encourages professional societies to continue to enhance scientific rigor and transparency within their specific disciplines and share alternative strategies developed to address these issues.

The full report is available for download on the FASEB website.

This week, GENETICS and protocols.io are launching a partnership to improve the materials & methods sections of published papers. The journal GENETICS, published by the Genetics Society of America, is encouraging authors to publish detailed methods on protocols.io, in parallel with their article’s publication in GENETICS. Also, as part of the partnership, several accepted manuscripts or articles will be selected each month by the editors of GENETICS, and protocols.io will then digitize these protocols. See the initial set of GENETICS protocols here.)

Tracey A. DePellegrin, Executive Editor of GSA Journals, talks with Lenny Teytelman, founder protocols.io, about this technology innovation.

TAD: Our integration with protocols.io means that protocols – detailed methods – will no longer be simply static PDFs or a series of printed steps. It seems to me that they become multi-dimensional!

Lenny Teytelman

LT: Exactly. Scientists will be able to execute the protocols on the web or on our mobile apps, step-by-step. Most importantly, the authors and readers can keep the methods up-to-date and share tweaks and optimizations on our open access platform.

TAD: Though GENETICS is turning 100 in a few months, a lot of people are surprised that we’ve been early adopters (in many cases, the first – like model organism database linking to articles or publishing early online) of some great technological innovations in communicating science – many of them created by geneticists.

LT: As a geneticist, having gone to countless GSA meetings as a graduate student and postdoc, it’s hard for me to imagine a better journal than GENETICS as the first partnership between protocols.io and a publisher. Through our partnership, many of your methods will be available on mobile phones, and I think this is one of several great examples of efforts on your part to improve the publishing experience and the published results. Together with the recent announcement that references do not have to be formatted on submission and partnership with bioRxiv as examples – I really do think that the GENETICS editors are actively pushing to improve science communication.

TAD: Tell us more about the impetus behind protocols.io. What’s your scientific muse?

LT: In a nutshell, I literally spent the first year and a half of my postdoc at MIT, correcting a single step of the RNA-FISH microscopy method that I was using. Instead of a microliter of an enzyme, we needed 5, and instead of 15 minutes incubation with this chemical, we needed an hour. I discussed this in a lab meeting, and researchers in our lab had to redo their experiments. I called Arjun Raj, the inventor of the method, and he put this info on his website’s FAQ. But the crazy part is that I had no effective way of letting anyone else using this technique know about my discovery. It’s not a new method, but a correction of a previously-published one, and neither I nor Arjun can go to the original Nature Methods publication and put a post-it note on that step, warning people to follow the new instructions. So I get zero credit for this year and a half, but more importantly, everyone else is either getting misleading data or has to spend a year or two rediscovering what I know. My example may be extreme, but it’s an experience familiar to most researchers when something simple that should take a week takes months due to some important tweak of a protocol.

TAD: A big part of making a difference in scientific communication I think does involve connecting scientists with what they need, when they need it. So putting up-to-date information ‘out there’ is important, but you mentioned before that’s only half the battle. Tell me about your approach here.

LT: What we didn’t realize is that more often than not, this knowledge is never even published in the first place. Materials and methods sections are full of “as reported elsewhere”, “we used a slightly modified version”, “detailed protocol available upon e-mail request.” So, I hope that our partnership will shift invisible supplementary protocols into a beautiful web/mobile user interface and will move many valuable protocols out of personal notebooks and into the public domain.

TAD: Two important areas in science right now – transparency and reproducibility – you’ve discussed as being improved with protocols.io. Can you elaborate?

LT: Over the last few years, there has been a lot of attention paid to this issue in academic and industry circles, as well as in the mainstream media. And we have all realized that reproducing work that others have published is very hard. I am glad the issue is in the spotlight, but frankly, some of the proposed solutions scare me. So instead of extra burdens on the scientists who are already overwhelmed, I am a big fan of leveraging technology to both help the researchers and to improve reproducibility smartly. (More thoughts from me on this topic here.) In the case of protocols.io, sharing the methods in fine detail will help others to reproduce the experiments in their lab. And while we all make mistakes, having a central place where we can correct them and share improvements and modifications should also ensure greater efficiency and reproducibility (more on the importance of protocols here and here).

TAD: What’s a typical use case?

LT: Today, we have about a thousand protocols that are private or shared within groups, and 200 publicly-shared methods. So at this point, most of the utility comes from the ability to “run” the public and private protocols, following them step-by-step as one performs the experiment. Many researchers run their protocols on a phone or tablet device directly at the bench, noting any tweaks and deviations in the app. We record a snapshot of exactly how they executed the protocol on that day in a cloud-synchronized journal, so it’s easy to instantly look up the details later, when you are writing the paper or answering the reviewer’s questions. However, the most important use case will happen when we reach critical mass in a year or so.

TAD:  Tell me about that future use case. What have you got up your sleeve for protocols.io in say a year?

LT: We are growing very quickly in terms of content and users, but we are still in the content-seeding stage. It’s critical for us to seed protocols.io with the commonly-used protocols. About 1,000 core techniques in molecular biology cover 70-80% of all the work at the bench. Once we get this initial set of widely-used methods into the platform, the barrier to adoption will be infinitely lower. Scientists will be able to immediately start using and commenting on the protocols, without having to first spend time on entering them in the first place. Getting content has been one of the challenges at the heart of getting our repository off the ground, which is why we are so excited about this partnership with GENETICS. I genuinely believe this partnership is a key step on our path to the ultimate use case – saving somebody a week, month or a year of unnecessary effort because they discovered the correction or optimization on protocols.io.