1. Generate evidence to support some claims 2. Don't automate the creation of said evidence Congratulations, you've just contributed to the reproducibility crisis! #reproducibility #openscience

So much brilliant work never makes it into a paper. The code, the data, the long nights helping others debug. At pyOpenSci, we believe that code, data, and community are the pulse. Research advances quickly when we build together & openly. Join us. 💛 bit.ly/pyos-volunteer #openscience #opensource

Don't be ashamed of "messy" code. If it works, it's good. Share it. #openscience #reproducibility

Reading through some slides from 2013 titled "how to succeed in reproducible research without really trying". It's true we have all the tools needed for researchers to build their own reproducible workflows, but still many do not. Maybe the tools are still too hard to learn and use!

Programming tip: Name classes after the data they encapsulate, not the actions they perform on that data. For example, instead of SchemaProcessor, just call it Schema: processed_schema = Schema().process() #programming #oop #softwareengineering

Hot take: Notebooks are fine in production as long as they're part of a reproducible pipeline docs.calkit.org/notebooks/ #reproducibility #datascience #openscience

Please don't number your scripts. Refer back to (2) and use a pipeline (like Calkit's of course)! www.nature.com/articles/d41... #reproducibility #automation #openscience

In a newly released arXiv preprint, we explore how open science practice like sharing data, code and preprints relate to citation impact in French-authored research over a 3-year period. Thanks to @ouvrirlascience.bsky.social for highlighting its national importance. 🔗 Read more: plos.io/3Vmykrj

Reproducibility tip: Any figure, dataset, ML model, etc., should not be shared until it is produced with an automated, version-controlled pipeline. #reproducibility #openscience

While profiling some CUDA code on a SLURM cluster I realized I was not working in a very reproducible way, which could become a problem down the road if I ever needed to know how a certain result was generated, so Calkit now has SLURM integration: docs.calkit.org/pipeline/slu...

Julia should have an option that automatically does the same thing as: export JULIA_LOAD_PATH=@:@stdlib julia --project=. -e 'using Pkg; Pkg.instantiate()' before running any command. Maybe an enhanced reproducibility mode option, like --repro? #julialang #reproducibility

I don't know why, but I always found it hard to remember the process for adding an SSH key to GitHub, so I made a wizard for it: calkit config github-ssh (might be buggy, but still an improvement over manually running commands from the docs)

Moving pieces of code farther apart from each other (into different packages, modules, repos) doesn't guarantee you've decoupled them. In fact, if you haven't, you probably just made your life a whole lot harder. #softwareengineering

If you're a leader of knowledge workers you should be giving teams fewer, vaguer goals. Handing out well-defined projects and tasks to individuals is a waste of their abilities.

How much "waste" do you have in your scientific workflow? For example, do you manually rerun plotting scripts/notebooks after updating processing logic? Do you then manually re-upload these figures to Overleaf? Want to automate this stuff away? Reach out and I will help!

Code, data, config files, etc. all must be shared in order to describe computational methods with sufficient detail. #reproducibility #openscience

Anyone have any good references that examine the relationship between computational reproducibility and time to publication? I'd assume more automated, reproducible workflows will help studies get through peer review more quickly. #openscience #reproducibility

Calkit now has its own GitHub Action to run your project's pipeline and optionally commit and push results: docs.calkit.org/tutorials/gi...

In software engineering, it's well established that it's worth the effort to automate testing and deployment, but in research, it's not. That is, research computations are almost never fully automated. There's almost always some list of manual steps like data transfer or interactive work involved.

Hot take: YAML is fine.

Graphic design is not my passion. Anyone want to collaborate on an infographic to explain the value of fully automating and version controlling research projects? docs.google.com/drawings/d/1... #openscience #reproducibility #automation

📢 Calkit now supports Julia environments and pipeline stages. Here's an example project: github.com/calkit/examp... #julialang #openscience #reproducibility

Interactive workflows are great for doing things once, not so great for doing things twice. #reproducibility

Naming files like 01_do-something.sh, 02_do-something-else.sh is a bit of a code smell. The order of execution in a project should be automated, not left to the user to follow a sequence of file names.

Calkit 0.27 introduces parameterization for notebooks run in pipeline stages, and these can even be iterated over: github.com/calkit/calki...

Getting started with Julia. First impression is that there is a lot done interactively from the REPL, which feels weird and antithetical to reproducibility. It's also strange that Julia will load packages from both project-level and system-level environments simultaneously.

To prevent being scooped: Instead of working in a closed, non-transparent way, why not make it easier to collaborate? #openscience

Too few researchers work reproducibly because it costs too much for too little benefit. We need to make it easier in such a way that helps researchers get papers out the door faster.

Put succinctly, Calkit's goal is to make "repro pack" READMEs unnecessary #openscience #reproducibility

Classes should be named after the data they encapsulate, not the actions they perform #softwarenengineering

When you describe the computational methods in your paper without sharing the code and data: #openscience #reproducibility

One major cause of the reproducibility crisis is that the way we describe computational methods in research articles has not kept pace with the increase in complexity of the methods themselves. Human language and mathematical formulas are not adequate in many cases.

"One button" reproducibility should be the standard #openscience #reproducibility

Snakemake users, tell me what you like and don't like about it. Is it too complex for non-software-oriented researchers? What's missing?

Tips for using Jupyter notebooks as part of a reproducible workflow (one that goes from raw data to research article with a single command): docs.calkit.org/notebooks/ #openscience #jupyter #reproducibility

If you're asked to review a journal article and don't have access to the code and data, or do but can't run it, do you reject the article? #openscience

Why don't scientists work reproducibly? All they need to do is learn an ever-evolving suite of complex tools built for software engineers, learn software engineering and architecture best practices, automated testing, etc. Super easy stuff and fun for everyone! #openscience

The "interest rate" on tech debt is proportional to how much will depend on it. If it's a function only used internally, not sweat. If it's an interface to a library you plan to have used by millions, you should really try to get it right the first time. #programming #software

You don't need to run every single thing you write through ChatGPT

Closed science is rent seeking behavior

How can we help researchers fall into the "pit of success" when it comes to computational reproducibility? I don't think I've come across a single code/data repo that's actually reproducible with a single command. blog.codinghorror.com/falling-into...

Calkit now has its own pipeline syntax that forces you to define an environment for every stage, but manages those environments for you automatically. No more pip installs, Docker builds, etc. Your project will just be reproducible. Docs: docs.calkit.org/pipeline/ #reproducibility #openscience

reproducible + (lots of effort) != reproducible

Engineering is simply predicting what will work and what won't.

"Restoring gold standard science" Lots to say here. It's amazing this administration would accuse others of undermining trust in science, but nevertheless, requiring all agencies to do open science is a good thing. Let's see if it actually happens. www.whitehouse.gov/presidential...

If you share all of your code and data but users can't rerun with a single command (including environment setup), that's commendable, but your project is not reproducible. In other words, don't write a non-automated "pipeline" as a list of manual steps in your README! #reproducibility #openscience

@russpoldrack.org's Substack led me down a reproducibility rabbit hole this morning, ending at doi.org/10.1190/1.18... One qualification I've been using for a project to be considered reproducible is that it only takes a single command (no lists of instructions for humans in the README!)

Does every Python developer go through a phase of overusing/abusing classes?

I have a hunch that when it comes to #reproducibility, the more automated a project is, the more reproducible it will be. Automation >> documentation www.nature.com/articles/d41...