I built a CLI that hashes your ML accuracy claims before the experiment runs
A developer created a command-line interface (CLI) tool called 'falsify' to hash machine learning accuracy claims before experiments are conducted. This tool aims to ensure that accuracy claims are verifiable and not subject to post-experiment modifications. By implementing a pre-registration process similar to those used in psychology and medicine, the CLI helps maintain the integrity of accuracy metrics in machine learning.
- ▪The CLI hashes accuracy claims to prevent changes after results are known.
- ▪It uses YAML canonicalization to ensure consistent hashing of specifications.
- ▪The tool distinguishes between claims that are verified and those that are falsified or tampered with.
Opening excerpt (first ~120 words) tap to expand
try { if(localStorage) { let currentUser = localStorage.getItem('current_user'); if (currentUser) { currentUser = JSON.parse(currentUser); if (currentUser.id === 3889537) { document.getElementById('article-show-container').classList.add('current-user-is-article-author'); } } } } catch (e) { console.error(e); } sk8ordie84 Posted on Apr 29 I built a CLI that hashes your ML accuracy claims before the experiment runs #python #opensource #machinelearning #showdev I built a CLI that hashes your ML accuracy claims before the experiment runs Last month, a customer told me our model's accuracy on their data was 71%, not the 94% we had shipped on the landing page. I went back to the eval notebook. The threshold was still 0.94. The test set was named the same thing.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at DEV.to (Top).
Discussion
0 commentsMore from DEV.to (Top)
