Ask a physician why they trust a clinical calculator and the honest answer is often that it appeared in a textbook, a colleague used it, and it has been around for years. Very few could tell you the size of the cohort it was derived from, whether it was ever externally validated, or which populations it was tested in.
That is not a criticism of physicians. The information mostly is not surfaced anywhere they would see it. As health systems researcher and gastroenterologist Shazia Siddique put it, in clinical practice, a lot of the tools we use, we genuinely have no idea how limited they are in their validation. Siddique is now chief medical officer at MDCalc, which is launching a quality-rating system covering the more than 800 clinical tools and calculators doctors use through its site to assess disease risk, transplant eligibility, and more.
To understand why that matters, look at what a systematic review of those tools actually found.
The number that should stop you: cohorts of five
Researchers publishing in npj Digital Medicine collected data on all 690 clinical decision instruments available on MDCalc as of March 2023, each with its own development study. The variation they documented is remarkable.
Development cohort sizes ranged from a minimum of 5 to a maximum of 490,768, with a median of 853. Publication dates ran from 1917 to the present, median 2017. The number of predictor variables ranged from a handful to 48.
Sit with the low end. Some tools that clinicians consult at the bedside, on real patients, in consequential decisions, were derived from studies of a few dozen people or fewer. That does not automatically make them wrong; some clinical relationships are strong enough to detect in small samples, and a tool derived from a small cohort may have been extensively validated since. But the physician using it typically cannot tell the difference, because the interface looks identical whether the underlying evidence is a 490,000-patient cohort or five.
That is the gap the rating system addresses. As MDCalc frames the goal, a clinician pulling up a calculator should be able to see at a glance whether it is evidence-based, endorsed by a professional society, validated across diverse patient populations, and practical enough to actually use under pressure.
The bias problem is specific, not abstract
The rating system's explicit focus on algorithmic bias and population-level harm responds to a documented, mechanical problem rather than a vague concern.
The npj study found the demographics underlying these tools skewed, with White over-representation and Asian under-representation relative to population benchmarks, and the disparities worsen when compared against worldwide rather than US populations. A calculator derived largely from one population may simply perform worse in others, and nothing in the tool tells you so.
Then there is race as a variable inside the equations. MDCalc co-founder Graham Walker described the issue precisely: for decades, race has been included in clinical equations but often used as a surrogate or shortcut for other factors. That is the crux. When a formula includes a race coefficient, it is usually standing in for something else, socioeconomic conditions, access to care, environmental exposure, unmeasured biology, and the shortcut hard-codes a population-level average into an individual patient's result.
The most famous demonstration of how badly this can go is an algorithm that used prior healthcare use as a proxy for medical need, which produced racial disparities in care because people with less access to care generate less spending, which the algorithm read as being less sick. The proxy was reasonable-sounding and the consequence was systematic.
Walker's second point is the one that makes this urgent now: when this gets automated, it scales impact. A flawed calculator used by one physician affects their patients. The same calculator embedded in electronic health records and AI systems applies its flaw to every patient in the system, silently, at machine speed, without anyone re-examining the assumption. Clinical algorithms are becoming infrastructure, and infrastructure is exactly where unexamined assumptions do the most damage.
How they built the ratings, and why the method matters
The criteria were developed using Delphi consensus methodology, a structured process that moves expert opinion through iterative rounds of feedback until genuine consensus is reached, and the team tested the criteria against real calculators already in use rather than only in theory, which helped refine the weighting.
That second detail is more important than it sounds. Rating frameworks designed in the abstract tend to produce elegant criteria that turn out to be unmeasurable or that rate every real tool the same way. Testing against the existing library forces the framework to discriminate among things clinicians actually use.
The system assesses tools across scientific soundness, clinical relevance, and usability and feasibility. Including usability alongside validity is a practical concession worth noting: a perfectly validated instrument requiring 20 inputs will not be used in an emergency department, and a rating system that ignored that would be rating something other than real-world clinical value.
The independent research community has been asking for something like this. The npj authors recommended that platforms implementing these instruments adopt standardized reporting frameworks such as a "model card," augmented with a bias analysis highlighting cohort demographics, external validation data, and known disparities. The rating system is a version of that recommendation, implemented by the platform with the most reach.
The obvious objection
Worth stating plainly: MDCalc is rating tools it hosts, which makes it both the venue and the judge. A rating system controlled by a platform has structural tension, low scores reflect on the library the platform maintains and monetizes.
Several things mitigate that. The methodology is published rather than proprietary. The company launched a nonprofit, the MDCalc Institute, and joined ENGAGE, a $2.6 million initiative coordinated by the Council of Medical Specialty Societies, funded by the Doris Duke Foundation, with NEJM Group and the Association for Clinical and Translational Science as partners, aimed at modernizing how race is handled in clinical equations. Putting the work inside a multi-institution coalition with specialty societies and a major journal is a meaningful check on unilateral grading.
And the alternative is worse. No regulator rates clinical calculators. The FDA's device authority does not reach most of them. Journals publish development studies but do not track how a tool performs a decade later or whether anyone validated it elsewhere. In the absence of any external system, a transparent platform-run rating with independent partners is a genuine improvement over the current default, which is nothing.
What it changes
The realistic effect is not that bad calculators disappear. It is that the evidence behind a tool becomes visible at the moment of use, which changes clinical behavior at the margin. A physician choosing between two risk scores who can see that one was validated across diverse populations and the other was derived from a small homogeneous cohort will, reasonably often, pick the first.
There is a second-order effect worth watching. If a widely used platform starts publicly scoring tools on external validation and demographic representation, that creates an incentive for researchers developing new instruments to meet those standards, because a poorly rated tool will not get adopted. Rating systems shape what gets built, not just what gets chosen.
The broader lesson extends past calculators, and it lands squarely on clinical AI. Medicine is rapidly embedding algorithmic decision support into workflows, and the field is discovering, with these decades-old calculators, that it never built the infrastructure to track how well the tools work or in whom.
Getting a rating framework working on 800 relatively simple, transparent calculators is a useful rehearsal for a problem that is about to get much bigger.