rulekit

R&D spending has outpaced drug approvals.

by Emmanuel Cuyugan

In a 2012 analysis, drug approvals per inflation-adjusted R&D dollar were about 80 times lower than in 1950. This historical trend is known as Eroom's law.

That measure relates research spending to approved drugs across the industry. It does not isolate the causes of the decline or predict the cost of a particular trial.

RuleKit examines how teams review eligibility interpretations and amendment effects. This prototype has not measured cost savings.

The walkthrough uses invented documents and synthetic patients.

Scannell et al., Nature Reviews Drug Discovery, 2012

Source wording, an explicit rule and a traceable result. Missing evidence remains unknown.

01the clinical side

Protocol amendments carry implementation costs.

A 2016 study reported median direct costs of $141K in Phase II and $535K in Phase III for a substantial amendment.

These are amendment costs, not savings demonstrated by RuleKit. The study examined 836 protocols from 15 companies; the direct cost analysis used a subset of 52 protocols.

Sponsors judged 45% of substantial amendments avoidable. That describes the share of amendments, not the share of spending, and does not establish that eligibility logic caused them.

Getz et al., Therapeutic Innovation & Regulatory Science, 2016

“At least 30” includes 30. “Over 30” does not.

Compare the source criterion with its checklist or query, then test the boundary.

A criterion may include a threshold, timing requirements and clinical judgement. Each needs to be represented explicitly or recorded as unmodeled.

source protocol "…participants are required to have an eGFR of at least 30 at the time of screening…"

screening spreadsheet egfr_min 30 ✓

nurse's paper checklist ☐ eGFR over 30

database query WHERE egfr > 30

site training slides eGFR above 30

proposed amendment requirement raised to at least 40

This illustration uses invented documents and synthetic patients. The source includes eGFR 30, but the query uses a strict greater-than comparison and excludes it.

The amendment adds an exclusion. Review the resulting rule set and the affected synthetic cases together.

protocol v3.1 · after the amendment

includeeGFR at least 30

includeage 18 or older

excludepregnancy

excludeeGFR below 45 at screeningadded

Version 3.1 adds an exclusion for eGFR below 45 at screening. The following examples show how it changes the evaluated boundary.

02the engine

Keep source wording beside the rule.

Each YAML rule stores the protocol sentence alongside its executable interpretation.

the engine · criterion representation

Make each comparison explicit.

The synthetic criterion becomes a comparison: fact egfr, operator gte, value 30.

synthetic demo · inclusion I3

“eGFR at least 30 mL/min/1.73m²”

1- id: egfr-min

2 kind: inclusion

3 verbatim: "eGFR at least 30 mL/min/1.73m²"

4 when: { fact: egfr, op: gte, value: 30, unit: mL/min/1.73m2 }

step 1 · one synthetic criterionStart with the original wording of a synthetic criterion.

step 2 · three parts of the comparisonTiming needs its own representation. A threshold alone does not check when a measurement was taken.

step 3 · fact, operator and valueUse egfr for the fact, gte for greater than or equal to, and 30 for the threshold.

step 4 · preserve the sourceThe rule includes the measurement unit and preserves the source sentence in verbatim.

line 3 · source wordingverbatim preserves the source sentence for comparison with the rule.

line 4 · executable conditionwhen holds the executable condition, including the fact, operator, threshold and unit.

human authorshipA person writes and reviews the rule. Criteria that the language cannot express remain visible as unmodeled text.

Version control records changes to the wording and logic. It does not establish which version a site is authorized to use.

the engine · idea one

Keep missing evidence unresolved.

Without eGFR, the renal criterion returns unknown.

one synthetic patient · what the chart contains Synthetic patient: age 63, LVEF 38%, eGFR missing.

step 1 · missing measurementThis synthetic record contains age and LVEF but no eGFR measurement. The renal criterion cannot be resolved from these facts.

a missing-data failure modeMissing evidence is distinct from a negative finding. Treating the two as equivalent can incorrectly pass an exclusion.

rulekitThe criterion returns unknown. Its trace identifies the missing evidence for review.

eligible result?

unknown refused

overall resultEvery criterion must pass for an eligible result. Any failure makes the result ineligible. Otherwise, an unknown leaves it undetermined.

02the engine · idea two

Inspect inclusions and exclusions together.

An inclusion requires eGFR ≥30. An exclusion applies below 45.

rule 1 inclusion

"eGFR at least 30" passes the inclusion at 30 and above.

rule 2 exclusion · added in v3.1

"eGFR below 45 at screening" triggers the exclusion below 45.

two clauses, pages apartEvaluate both criteria against the same measurement. Passing an inclusion does not override an exclusion.

rule 1, drawnThe inclusion passes at 30 and above.

rule 2, drawnThe exclusion applies to values below 45.

the evaluated bandValues from 30 up to, but excluding, 45 pass the inclusion and trigger the exclusion. The overall result is ineligible. This is ordinary exclusion behavior. rules check does not report a contradiction or fail the build.

boundary result

At eGFR 30, this example passes the inclusion and triggers the exclusion in both views.

02the engine · idea three

Compare the same results across tools.

The CLI report and workbench share the evaluator and attrition logic.

Pin the engine version, rules, facts and evaluation date to compare results. Each patient trace shows the criteria behind its result.

chapter 03 · the AI part

Source quotes and human review precede evaluation.

03the AI part

Review proposed facts against their source.

An LLM can propose facts from a note. Each proposal must include an exact source quote and receive human confirmation before evaluation.

The note and patient here are synthetic. A matching quote supports review; it does not prove the proposed value is clinically correct.

Confirm a proposed fact

Synthetic note and patient

LAKESHORE GENERAL (FICTIONAL) · ECHOCARDIOGRAPHY REPORT
patient SYN-042 · study date 2026-03-12 · synthetic: true

CLINICAL CORRELATION
She reports marked limitation of physical activity, comfortable at rest, with breathlessness on climbing a single flight of stairs; symptoms consistent with NYHA class III heart failure. She was hospitalized for decompensated heart failure in November 2025.

nyha_class = III suggestedapproved 0.94 suggested by the AI

quote: "symptoms consistent with NYHA class III heart failure" · found word-for-word in the note above

Approve Edit value Reject review source context before confirming

quote check

eGFR = 280.88

quote: "renal function is severely impaired"

rejected
that sentence appears nowhere in the note

step 1 · synthetic source noteThe synthetic note describes NYHA class III heart failure. The proposal links that value to the relevant passage.

proposal and sourceThe reviewer inspects the proposed value, the exact quote and its context before confirming, editing or rejecting it.

human confirmationConfirmation makes the fact available to the evaluator. This demo stores a generic reviewer label and timestamp locally; it has no authenticated review or durable audit trail.

reject unmatched evidenceAn unmatched quote causes rejection. Model confidence is not a clinical probability or an eligibility score.

chapter 04 · what exists

Trace each cohort result. This example contains ten synthetic patients: seven ineligible, three undetermined and none eligible. Open a result to inspect failed criteria and unresolved evidence. These counts describe the demo cases, not a trial population.

04what exists

Run the prototype locally.

The repository includes a rule engine, CLI reports, an amendment workbench and a fact review component. Bundled demos use synthetic patients and need no account or live model call. There is no EHR connector.

automated regression suites configured for pushes and pull requests

trial rule packs. COMMANDER HF uses public registry text: 12 criteria, 8 executable (2 explicitly partial), and 4 unmodeled.

synthetic patients: 100 generated records and 10 hand-written edge cases.

1 + 1

a CLI report and an interactive workbench with five views, including patient traces and amendment comparison

Inspect results in the workbench

rulekit DEMO-HF-001 10 synthetic patients

Funnel Thresholds Amendment Checks Review

Screening funnelwho gets removed, rule by rule

age 18 or older

"Age 18 years or older"

removes 1

kidney safety

"eGFR below 45 at screening"

removes 4

severity grade

kept as text · needs chart review

3 pending

7 screen fail 3 need a human 0 potentially eligible

An illustrative workbench view of ten synthetic cases. In the workbench, selecting a result opens the patient trace.

Evaluation scope

Automated regression tests cover the engine and interfaces. Real screening workflows and language coverage still need evaluation with clinical teams.

05the build

Review the implementation and its limits.

The next evaluation needs clinical operations experience: realistic screening steps, ambiguous criteria and amendment review requirements.

Development so far uses public registry text and synthetic patients. The project has not been evaluated in a real screening workflow.

YAML rules feed a TypeScript evaluator shared by the CLI and React workbench. Human confirmation is required before AI-proposed facts enter evaluation.

The draft rule format defines behavior that the engine can be checked against. The repository includes regression testskeep fixed bugs fixed for boundary cases and earlier implementation errors. Three examples:

conflict detectionRegression cases in tests/adversarial/conflicts.test.ts check for false positives in the static conflict checker.

consistent countsThe report and workbench share src/core/attrition.ts. Regression tests compare their counts over the same inputs.

missing evidencetests/adversarial/kleene.test.ts checks three-valued logic, including that an absent value stays unknown.

The repository includes regression tests for these failure modes.

A useful next step is one trial pack reviewed with a clinical operations collaborator, including an explicit record of criteria that remain unmodeled.

chapter 06 · close

Source criteria and reviewed facts feed the evaluator. The report and workbench expose the resulting traces.

06close

Help test this against a screening workflow.

⚠ not medical software This prototype is not validated, not cleared by any regulator, and not for clinical, feasibility, or research screening use. The demonstration uses invented patients, notes, clinicians and institutions.

Clinical operations and trial managers: review a public or invented criterion and help define a useful amendment review. Engineers: inspect the rule format, traces and tests. clinical and engineering review

Discuss a clinical workflow · Set up the project

Explore the RuleKit repository

pass fail unknown

Each criterion returns pass, fail or unknown.

models propose·humans confirm·rules evaluate

AI-proposed facts stay outside evaluation until a person confirms them.

by Emmanuel Cuyugan

rulekit · Apache-2.0 · set in Public Sans, Spline Sans Mono & Martian Mono
Trial pack source criteria: ClinicalTrials.gov

see the 3D version · same story, staged as a walk through a research clinic. Needs WebGL2 and a little motion tolerance.