An AI system helped Pakistani judges clear massive backlogs at $38.50 return per dollar invested

What happened

Researchers from ETH Zurich, the New Economic School, and Imperial College London ran a large-scale field experiment with Pakistan's judiciary. The randomized trial covered 1,559 judges across 118 courts, roughly half of all Pakistani trial court judges. The tool was JudgeGPT, an AI assistant built on OpenAI's GPT-4 and designed for Pakistani trial courts.

The researchers reviewed anonymized chat logs from about 1,500 judges. Legal research, text editing, and text generation were the most common tasks. Around 60 percent of queries sought information about laws, procedures, or legal concepts. Trained judges used JudgeGPT more for editing and summarizing text, tasks where language models are more reliable. They asked fewer broad legal questions, where hallucination risk is higher.

The researchers say the training steered judges toward limited support tasks while leaving the final decisions in their hands. Only about a fifth of requests involved what the researchers call "substantive AI delegation," where judges asked JudgeGPT to evaluate decisions or draft reasoning on its own. Training made judges more likely to decide cases themselves and only use AI to write up their reasoning.

Why it matters

It uses retrieval augmented generation to search a database of 129,235 documents, including 128,292 court rulings and 943 Pakistani laws. When a judge enters a query, JudgeGPT picks the ten most relevant passages and generates a cited answer. The researchers split judges into three groups. One got JudgeGPT access plus targeted training: six 90-minute lectures over three weeks, taught by ETH Professor Elliott Ash after court hours.

Judges learned which tasks suited the tool, where it fell short, and how to check its output. A second group got the same AI access but only a general seminar on technology and law. The control group attended that seminar with no JudgeGPT access. AI access alone did little. Judges with targeted training used JudgeGPT four times as much as those in the general seminar group.

After 40 weeks, trained judges averaged nearly 60 logins and over 200 prompts. The comparison group averaged about 20 logins and fewer than 50 prompts. Districts with more trained judges resolved more cases. 3 percent bump. Even districts in the bottom quartile still cleared about 616 more cases. Ruling quality held steady or improved.

The appeal rate per 1,000 resolved cases fell slightly, and judges worked the same hours with no change in work-life balance. 50 per dollar invested, based on what it would cost to hire enough extra judges to match the same output. Even conservative estimates put the return at "at least" $10 per dollar. A review of roughly 4,000 court judgments found more AI-flagged text, as expected.

But readability, length, and the number of legal arguments held steady. An LLM-based quality check, validated by two Pakistani lawyers, showed a slight improvement. Rulings from trained judges were rated better in 59 percent of pairwise comparisons, up from 42 percent in the control group. The study found no evidence that AI use increased gender or religious bias in judicial language.

What to watch

The researchers stress that their findings don't support replacing judges with AI. What the experiment shows, they say, is that AI can boost public sector productivity, but only when paired with training that steers users toward the right tasks. Without it, most of the gains disappear. The productivity gains in this study are likely a floor, not a ceiling.

JudgeGPT ran on GPT-4, a pre-reasoning model that was the best fit for the project at the time. Today's reasoning models write better, hallucinate less, and handle complex tasks more reliably.