Written by: Wiktor Stefański, Head of People & Operations, Digital Colliers
The FT piece landed last week and the pattern won't surprise anyone who's watched law firms chase AI for the past eighteen months. Several international firms are building bespoke tools to protect their IP. They're using off-the-shelf models for low-stakes work and reserving custom tooling for the sensitive stuff. The split makes sense. The execution almost never does.
The problem isn't the model. It's not the budget. It's that bespoke AI only pays back if your matter data already lives in a classification schema that separates confidential from routine. Most firms skipped that step. They're trying to fine-tune a model on an archive where every draft sits in the same bucket. You can't train on what you can't separate.
The POC Graveyard
We know the base rate. 88% of AI proof-of-concepts never reach widescale deployment. For every 33 POCs a company launches, only four graduate to production. Law firms aren't exempt from that math. The operators I talk to report the same pattern. The vendor demo looks great. The pilot runs on hand-picked matters. Then someone tries to scale it across the practice and the thing falls over because nobody tagged which precedents can safely feed the training data.
The hallucination risk is real. Court cases involving AI-fabricated citations climbed from 87 to over 1,300 in eleven months. That's the visible tip. Most hallucinations don't make it to a sanctions hearing because someone caught them in review. But review is expensive and it scales badly. If you're checking every output line by line you've just built a very expensive spellchecker.
What Matter Classification Actually Means
A classification schema answers one question: which matters can this system learn from and which ones must stay locked down. It's not a three-year IT project. It's a taxonomy with three to five sensitivity tiers and a rule for slotting each matter into one of them.
Most firms already have the raw ingredients. They have conflict-check systems. They have privilege logs. They have matter codes that distinguish M&A from employment disputes. The problem is that none of those systems talk to each other and none of them were designed to answer "can we fine-tune a model on this matter's drafts."
The minimum viable schema looks like this. Tier one is public or near-public work where confidentiality isn't the blocker. Tier two is standard commercial matters with run-of-the-mill confidentiality terms. Tier three is anything with heightened confidentiality, regulatory sensitivity, or reputational risk. Tier four is the vault. Government investigations, high-profile litigation, anything with a super-strict NDA. You need a person with domain expertise to slot each matter into a tier. You can't fully automate that judgment call yet.
The Archive Problem
Once you have the schema you hit the second unglamorous task. Your drafting archive needs a lineage-clean subset. That means knowing which documents were human-written, which ones went through which review gates, and which ones touched which matters. Most document management systems weren't built for this. They track versions and permissions but they don't track whether a draft was attorney work product or a paralegal's first swing or something someone pasted in from an AI tool.
The winning move is to start fresh with new matters. Tag them at intake. Route the drafts through a clean pipeline. After twelve months you have a corpus you can trust. That's long enough to train a useful model and short enough that partners will actually wait for it. Retrofitting the back catalogue is a PhD project. Skip it unless you're drowning in time and budget.
How This Unlocks Fine-Tuning
Once you have classified matters and a lineage-clean archive, bespoke tooling stops being a science experiment. You can fine-tune a model on tier-one and tier-two matters and know you're not leaking tier-three secrets into the weights. You can build a drafting assistant that suggests clauses from your own precedents instead of generic boilerplate scraped from the internet. You can run the assistant inside your perimeter and keep the training data off someone else's servers.
The regulatory tailwind is coming. The UK SRA issued AI guidance back in November 2023. The ABA published Formal Opinion 512 last year, which says you can't bill hours that AI actually saved. The EU AI Act high-risk obligations kick in late 2027 and the fines run up to €15M or 3% of global turnover. Firms that built the classification layer early won't be scrambling when the rules tighten. Firms that skipped it will be explaining to the regulator why they can't produce an audit trail.
The pattern I keep seeing is that partners want the bespoke model but they don't want to fund the prerequisite work. They'll approve the vendor contract but not the six-month classification project that makes the vendor contract useful. That's how you end up in the 95% of enterprise GenAI pilots that deliver zero measurable P&L impact. You bought the car but you never paved the road.