What a Private LLM Deployment Actually Costs a Mid-Sized Law Firm

Most firms that have crossed the private-LLM threshold have already settled the architecture question. They know they want tenant isolation, their own retrieval index, and a model that will not leak a draft complaint into someone else's training set. What they do not know, when the first vendor deck lands, is whether a credible year-one budget starts with a one or a four. The gap between those two numbers is almost entirely a function of five decisions, most of them made before procurement sees a quote.

This piece builds a line-item cost model for a firm of 150 to 500 attorneys, names the levers that move the total by six figures, and sketches the payback math a CFO can defend to an executive committee. It assumes the architectural work covered in the firm's on-prem vs VPC post is already done.

What a Year-One Budget Actually Contains

A private LLM program is not a software subscription with implementation attached. It is five parallel workstreams, each with its own vendor pattern and failure mode. Treating any one of them as a rounding error is how year-two budgets double.

• Infrastructure. GPU capacity, storage, networking, and the observability stack that proves what the model did.
• Model and software licensing. Base model access, embedding models, orchestration frameworks, and any commercial legal corpus a vendor layers on top.
• Integration. Document management, time and billing, matter management, email, and the identity layer that gates all of it.
• Governance. Policy authoring, red-teaming, audit logging, retention controls, and the committee time to approve all of the above.
• Change management. Training, practice-group champions, workflow redesign, and the hours lost while attorneys get fluent.

The relative weights shift by firm, but the shape is remarkably consistent. Infrastructure and licensing together usually consume 45 to 60 percent of year one. Integration and change management together consume most of the rest. Governance is small as a dollar figure and oversized as a risk control.

Infrastructure Is Where the Range Is Widest

The honest answer to "what does the compute cost" is that it depends on model size, concurrency, and whether the firm is renting GPUs or amortizing them. For a 70B-class open-weight model, which is the current sweet spot for firms that want strong reasoning without hyperscaler lock-in, hardware alone is substantial. Enterprise GPUs run roughly $25,000 to $30,000 per H100 and $30,000 to $40,000 per B200 when bought outright, and a production configuration typically needs several of them for redundancy and throughput.

Renting shifts the problem rather than solving it. H100 cloud rates land near $7 to $12 per hour on specialized GPU clouds and $20 to $40 per hour on hyperscalers, with clusters of 8 to 16 H100s capable of exceeding $50,000 per month at steady use. A community-reported benchmark puts an AWS p4d.24xlarge running Llama-3 70B on roughly $287,000 per year at continuous 24/7 utilization. Few firms actually need 24/7, which is why reserved capacity, autoscaling, and off-hours spin-down are the three dials that most reduce real spend.

Storage and networking are smaller in absolute terms but have a sharp edge. Vector databases for a firm's historical work product grow quickly, and egress fees punish any design that routes retrieval traffic across cloud boundaries. Firms that treat the retrieval index as a first-class asset, with versioned knowledge stores and clear provenance, tend to pay less overall because they stop re-indexing the same corpus every quarter.

Licensing, Model Choice, and the Fine-Tune Question

Model licensing splits cleanly along one line: open-weight families carry no license fees but shift the operational burden in-house, while commercial editions bundle support and indemnity at a price. Open-weight families such as LLaMA, Mistral, Qwen, and Gemma carry no license fees but require in-house staff and infrastructure, which is why the apparent savings often evaporate into headcount.

Commercial legal AI stacked on top of the base model is a separate line and the one most finance teams underestimate. Published seat pricing for enterprise legal AI platforms runs well into four figures per attorney per year. One industry comparison reports Harvey AI at $300,000+ annually and CoCounsel at $60,000 to $150,000+ for a representative firm, and other reporting puts Harvey roughly between $1,000 and $2,000 per seat per month at mid-market firms, dropping toward $100 to $200 per seat at Am Law 100 scale. For a 250-attorney firm those numbers compound quickly, and they sit on top of, not instead of, the private hosting bill.

Fine-tuning is the smaller line most people worry about more than they should. A LoRA adapter on a 7B model typically costs $1,000 to $3,000 as a one-off, while a full fine-tune of a 7B model runs $12,000 or more. For most firms, retrieval-augmented generation on a well-governed corpus outperforms a fine-tune and avoids a recurring retraining cadence. Reserving fine-tune budget for one or two genuinely proprietary workflows (a firm-specific drafting style, a niche practice group's taxonomy) is almost always the correct allocation.

Integration Is the Line Item Most Underestimated

A private LLM that cannot see the firm's documents, matters, and time entries is a very expensive chatbot. The integration budget covers connectors to iManage or NetDocuments, the practice management platform, Microsoft 365 or Google Workspace, and the identity provider. It also covers the less visible work: field mapping, permissions mirroring so the model never surfaces documents the user cannot see, and the regression tests that catch when a connector silently changes behavior after a vendor update.

Reasonable budgeting for a 150 to 500 attorney firm sits between $75,000 and $250,000 in year one for integration work, depending on how many systems participate and whether the firm has in-house engineering to co-build. A mid-point estimate assumes four to six integrations, matching the ratios published for broader enterprise AI programs where integrations typically consume 15 to 25 percent of the project.

Workflow orchestration is the companion line. A model that drafts a response memo but cannot route it to the correct reviewer, log the version, and respect the matter's conflict posture creates work rather than removes it. Firms that invest in explicit AI workflow orchestration up front avoid the common failure mode where every practice group reinvents a slightly different pipeline.

Governance Costs Less Than Partners Fear and Matters More

Governance is where the budget feels soft and the risk is hardest. A functional program needs a written AI use policy, a model-risk committee that actually meets, logging that survives a bar complaint, retention rules that match the firm's existing records policy, and a sandbox for new use cases that is isolated from production matters.

Dollars-wise, this is modest: typically $40,000 to $120,000 in year one for tooling and outside counsel review, plus internal time. The leverage comes from avoiding a single disclosable incident. Stanford's reliability study of leading AI legal research tools found that even retrieval-grounded commercial legal AI tools hallucinate at material rates, which is the empirical reason verification workflows and human-in-the-loop review are non-negotiable. Policy-as-code, versioned prompts, and an auditable retrieval log are the controls that let a firm defend its process after the fact rather than reconstruct it.

Change Management Is the Hidden Determinant of Payback

The last line item is the one that most reliably decides whether year two looks like a success. Training budgets of $50,000 to $150,000 for a 250-attorney firm are typical, covering role-based curricula, practice-group pilots, and compensated champion time. Firms that under-fund this line end up with excellent infrastructure that half the associates use for formatting tasks.

The payback math is less speculative than it sounds. If a private LLM saves even 30 minutes per attorney per day on tasks the firm would otherwise bill at a blended rate, a 250-attorney firm recovers thousands of hours per month. Writing down even a conservative fraction of those hours as realized value puts payback inside year one for most mid-sized deployments, provided adoption is real.

The Five Decisions That Swing the Total

Across every engagement, five choices move the budget by six figures or more. They are worth deciding before the first vendor conversation rather than during it.

1. Hosting posture. Dedicated VPC inside a hyperscaler is typically the lowest-friction path. On-prem is defensible for firms with existing data-center footprint and strict residency requirements. A private hybrid deployment splits the difference and is often the right answer for firms with mixed matter sensitivity.
2. Model class. A well-tuned 13B model handles a surprising amount of firm work. Reserving 70B+ capacity for the matters that genuinely need it, via intelligent routing, often cuts compute 40 to 60 percent against a one-size-fits-all deployment.
3. Build vs. buy on the application layer. Buying a commercial legal AI product on top of private infrastructure doubles the licensing line but compresses the integration and training lines. Building compresses licensing and expands engineering.
4. Number of integrations in year one. Each additional connector adds cost and risk. A disciplined year-one scope (DMS, identity, one practice management system) outperforms an ambitious one that misses go-live.
5. Adoption investment. The difference between a 20 percent and 70 percent active-use rate is the difference between a cost center and a return. This is the cheapest line item to grow and the most expensive to skip.

What to Do With the Number

A credible year-one all-in budget for a 150 to 500 attorney firm, built properly, lands between roughly $400,000 and $1.6 million depending on the five decisions above. The lower end assumes a VPC posture, a mid-size open-weight model with selective routing, a focused integration scope, and real adoption investment. The upper end assumes on-prem hardware, a commercial legal AI layer, broad integration, and premium support contracts.

The right next step for most firms is not another explainer. It is a scoped pricing conversation that pressure-tests these line items against the firm's actual matter mix, document volume, and risk appetite. A short diagnostic that maps intended use cases to infrastructure sizing, integration count, and governance posture usually produces a defensible budget within two to three weeks. Firms ready for that conversation can start with a product walkthrough or a direct pricing discussion, with the five decisions above already provisionally answered.