How To Choose Your Hardware Stack for Private Legal AI

Most firm IT leaders start the private AI conversation at the model layer: which open-weights LLM to run, how to retrieve from the DMS, where to put the audit trail. The hardware beneath those choices gets a quick nod and a vendor quote. In practice, the hardware stack is where the economics, the latency SLAs, and the residency posture of a private legal AI deployment are actually decided.

A 70B-class model that answers brief-drafting prompts in two seconds for one partner can take forty for twelve concurrent associates on the same GPU. A rack sized for last year's air-cooled servers may not accept this year's accelerators at all. The intent of this guide is to give managing partners, general counsel, and firm IT a sizing framework concrete enough to issue a defensible RFP rather than defer to a reference architecture written by the vendor that benefits from the answer.

What should the stack look like for your caseload?

Start From Caseload, Not From a GPU Catalog

Hardware sizing is downstream of three numbers the firm already knows or can estimate: the volume of documents that will pass through retrieval and generation in a working week, the peak concurrency of attorney and paralegal sessions, and the residency obligations attached to the matters involved. Everything else, from VRAM budget to networking fabric, is derived from those.

Document volume sets the embedding and indexing load. A litigation shop ingesting several hundred thousand pages of discovery per matter has a very different storage and CPU profile than a transactional group reviewing a few thousand pages of diligence. Concurrency sets GPU count: a single H100-class accelerator can serve one streaming decode of a 70B model acceptably, but KV-cache contention collapses throughput once eight to twelve simultaneous sessions arrive. Residency obligations, in turn, decide whether the stack sits inside the firm's own cage, inside a private tenancy at a hyperscaler, or straddles both. The on-prem versus VPC analysis on this site covers that split in detail; this piece assumes it has already been made and the question is which boxes to buy or rent.

GPU Class and VRAM Budget

The dominant hardware constraint for inference is GPU memory, not raw compute. A reasonable planning rule for self-hosted models is roughly 0.5 GB of VRAM per billion parameters at 4-bit quantization, with FP16 doubling that figure. A 70B-class model therefore wants somewhere between 35 GB and 140 GB of VRAM for weights alone, before any KV cache for long filings or multi-document context.

Three accelerator classes dominate current firm deployments. The NVIDIA H100 SXM5 ships with 80 GB of HBM3 at roughly 3.35 TB/s, with an H100 NVL variant offering 94 GB. The H200 lifts that to 141 GB of HBM3e at 4.8 TB/s, which lets a 70B dense model and a usable KV cache coexist on a single card. The emerging Blackwell generation pushes further, but for most firms the practical choice in 2026 is between H100-class and H200-class hardware.

Translating that to a caseload: a 70B model at FP8 typically needs two to four 80 GB GPUs, or a single 80 GB card with INT4 quantization for lower-concurrency serving. A firm supporting fifty to one hundred concurrent attorney sessions on a 70B assistant should plan on an eight-GPU node, with the choice between H100 and H200 driven by context length. If the workload routinely passes 32K-token briefs, the H200's additional headroom for KV cache is worth the premium; if the median prompt is a short memo query against a retrieved snippet, H100 density often wins on dollars per token.

CPU, Memory Channels, and NVMe Tiers

GPU sizing dominates the invoice, but a stack bottlenecked at the host is a stack that underperforms its spec sheet. Three host-side ratios matter.

First, CPU-to-accelerator ratio. Retrieval, reranking, tokenization, and policy enforcement run on the host. A sensible planning figure is sixteen to twenty-four modern server cores per accelerator in a RAG-heavy workload, enough to keep embedding and chunking from starving the GPU. Second, memory channels and capacity. Eight-channel DDR5 is the current baseline; a 70B deployment benefits from 1 TB or more of system RAM once vector indices, cached embeddings, and OS page cache for document stores are counted. Third, NVMe tiering. A legal AI stack typically wants a hot tier of PCIe Gen5 NVMe for the active vector index and KV cache spill, a warm tier for the document corpus and audit logs, and a capacity tier for cold matter archives governed by the firm's retention policy. Pairing that storage design with a disciplined data retention program keeps the audit surface honest and prevents the hot tier from silently accumulating privileged material that should have aged out.

Networking Fabric and Rack Power

Multi-GPU inference and training both depend on east-west bandwidth. Within a node, NVLink and NVSwitch carry weight shards across GPUs; between nodes, the fabric is typically 400 Gb/s InfiniBand NDR or an equivalent RoCEv2 Ethernet build. Firms with modest ambitions, serving one or two models from a single node, can defer this complexity. Firms planning to fine-tune on their own corpus, or to run frontier open-weights models across multiple nodes, cannot.

Rack power is the quieter constraint. Average data center rack density rose from about 16 kW in 2025 to 27 kW in 2026, with next-generation NVIDIA platforms pushing toward 246 kW per rack. Most law firm colocation contracts were written when 8 kW per rack was comfortable. Before signing a hardware order, firm IT should confirm with the colo provider that the cage can accept the proposed density and that liquid cooling, if required, is supported on the rows allocated to the firm. A stranded order of H200 servers waiting for a cooling retrofit is an expensive form of hedging.

HSM Selection and Confidential Computing

Key custody is the hardware decision that most directly affects whether the deployment survives a client security questionnaire. For firms serving regulated clients, the practical bar is a hardware security module validated at FIPS 140-3 Level 3, which enforces tamper response and identity-based authentication rather than the role-based authentication acceptable at Level 2. That validation level is the same bar certificate authorities use to protect root signing keys, and it maps cleanly to the key management expectations in NIST SP 800-171, whose fourteen requirement families define protection of Controlled Unclassified Information in nonfederal systems.

Confidential computing at the accelerator layer is the newer piece. NVIDIA's confidential computing mode on Hopper, Blackwell, and Rubin GPUs moves model weights and in-use data into a hardware-rooted trusted execution environment without code changes. Independent measurements put the overhead at under 7% for LLM inference, dominated by PCIe bus encryption rather than compute. For firms whose matter mix includes government, healthcare, or financial-services work, a stack that pairs FIPS 140-3 Level 3 HSMs for key custody with GPU TEEs for in-use isolation is now the defensible baseline. These controls sit alongside the broader legal AI governance program; hardware isolation reduces the attack surface but does not replace the policy, logging, and attorney oversight that governance frameworks require.

Model Licensing Changes the Hardware Math

Hardware decisions and model licensing are not independent. A firm choosing Meta's Llama 3.3 70B inherits the Llama Community License, which carries attribution, redistribution, acceptable-use, and scale conditions distinct from Apache 2.0 or MIT. Those conditions do not forbid commercial deployment, but they do require review by firm counsel and, in some cases, flow-down obligations to clients whose data touches the model. A stack sized around a 70B dense model is a different capital commitment from a stack sized around a smaller Apache-licensed model augmented with retrieval and a private LLM fine-tune. The hardware sizing framework in this piece applies to both, but the licensing posture should be settled before VRAM budgets are locked.

A Repeatable Sizing Procedure

The point of the framework is to make the RFP conversation specific. A firm can walk through it in a single session.

Each step produces an output the next step consumes: concurrency drives GPU count, model choice drives VRAM, residency and client mix drive HSM and TEE selection, and the resulting rack power drives the colocation conversation. Teams that work this sequence end up with an RFP that vendors cannot answer with a brochure, and with a configuration that matches the firm's actual caseload rather than a reference architecture written for a hyperscaler's demo.

1. 11. Estimate caseloadWeekly document volume, peak concurrent attorney sessions, residency obligations by matter type.
2. 22. Pick the model classDense 70B vs smaller Apache-licensed model plus RAG; settle licensing posture here.
3. 33. Size VRAM and GPU countWeights plus KV cache at target context length; derive GPUs per node from concurrency.
4. 44. Spec the host16–24 cores per GPU, 8-channel DDR5, 1 TB+ RAM, tiered NVMe for index, corpus, archive.
5. 55. Choose the fabricSingle-node NVLink only, or 400 Gb/s InfiniBand/RoCEv2 for multi-node fine-tune and serving.
6. 66. Lock key custody and isolationFIPS 140-3 Level 3 HSM in the signing path; GPU TEE on for regulated client matters.
7. 77. Confirm rack power and coolingValidate kW per rack and liquid-cooling support with the colo before issuing the order.
8. 88. Issue the RFPVendors answer against firm-specific numbers, not a reference architecture.

What To Buy, and When To Rent

At sustained volume, on-premise inference tends to run 8 to 18 times cheaper per token than cloud APIs. That crossover does not arrive at the first pilot. Firms with uneven demand, or without the colocation and operations footprint to run accelerated hardware in-house, often get further with a private hybrid deployment that keeps retrieval and policy enforcement on firm infrastructure while bursting inference to a dedicated tenancy for peak load. The capital commitment of a full on-prem build is only justified once concurrent sessions, token volume, and the matter mix all point the same direction. For a worked example of where that crossover sits, the piece on private LLM cost for a mid-sized firm covers the arithmetic.

Hardware is not the strategy; the strategy is governed, auditable AI that attorneys will actually use on billable work. The hardware is what makes that strategy feasible at the latency, concurrency, and confidentiality the firm has committed to its clients. Treat the stack as an engineering problem with specific inputs, size it against the caseload rather than against a vendor's reference diagram, and the RFP will come back with answers the partnership can defend.