On-Premise LLM Deployment for Marketing Personalization: Models, Hardware, Latency, Cost

On-premise LLM deployment means running an open-weight language model on hardware you control, in your own data centre or private-cloud account, so prompts and customer data never reach a third-party model provider. It is justified by data residency, regulated-data scope, sustained volume or latency, and it is a poor fit for low-volume, low-sensitivity work. This guide covers models, licences, serving, hardware and cost for a customer data platform (CDP) team.

Who this is for: architects, ML platform leads and DPOs deciding whether to run models privately for segmentation, content, summarization and agent workloads. You will get a decision table, a licence and model snapshot taken on 17 October 2026, sizing rules of thumb, a cost method (not invented totals) and a deployment checklist. Part of the AI hub.

When private inference is justified, and when it is not

Four drivers recur in regulated programmes: residency or sovereignty requirements that fix where processing may happen; regulated data in prompts such as protected health information, cardholder data or privileged client data; sustained volume, where owned capacity can beat per-token pricing; and latency or availability needs that rule out an external dependency. Terms used here: on-premise is hardware you operate in your own or a colocation facility; private cloud is a single-tenant environment in your own cloud account; self-hosted means you operate the software; sovereign means operators and jurisdiction are constrained by law.

The market data points both ways. Broadcom's Private Cloud Outlook 2026 (9 June 2026, 1,800 respondents) reports that 56% run or plan production AI inference on private cloud, that public cloud as the primary inference environment fell from 56% to 41%, and that sovereignty and residency were the leading geopolitical factor at 54% 1; Broadcom sells private cloud software, so read it as a vendor survey. Menlo Ventures' December 2025 survey of 495 US enterprise decision-makers found open-weight models at about 11% of enterprise LLM usage, down from 19%, with Llama the most widely adopted open model 2. Private inference is a growing architecture choice, but most enterprise LLM usage still goes to hosted models.

Which marketing and customer-data tasks suit private inference

Good candidates are tasks that read personal or confidential text and produce drafts a human reviews: turning a brief into a segment definition, summarizing support or call transcripts, classifying free-text feedback, drafting content variants under brand and legal constraints, explaining an anomaly in a delivery report, and generating embeddings for semantic search. Poor candidates are tasks that need a specific frontier capability, run rarely, or touch no personal data.

LLM data residency: where the prompt, the logs and the weights live

Residency has three parts. Processing location is where the model call runs: on Azure, Global and DataZone deployments may process outside the resource's geography, standard ones stay within it 3. Storage location covers stored data, logs and abuse-monitoring samples, which Azure keeps in the designated geography 3. Weights location matters for private inference: the model files sit in your registry, not with a provider. A hosted endpoint can satisfy the first two and still involve provider-side review of flagged samples 3; private inference removes that path. For country-level rules see data residency requirements by country.

Open-weight model options and licences, as of 17 October 2026

Releases are frequent, so treat this as a snapshot taken from model cards and licence pages. Read the licence file of the exact checkpoint you deploy; licences can differ within one family.

Reading the licences

Serving stacks

In a regulated deployment the serving layer is part of your supply chain. Pin versions, mirror weights and containers into an internal registry, scan images, verify weight hashes, and keep an offline mirror for air-gapped sites.

Reference architecture and hosting options

An on-premise LLM architecture has six parts:

• Gateway: authentication, per-application quotas, PII redaction, routing and logging.
• Serving pool: vLLM or NIM containers on GPU nodes, scaled by model and priority.
• Model registry: approved weights and container images, with hashes.
• Retrieval layer: an index over approved customer-data views, applying the same masking and access rules as the tool servers.
• Observability: latency percentiles, queue depth, GPU memory and error rates.
• Audit ledger: model, weights hash and request metadata per call.

Hosting options differ in who owns the hardware, not in the architecture: your own data centre, colocation racks, GPU instances in a private-cloud account you control, or a sovereign cloud region. Pick by where you can meet residency, power, lead-time and staffing constraints.

Hardware sizing basics

These are rules of thumb for first-pass planning. Measure with your own prompts before buying.

• Weights. Memory is roughly parameters times bytes per parameter: 2 for BF16, 1 for FP8 or INT8, about 0.5 for 4-bit plus overhead for scales.
• Mixture of experts. All experts must be resident, so memory follows total parameters; speed follows active parameters.
• KV cache. Per token per sequence it is about 2 × layers × KV heads × head dimension × bytes per element (read the values from the model's config). It scales with context length and concurrent sequences, and often decides how many requests a GPU can serve.
• Headroom. Serving engines expose a memory-fraction setting; the Mistral Small 4 card's vLLM example sets 0.8 8.
• Decode speed ceiling. Each generated token reads the active weights, so single-sequence tokens per second is at most memory bandwidth divided by bytes of active weights. An 8B model in BF16 (16 GB) on an H100 SXM (3.35 TB/s) gives a ceiling near 209 tokens per second; real figures are lower. Batching shares weight reads across requests and raises total throughput. Prompt processing is compute-bound.

Quantization lowers memory and often raises throughput, and vendors publish FP8 and NVFP4 variants of several models above 733. Quality impact depends on the task, so evaluate the exact quantized checkpoint.

Latency targets for personalization use cases

The figures below are planning targets, our editorial assumptions rather than benchmarks. Validate them with users.

Cost comparison framework

This is a method, and the inputs must be yours.

1. Profile the workload: requests per month, input and output tokens per request, peak concurrency, expected growth.
2. API cost per month = requests × (input tokens × input price + output tokens × output price) + any committed or provisioned capacity fees. Use the vendor's current price page.
3. Owned cost per month = hardware amortized over its life + power and cooling + facility or colocation + software and support (an NVIDIA AI Enterprise subscription if you choose Production Branch NIM 25) + staff time for platform and on-call + security and compliance work + standby capacity for failure.
4. Capacity: measure tokens per second per GPU at your latency target with realistic concurrency; GPUs needed = peak tokens per second ÷ per-GPU rate, plus N+1.
5. Effective owned price per million tokens = owned monthly cost ÷ (tokens served per month ÷ 1,000,000). Break-even is where this equals your blended API price; utilization is the lever.
6. Sensitivity: rerun at 20%, 40%, 60% and 80% utilization, with and without quantization, and with model upgrades every few months (Qwen3.8 and DeepSeek V4.1 both shipped in 2026 1014).
7. Non-price factors: residency, contractual exposure, latency and outage independence, weighted by the risk you carry.

For the wider platform picture see the three-year cost model and per-conversation AI pricing versus owned inference.

Guardrails and evaluation

• Filters reduce risk; architecture limits damage. Redact PII and card data before the model, validate outputs against schemas (vLLM supports structured outputs 21), and use safety classifiers such as the Qwen3Guard models Qwen publishes 33. The limits come from least-privilege tools and approvals, covered in running AI agents on customer data.
• Evaluate on your tasks: golden sets per use case, rerun on every model, quantization or engine change, with a human-reviewed sample and shadow runs before cut-over.
• Frameworks to anchor on: the OWASP Top 10 for LLM Applications 32 and NIST's Generative AI Profile (NIST-AI-600-1, 26 July 2024) 31.
• Record lineage: log model name, version and weights hash with every call; see Explainable AI for Customer Decisions.

Deployment checklist

1. Scope the workloads and data classes; decide which tasks justify private inference.
2. Confirm legal requirements: residency, business associate or processing agreements, DPIA.
3. Shortlist two or three models per task and read each exact checkpoint's licence.
4. Evaluate on your prompts, including the quantized variants you intend to run.
5. Choose the serving engine, pin versions, and avoid archived or maintenance-mode projects.
6. Size memory (weights, KV cache, headroom) and throughput at your latency target; add N+1.
7. Mirror weights and containers into an internal registry; verify hashes; scan images.
8. Isolate the network: endpoints reachable only from the gateway; no internet egress from inference nodes.
9. Authenticate callers with per-application credentials, quotas and rate limits.
10. Redact and validate at the gateway; use structured outputs where possible.
11. Log lineage: model, weights hash, engine and prompt versions per call, without raw personal data in general logs.
12. Monitor latency percentiles, queue depth, GPU memory and error rates; forecast capacity.
13. Plan upgrades: shadow-test a new model, keep the previous weights for rollback.
14. Test failure modes: GPU or node loss, cold-start time, overload; define a fallback to rules.

---

If you are deciding where inference should run for a regulated customer-data stack, the PrivyCDP team can help you test the options against your residency, hardware and operating constraints in a deployment qualification conversation. See also on-premise, private cloud and self-hosted customer data platforms, the data residency requirements by country and the AI hub.