The 2017 paper that introduced the transformer architecture reported training its larger model on eight NVIDIA P100 GPUs for three and a half days. That is roughly 672 GPU-hours, an amount of compute a single researcher could book on a cloud provider today for a few hundred dollars.

Seven years later, DeepSeek published the training budget for DeepSeek-V3: 2.788 million H800 GPU-hours. That is about 4,100 times the compute of the original transformer, and DeepSeek-V3 was widely reported as one of the cheap frontier-class models.

Almost every number in this space is an estimate, and the people producing the estimates say so. What follows separates the figures that come from primary disclosures, the figures that come from careful third-party modelling, and the figures that are guesswork dressed up as reporting.

What the bill actually consists of

People say “training cost” and mean at least four different things.

• Final training run, amortized. The share of hardware depreciation plus electricity attributable to the one run that produced the released model.
• Hardware acquisition. What it cost to buy the cluster in the first place, most of which serves many runs over several years.
• Full development cost. The final run plus failed runs, ablations, data work, and researcher salaries.
• Cloud-equivalent rental. What it would cost to rent the same compute at list prices, which is what most public “cost” numbers actually measure.

These differ by more than an order of magnitude, which is why two credible sources can quote wildly different figures for the same model without either being wrong.

The most careful public estimates

The best-documented work here is from Epoch AI, published as The Rising Costs of Training Frontier AI Models (Cottier et al., 2024) and summarised on their blog in June 2024. They modelled costs three ways rather than picking one, which is the main reason to trust the shape of their results even where you doubt the precision.

Their headline findings:

• Amortized hardware and energy cost for frontier final training runs has grown at roughly 2.4x per year since 2016, with a 90% confidence interval of 2.0x to 2.9x.
• GPT-4 carried the largest amortized hardware and energy cost in their sample, estimated at around $40 million.
• Gemini Ultra came in near $30 million on the same basis.
• Hardware acquisition cost is one to two orders of magnitude higher than the amortized figure. Their estimate for the GPT-4 training cluster is around $800 million.
• The cloud-rental method produced a similar growth rate, 2.6x per year, but costs roughly twice as high.
• If the trend holds, the largest training runs pass $1 billion by 2027.

Every one of those is an external estimate. Neither OpenAI nor Google has published an audited training cost for any frontier model. Treat the numbers as well-argued modelling with stated uncertainty, not as disclosure.

Where the money goes

The breakdown is more interesting than the total, and it surprises most people.

Component	Share of full development cost
AI accelerator chips	47–67%
R&D staff compensation	29–49%
Server components (non-accelerator)	15–22%
Cluster interconnect	9–13%
Energy	2–6%

Two things stand out. Salaries rival hardware. A few hundred researchers at frontier-lab compensation is genuinely comparable to a GPU cluster, which explains the aggressive hiring markets of the past few years.

And electricity, the thing that dominates public discussion of AI cost, is a rounding error in a training budget. The energy story is real, but it is an infrastructure and grid story rather than a line item that decides whether a model gets built.

The one budget a lab actually published

DeepSeek broke ranks. The DeepSeek-V3 technical report includes a table with GPU-hours by phase: 2,664K for pre-training, 119K for context extension, and 5K for post-training, totalling 2,788K H800 GPU-hours. At an assumed rental price of $2 per GPU-hour, they put the total at $5.576 million. The model has 671 billion total parameters with 37 billion active per token, trained on 14.8 trillion tokens.

The authors attach an explicit caveat that got lost in most coverage: the figure covers only the official training of DeepSeek-V3 and excludes prior research and ablation experiments on architectures, algorithms and data. It is the cost of the successful run, not the cost of the programme that produced it. The rented-price assumption is also theirs, not a market quote.

Meta disclosed differently. The Llama 3 paper reports that the 405-billion-parameter model was trained on up to 16,000 H100 GPUs, each rated at 700W, using 3.8 x 10^25 FLOPs on 15.6 trillion tokens. No dollar figure, but the physics is instructive: 16,000 GPUs at 700W is 11.2 megawatts of accelerator draw alone, before servers, networking or cooling. The paper also notes power fluctuations across the data centre on the order of tens of megawatts as GPUs synchronise.

The line item nobody can price: data

Both disclosed training runs above used a similar quantity of text. DeepSeek-V3 pre-trained on 14.8 trillion tokens; Llama 3 on 15.6 trillion. What neither paper says is what that text cost to obtain.

This is the least transparent part of the whole budget. Epoch AI’s cost breakdown covers accelerators, servers, interconnect, energy and staff, but does not isolate data acquisition as a category, and no frontier lab publishes what it pays for licensed corpora. Content licensing agreements between AI companies and publishers have been announced repeatedly since 2023, but terms are almost always confidential, so any specific per-deal figure you see reported should be treated as sourcing from someone who was not a party to the contract.

There are also real costs that never appear as a licence fee: deduplication and filtering pipelines, human annotation for post-training, and the specialist contractors who write demonstrations and preference rankings. Some of that hides inside the staff and contractor lines. Some of it sits with vendors whose invoices nobody outside the labs sees.

The honest summary is that data is a genuine cost, plausibly a large one for post-training in particular, and there is no public figure worth quoting.

The electricity question, in proportion

The International Energy Agency’s Energy and AI report, published April 2025, put global data centre electricity consumption at about 415 TWh in 2024, roughly 1.5% of world electricity, split approximately 45% United States, 25% China, 15% Europe. It projects consumption more than doubling to around 945 TWh by 2030, comparable to Japan’s current total output, with AI named as the most important driver.

The report is careful about proportion. Data centres account for around one tenth of global electricity demand growth to 2030, less than industrial motors, air conditioning or electric vehicles. In advanced economies specifically, the share of demand growth exceeds 20%, and US data centre consumption is projected to overtake aluminium, steel, cement, chemicals and other energy-intensive goods combined by the end of the decade.

Note what that growth is mostly about. Training runs are episodic. Serving billions of daily queries is continuous.

Why inference is the cost that matters now

You will hear that inference has overtaken training as the dominant AI cost. I could not find an audited figure confirming that for the industry as a whole, and anyone quoting a precise ratio is probably guessing. But the structural reasons are solid and worth understanding.

A training run is paid once and amortized across the model’s entire service life. Inference is paid per request, forever, and scales with adoption rather than with ambition.

Reasoning models changed the arithmetic sharply. OpenAI’s reasoning documentation confirms that internal reasoning tokens consume context window space and are billed as output tokens even though callers never see them. Output tokens typically cost five times input tokens. A single hard question can now generate thousands of billed tokens the user never reads.

Serving is also technically awkward. Epoch AI’s June 2025 analysis of inference economics found network latency, not bandwidth, is the critical bottleneck for fast generation, and that on H100 hardware pipeline parallelism becomes impractical when serving faster than roughly 24 milliseconds per token. You cannot buy your way to arbitrary speed; past a point you are paying more per token for less efficiency.

The price collapse, measured properly

Comparing list prices across years is misleading because model quality changes. The right question is what it costs to reach a fixed capability level.

Epoch AI ran that analysis in March 2025 across six benchmarks including MMLU, GPQA Diamond, MATH and HumanEval. Their finding: the price to reach a given performance level fell between 9x and 900x per year depending on the benchmark. For GPT-4-level performance on PhD-level science questions, the price fell roughly 40x per year.

Current list prices show the resulting spread. Anthropic’s published API pricing as of 2026 runs from $1 per million input tokens for Claude Haiku 4.5 up to $10 for the Fable and Mythos 5.1 tier, with output at five times input across the range. OpenAI’s pricing page shows a similar spread, from $0.20 per million input tokens for its smallest current model to $10 for its flagship.

That is a 50-fold price range within a single vendor’s catalogue. Both vendors also bill cached input at roughly 10% of the standard rate and offer around 50% off for batch processing.

What this means for you

If you are building a company on top of these models rather than training them, the cost structure has several practical implications.

• Training a frontier model is not a startup strategy. With final runs at tens of millions amortized, clusters in the hundreds of millions, and staff costs matching hardware, this is a capital game. Epoch’s billion-dollar-by-2027 projection makes that clearer, not less.
• Fine-tuning an open-weights model is a different order of magnitude. Adapting an existing checkpoint costs thousands to low six figures, not tens of millions.
• Your dominant cost is inference, and it grows with success. Model it per active user before you price your product, not after.
• Model selection is your biggest lever. A 50x price spread within one vendor means routing easy requests to a small model and hard ones to a large model is usually a bigger saving than any prompt optimisation.
• Use caching and batching. Roughly 90% off repeated prefixes and 50% off asynchronous work are the two largest discounts available, and both are published, not negotiated.
• Do not build a business on today’s prices staying still. They have fallen fast, which is good for buyers and brutal for anyone whose margin is a markup on tokens.

Frequently asked questions

How much did GPT-4 cost to train?

Nobody outside OpenAI knows. Epoch AI’s modelling puts the amortized hardware and energy cost of the final run near $40 million and the acquisition cost of the cluster near $800 million. Those are estimates with stated uncertainty, and full development cost including salaries and failed runs would be higher again.

Was DeepSeek-V3 really trained for $5.6 million?

That is the figure in DeepSeek’s own paper, and it is real for what it measures: 2.788 million H800 GPU-hours at an assumed $2 per hour. The paper explicitly excludes prior research and ablation experiments. It is the cost of one successful run, not of the research programme.

Is electricity the main cost of AI?

Not of training. Epoch’s breakdown puts energy at 2 to 6% of full development cost. Electricity matters enormously at the level of grid planning and data centre siting, but it does not drive the training budget.

Why do inference prices keep falling?

Better hardware, better serving software, smaller distilled models reaching yesterday’s quality, and competition among vendors. Epoch measured declines of 9x to 900x per year for fixed capability levels depending on the task.

Will training costs keep rising forever?

Unclear, and reasonable people disagree. Epoch’s 2.4x annual trend points to billion-dollar runs by 2027, but efficiency gains of the kind DeepSeek demonstrated cut the other way. The trend is an extrapolation, not a forecast anyone is committed to.

Reading the numbers you will see quoted

When someone cites a training cost, ask which of the four costs they mean, who produced the estimate, and whether it includes failed runs and salaries. Most viral figures are amortized final-run estimates presented as if they were total programme costs, which understates them, or cluster acquisition costs presented as if they were per-model, which overstates them.

The verifiable picture in 2026 is this. Frontier training runs cost tens of millions in amortized compute and energy, sit on clusters costing hundreds of millions, and are roughly half hardware and half people. Costs have risen about 2.4x a year for a decade. Electricity is a small fraction of that bill and a large fraction of the public conversation.

Meanwhile the cost of using a model of any given quality has collapsed by one to two orders of magnitude annually. For everyone who is not training frontier models, that second trend is the one that decides what is buildable.

Sources

• arXiv — The Rising Costs of Training Frontier AI Models (Cottier et al.)
• Epoch AI — How Much Does It Cost to Train Frontier AI Models?
• arXiv — DeepSeek-V3 Technical Report
• arXiv — The Llama 3 Herd of Models
• International Energy Agency — Energy and AI, Executive Summary
• Epoch AI — LLM Inference Price Trends
• Epoch AI — Inference Economics of Language Models
• Epoch AI — Notable AI Models Dataset
• Anthropic — Model Pricing
• OpenAI — API Pricing
• OpenAI — Reasoning Models Guide
• NeurIPS Proceedings — Attention Is All You Need

Image credit: Photo: Trower, NASA — Public domain (via Wikimedia Commons)