CPU, GPU, NPU: which chip actually runs your AI application?

An AI application can use a CPU, a GPU and an NPU during the same workflow. The chip’s name is not enough to predict speed or consumption: check what the software actually assigns to it, what still runs elsewhere and the cost of transfers between stages. The first useful question is therefore what work needs to be done.

This article provides a framework for reading a demonstration or a technical specification. It recommends no computer and ranks no manufacturers. Results depend on the model, its numerical representation, the execution engine and the observed workload.

Three roles, with boundaries that depend on the system

The CPU, a general-purpose processor, runs application logic and orchestrates varied tasks, among other roles. A GPU has an organisation suited to many parallel computations. An NPU is an accelerator specialised for certain neural network operations; its actual capabilities vary with its architecture and software.

The CUDA documentation on heterogeneous systems explicitly describes CPU-GPU cooperation: a program starts on the CPU, which can launch code on the GPU and organise memory exchanges. This documentation concerns CUDA and does not describe every architecture, but it is enough to show why “running on a GPU” does not mean “without a CPU”.

For an NPU, the OpenVINO 2025 documentation provides a concrete example of dependence on software, drivers and supported model shapes. Its restrictions are specific to that version and hardware; they should not be attributed to every NPU.

Follow a document from arrival to response

Imagine an application that receives a photograph of an invoice, extracts the information and stores it in business software. Recognition may use an accelerator, but reading the file, resizing it, checking fields and performing the final write also require work.

This is an educational scenario, not an observation of a product. For each stage, it lets you ask: which component executes it, how long does it take and on what data? If downloading or an external call accounts for most of the wait, accelerating the neural computation does not solve that delay.

Stage to observe	Technical question expressed as a concrete need
Preparation	How long before the input is ready for the model?
Inference	Does the exact model work on the advertised accelerator?
Transfer	Must the data change memory location or format?
Validation	Does the result meet the document’s constraints?
Delivery	When does the user receive a usable result?

The guide to multimodal AI and documents (in French) helps define the expected quality. A fast but incorrect extraction is not the same service as a validated extraction.

A simple calculation reveals a speedup with limited effect

Consider a fictional application with a total duration of 100 milliseconds: 30 for model computation and 70 for the other stages. An accelerator makes model computation three times faster without changing anything else. It then takes 10 milliseconds, bringing the total to 80 milliseconds.

The isolated computation is three times faster; the complete application is only 1.25 times faster, since 100 ÷ 80 = 1.25. The values are invented to illustrate an arithmetic breakdown. Any additional transfers would also have to be included.

You can repeat this exercise with measured timings. It indicates where an investment deserves investigation. It does not assume every stage remains constant: a real change may also affect parallelism or preparation. In that case, measure the whole workflow again rather than retain assumptions that are no longer true.

Check the model that runs, not just the one that imports

A model name appearing in a catalogue does not guarantee that all its operations run on the intended chip. Request the effective configuration: engine version, driver, numerical formats, input dimensions and any fallback to another computing unit.

Fallback can be useful for compatibility, but it changes how the test should be interpreted. If part of the work returns to the CPU, the performance belongs to the hybrid system. It should not be presented as an independent measurement of the NPU or GPU.

In the cited OpenVINO 2025 documentation, static shapes and the compatibility of precompiled models across versions have explicit limitations. This illustrates what to look for in the documentation for the chosen configuration, not a prediction about a later version.

Quantisation (in French) introduces another parameter. Two demonstrations using different precisions may be running different variants of the model. Output quality must then be compared, rather than automatically treating the advertised operations as equivalent.

Measure energy over a complete task

Power in watts describes a rate of consumption. Energy per task also depends on duration. In a fictional constant-power measurement, 20 watts for two seconds gives 40 joules; 10 watts for five seconds gives 50 joules. Lower power alone therefore does not establish lower energy per outcome.

On a real machine, specify the boundary: accelerator alone, whole processor or device at the socket. Also describe the starting state, background work and number of tasks. If idle consumption is subtracted, publish that convention and the measurement used.

These calculations do not rank NPUs against GPUs. They show the information needed to examine an efficiency claim. A local measurement lasting a few seconds does not automatically characterise a service operating all day, nor the impacts of manufacturing the equipment.

Prepare a test that supports a decision

Choose inputs that reflect the future use: image dimensions, text lengths, arrival frequency and time constraints. Test the complete result on an identified machine. Document compilation and loading stages separately, because they may matter on first use even if their cost is spread over later runs.

Compare configurations using the same accuracy criterion. Record errors and rejected tasks. Then examine total duration, variability, memory and energy within a consistent boundary. This procedure is a methodological recommendation by Pnebbula, not a hardware certification protocol.

The right choice might be to keep existing equipment, modify the software or use an accelerator for only part of the workflow. Without measurements of the relevant workload, these remain options to test. An acronym on a specification sheet cannot decide between them.

Sources and scope

Consulted on 22 October 2026: NVIDIA, CUDA Programming Model, archive 13.2.2; Intel, OpenVINO 2025, NPU Device. These are manufacturers’ technical sources, used to describe their operation and limitations without adopting their comparative promises. The timing and energy examples are fictional.