Why doubling the number of GPUs does not always halve computation time

Adding GPUs increases available resources, but the program must be able to use them together. Some operations remain sequential, data must move and units may wait for their peers. The speed of a distributed computation therefore depends on how work is divided and on the interconnect, not just the number of chips.

This distinction helps when reading a computing capacity announcement or preparing a test on shared infrastructure. First clarify the objective: solve the same problem faster, process more independent problems or make it possible to run a model that does not fit on a single unit.

Three objectives lead to three different comparisons

Keeping exactly the same workload while increasing resources tests how much it can be accelerated. Increasing the workload at the same time tests the ability to handle more work. Infrastructure can achieve the second objective without halving the duration of the first.

The Cornell course on Amdahl’s law explains this distinction between a fixed-size problem and a larger one. The portion that cannot be parallelised limits the speedup of fixed work. This is a simplified theoretical model, not a result measured on an AI server.

A third case concerns capacity: several GPUs can accommodate a model too large for one by distributing it. In this situation, asking only “how many times faster?” may miss the first intended benefit. The solution should also be compared with another realistic way of executing the same work.

Communication is part of the computation

NVIDIA’s NCCL library provides communication operations between GPUs, within one machine or across several machines. Among them, AllReduce combines results and redistributes them to the participants; other operations gather or broadcast data. NCCL takes topology into account and supports different interconnects, including PCIe, NVLink and networks between machines.

This documentation establishes the role of communication in that environment. It does not justify concluding that a link with a particular name guarantees a specific performance level. The actual route, other transfers and software remain decisive.

To picture the difference, imagine two ways of organising a workload. In the first, each GPU processes an independent image and returns its result. In the second, several GPUs must regularly exchange intermediate results to continue the same task. The second scenario requires more coordination. The precise volume and frequency depend on the program.

A simple bound before the network details

Here is a fictional example. A job takes 100 seconds on one resource. We assume 80 seconds can be divided perfectly, while 20 seconds remain sequential. We deliberately ignore all communication to calculate an ideal limit.

Identical resources	Fictional ideal duration	Speedup relative to 100 s
1	100 s	1×
2	60 s	About 1.67×
4	40 s	2.5×
8	30 s	About 3.33×

The formula is: 20 + 80 ÷ number of resources. Even with an enormous quantity of resources, the sequential 20 seconds remain in this model. Communication cannot be ignored in a real measurement, but its duration should not simply be added if it overlaps with computation: an execution timeline is needed.

The example distinguishes a bound from a measurement. It gives neither the efficiency of a real network nor the parallelisable fraction of an existing model. Those parameters must be observed on the workload being studied.

Bandwidth and latency are not interchangeable

Bandwidth describes a transferable volume per unit of time; latency concerns the delay of an exchange. A program performing many small, dependent exchanges faces a different problem from one moving large blocks. Reducing the number of exchanges can sometimes matter as much as increasing a link’s maximum throughput.

Also examine topology, meaning how the components are connected. Two GPUs installed in the same system do not necessarily have an identical path to every neighbour. The NCCL documentation describes how it adapts to that organisation; this is a reason to measure the actual allocated group rather than a machine that looks similar on paper.

The guide to AI chip memory (in French) describes transfers within the computing chain. Here we add another boundary: results must also cross links between units or machines.

What a scaling curve must keep consistent

To compare one, two and then four GPUs, document the exact work, hardware, software engine, numerical precision and settings. If the batch size changes, disclose it. If a training experiment targets a quality level, verify that level rather than comparing only the time for one iteration.

Two measures answer different questions: elapsed time and resource time consumed. In a fictional example, a 60-second job on two GPUs represents 120 GPU-seconds, compared with 100 GPU-seconds for 100 seconds on one GPU. The delay has fallen, but the amount of reserved resource has increased. This unit does not measure energy: that would require recording power over time.

For a decision, add cost under the actual contract and resource availability. Reservation or queueing delays can matter more than a few seconds saved during execution. Separating these stages helps locate what is actually slowing the project.

Before buying more, locate the wait

Pnebbula’s proposed protocol is to record a timeline that distinguishes preparation, computation, communication and synchronisation. Check whether a unit is waiting for data, whether all units are waiting for the slowest one or whether work is unevenly distributed. An isolated utilisation rate does not explain the cause.

Then test one change at a time: batch size, distribution, data organisation or resource count. Retain a baseline result and the quality achieved. If the program is primarily limited by a sequential stage, buying more GPUs without changing that stage may leave the delay almost unchanged.

European AI Factories (in French) provide context for shared computing facilities. Access to substantial capacity remains distinct from evidence that your application uses it efficiently. Useful evidence is a reproducible measurement with the same objective and an explicit scope.

Sources and scope

Consulted on 22 October 2026: Cornell Center for Advanced Computing, Amdahl’s Law; NVIDIA, Overview of NCCL. The duration and GPU-second calculations are fictional. No computing centre result or commercial cost is attributed to these examples.