Why are warmup forks useful in JMH?

Asked 3 months ago
Viewed 5 times

In short my question is:

What do forked warmups offer that within-fork warmups do not? What is the thing that makes executing forked warmups useful?

JMH conveniently allows you to fork a new JVM instance in which your benchmarks are run. This is useful because as explained in this and this question, it allows you to run your benchmakrs with a clean slate: a fresh OS process where the state of both the JVM and the OS/system has no previous decisions/observations for the process which runs the benchmarks.

In addition, JMH allows you to perform a number of initial "warmup" runs of your benchmarks in the forked process, the results of which are discared. These are useful to let the system "settle" with decisions made during execution (OS makes memory paging decisions, JIT compiles VM code to native, etc).

There are actually though two ways this can be done: using @Warmup annotation and using the warmups parameter of the @Fork annotation. As explained in what is the difference between warmup attribute in Fork and Warmup annotation in jmh? the former executes some initial iterations within each fork, whereas the latter executes some entire forks that are discarded.

I am having trouble understanding why the forked warmups were introduced as a feature. A forked process presents a fresh challenge to the the OS/JVM removing any bias that may arise from previously executed code in the existing process. Having the forked process execute enough warmup iterations also means that any "initial optimizations" the OS/JVM perform, are given time to "settle".

So then, why would I want to run some entire forks that are discarded all together by adding @Fork(warmups = N)? I'm guessing it has to do with the OS/system being given time to settle on something (as the JVM does not stick around after each fork to use the information from the previous warm-up forks) but what is it about a process being created/torn-down a few times that helps improve results? And why would a generous @Warmup annotation not achieve the same if a single fork were to be used?

asked 3 months ago

Correct Answer

Warmup forks can do things like:

  • prime directory and file caches by opening and files
  • prime the virtual memory system by causing dirty pages to be written and adding pages to the free pool,
  • and so on.

These things can definitely improve benchmark consistency.

Why do it multiple times?

It is not clear to me why you would want to do that. But having the ability to do it just in case it makes a difference seems like a good thing.

Presumably the warmup fork feature was added because at least some people wanted it / asked for it. And if you can do a single warmup fork run, an option to do it N times is trivial add.

answered 3 months ago