Improving Indexing Performance By Adding Threads And Cores

How can indexing performance by improved?

The Retain Server is multithreading and will use additonal cores. For every additional core added beyond 8, the Indexer begins to launch a corresponding thread:

For example, when there are 5 cores, Retain Server has 4 threads and the Indexer has 3 threads.

Cores to Threads

Cores	Retain Server Threads	Indexer Threads
2	1	3
3	2	3
4	3	3
5	4	3
6	4	3
7	4	3
8	4	4
9	4	5

One thread will be added per CPU/core and the sweet spot seems to be near 8 CPUs/cores.The calculation is the number of CPU cores minus 4. An additional thread will be added for each CPU/core thereafter.

For example, if you add 48 cores to your Retain server there will be 4 Server threads and 44 Indexer threads.

This can significantly increase the performance of indexing if RAM or disk I/O are not the bottlenecks. If they are, you probably won't see any change; however, it is still recommended in order to avoid CPU spikes and improve overall performance.

Changing Threads Manually

CPU time is being sliced between the threads, so the limitation becomes the overhead of context switching between threads.

The number of indexing threads can be set manually in solrcloud.indexing.properties found in ~\Beginfinite\Retain\RetainServer\WEB-INF\classes\config

Remove the # comment symbol at the beginning of the line with indexingThreads=0

indexingThreads has a valid range of 1-24

This overrides the number of indexing threads. You can set indexingThreads=1 and only 1 thread will run no matter how many CPU/core are available. The recommendation is to go no higher than 10-20 times the number of CPU cores. There is a hardcoded maximum of 24 threads using this method.

See also: Configuring more indexing threads

This article was originally published in the GWAVA knowledgebase as article ID 2834.