What is Thread Pool?

What is a Thread Pool?

As the name suggests, a Thread Pool is a pool or a collection of threads.

Let us take an example of a book library to understand thread pools better. A user comes to a library and requests for a book, if the book is not present in the library, the book has to be purchased first before allocating it to the user. However if the library anticipates in advance that there would be a demand for the book and stores the book before any user requests for it, they can save on the time delay of purchasing the book. Once the user visits the library, they can be immediately allocated the book. After consumption, the book will be returned to the library, ready for allocation to other users

The same concept can be applied to threads. Let us assume we have two tasks T1 and T2 which need to be executed one after the other. T1 requests the compiler/ application to spawn a thread where the task can execute, the compiler creates one, T1 does its processing, and then the compiler destroys one. Now T2 has to process, the same processs happens again. The total time of computation is as follows:

If the threads were already created in advance, and we do not destroy them after their use, instead return them to the pool, we can save up on a lot of compute time.

Thread Pool Architecture

The following diagram illustrates how a thread pool works

There are 3 threads in the pool: T1, T2 and T3.

Service1 and Service2 request for a thread from the thread pool, and are allocated thread T1 and T2 respectively

Service3 has finished its request and is ready to return thread T3 to the pool

Service4 has a request but since there are no threads available in the pool, its request cannot be served. The request will be parked to a waiting queue

Thread Pool in Java

In Java, a thread pool is created using ExecutorService

This will create a pool of 3 threads

Using the execute() function, a thread can be requested from the executorService

The Ideal Thread Pool

How many threads should be present in the pool? Let us understand the extreme cases first

• If we create a lot of threads in the pool, we will face high memory utilization and degraded performance of the application
• If we create too few threads, we would not get the benefits of using a thread pool

Java has an interesting way of creating a Thread pool using Executors.newChachedThreadPool(). This implementation creates an initial pool of zero threads, whenever there is a demand, more threads are created and added to the pool. Threads which remain idle for more than 60 seconds are removed from the pool

The parameters like number of threads in the pool, TTL of idle threads can be controlled. We can instantiate the pool with a starting number of threads, and more threads will be created dynamically if needed

How to know the starting thread count?
There is a formula to compute the ideal number of threads your pool should have:

Wait Time: The total time spent when no processing is done, for example I/O operations, network calls Service Time: Actual processing time

For example, in a quad-core CPU, if a microservice spends 500ms on spent on I/O operations, and 50 ms on actual compute,