Setting up Worker Pools with Poolboy in Elixir

One of the great benefits of the way processes work in Elixir is being able to easily opt in to synchronous or asynchronous behavior. Consider the following 2 examples, which takes a list of numbers and applies the :math.sqrt/1 function.

Single process:

  def run_sync do
    1..10
    |> Enum.map(fn(x) ->
      :math.sqrt(x)
    end)
  end

Using Task to run each computation in a separate process:

  def run_async do
    1..10
    |> Enum.map(fn(x) -> 
      Task.async(fn -> :math.sqrt(x) end)
    end)
    |> Enum.each(fn(task) -> 
      task
      |> Task.await(30_000)
    end)
  end

In the first example, each computation will take place once the previous has completed. In the second, all the computations will take place simultaneously.

There are cases where we will want to limit the number of processes that are created. Examples include making a worker pool to perform background jobs, making calls to external APIs, and connection pools for connecting to databases.

In the following series of videos we’ll cover how to set up a worker pool using poolboy, and how to create a connection pool for Redis using the Redix library.