⛵ crew#

Assemble a crew of worker processes to tackle your computational tasks.

A lightweight, generic worker/task pool for Gleam that distributes work across multiple managed worker processes. The pool manages a queue of work items and distributes them to idle workers; when no workers are available, work is queued until a worker becomes free.

Worker pools are named and can be used from multiple sources concurrently, making them useful for limiting concurrency globally in a system. crew offers blocking and non-blocking APIs for a variety of use-cases.

Installation#

gleam add crew@2

Example#

import crew
import gleam/erlang/process
import gleam/int
import gleam/list

pub fn main() {
  // Start a pool with 4 workers
  let pool_name = process.new_name("image_pool")
  let assert Ok(_) =
    crew.new(pool_name, process_image)
    |> crew.fixed_size(4)
    |> crew.start

  // Process images concurrently
  let images =
    list.range(1, 20)
    |> list.map(fn(i) { "photo" <> int.to_string(i) <> ".jpg" })
    |> crew.call_parallel(pool_name, 6000, _)

  echo images
}

fn process_image(filename: String) -> String {
  // Simulate image processing work
  process.sleep(1000)
  "Processed: " <> filename
}

See also#

This worker pool is inspired by poolboy and lifeguard, adapted for Gleam's type system and actor model.