Sources and Engine

LLMDB.Engine is the build-time ETL pipeline. It loads only the sources passed to it (or returned by LLMDB.Config.sources!/0), normalizes and validates their data, merges the source layers, enriches the result, and emits a canonical snapshot. Consumer runtime loading is separate: LLMDB.load/1 reads a packaged or explicitly configured snapshot, then applies consumer filters and indexes.

Source Behaviour

Sources implement LLMDB.Source:

@callback load(opts :: map()) :: {:ok, data :: map()} | {:error, term()}
@callback pull(opts :: map()) ::
            :noop | {:ok, cache_path :: String.t()} | {:error, term()}

pull/1 is optional. Remote sources use it to refresh a local cache; load/1 reads that cache and performs no network request.

Canonical Format

Each outer key is a provider ID string. Its value is a provider map with atom keys and a :models list. Model maps also use atom keys:

%{
  "openai" => %{
    id: :openai,
    name: "OpenAI",
    base_url: "https://api.openai.com/v1",
    models: [
      %{
        id: "gpt-4o",
        provider: :openai,
        name: "GPT-4o",
        capabilities: %{chat: true}
      }
    ]
  }
}

Use LLMDB.Source.assert_canonical!/1 for the fast shape assertion. Full Zoi validation happens in the Engine.

Built-in Sources

Remote cached sources

Models.dev, OpenRouter, OpenAI, Anthropic, Google, xAI, ZenMux, and Llmfit have source adapters. Run mix llm_db.pull to refresh their repository-local cache; the Engine subsequently reads that cache through each source's load/1.

Local TOML

{LLMDB.Sources.Local, %{dir: "priv/llm_db/local"}}

The directory contains one subdirectory per provider. Each provider directory contains provider.toml and model TOML files. The adapter injects the provider ID from the directory name.

Configuring Build-time Sources

config :llm_db,
  sources: [
    {LLMDB.Sources.ModelsDev, %{}},
    {LLMDB.Sources.Local, %{dir: "priv/llm_db/local"}}
  ]

Sources are processed in order and later sources have higher precedence. The Engine does not implicitly place the packaged runtime snapshot beneath these sources. Passing no sources produces an empty Engine catalog with a warning; this does not change the independent runtime default of loading the packaged snapshot.

ETL Pipeline

LLMDB.Engine.run/1 performs these build-time stages:

1. Ingest — load configured sources and assert canonical shape.
2. Normalize — normalize provider IDs, dates, modalities, and source fields.
3. Validate — parse provider/model data with the Zoi schemas.
4. Merge — apply last-source-wins precedence and field merge rules.
5. Finalize — enrich records and nest models beneath providers.
6. Ensure viable — warn when the resulting catalog is empty.

Allow/deny filters, provider preferences, custom runtime overlays, and runtime indexes are LLMDB.load/1 concerns, not Engine stages.

Supported Maintainer Tasks

• mix llm_db.pull — fetch and cache upstream source data.
• mix llm_db.build — run ETL and build canonical snapshot artifacts.
• mix llm_db.snapshot.publish — publish an immutable snapshot release.

These tasks are maintainer tooling. Downstream applications consume the snapshot shipped with the package.

Custom Source Example

defmodule MyApp.InternalModels do
  @behaviour LLMDB.Source

  @impl true
  def load(_opts) do
    {:ok,
     %{
       "internal" => %{
         id: :internal,
         name: "Internal",
         models: [
           %{
             id: "custom-gpt",
             provider: :internal,
             capabilities: %{chat: true}
           }
         ]
       }
     }}
  end
end

config :llm_db, sources: [{MyApp.InternalModels, %{}}]

Custom sources are supported build-time extensions. Their output contract is stable, while Engine implementation modules are internal and may change behind that contract.