Sources and Engine

Copy Markdown View Source

LLMDB.Engine is the build-time ETL pipeline. It loads only the sources passed to it (or returned by LLMDB.Config.sources!/0), normalizes and validates their data, merges the source layers, enriches the result, and emits a canonical snapshot. Consumer runtime loading is separate: LLMDB.load/1 reads a packaged or explicitly configured snapshot, then applies consumer filters and indexes.

Source Behaviour

Sources implement LLMDB.Source:

@callback load(opts :: map()) :: {:ok, data :: map()} | {:error, term()}
@callback pull(opts :: map()) ::
            :noop | {:ok, cache_path :: String.t()} | {:error, term()}

pull/1 is optional. Remote sources use it to refresh a local cache; load/1 reads that cache and performs no network request.

Canonical Format

Each outer key is a provider ID string. Its value is a provider map with atom keys and a :models list. Model maps also use atom keys:

%{
  "openai" => %{
    id: :openai,
    name: "OpenAI",
    base_url: "https://api.openai.com/v1",
    models: [
      %{
        id: "gpt-4o",
        provider: :openai,
        name: "GPT-4o",
        capabilities: %{chat: true}
      }
    ]
  }
}

Use LLMDB.Source.assert_canonical!/1 for the fast shape assertion. Full Zoi validation happens in the Engine.

Built-in Sources

Remote cached sources

Models.dev, OpenRouter, OpenAI, Anthropic, Google, Meta, xAI, ZenMux, LLM API, and Llmfit have source adapters. Run mix llm_db.pull to refresh their repository-local cache; the Engine subsequently reads that cache through each source's load/1.

Meta Model API discovery

Set META_API_KEY in your environment, then run:

mix llm_db.pull --source meta
mix llm_db.build --install

The source requests GET https://api.meta.ai/v1/models with bearer authentication. It stores a cache and provenance manifest under priv/llm_db/remote. Configure :meta_cache_dir to change this directory. Meta.load/1 reads the cache without a network request or credentials. Tests can pass :api_key, :cache_dir, :url, and :req_opts; no live key is required for tests.

Add the repository Actions secret META_API_KEY to enable Meta discovery in the scheduled build-metadata.yml workflow. The workflow already passes this secret to mix llm_db.pull. An absent or empty key skips the source. HTTP and transport failures preserve the last good cache and make the pull task report a failure. Returned errors do not contain credentials or response bodies.

The registry list is specific to a team. A new response does not remove models from the previous cache or the reviewed catalog. Discovery records preserve id, created, and owned_by. created stays registry metadata; it is not a public release date. The source does not require optional client metadata and does not infer prices, limits, or capabilities from a sparse response.

The local Meta overlay has higher precedence. It supplies the reviewed Muse Wonder prices and capabilities even without a cache. Unreviewed registry entries stay catalog-only. Muse Code and subscription-only Muse Wonder Full are excluded from direct API discovery. The reviewed sources do not publish a maximum output limit for Muse Wonder, so that field remains unset.

The overlay sets extra.unpublished_limits = ["output"]. This build-time directive removes an unverified output limit inherited from another source. It can name context, input, or output. Other limits and reviewed prices remain available.

References: Meta model registry, registry schema, and Meta prices.

The LLM API source reads the public https://api.llmapi.ai/v1/models catalog. Run mix llm_db.pull --source llmapi to refresh it without credentials, then mix llm_db.build --install to rebuild the packaged snapshot. Runtime loading does not make network requests.

LLM API model contracts use the shared OpenAI Chat Completions adapter only when every advertised fallback route supports Chat. Media models and models with Responses-only routes remain catalog-only. Capabilities are intersected across fallback routes, and unpublished output limits remain absent. Object contracts require tools, tool choice, and the max_tokens parameter used by the shared adapter's default object request.

Published one-hour cache-write prices have separate duration-scoped components. When supported input modalities have different token rates, the source retains the base cost summary and raw fees but excludes the flat token.input component. Complete input pricing then requires a token split by modality; extra.pricing_input_split_required identifies these models.

Local TOML

{LLMDB.Sources.Local, %{dir: "priv/llm_db/local"}}

The directory contains one subdirectory per provider. Each provider directory contains provider.toml and model TOML files. The adapter injects the provider ID from the directory name.

Configuring Build-time Sources

config :llm_db,
  sources: [
    {LLMDB.Sources.ModelsDev, %{}},
    {LLMDB.Sources.Local, %{dir: "priv/llm_db/local"}}
  ]

Sources are processed in order and later sources have higher precedence. The Engine does not implicitly place the packaged runtime snapshot beneath these sources. Passing no sources produces an empty Engine catalog with a warning; this does not change the independent runtime default of loading the packaged snapshot.

ETL Pipeline

LLMDB.Engine.run/1 performs these build-time stages:

  1. Ingest — load configured sources and assert canonical shape.
  2. Normalize — normalize provider IDs, dates, modalities, and source fields.
  3. Validate — parse provider/model data with the Zoi schemas.
  4. Merge — apply last-source-wins precedence and field merge rules.
  5. Finalize — enrich records and nest models beneath providers.
  6. Ensure viable — warn when the resulting catalog is empty.

Allow/deny filters, provider preferences, custom runtime overlays, and runtime indexes are LLMDB.load/1 concerns, not Engine stages.

Supported Maintainer Tasks

These tasks are maintainer tooling. Downstream applications consume the snapshot shipped with the package.

Custom Source Example

defmodule MyApp.InternalModels do
  @behaviour LLMDB.Source

  @impl true
  def load(_opts) do
    {:ok,
     %{
       "internal" => %{
         id: :internal,
         name: "Internal",
         models: [
           %{
             id: "custom-gpt",
             provider: :internal,
             capabilities: %{chat: true}
           }
         ]
       }
     }}
  end
end

config :llm_db, sources: [{MyApp.InternalModels, %{}}]

Custom sources are supported build-time extensions. Their output contract is stable, while Engine implementation modules are internal and may change behind that contract.