llm-simple

An experimental Haskell LLM toolbox for trying out providers, tool loops, and workspace agents locally. It is meant to get you from zero to a working agent quickly — not to be a production agent platform.

The API stays small on purpose: JSON model and provider catalogs, bundled filesystem tools, and a straight line from loadModelOrThrow to generateText. Under that surface you still get multi-provider gateways, fallbacks, streaming, structured output, and sandboxed workspace tools — enough to prototype and compare ideas without assembling the pieces yourself.

Status: early 0.4.x release. APIs may change.

Features

• Providers: OpenAI, Claude, Gemini, Ollama, DeepSeek
• Generate: single-request text, streaming, and structured object generation with model fallbacks, timeouts, retries, and throttling
• Agent: multi-round tool loops (generateText, streamText), plus generateObject / generateObjectUntyped for a single structured-output call with the agent's system prompt and context window (no tools are sent or run)
• Tools: filesystem tool suite with workspace path sandboxing
• Load: JSON model and provider catalogs, API-key gateway initialization
• Observability: hooks, logging, and generation lifecycle events

Install

cabal build

Requires GHC 9.6–9.12 with GHC2021 (see tested-with in llm-simple.cabal).

Tested on Linux and macOS. Windows is not supported and not tested.

Quick start

1. Create a .env file with API keys for the providers you use:

OPENAI_API_KEY=...
CLAUDE_API_KEY=...
GEMINI_API_KEY=...
DEEPSEEK_API_KEY=...

Ollama needs no API key. It is used through its OpenAI-compatible endpoint (/v1/chat/completions); tested with Ollama 0.32.4. Streamed responses request usage via stream_options.include_usage; an Ollama version that does not support it streams without usage.

The library reads keys from the process environment. Load .env yourself in main (the example executable does this), or call loadGatewaysWithDotenv when building gateways manually.

2. Configure models in model-catalog.json (an example is included in the repository and source distribution).

3. Create a workspace directory for filesystem tools (the example uses ./user-workspace/).

4. Run the example executable:

cabal run llm-simple

The example loads .env, loads one model from the catalog (deepseek4flash, so only DEEPSEEK_API_KEY is needed; change the name in app/Main.hs to use another, e.g. llama_3_2 for a local Ollama), wires up filesystem tools, and runs a small agent that asks about files in the workspace.

Configuration

The model and provider catalogs are trusted input, like code: they decide where requests and API keys are sent (baseUrl, baseUrlEnv) and which environment variables are read. Do not load catalogs from an untrusted source. The same holds for getSystemPrompt (LLM.Load.Utils), which reads any file named by a file: value, outside any sandbox.

Model catalog

Models are defined in a JSON array. Each entry maps a logical config name to a provider and model, plus runtime settings:

A provider is only available if its API key is set and not empty or only whitespace (except Ollama, which is always available).

Each modelConfigName must be unique, maxTokens must be positive, and requestTimeout, throttleDelay, retryCount, jitterBackoff and the prices must not be negative. Otherwise loading the catalog fails with a LoadModelCatalogError that names the entry, whichever model you ask for.

Capability flags are declarations used before each fallback candidate is called: image parts require vision; enabled thinking requires thinking. Unsupported candidates fail with UnsupportedCapability and the next fallback is tried. Cache hints may be ignored by providers that do not support prompt caching (they do not change prompt semantics).

Image input

User messages may mix text and images:

import LLM (imageUrlPart, imageBase64Part, textPart, UserMessage)

let Right png = imageBase64Part "image/png" "<base64>"
    msg = UserMessage [imageUrlPart "https://example.com/photo.jpg", png, textPart "describe"]

imageBase64Part validates MIME type (image/jpeg|png|gif|webp|heic|heif) and base64 payload locally. Use a model with "capabilities": {"vision": true}.

Prompt cache breakpoints

Mark stable content with cacheEphemeral (or withCacheHint CacheEphemeral). Claude serializes that as cache_control: {"type":"ephemeral"} on the content block (provider default TTL). Other providers omit the marker and send the same content. Hints persist in conversation JSON and through agent tool rounds.

import LLM (cacheEphemeral, textPart, UserMessage)

let msg =
      UserMessage
        [ cacheEphemeral (textPart "large stable context…"),
          textPart "question about the context"
        ]

UserTurn "text" still matches only a single unannotated text part; use UserMessage when parts carry cache hints.

Provider catalog

Providers are defined in providers.json (bundled with the package as a reference). Built-in providers are always available. When you load a model catalog, the loader looks for providers.json in the same directory; if present, its entries are merged on top of the built-ins — adding new providers or overriding built-ins with the same providerName. Within the file, each providerName must be unique (LoadProviderCatalogError otherwise).

To add an OpenAI-compatible provider (OpenRouter, Groq, Together, etc.), add an entry with "protocol": "openai", set baseUrl and apiKeyEnv, then reference providerName from your model catalog.

Optional base URL overrides for built-in providers: OPENAI_BASE_URL, CLAUDE_BASE_URL, GEMINI_BASE_URL, DEEPSEEK_BASE_URL, OLLAMA_BASE_URL. Do not include the protocol path (/v1/... or /v1beta/...) in baseUrl; the library appends it. A provider that sends an API key must use an https:// base URL, or http:// to a loopback host (localhost, 127.0.0.0/8, ::1); any other http:// base URL makes the provider unavailable, so the key is never sent unencrypted. Keyless providers (Ollama, or entries without apiKeyEnv) may use any http:// host. loadGateways and loadGatewaysWithDotenv always use built-in provider defaults. Custom providers from a co-located providers.json apply when loading models via loadModelOrThrow or loadModelsOrThrow.

Loading models

import Configuration.Dotenv (defaultConfig, loadFile)
import Control.Exception (IOException, catch)
import LLM.Load (loadModelsOrThrow)

main = do
  loadFile defaultConfig `catch` \(_ :: IOException) -> pure ()
  (gemini, deepseek) <-
    loadModelsOrThrow "./model-catalog.json" ("gemini_lite", "deepseek4flash")
  ...

loadModelsOrThrow takes a filepath and a tuple of config names, returning a tuple of ModelConfig values. Use loadModelOrThrow for a single model. On failure it throws a catchable LoadConfigError.

Usage

Agent with tools

An agent runs a loop: the model may call tools, results are fed back, and the loop continues until the model finishes or agMaxToolRounds is reached. After the last allowed round, the agent makes one more model call that asks for a text answer; if the model still calls tools, they are not run and the result is GErrToolExceeded.

Configuration mistakes are logged as warnings (onLog) at the start of a run instead of failing it: a name in agTools that the tool map lacks, a repeated name, a tool named get_history while agContextWindow is set (the built-in one wins), or a non-positive agContextWindow (no windowing). validateAgent returns the same warnings.

import Configuration.Dotenv (defaultConfig, loadFile)
import Control.Exception (IOException, catch)
import Heptapod (generate)
import LLM.Agent (Agent (..), RuntimeArgs (..), generateText, noEventObserver)
import LLM.Core (pattern UserTurn)
import LLM.Generate (ModelWithFallbacks (..), Hooks (..), llmHooks, noHooks)
import LLM.Load (fsTools, loadModelsOrThrow)

main :: IO ()
main = do
  loadFile defaultConfig `catch` \(_ :: IOException) -> pure ()

  (gemini, deepseek) <-
    loadModelsOrThrow "./model-catalog.json" ("gemini_lite", "deepseek4flash")

  let models = ModelWithFallbacks { mwfModel = gemini, mwfFallbacks = [deepseek] }

  toolMap <- fsTools "./user-workspace/"
  genId <- generate

  let agent =
        Agent
          { agName = "friendly-assistant",
            agSystemPrompt = Nothing,
            agTools = ["directory_tree"],
            agMaxToolRounds = 10,
            agContextWindow = Nothing
          }
      rt =
        RuntimeArgs
          { rtGenerationId = genId,
            rtAbortSignal = Nothing,
            rtLLMHooks = llmHooks noHooks,
            rtHooks = noHooks,
            rtOnEvent = noEventObserver,
            rtReadonly = False
          }

  r <- generateText agent models toolMap rt [UserTurn "what files are here?"]
  print r

Set rtReadonly = True to omit mutating tools from the schema sent to the model and to block them at execution time.

Every request's messages are checked with validateTurn (for example, no tool-call parts in a user turn) and validateConversation (tool results follow the assistant turn whose calls they answer, each call answered exactly once); each problem is logged as a warning and the request is still sent. From 0.5 it will be an error.

Log messages of an agent run (onLog) start with [<agName>] ; leave agName empty for no prefix.

Hooks are for observation only. An exception a hook throws is caught and dropped (asynchronous exceptions still propagate), so a hook cannot stop a run.

For local debugging, defaultDebugHooks enables stderr logging and writes request/response JSON to ./dumps — avoid that in production or shared environments. Each dump is a new file readable only by its owner; dumps hold prompts, tool arguments and results, and model output, and are never deleted by the library.

Single request (no agent loop)

For one-shot generation without a tool loop, use the Generate layer directly:

import LLM.Generate (GenRequest (..), generateTextWithFallbacks)

result <- generateTextWithFallbacks genRequest models

Also available: streamTextWithFallbacks, genObject, and genObjectUntyped.

Custom tools

Define a Tool with a ToolDef (sent to the model) and an execution function, then add it to a ToolMap. Use toTool to convert typed tool definitions. See LLM.Agent.ToolUtils and the modules under LLM.Tools.

Built-in filesystem tools

fsTools registers these tools, all scoped to a workspace root:

Use fsTools' to map tool results through a custom embedding function.

Filesystem sandbox

Filesystem tools enforce workspace-relative path containment: .. escapes, absolute paths outside the workspace, and symlinks pointing outside are rejected. A symlink is judged by where it points, including when its target does not exist yet: writing through a dangling link that leads out of the workspace is refused and does not create the target. A link that resolves inside the workspace is followed, and the operation lands on its target. Recursive tools skip symlinks rather than following them. A .. that would step back out of a symlink (link/..) is refused, since it names a different file by name than the operating system would open. A path that cannot be checked for being a symlink (for example, permission denied) is treated as one.

Errors that reach the model name paths relative to the workspace, so the model does not learn where the workspace lives on the host. For any tool, the library drops an absolute file name from an IOException before it shows the error to the model.

Threat model. The sandbox keeps the model inside the workspace. Through the tools, a model can only reach paths inside it: .., absolute paths and symlinks are checked as described above. The tools cannot create symlinks, and an agent runs its tool calls one at a time, so a model cannot change the workspace while a check it relies on is in progress.

The sandbox does not protect against other processes that write to the workspace while the agent runs. A path is checked and then opened. A process that swaps a directory for a symlink between those two steps (a time-of-check/time-of-use race) can make a tool read or write outside the workspace. Hard links to files outside the workspace are not detected either. Such a process usually runs as the same user and can reach those files directly, so this only matters if it has less access than the agent. If a less-trusted process can write to the workspace, isolate the agent at the OS level (a container, or a separate user).

Limitations (pre-1.0):

• The sandbox is path-scoped, not kernel-level isolation. Use a dedicated, disposable workspace directory — not your home directory or a production tree.
• There is no denylist for sensitive in-workspace files (.env, .git/config, etc.). Anything inside the workspace is accessible to tools the model can call.
• Read tools enforce byte caps: whole-file and edit tools use a 1 MiB limit; read_file_paginated accepts source files up to 10 MiB and returns at most 100 KiB per page.
• Search and listing tools bound their work: find_files, grep and directory_tree look at no more than 20,000 entries per call, grep reads no more than 64 MiB of files per call, and each says so when it stops; tool output is capped at 100 KiB.
• rtReadonly omits mutating tools from the model schema and blocks them at execution time if they are invoked anyway.
• writefile and the replace tools replace the file as a whole (a temporary file renamed over it), so a failed write leaves the old content. The file keeps its mode bits but gets a new inode, so hard links to it keep the old content. Concurrent edits of one file from the same process are serialized; edits from other processes are not.

Module layout

Import LLM for the common surface, or import sub-modules directly for agent loops, providers, and advanced configuration. Sub-module APIs are exposed but considered experimental before 1.0.

Note: Workflow support was removed from this repo and moved elsewhere.

Testing

cabal test

Tests replay recorded provider responses from test/fixtures/ so they run without live API calls.

License

BSD-3-Clause. See LICENSE.