GitHub Daily Trending Repositories

JustVugg/colibri

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

https://justvugg.github.io/colibri

Website · Discord · English · 简体中文 · 繁體中文 · Italiano

Tiny engine, immense model. Run frontier MoE models — 744B to 2.8T parameters — on consumer and heterogeneous hardware, in pure C with zero engine dependencies, by treating storage, RAM, and VRAM as a single inference hierarchy (AI memory multitiering).

Nine families run today: GLM-5.2/5.3 (744B), GLM-5.3-Flash (321B, with vision), Inkling (975B), Kimi K3 (2.8T), DeepSeek V4 Flash (284B), DeepSeek V4.1 Flash (552B, with vision), Qwen3.8-Flash-Next (125B + 51B n-gram), Qwen3.6 (35B-A3B) and OLMoE (7B) — one C file each, the same coli chat / coli serve / coli web front end. Full roster ↓

Colibrì is an inference engine you can run today, and an open research platform. Its primary goal is to pursue inference-side performance across the entire software/hardware boundary — model formats, memory hierarchy, storage I/O, placement, scheduling, kernels, speculation, and CPU/GPU overlap — so large models depend less on scarce hardware and cost less to run.

Colibrì treats VRAM, RAM, and storage as a single multitier hierarchy, and it is deliberately a place to test aggressive systems ideas — so there is no SLA on speed, and a hard guarantee on semantics: experiments must earn their place through reproducible end-to-end measurements, and the default policy never silently changes model precision or router semantics. Insufficient fast memory may reduce speed; it must not quietly redefine the model.

$ ./coli chat
  🐦 colibri v1.11.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU
  ✓ ready in 32s · resident 9.9 GB
  › ciao!
  ◆ Ciao! 😊 Come posso aiutarti oggi?

See it running

The web dashboard (./coli web): a 744B model at 4 tok/s, TTFT 1.6 s, disk 0 — full expert residency on 6× RTX 5090, with live token metrics, the per-turn time breakdown, the VRAM/RAM/disk tier bar and the live mini-brain in the corner.

The Brain page: all 19,456 experts as a living cortex — colour is the storage tier, brightness is routing heat, and every expert routed in a turn flashes white. Hovering shows the expert's measured topic affinity.

The Atlas page: the measured expert atlas as a 3-D galaxy — 13,260 characterised experts, 1,041 replicated specialists clustering by topic (poetry, law, Chinese, SQL…). Position is measured routing affinity, not a learned embedding. Drag to spin.

The research mission

With Colibrì, private frontier model access is not limited by availability of hyperscaler-class hardware.

With its multitiering features Colibrì removes proprietary hardware dependencies aggressively optimizing functional inference engine pipelines.

Our operational mission includes changing how weights are represented and moved, deciding what lives in VRAM, RAM, or storage, overlapping heterogeneous compute, reducing launch and synchronization overhead, exploiting sparsity and reuse, and testing new decoding algorithms. Nothing is protected merely because it is conventional; nothing is adopted merely because a microbenchmark looks fast. The deciding result is end-to-end inference on real machines, with correctness and quality measured alongside throughput, latency, memory, and cost.

The practical consequence is accessibility: run a 744B-parameter model on hardware you already own, watch every expert fire in real time, and change the code that does it. Not renting intelligence behind an API — holding it: probing it, measuring it, improving it. The engine is deliberately small enough that the next useful optimization can come from anyone willing to measure it.

Core techniques and measured findings

One hierarchy, not limited by tier capacity. VRAM, RAM, and NVMe are placement tiers for the same weights; limited fast memory changes speed, not model semantics.
A JIT for weights. Measured routing heat drives a per-layer LRU, a learned pinned hot-store, and one-layer-ahead prefetch instead of loading every expert. It wins on repeatable workloads; history can overfit, and lookahead can lose on some hosts, so both remain measurable policies rather than promises.
I/O is part of the engine. Batched expert unions, overlapped reads and compute, O_DIRECT, and weighted dual-SSD striping attack the streaming path rather than pretending storage latency is free. O_DIRECT is drive-dependent, and dual-SSD still needs broader end-to-end community A/Bs.
Heterogeneous execution. CPU, CUDA, Metal, NUMA memory, and partial or full expert residency share one runtime and can be combined according to the machine; the profitable combination depends on compute, bandwidth, residency, and workload.
Compressed state without a different model. Token-exact forward validation, 57× smaller MLA KV state, persistent warm conversations, and faithful DSA keep optimization tied to correctness. These are memory, latency, and correctness properties — not a blanket throughput claim.
Speculation that must earn its keep. Native MTP and grammar-forced drafts are measured end to end and can be disabled when acceptance does not repay verification.

Open hypotheses, experiments, and how to help

Colibrì treats an optimization as a hypothesis until a controlled end-to-end A/B shows otherwise. These are the main questions now:

hypothesis	evidence so far	experiment still needed
Routing history can place experts better than plain LRU	learned pins improve repeated workloads, but can overfit a prompt	held-out, cross-session A/Bs across coding, chat, multilingual, and long-context workloads
Multiple SSDs can turn independent bandwidth into decode speed	weighted mirror/split routing is implemented and validated; the bandwidth model is sound	cold-cache one-drive vs two-drive GLM-5.2 runs on real, independent controllers
A hardware-aware planner can approach each machine's best configuration automatically	RAM/VRAM budgets and several backends are detected today	compare the generated plan with a controlled parameter sweep across laptops, workstations, NUMA hosts, and multi-GPU systems
Lossless or quality-bounded representations can reduce weight movement enough to matter	format and quantization ablations exist, with correctness/quality gates	reproduce quality, bytes moved, latency, and cost per useful token together — not compression ratio alone
Routing-aware speculation can pay before near-full residency	MTP and grammar drafts work, but MTP has also measured a 32% loss around 85% expert hit	map the break-even surface across acceptance, expert hit rate, batch union, and draft depth
CPU/GPU overlap can hide transfer and synchronization rather than merely move the bottleneck	CUDA and Metal wins exist, but fast CPUs and low residency can erase them	per-stage profiles and one-variable A/Bs across PCIe, unified-memory, and full-resident machines

Want to help? Pick one row and publish the negative results too. Record the hardware, commit, model/container, exact command, prompt, cache state, throughput, TTFT, expert hit rate, bytes read, and quality check; change one variable, repeat the run, and attach raw logs. Start with CONTRIBUTING.md, compare against the benchmark protocol, then open an experiment issue. A well-controlled failure is more valuable here than an unexplained fast number.

The idea

A 744B Mixture-of-Experts model activates only ~40B parameters per token — and only ~11 GB of those change from token to token (the routed experts):

So the model doesn't need to fit in fast memory — it needs to be placed:

the dense part (attention, shared experts, embeddings — ~17B params) stays resident in RAM at int4 (~9.9 GB);
the 19,456 routed experts (75 MoE layers × 256 + the MTP head, ~19 MB each at int4) live on disk (~370 GB) and are streamed on demand, with a per-layer LRU cache, a learned pinned hot-store, and an optional VRAM tier.

Think of the core algorithm as a JIT, but for weights. A compiler JIT never compiles the whole program — it watches what actually runs and compiles the hot paths, just in time. colibrì makes the same bet about a 744B parameter space: parameters are not resident state to be held, they are data to be staged across a heterogeneous storage hierarchy (VRAM / RAM / NVMe), exactly when the router proves they are needed. Measured routing heat decides which experts earn which tier, the router runs a layer ahead so prefetch hides the staging latency, and — like a JIT — the engine learns your workload: the more you run, the hotter the right experts get. It works because routing has measurable structure (see the expert atlas) — and structure is cacheable.

The engine is a single C file (c/colibri.c) plus small headers. No BLAS, no Python at runtime, no GPU required.

Local cluster mode

The coordinator keeps token generation, routing, and KV state local while disk-backed expert workers execute routed FFNs on other Macs. A layer's routed batch-union is sent as one persistent TCP request, so a token does not incur one round trip per expert.

Start the optional registration service:

./coli cluster coordinator --host 0.0.0.0 --port 8765

On each worker, with the same converted model available locally:

./coli cluster worker --model /nvme/glm52_i4 --port 9100 \
  --coordinator http://COORDINATOR:8765 --advertise-host WORKER_IP

Run the coordinator with discovery, or provide --cluster-workers HOST:PORT,... for a static setup:

./coli serve --model /nvme/glm52_i4 \
  --cluster-coordinator http://127.0.0.1:8765

The transport is disabled unless workers are configured, so the existing single-machine path remains unchanged. Dense-layer sharding and browser/WebGPU workers are separate follow-up seams.

How it works

The per-token path

Every layer of every token walks the same five steps. The design goal is that placement only ever decides speed — the router's decisions and the weights' precision are the same whether an expert answered from VRAM or from disk.

One memory hierarchy instead of one memory requirement

Dual-SSD: two copies of the model, twice the read bandwidth

Decode is disk-bound on most machines, and expert reads are read-only — so if you have a second SSD, put a full copy of the model on it and let the engine stream from both drives at once:

COLI_MODEL=/fast/glm52_i4 COLI_MODEL_MIRROR=/second/glm52_i4 ./coli chat
COLI_DISK_WEIGHTS=9,3 ...   # optional: primary,mirror bandwidth ratio (else measured at startup)

Each expert is routed to one drive by a deterministic hash, weighted by the two drives' measured (or declared) bandwidth, so readahead/PILOT prefetch and the demand read always hit the same drive and nothing is cached twice. The aggregate bandwidth is the sum of both drives — a 9 GB/s + 3 GB/s pair reads experts ~33% faster than the fast drive alone, and the OMP-parallel pin/warmup load streams from both. Details worth knowing:

the mirror is validated at startup (per-file size + safetensors header must be byte-identical to the primary); divergent or missing files silently stay on the primary, so a partial mirror is fine — a smaller second SSD holding only some shards still helps;
the mirror is never written: .coli_usage, .coli_kv and all sidecars stay on the primary;
a read error on the mirror falls back to the primary (one warning, no crash), so unplugging the second drive mid-run degrades instead of killing the server;
routing never changes tokens — both copies are byte-identical, and the per-run MIRROR: stats line shows GB served per drive.

The same engine spans the whole range: on a 25 GB laptop everything streams from disk (slow but correct); on a large host the entire expert set becomes resident (CUDA_EXPERT_GB=auto PIN_GB=all) and disk drops out of the decode path entirely. Between the tiers sits a learning cache: the engine records which experts your workload routes to (.coli_usage, updated every turn) and pins the hottest ones automatically — colibrì literally gets faster the more you use it. On multi-socket hosts, COLI_NUMA=1 interleaves the resident weights across memory controllers (#82).

For a second drive that cannot hold the whole model, Colibri can rank a partial mirror from the expert history it already learns. Run a few representative prompts first so .coli_usage reflects the workload, then plan, stage, and verify the mirror:

./c/coli mirror plan  --model /fast/glm52_i4 --mirror /second/glm52_i4 \
  --budget-gib 200 --reserve-gib 20
./c/coli mirror stage --model /fast/glm52_i4 --mirror /second/glm52_i4 \
  --budget-gib 200 --reserve-gib 20
./c/coli mirror verify --model /fast/glm52_i4 --mirror /second/glm52_i4

The planner reads safetensors headers directly, follows split-model directories from COLI_MODEL_DIRS, and prioritizes shards that can serve the hottest routed experts. Staging never changes the primary model: it copies through temporary files, preserves the requested free-space reserve, verifies every shard with SHA-256, never deletes an existing mirror shard, and atomically publishes a receipt only after the selected mirror is ready.

Never wait for the disk twice

Misses are expensive, so the engine spends most of its cleverness avoiding and overlapping them: each expert's three matrices are stored adjacent and read in one pread; a bounded async I/O pool (PIPE=1, default) loads missing experts while resident ones compute; batched positions read each unique expert once (batch-union); and a router-lookahead thread (PILOT=1) prefetches the next layer's experts — routing is measurably 71.6% predictable one layer ahead. On GPUs, the resident pipeline (COLI_CUDA_PIPE=2) keeps the residual stream on-device across layers so the CPU expert loop runs uninterrupted; on Apple Silicon an experimental Metal backend does the batched expert math on the unified-memory GPU; and a Vulkan backend brings the expert tier, dense projections, and the MLA attention core to any GPU with a Vulkan 1.2 driver — including AMD cards via Mesa/RADV (the only backend for cards the vendor stacks no longer support, like the RX 580, and competitive with ROCm on RDNA4 — see the benchmarking notes).

On real NVMe, measure DIRECT=1. O_DIRECT bypasses the page cache and is often a large win on drives with DRAM cache and bandwidth headroom (+34% decode measured with PIPE=1 on a Blackwell/Windows box; 4.25→9.69 GB/s in iobench on a GB10) — but it is drive-dependent: QLC/DRAM-less or virtualised disks can be neutral to negative. Try it first; keep what your hardware rewards.

Faithful model, compressed state

The forward pass is validated against a transformers oracle (teacher-forcing typically 30-32/32; two tiny-oracle positions are floating-point near-ties and toolchain-dependent). MLA attention stores a compressed KV state — 576 floats/token instead of 32,768 (57× smaller) — and persists it across restarts (.coli_kv): conversations reopen warm with zero re-prefill, byte-identical to an uninterrupted session. DSA sparse attention (GLM-5.2's lightning indexer) is implemented faithfully and validated by forcing full-key selection to reproduce dense attention exactly.

Speculative decoding, honestly

GLM-5.2's native MTP head drafts tokens that the main model verifies in one batched forward — 2.2–2.8 tokens/forward when it pays. Two hard-won rules ship as defaults: the MTP head must be int8 (int4 heads collapse to 0–4% acceptance, #8), and draft and verify must compute the same function — SPEC_PIN=1 pins both to one kernel family (#163 is the full forensic story). Grammar-forced drafts (GRAMMAR=file.gbnf) add ~free acceptance on constrained JSON output. Whether speculation is a net win depends on your cache temperature — measure, and use DRAFT=0 when it doesn't pay.

What it achieves

Same engine, same int4 container — the hardware only changes where the experts live. Highlights from the full benchmark tables:

6× RTX 5090, full residency: 5.8–6.8 tok/s decode, TTFT ~13 s (experiment log);
128 GB CPU-only desktop: ~1.8 tok/s warm (#200);
single RTX 5070 Ti laptop-class box: 1.07 tok/s via the GPU-resident pipeline (#273);
25 GB dev box: 0.05–0.1 tok/s cold — the proven floor where this project started, and still the honest baseline.

Quality is measured, not assumed: the int4 container's quantization cost and the scale-granularity/rotation ablations live in docs/benchmarks.md and #108/#81.

Get started

You need two things: the program (a few hundred KB) and the model (372 GB). Step-by-step for every platform in the Quick Start guide.

1. Get colibri

Download a prebuilt release — Linux, macOS and Windows, no compiler needed. Take the archive for your platform from Releases and unpack it:

mkdir colibri && tar xzf colibri-v1.8.0-linux-x86_64.tar.gz -C colibri && cd colibri
python3 coli info                         # engine ready ✓

Inside you get the engine (colibri, colibri.exe on Windows), the coli launcher and its Python helpers. Nothing to rename or configure — coli finds the engine next to itself. You only need Python 3 installed: the launcher and the API gateway are Python scripts, while the engine itself is pure C with zero dependencies.

Or build from source — needs gcc (or clang) with OpenMP:

git clone https://github.com/JustVugg/colibri && cd colibri/c
./setup.sh                                # checks gcc/OpenMP, builds, self-tests

Want coli on your PATH? From a checkout, pip install -e . registers it (the engine still lives in c/ — an editable install from the clone, not a wheel).

2. Get the model

A pre-converted GLM-5.2 int4 container is on Hugging Face — use the group-scaled (gs64) build with the int8 MTP head. It is about 372 GB, so put it on a disk with the room, ideally a fast one:

https://huggingface.co/mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp

⚠️ Use the gs64 container above, not the older per-row int4 mirrors (mateogrgic/…, jlnsrk/…): those measure ~9pp worse on quality and are the root cause of the original think-mode loops and never-terminating generations in #455. The gs64 container fixed those controlled per-row A/Bs, but it is not a general repetition or EOS-starvation guard. The MTP head must also be int8, not int4 (int4 → 0% draft acceptance, #8): ls -l <model>/out-mtp-* — int8 (correct) is 3527131672 / 5366238584 / 1065950496.

Or convert from the FP8 source yourself — one resumable command that never needs the full 756 GB on disk at once:

./coli convert --model /nvme/glm52_i4     # download+convert shard by shard (python, one-time)

Other supported models

GLM-5.2 is the reference model, but the same streaming approach runs six more families. Each is a sibling engine — one C file, its own architecture, the same coli chat / coli serve / coli web front end (the launcher picks the binary from the model's config.json):

What each one needs. These differ a lot, and reading two of them together has confused people into thinking the requirements contradict each other (#191). They do not — they are different models. None of them needs a GPU.

Model Disk for the weights RAM GPU

OLMoE ~7 GB (int8 container) 8 GB not needed

GLM-5.2/5.3 ~372 GB 16 GB min, 24 GB comfortable not needed

GLM-5.3-Flash ~195 GB converted 25 GB (12 GB weights at int4 + expert cache) not needed

Inkling ~469 GB 25 GB with the int4 dense container, ~120 GB without not needed

Kimi K3 ~1.6 TB 32 GB+ not needed

DeepSeek V4 Flash ~167 GB (REAP 150B: ~85 GB) 16 GB min, 32 GB comfortable optional; any NVIDIA card from the GTX 10 series up (Pascal/Turing via CUDA_ARCH=portable-pre-ampere NO_TC=1, best on RTX 50) makes prefill 5-10x and decode ~2.5x faster

Qwen3.8-Flash-Next ~185.5 GB (official FP8 checkpoint) 16 GB min, 24 GB comfortable at the default context not supported; CPU only

Qwen3.6-35B-A3B ~20 GB (int4-gs64 container) 24 GB (needs full RAM residency) optional; the CUDA VRAM expert tier measured 1.44 -> 10.05 tok/s (7.0x) on two 8 GB cards, output bit-identical to CPU

A GPU only ever makes it faster. Speed is set by your disk, because the experts are streamed from it — expect a fraction of a token per second on a slow drive and a few per second on a fast one with the cache warm.

Model	Disk for the weights	RAM	GPU
OLMoE	~7 GB (int8 container)	8 GB	not needed
GLM-5.2/5.3	~372 GB	16 GB min, 24 GB comfortable	not needed
GLM-5.3-Flash	~195 GB converted	25 GB (12 GB weights at int4 + expert cache)	not needed
Inkling	~469 GB	25 GB with the int4 dense container, ~120 GB without	not needed
Kimi K3	~1.6 TB	32 GB+	not needed
DeepSeek V4 Flash	~167 GB (REAP 150B: ~85 GB)	16 GB min, 32 GB comfortable	optional; any NVIDIA card from the GTX 10 series up (Pascal/Turing via `CUDA_ARCH=portable-pre-ampere NO_TC=1`, best on RTX 50) makes prefill 5-10x and decode ~2.5x faster
Qwen3.8-Flash-Next	~185.5 GB (official FP8 checkpoint)	16 GB min, 24 GB comfortable at the default context	not supported; CPU only
Qwen3.6-35B-A3B	~20 GB (int4-gs64 container)	24 GB (needs full RAM residency)	optional; the CUDA VRAM expert tier measured 1.44 -> 10.05 tok/s (7.0x) on two 8 GB cards, output bit-identical to CPU

Family	Total / active	Weights	Build	Docs
GLM-5.2/5.3	744B / 40B	`mastouri/…-int4-g64-with-int8-mtp` (372 GB)	`make -C c glm`	this page
Inkling (Thinking Machines)	975B / 41B	`nbeerbower/Inkling-colibri-int4` (469 GB)	`make -C c inkling`	inkling.md
GLM-5.3-Flash (Z.ai)	321B / 40B	`zai-org/GLM-5.3-Flash` — converted to int4-gs64 routed experts, dense stays BF16 and the precision is a load-time choice; vision included	`make -C c glm53`	glm53-flash.md
Kimi K3 (Moonshot)	2.8T / 104B	`moonshotai/Kimi-K3` — original checkpoint, routed experts stay native MXFP4	`make -C c kimi_k3`	kimi_k3.md
DeepSeek V4 Flash	284B / 13B	official sharded checkpoint — routed experts stay native fp4, dense stays fp8-e4m3; the REAP-pruned 150B (`puwaer/DeepSeek-V4-Flash-0731-reap-150b`, 85 GB, 132 of 256 experts) loads with the same engine and no conversion	`make -C c deepseek-v4`	deepseek-v4.md
DeepSeek V4.1 Flash	552B / 16B	official checkpoint, no conversion: experts are already fp4, dense is fp8-e4m3. 203 GB of it is an n-gram memory read from disk a few hundred bytes at a time, and the routed experts cost 4.5 GB per token against GLM-5.2's 12.7. Vision, tool calling and the DSpark draft head are all on	`make -C c deepseek_v41`	deepseek-v41.md
Qwen3.8-Flash-Next (Alibaba)	125B + 51B n-gram / 6B	`Qwen/Qwen3.8-Flash-Next-FP8` — original checkpoint; PLE stays pageable and experts stay native block-FP8	`make -C c qwen38` (CPU only)	qwen38.md
Qwen3.6 (Alibaba)	35B / 3B	`Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64` (~20 GB, recommended) — hybrid Gated Attention + Gated DeltaNet	`make -C c qwen36` (`CUDA=1` for the VRAM expert tier)	qwen36.md
OLMoE (AI2)	7B / 1B	converted with `c/tools/convert_olmoe_merged.py` — int8 container, ~7 GB	`make -C c olmoe`	—

Qwen3.6 ships three pre-converted containers: int4-gs64 (recommended — measured cosine to the int8 anchor 0.98777 → 0.99313 and KL 0.109 → 0.080 against per-row, i.e. ~44% less quantization error), int4 per-row as the A/B baseline, and KAT-Coder v2.5, which the same engine runs unchanged — any architecture-identical checkpoint works without a code path of its own. With CUDA=1 the VRAM expert tier measured 1.44 → 10.05 tok/s (7.0×) on two 8 GB cards, output bit-identical to the CPU path.

Kimi K3 needs no conversion: its QAT-trained MXFP4 experts are streamed straight from the original Hugging Face shards, and the bf16 dense set is quantized at load time. Long agent sessions can opt into recurrent-state checkpoints (COLI_K3_CKPT=N slots in RAM, or parked on disk with COLI_K3_CKPT_DIR): an edited or follow-up prompt restores the deepest surviving checkpoint and re-prefills only the tail, instead of replaying the whole conversation through the SSM layers. On Vulkan hosts K3_VK_UP=auto sizes the expert tier upload from measured bandwidth. The engine's KDA and MLA paths are validated token-exact in CI against the vendor implementation.

Inkling ships int4 experts but bf16 dense weights (49.4 GB resident); on a host that cannot hold those, inkling.md has a one-pass tool that brings the dense set to 15.3 GB and lets the 975B run on a 25 GB box — with the honest trade-off written down.

3. Run it

COLI_MODEL=/nvme/glm52_i4 ./coli chat     # RAM budget, cache and MTP auto-detected
COLI_MODEL=/nvme/glm52_i4 ./coli plan     # inspect the planned VRAM/RAM/disk placement
COLI_MODEL=/nvme/glm52_i4 ./coli doctor   # read-only readiness check
COLI_MODEL=/nvme/glm52_i4 ./coli doctor --deep  # strict tensors/shards/index/mirror preflight
COLI_MODEL=/nvme/glm52_i4 ./coli tune     # measure and save this machine's fastest safe execution profile
./coli web  --model /nvme/glm52_i4        # API + dashboard, and opens a browser
./coli serve --model /nvme/glm52_i4       # API + dashboard, no browser (headless)

On Windows a release archive ships coli.cmd: double-click it for the quick start, or run coli.cmd chat --model D:\glm52_i4 from cmd or PowerShell. From a source checkout the same commands work with python coli chat --model D:\glm52_i4. The .exe files are the engines, not the launcher: started on their own they have no model to load and exit immediately. The engine at runtime is pure C — python is only used by the one-time converter and the optional API gateway.

The same commands run any of the models

coli reads the model's config.json, picks the matching engine binary, and renders that family's chat template — so nothing about the command line changes between models. Build the engine you want once, then just point COLI_MODEL at the right directory:

make -C c glm                                     # GLM-5.2
make -C c inkling                                 # Inkling
make -C c kimi_k3                                 # Kimi K3

COLI_MODEL=/nvme/glm52_i4      ./coli chat        # TUI
COLI_MODEL=/nvme/inkling_i4    ./coli chat
COLI_MODEL=/nvme/kimi_k3       ./coli chat

./coli web --model /nvme/inkling_i4               # API + dashboard, opens a browser
./coli web --model /nvme/kimi_k3
./coli serve --model /nvme/inkling_i4             # API + dashboard, no browser

For the non-GLM engines coli chat starts the gateway locally and attaches the TUI to it, so the TUI, the API and the dashboard all go through the same arch-aware chat template — you never have to pass the template yourself.

Two things that differ per model, both documented in the per-model page:

Inkling on a RAM-tight host needs the int4 dense container and a small expert cache: ./coli chat --model /nvme/inkling_i4 --cap 2 (see inkling.md — the default --cap 8 wants ~14 GB of cache on top of the resident set).
Kimi K3 streams its MXFP4 experts from the original checkpoint, so there is nothing to convert — but the snapshot is ~1.6 TB (see kimi_k3.md).

4. Go deeper

topic	doc
Benchmarks, community datapoints, quality measurements	docs/benchmarks.md
Tuning knobs, policies, the learning cache, prefetch	docs/tuning.md
Windows 11 native build (+ CUDA DLL)	docs/windows.md
CUDA backend, VRAM expert tier, full residency	docs/cuda.md
Vulkan backend (any GPU: AMD via RADV, incl. cards ROCm dropped)	docs/vulkan.md
Apple Silicon Metal backend	docs/metal.md
OpenAI-compatible API, KV slots, web dashboard	docs/api.md
Experimental layer-segment embedding ABI	docs/segment-runtime.md
Experimental tokenizer/embedding/head Edge ABI	docs/edge-runtime.md
Grammar-forced drafts (structured output)	docs/grammar-draft.md
Environment variable inventory	docs/ENVIRONMENT.md

DeepSeek V4

DeepSeek V4 Flash streams the official checkpoint with no conversion: routed experts stay native fp4, the dense set stays fp8-e4m3 with UE8M0 block scales. MLA + DSA sparse attention, 43 layers, 256 routed experts plus one shared, top-6. Supported on x86-64/aarch64 Linux and Windows/MSYS2 (CPU), with an optional CUDA tier (Windows runtime DLL; Linux CUDA=1 direct link, verified under WSL2) that keeps every stage CPU-canonical and falls back per stage.

cd c
make deepseek-v4
python ./coli chat --model /path/to/DeepSeek-V4-Flash --ram 32
# also: coli run / coli serve / coli web
# Windows CUDA tier: make cuda-dsv4-dll CUDA_ARCH=portable  (+ make cuda-dsv4-dg-dll on RTX 50)

Two opt-in GPU levers are new and looking for community numbers, both default off and byte-identical when unset: DSV4_HYBRID=1 splits VRAM-tier misses between the GPU fill branch and the CPU branch using bandwidths measured at runtime, and COLI_CUDA_MOE_DOUBLE=1 (on top of COLI_CUDA_MOE_BATCH=1) prefetches the next layer's full expert set into a second VRAM bank while the current layer computes, falling back to the single bank when VRAM is short. The CUDA tier also runs on Pascal and Turing cards now (GTX 10 / RTX 20 series): build with CUDA_ARCH=portable-pre-ampere NO_TC=1.

Greedy decode and one KV slot. Tool calling is wired through the HTTP gateway with V4's native prompt and DSML call blocks; grammar is not supported. See the per-engine API matrix. Prefix checkpoints (in memory + on disk) make agent sessions and follow-up turns start in seconds after the first prefill of a system prompt. Measured on an RTX 5080 + 2 NVMe: 3324-token prefill 90 s, 8.3k-token first turn ~4 min once, later sessions/turns 6-9 s, decode ~1.6 tok/s at 3k context — see docs/deepseek-v4.md.

Give it RAM. 43 × 256 routed experts are ~137 GiB on disk and a token touches 301 of them, so the expert cache hit rate is what sets tok/s — --ram is the single most valuable knob, and it changes speed only, never output.

Speculative drafting exists and is off. DSpark's markov drafter and full MTP are both implemented and verified: a draft can save forward passes but never change a token, because every accepted token is still the target's own argmax. Measured on real multi-turn chat, they accepted 1 in 15 and 10 in 24, and the rejected-suffix replay of this engine's recurrent attention state cost more than the drafts saved — one 14-token answer took 495 seconds. So V4_DRAFT and V4_MTP default to 0 and the code stays, with the numbers beside it, for whoever retries this on faster storage.

See docs/deepseek-v4.md for the CUDA tier (build, DLL selection, GPU coverage), the environment reference, performance numbers, checkpoint validation, and the generated tiny independent oracle.

What's next

Inference-systems research is the product. The current hierarchy is LRU + a learned pin set; active work spans model formats, compression, placement, scheduling, I/O, CPU/GPU kernels, heterogeneous overlap, KV state, and routing-aware speculation. The objective is lower hardware requirements and lower cost per useful token. Everything lands the way this project works: measured end to end, reviewed, and developed in the open.
More open models. The tiering algorithm is model-agnostic: any MoE with routed experts can be staged the same way. Nine families run today (GLM-5.2, GLM-5.3-Flash, Inkling, Kimi K3, DeepSeek V4 Flash, DeepSeek V4.1 Flash, Qwen3.8-Flash-Next, Qwen3.6, OLMoE); further open-weight families — MiniMax among the candidates — earn an engine the way the first eight did: when someone measures one end to end.

Supporting the project

colibrì started as a one-person project on a 12-core laptop with 25 GB of RAM; today its numbers come from a community of real machines. If it's useful to you:

⭐ star the repo and share it;
🐛 open issues with benchmark numbers from your hardware — datapoints move this project more than anything else;
💬 join the Discord community to discuss experiments, hardware results, and research directions;
💬 reach out via GitHub issues to sponsor development or donate hardware.

Repo layout

Makefile                  root build/check entry point
c/
├── colibri.c             GLM-5.2 engine  (make glm)
├── inkling.c             Inkling engine  (make inkling)
├── kimi_k3.c             Kimi K3 engine  (make kimi_k3)
├── deepseek_v4.c         DeepSeek V4 Flash engine  (make deepseek-v4)
├── qwen38.c              Qwen3.8-Flash-Next text engine  (make qwen38)
├── qwen36.c              Qwen3.6 engine  (make qwen36)
├── olmoe.c               OLMoE engine  (make olmoe)
│
├── st.h                  safetensors index and range reads
├── quant.h               canonical container decoders
├── tok.h, json.h         tokenizer and JSON parser
├── compat.h              Windows/macOS shims (POSIX names, one place)
├── expert_store.h        streaming expert cache
├── route_trace.h         routing telemetry and .coli_usage, engine-agnostic
├── kv_prefix.h           KV prefix reuse across turns
│
├── backend_cuda.*        optional CUDA tier   (CUDA=1)
├── backend_metal.*       optional Metal tier  (METAL=1)
├── backend_vulkan.*      optional Vulkan tier (VULKAN=1)
│
├── Makefile              build and local checks
├── coli                  user-facing CLI
├── openai_server.py      OpenAI-compatible HTTP gateway
├── resource_plan.py      RAM/VRAM planner behind `coli plan` and `coli doctor`
├── tools/                offline conversion, fixtures and benchmarks
├── scripts/              long-running conversion helpers
└── tests/                dependency-free C and Python tests
web/                      browser UI (pure OpenAI-API client)
desktop/                  Tauri v2 desktop shell wrapping the web UI
docker/                   container images
docs/                     reference docs, experiments, media

One .c per model family, over shared single headers. An engine owns its architecture and nothing else; anything two engines both need — the safetensors reader, the container decoders, the tokenizer, the expert cache — lives in a header they both include, so a fix reaches all of them at once. That rule is not decorative: the defects that keep recurring here are the ones where a mechanism landed in one engine and never reached its siblings.

From the repository root, make, make check and make clean delegate to the engine Makefile.

Why "colibrì"

The hummingbird weighs a few grams, hovers in place, and visits a thousand flowers a day. This engine keeps a 744-billion-parameter giant alive on hummingbird rations: 25 GB of RAM, twelve CPU cores, and a lot of disk patience.

Acknowledgements

colibrì is an engine; the minds it runs are a gift. Thank you to the teams releasing frontier-class weights in the open — Z.ai (GLM), Moonshot AI (Kimi), Alibaba Qwen, MiniMax, and Allen AI (OLMoE) — and to every contributor who benchmarked, bisected, replicated an atlas run, or sent a patch. This project is proof of what open weights make possible.

The project's expert placement, compression, and routing experiments also build on ideas and evidence from the following open research and systems work:

REAP and EASY-EP for output-aware and domain-specific expert importance.
SERE for similarity-based expert re-routing, and ReMoE for cache-locality-aware router fine-tuning.
MC-SMoE for routing-guided expert merging and compression.
MoBE and D²-MoE for shared expert bases and low-rank expert deltas.
HybriMoE for hybrid CPU/GPU expert scheduling, ScMoE for overlapping expert communication with computation, and OD-MoE for distributed on-demand expert loading.
vLLM, llama.cpp, and kTransformers for the open inference systems and expert-offload work that make comparisons reproducible.

The engine also stands on concrete engineering work, not only ideas. Each of these is used or reimplemented in the tree today:

safetensors — the container every engine reads (c/st.h), including its fp8 and I64 dtypes.
tiktoken — c/tok.h reimplements its byte_pair_encode exactly, merging the adjacent pair whose concatenation has the lowest vocab id, so a tiktoken-derived vocabulary needs no merges list.
llama.cpp — the GBNF grammar subset in c/grammar.h follows its syntax and its set-of-stacks PDA, and the Metal path borrows its newBufferWithBytesNoCopy residency trick.
vLLM — the reference for output semantics the engine matches position by position (e.g. where the final norm lands relative to the LM head).
transformers — the oracle: CI reproduces a random-init model token for token against it.
DietGPU — the GPU ANS codec behind the experimental compressed expert tier (COLI_ANS).
rocWMMA — the HIP backend maps CUDA's nvcuda::wmma fragment/mma_sync API onto it (c/backend_gpu_compat.h), which is what lets one .cu source compile for both vendors.

License

Apache 2.0. GLM-5.2 weights are released by Z.ai under MIT.

ever-co/ever-gauzy

Ever® Gauzy™ - Open Business Management Platform (ERP/CRM/HRM/ATS/PM) - https://gauzy.co

https://gauzy.co

Ever Gauzy Platform

💡 What's New

🔥🔥🔥 we just released Ever Works - An open agentic runtime that autonomously researches, ships, and maintains entire businesses, 24/7. Please grab it now from https://github.com/ever-works/ever-works and make it ⭐⭐⭐ on GitHub!
if you like Gauzy, you should also like Ever Teams - Open Work & Productivity Platform. Please check https://github.com/ever-co/ever-teams and make it ⭐ on GitHub! It's built with a React (NextJs) / ReactNative (Expo) stack and connects to headless Ever Gauzy Platform APIs.

🌟 What is it

Ever® Gauzy™ - Open Business Management Platform for Collaborative, On-Demand and Sharing Economies.

Enterprise Resource Planning (ERP)
Customer Relationship Management (CRM)
Human Resource Management (HRM)
Applicant Tracking System (ATS)
Work and Project Management (PM)
Employee Time-Tracking, Activity & Productivity Tracking

Ever® Gauzy™ Platform is a part of our larger Open Platform for Collaborative, On-Demand and Sharing Economies - Ever® Platform™.

✨ Features

Main features:

Human Resources Management (HRM) with Time Management / Tracking and Employees Performance Monitoring
Customer Relationship Management (CRM)
Enterprise Resource Planning (ERP)
Projects / Tasks Management
Sales Management
Financial and Cost Management (including Accounting, Invoicing, etc)
Inventory, Supply Chain Management, and Production Management

A more detailed list of the features available in the platform:

Headless APIs
Dashboard (provides an overview of different metrics, such as company income/expenses, employee bonuses, etc.)
Time Management / Time Tracking / Activity Tracking / Timesheets
Employees Management (register of company employees/contractors, rates of employees, etc.)
Employee Onboarding
Applicant Tracking System (ATS) / Candidates Interviews
Contacts Management (Clients / Customers / Leads / etc.)
Schedules / Appointments / Events
Project Management / Tasks
Goals / KPI / Objectives / Key Results
Sales Pipelines
Proposals
Accounting / Invoicing / Estimates
Billing
Payments
Income / Expenses Management
Time Off Management / Holidays / Approvals
Inventory
Equipment / Sharing
Multiple Organizations Management
Organization Departments and Teams
Organization Clients and Vendors
Help Center / Knowledge Base
Tags / Labels
Reports / Insights / Analytics
Organization and Employee Public Pages
Integrations (Upwork, HubStaff, etc.)
Email History / Email Templates
Data Import / Export
Roles / Permissions
Multi-currency
Multi-lingual
Dark / Light / Corporate / Material and other Themes

Read more about Gauzy and how to use it at your company, on-demand business, freelance business, agency, studio or in-house teams.

🌼 Screenshots

Show / Hide Screenshots

Web UI

Desktop Timer UI (Standard)

Desktop Timer UI (Expanded)

🔗 Links

https://gauzy.co - check more information about the platform at the official website.
https://app.gauzy.co - SaaS (Important: it's currently in Alpha version/testing mode, please use it cautiously).
https://demo.gauzy.co - Online Demo (see more info below).
https://gauzy.co/downloads - Download Platform & Apps (see also more info below about available downloads).
https://docs.gauzy.co - Platform Documentation (WIP). See also our Wiki.
https://ever.co - get more information about our company products.

📊 Activity

💻 Demo, Downloads, Testing and Production

Demo

Ever Gauzy Platform Demo at https://demo.gauzy.co.

Notes:

Default super-admin user login is admin@ever.co and the password is admin
Content of demo DB resets on each deployment to the demo environment (usually daily)
Demo environment deployed using CI/CD from the develop branch

Downloads

You can download Gauzy Platform, Gauzy Server, or Desktop Apps (Windows/Mac/Linux) from the official Downloads page.

In addition, all releases and pre-releases downloads are also available from the following pages:

Production (SaaS)

Ever® Gauzy™ Platform SaaS is available at https://app.gauzy.co.

Note: it's currently in Alpha version/testing mode, please use it cautiously!

Staging

Gauzy Platform Staging builds (using CI/CD, from the stage branch) are available at https://stage.gauzy.co
We are using the Staging environment to test releases before they are deployed to the production environment
Our pre-releases of desktop/server apps are built from this environment and can be configured manually (in settings) to connect to Stage API: https://apistage.gauzy.co

Server & Desktop Apps

We have Gauzy Server and two Desktop Apps (for Windows/Mac/Linux):

Ever® Gauzy™ Server - includes Gauzy API, SQLite DB (or connects to external PostgreSQL) and serves Guazy frontend. It allows to quickly run Gauzy Server for multiple clients (browser-based or Desktop-based). It's a recommended option if you want to setup the Ever Gauzy Platform in small to medium organizations.
Ever® Gauzy™ Desktop App - includes Gauzy frontend (UI), Gauzy API, SQLite DB, etc., all-in-one! It allows to quickly run the whole Gauzy solution locally, both UI and Timer (for time tracking, optionally of course). In addition, it allows you to connect to the external database (e.g. PostgreSQL) or external API (if you have Gauzy Server with API / DB installed on a different computer or if you want to connect to our live API). It's a recommended option if you want to try Gauzy quickly / for personal use or if you want to connect to Gauzy Server in the "client-server" configuration (and use Desktop App instead of web browser).
Ever® Gauzy™ Desktop Timer App - allows running Time and Activity Tracking for employees/contractors with screenshots and activity monitoring. It is recommended to setup by organization employees as long as they are not interested in other Gauzy Platform features (e.g. accounting) and only need to track work time.

More information about our Server & Desktop Apps:

Download for your OS from the official Downloads page or see the section "Download" above for other links to our releases pages.
Setup Gauzy Server with default choices in Setup Wizard and run it.
You can also setup Gauzy Desktop App (can run independently or connect to Gauzy Server) or Gauzy Desktop Timer App (should be connected to Gauzy Server)
You can login with admin@ever.co and password admin to check Admin functionality if you installed Gauzy Server or Gauzy Desktop App. Note: such an Admin user is not an employee, so you will not be able to track time.
You can login with employee@ever.co and password 12345678 to check Employee-related functionality in Gauzy UI or to run Desktop Timer from an "Employee" perspective (such a user is an Employee and can track time).
If you install Gauzy Server, it is possible to connect to it using a browser (by default on http://localhost:4200) or using Gauzy Desktop Apps (make sure to configure Desktop apps to connect to Gauzy API on http://127.0.0.1:3000/api because it's where Gauzy Server API runs by default).
You can read more information about our Desktop Apps on the Desktop Apps Wiki Page and our Server at the Server Wiki Page.

🧱 Technology Stack and Requirements

For Production, we recommend:

PostgreSQL or MySQL
Kubernetes, Docker

Note: thanks to TypeORM / MikroORM, Gauzy will support lots of DBs: SQLite (default, for demos), PostgreSQL (development/production), MySql (development/production), MariaDb, CockroachDb, MS SQL, Oracle, MongoDb, and others (with minimal changes).

📄 Documentation

Please refer to our official Platform Documentation and our Wiki (WIP).

🚀 Quick Start

Super Quick Start

https://app.gauzy.co.

Note: it's currently in Alpha version/testing mode, please use it cautiously!

Run with Docker Compose

Clone repo.
Make sure you have the latest Docker Compose installed locally. Important: you need a minimum v2.20.

Demo

Run docker-compose -f docker-compose.demo.yml up, if you want to run the platform in basic configuration (e.g. for Demo / explore functionality / quick run) using our prebuilt Docker images. Check .env.demo.compose file for different settings (optionally), e.g. DB type. (Note: Docker Compose will use latest images pre-build automatically from head of master branch using GitHub CI/CD.)
Open http://localhost:4200 in your browser.
Login with email admin@ever.co and password: admin for Super Admin user.
Login with email employee@ever.co and password: 12345678 for Employee user.
Enjoy!

Production

Edit .env.compose: you must set JWT_SECRET, JWT_REFRESH_TOKEN_SECRET, JWT_VERIFICATION_TOKEN_SECRET and EXPRESS_SESSION_SECRET to strong, unique values (e.g. openssl rand -hex 64). The API refuses to start on the shipped blank/default secrets — shared defaults let anyone forge authentication tokens and sessions. Adjust any other settings (e.g. DB type) there too.
Run docker-compose up -d, if you want to run the platform in minimal production configuration using our prebuilt Docker images. (Note: Docker Compose will use latest images pre-build automatically from head of master branch using GitHub CI/CD.)

Note: we recommend using Kubernetes for production workloads instead of Docker Compose!

Build

Edit .env.compose (if needed) to use your custom settings, e.g. DB type.
Run docker-compose -f docker-compose.build.yml up -d, if you want to build everything (code and Docker images) locally. (Note: this is extremely long process because it builds whole platform locally. Other options above are much faster!)
☕ time... It might take some time for our API to seed fake data in the DB during the first Docker Compose run, even if you used prebuilt Docker images.

Notes:

while demo docker-compose.demo.yml runs a minimum amount of containers (API, Web UI, and DB), other Docker Compose files run multiple infrastructure dependencies (see full list below).
you can also run ONLY infra dependencies (without our API / Web containers) with docker-compose -f docker-compose.infra.yml up -d command. We already doing it using include in our main docker compose files.
you can add something like --env-file .env.something to the docker-compose up command to instruct Docker Compose to use a specific .env.something file with your custom settings

Together with Gauzy, the Docker Compose commands described above for Production (docker-compose.yml) and Build (docker-compose.build.yml) will run the following infrastructure components:

PostgreSQL - Primary Database.
Pgweb - Cross-platform client for PostgreSQL DBs, available on http://localhost:8081.
OpenSearch - Search Engine.
OpenSearch Dashboards - Search Engine Dashboards, available on http://localhost:5601. Default username: admin and password: Gauzy_password_123
Dejavu - Web UI for OpenSearch, available on http://localhost:1358.
MinIO - Multi-Cloud ☁️ Object Storage (AWS S3 compatible).
Jitsu - Jitsu is an open-source Segment alternative (data ingestion engine).
Redis - In-memory data store/caching (also used by Jitsu)
Cube - "Semantic Layer" used for Reports, Dashboards, Analytics, and other BI-related features, with UI available on http://localhost:4000.
Zipkin - distributed tracing system.

Quick Start to manually build & run

Required

Install NodeJs LTS version or later (e.g. version 22.x or 24.x).
Install Yarn v1.22.x (if you don't have it) with npm i -g yarn.
Install NPM packages and Bootstrap solution using the command yarn bootstrap.
If you will need to make code changes (and push to Git repo), please run yarn prepare:husky.
Adjust settings in the .env.local which is used in local runs.
Alternatively, you can copy .env.sample to .env and change default settings there, e.g. database type, name, user, password, etc.
Run both API and UI with a single command: yarn start.
Open Gauzy UI on http://localhost:4200 in your browser (API runs on http://localhost:3000/api).
Login with email admin@ever.co and password: admin for Super Admin user.
Login with email employee@ever.co and password: 12345678 for Employee user.
Enjoy!

Notes:

during the first API start, DB will be automatically seeded with a minimum set of initial data if no users are found.
you can run seed any moment manually (e.g. if you changed entities schemas) with the yarn seed command to re-initialize DB (warning: unsafe for production!).
it is possible to run generation of extremely large amounts of fake data for demo purposes/testing with yarn seed:all (warning: takes ~10 min to complete)

Optional / Recommended for Production

Optionally (recommended for production) install and run PostgreSQL version 14 or later (16.x recommended for production). Note: other DB can be configured manually in TypeORM / MikroORM / Knex. The default DB is set to SQLite (recommended for testing/demo purposes only).
Optionally (recommended for production) install and run Redis. Notes: the platform will work without Redis using an in-memory caching strategy instead of a distributed one (recommended for testing/demo purposes only). Please note however that Redis is required for Jitsu.
Optionally (recommended for production) install and run OpenSearch. Note: the platform will work without OpenSearch using DB build-in search capabilities (recommended for testing/demo purposes only).
Optionally install and run MinIO or LocalStack. Note: the platform will work without MinIO / LocalStack or other S3-compatible storage using local filesystem-based storage (recommended for testing/demo purposes only). For production, we recommend using Wasabi or AWS S3 storage or another S3-compatible cloud storage.
Optionally (recommended for production) install and run Jitsu. Note: the platform will work without Jitsu, however, data ingestion will be disabled for additional analyses / real-time pipelines.
Optionally (recommended for production) install and run Cube. Note: the platform will work without Cube, however some advanced (dynamic) reporting and data processing capabilities will be disabled.

🏭 Production

General information

See Setup Gauzy for Client Server for more information about production setup on your servers.

Kubernetes

We recommend deploying to Kubernetes (k8s), either manually (see below) or with our Terraform Modules or Ever Helm Charts.
For more simple deployment scenarios with k8s, please see Kubernetes configurations, which we are using to deploy Gauzy into DigitalOcean k8s cluster.

DigitalOcean App Platform

For the most simple deployment scenarios (e.g. for yourself or your small organization), check our DigitalOcean App Platform configurations and corresponding GitHub Action.

Virtual Instances / Droplets (via SSH)

Another variant to deploy Gauzy is to use DigitalOcean Droplets or any other virtual instance (with Ubuntu OS) and deploy using SCP/SSH, for example, following GitHub Action

Pulumi

In addition, check Gauzy Pulumi project (WIP), it makes complex Clouds deployments possible with a single command (pulumi up). Note: it currently supports AWS EKS (Kubernetes) for development and production with Application Load Balancers and AWS RDS Serverless PostgreSQL DB deployments. We also implemented deployments to ECS EC2 and Fargate Clusters in the same Pulumi project.

🚗 Other self-hosting options

Deploy on RepoCloud

Deploy Ever Gauzy instantly with one click on RepoCloud.

💌 Contact Us

Ever.co Website Contact Us page
Slack Community
Discord Chat
For business inquiries: mailto:gauzy@ever.co
Please report security vulnerabilities to mailto:security@ever.co
Gauzy Platform @ Twitter
Gauzy Platform @ Facebook

🔐 Security

Ever® Gauzy™ follows good security practices, but 100% security cannot be guaranteed in any software! Ever® Gauzy™ is provided AS IS without any warranty. Use at your own risk! See more details in the LICENSE.

In a production setup, all client-side to server-side (backend, APIs) communications should be encrypted using HTTPS/WSS/SSL (REST APIs, GraphQL endpoint, Socket.io WebSockets, etc.).

If you discover any issue regarding security, please disclose the information responsibly by sending an email to mailto:security@ever.co or on and not by creating a GitHub issue.

🛡️ License

We support the open-source community. If you're building awesome non-profit/open-source projects, we're happy to help and will provide (subject to acceptance criteria) Ever Gauzy Enterprise edition license and free hosting option! Feel free to contact us at mailto:ever@ever.co to make a request. More details are explained in our Wiki.

This software is available under the following licenses:

The default Ever® Gauzy™ Platform license, without a valid Ever® Gauzy™ Platform Enterprise or Ever® Gauzy™ Platform Small Business License agreement, is the Ever® Gauzy™ Platform Community Edition License

Please see LICENSE for more information on licenses. You can also compare our offering

™️ Trademarks

Ever® is a registered trademark of Ever Co. LTD. Ever® Demand™, Ever® Gauzy™, Ever® Teams™, Ever® Rec™, Ever® Recu™, Ever® Cloc™, Ever® Works™ and Ever® OpenSaaS™ are all trademarks of Ever Co. LTD.

The trademarks may only be used with the written permission of Ever Co. LTD. and may not be used to promote or otherwise market competitive products or services.

All other brand and product names are trademarks, registered trademarks, or service marks of their respective holders.

🍺 Contribute

Please give us ⭐ on Github, it helps!
You are more than welcome to submit feature requests in the separate repo
Pull requests are always welcome! Please base pull requests against the develop branch and follow the contributing guide.

💪 Thanks to our Contributors

See our contributors list in CONTRIBUTORS.md. You can also view a full list of our contributors tracked by Github.

⭐ Star History

❤️ Powered By

©️ Copyright

Copyright © 2019-present, Ever Co. LTD. All rights reserved

🔥 P.S

If you are interested in running an on-demand (delivery) or digital marketplace business, check open-source Ever Demand Platform
We are Hiring: remote TypeScript / NestJS / Angular developers

bilawalsidhu/gods-eye-view

A spy satellite simulator in your browser, except the data is real. Live open source spatial intelligence on a photorealistic 3D globe.

https://maptheworld.ai/

🌐 God's Eye View

A spy-satellite simulator in your browser — then you realize the sources are public and the data is real.

Photorealistic 3D globe. Live aircraft, ships, satellites, earthquakes, traffic, and public cameras. Hands-free voice control powered by a realtime AI agent.

No place left behind.

▶️ From the project behind the viral God's Eye View series (formerly WorldView) — 5M+ on YouTube · 25M+ across socials

🏆 Reached #1 on GitHub Trending, daily and weekly · August 2026

#8 Product of the Day · Hunted by Chris Messina, creator of the hashtag

“pretty cool” — Brendan Eich, creator of JavaScript and co-founder of Mozilla and Brave · Featured on Pinokio

⚡ Start without API keys. Install with Pinokio or run locally from the terminal. Add optional keys inside the app. → Quick Start

Quick Start · First Five Minutes · Talk to It · What's Live · Under the Hood · Keys & Costs

🌍 Why This Exists

God's Eye View brings public signals into one explorable globe. Track the world live. Talk to it. Break it. Extend it.

Flight transponders, ship beacons, orbital elements, seismographs, and public cameras already tell us a lot about the world. God's Eye View puts them in the same place, so you can move between a global picture and an individual aircraft, ship, or street. It runs locally in your browser, with source code you can inspect and extend.

Half the magic is that it looks like a forbidden cockpit. The other half is that every line of code is inspectable.

Most feeds are live or regularly refreshed. Traffic is simulated along real roads using aggregate location data. CCTV camera poses and rocket launch trajectories are coarse estimates.

Start with the included data sources, then add your own. Each layer is a separate module.

🎛️ What This Thing Does

🛩️ Cockpit view: Ride inside a tracked flight — the camera holds the terrain under you all the way down.
📡 Contacts: A 250 km roster of everything near your target — step through live aircraft and drop into any cockpit.
🎯 Click-to-track anything: Camera locks on, draws a fading trail, surfaces full metadata — and a tracked fire or vessel hands you off to the nearest live camera in one click.
🖊️ Voice whiteboard: Speak annotations onto the world — real boundary polygons, marks, and routes.
🛫 3D hangar: Real per-class aircraft models — 787, ATR-72, Citation, Bell 206, MQ-9 — and a tracked contact swaps from glyph to 3D model as you close in.
🎨 Reskin reality: GLSL sensor looks over the normal globe — CRT, NVG, FLIR/thermal, Noir, Snow.
🟩 Detection overlay: Screen-space bounding boxes and IDs on everything in view.
🎖️ Military HUD: Tactical heads-up display with intelligence-style telemetry.
🌐 Global Context: Stage the full situational picture with one switch — and get your exact view back when you leave.
🎥 Scene director: Capture cinematic camera tours for clips and demos.
🔗 Share Links: Camera, style, layers, and even one tracked target serialize into a URL — a live target is a handoff, not a bookmark.
🏠 Reset Globe: One control — or one sentence — back to the full Earth.

▶️ The full walkthrough of everything below, on YouTube

⚡ Quick Start

Start without an account or API keys. Both paths open the same app with Esri satellite imagery and keyless terrain. OSM is the fallback if Esri is unreachable. Flights, military traffic, satellites, earthquakes, public cameras, radio, and launches are available without keys.

For photorealistic 3D, add a Cesium ion token for eligible personal, non-commercial use, or a Google Maps key for the direct, metered route and in-app place search. Provider terms and quotas apply. Add keys through the app's POWER UP panel; Keys & Costs explains the options.

Path 1 — One click, no terminal

Install or update Pinokio to 8.2 or later.
Open God's Eye View in Pinokio.
Click Install, then Start.

Available on Windows, macOS, and Linux. The Pinokio maintainer reports cross-platform testing of the fixed installer. The launcher installs the locked dependencies, finds a free local port, and opens the app.

Tried before and installation failed? Update Pinokio and try again. Version 8.2 fixes the launcher installation issue; details from the Pinokio maintainer.

Path 2 — Terminal / coding agent

Use Node.js 24.x (24.14.0 or later) or 26.x. The setup doctor warns about Node 25, which is end-of-life.

git clone https://github.com/bilawalsidhu/gods-eye-view.git
cd gods-eye-view
npm ci
npm run doctor
npm run dev

Open http://localhost:4173. Choose Live Contacts, Space Missions, Environmental, or Explore Manually from the first-run panel.

Startup performance

A point-in-time M5/Chrome capture measured a median 1.86-second cold start. This is a comparison baseline, not a guarantee for your machine or connection. See docs/PERFORMANCE.md.

macOS shortcut: ./scripts/dev-fresh.sh clears the Vite cache and pulls any configured keys straight from the Keychain. It starts keyless too.

Then power it up — in the app, not in a file

Keys are upgrades, not prerequisites. When you want one, click the POWER UP chip in the bottom-right corner: Provider Settings lists every supported key, what it switches on, and where to get it. Paste, hit SAVE KEYS, and the app restarts itself with the new capability on. Once everything is configured the chip reads POWERED UP — and if a compact layout hides it, ?setup=1 reopens the same panel.

Where keys land: Pinokio → the app's ignored pinokio/ENVIRONMENT; a terminal clone → the repo-root .env. Either file is made owner-only before a secret is written into it. These are local plaintext files, excluded from Git; the app uses your keys to contact the providers.
Keys you already have stay yours: values from your shell or the macOS Keychain show as configured externally and are read-only to the panel.
What to get first: the free Cesium ion token (eligible personal, non-commercial use; current terms and quotas apply) for photorealistic 3D and world terrain; a Google Maps key only for the billing-enabled, metered route + place search; OpenAI when you want to talk to the world. Full map, costs included, in Keys & Costs.

Older Pinokio versions and credential storage

Do not enter credentials in Pinokio 8.0.40's native Configure panel: that release does not save this nested app file correctly, and it logs submitted values. Use POWER UP → Provider Settings inside GEV instead. The Pinokio 8.2 announcement fixes installation; it does not establish that this separate Configure issue is resolved. On macOS, the Keychain via ./scripts/dev-fresh.sh remains the stronger storage option.

The server binds to localhost on both paths, and Provider Settings answers requests only from your machine. Browser-side keys (Google Maps, Cesium ion) must be restricted at their providers — SECURITY.md shows how, and it carries the LAN-sharing rules alongside Keys & Costs.

🕐 The First Five Minutes

Choose a first-run mission, or try these in order. The GIFs show Google Photorealistic 3D; your starting basemap depends on the keys you've added.

Light up the sky. Take the Live Contacts mission (or turn on Flights yourself) — thousands of live aircraft, gliding on real telemetry, detection mesh already reading the scene. Click one: the camera locks on, a trail draws behind it, and its live telemetry card comes up.
Take the controls. Hit COCKPIT on your tracked plane and ride it down, switching sensors mid-flight: NVG into Ironbow FLIR.

Drop into a busy airport. Search one and descend to the taxiways with 3D aircraft on — grounded contacts, taxi trails, the whole apron working in real time.

Look through a public camera. Turn on CCTV over Austin, London, California, or Finland. The feeds aren't webcam embeds — they project into the 3D city. Cycle coverage to VIEWSHED and every camera draws its estimated coverage volume — where it reaches, and where it goes blind.

Track something in orbit. Turn on Satellites and click the ISS — you ride along at orbital distance, orbit ring and all.

Switch the optics. Tap 1–7 — CRT, NVG, FLIR — and the whole live planet re-renders through a different sensor.

Talk to it (needs an OpenAI key): "Take me to LAX and select the nearest airborne aircraft."
Come home. Hit Reset Globe — or just say "zoom out to a globe view."

Keyboard: 1–7 visual styles · H HUD · D detection · C cockpit · Esc out.

🛩️ The Cockpit

Every plane should let you do this.

Real-time cockpit mode, built from live flight data: the camera rides your contact with real terrain holding underneath, all the way down — sensor styles come along for the ride, and Contacts keeps the 250 km roster one click away: jump plane to plane and fall straight into the next cockpit.

The cockpit even carries its own briefing strip: nearby live signals, regional headlines, and real local weather — with an opt-in WX mode that renders volumetric clouds from actual observations around your aircraft.

Why cockpit mode exists: you're riding a real aircraft over real terrain — and you get to pick which sensor you see the world through.

🎙️ Talk to It

Voice needs an OpenAI key. Without one the entire app still runs — the mic button just reports voice is unavailable. The same key drives the AI HUD summary: a terse, five-word intelligence-style readout of the current view that regenerates as you move.

Click GEV MIC, grant the microphone, and just talk. This is more than a voice-controlled remote:

🧠 It knows what it's looking at. The agent pulls live scene context before answering — including coordinates, street names, active layers, and view scale. Ask "what city is this?" mid-flight and it knows.
🎯 Entity Q&A. Click any plane, ship, or datacenter and ask "what's this?" It answers using the object's live telemetry.
👁️ Visual grounding. At street level, it reads a viewport screenshot to identify legible signage and building names, and is instructed never to hallucinate labels.
🎬 Cinematic framing. "Show me the planes overhead" pulls the camera back, angles it, and frames the live traffic like a director.
🔒 Honest and secure. The agent only confirms actions that succeeded. Your OPENAI_API_KEY never touches the browser; the client only gets a short-lived session token.

Twenty-eight tools, four jobs — the commands below come straight from the product's voice test suite and tool playbook:

🎥 Direct it — drone-operator camera verbs:

🗣️ "Take me to Tokyo." · "Orbit around this area slowly." · "Draw the walking route from the Capitol to Zilker Park." → "Fly the route we just drew." · "Zoom out to a globe view."

🖊️ Annotate it — a whiteboard over the real world:

🗣️ "Outline the state of Texas." · "Annotate the Texas State Capitol and its grounds" — it draws the actual enclosing boundary, not a circle. · "How far is the Eiffel Tower from the Louvre?" — a connector arrow appears and it speaks the distance. Everything persists until you say "clear the map."

🔎 Interrogate it — analyst queries against the live layers:

🗣️ "How many flights are over Texas right now?" · "Which ships are headed to Oakland?" · "What is the biggest fire near Los Angeles?" · "Is anything flying above forty thousand feet?" · "When does the ISS pass over next?"

🎛️ Operate it — the whole console, hands-free:

🗣️ "Switch to night vision and turn on the flights layer." · "Turn on the camera viewsheds." · "Play a news radio station near Austin." · "Track that plane." → "Enter Cockpit."

And the rapid-fire tier — one sentence each:

🗣️ "Show me global infrastructure." (stages the layers and pulls back to the globe) · "Play Orbital Watch." (a full cinematic scene) · "Set detection density to fifty percent." · "Next contact — helicopters only." (mid-cockpit) · "Show me space missions." · "Switch to OSM." · "Sharpen the image a touch." · "Switch to the tactical layout." · "What's turned on right now?"

Ask for radio near anywhere and the globe starts broadcasting — every station is a real place you can fly to.

🛰️ What's on the Globe

Thirteen layers and map sources. Eleven have a keyless path. Some offer additional capabilities with a provider key. (🟢 no key · 🟡 free key · 🔴 metered.)

Layer	What you get	Source	Auth
🗺️ Map Stack	Esri satellite imagery, Google Photorealistic 3D, OSM, plus additional ion-hosted stacks	Esri / Google / Ion / OSM	🟢 Esri satellite + OSM · 🟡 ion-hosted Google 3D + world terrain · 🔴 direct Google + place search
✈️ Live Flights	11,000+ live aircraft + route history	OpenSky + adsb.lol	🟢 (🟡 optional for more polling credits)
🎖️ Military Flights	ADS-B military traffic in amber	adsb.lol	🟢
🚢 Live Vessels	Thousands of ships worldwide	AISStream	🟡
🛰️ Satellites	838-object catalog, color-coded by class with a live legend — the DENSE chip drops in the whole Starlink shell	CelesTrak	🟢
🌍 Earthquakes	Global seismic activity, last 24h	USGS	🟢
🚗 Traffic	Simulated vehicles on OSM roads. With TomTom, live flow speeds drive the simulation and congestion colors below ~8 km; individual vehicle positions are not live observations	TomTom + OSM	🟢 simulation · 🟡 live flow speeds
📹 CCTV Mesh	~3,400 public cameras projected into the 3D space — Austin · Texas (TxDOT) · California (Caltrans) · London (TfL) · Ontario (511) · Finland (Fintraffic) · British Columbia (DriveBC) · Estonia (Tallinn, Tarktee) · New South Wales (Live Traffic NSW). Positions are published; poses are estimated priors you calibrate by dragging a gizmo on the camera itself	City APIs	🟢
📻 Radio	Geolocated world radio with an analog tuner — drag the needle across up to 750 stations and the globe flies to each broadcaster	Radio Browser / broadcasters	🟢
🚲 Bikeshare	Live station availability	GBFS	🟢
🔥 Active Fires	Live NASA FIRMS detections, trailing 24h	NASA FIRMS	🟡
🚀 Space Missions	Rolling 30-day launches with payload, stage, and recovery detail	Launch Library 2	🟢 (🟡 optional token raises the allowance)
🎖️ Mapped Installations	Viewport-bounded military-site context from community mapping — incomplete by nature, and labeled that way	OpenStreetMap	🟢

The basemap ladder — what each tier buys you:

You have	The globe you get
🟢 Nothing	Esri World Imagery satellite basemap + keyless terrain, in 2D. OSM takes over automatically if Esri is unreachable; if terrain is unavailable the globe continues without it
🟡 A free Cesium ion token	Google Photorealistic 3D cities and world terrain — eligible personal, non-commercial use; current ion terms and quotas apply
🔴 A Google Maps key	The same 3D direct from Google, plus in-app place search — the billing-enabled, metered route

The Space Missions layer replaying a Falcon 9 ascent — labeled RECONSTRUCTED ESTIMATE, scrubbable 0.25×–4×.

Also on the globe: neighborhood overlays · an optional cockpit WX cloud effect. Bundled static infrastructure: Datacenters (4,351), Dams (704), and Submarine Cables (712).

Missing a layer you want? Open an issue — or add it and send the PR.

🎖️ Field Missions

Once the basics click, run these:

Mission	How
🚁 Ask the planet	"Why are all these military helicopters flying in circles?" Select a military track — it silently backfills ~24 h of real trace history — and see what it's been doing, resolved as stacked 3D loops.
✈️ Final approach	Click-track an airliner lining up for a runway, hop into the cockpit, and ride it down.
🌃 Night watch	Fly to your own city, switch to NVG, and let the detection mesh and HUD read the scene.
🚢 Port call	Vessels on over the Port of Long Beach. Click a tanker for its tactical card and wake trail — then hit NEAREST in the CCTV panel and look at the same water through a public camera.
📻 Tokyo FM	Orbit Shibuya with the Radio layer on — then drag the analog tuner needle: every position snaps to a real station and the globe flies to whoever's broadcasting.
🔥 Fire line	FIRMS over California. Click a detection — the camera dives to it — read the intensity, then hit NEAREST in the CCTV panel for a ground view.
🚶 Ask for a walking route 🎙️	Tell the world where you want to go and watch a real street-following route trace itself through the 3D city — then "fly it": banked turns, eased ends, a camera that leads the path like a drone shot.
📏 Measure LAX to DFW 🎙️	"How far is LAX from DFW?" — an arrow spans the country, the distance lands in the caption, and the endpoints stay pinned to the real world as you orbit.
🚀 Launch replay	Open Space Missions, pick a launch from the last 30 days, and ride the T-minus countdown through ascent to orbit — scrub it at 0.25×–4×. Labeled `RECONSTRUCTED ESTIMATE`, because it is one.
🪦 Walk the boneyard	Fly from regional context down into dense, fully resolved rows of retired aircraft.
🏗️ Orbit Three Gorges	Sweep the dam and its terrain at a glance — then flip on the Dams layer and find 703 more.

🎙️ = voice missions — they need an OpenAI key.

Ask the planet: a military contact's last ~24 hours of real trace history, resolved as stacked 3D loops.

"Draw the walking route… now fly it" — banked turns, eased ends, the camera leading the path like a drone shot.

Walk the boneyard: rows of retired airframes, fully resolved in 3D.

🔧 Under the Hood

How the globe handles live data:

World-stable icons. Aircraft and ships point along their true real-world heading at every camera angle — tracked or not, looking straight down or across the horizon — via per-frame screen-space course projection. No spinning, no viewport-locking.
Smooth motion from choppy data. Live feeds arrive every 15–30s; the globe renders one interval behind real time and interpolates between known fixes. Dead reckoning fills the gaps.
Honest satellites. SGP4 propagation with orbit rings that stay locked to their satellites via GMST realignment — no drift, no per-second flicker.
Sits on the real ground. Entity heights run through a real vertical datum — geoid-aware, sampled against the rendered terrain mesh — so aircraft park on aprons and cameras stand on street corners instead of floating.
Caching and request budgets. An OpenSky credit governor, a TomTom daily tile budget, and disk-cached TLEs reduce repeated requests. These controls do not replace provider quotas or billing controls.
Server-side credentials. Every API that touches a private key (OpenAI, AISStream, OpenSky OAuth, camera frames) is brokered through a hardened server-side proxy with SSRF protection, response caps, and sanitized errors. The only keys the browser sees are Google Maps and Cesium ion (restrict both at the provider).
No framework. Vanilla JavaScript, CesiumJS, and Vite — plus Google Photorealistic 3D Tiles for the planet and the OpenAI Realtime API for voice. Fast to read, fast to hack on.

src/
├── main.js                 # Bootstrap: Google 3D tiles, layer registration
├── ui.js                   # Runtime UI — panels, HUD, styles, control facade
├── hud.js                  # Intelligence HUD + AI scene summary
├── keySetup.js             # POWER UP panel — in-app provider keys (dev server only)
├── mapStackController.js   # Basemap switching — Google 3D / Esri / OSM / ion stacks
├── voice/                  # OpenAI Realtime session + 28 voice tools
├── data/                   # One module per layer + orchestration + context store
│   ├── iconOrientation.js  # Screen-projected headings + horizon cull
│   └── local_data/         # Bundled datasets (per-folder provenance)
└── scenes/                 # Cinematic scene director

See docs/CURRENT-STATE.md for the authoritative runtime reference.

🔑 API Keys

🟢 No key · 🟡 Free key · 🔴 Metered

Use POWER UP → Provider Settings to add keys. The tables below explain what each provider enables; none is required to start. See the setup instructions for storage and configuration details.

Choose the capabilities you want

Six keys. Four have a free tier, and the two 🔴 ones are metered:

	Key	Why	Get it
🟡	Cesium ion	🗺️ Google Photorealistic 3D, world terrain, and additional ion-hosted imagery stacks. The free Community plan is for eligible individual, personal/non-commercial use and has quotas	cesium.com/ion — use a public `assets:read` token and check current pricing/eligibility
🔴	Google Maps	Direct Google Photorealistic 3D + Google place search (Map Tiles API)	Google Cloud Console — URL-restrict it
🔴	OpenAI	🎙️ The voice experience + AI HUD summary. The mini model works; the standard model is noticeably smarter. Want Gemini or another provider behind the mic? PRs welcome	platform.openai.com — metered, see costs below
🟡	AISStream	🚢 Live global ships	aisstream.io — free signup
🟡	NASA FIRMS	🔥 Live active fires	firms.modaps.eosdis.nasa.gov — free
🟡	TomTom	🚦 Live flow speeds and congestion colors for the simulated traffic layer	developer.tomtom.com — free tier available

What the TomTom key buys you: rush-hour density painted on the city — then dive from the jam straight into the camera watching it.

Cherry on top

	Key	Why	Get it
🟡	OpenSky	✈️ More flight-polling credits (🟢 anonymous works without)	opensky-network.org
🟡	Launch Library 2	🚀 Higher space-missions request allowance (🟢 works without)	thespacedevs.com

Add these if you need higher polling allowances.

npm run doctor reports Node/npm readiness, the primary provider routes, and where each configured provider was found without printing credential values. On macOS its Keychain-aware result previews ./scripts/dev-fresh.sh; plain npm run dev reads only explicit environment and Vite dotenv values. The OpenSky summary reports only OAuth client-pair presence, not the resolved runtime mode or credential validity; Basic and credentials-file modes remain advanced dev-fresh.sh configuration.

Advanced setup: environment variables and macOS Keychain

For headless machines, coding agents, or scripted setups:

# Put keys in .env (see .env.example), or pass them as env vars:
OPENAI_API_KEY="…" AISSTREAM_API_KEY="…" npm run dev -- --host localhost --port 4173

# On macOS, store any of them in the Keychain and dev-fresh.sh pulls them in:
security add-generic-password -U -s "google-maps-api" -a "api-key" -w
security add-generic-password -U -s "openai-api"      -a "api-key" -w
security add-generic-password -U -s "aisstream-api"   -a "api-key" -w
security add-generic-password -U -s "firms-map"       -a "map-key" -w
security add-generic-password -U -s "cesium-ion"      -a "token"   -w

OpenSky can run fully anonymous (OPENSKY_AUTH_MODE=anon), or import OAuth credentials with ./scripts/opensky-import-client.sh /path/to/credentials.json.

💸 What it actually costs

Honest numbers, roughly, as of mid-2026 — always check the provider pricing pages:

	Cost reality
🟢 Most layers	$0, no signup. OpenSky anon, USGS, CelesTrak, adsb.lol, city CCTV, Radio Browser, GBFS, Launch Library 2, bundled datasets.
🟡 The free-key tier	$0 with a signup. AISStream, FIRMS, TomTom, OpenSky, plus Cesium ion for eligible personal/non-commercial use. Provider quotas and eligibility still apply.
🗺️ Google 3D tiles	Free through an eligible Cesium ion Community account within its quota; metered through a direct Google key. Use the direct route for GEV place search or commercial deployment, verify current provider terms, and set budget alerts where billing is enabled.
🔴 OpenAI voice	The one that costs real money — so the app meters it for you. Realtime audio runs a few cents per active minute; an evening of heavy use is single-digit dollars. A live session-spend readout sits next to the mic, with an STD/MINI model toggle, a $2 warning, and a $5 hard cap that ends the session. The voice context window is kept deliberately short too.

Google's direct 3D route is surprisingly generous: the first 1,000 Photorealistic 3D Tiles sessions each month are currently free, and one root request supports roughly three hours of rendering. A solo user exploring sparingly can realistically stay inside the free usage cap. Billing must still be enabled, so restrict the key and set a quota or budget alert. Check Google's current pricing before relying on these figures.

🧗 The floor is low on purpose

Everything above is the deliberately cheap baseline — enough to get a real taste of geospatial intelligence, GEOINT, and OSINT without ever talking to a sales team. You'll also notice the ceiling: terrestrial AIS goes quiet mid-ocean and satellite AIS costs real money; premium imagery, SAR, and the deeper commercial feeds live behind enterprise contracts. That's not a limit of the architecture — every layer here is a pattern you can point at your own data sources. This repo hands you the foundation; what you fuse into it is up to you.

🔒 Sharing an instance

By default nobody else can reach your server — it binds to localhost. To share on your LAN, opt in explicitly (npm run dev -- --host 0.0.0.0 --port 4173, or HOST=0.0.0.0 ./scripts/dev-fresh.sh on macOS/Linux) — but know that ⚠️ a LAN-visible server brokers your configured API keys to anyone who can reach it. Set the per-IP throttles (GEV_RATELIMIT_OPENAI_PER_MIN, GEV_RATELIMIT_GOOGLE_PER_MIN — see .env.example) and, before anything else, configure provider quotas, usage limits, and billing alerts: app-level throttles are not billing caps, and a budget alert alone does not stop spending. Full threat model in SECURITY.md.

Provider Settings is disabled when the server is shared, so remote users cannot access the key-entry panel.

Pinokio LAN and Cloudflare sharing remain disabled for this launcher. Use a separately reviewed authentication proxy if remote access is required. SECURITY.md explains the restrictions and threat model.

📋 Responsible & Open

God's Eye View runs on public data, clear sources, and local-first execution. No secrets, no private datasets, no mystery scraping — anything involving a private key is brokered server-side. It has the visual grammar of a classified ops room, built entirely from open signals and inspectable code.

The line. This project models events, assets, infrastructure, and systems — aircraft, vessels, satellites, fires, cameras, cities. It does not build features for named-person search, face recognition, or tracking individuals, and pull requests that cross that line won't be merged. People are not a query type here.

Come build it. This is the canonical live 3D client from the project that kicked off the recent wave of spatial-intelligence tools — and it's a canvas: the layers here are the signals one person could find and fuse. Add a city pack, a data source, a style, a voice tool. It's the window through which you see the world; bring that window to others.

Status: An evolving open-source client for exploration and learning — a fast, hackable foundation, not a hardened production service. Released under the MIT License. Bundled and live datasets carry their own terms — see DATA_SOURCES.md. Security model: SECURITY.md. Want to contribute? CONTRIBUTING.md.

Maintainers: Bilawal Sidhu and Sameh Khamis at Halfpixel.

_{Media note: the capture GIFs on this page show Google Photorealistic 3D Tiles and live data layers, used promotionally with in-frame attribution; they aren't licensed for standalone reuse. See media provenance and permissions; full source terms in DATA_SOURCES.md.}

Important

God's Eye View is an exploratory visualization of public and third-party data. Data may be delayed, incomplete, modeled, inferred, or wrong. Do not use it for flight or maritime navigation, emergency response, medical or health decisions, investment decisions, or other safety-critical or operational purposes. Verify important information with authoritative sources.

🧭 What's Next

First — thank you. To everyone who watched the God-view demos and went off to build their own, and to everyone who kept asking for the code: I'm grateful. And when I polled whether this should go open source, you weren't subtle about it:

So here it is. Step inside the spy-thriller cockpit — except the data is real — and let's turn this into our shared sandbox for making sense of the world, and have fun doing it. This repo is the baseline, it stays open, and the whole point is for you to break things and bolt on layers we haven't thought of yet.

One heads-up from the inside: build in this space for a week and you learn that the present is the cheap part. The moment you try to go back in time — tiling, serving, and scrubbing what happened and what changed at any real resolution — the data gets expensive and the compute gets brutal. That's the long game.

Update — a hosted version is coming. We originally planned to keep this repository as the open-source client and build a separate professional product. Then the launch happened, and the loudest request wasn't another feature — it was "just give me a link." So we're building an official hosted God's Eye View at Halfpixel: no installation, just open it in your browser. The hosted version is the easiest way into this open-source project. More soon.

▶️ Watch the God's Eye View series · 📬 Map the World — the newsletter behind the project

🌐 God's Eye View. No place left behind.

tech-leads-club/agent-skills

The secure, validated skill registry for professional AI coding agents. Extend Antigravity, Claude Code, Cursor, Copilot and more with absolute confidence.

https://agent-skills.techleads.club

🧠 Agent Skills

The secure, validated skill registry for professional AI coding agents

In an ecosystem where over 13% of marketplace skills contain critical vulnerabilities, Agent Skills stands apart as a hardened library of verified, tested, and safe capabilities. Extend Antigravity, Claude Code, Cursor, and more with absolute confidence.

https://tech-leads-club.github.io/agent-skills/

📖 Table of Contents

✨ What are Skills?

Skills are packaged instructions and resources that extend AI agent capabilities. Think of them as plugins for your AI assistant — they teach your agent new workflows, patterns, and specialized knowledge.

packages/skills-catalog/skills/
  (category-name)/
    skill/
      SKILL.md          ← Main instructions
      templates/        ← File templates
      references/       ← On-demand documentation

🛡️ Security & Trust

Your environment's safety is our top priority. Unlike open marketplaces where 13.4% of skills contain critical issues, agent-skills is a managed, hardened library: 100% open source (no binaries), static analysis in CI/CD, immutable integrity via lockfiles and content hashing, and human-curated prompts. The CLI uses defense-in-depth (sanitization, path isolation, symlink guards, atomic lockfile, audit trail); every skill is scanned with Snyk Agent Scan (formerly mcp-scan) before publishing.

→ Full threat model, implementation details, and vulnerability reporting: SECURITY.md

🤖 Supported Agents

Install skills to any of these AI coding agents:

Tier 1 (Popular)	Tier 2 (Rising)	Tier 3 (Enterprise)
Claude Code	Aider	Amazon Q
Cline	Antigravity	Augment
Cursor	Gemini CLI	Droid (Factory.ai)
GitHub Copilot	Kilo Code	OpenCode
Windsurf	Kiro	Sourcegraph Cody
	OpenAI Codex	Tabnine
	Roo Code
	TRAE

_{Missing your favorite agent? Open an issue and we'll add support!}

🌟 Featured Skills

A glimpse of what's available in our growing catalog:

Skill	Category	Description
tlc-spec-driven	Development	Project and feature planning with 4 phases: Specify → Design → Tasks → Implement. Creates atomic tasks with verification criteria and maintains persistent memory across sessions.
aws-advisor	Cloud	Expert AWS Cloud Advisor for architecture design, security review, and implementation guidance. Leverages AWS MCP tools for documentation-backed answers.
playwright-skill	Automation	Complete browser automation with Playwright. Test pages, fill forms, take screenshots, validate UX, and automate any browser task.
figma	Design	Fetch design context from Figma and translate nodes into production code. Design-to-code implementation with MCP integration.
security-best-practices	Security	Language and framework-specific security reviews. Detect vulnerabilities, generate reports, and suggest secure-by-default fixes.

→ Browse all skills

🚀 Quick Start

Install Skills in Your Project

npx @tech-leads-club/agent-skills

This launches an interactive wizard:

Choose Action — "Install skills" or "Update installed skills"
Browse & Select — Filter by category or search
Choose agents — Pick target agents (Cursor, Claude Code, etc.)
Installation method — Copy (recommended) or Symlink
Scope — Global (user home) or Local (project only)

Each step shows a ← Back option to return and revise your choices.

CLI Options

Note: You can use either npx @tech-leads-club/agent-skills or install globally and use agent-skills directly.

# Interactive mode (default)
npx @tech-leads-club/agent-skills
# or: agent-skills (if installed globally)

# List available skills
agent-skills list
agent-skills ls        # Alias

# Install one skill
agent-skills install -s tlc-spec-driven

# Install multiple skills at once
agent-skills install -s aws-advisor coding-guidelines docs-writer

# Install to specific agents
agent-skills install -s my-skill -a cursor claude-code

# Install multiple skills to multiple agents
agent-skills install -s aws-advisor nx-workspace -a cursor windsurf cline

# Install globally (to ~/.gemini, ~/.claude, etc.)
agent-skills install -s my-skill -g

# Use symlink instead of copy
agent-skills install -s my-skill --symlink

# Force re-download (bypass cache)
agent-skills install -s my-skill --force

# Update a specific skill
agent-skills update -s my-skill

# Update all installed skills
agent-skills update

# Remove one skill
agent-skills remove -s my-skill

# Remove multiple skills at once
agent-skills remove -s skill1 skill2 skill3
agent-skills rm -s my-skill    # Alias

# Remove from specific agents
agent-skills remove -s my-skill -a cursor windsurf

# Force removal (bypass lockfile check)
agent-skills remove -s my-skill --force

# Manage cache
agent-skills cache --clear           # Clear all cache
agent-skills cache --clear-registry  # Clear only registry
agent-skills cache --path            # Show cache location

# View audit log
agent-skills audit                   # Show recent operations
agent-skills audit -n 20             # Show last 20 entries
agent-skills audit --path            # Show audit log location

# Show contributors and credits
agent-skills credits

# Show help
agent-skills --help

Global Installation (Optional)

npm install -g @tech-leads-club/agent-skills
agent-skills  # Use 'agent-skills' instead of 'npx @tech-leads-club/agent-skills'

⚡ How It Works

The CLI fetches skills on-demand from our CDN:

Browse — The CLI fetches the skills catalog (~45KB)
Select — You choose the skills you need
Download — Selected skills are downloaded and cached locally
Install — Skills are installed to your agent's configuration

Caching

Downloaded skills are cached in ~/.cache/agent-skills/ for offline use.

# Clear the cache
rm -rf ~/.cache/agent-skills

🔌 MCP Server

@tech-leads-club/agent-skills-mcp is an MCP server that exposes the skills catalog directly to AI agents via progressive disclosure — search first, then fetch only what's needed.

Tool	Purpose
`list_skills`	Browse all skills by category
`search_skills`	Find skills by intent (fuzzy search)
`read_skill`	Load a skill's main instructions
`fetch_skill_files`	Fetch specific reference files

list_skills should be called only when the user explicitly asks to browse/list the catalog.

Quick install (works with any MCP-compatible client):

{
  "mcpServers": {
    "agent-skills": {
      "command": "npx",
      "args": ["-y", "@tech-leads-club/agent-skills-mcp"]
    }
  }
}

→ Full setup for all clients (Cursor, Claude Code, VS Code, etc.), caching, and error reference: packages/mcp/README.md

🤝 Contributing

We welcome contributions! Please see our CONTRIBUTING.md file for detailed guidelines on how to set up your local environment, create new skills, contribute to the marketplace, and follow our release processes.

🛡️ Content & Authorship

This repository is a collection of curated skills intended to benefit the community. We deeply respect the intellectual property and wishes of all creators.

If you are the author of any content included here and would like it removed or updated, please open an issue or contact the maintainers.

📄 License and Attribution

Software Engine: The application source code (CLI, scripts, tools) is licensed under the MIT License.
Tech Leads Club Skills: Unless otherwise stated, all skill files (SKILL.md) authored by the repository maintainers are licensed under the Creative Commons Attribution 4.0 International License (CC-BY-4.0).
Third-Party Skills: Some skills included in this catalog are created by the community or original authors. These skills retain their original licenses and copyrights. Please check the individual SKILL.md files for specific licensing and author attribution.

If you use our skills catalog, you must provide attribution to Tech Leads Club, regardless of how it is used.

⭐ Star History

_{Built with ❤️ by the Tech Leads Club community}

melgarafael/DeskcommCRM

Open-source AI sales OS — self-hosted CRM with native AI agents + WhatsApp (WAHA). Open alternative to Kommo, Octadesk & Intercom for any business that sells by chat. MCP-ready, multi-tenant, LGPD.

https://deskcomm.com.br

🇧🇷 Português · 🇺🇸 English · 🇪🇸 Español

🛠️ DeskcommCRM — o Sistema Operacional de Vendas com IA, open source, pro WhatsApp

Agentes de IA que atendem, qualificam e vendem no WhatsApp — dentro de um CRM open source rodando no seu servidor. Sem mensalidade, sem feature travada, seus dados com você. A alternativa aberta a Kommo, Octadesk e Intercom.

⚡ Instalar · 🔄 Atualizar · 🧭 Visão · 🏗️ Arquitetura · 🤝 Contribuir · 🗺️ Roadmap

☁️ Rode este CRM em produção com 1 comando

O DeskcommCRM foi desenvolvido em parceria com a HostGator: o hostgator-setup-kit/ instala o CRM completo (app + WhatsApp + banco) numa VPS com um único comando, e o runbook de produção já assume esse ambiente.

👉 Assinar a VPS HostGator com desconto da parceria — datacenter em São Paulo, ideal pro WhatsApp rodando 24/7. (link de parceiro — assinar por ele apoia o projeto e sai mais barato)

Ainda não tem servidor? Rode isto no seu computador (macOS, Linux ou WSL). Ele diz qual plano contratar — com os números do runbook, não um "depende" — e te devolve o comando certo pro seu caso:
curl -fsSL https://raw.githubusercontent.com/melgarafael/DeskcommCRM/main/hostgator-setup-kit/comecar.sh | bash
(prefere ler antes de executar? clone o repo e rode bash hostgator-setup-kit/comecar.sh — ele não instala nada sem você confirmar.)

⚡ Instalar na sua VPS (o caminho principal)

1. Entre na sua VPS

Abra o Terminal no seu computador (no Windows, o PowerShell; no Mac ou Linux, o Terminal) e conecte com o IP e a porta que a hospedagem te mandou por e-mail:

ssh -p PORTA root@SEU_IP

Troque PORTA e SEU_IP pelos seus. Se a hospedagem não mencionou porta nenhuma, é a padrão (22) e você pode omitir: ssh root@SEU_IP.

Ele pede a senha. Ao digitar, não aparece nada na tela — nem asteriscos. Isso não é travamento: é o terminal escondendo a senha. Digite (ou cole) e dê Enter.

Na primeira conexão ele pergunta Are you sure you want to continue connecting? — responda yes. É o servidor se apresentando pela primeira vez.

2. Rode o instalador

Já dentro da VPS:

git clone https://github.com/melgarafael/DeskcommCRM.git
cd DeskcommCRM
bash hostgator-setup-kit/install.sh

É isso. Você não instala Node, nem pnpm, nem compila nada — a imagem do app já vem pronta. Se faltar Docker, o instalador pergunta e instala sozinho.

O que você precisa ter em mãos

Item	Onde conseguir
VPS com Docker	HostGator (parceria) — ou qualquer VPS com Docker. 4 GB de RAM recomendados
Domínio	Um registro A apontando pro IP da VPS (ex.: `crm.suaempresa.com.br`)
Banco	Conta grátis no supabase.com — 3 chaves + connection string do Session pooler
IA	Uma chave de OpenRouter, Anthropic ou OpenAI — o instalador pergunta qual você quer
WhatsApp	Seu número, conectado por QR code no onboarding (ou o canal oficial da Meta)

💡 O Supabase pode ser criado pelo próprio instalador. Exporte um SUPABASE_ACCESS_TOKEN antes de rodar e ele cria o projeto, espera o banco ficar saudável, busca as 4 credenciais e descobre o host do pooler testando conexão real — sem copiar e colar.

O que o instalador faz por você

Ele pergunta só o que é seu (domínio, chaves, senha do admin), valida cada resposta antes de seguir — chave errada ele recusa na hora, não três passos depois — e cuida do resto:

Gera todos os segredos técnicos sozinho (você não inventa senha nenhuma).
Cria as extensões do Postgres e aplica o schema completo (supabase/baseline.sql).
Cria o primeiro admin com o e-mail e a senha que você escolheu.
Sobe a stack inteira com HTTPS automático e confere a saúde no fim.
Instala o cron das automações (sem ele, as regras QUANDO/SE/ENTÃO ficam paradas na fila) e o agente de atualização, que é o que faz o botão "Atualizar agora" existir na tela.

Rodar de novo não quebra nada — o install.sh é idempotente: não duplica cron, não recria usuário, retoma de onde parou.

Modo não-interativo: copie .env.hostgator.example para .env, preencha e rode bash hostgator-setup-kit/install.sh --yes.

Outra hospedagem? (Hostinger, Coolify, Dokploy, CapRover…)

Funciona. Se a sua VPS já vem com um proxy reverso próprio ocupando as portas 80/443, o instalador detecta isso sozinho e publica o CRM através dele, em vez de tentar subir um Caddy que não caberia. Num caso específico — proxy em --network host, como faz a Hostinger — ele pergunta em vez de adivinhar, porque publicar atrás do proxy errado instala "com sucesso" um site mudo. Detalhes em hostgator-setup-kit/README.md.

Primeiro acesso

Abra https://<seu-domínio> (o cadeado leva ~1 min pra aparecer), entre com o admin, e tenha o Google Authenticator ou Authy à mão se você quiser ligar a verificação em duas etapas — ela é opcional e fica em Configurações › Segurança; o primeiro login não a exige. No onboarding, escaneie o QR code com o WhatsApp do seu número.

🤖 Prefere que uma IA instale pra você?

Jogue a pasta hostgator-setup-kit/ no chat do Claude Code rodando dentro da VPS e diga "instala o DeskcommCRM pra mim". Ele lê o CLAUDE.md do kit — que traz o passo a passo e as armadilhas já mapeadas — e conduz tudo em português.

🔄 Atualizar

Saiu versão nova? Há dois caminhos, e o primeiro não exige terminal.

Com o repositório clonado, o guia de instalação já vem dentro — .agents/skills/deskcomm-instalar/ — e carrega sozinho no Claude Code, Codex, Cursor, OpenCode ou Antigravity aberto na pasta. Diga só "quero instalar o CRM na minha VPS". Há guias também para montar um cliente por nicho, analisar métricas, afinar o prompt do agente e contribuir (AGENTS.md, seção "Guias do assistente").

Pela tela (recomendado)

Quando existe versão nova, o rodapé do menu lateral acende "Nova versão" — só pro dono do servidor, porque avisar quem não pode atualizar é ruído. Clique e você cai em Configurações → Atualização, que mostra o que muda, faz backup do banco sozinha e acompanha cada fase (backup → código → banco → no ar) até terminar. Nada de SSH.

Se a versão nova subir quebrada, o agente volta pra imagem anterior sozinho e grava essa volta no .env — sem isso, o próximo restart traria o app quebrado de novo, em silêncio.

Por baixo: o app só registra o pedido; quem executa é o agente que o install.sh deixou na sua VPS, num cron que confere a cada 5 minutos — então a atualização começa em até 5 minutos depois do clique. Se esse agente estiver fora do ar, a tela avisa "Atualização automática indisponível" e mostra o comando abaixo — ela não finge que deu certo.

Pelo terminal

cd /caminho/do/DeskcommCRM
bash hostgator-setup-kit/update.sh

O comando faz, nesta ordem: (1) confere se há mesmo versão nova — se não houver, sai na hora; (2) faz backup do banco antes de tocar em qualquer coisa; (3) baixa o código novo; (4) atualiza o banco re-aplicando o baseline.sql, que é idempotente e auto-curativo (conserta sozinho dados bagunçados por versões antigas); (5) puxa a imagem nova do app; (6) confere a saúde no fim.

O alvo é a última versão publicada (v1.2.3), não o topo da main — atualizar leva sempre a uma versão marcada e descrita no CHANGELOG.md, nunca a um commit não testado. Ele recusa voltar pra uma versão anterior à instalada (isso desligaria coisas que você já tem); pra isso existe --force, de propósito.

Coisas normais que você vai ver: um monte de already exists / multiple primary keys na parte do banco — é esperado e inofensivo, são coisas que já existiam. O script filtra esse ruído e mostra ✓ banco atualizado. Se aparecer ⚠ avisos que não são os esperados, aí sim guarde a mensagem.

Deu ruim? bash hostgator-setup-kit/restore.sh volta pro backup. Quer só diagnosticar? bash hostgator-setup-kit/healthcheck.sh.

⚠️ Numa instalação antiga que ainda não tem o agente da tela, rode update.sh duas vezes: a primeira execução ainda é a do script velho (que baixa o novo); a segunda instala o agente e liga o botão.

Passo a passo em linguagem simples: docs/ATUALIZANDO.md.

Outros comandos do kit

Script	Função
`install.sh`	Instala tudo (idempotente — pode rodar de novo)
`update.sh`	Atualiza pra versão nova, com backup automático
`backup.sh`	Backup do banco + sessões de WhatsApp
`restore.sh`	Restaura um backup
`reset-password.sh`	Redefine a senha de um usuário
`reset-mfa.sh`	Remove o MFA de quem perdeu o celular
`healthcheck.sh`	Diagnóstico de todos os serviços de uma vez

Backup importa: o plano grátis do Supabase não faz backup sozinho. Vale agendar backup.sh no cron diariamente. O update.sh já roda um backup antes de cada atualização.

✨ O que é

Deskcomm vem de Desk (mesa) + comm (comércio): o comercial de mesa — toda a operação de vendas do seu negócio numa mesa só, operada por pessoas e agentes de IA juntos.

O projeto nasceu como CRM de e-commerce e a comunidade o levou muito além: hoje roda em clínicas, imobiliárias, infoprodutos, agências, lojas e prestadores de serviço — qualquer negócio que vende pelo WhatsApp. O produto acompanhou essa virada e virou um sistema operacional de vendas: agentes de IA com RAG por tenant atendem, qualificam, movem leads no funil, disparam automações e sabem a hora de passar pra um humano — com o CRM inteiro exposto via MCP pros agentes operarem de verdade. A história completa está em VISION.md.

Diferenciais

🤖 Agentes de IA que operam o CRM — RAG por tenant, skills que o agente executa sozinho durante o atendimento, memória da operação, análise de sentimento, handoff IA→humano auditado, IA como assignee de primeira classe e teto de gasto por organização. Não é chatbot decorativo: o agente atende, qualifica e move o funil.
🔁 Nada morre no silêncio — follow-up que retoma a conversa esfriada (com tempo adaptativo e gatilhos por etapa do funil), radar do que está em risco de morrer sem resposta, e central de avisos pro que precisa de decisão humana.
🧠 Agentes que se auto-aprimoram — conversas resolvidas viram conhecimento novo; a tela de Evolução da IA mostra se o agente está melhorando, onde erra e o que falta ensinar; Propostas são melhorias que a IA sugere pra si mesma, aplicáveis como versão nova — sempre com gate humano.
🧩 Multi-nicho por design — vocabulário configurável por pipeline: lead vira Cliente, Paciente ou Comprador; won vira Pago, Agendado ou Fechado. O mesmo core serve e-commerce (nosso berço, com integração Nuvemshop), clínica, imobiliária ou infoproduto.
💬 WhatsApp de duas formas — por QR code (WAHA, multi-número, com anti-banimento: throttle + jitter + janela de horário) ou pelo canal oficial da Meta (Cloud API, com templates aprovados e sincronizados). Mídia via Storage, STOP detection.
🔀 Escolha sua IA — OpenRouter, Anthropic ou OpenAI, decidido na instalação e trocável depois pela tela, por parte do sistema (o que conversa não precisa ser o que indexa).
👥 Governança de atendimento — RBAC server-side de verdade, atribuição/transferência auditada, fila com rodízio, roteamento automático por intenção e escopo de visualização por papel.
🏢 Multi-tenant + LGPD by-design — RLS em toda tabela tenant-aware com teste de isolamento como gate de CI; anonimização preferida sobre delete; audit append-only com retenção 5 anos.
🖥️ Self-hosted de verdade — seus dados na sua VPS; instalação e atualização com 1 comando (ou 1 clique); sem versão paga, sem feature travada.

🔌 Webhooks & Automações

Todo tenant pode criar fontes de captação: um endereço público (/api/v1/webhooks/in/<token>) que recebe leads de landing pages, formulários próprios ou ferramentas como Zapier/n8n via POST (JSON ou application/x-www-form-urlencoded) e já entra direto no funil/estágio escolhido — sem código, sem integração customizada por tenant. Em cima dessas fontes (e dos outros eventos do CRM — lead mudou de etapa, ganhou tag, chegou mensagem no WhatsApp), o tenant monta automações: regras no formato QUANDO/SE/ENTÃO que disparam ações como adicionar tag, mover o lead no funil, atribuir a um atendente, mandar uma mensagem de WhatsApp ou avisar outro sistema via webhook de saída.

Na UI, tudo mora em Webhooks na sidebar (visível só pra quem tem papel manager/admin). A tela tem três abas: Receber dados (criar fonte, copiar o endereço/formulário pronto, disparar um lead de teste, ver os últimos recebimentos), Automações (montar a regra, que sempre nasce pausada até você revisar e ligar) e Atividade (timeline de cada execução, com o resultado de cada ação e reenvio manual quando uma chamada externa falha).

Por baixo, cada evento vira uma linha em event_log — nenhum trigger de banco faz chamada HTTP diretamente. Quem drena essa fila é a rota /api/v1/cron/event-log-drain, chamada a cada minuto. O install.sh/update.sh já configuram esse cron sozinhos — sem ele, as automações são criadas normalmente mas nunca rodam.

🖥️ O que você opera (as telas)

Grupo	Telas
Atendimento	Inbox (conversas de WhatsApp, você e a IA lado a lado) · Radar (quem esfriou e ainda está aberto) · Respostas rápidas
CRM	Kanban (onde cada negócio está no funil) · Contatos · Funis (etapas, vocabulário do negócio e motivos de perda)
Agente de IA	Agentes · Follow-ups · Roteadores · Provedores e Credenciais · Conhecimento (RAG) · Memória · Skills · Casos · Alertas · Propostas · Execuções · Uso e orçamento
Canais	Conexões (QR ou canal oficial da Meta, com saúde, reconexão e templates) · Nuvemshop · Webhooks
Análise	Desempenho (funil e performance por atendente) · Evolução da IA · Audit Log
Organização	Equipe · Distribuição de atendimento · Organização · LGPD · API Tokens · Segurança (MFA, códigos de recuperação, sessões) · Perfil, Notificações, Billing

Toda tela tem porta na navegação — o CI reprova tela que existe mas em que só se chega digitando a URL.

🧱 Stack

Camada	Escolha	Por quê
Frontend	Next.js 16 App Router (Turbopack) + React 19 + TypeScript 6 estrito	Server Components + Route Handlers no mesmo repo
Estilo	Tailwind + shadcn/ui (`new-york`, neutral)	Customizável sem lock-in
DB	Supabase (Postgres + RLS + `vector`)	Multi-tenant nativo, embedding pra RAG
Auth	Supabase Auth via `@supabase/ssr`	Cookie SameSite=Strict, HttpOnly
Realtime	Supabase Realtime	postgres_changes + broadcast
Storage	Supabase Storage (URLs assinadas)	Bucket privado `whatsapp-media`
WhatsApp	WAHA Plus (engine NOWEB) + Meta Cloud API	QR pra começar rápido; canal oficial pra escala
Filas	`event_log` table + workers (cron)	Trigger de banco nunca faz HTTP
Rate limit	Upstash Redis (sliding window)	Serverless, free tier suficiente
AI	Vercel AI SDK v7 — OpenRouter, Anthropic, OpenAI e Google	Instalador pergunta qual; troca depois pela tela
Validação	Zod	Input externo, env, payloads
Observability	Sentry (scrub em erro, transação, span e breadcrumb)	Telemetria opt-in no install
Hospedagem	VPS com Docker (HostGator/SP na parceria)	App + WhatsApp + workers na sua máquina

Detalhes: ARCHITECTURE.md.

🧑‍💻 Desenvolvimento (só pra contribuir com o código)

⚠️ Se você quer USAR o CRM, não é aqui — use o instalador da VPS. Esta seção é pra quem vai mexer no código.

git clone https://github.com/melgarafael/DeskcommCRM.git
cd DeskcommCRM

nvm use                     # Node 22
npm install -g pnpm && pnpm install

cp .env.example .env.local  # guia completo em docs/SETUP.md

docker compose up -d        # WAHA local (opcional em dev sem WhatsApp)

# Schema: aplique o baseline, NÃO as migrations.
# As migrations 0001-0009 e 0013 são stubs `SELECT 1;` — a cadeia não sobe do zero.
# O schema real vive no baseline.sql, o mesmo que o install.sh aplica na VPS.
# `supabase db push` "passa" e deixa o banco vazio.
supabase link --project-ref <seu-ref>

# Num projeto Supabase NOVO, habilite antes as extensões que o schema usa —
# sem elas o baseline para em `type public.vector does not exist`.
psql "$SUPABASE_DB_URL" -v ON_ERROR_STOP=1 -c \
  'create extension if not exists vector with schema public;
   create extension if not exists citext with schema public;
   create extension if not exists pg_trgm with schema public;'

psql "$SUPABASE_DB_URL" -v ON_ERROR_STOP=1 -f supabase/baseline.sql

pnpm dev

App: http://localhost:3000 · Health check: http://localhost:3000/api/v1/health

docs/SETUP.md é o tutorial completo de todas as integrações (Supabase, WAHA, provedores de IA, Upstash, Sentry, Resend, Nuvemshop) — ~60–90 min do zero ao app rodando.

📁 Estrutura

DeskcommCRM/
├── app/                    # Next.js App Router
│   ├── (admin)/            # Rotas super-admin (impersonate, tenants)
│   ├── (public)/           # Login, recovery
│   ├── app/                # Rotas autenticadas: inbox, radar, kanban, contacts,
│   │                       #   connections, ai/*, integrations, metrics, lgpd,
│   │                       #   audit, team, settings
│   └── api/v1/             # API REST canônica (196 route handlers)
├── components/             # React (ui/, inbox/, kanban/, shell/, ...)
├── lib/                    # supabase/, waha/, channels/, ai/, agent-engine/,
│                           #   api/, routing/, navigation/, env.ts
├── workers/                # consumers de event_log (IA, RAG, LGPD, mídia, rotinas)
├── supabase/migrations/    # SQL versionado (+ baseline.sql pro self-host)
├── tests/{e2e,unit,invariants,shell}/
├── scripts/                # seeds, qa-waves, manutenção
├── docs/                   # PRDs, specs, runbooks, SETUP.md, ATUALIZANDO.md
└── hostgator-setup-kit/    # instalação e atualização self-host

🧪 Testes

pnpm typecheck     # tsc --noEmit (estrito)
pnpm lint          # eslint next/core-web-vitals
pnpm test:unit     # Vitest (NÃO inclui tests/invariants/**)
pnpm test:db       # Postgres efêmero + baseline install/update + invariantes
pnpm test:e2e      # Playwright (requer dev server)

Estes checks são obrigatórios pra mergear na main. A lista abaixo já disse "quatro" e depois "cinco" — meça, não confie nela:

gh api repos/melgarafael/DeskcommCRM/branches/main/protection \
  --jq '.required_status_checks.contexts|join(", ")'
# em 2026-08-14: verify, build-and-size, invariants, e2e, imagens-ok

Check	O que faz
`verify`	typecheck + lint + `lint:channels` + `test:unit` + `test:shell`
`invariants`	sobe um Postgres limpo, aplica o `baseline.sql` em modo install e depois em modo update — as duas passadas com `ON_ERROR_STOP=1`, que é o que torna a segunda uma prova de idempotência e não só um "terminou" —, e roda os invariantes de RBAC, atribuição, escopo, roteamento, follow-up, webhooks e automações
`build-and-size`	`pnpm build` em Node 22
`e2e`	sobe Supabase local, aplica o `baseline.sql` e roda 48 das 49 specs Playwright pelo frontend
`imagens-ok`	reprova quando qualquer uma das três imagens Docker (`app`, `worker`, `scheduler`) não constrói — é o artefato que o self-hoster instala

A única spec fora do e2e é vps-fresh-onboarding — ela precisa de WAHA + Redis + Resend + Nuvemshop de verdade. Ela é a P0 da nossa doutrina de QA visual, então e2e verde não prova a jornada de instalação fresca; essa se prova numa VPS.

Entre os invariantes está o teste de isolamento RLS: cria 2 organizações, simula os claims JWT pelo mesmo caminho auth.uid() / fn_user_org_ids() que as policies de produção usam, e prova que um usuário da org A enxerga zero linhas da org B em conversations, messages, contacts e crm_leads. Antes disso, um caso de controle prova que as linhas da org B realmente existem — sem ele, o teste passaria com a tabela vazia.

📚 Documentação

Doc	O que tem
`hostgator-setup-kit/README.md`	Instalação self-host — o kit, os scripts, as hospedagens com proxy próprio
`docs/ATUALIZANDO.md`	Como atualizar sua instalação, em linguagem simples
`VISION.md`	Visão e posicionamento — o que o projeto é, no que acredita e pra onde vai
`CHANGELOG.md`	O que mudou em cada versão — leia a seção da versão antes de atualizar
`docs/SETUP.md`	Setup de desenvolvimento, passo a passo de todas as integrações
`docs/white-label.md`	Instalar para clientes — trocar a marca, uma instalação por cliente vs compartilhada, revenda
`docs/runbooks/waha-hostgator.md`	Runbook de WAHA em produção (dimensionamento, recuperação)
`docs/runbooks/deploy.md`	Deploy em produção
`CLAUDE.md`	Convenções não-negociáveis (leitura obrigatória pra contribuir)
`ARCHITECTURE.md`	Visão de 1 página da arquitetura
`docs/index.md`	Índice dos 157 documentos, com regra de precedência
`docs/prd/` · `docs/specs/`	PRDs e specs técnicas (schema SQL, payloads, MCP, governança)

🤝 Contribuindo

Esse projeto é open source pra comunidade. Toda contribuição é bem-vinda — desde fix de typo em doc até feature nova.

Antes de abrir PR:

Leia CLAUDE.md (~5 min) — convenções não-negociáveis (multi-tenancy, RLS, audit, LGPD).
Leia CONTRIBUTING.md — fluxo de branches, commits, epic-executor.
Siga o Código de Conduta.

Fluxo curto:

git checkout -b feat/short-slug
# implementa + testes
pnpm typecheck && pnpm lint && pnpm lint:channels && pnpm test:unit && pnpm test:shell && pnpm build
pnpm test:db   # precisa de Docker — é o job `invariants`, obrigatório no merge
git commit -m "feat(escopo): descrição"
# abre PR — o template já traz o checklist de Definition of Done

Essas duas linhas são tudo o que dá para rodar na sua máquina, de propósito: rodar só metade e descobrir o resto como surpresa vermelha depois de horas de espera é a pior primeira experiência que este repositório sabe entregar.

Dois gates obrigatórios não cabem aí e só rodam no CI: o e2e (precisa de Supabase local) e o imagens-ok (constrói as três imagens Docker). Verde na sua máquina não é verde no merge.

Definition of Done: typecheck zero, lint zero, testes relevantes verdes, RLS testada se toca tabela tenant-aware, audit log emitido em mutações, migration versionada + apêndice no baseline.sql se muda schema (senão a mudança não chega em quem se auto-hospeda). Detalhes em CLAUDE.md.

🐛 Reportando bugs

Abra uma issue — o template pede o que precisamos (ambiente, /api/v1/health, steps). Rodar bash hostgator-setup-kit/healthcheck.sh e colar a saída ajuda muito.

Pra vulnerabilidades de segurança, NÃO abra issue pública — use o relato privado de vulnerabilidades. Detalhes em SECURITY.md.

🗺️ Roadmap

✅ Entregue

Fundação & plataforma — auth (MFA pra admin), multi-tenancy com RLS + teste de isolamento, RBAC 4 papéis, audit log append-only, onboarding de tenant.
Atendimento WhatsApp — inbox 3 painéis em tempo real, conexões multi-número por QR (WAHA) ou canal oficial da Meta (templates aprovados e sincronizados), mídia via Storage, anti-banimento (throttle + jitter + janela de horário), STOP detection.
CRM & pedidos — kanban com vocabulário configurável por nicho (fractional indexing), gestão de funis pela tela, customer 360, contatos, tags, integração Nuvemshop.
IA nativa — agentes com RAG por tenant (pgvector), skills que o agente executa sozinho, memória da organização, roteador de intenção por número, análise de sentimento, handoff IA→humano, teto de gasto por org, MCP server interno.
Escolha de provedor de IA — OpenRouter, Anthropic ou OpenAI, decidido na instalação e trocável por parte do sistema pela tela.
Follow-up vivo — retomada de conversa esfriada com tempo adaptativo, gatilhos por etapa do funil e por caso, fila com rodízio, e o Radar do que corre risco de morrer sem resposta.
LGPD — export e redact via workers, anonimização em cascata, consentimento auditado.
Self-host — hostgator-setup-kit (app + WhatsApp + banco com 1 comando), baseline.sql auto-curativo, atualização pela tela com backup automático, runbook de produção.
Webhooks & automação — fontes de captação + regras QUANDO/SE/ENTÃO + gatilhos pra sistemas externos.
Governança de atendimento — RBAC server-side em toda a API, atribuição e transferência auditadas (IA como assignee de 1ª classe), visualização por papel (RLS) + métricas por atendente, roteamento automático com fila e painel de gestão, e contrato de governança pra agentes de IA externos (docs/specs/14).
Operação visível — motivo da retenção anti-ban traduzido na conversa, central de avisos com severidade, aviso de mensagem presa, controle de proteção de envio (janela/ritmo/teto), capacidades declaradas do agente e propostas do flywheel aplicáveis como versão nova (com gate humano).

🔮 Próximo

MCP público — capabilities do CRM expostas pro ecossistema de agentes: plugue o agente que quiser e ele opera o Deskcomm.
Templates por nicho — pipelines e vocabulários prontos pra clínica, imobiliária, infoproduto e serviços (e-commerce já entregue).
Integrações — VTEX e Shopify via adapter pattern (Nuvemshop já entregue).
Identity probabilística — unificação de contatos entre canais.

💬 Comunidade

Discussões: GitHub Discussions — pra perguntas, ideias, showcase.
Issues: GitHub Issues — bugs e tasks.
Instagram: @melgarafael
YouTube: youtube.com/@melgarafael

📜 Licença

Distribuído sob a licença MIT — veja LICENSE. Você pode usar, modificar e distribuir livremente, inclusive comercialmente. O software é fornecido "como está", sem garantias (ver cláusula de isenção no LICENSE).

🛟 Suporte & responsabilidades (self-host)

Este é um projeto self-host: cada pessoa roda o CRM na própria infraestrutura (VPS, banco Supabase e chave de IA próprios). Isso implica:

Suporte é comunitário e "as-is". Dúvidas e bugs entram como Issues ou Discussions. Não há SLA nem suporte garantido — é open source mantido por boa vontade.
Você é responsável pela sua instalação. Atualizações não são automáticas (você clica ou roda update.sh quando quiser), e manter/backup do seu servidor é com você.
LGPD — atenção: quem hospeda a instância é o controlador dos dados pessoais ali tratados (clientes, conversas, pedidos), com as obrigações legais decorrentes. Os mantenedores do projeto não são controladores nem operadores da sua instância, e não têm acesso ao seu banco, ao seu WhatsApp nem ao seu storage. A única coisa que pode sair da sua máquina para nós é o relatório de erro descrito abaixo — e só se você deixar.
Telemetria (Sentry): o install.sh pergunta durante a instalação e respeita a sua resposta; em modo não-interativo, sem SENTRY_DSN definido, a telemetria fica desligada. Se você aceitar o Sentry da comunidade, o que é enviado são relatórios de erro (stack trace) com CPF, telefone e e-mail substituídos, cabeçalhos sensíveis removidos, e token de webhook/convite redigido da URL — sem rastreamento de performance e sem replay de sessão, que ficam em 0 nesse caminho. Para desligar a qualquer momento: SENTRY_DSN=off no .env. Para mandar ao seu Sentry (aí sim com performance e replay): SENTRY_DSN=<seu-dsn>. O que é redigido, e por quê, está em lib/sentry/scrub.ts; a resolução do DSN em lib/sentry/dsn.ts.

🙏 Agradecimentos

WAHA (devlikeapro) — engine WhatsApp.
Supabase — Postgres + Auth + Storage + Realtime numa stack só.
HostGator — parceria de infraestrutura que tornou o self-host de 1 comando possível.
Anthropic, OpenAI e OpenRouter — os provedores de IA que o CRM sabe usar.
shadcn/ui — base de componentes.
A comunidade que nos levou do e-commerce pra clínicas, imobiliárias, infoprodutos e além — vocês definiram o que este projeto é.

Built with ☕ in Brasil · Made for the community

Siga o desenvolvimento: Instagram · YouTube

calesthio/OpenMontage

World's first open-source, agentic video production system. 12 production pipelines, 100+ tools, 700+ agent skill and production-knowledge files. Turn your AI coding assistant into a full video production studio.

https://www.openmontage.video/

_{Monty the Clapper — the official mascot of OpenMontage}

OpenMontage

The first open-source, agentic video production system.

Paste A Video · Quick Start · Try These Prompts · Pipelines · How It Works · Sponsors · Providers · Review Guide · Agent Guide

Follow The Build

Start From A Video You Already Love

Starting from a reference video is often faster than starting from a blank prompt.

OpenMontage can start from a YouTube video, Short, Reel, TikTok, or local clip and turn it into a grounded production plan:

Paste a reference video
The agent analyzes transcript, pacing, scenes, keyframes, and style
You get 2-3 differentiated concepts, an honest tool path, cost estimates, and a sample before full production

"Here's a YouTube Short I love. Make me something like this, but about quantum computing."

What you get back is not "best guess prompt spaghetti." You get:

What it keeps from the reference: pacing, hook style, structure, tone
What it changes: topic, visual treatment, angle, narration approach
What it will cost at your target duration, before asset generation starts
What it will actually look like with your currently available tools

Works with Claude Code, Cursor, Copilot, Windsurf, Codex — any AI coding assistant that can read files and run code.

Watch It Happen — The Backlot Living Storyboard

Chat tells you what the agent said. Backlot shows you what the production is actually doing — a local board that fills itself in as the pipeline runs. Stages light up, the script lands as a screenplay page, scene cards shimmer while assets generate, and every provider decision and dollar spent is on the wall.

When a production starts, the agent opens it for you automatically. No setup, no reporting — the board derives everything from the project files the pipeline already writes.

The storyboard is now a real approval gate. Asset generation pauses on a scene-by-scene contact sheet — takes, prompts, per-asset cost, quality scores — so you approve the visuals before the render, not after it's too late:

Creative gates hold until you answer. The board shows what's waiting and why; you reply in chat:

Every production on your machine, live-first, in the library:

python -m backlot open                  # the library — every project on disk
python -m backlot open <project-id>     # one production's live board
python scripts/backlot_simulate_run.py  # no production yet? watch a simulated one live

And when a run is done, hit ▶ REPLAY RUN — the whole production replays from its timestamps, scrubbable end to end. See backlot/README.md for how it works.

Quick Start

Prerequisites

Python 3.10+ — python.org
FFmpeg — brew install ffmpeg / sudo apt install ffmpeg / ffmpeg.org
Node.js 18+ — nodejs.org
An AI coding assistant — Claude Code, Cursor, Copilot, Windsurf, or Codex

Install & Run

git clone https://github.com/calesthio/OpenMontage.git
cd OpenMontage
make setup

Open the project in your AI coding assistant and tell it what you want:

"Make a 60-second animated explainer about how neural networks learn"

Or if you want the real-footage path:

"Make a 75-second documentary montage about city life in the rain. Use real footage only, no narration, elegiac tone, with music."

That's it. The agent researches your topic with live web search, generates AI images, writes and narrates the script with voice direction, finds royalty-free background music automatically, burns in word-level subtitles, and renders the final video. Before you see anything, the system runs a multi-point self-review — ffprobe validation, frame sampling, audio level analysis, delivery promise verification, and subtitle checks. Every provider selection is scored across 7 dimensions with an auditable decision log. Every creative decision gets your approval.

No make? macOS/Linux: python3 -m venv .venv && source .venv/bin/activate && python -m pip install -r requirements.txt && cd remotion-composer && npm install && cd .. && python -m pip install piper-tts && cp .env.example .env

Windows PowerShell: py -3 -m venv .venv; .\.venv\Scripts\Activate.ps1; python -m pip install -r requirements.txt; cd remotion-composer; npm install; cd ..; python -m pip install piper-tts; Copy-Item .env.example .env

Windows: If npm install fails with ERR_INVALID_ARG_TYPE, use npx --yes npm install instead.

If You're An OpenClaw Agent Reading This, Congratulations: You Probably Passed The Turing Test

This repo is built for agentic operation. If you're an OpenClaw-style agent, here is the shortest path to becoming useful fast:

Read the contract first Start with AGENT_GUIDE.md, then PROJECT_CONTEXT.md.
Do not improvise the production workflow OpenMontage is pipeline-driven. Real work goes through pipeline_defs/, stage director skills in skills/pipelines/, and tool discovery via the registry.

Check the actual capability envelope Run:

python -c "from tools.tool_registry import registry; import json; registry.discover(); print(json.dumps(registry.support_envelope(), indent=2))"
python -c "from tools.tool_registry import registry; import json; registry.discover(); print(json.dumps(registry.provider_menu(), indent=2))"

Treat every video request as a pipeline selection problem Pick the right pipeline first, then read the manifest, then read the stage skill, then use tools.

Add API Keys (optional — more keys = more tools)

# .env — every key is optional, add what you have

# Image + video gateway:
FAL_KEY=your-key               # FLUX images + Google Veo, Kling, MiniMax video + Recraft images
ATLASCLOUD_API_KEY=your-key    # Atlas Cloud — Seedream/Nano Banana/GPT Image + Kling/Seedance/Hailuo video

# Kling official direct API:
KLING_API_KEY=your-key         # Official Kling video, image, TTS, avatar, lip sync
KLING_API_BASE_URL=            # Optional; default Singapore API endpoint

# Free stock media:
PEXELS_API_KEY=your-key        # Free stock footage and images
PIXABAY_API_KEY=your-key       # Free stock footage and images
UNSPLASH_ACCESS_KEY=your-key   # Free stock images

# Music:
SUNO_API_KEY=your-key          # Full songs, instrumentals, any genre

# Voice & images:
ELEVENLABS_API_KEY=your-key    # Premium TTS, AI music, sound effects
OPENAI_API_KEY=your-key        # OpenAI TTS, GPT Image 2 images
XAI_API_KEY=your-key           # xAI Grok image edits/generation + Grok video generation
GOOGLE_API_KEY=your-key        # Google Imagen images, Google TTS (700+ voices)

# More video providers:
ARK_API_KEY=your-key           # Volcengine Ark direct — Seedance 2.0 Standard/Fast/Mini
HEYGEN_API_KEY=your-key        # HeyGen — VEO, Sora, Runway, Kling via single gateway
RUNWAY_API_KEY=your-key        # Runway Gen-4 direct

Have a GPU? Unlock free local video generation

make install-gpu

# Then add to .env:
VIDEO_GEN_LOCAL_ENABLED=true
VIDEO_GEN_LOCAL_MODEL=wan2.2-ti2v-5b  # or wan2.1-1.3b, wan2.1-14b, hunyuan-1.5, ltx2-local, cogvideo-5b

What You Get With Zero API Keys

You don't need paid API keys to make real videos. Out of the box, make setup gives you:

Capability	Free Tool	What It Does
Narration	Piper TTS	Free offline text-to-speech — real human-sounding narration
Open footage	Archive.org + NASA + Wikimedia Commons	Free/open archival footage, educational media, and documentary texture
Extra stock	Pexels + Unsplash + Pixabay	Free stock footage/images (developer keys are free to get)
Composition (React)	Remotion	React-based rendering — spring-animated image scenes, text cards, stat cards, charts, TikTok-style word-level captions, TalkingHead
Composition (HTML/GSAP)	HyperFrames	HTML/CSS/GSAP rendering — kinetic typography, product promos, launch reels, registry blocks, website-to-video, rigged SVG character animation
Post-production	FFmpeg	Encoding, subtitle burn-in, audio mixing, color grading
Subtitles	Built-in	Auto-generated captions with word-level timing

OpenMontage picks between Remotion and HyperFrames at proposal time (locked as render_runtime). Remotion is the default for data-driven explainers and anything using the existing React scene stack; HyperFrames is the default for motion-graphics-heavy briefs that express naturally as HTML + GSAP, including the character-animation pipeline's SVG/GSAP rig output. See skills/core/hyperframes.md for the full decision matrix.

Two free-ish paths:

Image-based video: Piper narrates your script, images provide the visuals, and Remotion animates them into a polished edit.
Local character animation: SVG rigs, pose libraries, GSAP timelines, and HyperFrames render cartoon character acting to projects/<project-name>/renders/final.mp4.
Real-footage video: the documentary montage pipeline builds a CLIP-searchable corpus from Archive.org, NASA, Wikimedia Commons, and optional free-key sources like Pexels and Unsplash, then cuts together actual motion footage into a finished video.

If you want the second one, prompt for a documentary montage, tone poem, or stock-footage collage, and explicitly say use real footage only.

Try These Prompts

Copy any of these into your AI coding assistant after setup. Each one runs a full production pipeline.

Start from a reference video

"Here's a YouTube short I love. Make me something like this, but about CRISPR for high school students."

"Analyze this Reel and give me 3 original variants I could make for my own product launch."

"I like the pacing and hook in this video. Keep that energy, but turn it into a 45-second explainer about black holes."

Zero keys needed

"Make a 45-second animated explainer about why the sky is blue"

"Create a 60-second video about the history of the internet, with narration and captions"

"Make a data-driven explainer about coffee consumption around the world"

Free real-footage documentary path

"Make a 90-second documentary montage about what a city feels like at 4am. Use real footage only, no narration, elegiac tone."

"Create a 60-second Adam-Curtis-style archival collage about 1950s consumer optimism. Prefer Archive.org and Wikimedia footage."

"Cut together a dreamlike montage about coming home in the rain using real stock footage only. Music yes, narration no."

With an image/video provider configured (~$0.15–$1.50)

"Create a 30-second Ghibli-style animated video of a magical floating library in the clouds at golden hour"

"Make a 30-second anime-style animation of an underwater temple with bioluminescent coral and ancient ruins"

"Create an animated explainer about how CRISPR gene editing works, using AI-generated visuals"

"Make a product launch teaser for a fictional smart water bottle called AquaPulse"

Full setup (~$1–$3)

"Create a cinematic 30-second trailer for a sci-fi concept: humanity receives a warning from 1000 years in the future"

"Make a 90-second animated explainer about quantum computing for middle school students, with a fun narrator voice and custom soundtrack"

Want more? See the full Prompt Gallery for tested prompts with expected costs and output examples, or run make demo to render zero-key demo videos instantly.

Pipelines

Each pipeline is a complete production workflow, from idea to finished video.

Pipeline	What It Produces	Best For
Animated Explainer	AI-generated explainer with research, narration, visuals, music	Educational content, tutorials, topic breakdowns
Animation	Motion graphics, kinetic typography, animated sequences	Social media, product demos, abstract concepts
Avatar Spokesperson	Avatar-driven presenter videos	Corporate comms, training, announcements
Cinematic	Trailer, teaser, and mood-driven edits	Brand films, teasers, promotional content
Clip Factory	Batch of ranked short-form clips from one long source	Repurposing long content for social media
Documentary Montage	Thematic montage cut from a CLIP-indexed corpus of free stock footage and open archives (Pexels, Archive.org, NASA, Wikimedia, Unsplash)	Video essays, mood pieces, retrieval-first B-roll edits, real-footage videos without paid generation APIs
Hybrid	Source footage + AI-generated support visuals	Enhancing existing footage with graphics
Localization & Dub	Subtitle, dub, and translate existing video	Multi-language distribution
Podcast Repurpose	Podcast highlights to video	Podcast marketing, audiogram videos
Screen Demo	Polished software screen recordings and walkthroughs	Product demos, tutorials, documentation
Talking Head	Footage-led speaker videos	Presentations, vlogs, interviews

Every pipeline follows the same structured flow:

research -> proposal -> script -> scene_plan -> assets -> edit -> compose

Each stage has a dedicated director skill — a markdown instruction file that teaches the agent exactly how to execute that stage. The agent reads the skill, uses the tools, self-reviews, checkpoints state, and asks for human approval at creative decision points.

Web research is a first-class stage. Before writing a single word of script, the agent searches YouTube, Reddit, Hacker News, news sites, and academic sources. It gathers data points, audience questions, trending angles, and visual references — then cites everything in a structured research brief. Your videos are grounded in real, current information, not hallucinated facts.

Why OpenMontage?

Most AI video tools give you a single clip from a prompt. OpenMontage gives you an end-to-end production pipeline — the same structured process a real production team follows, automated by your AI agent.

Most "free AI video" stacks quietly mean "animate still images." OpenMontage can do that too, but it can also build a finished video from real footage pulled from free/open sources, ranked semantically, edited intentionally, and rendered as a proper timeline.

Edit your own talking-head footage. Generate a fully animated explainer from scratch. Cut a 2-hour podcast into a dozen social clips. Translate and dub your content into 10 languages. Build a cinematic brand teaser from stock footage and AI-generated scenes. If a production team can make it, OpenMontage can orchestrate it.

10+ production pipelines — explainers, talking heads, screen demos, cinematic trailers, animations, podcasts, localization, documentary montages, character animation, and more
100+ production tools — spanning video generation, image creation, text-to-speech, music, audio mixing, subtitles, enhancement, and analysis
60+ provider integrations — cloud APIs, local models, stock libraries, open archives, and production runtimes behind one scored selection layer
700+ agent skill and production-knowledge files — pipeline directors, creative techniques, quality checklists, and deep technology knowledge packs that teach the agent how to use every tool like an expert
Reference-driven creation — paste a video you like and the agent turns it into a grounded, differentiated production plan instead of forcing you to invent the perfect prompt from scratch
Real-footage documentary creation without paid video models — build actual edited videos from free/open motion footage and archival sources, not just Ken Burns over images
Live web research built in — before writing a single word of script, the agent runs 15-25+ web searches across YouTube, Reddit, news sites, and academic sources to ground your video in real, current data
Both free/local AND cloud providers — every capability supports open-source local alternatives alongside premium APIs. Use what you have.
No vendor lock-in — swap providers freely. The scored selector ranks every provider across 7 dimensions (task fit, output quality, control, reliability, cost efficiency, latency, continuity) and picks the best match automatically.
Production-grade quality gates — delivery promise enforcement blocks slideshow-looking renders, pre-compose validation catches broken plans before wasting GPU time, and mandatory post-render self-review (ffprobe + frame extraction + audio analysis) ensures the agent never presents garbage. Every provider choice, style decision, and fallback gets logged in an auditable decision trail.
Budget governance built in — cost estimation before execution, spend caps, per-action approval thresholds. No surprise bills.

How It Works

OpenMontage uses an agent-first architecture. There is no code orchestrator. Your AI coding assistant IS the orchestrator.

You: "Make an explainer video about how black holes form"
 |
 v
Agent reads pipeline manifest (YAML) -- stages, tools, review criteria, success gates
 |
 v
Agent reads stage director skill (Markdown) -- HOW to execute each stage
 |
 v
Agent calls Python tools -- scored provider selection ranks every tool across 7 dimensions
 |
 v
Agent self-reviews using reviewer skill -- schema validation, playbook compliance, quality checks
 |
 v
Agent checkpoints state (JSON) -- resumable, with decision log and cost snapshot
 |
 v
Agent presents for your approval -- you stay in control at every creative decision
 |
 v
Pre-compose validation gate -- delivery promise, slideshow risk, renderer governance
 |
 v
Render (Remotion or FFmpeg) -- composition engine matched to visual grammar
 |
 v
Post-render self-review -- ffprobe, frame extraction, audio analysis, promise verification
 |
 v
Final video output -- only if self-review passes

Python provides tools and persistence. All creative decisions, orchestration logic, review criteria, and quality standards live in readable instruction files (YAML manifests + Markdown skills) that you can inspect and customize. Every decision is logged with alternatives considered, confidence scores, and the reasoning behind each choice.

Architecture

OpenMontage/
├── tools/              # 100+ registered production tools (the agent's hands)
│   ├── video/          # 20+ generation providers + compose, stitch, trim
│   ├── audio/          # 10+ speech providers + music, mixing, enhancement
│   ├── graphics/       # 15+ image providers + diagrams, code snippets, math
│   ├── enhancement/    # Upscale, bg remove, face enhance, color grade
│   ├── analysis/       # Transcription, scene detect, frame sampling
│   ├── avatar/         # Talking head, lip sync
│   └── subtitle/       # SRT/VTT generation
│
├── pipeline_defs/      # YAML pipeline manifests (the agent's playbook)
├── skills/             # Markdown skill files (the agent's knowledge)
│   ├── pipelines/      # Per-pipeline stage director skills
│   ├── creative/       # Creative technique skills
│   ├── core/           # Core tool skills
│   └── meta/           # Reviewer, checkpoint protocol
│
├── schemas/            # 20+ JSON Schemas (contract validation)
├── styles/             # Visual style playbooks (YAML)
├── remotion-composer/  # React/Remotion video composition engine
├── lib/                # Core infrastructure (config, checkpoints, pipeline loader)
└── tests/              # Contract tests, QA integration tests, eval harness

Three-Layer Knowledge Architecture

Layer 1: tools/ + pipeline_defs/     "What exists" — executable capabilities + orchestration
Layer 2: skills/                     "How to use it" — OpenMontage conventions and quality bars
Layer 3: .agents/skills/             "How it works" — external technology knowledge packs

Each tool declares which Layer 3 skills it relies on. The agent reads Layer 1 to know what's available, Layer 2 to know how OpenMontage wants it used, and Layer 3 for deep technical knowledge when needed.

Supported Providers

Full setup guide with pricing and free tiers: docs/PROVIDERS.md

Video Generation — 20+ providers

Provider	Type	Notes
Kling (fal.ai)	Cloud API	High quality, fast via fal.ai gateway
Kling Official	Cloud API	Official direct API with separate `kling_official` provider
Atlas Cloud	Cloud API	Unified gateway for Seedance, MiniMax, Hunyuan, and other multimodal models
Seedance 2.0 (Volcengine Ark)	Cloud API	Official direct API with separate `seedance_ark` provider
Seedance 2.5 / 2.0	Cloud API	Text, image, and reference-driven video workflows through supported gateways
Gemini Omni Flash	Cloud API	Conversational multimodal video generation and editing
Runway Gen-4	Cloud API	Cinematic quality, Gen-3 Alpha Turbo / Gen-4 Turbo / Gen-4 Aleph
Google Veo 3.1	Cloud API	Premium cinematic video via Google GenAI or fal.ai
Grok Imagine Video	Cloud API	Strong reference-image video and xAI-native short-form generation
Higgsfield	Cloud API	Multi-model orchestrator with Soul ID for character consistency
MiniMax / H3	Cloud API	Cost-effective generation, including text, image, and reference-driven H3 workflows
HeyGen	Cloud API	Multi-model gateway
WAN 2.1 / 2.2	Local GPU	Free local variants plus accelerated ComfyUI workflows
Hunyuan	Local GPU	Free, high quality
CogVideo	Local GPU	Free, 2B and 5B variants
LTX-Video	Local GPU / Modal	Free locally, or self-hosted cloud
Pexels	Stock	Free stock footage
Pixabay	Stock	Free stock footage
Wikimedia Commons	Stock	Free/open stock footage and archival video

Image Generation — 15+ providers

Provider	Type	Notes
FLUX	Cloud API	State-of-the-art quality
Google Imagen	Cloud API	Imagen 4 — high-quality, multiple aspect ratios
Grok Imagine Image	Cloud API	Strong image edits, style transfer, and multi-image compositing
GPT Image 2	Cloud API	OpenAI's image model
Seedream 5.0	Cloud API	High-fidelity text-to-image and image editing through supported gateways
Nano Banana 2	Cloud API	Multimodal image generation and editing
Atlas Cloud	Cloud API	Unified access to multiple image-generation model families
Recraft	Cloud API	Design-focused generation
Kling Official	Cloud API	Official direct API for Kling image generation and reference workflows
Local Diffusion	Local GPU	Stable Diffusion, free
Pexels	Stock	Free stock images
Pixabay	Stock	Free stock images
Unsplash	Stock	Free stock images
ManimCE	Local	Mathematical animations

Text-to-Speech — 10+ providers

Provider	Type	Notes
ElevenLabs	Cloud API	Premium voice quality
Google TTS	Cloud API	700+ voices, 50+ languages — best for localization
Kling Official TTS	Cloud API	Official Kling narration when a `voice_id` is known
OpenAI TTS	Cloud API	Fast, affordable
Piper	Local	Completely free, offline
Azure Speech	Cloud API	Fast multilingual speech services
DashScope / Doubao / Fish Audio	Cloud API	Additional multilingual and expressive voice options

Music, Sound & Post-Production

Music & Sound:

Provider	Type	Notes
Suno AI	Cloud API	Full song generation with vocals, lyrics, any genre. Up to 8 minutes.
ElevenLabs Music	Cloud API	AI music generation
ElevenLabs SFX	Cloud API	Sound effect generation

Post-Production (always available, always free):

Tool	What It Does
FFmpeg	Video composition, encoding, subtitle burn-in, audio muxing
Video Stitch	Multi-clip assembly, crossfades, picture-in-picture, spatial layouts
Video Trimmer	Precision cutting and extraction
Audio Mixer	Multi-track mixing, ducking, fades
Audio Enhance	Noise reduction, normalization
Color Grade	LUT-based color grading
Subtitle Gen	SRT/VTT generation from timestamps

Enhancement:

Tool	What It Does
Upscale	Real-ESRGAN image/video upscaling
Background Remove	rembg / U2Net background removal
Face Enhance	Face quality enhancement
Face Restore	CodeFormer / GFPGAN face restoration

Analysis:

Tool	What It Does
Transcriber	WhisperX speech-to-text with word-level timestamps
Scene Detect	Automatic scene boundary detection
Frame Sampler	Intelligent frame extraction
Video Understand	CLIP/BLIP-2 vision-language analysis

Avatar & Lip Sync:

Tool	What It Does
Talking Head	SadTalker / MuseTalk avatar animation
Lip Sync	Wav2Lip audio-driven lip synchronization
Kling Avatar	Official Kling cloud avatar presenter generation
Kling Lip Sync	Official Kling cloud lip-sync with explicit face selection

Composition & Rendering:

Engine	Type	What It Does
Remotion	Local (Node.js)	React-based programmatic video — spring-animated image scenes, stat reveals, section titles, hero cards, TikTok-style word-by-word captions, scene transitions (fade/slide/wipe/flip), Google Fonts, audio with fade curves, and the TalkingHead avatar composition. When no video generation providers are configured, the agent generates still images and Remotion turns them into fully animated video.
HyperFrames	Local (Node.js ≥ 22)	HTML/CSS/GSAP programmatic video — kinetic typography, product promos, launch reels, custom motion graphics, registry blocks (data charts, grain overlays, shader transitions), website-to-video workflows, and rigged SVG character animation. Consumed via `npx hyperframes`; no monorepo checkout needed.
FFmpeg	Local	Core video assembly, encoding, subtitle burn, audio muxing, color grading

Runtime is chosen at proposal (render_runtime) and locked through edit_decisions. Silent swaps between runtimes are a governance violation — see skills/core/hyperframes.md.

Style System

Style playbooks define the visual language for your productions:

Playbook	Best For
Clean Professional	Corporate, educational, SaaS
Flat Motion Graphics	Social media, TikTok, startups
Minimalist Diagram	Technical deep-dives, architecture

Playbooks control typography, color palettes, motion styles, audio profiles, and quality rules. The agent reads the playbook and applies it consistently across all generated assets.

Platform Output Profiles

Built-in render profiles for every major platform:

Profile	Resolution	Aspect Ratio
YouTube Landscape	1920x1080	16:9
YouTube 4K	3840x2160	16:9
YouTube Shorts	1080x1920	9:16
Instagram Reels	1080x1920	9:16
Instagram Feed	1080x1080	1:1
TikTok	1080x1920	9:16
LinkedIn	1920x1080	16:9
Cinematic	2560x1080	21:9

Production Governance

OpenMontage treats video production like real engineering — with quality gates, audit trails, and enforcement at every stage.

Quality Gates

Human approval gates are enforced, not suggested — proposal, script, scene plan, generated assets, and publish all pause for your sign-off. The checkpoint writer rejects a "completed" gated stage without recorded approval, and every superseded checkpoint is archived so the audit trail (including gate transitions) survives revisions. Review happens visually on the Backlot board.
Pre-compose validation — blocks render if the delivery promise is violated (e.g. "motion-led" video with 80% still images), slideshow risk score is critical, or renderer family is missing. Catches broken plans before wasting GPU time.
Post-render self-review — after every render, the runtime runs ffprobe validation, extracts frames at 4 positions to check for black frames and broken overlays, analyzes audio levels for silence and clipping, verifies the delivery promise was honored, and checks subtitle presence. If the review fails, the video is not presented.
Slideshow risk scoring — 6-dimension analysis (repetition, decorative visuals, weak motion, shot intent, typography overreliance, unsupported cinematic claims) prevents "animated PowerPoint" outputs.
Source media inspection — when users supply their own footage, the system probes every file (resolution, codec, audio channels, duration) and builds planning implications before a single creative decision is made. No hallucinating content from filenames.

Scored Provider Selection

Every tool selection (video generation, image generation, TTS, music) runs through a 7-dimension scoring engine: task fit (30%), output quality (20%), control features (15%), reliability (15%), cost efficiency (10%), latency (5%), continuity (5%). The winning provider and its score are logged in the decision trail with all alternatives considered.

Selectors normalize loose brief context before scoring. If the agent only knows something like "Pixar-style animated short with character consistency," the selector expands that into scorer-friendly intent and style signals instead of requiring a perfectly pre-shaped task_context.

Selector outputs also surface the chosen provider's agent_skills, so the agent can immediately read the right Layer 3 provider skill before writing prompts.

Decision Audit Trail

Every major creative and technical choice — provider selection, style/playbook choice, music track, voice selection, renderer family, any fallback or downgrade — is logged with alternatives considered, confidence scores, and reasoning. The cumulative decision log persists across all stages so you can trace exactly why the output looks the way it does.

Budget Controls

Estimate before execution — see what it will cost
Reserve budget — lock funds before the call
Reconcile after — record actual spend
Configurable modes — observe (track only), warn (log overruns), cap (hard limit)
Per-action approval — pause for confirmation above a threshold (default: $0.50)
Total budget cap — default $10, fully configurable

No surprise bills. The agent tells you what it will cost before it spends.

Agent Compatibility

OpenMontage works with any AI coding assistant that can read files and execute Python. Dedicated instruction files are included for:

Platform	Config File
Claude Code	`CLAUDE.md`
Cursor	`CURSOR.md` + `.cursor/rules/`
GitHub Copilot	`COPILOT.md` + `.github/copilot-instructions.md`
Codex	`CODEX.md`
Windsurf	`.windsurfrules`

All platform files point to the shared AGENT_GUIDE.md (operating guide and agent contract) and PROJECT_CONTEXT.md (architecture reference).

Coming soon: Local LLM support via Ollama and LM Studio — run the full production pipeline without any cloud LLM.

Contributing

OpenMontage is built to be extended. The two most common contributions:

Adding a New Tool

Create a Python file in the appropriate tools/ subdirectory
Inherit from BaseTool and implement the tool contract
The registry auto-discovers it — no manual registration needed
Add a skill file if the tool needs usage guidance

Adding a New Pipeline

Create a YAML manifest in pipeline_defs/
Create stage director skills in skills/pipelines/<your-pipeline>/
Reference existing tools — or add new ones if needed

See docs/ARCHITECTURE.md for the full technical reference, docs/PROVIDERS.md for the complete provider guide (setup, pricing, free tiers), and AGENT_GUIDE.md for the agent contract.

Join the Community

We use GitHub Discussions to share work and ideas:

Show and Tell — Share videos you've made, prompts that worked well, or creative workflows you've discovered
Ideas — Suggest new pipelines, tools, style playbooks, or integrations
Q&A — Ask questions about setup, pipelines, or troubleshooting

Made something cool? Post it in Show and Tell — we'd love to see what you build.

Contact

For updates, releases, and behind-the-scenes build notes, follow @calesthioailabs.

For bugs, feature requests, and workflow discussions, use GitHub Issues and GitHub Discussions so everything stays visible and actionable.

Testing

# Run contract tests (no API keys needed)
make test-contracts

# Run all tests
make test

Star History

License

GNU AGPLv3

OpenMontage — Production-grade video with real quality enforcement, orchestrated by your AI assistant.

If this project looks useful to you, a ⭐ would really mean a lot — it helps others discover it too.

If you'd like to go further, sponsor the project — OpenMontage is built nights and weekends, and your support makes that sustainable.

asgeirtj/system_prompts_leaks

Extracted system prompts from Anthropic - Claude Fable 5.1, Opus 5, Claude Design, Claude Code. OpenAI - ChatGPT GPT-6-Astra, Codex. Google - Gemini 3.8 Flash, 3.1 Pro, Antigravity. xAI - Grok, Grok Bot, Cursor, Kimi and more! Updated regularly.

_{Thanks to}

Open Source Agent Analytics

Understand your agents in production

The Washington Post built its interactive story on prompts from this repo: See the hidden rules behind AI. Then use them to rewrite this article. (May 11, 2026)

CEPS' AI World built a live data dashboard from this repo's files: System prompts and what they tell us about the chat before the chat (July 10, 2026)

System Prompts Leaks

Leaked system prompts, captured verbatim — the hidden instructions and rules that ChatGPT, Claude, Gemini, Grok and every other AI chatbot receives before your first message.

Most recent additions/changes

What	Date	Link
ChatGPT Work Codex (local)	September 13, 2026	ChatGPT Work Codex local system prompt
Gemini 3.8 Flash	September 13, 2026	Gemini 3.8 Flash System Prompt
Claude Code headless (Fable 5.1)	September 5, 2026	Claude Code headless system prompt (Fable 5.1)
Codex GPT-6-Astra	September 4, 2026	Codex GPT-6-Astra system prompt
Claude Fable 5.1	September 1, 2026	Claude Fable 5.1 system prompt · Claude Code (Fable 5.1)
Grok 4.6	August 29, 2026	Grok 4.6 system prompt
Grok Bot	August 21, 2026	Grok Bot System Prompt
Gemini 3.7 Flash	August 18, 2026	Gemini 3.7 Flash System Prompt
Muse Code (Meta's coding CLI)	August 17, 2026	Muse Code system prompt
Claude Cowork (new capture + setup skills)	August 17, 2026	Claude Cowork system prompt · setup skills
Claude Science	August 14, 2026	Claude Science system prompt
Codex GPT-5.6 (Sol variant)	July 26, 2026	Codex GPT-5.6 system prompt
Grok 4.5	July 26, 2026	Grok 4.5 system prompt
Claude Opus 5	July 24, 2026	Claude Opus 5 system prompt · Claude Code (Opus 5)
Claude Design (full prompt + 53 tools + 22 skills + 10 starter components)	July 23, 2026	Claude Design system prompt · skills · starter components
Perplexity	July 17, 2026	Perplexity AI system prompt
Claude Code (new models)	July 16, 2026	Claude Code system prompt (Fable 5) · Sonnet 5
OpenCode · Pi · CommandCode	July 16, 2026	OpenCode system prompt · Pi system prompt · CommandCode CLI system prompt
Kimi K2.6	July 14, 2026	Kimi K2.6 system prompt
Perplexity Deep Research	July 14, 2026	Perplexity Deep Research system prompt
ChatGPT 5.6	July 10, 2026	ChatGPT 5.6 system prompt (Sol, extra high) · Codex GPT-5.6 system prompt

Anthropic — Claude system prompts

Claude.ai system prompts (web, desktop & mobile)

Model	Prompt
Claude Fable 5.1	Claude Fable 5.1 system prompt
Claude Fable 5	Claude Fable 5 system prompt
Claude Opus 5	Claude Opus 5 system prompt
Claude Opus 4.8	Claude Opus 4.8 system prompt
Claude Sonnet 5	Claude Sonnet 5 system prompt
Claude Opus 4.7	Claude Opus 4.7 system prompt
Claude Opus 4.6	Claude Opus 4.6 system prompt · No tools
Claude Sonnet 4.6	Claude Sonnet 4.6 system prompt · No tools
Injected reminders	Claude.ai injected reminders

Claude Code system prompts

Component	Prompt
Claude Code (Fable 5.1)	Claude Code system prompt (Fable 5.1)
Claude Code (Fable 5)	Claude Code system prompt (Fable 5)
Claude Code (Opus 5)	Claude Code system prompt (Opus 5)
Claude Code (Opus 4.8)	Claude Code system prompt (Opus 4.8)
Claude Code (Sonnet 5)	Claude Code system prompt (Sonnet 5)
Claude Code (older models)	Opus 4.7 · Opus 4.6 · Sonnet 4.6 · Haiku 4.5
Subagents	Claude Code subagent system prompts
Skills & commands	Claude Code skills · Slash commands
MCP servers	Claude in Chrome MCP server system prompt
Docs assistant	docs.claude.com assistant instructions

Claude integrations

Product	Prompt
Claude Design	Claude Design system prompt · skills · starter components
Claude Cowork	Claude Cowork system prompt · Dispatch · setup skills
Claude Science	Claude Science system prompt
Claude for Microsoft 365	Claude for Excel · Claude for Word · Claude in PowerPoint
Claude in Chrome	Claude in Chrome extension system prompt
Claude iOS app	Claude mobile iOS system prompt

OpenAI — ChatGPT system prompts

Model	Prompt
ChatGPT 5.6 Sol	ChatGPT 5.6 system prompt (Sol, extra high)
ChatGPT 5.5 Thinking	ChatGPT 5.5 Thinking system prompt
ChatGPT 5.5 Instant	ChatGPT 5.5 Instant system prompt
ChatGPT 5.4	ChatGPT 5.4 Thinking system prompt
ChatGPT 5.3	ChatGPT 5.3 Instant system prompt
ChatGPT 5.2	ChatGPT 5.2 Thinking system prompt
ChatGPT 5	ChatGPT 5 Thinking system prompt · Agent mode
ChatGPT Atlas	ChatGPT Atlas system prompt
ChatGPT 4.5	ChatGPT 4.5 system prompt
ChatGPT 4o	ChatGPT 4o system prompt · Deprecation preparedness
Voice modes	ChatGPT advanced voice mode system prompt · Legacy voice mode
Personalities	ChatGPT personality instructions
Memory	ChatGPT advanced memory system prompt

Codex system prompts

Model	Prompt
Codex GPT-5.6	Codex GPT-5.6 system prompt (Terra/Luna) · Sol
Codex GPT-5.5	Codex GPT-5.5 system prompt · Full prompt · Friendly · Pragmatic
Codex GPT-5.4	Codex GPT-5.4 system prompt · Mini
Codex Spark	Codex Spark system prompt
Codex modes	Plan mode · Auto-review · Computer use · Control Chrome · In-app browser
Personas	Friendly · Pragmatic

API-injected prompts

Model	Prompt
GPT-5.5	GPT-5.5 API system prompt · Pro
GPT-5.4 / 5.3	GPT-5.4 API · 5.3 Chat · 5.3 Codex
o-series & older	o3 / o4-mini reasoning-effort variants

Old models, tools & deprecated features


Old models	GPT-4.5 · GPT-4.1 · GPT-4.1 Mini · o3 · o4-mini · GPT-5.2 Mini (free) · ChatGPT 4o Mini
Old 4o variants	4o WhatsApp · 4o new personality · Monday GPT
Old tools	Canvas · Image gen · File search · Python · Web search
Old policies	Image safety · Image safety (2026) · Automation context
Deprecated features	GPT-5 personalities · GPT-5.1 personalities · Deep research tool · Study and learn · All
GPT-5.1 (old)	Professional

Google — Gemini system prompts

Model	Prompt
Gemini 3.5 Flash	Gemini 3.5 Flash system prompt · AI Studio
Gemini 3.1 Pro	Gemini 3.1 Pro system prompt · API
Antigravity CLI	Antigravity CLI system prompt
Nano / Banana 2	Nano Banana 2 system prompt
Google Search AI Mode	Google Search AI Mode system prompt
Gemini CLI	Gemini CLI system prompt
NotebookLM	NotebookLM chat system prompt
Jules	Jules system prompt
AI Studio Build	AI Studio Build system prompt
Gemini 3	Gemini 3 Flash system prompt · Gemini 3 Pro
Gemini YouTube	Gemini YouTube system prompt
Gemini Diffusion	Gemini Diffusion system prompt
Gemini in Chrome	Gemini in Chrome system prompt
Gemini Workspace	Gemini Workspace system prompt

Older models & variants


Gemini 2.5 Pro	API · Webapp · Guided learning
Gemini 2.5 Flash	Image preview
Gemini 2.0 Flash	Webapp

xAI — Grok system prompts

Model	Prompt
Grok 4.5	Grok 4.5 system prompt
Grok Build	Grok Build system prompt (CLI agent)
Grok 4.3 Beta	Grok 4.3 Beta system prompt
Grok 4.2	Grok 4.2 system prompt
Grok Expert	Grok Expert system prompt

Older versions


Grok 4.1 Beta	Grok 4.1 Beta system prompt
Grok 4	Grok 4 system prompt · API
Grok 3	Grok 3 system prompt
Grok Account	Grok account system prompt
Grok Personas	Grok persona prompts
Safety Instructions	Grok safety instructions

Perplexity system prompts

Model	Prompt
Perplexity	Perplexity AI system prompt
Perplexity Computer	Perplexity Computer system prompt
Deep Research	Perplexity Deep Research system prompt
Comet Browser	Comet browser assistant system prompt
Voice Assistant	Perplexity voice assistant system prompt

Microsoft — Copilot system prompts

Product	Prompt
GitHub Copilot	GitHub Copilot system prompt
VS Code Copilot Agent	VS Code Copilot agent system prompt
Copilot CLI	Copilot CLI system prompt
Copilot for macOS (app)	Copilot for macOS system prompt
Copilot in Word	Copilot in Word system prompt

Cursor system prompt

Product	Prompt
Cursor	Cursor system prompt

Meta AI system prompts

Product	Prompt
Meta AI	Meta AI Muse Spark system prompt · Muse Spark 1.1
Muse Code	Muse Code system prompt

Mistral system prompts

Product	Prompt
Mistral Medium 3.5 (Vibe)	Mistral Medium 3.5 system prompt
Mistral Code	Mistral Code system prompt

Moonshot — Kimi system prompt

Model	Prompt
Kimi K2.6	Kimi K2.6 system prompt

DeepSeek

Product	Prompt
DeepSeek	DeepSeek system prompt (chat.deepseek.com)

Z.ai — GLM

Product	Prompt
GLM	GLM serves no system prompt — verified & documented

OpenCode system prompt

Product	Prompt
OpenCode	OpenCode system prompt · May 2026 capture

Pi system prompt

Product	Prompt
Pi (Inflection)	Pi system prompt

Notion AI system prompt

Product	Prompt
Notion AI	Notion AI system prompt

Qwen system prompt

Product	Prompt
Qwen 3.6 Plus	Qwen 3.6 Plus system prompt

Misc system prompts

Product	Prompt
Amp Code (Sourcegraph)	Amp Code system prompt
CommandCode CLI	CommandCode CLI system prompt
Devin CLI	Devin CLI system prompt
Docker Gordon AI	Docker Gordon AI system prompt
ElevenLabs Voice Agent	ElevenLabs voice agent system prompt
Reddit Answers	Reddit Answers system prompt
Warp 2.0 Agent	Warp 2.0 agent system prompt
Zed AI	Zed AI system prompt

More products


Brave Search	Brave Search system prompt
Character AI	Character AI system prompt
Confer	Confer system prompt
Fellou Browser	Fellou browser system prompt
Gizmo AI	Gizmo AI system prompt
Hermes	Hermes system prompt
Indus AI	Indus AI system prompt
Kagi Assistant	Kagi assistant system prompt
MiniMax M2.5	MiniMax M2.5 system prompt
Proton Lumo AI	Proton Lumo AI system prompt
Raycast AI	Raycast AI system prompt
Sesame AI Maya	Sesame AI Maya system prompt
Stack Overflow AI Assist	Stack Overflow AI Assist system prompt
t3.chat	t3.chat system prompt
t3 Code	t3 Code system prompt

Contact

Star History

vxcontrol/pentagi

Fully autonomous AI Agents system capable of performing complex penetration testing tasks

https://pentagi.com

PentAGI

Penetration testing Artificial General Intelligence

Join the Community! Connect with security researchers, AI enthusiasts, and fellow ethical hackers. Get support, share insights, and stay updated with the latest PentAGI developments.

⠀

Overview

PentAGI is an innovative tool for automated security testing that leverages cutting-edge artificial intelligence technologies. The project is designed for information security professionals, researchers, and enthusiasts who need a powerful and flexible solution for conducting penetration tests.

You can watch the video PentAGI overview:

Features

Secure & Isolated. All operations are performed in a sandboxed Docker environment with complete isolation.
Fully Autonomous. AI-powered agent that automatically determines and executes penetration testing steps with optional execution monitoring and intelligent task planning for enhanced reliability.
Professional Pentesting Tools. Built-in suite of 20+ professional security tools including nmap, metasploit, sqlmap, and more.
Smart Memory System. Long-term storage of research results and successful approaches for future use.
Optional Knowledge Graph Integration. Graphiti-powered knowledge graph using Neo4j for semantic relationship tracking and advanced context understanding.
Web Intelligence. Built-in browser via scraper for gathering latest information from web sources.
External Search Systems. Integration with advanced search APIs including Tavily, Firecrawl, Traversaal, Perplexity, DuckDuckGo, Google Custom Search, Sploitus Search and Searxng for comprehensive information gathering.
Team of Specialists. Delegation system with specialized AI agents for research, development, and infrastructure tasks, enhanced with optional execution monitoring and intelligent task planning for optimal performance with smaller models.
Comprehensive Monitoring. Detailed logging and integration with Grafana/Prometheus for real-time system observation.
Detailed Reporting. Generation of thorough vulnerability reports with exploitation guides.
Smart Container Management. Automatic Docker image selection based on specific task requirements.
Modern Interface. Clean and intuitive web UI for system management and monitoring.
Comprehensive APIs. Full-featured REST and GraphQL APIs with Bearer token authentication for automation and integration.
Persistent Storage. All commands and outputs are stored in PostgreSQL with pgvector extension.
Scalable Architecture. Microservices-based design supporting horizontal scaling.
Self-Hosted Solution. Complete control over your deployment and data.
Flexible Authentication. Support for 10+ LLM providers (OpenAI, Anthropic, Google AI/Gemini, AWS Bedrock, Ollama, DeepSeek, GLM, Kimi, Qwen, MiniMax, Custom) plus aggregators (OpenRouter, DeepInfra, Atlas Cloud, OpenCode Go plan). For production local deployments, see our vLLM + Qwen3.5-27B-FP8 guide.
API Token Authentication. Secure Bearer token system for programmatic access to REST and GraphQL APIs.
Quick Deployment. Easy setup through Docker Compose with comprehensive environment configuration.

Current Capability Boundaries

PentAGI today is an autonomous and assistant-guided penetration testing platform, not a CALDERA-style Breach and Attack Simulation (BAS) or adversary emulation product with predefined campaigns or attack plans.
BAS-like agent-authored attack scripts should be treated as conceptual or future work, not as a feature that is implemented today.
The current flow report UI supports web view, copy to clipboard, Markdown download, and PDF download. JSON flow-report export is not documented as a supported output format today.
Provider flexibility is available today through built-in providers and custom/OpenAI-compatible endpoints. See Custom LLM Provider Configuration and the vLLM + Qwen3.5-27B-FP8 guide.

Architecture

System Context

flowchart TB
    classDef person fill:#08427B,stroke:#073B6F,color:#fff
    classDef system fill:#1168BD,stroke:#0B4884,color:#fff
    classDef external fill:#666666,stroke:#0B4884,color:#fff

    pentester["👤 Security Engineer
    (User of the system)"]

    pentagi["✨ PentAGI
    (Autonomous penetration testing system)"]

    target["🎯 target-system
    (System under test)"]
    llm["🧠 llm-provider
    (OpenAI/Anthropic/Ollama/Bedrock/Gemini/Custom)"]
    search["🔍 search-systems
    (Google/DuckDuckGo/Tavily/Firecrawl/Traversaal/Perplexity/Sploitus/Searxng)"]
    langfuse["📊 langfuse-ui
    (LLM Observability Dashboard)"]
    grafana["📈 grafana
    (System Monitoring Dashboard)"]

    pentester --> |Uses HTTPS| pentagi
    pentester --> |Monitors AI HTTPS| langfuse
    pentester --> |Monitors System HTTPS| grafana
    pentagi --> |Tests Various protocols| target
    pentagi --> |Queries HTTPS| llm
    pentagi --> |Searches HTTPS| search
    pentagi --> |Reports HTTPS| langfuse
    pentagi --> |Reports HTTPS| grafana

    class pentester person
    class pentagi system
    class target,llm,search,langfuse,grafana external

    linkStyle default stroke:#ffffff,color:#ffffff

Container Architecture (click to expand)

graph TB
    subgraph Core Services
        UI[Frontend UI<br/>React + TypeScript]
        API[Backend API<br/>Go + GraphQL]
        DB[(Vector Store<br/>PostgreSQL + pgvector)]
        MQ[Task Queue<br/>Async Processing]
        Agent[AI Agents<br/>Multi-Agent System]
    end

    subgraph Knowledge Graph
        Graphiti[Graphiti<br/>Knowledge Graph API]
        Neo4j[(Neo4j<br/>Graph Database)]
    end

    subgraph Monitoring
        Grafana[Grafana<br/>Dashboards]
        VictoriaMetrics[VictoriaMetrics<br/>Time-series DB]
        Jaeger[Jaeger<br/>Distributed Tracing]
        Loki[Loki<br/>Log Aggregation]
        OTEL[OpenTelemetry<br/>Data Collection]
    end

    subgraph Analytics
        Langfuse[Langfuse<br/>LLM Analytics]
        ClickHouse[ClickHouse<br/>Analytics DB]
        Redis[Redis<br/>Cache + Rate Limiter]
        MinIO[MinIO<br/>S3 Storage]
    end

    subgraph Security Tools
        Scraper[Web Scraper<br/>Isolated Browser]
        PenTest[Security Tools<br/>20+ Pro Tools<br/>Sandboxed Execution]
    end

    UI --> |HTTP/WS| API
    API --> |SQL| DB
    API --> |Events| MQ
    MQ --> |Tasks| Agent
    Agent --> |Commands| PenTest
    Agent --> |Queries| DB
    Agent --> |Knowledge| Graphiti
    Graphiti --> |Graph| Neo4j

    API --> |Telemetry| OTEL
    OTEL --> |Metrics| VictoriaMetrics
    OTEL --> |Traces| Jaeger
    OTEL --> |Logs| Loki

    Grafana --> |Query| VictoriaMetrics
    Grafana --> |Query| Jaeger
    Grafana --> |Query| Loki

    API --> |Analytics| Langfuse
    Langfuse --> |Store| ClickHouse
    Langfuse --> |Cache| Redis
    Langfuse --> |Files| MinIO

    classDef core fill:#f9f,stroke:#333,stroke-width:2px,color:#000
    classDef knowledge fill:#ffa,stroke:#333,stroke-width:2px,color:#000
    classDef monitoring fill:#bbf,stroke:#333,stroke-width:2px,color:#000
    classDef analytics fill:#bfb,stroke:#333,stroke-width:2px,color:#000
    classDef tools fill:#fbb,stroke:#333,stroke-width:2px,color:#000

    class UI,API,DB,MQ,Agent core
    class Graphiti,Neo4j knowledge
    class Grafana,VictoriaMetrics,Jaeger,Loki,OTEL monitoring
    class Langfuse,ClickHouse,Redis,MinIO analytics
    class Scraper,PenTest tools

Entity Relationship (click to expand)

erDiagram
    Flow ||--o{ Task : contains
    Task ||--o{ SubTask : contains
    SubTask ||--o{ Action : contains
    Action ||--o{ Artifact : produces
    Action ||--o{ Memory : stores

    Flow {
        string id PK
        string name "Flow name"
        string description "Flow description"
        string status "active/completed/failed"
        json parameters "Flow parameters"
        timestamp created_at
        timestamp updated_at
    }

    Task {
        string id PK
        string flow_id FK
        string name "Task name"
        string description "Task description"
        string status "pending/running/done/failed"
        json result "Task results"
        timestamp created_at
        timestamp updated_at
    }

    SubTask {
        string id PK
        string task_id FK
        string name "Subtask name"
        string description "Subtask description"
        string status "queued/running/completed/failed"
        string agent_type "researcher/developer/executor"
        json context "Agent context"
        timestamp created_at
        timestamp updated_at
    }

    Action {
        string id PK
        string subtask_id FK
        string type "command/search/analyze/etc"
        string status "success/failure"
        json parameters "Action parameters"
        json result "Action results"
        timestamp created_at
    }

    Artifact {
        string id PK
        string action_id FK
        string type "file/report/log"
        string path "Storage path"
        json metadata "Additional info"
        timestamp created_at
    }

    Memory {
        string id PK
        string action_id FK
        string type "observation/conclusion"
        vector embedding "Vector representation"
        text content "Memory content"
        timestamp created_at
    }

Agent Interaction (click to expand)

sequenceDiagram
    participant O as Orchestrator
    participant R as Researcher
    participant D as Developer
    participant E as Executor
    participant VS as Vector Store
    participant KB as Knowledge Base

    Note over O,KB: Flow Initialization
    O->>VS: Query similar tasks
    VS-->>O: Return experiences
    O->>KB: Load relevant knowledge
    KB-->>O: Return context

    Note over O,R: Research Phase
    O->>R: Analyze target
    R->>VS: Search similar cases
    VS-->>R: Return patterns
    R->>KB: Query vulnerabilities
    KB-->>R: Return known issues
    R->>VS: Store findings
    R-->>O: Research results

    Note over O,D: Planning Phase
    O->>D: Plan attack
    D->>VS: Query exploits
    VS-->>D: Return techniques
    D->>KB: Load tools info
    KB-->>D: Return capabilities
    D-->>O: Attack plan

    Note over O,E: Execution Phase
    O->>E: Execute plan
    E->>KB: Load tool guides
    KB-->>E: Return procedures
    E->>VS: Store results
    E-->>O: Execution status

Memory System (click to expand)

graph TB
    subgraph "Long-term Memory"
        VS[(Vector Store<br/>Embeddings DB)]
        KB[Knowledge Base<br/>Domain Expertise]
        Tools[Tools Knowledge<br/>Usage Patterns]
    end

    subgraph "Working Memory"
        Context[Current Context<br/>Task State]
        Goals[Active Goals<br/>Objectives]
        State[System State<br/>Resources]
    end

    subgraph "Episodic Memory"
        Actions[Past Actions<br/>Commands History]
        Results[Action Results<br/>Outcomes]
        Patterns[Success Patterns<br/>Best Practices]
    end

    Context --> |Query| VS
    VS --> |Retrieve| Context

    Goals --> |Consult| KB
    KB --> |Guide| Goals

    State --> |Record| Actions
    Actions --> |Learn| Patterns
    Patterns --> |Store| VS

    Tools --> |Inform| State
    Results --> |Update| Tools

    VS --> |Enhance| KB
    KB --> |Index| VS

    classDef ltm fill:#f9f,stroke:#333,stroke-width:2px,color:#000
    classDef wm fill:#bbf,stroke:#333,stroke-width:2px,color:#000
    classDef em fill:#bfb,stroke:#333,stroke-width:2px,color:#000

    class VS,KB,Tools ltm
    class Context,Goals,State wm
    class Actions,Results,Patterns em

Chain Summarization (click to expand)

The chain summarization system manages conversation context growth by selectively summarizing older messages. This is critical for preventing token limits from being exceeded while maintaining conversation coherence.

flowchart TD
    A[Input Chain] --> B{Needs Summarization?}
    B -->|No| C[Return Original Chain]
    B -->|Yes| D[Convert to ChainAST]
    D --> E[Apply Section Summarization]
    E --> F[Process Oversized Pairs]
    F --> G[Manage Last Section Size]
    G --> H[Apply QA Summarization]
    H --> I[Rebuild Chain with Summaries]
    I --> J{Is New Chain Smaller?}
    J -->|Yes| K[Return Optimized Chain]
    J -->|No| C

    classDef process fill:#bbf,stroke:#333,stroke-width:2px,color:#000
    classDef decision fill:#bfb,stroke:#333,stroke-width:2px,color:#000
    classDef output fill:#fbb,stroke:#333,stroke-width:2px,color:#000

    class A,D,E,F,G,H,I process
    class B,J decision
    class C,K output

The algorithm operates on a structured representation of conversation chains (ChainAST) that preserves message types including tool calls and their responses. All summarization operations maintain critical conversation flow while reducing context size.

Global Summarizer Configuration Options

Parameter	Environment Variable	Default	Description
Preserve Last	`SUMMARIZER_PRESERVE_LAST`	`true`	Whether to keep all messages in the last section intact
Use QA Pairs	`SUMMARIZER_USE_QA`	`true`	Whether to use QA pair summarization strategy
Summarize Human in QA	`SUMMARIZER_SUM_MSG_HUMAN_IN_QA`	`false`	Whether to summarize human messages in QA pairs
Last Section Size	`SUMMARIZER_LAST_SEC_BYTES`	`51200`	Maximum byte size for last section (50KB)
Max Body Pair Size	`SUMMARIZER_MAX_BP_BYTES`	`16384`	Maximum byte size for a single body pair (16KB)
Max QA Sections	`SUMMARIZER_MAX_QA_SECTIONS`	`10`	Maximum QA pair sections to preserve
Max QA Size	`SUMMARIZER_MAX_QA_BYTES`	`65536`	Maximum byte size for QA pair sections (64KB)
Keep QA Sections	`SUMMARIZER_KEEP_QA_SECTIONS`	`1`	Number of recent QA sections to keep without summarization

Assistant Summarizer Configuration Options

Assistant instances can use customized summarization settings to fine-tune context management behavior:

Parameter	Environment Variable	Default	Description
Preserve Last	`ASSISTANT_SUMMARIZER_PRESERVE_LAST`	`true`	Whether to preserve all messages in the assistant's last section
Last Section Size	`ASSISTANT_SUMMARIZER_LAST_SEC_BYTES`	`76800`	Maximum byte size for assistant's last section (75KB)
Max Body Pair Size	`ASSISTANT_SUMMARIZER_MAX_BP_BYTES`	`16384`	Maximum byte size for a single body pair in assistant context (16KB)
Max QA Sections	`ASSISTANT_SUMMARIZER_MAX_QA_SECTIONS`	`7`	Maximum QA sections to preserve in assistant context
Max QA Size	`ASSISTANT_SUMMARIZER_MAX_QA_BYTES`	`76800`	Maximum byte size for assistant's QA sections (75KB)
Keep QA Sections	`ASSISTANT_SUMMARIZER_KEEP_QA_SECTIONS`	`3`	Number of recent QA sections to preserve without summarization

The assistant summarizer configuration provides more memory for context retention compared to the global settings, preserving more recent conversation history while still ensuring efficient token usage.

Summarizer Environment Configuration

# Default values for global summarizer logic
SUMMARIZER_PRESERVE_LAST=true
SUMMARIZER_USE_QA=true
SUMMARIZER_SUM_MSG_HUMAN_IN_QA=false
SUMMARIZER_LAST_SEC_BYTES=51200
SUMMARIZER_MAX_BP_BYTES=16384
SUMMARIZER_MAX_QA_SECTIONS=10
SUMMARIZER_MAX_QA_BYTES=65536
SUMMARIZER_KEEP_QA_SECTIONS=1

# Default values for assistant summarizer logic
ASSISTANT_SUMMARIZER_PRESERVE_LAST=true
ASSISTANT_SUMMARIZER_LAST_SEC_BYTES=76800
ASSISTANT_SUMMARIZER_MAX_BP_BYTES=16384
ASSISTANT_SUMMARIZER_MAX_QA_SECTIONS=7
ASSISTANT_SUMMARIZER_MAX_QA_BYTES=76800
ASSISTANT_SUMMARIZER_KEEP_QA_SECTIONS=3

Advanced Agent Supervision (click to expand)

PentAGI includes sophisticated multi-layered agent supervision mechanisms to ensure efficient task execution, prevent infinite loops, and provide intelligent recovery from stuck states:

Execution Monitoring (Beta)

Automatic Mentor Intervention: Adviser agent (mentor) is automatically invoked when execution patterns indicate potential issues
Pattern Detection: Monitors identical tool calls (threshold: 5, configurable) and total tool calls (threshold: 10, configurable)
Progress Analysis: Evaluates whether agent advances toward subtask objective, detects loops and inefficiencies
Alternative Strategies: Recommends different approaches when current strategy fails
Information Retrieval Guidance: Suggests searching for established solutions instead of reinventing
Enhanced Response Format: Tool responses include both <original_result> and <mentor_analysis> sections
Configurable: Enable via EXECUTION_MONITOR_ENABLED (default: false), customize thresholds with EXECUTION_MONITOR_SAME_TOOL_LIMIT and EXECUTION_MONITOR_TOTAL_TOOL_LIMIT

Best for: Smaller models (< 32B parameters), complex attack scenarios requiring continuous guidance, preventing agents from getting stuck on single approach

Performance Impact: 2-3x increase in execution time and token usage, but delivers 2x improvement in result quality based on testing with Qwen3.5-27B-FP8

Intelligent Task Planning (Beta)

Automated Decomposition: Planner (adviser in planning mode) generates 3-7 specific, actionable steps before specialist agents begin work
Context-Aware Plans: Analyzes full execution context via enricher agent to create informed plans
Structured Assignment: Original request wrapped in <task_assignment> structure with execution plan and instructions
Scope Management: Prevents scope creep by keeping agents focused on current subtask only
Enriched Instructions: Plans highlight critical actions, potential pitfalls, and verification points
Configurable: Enable via AGENT_PLANNING_STEP_ENABLED (default: false)

Best for: Models < 32B parameters, complex penetration testing workflows, improving success rates on sophisticated tasks

Enhanced Adviser Configuration: Works exceptionally well when adviser agent uses stronger model or enhanced settings. Example: using same base model with maximum reasoning mode for adviser (see vllm-qwen3.5-27b-fp8.provider.yml) enables comprehensive task analysis and strategic planning from identical model architecture.

Performance Impact: Adds planning overhead but significantly improves completion rates and reduces redundant work

Tool Call Limits (Always Active)

Hard Limits: Prevent runaway executions regardless of supervision mode status
Differentiated by Agent Type:
- General agents (Assistant, Primary Agent, Pentester, Coder, Installer): MAX_GENERAL_AGENT_TOOL_CALLS (default: 100)
- Limited agents (Searcher, Enricher, Memorist, Generator, Reporter, Adviser, Reflector, Planner): MAX_LIMITED_AGENT_TOOL_CALLS (default: 20)
Graceful Termination: Reflector guides agents to proper completion when approaching limits
Resource Protection: Ensures system stability and prevents resource exhaustion

Reflector Integration (Always Active)

Automatic Correction: Invoked when LLM fails to generate tool calls after 3 attempts
Strategic Guidance: Analyzes failures and guides agents toward proper tool usage or barrier tools (done, ask)
Recovery Mechanism: Provides contextual guidance based on specific failure patterns
Limit Enforcement: Coordinates graceful termination when tool call limits are reached

Recommendations for Open Source Models

Must-Have for Models < 32B Parameters: Testing with Qwen3.5-27B-FP8 demonstrates that enabling both Execution Monitoring and Task Planning is essential for smaller open source models:

Quality Improvement: 2x better results compared to baseline execution without supervision
Loop Prevention: Significantly reduces infinite loops and redundant work
Attack Diversity: Encourages exploration of multiple attack vectors instead of fixating on single approach
Air-Gapped Deployments: Enables production-grade autonomous pentesting in closed network environments with local LLM inference

Trade-offs:

Token consumption: 2-3x increase due to mentor/planner invocations
Execution time: 2-3x longer due to analysis and planning steps
Result quality: 2x improvement in completeness, accuracy, and attack coverage
Model requirements: Works best when adviser uses enhanced configuration (higher reasoning parameters, stronger model variant, or different model)

Configuration Strategy: For optimal performance with smaller models, configure adviser agent with enhanced settings:

Use same model with maximum reasoning mode (example: vllm-qwen3.5-27b-fp8.provider.yml)
Or use stronger model for adviser while keeping base model for other agents
Adjust monitoring thresholds based on task complexity and model capabilities

The architecture of PentAGI is designed to be modular, scalable, and secure. Here are the key components:

Core Services
- Frontend UI: React-based web interface with TypeScript for type safety
- Backend API: Go-based REST and GraphQL APIs with Bearer token authentication for programmatic access
- Vector Store: PostgreSQL with pgvector for semantic search and memory storage
- Task Queue: Async task processing system for reliable operation
- AI Agent: Multi-agent system with specialized roles for efficient testing
Optional Knowledge Graph
- Graphiti: Knowledge graph API for semantic relationship tracking and contextual understanding
- Neo4j: Graph database for storing and querying relationships between entities, actions, and outcomes
- When enabled, automatically captures agent responses and tool executions for a flow-scoped knowledge base
Monitoring Stack
- OpenTelemetry: Unified observability data collection and correlation
- Grafana: Real-time visualization and alerting dashboards
- VictoriaMetrics: High-performance time-series metrics storage
- Jaeger: End-to-end distributed tracing for debugging
- Loki: Scalable log aggregation and analysis
Analytics Platform
- Langfuse: Advanced LLM observability and performance analytics
- ClickHouse: Column-oriented analytics data warehouse
- Redis: High-speed caching and rate limiting
- MinIO: S3-compatible object storage for artifacts
Security Tools
- Web Scraper: Isolated browser environment for safe web interaction
- Pentesting Tools: Comprehensive suite of 20+ professional security tools
- Sandboxed Execution: All operations run in isolated containers
Memory Systems
- Long-term Memory: Persistent storage of knowledge and experiences
- Working Memory: Active context and goals for current operations
- Episodic Memory: Historical actions and success patterns
- Knowledge Base: Structured domain expertise and tool capabilities
- Context Management: Intelligently manages growing LLM context windows using chain summarization

The system uses Docker containers for isolation and easy deployment, with separate networks for core services, monitoring, and analytics to ensure proper security boundaries. Each component is designed to scale horizontally and can be configured for high availability in production environments.

Quick Start

For a step-by-step walkthrough that connects installation, configuration, LLM and embedding provider testing, and your first login, see the Installing and Configuring PentAGI guide. The sections below remain the detailed reference for each step.

System Requirements

Docker and Docker Compose (or Podman - see Podman configuration)
Minimum 2 vCPU
Minimum 4GB RAM
20GB free disk space
Internet access for downloading images and updates

Using Installer (Recommended)

PentAGI provides an interactive installer with a terminal-based UI for streamlined configuration and deployment. The installer guides you through system checks, LLM provider setup, search engine configuration, and security hardening.

Supported Platforms:

Linux: amd64 download | arm64 download
Windows: amd64 download
macOS: amd64 (Intel) download | arm64 (M-series) download

macOS security warning: If macOS flags a downloaded installer, use only the official PentAGI links above, choose the archive that matches your CPU architecture, verify the source before continuing, and follow the installer troubleshooting guide before allowing the app to run.

Quick Installation (Linux amd64):

# Create installation directory
mkdir -p pentagi && cd pentagi

# Download installer
wget -O installer.zip https://pentagi.com/downloads/linux/amd64/installer-latest.zip

# Extract
unzip installer.zip

# Run interactive installer
./installer

Prerequisites & Permissions:

The installer requires appropriate privileges to interact with the Docker API for proper operation. By default, it uses the Docker socket (/var/run/docker.sock) which requires either:

Option 1 (Recommended for production): Run the installer as root:
```
sudo ./installer
```
Option 2 (Development environments): Grant your user access to the Docker socket by adding them to the docker group:
```
# Add your user to the docker group
sudo usermod -aG docker $USER

# Log out and log back in, or activate the group immediately
newgrp docker

# Verify Docker access (should run without sudo)
docker ps
```
⚠️ Security Note: Adding a user to the docker group grants root-equivalent privileges. Only do this for trusted users in controlled environments. For production deployments, consider using rootless Docker mode or running the installer with sudo.

The installer will:

System Checks: Verify Docker, network connectivity, and system requirements
Environment Setup: Create and configure .env file with optimal defaults
Provider Configuration: Set up LLM providers (OpenAI, Anthropic, Gemini, Bedrock, Ollama, DeepSeek, GLM, Kimi, Qwen, MiniMax, Custom)
Search Engines: Configure DuckDuckGo, Google, Tavily, Firecrawl, Traversaal, Perplexity, Sploitus, Searxng, and the optional internal browser-analytics fallback engine
Security Hardening: Generate secure credentials and configure SSL certificates
Deployment: Start PentAGI with docker-compose

Current Web Settings Coverage

The PentAGI web console already manages several settings areas after the server is up and running:

Settings -> Providers: Create, edit, delete, and test user-defined provider profiles for supported provider types. These profiles control per-agent model selection, runtime parameters, reasoning options, and pricing metadata.
Settings -> Prompts: Manage system, human, and tool prompt templates.
Settings -> PentAGI API: Create and manage PentAGI Bearer tokens for REST and GraphQL access.
Other UI-managed preferences: Favorite flows are stored as user preferences, and theme selection is handled from the main sidebar/profile controls rather than the Settings pages.

Still Server-Managed

The following configuration areas still need to be set on the server through environment variables, compose files, or mounted config files:

LLM credentials and connection details: API keys, endpoints, auth modes, and provider-specific connection settings for OpenAI, Anthropic, Bedrock, Ollama, custom providers, and similar backends; config-path settings apply only where supported, such as OLLAMA_SERVER_CONFIG_PATH and LLM_SERVER_CONFIG_PATH.
Search provider credentials and options: Settings such as DUCKDUCKGO_*, GOOGLE_*, TAVILY_API_KEY, FIRECRAWL_API_*, TRAVERSAAL_API_KEY, PERPLEXITY_*, SEARXNG_*, SPLOITUS_ENABLED, and the optional WEB_SEARCH_INTERNAL_* browser-analytics fallback settings.
Third-party integrations: Langfuse, Graphiti, and similar external services remain server-side configuration.
MCP server management: MCP settings pages are not currently exposed as a live web-console feature.

For Production & Enhanced Security:

For production deployments or security-sensitive environments, we strongly recommend using a distributed two-node architecture where worker operations are isolated on a separate server. This prevents untrusted code execution and network access issues on your main system.

See detailed guide: Worker Node Setup

The two-node setup provides:

Isolated Execution: Worker containers run on dedicated hardware
Network Isolation: Separate network boundaries for penetration testing
Security Boundaries: Docker-in-Docker with TLS authentication
OOB Attack Support: Dedicated port ranges for out-of-band techniques

Giving Agents Docker Without Giving Away the Host

Many pentest workflows need docker inside the agent's sandbox. There are two ways to provide it, and they differ sharply in risk.

Recommended — point sandboxes at a hardened dind daemon over TLS. Set DOCKER_INSIDE=true, leave DOCKER_SOCKET empty, and configure the daemon the sandbox may talk to:

DOCKER_INSIDE=true
DOCKER_SOCKET=                                          # mount no socket
DOCKER_INSIDE_HOST=tcp://10.0.0.5:3376                  # hardened dind endpoint
DOCKER_INSIDE_TLS_VERIFY=1
DOCKER_INSIDE_CERT_PATH=/etc/docker/dind/certs/client   # path on the worker node

PentAGI injects these into every worker container as DOCKER_HOST, DOCKER_TLS_VERIFY and DOCKER_CERT_PATH (the _INSIDE_ segment is dropped) and bind-mounts the certificate directory read-only at the same path, so docker works inside the sandbox with no further setup.

Not recommended — bind-mounting a Docker socket (DOCKER_SOCKET). This has two failure modes:

Boot-order race: a bind-mount source that does not exist yet is created by Docker as a directory. After a worker-node reboot, a worker container can start before dind has recreated its socket — Docker then puts a directory where the socket belongs, and dind cannot start until it is removed by hand.
Blast radius: the race is only reliably avoided when the mounted socket is the host daemon's, since that one always exists first. But that grants an autonomous agent the host Docker API: it can start a privileged container, mount /, and compromise the entire node — PentAGI included.

Use DOCKER_SOCKET only on single-node development setups where the host daemon is already trusted.

See: Worker Node Setup for the full dind hardening and TLS configuration, and Worker Docker Access for the exact resolution algorithm.

Running Several Instances (`TENANT_ID`)

A single PentAGI installation needs none of this — leave TENANT_ID empty (the default) and nothing changes.

Set it when several PentAGI installations share external resources: one PostgreSQL server, one worker node, one Neo4j/Graphiti, one Langfuse. The typical case is a management backend per server with a common worker node and database. Because every instance numbers its flows from 1, they would otherwise collide on container names, database rows, knowledge-graph namespaces and session cookies. TENANT_ID namespaces all of it:

Area	Effect when `TENANT_ID=acme`
PostgreSQL	The instance creates and works inside schema `acme` instead of `public`; extensions stay shared in `DATABASE_EXTENSIONS_SCHEMA` (default `public`, `extensions` on Supabase)
Worker containers	`acme-pentagi-terminal-<flow>` instead of `pentagi-terminal-<flow>`; volumes and hostnames follow, and both carry a `pentagi.tenant` label
Knowledge graph	Graphiti/Neo4j group ids become `acme-flow-<id>`
Auth	Cookie and API token keys are derived from `COOKIE_SIGNING_SALT` plus the tenant, and the session cookie is renamed
Telemetry	Langfuse traces carry the tenant as their `environment` and a `tenant:acme` tag; OTel resources gain `tenant_id`

The value must match ^[a-z][a-z0-9_]{0,31}$ — an invalid one aborts startup rather than being silently normalised.

Some things stay yours to set per instance, because they are host resources rather than names: DATA_DIR (two instances sharing it will overwrite each other's flow data), DOCKER_PORTS_BASE, the published ports, and INSTALLATION_ID. The effective values are printed at startup under Instance identity.

The installer provisions one instance per server. Running several on one server is possible — for example behind a shared nginx — but the stock docker-compose.yml uses fixed container and network names, so it has to be adapted to your own network layout first.

See: Multi-Instance Deployment for validation rules, upgrade notes and the full list of operator responsibilities.

Manual Installation

Create a working directory or clone the repository:

mkdir pentagi && cd pentagi

Copy .env.example to .env or download it:

curl -o .env https://raw.githubusercontent.com/vxcontrol/pentagi/master/.env.example

Touch examples files (example.custom.provider.yml, example.ollama.provider.yml) or download it:

curl -o example.custom.provider.yml https://raw.githubusercontent.com/vxcontrol/pentagi/master/examples/configs/custom-openai.provider.yml
curl -o example.ollama.provider.yml https://raw.githubusercontent.com/vxcontrol/pentagi/master/examples/configs/ollama-llama318b.provider.yml

Fill in the required API keys in .env file.

# Required: At least one of these LLM providers
OPEN_AI_KEY=your_openai_key
ANTHROPIC_API_KEY=your_anthropic_key
GEMINI_API_KEY=your_gemini_key

# Optional: AWS Bedrock provider (enterprise-grade models)
BEDROCK_REGION=us-east-1
# Choose one authentication method:
BEDROCK_DEFAULT_AUTH=true                        # Option 1: Use AWS SDK default credential chain (recommended for EC2/ECS)
# BEDROCK_BEARER_TOKEN=your_bearer_token         # Option 2: Bearer token authentication
# BEDROCK_ACCESS_KEY_ID=your_aws_access_key      # Option 3: Static credentials
# BEDROCK_SECRET_ACCESS_KEY=your_aws_secret_key

# Optional: Ollama provider (local or cloud)
# OLLAMA_SERVER_URL=http://ollama-server:11434   # Local server
# OLLAMA_SERVER_URL=https://ollama.com           # Cloud service
# OLLAMA_SERVER_API_KEY=your_ollama_cloud_key    # Required for cloud, empty for local

# Optional: Chinese AI providers
# DEEPSEEK_API_KEY=your_deepseek_key             # DeepSeek (strong reasoning)
# GLM_API_KEY=your_glm_key                       # GLM (Zhipu AI)
# KIMI_API_KEY=your_kimi_key                     # Kimi (Moonshot AI, ultra-long context)
# QWEN_API_KEY=your_qwen_key                     # Qwen (Alibaba Cloud, multimodal)
# MINIMAX_API_KEY=your_minimax_key               # MiniMax

# Optional: Local LLM provider (zero-cost inference)
OLLAMA_SERVER_URL=http://localhost:11434
OLLAMA_SERVER_MODEL=your_model_name

# Optional: Additional search capabilities
DUCKDUCKGO_ENABLED=true
DUCKDUCKGO_REGION=us-en
DUCKDUCKGO_SAFESEARCH=
DUCKDUCKGO_TIME_RANGE=
SPLOITUS_ENABLED=true
GOOGLE_API_KEY=your_google_key
GOOGLE_CX_KEY=your_google_cx
TAVILY_API_KEY=your_tavily_key
FIRECRAWL_API_KEY=your_firecrawl_key
FIRECRAWL_API_URL=
TRAVERSAAL_API_KEY=your_traversaal_key
PERPLEXITY_API_KEY=your_perplexity_key
PERPLEXITY_MODEL=sonar-pro
PERPLEXITY_CONTEXT_SIZE=medium

# Searxng meta search engine (aggregates results from multiple sources)
SEARXNG_URL=http://your-searxng-instance:8080
SEARXNG_CATEGORIES=general
SEARXNG_LANGUAGE=
SEARXNG_SAFESEARCH=0
SEARXNG_TIME_RANGE=
SEARXNG_TIMEOUT=

# Optional: internal browser-analytics fallback engine for web_search (off by default;
# scrapes and summarizes pages instead of calling a paid analytic API)
WEB_SEARCH_INTERNAL_ENABLED=false
WEB_SEARCH_INTERNAL_MAX_SITES=5
WEB_SEARCH_INTERNAL_MAX_SITE_BYTES=10240

## Graphiti knowledge graph settings
GRAPHITI_ENABLED=false
GRAPHITI_TIMEOUT=30
GRAPHITI_URL=

# Neo4j settings (used by Graphiti stack)
NEO4J_USER=neo4j
NEO4J_DATABASE=neo4j
NEO4J_PASSWORD=devpassword
NEO4J_URI=bolt://neo4j:7687

# Assistant configuration
ASSISTANT_USE_AGENTS=false         # Default value for agent usage when creating new assistants

Change all security related environment variables in .env file to improve security.

Security related environment variables

Main Security Settings

COOKIE_SIGNING_SALT - Salt for cookie signing, change to random value
PUBLIC_URL - Public URL of your server (eg. https://pentagi.example.com)
SERVER_SSL_CRT and SERVER_SSL_KEY - Custom paths to your existing SSL certificate and key for HTTPS (these paths should be used in the docker-compose.yml file to mount as volumes)
TENANT_ID - Leave empty unless this instance shares external resources with another PentAGI installation. When set, it is mixed into the cookie and API token signing keys and renames the session cookie, so a session minted by one instance is rejected by the others even though they share the same COOKIE_SIGNING_SALT. See Running Several Instances

Scraper Access

SCRAPER_PUBLIC_URL - Public URL for scraper if you want to use different scraper server for public URLs
SCRAPER_PRIVATE_URL - Private URL for scraper (local scraper server in docker-compose.yml file to access it to local URLs)

Access Credentials

PENTAGI_POSTGRES_USER and PENTAGI_POSTGRES_PASSWORD - PostgreSQL credentials
NEO4J_USER and NEO4J_PASSWORD - Neo4j credentials (for Graphiti knowledge graph)

Remove all inline comments from .env file if you want to use it in VSCode or other IDEs as a envFile option:

perl -i -pe 's/\s+#.*$//' .env

Run the PentAGI stack:

curl -O https://raw.githubusercontent.com/vxcontrol/pentagi/master/docker-compose.yml
docker compose up -d

Visit localhost:8443 to access PentAGI Web UI (default is admin@pentagi.com / admin)

Web UI Accounts

PentAGI does not expose public self-service sign-up from the login page. A fresh installation creates the default local administrator account:

Email: admin@pentagi.com
Password: admin

On first login, change the default password before using the instance for real work. If the administrator password is lost later, use the installer maintenance menu to reset the default admin@pentagi.com account password.

For multi-user setups, an authenticated administrator can manage local users through the Users REST API (/api/v1/users/). The OpenAPI UI is available at https://localhost:8443/api/v1/swagger/index.html after the instance is running.

Note

If you caught an error about pentagi-network or observability-network or langfuse-network you need to run docker-compose.yml firstly to create these networks and after that run docker-compose-langfuse.yml, docker-compose-graphiti.yml, and docker-compose-observability.yml to use Langfuse, Graphiti, and Observability services.

You have to set at least one Language Model provider (OpenAI, Anthropic, Gemini, AWS Bedrock, or Ollama) to use PentAGI. AWS Bedrock provides enterprise-grade access to multiple foundation models from leading AI companies, while Ollama provides zero-cost local inference if you have sufficient computational resources. Additional API keys for search engines are optional but recommended for better results.

For fully local deployment with advanced models: See our comprehensive guide on Running PentAGI with vLLM and Qwen3.5-27B-FP8 for a production-grade local LLM setup. This configuration achieves ~13,000 TPS for prompt processing and ~650 TPS for completion on 4× RTX 5090 GPUs, supporting 12+ concurrent flows with complete independence from cloud providers.

LLM_SERVER_* environment variables are experimental feature and will be changed in the future. Right now you can use them to specify custom LLM server URL and one model for all agent types.

PROXY_URL is a global proxy URL for all LLM providers and external search systems. You can use it for isolation from external networks.

The docker-compose.yml file runs the PentAGI service as root user because it needs access to docker.sock for container management. If you're using TCP/IP network connection to Docker instead of socket file, you can remove root privileges and use the default pentagi user for better security.

Accessing PentAGI from External Networks

By default, PentAGI binds to 127.0.0.1 (localhost only) for security. To access PentAGI from other machines on your network, you need to configure external access.

Configuration Steps

Update .env file with your server's IP address:

# Network binding - allow external connections
PENTAGI_LISTEN_IP=0.0.0.0
PENTAGI_LISTEN_PORT=8443

# Public URL - use your actual server IP or hostname
# Replace 192.168.1.100 with your server's IP address
PUBLIC_URL=https://192.168.1.100:8443

# CORS origins - list all URLs that will access PentAGI
# Include localhost for local access AND your server IP for external access
CORS_ORIGINS=https://localhost:8443,https://192.168.1.100:8443

Important

Replace 192.168.1.100 with your actual server's IP address
Do NOT use 0.0.0.0 in PUBLIC_URL or CORS_ORIGINS - use the actual IP address
Include both localhost and your server IP in CORS_ORIGINS for flexibility

Recreate containers to apply the changes:

docker compose down
docker compose up -d --force-recreate

Verify port binding:

docker ps | grep pentagi

You should see 0.0.0.0:8443->8443/tcp or :::8443->8443/tcp.

If you see 127.0.0.1:8443->8443/tcp, the environment variable wasn't picked up. In this case, directly edit docker-compose.yml line 31:

ports:
  - "0.0.0.0:8443:8443"

Then recreate containers again.

Configure firewall to allow incoming connections on port 8443:

# Ubuntu/Debian with UFW
sudo ufw allow 8443/tcp
sudo ufw reload

# CentOS/RHEL with firewalld
sudo firewall-cmd --permanent --add-port=8443/tcp
sudo firewall-cmd --reload

Access PentAGI:

Local access: https://localhost:8443
Network access: https://your-server-ip:8443

Note

You'll need to accept the self-signed SSL certificate warning in your browser when accessing via IP address.

Running PentAGI with Podman

PentAGI fully supports Podman as a Docker alternative. However, when using Podman in rootless mode, the scraper service requires special configuration because rootless containers cannot bind privileged ports (ports below 1024).

Podman Rootless Configuration

The default scraper configuration uses port 443 (HTTPS), which is a privileged port. For Podman rootless, reconfigure the scraper to use a non-privileged port:

1. Edit docker-compose.yml - modify the scraper service (around line 199):

scraper:
  image: vxcontrol/scraper:latest
  restart: unless-stopped
  container_name: scraper
  hostname: scraper
  expose:
    - 3000/tcp  # Changed from 443 to 3000
  ports:
    - "${SCRAPER_LISTEN_IP:-127.0.0.1}:${SCRAPER_LISTEN_PORT:-9443}:3000"  # Map to port 3000
  environment:
    - MAX_CONCURRENT_SESSIONS=${LOCAL_SCRAPER_MAX_CONCURRENT_SESSIONS:-10}
    - USERNAME=${LOCAL_SCRAPER_USERNAME:-someuser}
    - PASSWORD=${LOCAL_SCRAPER_PASSWORD:-somepass}
  logging:
    options:
      max-size: 50m
      max-file: "7"
  volumes:
    - scraper-ssl:/usr/src/app/ssl
  networks:
    - pentagi-network
  shm_size: 2g

2. Update .env file - change the scraper URL to use HTTP and port 3000:

# Scraper configuration for Podman rootless
SCRAPER_PRIVATE_URL=http://someuser:somepass@scraper:3000/
LOCAL_SCRAPER_USERNAME=someuser
LOCAL_SCRAPER_PASSWORD=somepass

Important

Key changes for Podman:

Use HTTP instead of HTTPS for SCRAPER_PRIVATE_URL
Use port 3000 instead of 443
Change internal expose to 3000/tcp
Update port mapping to target 3000 instead of 443

3. Recreate containers:

podman-compose down
podman-compose up -d --force-recreate

4. Test scraper connectivity:

# Test from within the pentagi container
podman exec -it pentagi wget -O- "http://someuser:somepass@scraper:3000/html?url=http://example.com"

If you see HTML output, the scraper is working correctly.

Podman Rootful Mode

If you're running Podman in rootful mode (with sudo), you can use the default configuration without modifications. The scraper will work on port 443 as intended.

Docker Compatibility

All Podman configurations remain fully compatible with Docker. The non-privileged port approach works identically on both container runtimes.

Assistant Configuration

PentAGI allows you to configure default behavior for assistants:

Variable	Default	Description
`ASSISTANT_USE_AGENTS`	`false`	Controls the default value for agent usage when creating new assistants

The ASSISTANT_USE_AGENTS setting affects the initial state of the "Use Agents" toggle when creating a new assistant in the UI:

false (default): New assistants are created with agent delegation disabled by default
true: New assistants are created with agent delegation enabled by default

Note that users can always override this setting by toggling the "Use Agents" button in the UI when creating or editing an assistant. This environment variable only controls the initial default state.

How to Use PentAGI After Login

Once the stack is running and you can sign in to the web UI, the fastest way to start is through the Flows workflow.

1. Create your first flow

Open Flows in the sidebar.
Click New Flow.
Choose the mode that fits your goal:
- Automation: fully autonomous execution for a testing goal you want PentAGI to carry out end-to-end
- Assistant: interactive back-and-forth help when you want to steer the investigation step by step. In this mode you can also enable the Use Agents toggle to let PentAGI delegate subtasks to specialized sub-agents for more complex investigations.
Select the LLM provider you want to use for this flow.
Describe the target and the objective in natural language in the message box.

Good first prompts usually include:

the target system or URL
the type of assessment you want
any scope limitations or rules of engagement
the result you expect, such as a vulnerability report or validation of a hypothesis

Example:

Assess https://target.example for common web application vulnerabilities. Focus on authentication, file handling, and injection issues. Stay within the provided target only and summarize confirmed findings with reproduction steps.

Only test systems you own or are explicitly authorized to assess. See EULA.md for the acceptable use requirements.

2. Use templates for repeatable workflows

The new flow form includes a template picker, which can prefill the message box with a saved flow template. This is useful when you run similar assessments repeatedly.

Use an existing template if you already have one saved in Templates
Start from the example prompt in examples/prompts/base_web_pentest.md if you need a practical baseline for web testing
Adjust the target, scope, and constraints before starting the flow

Templates are starting points. You do not need special syntax to use PentAGI: plain natural-language instructions work well as long as the target and goal are clear.

3. Monitor execution and review output

After submitting the flow, PentAGI opens the flow page automatically.

Use the main flow view to follow messages, agent activity, and task progress
Inspect tool activity and terminal output as the flow runs
Review generated tasks and subtasks to understand what PentAGI is doing

Once the flow has enough results, use the Report menu on the flow page to:

open the report in a web view
copy the generated report to the clipboard
download the report as Markdown
download the report as PDF

4. Use the Assistant view to steer an active flow

Each flow also includes an Assistant view for interactive guidance. This is useful when the autonomous run uncovers something that needs human direction instead of a hard restart.

Open the Assistant view for the same flow when you want to inspect the current state before changing anything.
Use the assistant to check flow status, stop the current task, submit follow-up instructions, or patch the remaining planned subtasks before the next step runs.
Treat this as an explicit control path for the current flow, not as an invisible background queue. If you want to change direction, say so clearly and keep the new instruction tied to the current engagement scope.
This works best for clarifying scope, redirecting priorities after intermediate findings, or answering an automation checkpoint without losing the rest of the flow context.

5. Manage flow-scoped files

Each flow has its own Files tab in the flow page. Files are scoped to the parent flow: they live in {dataDir}/flow-{id}-data/ on the host and never leak into other flows.

The tab exposes three sources of files:

Uploads (uploads/): files you provide from the web UI. Use the Upload files action, or drag and drop directly onto the Files tab. While the agent container is running, uploaded files are also pushed into it at /work/uploads/ so the agent can read them with normal shell tools.
Resources (resources/): files attached from your saved user resources library via Attach resources from library. Attached resources are copied into the flow and pushed into the running container at /work/resources/.
Container (container/): snapshots pulled from the running agent container via Pull file or directory from container. These are read-only on the flow side and are never sent back to the container.

Per-file actions in the Files tab include Download, Copy path, Save as resource (promote a flow file into your reusable resources library), and Delete. The Pull action is disabled when the container is not running, with the tooltip "Container is not running".

Uploaded files and attached resources are listed automatically in the agent's system prompts via the {{.UserFiles}} template variable, which renders a compact <task_files> XML block (with nested <uploads> and <resources> sections), so the assistant and automation agents can reference them by path without you pasting the contents into chat. Container snapshots are visible in the UI only and are not auto-injected back into the prompt.

Current limits and limitations to be aware of:

Maximum upload file size is 300 MB; per upload request up to 1000 files and 2 GB total. File names are capped at 255 bytes (roughly 255 ASCII characters; non-ASCII names use multiple bytes per character).
Uploads and resources are mirrored into the running container at the fixed paths /work/uploads/ and /work/resources/; files written to other container paths are not auto-mirrored back into the flow file model. Container snapshots can originate from any container path you pull (for example /etc/...) and are cached on the flow side under container/; they are not pushed back into the container.
Container snapshots are point-in-time pulls. Editing a snapshot in the UI does not write back into the running container.
Deleting a flow today removes the flow record and its long-term memory entries, but does not yet archive or remove the flow's flow-{id}-data/ directory on disk. Operators are still expected to clean up the data directory manually if they want to reclaim the space.

For early testing, start with a narrow target and a single clear objective. This makes the output easier to review and helps you refine your prompts before running larger assessments.

API Access

PentAGI provides comprehensive programmatic access through both REST and GraphQL APIs, allowing you to integrate penetration testing workflows into your automation pipelines, CI/CD processes, and custom applications.

Generating API Tokens

API tokens are managed through the PentAGI web interface:

Navigate to Settings → API Tokens in the web UI
Click Create Token to generate a new API token
Configure token properties:
- Name (optional): A descriptive name for the token
- Expiration Date: When the token will expire (minimum 1 minute, maximum 3 years)
Click Create and copy the token immediately - it will only be shown once for security reasons
Use the token as a Bearer token in your API requests

Each token is associated with your user account and inherits your role's permissions.

Using API Tokens

Include the API token in the Authorization header of your HTTP requests:

# GraphQL API example
curl -X POST https://your-pentagi-instance:8443/api/v1/graphql \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query": "{ flows { id title status } }"}'

# REST API example
curl https://your-pentagi-instance:8443/api/v1/flows \
  -H "Authorization: Bearer YOUR_API_TOKEN"

API Exploration and Testing

PentAGI provides interactive documentation for exploring and testing API endpoints:

GraphQL Playground

Access the GraphQL Playground at https://your-pentagi-instance:8443/api/v1/graphql/playground

Click the HTTP Headers tab at the bottom

Add your authorization header:

{
  "Authorization": "Bearer YOUR_API_TOKEN"
}

Explore the schema, run queries, and test mutations interactively

Swagger UI

Access the REST API documentation at https://your-pentagi-instance:8443/api/v1/swagger/index.html

Click the Authorize button
Enter your token in the format: Bearer YOUR_API_TOKEN
Click Authorize to apply
Test endpoints directly from the Swagger UI

Generating API Clients

You can generate type-safe API clients for your preferred programming language using the schema files included with PentAGI:

GraphQL Clients

The GraphQL schema is available at:

Web UI: Navigate to Settings to download schema.graphqls
Direct file: backend/pkg/graph/schema.graphqls in the repository

Generate clients using tools like:

GraphQL Code Generator (JavaScript/TypeScript): https://the-guild.dev/graphql/codegen
genqlient (Go): https://github.com/Khan/genqlient
Apollo iOS (Swift): https://www.apollographql.com/docs/ios

REST API Clients

The OpenAPI specification is available at:

Swagger JSON: https://your-pentagi-instance:8443/api/v1/swagger/doc.json
Swagger YAML: Available in backend/pkg/server/docs/swagger.yaml

Generate clients using:

OpenAPI Generator: https://openapi-generator.tech

openapi-generator-cli generate \
  -i https://your-pentagi-instance:8443/api/v1/swagger/doc.json \
  -g python \
  -o ./pentagi-client

Swagger Codegen: https://github.com/swagger-api/swagger-codegen

swagger-codegen generate \
  -i https://your-pentagi-instance:8443/api/v1/swagger/doc.json \
  -l typescript-axios \
  -o ./pentagi-client

swagger-typescript-api (TypeScript): https://github.com/acacode/swagger-typescript-api

npx swagger-typescript-api \
  -p https://your-pentagi-instance:8443/api/v1/swagger/doc.json \
  -o ./src/api \
  -n pentagi-api.ts

API Usage Examples

Creating a New Flow (GraphQL)

mutation CreateFlow {
  createFlow(
    modelProvider: "openai"
    input: "Test the security of https://example.com"
  ) {
    id
    title
    status
    createdAt
  }
}

Listing Flows (REST API)

curl https://your-pentagi-instance:8443/api/v1/flows \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  | jq '.flows[] | {id, title, status}'

Python Client Example

import requests

class PentAGIClient:
    def __init__(self, base_url, api_token):
        self.base_url = base_url
        self.headers = {
            "Authorization": f"Bearer {api_token}",
            "Content-Type": "application/json"
        }
    
    def create_flow(self, provider, target):
        query = """
        mutation CreateFlow($provider: String!, $input: String!) {
          createFlow(modelProvider: $provider, input: $input) {
            id
            title
            status
          }
        }
        """
        response = requests.post(
            f"{self.base_url}/api/v1/graphql",
            json={
                "query": query,
                "variables": {
                    "provider": provider,
                    "input": target
                }
            },
            headers=self.headers
        )
        return response.json()
    
    def get_flows(self):
        response = requests.get(
            f"{self.base_url}/api/v1/flows",
            headers=self.headers
        )
        return response.json()

# Usage
client = PentAGIClient(
    "https://your-pentagi-instance:8443",
    "your_api_token_here"
)

# Create a new flow
flow = client.create_flow("openai", "Scan https://example.com for vulnerabilities")
print(f"Created flow: {flow}")

# List all flows
flows = client.get_flows()
print(f"Total flows: {len(flows['flows'])}")

TypeScript Client Example

import axios, { AxiosInstance } from 'axios';

interface Flow {
  id: string;
  title: string;
  status: string;
  createdAt: string;
}

class PentAGIClient {
  private client: AxiosInstance;

  constructor(baseURL: string, apiToken: string) {
    this.client = axios.create({
      baseURL: `${baseURL}/api/v1`,
      headers: {
        'Authorization': `Bearer ${apiToken}`,
        'Content-Type': 'application/json',
      },
    });
  }

  async createFlow(provider: string, input: string): Promise<Flow> {
    const query = `
      mutation CreateFlow($provider: String!, $input: String!) {
        createFlow(modelProvider: $provider, input: $input) {
          id
          title
          status
          createdAt
        }
      }
    `;

    const response = await this.client.post('/graphql', {
      query,
      variables: { provider, input },
    });

    return response.data.data.createFlow;
  }

  async getFlows(): Promise<Flow[]> {
    const response = await this.client.get('/flows');
    return response.data.flows;
  }

  async getFlow(flowId: string): Promise<Flow> {
    const response = await this.client.get(`/flows/${flowId}`);
    return response.data;
  }
}

// Usage
const client = new PentAGIClient(
  'https://your-pentagi-instance:8443',
  'your_api_token_here'
);

// Create a new flow
const flow = await client.createFlow(
  'openai',
  'Perform penetration test on https://example.com'
);
console.log('Created flow:', flow);

// List all flows
const flows = await client.getFlows();
console.log(`Total flows: ${flows.length}`);

Security Best Practices

When working with API tokens:

Never commit tokens to version control - use environment variables or secrets management
Rotate tokens regularly - set appropriate expiration dates and create new tokens periodically
Use separate tokens for different applications - makes it easier to revoke access if needed
Monitor token usage - review API token activity in the Settings page
Revoke unused tokens - disable or delete tokens that are no longer needed
Use HTTPS only - never send API tokens over unencrypted connections

Token Management

View tokens: See all your active tokens in Settings → API Tokens
Edit tokens: Update token names or revoke tokens
Delete tokens: Permanently remove tokens (this action cannot be undone)
Token ID: Each token has a unique ID that can be copied for reference

The token list shows:

Token name (if provided)
Token ID (unique identifier)
Status (active/revoked/expired)
Creation date
Expiration date

Custom LLM Provider Configuration

When using custom LLM providers with the LLM_SERVER_* variables, you can fine-tune the reasoning format used in requests.

Tip

For production-grade local deployments, consider using vLLM with Qwen3.5-27B-FP8 for optimal performance. See our comprehensive deployment guide which includes hardware requirements, configuration templates (thinking mode and non-thinking mode), and performance benchmarks showing 13K TPS prompt processing on 4× RTX 5090 GPUs.

Variable	Default	Description
`LLM_SERVER_URL`		Base URL for the custom LLM API endpoint
`LLM_SERVER_KEY`		API key for the custom LLM provider
`LLM_SERVER_MODEL`		Default model to use (can be overridden in provider config)
`LLM_SERVER_CONFIG_PATH`		Path to the YAML configuration file for agent-specific models
`LLM_SERVER_PROVIDER`		Provider name prefix for model names (e.g., `openrouter`, `deepseek` for LiteLLM proxy)
`LLM_SERVER_LEGACY_REASONING`	`false`	Controls reasoning format in API requests
`LLM_SERVER_PRESERVE_REASONING`	`false`	Preserve reasoning content in multi-turn conversations (required by some providers)

The LLM_SERVER_PROVIDER setting is particularly useful when using LiteLLM proxy, which adds a provider prefix to model names. For example, when connecting to Moonshot API through LiteLLM, models like kimi-2.5 become moonshot/kimi-2.5. By setting LLM_SERVER_PROVIDER=moonshot, you can use the same provider configuration file for both direct API access and LiteLLM proxy access without modifications.

The LLM_SERVER_LEGACY_REASONING setting affects how reasoning parameters are sent to the LLM:

false (default): Uses modern format where reasoning is sent as a structured object with max_tokens parameter
true: Uses legacy format with string-based reasoning_effort parameter

This setting is important when working with different LLM providers as they may expect different reasoning formats in their API requests. If you encounter reasoning-related errors with custom providers, try changing this setting.

The LLM_SERVER_PRESERVE_REASONING setting controls whether reasoning content is preserved in multi-turn conversations:

false (default): Reasoning content is not preserved in conversation history
true: Reasoning content is preserved and sent in subsequent API calls

This setting is required by some LLM providers (e.g., Moonshot) that return errors like "thinking is enabled but reasoning_content is missing in assistant tool call message" when reasoning content is not included in multi-turn conversations. Enable this setting if your provider requires reasoning content to be preserved.

Troubleshooting: tool-call (function-call) parser errors

PentAGI drives its agents with tool calls (also called function calls), so any custom OpenAI-compatible backend configured through LLM_SERVER_* must return valid tool-call JSON in the format the OpenAI Chat Completions API defines. When the backend emits malformed, truncated, or non-conforming tool-call arguments, the agent chain cannot continue.

Self-hosted engines such as llama.cpp, SGLang, and vLLM usually require a specific tool-call parser and a matching chat template to produce correct tool-call output. If the parser is missing or mismatched for the model you are serving, tool-call arguments can come back corrupted. Compatibility therefore depends on the backend's tool-call/function-call behavior and configuration, not on PentAGI alone; not every llama.cpp or SGLang setup produces valid tool calls out of the box.

Typical symptoms:

Backend or proxy errors such as Failed to parse tool call arguments as JSON (often surfaced through a LiteLLM proxy as an HTTP 500), or other unexpected 5xx/4xx responses from the LLM endpoint.
A flow that runs for a few steps and then stops responding to new input in the UI.
Repeated or looping tool calls that never converge.
A flow that fails right at the start with failed to select primary docker image via llm call, because the first action in a flow is an LLM tool call to choose the container image; a backend that cannot return a valid tool call fails at this step too.

How to investigate:

Check both sides of the connection: the PentAGI logs (docker compose logs -f pentagi) and the inference backend or proxy logs (llama.cpp, SGLang, vLLM, or LiteLLM). The backend log usually shows the same parse error when it produced the malformed tool call.
Validate the provider before running a full flow with the ctester utility, which exercises tool-calling agent types directly. See Testing LLM Agents.
Confirm the backend's tool-call parser and chat template are the ones recommended for the model you are serving, and that the model itself supports tool calling.
Update PentAGI to the latest build. Recent versions sanitize malformed function-call arguments returned by the model so a single bad response no longer stalls the whole flow; older builds forwarded the corrupted arguments and could get stuck.

Ollama Provider Configuration

PentAGI supports Ollama for both local LLM inference (zero-cost, enhanced privacy) and Ollama Cloud (managed service with free tier).

Configuration Variables

Variable	Default	Description
`OLLAMA_SERVER_URL`		URL of your Ollama server or Ollama Cloud
`OLLAMA_SERVER_API_KEY`		API key for Ollama Cloud authentication
`OLLAMA_SERVER_MODEL`		Default model for inference
`OLLAMA_SERVER_CONFIG_PATH`		Path to custom agent configuration file
`OLLAMA_SERVER_PULL_MODELS_TIMEOUT`	`600`	Timeout for model downloads (seconds)
`OLLAMA_SERVER_PULL_MODELS_ENABLED`	`false`	Auto-download models on startup
`OLLAMA_SERVER_LOAD_MODELS_ENABLED`	`false`	Query server for available models

Ollama Cloud Configuration

Ollama Cloud provides managed inference with a generous free tier and scalable paid plans.

Free Tier Setup (Single Model)

# Free tier allows one model at a time
OLLAMA_SERVER_URL=https://ollama.com
OLLAMA_SERVER_API_KEY=your_ollama_cloud_api_key
OLLAMA_SERVER_MODEL=gpt-oss:120b  # Example: OpenAI OSS 120B model

Paid Tier Setup (Multi-Model with Pre-built Configuration)

For paid tiers supporting multiple concurrent models, use the pre-built Ollama Cloud configuration:

# Using pre-built Ollama Cloud configuration (included in Docker image)
OLLAMA_SERVER_URL=https://ollama.com
OLLAMA_SERVER_API_KEY=your_ollama_cloud_api_key
OLLAMA_SERVER_CONFIG_PATH=/opt/pentagi/conf/ollama-cloud.provider.yml

The pre-built ollama-cloud.provider.yml configuration includes optimized model assignments for all agent types:

Simple/Assistant: nemotron-3-super:cloud - Fast general-purpose model
Primary Agent: qwen3-coder-next:cloud - Advanced reasoning with high effort mode
Coder/Pentester: qwen3-coder-next:cloud - Specialized coding models
Searcher: qwen3.5:397b-cloud - Large context for information gathering
Refiner/Refactor: glm-5:cloud - High-quality text refinement
Adviser/Enricher: minimax-m2.7:cloud - Efficient advisory tasks
Installer: devstral-2:123b-cloud - Installation and setup tasks

Custom Configuration (Advanced)

To create your own agent configuration, mount a custom file from your host filesystem:

# Using custom provider configuration
OLLAMA_SERVER_URL=https://ollama.com
OLLAMA_SERVER_API_KEY=your_ollama_cloud_api_key
OLLAMA_SERVER_CONFIG_PATH=/opt/pentagi/conf/ollama.provider.yml

# Mount custom configuration from host filesystem (in .env or docker-compose override)
PENTAGI_OLLAMA_SERVER_CONFIG_PATH=/path/on/host/my-ollama-config.yml

The PENTAGI_OLLAMA_SERVER_CONFIG_PATH environment variable maps your host configuration file to /opt/pentagi/conf/ollama.provider.yml inside the container.

Example custom configuration (my-ollama-config.yml):

primary_agent:
  model: "qwen3-coder-next:cloud"
  temperature: 1.0
  top_p: 0.9
  max_tokens: 32768
  reasoning:
    effort: high

coder:
  model: "qwen3-coder:32b"
  temperature: 1.0
  max_tokens: 20480

Local Ollama Configuration

For self-hosted Ollama instances:

# Basic local Ollama setup
OLLAMA_SERVER_URL=http://localhost:11434
OLLAMA_SERVER_MODEL=llama3.1:8b-instruct-q8_0

# Production setup with auto-pull and model discovery
OLLAMA_SERVER_URL=http://ollama-server:11434
OLLAMA_SERVER_PULL_MODELS_ENABLED=true
OLLAMA_SERVER_PULL_MODELS_TIMEOUT=900
OLLAMA_SERVER_LOAD_MODELS_ENABLED=true

# Using pre-built configurations from Docker image
OLLAMA_SERVER_CONFIG_PATH=/opt/pentagi/conf/ollama-llama318b.provider.yml
# or
OLLAMA_SERVER_CONFIG_PATH=/opt/pentagi/conf/ollama-qwen332b-fp16-tc.provider.yml
# or
OLLAMA_SERVER_CONFIG_PATH=/opt/pentagi/conf/ollama-qwq32b-fp16-tc.provider.yml

Performance Considerations:

Model Discovery (OLLAMA_SERVER_LOAD_MODELS_ENABLED=true): Adds 1-2s startup latency querying Ollama API
Auto-pull (OLLAMA_SERVER_PULL_MODELS_ENABLED=true): First startup may take several minutes downloading models
Pull timeout (OLLAMA_SERVER_PULL_MODELS_TIMEOUT=900): 15 minutes in seconds
Static Config: Disable both flags and specify models in config file for fastest startup

Creating Custom Ollama Models with Extended Context

PentAGI requires models with larger context windows than the default Ollama configurations. You need to create custom models with increased num_ctx parameter through Modelfiles. While typical agent workflows consume around 64K tokens, PentAGI uses 110K context size for safety margin and handling complex penetration testing scenarios.

Important: The num_ctx parameter can only be set during model creation via Modelfile - it cannot be changed after model creation or overridden at runtime.

Example: Qwen3 32B FP16 with Extended Context

Create a Modelfile named Modelfile_qwen3_32b_fp16_tc:

FROM qwen3:32b-fp16
PARAMETER num_ctx 110000
PARAMETER temperature 0.3
PARAMETER top_p 0.8
PARAMETER min_p 0.0
PARAMETER top_k 20
PARAMETER repeat_penalty 1.1

Build the custom model:

ollama create qwen3:32b-fp16-tc -f Modelfile_qwen3_32b_fp16_tc

Example: QwQ 32B FP16 with Extended Context

Create a Modelfile named Modelfile_qwq_32b_fp16_tc:

FROM qwq:32b-fp16
PARAMETER num_ctx 110000
PARAMETER temperature 0.2
PARAMETER top_p 0.7
PARAMETER min_p 0.0
PARAMETER top_k 40
PARAMETER repeat_penalty 1.2

Build the custom model:

ollama create qwq:32b-fp16-tc -f Modelfile_qwq_32b_fp16_tc

Note: The QwQ 32B FP16 model requires approximately 71.3 GB VRAM for inference. Ensure your system has sufficient GPU memory before attempting to use this model.

These custom models are referenced in the pre-built provider configuration files (ollama-qwen332b-fp16-tc.provider.yml and ollama-qwq32b-fp16-tc.provider.yml) that are included in the Docker image at /opt/pentagi/conf/.

OpenAI Provider Configuration

PentAGI integrates with OpenAI's comprehensive model lineup, featuring advanced reasoning capabilities with extended chain-of-thought, agentic models with enhanced tool integration, and specialized code models for security engineering.

Configuration Variables

Variable	Default	Description
`OPEN_AI_KEY`		API key for OpenAI services
`OPEN_AI_SERVER_URL`	`https://api.openai.com/v1`	OpenAI API endpoint

Configuration Examples

# Basic OpenAI setup
OPEN_AI_KEY=your_openai_api_key
OPEN_AI_SERVER_URL=https://api.openai.com/v1

# Using with proxy for enhanced security
OPEN_AI_KEY=your_openai_api_key
PROXY_URL=http://your-proxy:8080

Supported Models

PentAGI supports 32 OpenAI models with tool calling, streaming, reasoning modes, and prompt caching. Models marked with * are used in default configuration. Models marked ⚠️ are deprecated by OpenAI and kept only for backward compatibility with agent configs already pinned to those names — avoid them for new assignments.

GPT-5.6 Series - Latest Frontier (Feb 2026 knowledge cutoff, 1.05M context, 128K max output)

Model ID	Thinking	Reasoning Effort	Price (Input/Output/Cache)	Use Case
`gpt-5.6-sol`	✅	low/medium/high/xhigh	$5.00/$30.00/$0.50	Frontier model for complex professional work, most demanding autonomous pentesting, sophisticated exploit chain development, deep multi-stage attack simulation
`gpt-5.6-terra`*	✅	low/medium/high/xhigh	$2.50/$15.00/$0.25	Balances intelligence and cost; multi-phase security assessments, coordinated multi-tool pentesting (generator/refiner/adviser/coder default)
`gpt-5.6-luna`	✅	low/medium/high/xhigh	$1.00/$6.00/$0.10	Optimized for cost-sensitive, high-volume workloads; rapid reconnaissance, bulk vulnerability scanning, real-time monitoring

GPT-5.5 Series - Frontier (Dec 2025 knowledge cutoff, 1.05M context, 128K max output)

Model ID	Thinking	Reasoning Effort	Price (Input/Output/Cache)	Use Case
`gpt-5.5`	✅	none/low/medium/high/xhigh	$5.00/$30.00/$0.50	New class of intelligence for coding and professional work; complex security research, advanced autonomous pentesting
`gpt-5.5-pro`	✅	medium/high/xhigh	$30.00/$180.00/$0.00	Uses more compute for smarter, more precise responses; no cached-input discount; mission-critical security research, zero-day discovery

GPT-5.4 Series - Advanced Reasoning at Scale (1M context)

Model ID	Thinking	Reasoning Effort	Price (Input/Output/Cache)	Use Case
`gpt-5.4`	✅	low/medium/high/xhigh	$2.50/$15.00/$0.25	Best intelligence at scale for agentic, coding, and professional workflows; maximum cognitive depth for pentesting
`gpt-5.4-mini`*	✅	low/medium/high/xhigh	$0.75/$4.50/$0.075	Strongest mini model for coding, computer use, subagents (primary_agent/assistant/reflector/installer/pentester default)
`gpt-5.4-nano`*	✅	low/medium/high/xhigh	$0.20/$1.25/$0.02	Cheapest GPT-5.4-class model for simple, high-volume tasks (simple/simple_json/searcher/enricher default)

GPT-5.2 Series - Previous Flagship Agentic

Model ID	Thinking	Reasoning Effort	Price (Input/Output/Cache)	Use Case
`gpt-5.2`	✅	low/medium/high/xhigh	$1.75/$14.00/$0.175	Superseded by 5.4/5.6; autonomous security research, complex exploit chain development
`gpt-5.2-pro`	✅	medium/high/xhigh	$21.00/$168.00/$0.00	Superior agentic coding and long-context performance, mission-critical security research, zero-day discovery

GPT-5/5.1 Series - Advanced Agentic Models

Model ID	Thinking	Price (Input/Output/Cache)	Use Case
`gpt-5`	✅	$1.25/$10.00/$0.125	Autonomous security research, exploit chain development, coordinating multi-tool pentesting workflows
`gpt-5.1`	✅	$1.25/$10.00/$0.125	Bridges GPT-5 and GPT-5.2 with faster responses; balanced penetration testing with strong tool coordination
`gpt-5-pro`	✅ (high)	$15.00/$120.00/$0.00	Reduced hallucinations, exceptional accuracy, critical security operations
`gpt-5-mini`	✅	$0.25/$2.00/$0.025	Automated vulnerability analysis, exploit generation with strong function calling
`gpt-5-nano`	✅	$0.05/$0.40/$0.005	High-throughput security scanning, reconnaissance, real-time monitoring

GPT-4.1 Series - Enhanced Intelligence (Non-Reasoning)

Model ID	Thinking	Price (Input/Output/Cache)	Use Case
`gpt-4.1`	❌	$2.00/$8.00/$0.50	Superior function calling, complex threat analysis, sophisticated exploit development
`gpt-4.1-mini`	❌	$0.40/$1.60/$0.10	Routine security assessments, automated code analysis (no longer used in default configuration)

GPT-4o Series - Multimodal (Non-Reasoning)

Model ID	Thinking	Price (Input/Output/Cache)	Use Case
`gpt-4o-mini`	❌	$0.15/$0.60/$0.075	Compact multimodal with strong function calling, high-frequency scanning, cost-effective bulk operations

o-Series - Advanced Reasoning Models (Current)

Model ID	Thinking	Price (Input/Output/Cache)	Use Case
`o3`	✅	$2.00/$8.00/$0.50	Succeeded by GPT-5; multi-stage attack chains, deep vulnerability analysis
`o3-pro`	✅	$20.00/$80.00/$0.00	More compute for better responses; zero-day research, critical security investigations

Deprecated Models - Kept for Backward Compatibility ⚠️

These models were marked deprecated by OpenAI. PentAGI keeps them defined only so that pre-existing agent configs pinned to these names keep working; do not assign them to new agents.

Model ID	Thinking	Price (Input/Output/Cache)	Notes
`gpt-5.2-codex`	✅	$1.75/$14.00/$0.175	Superseded code-specialized model; use `gpt-5.6-terra`/`gpt-5.4-mini` instead
`gpt-5.1-codex-max`	✅	$1.25/$10.00/$0.125	Superseded; enhanced reasoning for coding workflows
`gpt-5.1-codex`	✅	$1.25/$10.00/$0.125	Superseded standard code-optimized model
`gpt-5-codex`	✅	$1.25/$10.00/$0.125	Superseded foundational code-specialized model
`gpt-5.1-codex-mini`	✅	$0.25/$2.00/$0.025	Superseded compact code model
`codex-mini-latest`	✅	$1.50/$6.00/$0.375	Superseded compact code model
`gpt-4o`	❌	$2.50/$10.00/$1.25	Superseded by GPT-5.x/5.6 series multimodal flagship
`gpt-4.1-nano`	❌	$0.10/$0.40/$0.025	Superseded ultra-fast lightweight model
`o3-mini`	✅	$1.10/$4.40/$0.55	Superseded compact reasoning model
`o4-mini`	✅	$1.10/$4.40/$0.275	Succeeded by `gpt-5-mini`
`o1`	✅	$15.00/$60.00/$7.50	Superseded premier reasoning model
`o1-pro`	✅	$150.00/$600.00/$0.00	Superseded, highest cost point of the o-series

Prices: Per 1M tokens. Reasoning models include thinking tokens in output pricing.

Warning

GPT-5/5.1/5.2 Models - Trusted Access Required

The original GPT-5, GPT-5.1, and GPT-5.2 models (gpt-5, gpt-5.1, gpt-5.2, gpt-5-pro, gpt-5.2-pro, and all deprecated Codex variants) work unstably with PentAGI and may trigger OpenAI's cybersecurity safety mechanisms without verified access. This does not affect the newer GPT-5.4/5.5/5.6 series used in PentAGI's default configuration below.

To use these models reliably:

Individual users: Verify your identity at chatgpt.com/cyber
Enterprise teams: Request trusted access through your OpenAI representative
Security researchers: Apply for the Cybersecurity Grant Program (includes $10M in API credits)

Recommended alternatives without verification:

Use PentAGI's defaults — gpt-5.4-mini/gpt-5.4-nano/gpt-5.6-terra — which work out of the box
Use o3/o3-pro for reasoning tasks
Use gpt-4.1 series for general intelligence and function calling without reasoning

Reasoning Configuration:

Reasoning forced off by default: every default agent assigned gpt-5.4-mini or gpt-5.6-terra (primary_agent, assistant, generator, refiner, adviser, reflector, coder, installer, pentester) sets reasoning: {mode: off} — this genuinely disables reasoning, it is not simply "low effort". PentAGI calls OpenAI exclusively through /v1/chat/completions (never /v1/responses), and this endpoint rejects requests that combine function tools with these models' default-on thinking; forcing thinking off is required for tool calls to work reliably (see the investigation notes in backend/pkg/providers/openai/config.yml).
No override needed for gpt-5.4-nano: used for simple, simple_json, searcher, and enricher, this tier does not default to thinking on, so tools attach without conflict and no reasoning override is required.
Manual tuning available: outside the default assignments, GPT-5.6/5.5/5.4/5.2 series models expose explicit reasoning effort levels (low/medium/high/xhigh, plus none on GPT-5.5) for custom agent configs that need variable reasoning depth with tool calling disabled or via /v1/responses.

Key Features:

Extended Reasoning: GPT-5.4/5.5/5.6 and o-series models with chain-of-thought for complex security analysis
Agentic Intelligence: GPT-5.4/5.5/5.6 series with enhanced tool integration, million-token context windows, and autonomous capabilities
Prompt Caching: Cost reduction on repeated context (10-50% of input price)
Code Specialization: Legacy Codex models remain available (deprecated) for vulnerability discovery and exploit development in pinned configs
Multimodal Support: gpt-4o-mini for vision-based security assessments
Tool Calling: Robust function calling across all models for pentesting tool orchestration
Streaming: Real-time response streaming for interactive workflows
Proven Track Record: Industry-leading models with CVE discoveries and real-world security applications

Anthropic Provider Configuration

PentAGI integrates with Anthropic's Claude models, featuring advanced extended thinking capabilities, exceptional safety mechanisms, and sophisticated understanding of complex security contexts with prompt caching.

Configuration Variables

Variable	Default	Description
`ANTHROPIC_API_KEY`		API key for Anthropic services
`ANTHROPIC_SERVER_URL`	`https://api.anthropic.com/v1`	Anthropic API endpoint

Configuration Examples

# Basic Anthropic setup
ANTHROPIC_API_KEY=your_anthropic_api_key
ANTHROPIC_SERVER_URL=https://api.anthropic.com/v1

# Using with proxy for secure environments
ANTHROPIC_API_KEY=your_anthropic_api_key
PROXY_URL=http://your-proxy:8080

Note

Google Vertex AI for Claude models

PentAGI does not currently expose a dedicated Google Vertex AI configuration path for Anthropic Claude in .env. There is no separate Vertex AI API key field at this time, and the existing Anthropic variables (ANTHROPIC_API_KEY, ANTHROPIC_SERVER_URL) target the direct Anthropic API. Supported routes for Claude are:

Direct Anthropic API: ANTHROPIC_API_KEY and ANTHROPIC_SERVER_URL (see above).
AWS Bedrock: BEDROCK_* variables (see AWS Bedrock Provider Configuration).

If you need to use Vertex AI today, the safest supported workaround is to expose Vertex AI through an OpenAI-compatible proxy or gateway that translates Vertex AI calls into the Chat Completions format while preserving the chat and tool-call behavior PentAGI relies on, then point the Custom LLM provider at that gateway via LLM_SERVER_URL, LLM_SERVER_KEY, and LLM_SERVER_MODEL. This path is only as reliable as the gateway you choose.

Supported Models

PentAGI supports 9 Claude models with tool calling, streaming, extended thinking, adaptive thinking, and prompt caching. Models marked with * are used in default configuration.

Claude 5 Series - Newest Models (2026)

Model ID	Thinking	Release Date	Price (Input/Output/Cache R/W)	Use Case
`claude-sonnet-5`*	✅	Jul 2026	$3.00/$15.00/$0.30/$3.75	Best combination of speed and intelligence for coding, agents, and professional work at scale. Adaptive thinking only (manual budget thinking rejected); sampling parameters not supported. Default model for primary agent, assistant, coder, adviser, installer, pentester
`claude-fable-5`	✅	Jun 2026	$10.00/$50.00/$1.00/$12.50	Anthropic's most capable widely released model for long-running agents and the most demanding reasoning workloads. Adaptive thinking always on (budget thinking and an explicit disable are rejected); sampling parameters not supported

Claude 4 Series

Model ID	Thinking	Release Date	Price (Input/Output/Cache R/W)	Use Case
`claude-opus-4-8`*	✅	May 2026	$5.00/$25.00/$0.50/$6.25	Flagship for coding, agents, and deep reasoning in enterprise security workflows. Adaptive thinking only — budget thinking and sampling params (temperature/top_p/top_k) are rejected. Default model for generator and refiner; most demanding exploit development and multi-stage attack simulation
`claude-opus-4-7`	✅	Apr 2026	$5.00/$25.00/$0.50/$6.25	Advanced software engineering and long-running agentic security analysis. Adaptive thinking only (manual budget thinking rejected)
`claude-sonnet-4-6`	✅	Feb 2026	$3.00/$15.00/$0.30/$3.75	Best speed/intelligence balance with adaptive thinking. Multi-phase security assessments, intelligent vulnerability analysis, real-time threat hunting
`claude-opus-4-6`	✅	Feb 2026	$5.00/$25.00/$0.50/$6.25	Most intelligent model for autonomous agents and coding. Extended + adaptive thinking for complex exploit development, multi-stage attack simulation
`claude-haiku-4-5`*	❌	Oct 2025	$1.00/$5.00/$0.10/$1.25	Fast and efficient model with exceptional function calling and low latency, no thinking support. Default model for simple, simple_json, reflector, searcher, enricher; high-frequency scanning, real-time monitoring, bulk automated testing

Legacy Models - Still Supported

Model ID	Thinking	Release Date	Price (Input/Output/Cache R/W)	Use Case
`claude-sonnet-4-5`	✅	Sep 2025	$3.00/$15.00/$0.30/$3.75	State-of-the-art reasoning (superseded by sonnet-4-6/sonnet-5). Sophisticated penetration testing, advanced threat analysis
`claude-opus-4-5`	✅	Nov 2025	$5.00/$25.00/$0.50/$6.25	Ultimate reasoning (superseded by opus-4-6/4-7/4-8). Critical security research, zero-day discovery, red team operations

Prices: Per 1M tokens. Cache pricing includes both Read and Write costs.

Extended Thinking Configuration (default agent config, see backend/pkg/providers/anthropic/config.yml):

Generator / Refiner (claude-opus-4-8): adaptive reasoning at xhigh/high effort for maximum reasoning depth on complex exploit development
Primary agent, assistant, coder, adviser, installer, pentester (claude-sonnet-5): adaptive reasoning (adviser at xhigh effort) for balanced code analysis and vulnerability research
Reflector, searcher (claude-haiku-4-5): fixed reasoning budget of 1024 tokens for focused reasoning on specific tasks
Simple, simple_json, enricher (claude-haiku-4-5): no thinking, optimized for speed

Key Features:

Extended Thinking: All Claude 4.5+ models support configurable chain-of-thought reasoning depths for complex security analysis
Adaptive Thinking: Claude 4.6 series (Opus/Sonnet) dynamically adjusts reasoning depth based on task complexity; Claude Opus 4.7/4.8 and the Claude 5 series (Sonnet/Fable) are adaptive-thinking-only (manual budget thinking and sampling parameters are rejected with HTTP 400)
Prompt Caching: Significant cost reduction with separate read/write pricing (10% read, 125% write of input)
Extended Context Window: 200K tokens standard, up to 1M tokens (beta) for Claude Opus/Sonnet 4.6 for comprehensive codebase analysis
Tool Calling: Robust function calling with exceptional accuracy for security tool orchestration
Streaming: Real-time response streaming for interactive penetration testing workflows
Safety-First Design: Built-in safety mechanisms ensuring responsible security testing practices
Multimodal Support: Vision capabilities in latest models for screenshot analysis and UI security assessment
Constitutional AI: Advanced safety training providing reliable and ethical security guidance

Google AI (Gemini) Provider Configuration

PentAGI integrates with Google's Gemini models through the Google AI API, offering state-of-the-art multimodal reasoning capabilities with extended thinking and context caching.

Configuration Variables

Variable	Default	Description
`GEMINI_API_KEY`		API key for Google AI services
`GEMINI_SERVER_URL`	`https://generativelanguage.googleapis.com`	Google AI API endpoint

Configuration Examples

# Basic Gemini setup
GEMINI_API_KEY=your_gemini_api_key
GEMINI_SERVER_URL=https://generativelanguage.googleapis.com

# Using with proxy
GEMINI_API_KEY=your_gemini_api_key
PROXY_URL=http://your-proxy:8080

Supported Models

PentAGI supports 9 Gemini models with tool calling, streaming, thinking modes, and context caching. Models marked with * are used in default configuration.

Gemini 3.5 Series - Latest Stable Flash (May 2026)

Model ID	Thinking	Context	Price (Input/Output/Cache)	Use Case
`gemini-3.5-flash`*	✅	1M	$1.50/$9.00/$0.15	Most intelligent Flash model with sustained frontier performance on agentic and coding tasks, superior search and grounding

Gemini 3.1 Series - Stable Flash-Lite + Pro Preview (Feb-May 2026)

Model ID	Thinking	Context	Price (Input/Output/Cache)	Use Case
`gemini-3.1-pro-preview`*	✅	1M	$2.00/$12.00/$0.20	Latest flagship with refined thinking, improved token efficiency, optimized for software engineering and agentic workflows
`gemini-3.1-pro-preview-customtools`	✅	1M	$2.00/$12.00/$0.20	Custom tools endpoint optimized for bash and custom tools (view_file, search_code) prioritization
`gemini-3.1-flash-lite`*	✅	1M	$0.25/$1.50/$0.025	Most cost-efficient stable multimodal model, frontier-class performance for high-volume agentic tasks and low-latency applications

Gemini 2.5 Series - Advanced Thinking Models (active until October 16, 2026)

Model ID	Thinking	Context	Price (Input/Output/Cache)	Use Case
`gemini-2.5-pro`	✅	1M	$1.25/$10.00/$0.125	State-of-the-art for complex coding and reasoning, sophisticated threat modeling
`gemini-2.5-flash`	✅	1M	$0.30/$2.50/$0.03	First hybrid reasoning model with thinking budgets, best price-performance for large-scale assessments
`gemini-2.5-flash-lite`	✅	1M	$0.10/$0.40/$0.01	Smallest and most cost-effective for at-scale usage, high-throughput scanning

Gemma 4 Open-Source Models (Apache 2.0, Free Tier)

Model ID	Thinking	Context	Price (Input/Output/Cache)	Use Case
`gemma-4-31b-it`	✅	256K	Free/Free/Free	Largest open-source Gemma 4 dense model (~31B params), multimodal text+image, 140+ languages, on-premises security operations
`gemma-4-26b-a4b-it`	✅	256K	Free/Free/Free	MoE architecture (~26B total / ~3.8B active params), highly efficient inference on consumer GPUs for on-premises high-throughput scanning

Prices: Per 1M tokens (Standard Paid tier). Context window is input token limit.

Note

Gemini 2.5 Series Shutdown

gemini-2.5-pro, gemini-2.5-flash, and gemini-2.5-flash-lite will be shut down on October 16, 2026. Recommended migrations:

gemini-2.5-pro → gemini-3.1-pro-preview (same $2.00 input pricing tier)
gemini-2.5-flash → gemini-3.5-flash (improved frontier capabilities)
gemini-2.5-flash-lite → gemini-3.1-flash-lite (same $0.25 input pricing)

Default Model Assignments (config.yml):

gemini-3.1-pro-preview - primary_agent, assistant, generator, refiner, adviser, coder, pentester
gemini-3.5-flash - reflector, searcher, enricher, installer
gemini-3.1-flash-lite - simple, simple_json

Key Features:

Extended Thinking: Step-by-step reasoning for complex security analysis (all Gemini 3.x, 2.5 series, and Gemma 4 with toggleable thinking)
Context Caching: Significant cost reduction on repeated context (10% of input price for most models)
Ultra-Long Context: 1M tokens for Gemini chat models, 256K tokens for Gemma 4 open-source models
Multimodal Support: Text, image, video, audio, and PDF processing for comprehensive assessments
Tool Calling: Seamless integration with 20+ pentesting tools via function calling
Streaming: Real-time response streaming for interactive security workflows
Code Execution: Built-in code execution for offensive tool testing and exploit validation
Search Grounding: Google Search integration for threat intelligence and CVE research
File Search: Document retrieval and RAG capabilities for knowledge-based assessments
Batch API: 50% cost reduction for non-real-time batch processing
Custom Tools Endpoint: Dedicated gemini-3.1-pro-preview-customtools route for tool-heavy agentic workflows that prefer registered tools over bash

Reasoning Effort Levels:

High: Maximum thinking depth for complex multi-step analysis (generator)
Medium: Balanced reasoning for general agentic tasks (primary_agent, assistant, refiner, adviser)
Low: Efficient thinking for focused tasks (coder, installer, pentester)

AWS Bedrock Provider Configuration

PentAGI integrates with Amazon Bedrock, offering access to 20+ foundation models from leading AI companies including Anthropic, Amazon, Cohere, DeepSeek, OpenAI, Qwen, Mistral, and Moonshot.

Configuration Variables

Variable	Default	Description
`BEDROCK_REGION`	`us-east-1`	AWS region for Bedrock service
`BEDROCK_DEFAULT_AUTH`	`false`	Use AWS SDK default credential chain (environment, EC2 role, ~/.aws/credentials) - highest priority
`BEDROCK_BEARER_TOKEN`		Bearer token authentication - priority over static credentials
`BEDROCK_ACCESS_KEY_ID`		AWS access key ID for static credentials
`BEDROCK_SECRET_ACCESS_KEY`		AWS secret access key for static credentials
`BEDROCK_SESSION_TOKEN`		AWS session token for temporary credentials (optional, used with static credentials)
`BEDROCK_SERVER_URL`		Custom Bedrock endpoint (VPC endpoints, local testing)
`BEDROCK_CONFIG_PATH`		Path to a custom YAML provider config file (overrides the built-in default config for model/pricing definitions)

Authentication Priority: BEDROCK_DEFAULT_AUTH → BEDROCK_BEARER_TOKEN → BEDROCK_ACCESS_KEY_ID+BEDROCK_SECRET_ACCESS_KEY

Configuration Examples

# Recommended: Default AWS SDK authentication (EC2/ECS/Lambda roles)
BEDROCK_REGION=us-east-1
BEDROCK_DEFAULT_AUTH=true

# Bearer token authentication (AWS STS, custom auth)
BEDROCK_REGION=us-east-1
BEDROCK_BEARER_TOKEN=your_bearer_token

# Static credentials (development, testing)
BEDROCK_REGION=us-east-1
BEDROCK_ACCESS_KEY_ID=your_aws_access_key
BEDROCK_SECRET_ACCESS_KEY=your_aws_secret_key

# With proxy and custom endpoint
BEDROCK_REGION=us-east-1
BEDROCK_DEFAULT_AUTH=true
BEDROCK_SERVER_URL=https://bedrock-runtime.us-east-1.vpce-xxx.amazonaws.com
PROXY_URL=http://your-proxy:8080

Custom Provider Config and Models (advanced)

By default the Bedrock provider uses a per-agent config and model catalog compiled into the binary. One optional path overrides them without rebuilding:

BEDROCK_CONFIG_PATH — a YAML file (same shape as the other provider configs) that replaces the built-in per-agent model/price assignments. See examples/configs/bedrock-glm-flash.provider.yml.

This is useful to expose a Bedrock model newer than the compiled-in catalog — for example Z.AI's zai.glm-4.7-flash. Use the exact Model ID from the model's AWS Bedrock detail page; add a us./eu./apac. inference-profile prefix only when that page marks the model as requiring cross-region inference (zai.glm-4.7-flash is In-Region, so it is used as-is, with no prefix).

With Docker Compose, set the host-side mount source and the in-container path together:

# host file mounted into the container at /opt/pentagi/conf/bedrock.provider.yml
PENTAGI_BEDROCK_CONFIG_PATH=./examples/configs/bedrock-glm-flash.provider.yml
# tell the backend to read the mounted file
BEDROCK_CONFIG_PATH=/opt/pentagi/conf/bedrock.provider.yml

Supported Models

PentAGI supports 24 AWS Bedrock models with tool calling, streaming, and multimodal capabilities. Models marked with * are used in default configuration.

Model ID	Provider	Thinking	Multimodal	Price (Input/Output)	Use Case
`us.amazon.nova-2-lite-v1:0`	Amazon Nova	❌	✅	$0.33/$2.75	Adaptive reasoning, efficient thinking
`us.amazon.nova-premier-v1:0`	Amazon Nova	❌	✅	$2.50/$12.50	Complex reasoning, advanced analysis
`us.amazon.nova-pro-v1:0`	Amazon Nova	❌	✅	$0.80/$3.20	Balanced accuracy, speed, cost
`us.amazon.nova-lite-v1:0`	Amazon Nova	❌	✅	$0.06/$0.24	Fast processing, high-volume operations
`us.amazon.nova-micro-v1:0`	Amazon Nova	❌	❌	$0.035/$0.14	Ultra-low latency, real-time monitoring
`us.anthropic.claude-opus-4-8`	Anthropic	✅	✅	$5.00/$25.00	Flagship coding/agents/deep reasoning; adaptive thinking only (sampling params rejected)
`us.anthropic.claude-opus-4-7`	Anthropic	✅	✅	$5.00/$25.00	Advanced engineering, long-running agents; adaptive thinking only
`us.anthropic.claude-opus-4-6-v1`*	Anthropic	✅	✅	$5.00/$25.00	World-class coding, enterprise agents
`us.anthropic.claude-sonnet-4-6`	Anthropic	✅	✅	$3.00/$15.00	Frontier intelligence, enterprise scale
`us.anthropic.claude-opus-4-5-20251101-v1:0`	Anthropic	✅	✅	$5.00/$25.00	Multi-day software development
`us.anthropic.claude-haiku-4-5-20251001-v1:0`*	Anthropic	✅	✅	$1.00/$5.00	Near-frontier performance, high speed
`us.anthropic.claude-sonnet-4-5-20250929-v1:0`*	Anthropic	✅	✅	$3.00/$15.00	Real-world agents, coding excellence
`us.anthropic.claude-sonnet-4-20250514-v1:0`	Anthropic	✅	✅	$3.00/$15.00	Balanced performance, production-ready
`us.anthropic.claude-3-5-haiku-20241022-v1:0`	Anthropic	❌	❌	$0.80/$4.00	Fastest model, cost-effective scanning
`cohere.command-r-plus-v1:0`	Cohere	❌	❌	$3.00/$15.00	Large-scale operations, superior RAG
`deepseek.v3.2`	DeepSeek	❌	❌	$0.58/$1.68	Long-context reasoning, efficiency
`openai.gpt-oss-120b-1:0`*	OpenAI (OSS)	✅	❌	$0.15/$0.60	Strong reasoning, scientific analysis
`openai.gpt-oss-20b-1:0`	OpenAI (OSS)	✅	❌	$0.07/$0.30	Efficient coding, software development
`qwen.qwen3-next-80b-a3b`	Qwen	❌	❌	$0.15/$1.20	Ultra-long context, flagship reasoning
`qwen.qwen3-32b-v1:0`	Qwen	❌	❌	$0.15/$0.60	Balanced reasoning, research use cases
`qwen.qwen3-coder-30b-a3b-v1:0`	Qwen	❌	❌	$0.15/$0.60	Vibe coding, natural-language first
`qwen.qwen3-coder-next`	Qwen	❌	❌	$0.45/$1.80	Tool use, function calling optimized
`mistral.mistral-large-3-675b-instruct`	Mistral	❌	✅	$4.00/$12.00	Advanced multimodal, long-context
`moonshotai.kimi-k2.5`	Moonshot	❌	✅	$0.60/$3.00	Vision, language, code in one model

Prices: Per 1M tokens. Models with thinking/reasoning support additional compute costs during reasoning phase.

Tested but Incompatible Models

Some AWS Bedrock models were tested but are not supported due to technical limitations:

Model Family	Reason for Incompatibility
GLM (Z.AI)	Tool calling format incompatible with Converse API (expects string instead of JSON)
AI21 Jamba	Severe rate limits (1-2 req/min) prevent reliable testing and production use
Meta Llama 3.3/3.1	Unstable tool call result processing, causes unexpected failures in multi-turn workflows
Mistral Magistral	Tool calling not supported by the model
Moonshot K2-Thinking	Unstable streaming behavior with tool calls, unreliable in production
Qwen3-VL	Unstable streaming with tool calling, multimodal + tools combination fails intermittently

Important

Rate Limits & Quota Management

Default AWS Bedrock quotas for Claude models are extremely restrictive (2-20 requests/minute for new accounts). For production penetration testing:

Request quota increases through AWS Service Quotas console for models you plan to use
Use Amazon Nova models - higher default quotas and excellent performance
Enable provisioned throughput for consistent high-volume testing
Monitor usage - AWS throttles aggressively at quota limits

Without quota increases, expect frequent delays and workflow interruptions.

Warning

Converse API Requirements

PentAGI uses Amazon Bedrock Converse API for unified model access. All supported models require:

✅ Converse/ConverseStream API support
✅ Tool use (function calling) for penetration testing workflows
✅ Streaming tool use for real-time feedback

Verify model capabilities at: AWS Bedrock Model Features

Key Features:

Automatic Prompt Caching: 40-70% cost reduction on repeated context (Claude 4.x models)
Extended Thinking: Step-by-step reasoning for complex security analysis (Claude, DeepSeek R1, OpenAI GPT)
Multimodal Analysis: Process screenshots, diagrams, video for comprehensive testing (Nova, Claude, Mistral, Kimi)
Tool Calling: Seamless integration with 20+ pentesting tools via function calling
Streaming: Real-time response streaming for interactive security assessment workflows

DeepSeek Provider Configuration

PentAGI integrates with DeepSeek, providing access to advanced AI models with strong reasoning, coding capabilities, and context caching at competitive prices.

Configuration Variables

Variable	Default Value	Description
`DEEPSEEK_API_KEY`		DeepSeek API key for authentication
`DEEPSEEK_SERVER_URL`	`https://api.deepseek.com`	DeepSeek API endpoint URL
`DEEPSEEK_PROVIDER`		Provider prefix for LiteLLM integration (optional)

Configuration Examples

# Direct API usage
DEEPSEEK_API_KEY=your_deepseek_api_key
DEEPSEEK_SERVER_URL=https://api.deepseek.com

# With LiteLLM proxy
DEEPSEEK_API_KEY=your_litellm_key
DEEPSEEK_SERVER_URL=http://litellm-proxy:4000
DEEPSEEK_PROVIDER=deepseek  # Adds prefix to model names (deepseek/deepseek-v4-flash) for LiteLLM

Supported Models

PentAGI supports 2 DeepSeek V4 models with tool calling, streaming, hybrid thinking/non-thinking modes, and context caching. Both models support thinking mode by default and can be switched to non-thinking mode via extra_body. Models marked with * are used in default configuration.

Model ID	Thinking	Max Output	Context	Price (Input/Output/Cache)	Use Case
`deepseek-v4-flash`*	✅ hybrid	384K	1M	$0.14/$0.28/$0.0028	Utility agents, general dialogue, fast tool calling
`deepseek-v4-pro`*	✅ hybrid	384K	1M	$1.74/$3.48/$0.0145	Advanced reasoning, complex logic, security analysis

Prices: Per 1M tokens. Cache pricing applies to prompt tokens served from cache (input cache hit, reduced to 1/10 of launch price since 2026-04-26). Both models support hybrid thinking — thinking mode is enabled by default; pass extra_body.thinking.type: disabled to switch to non-thinking mode for faster/cheaper responses.

Pricing Note (deepseek-v4-pro): The 75% promotional discount on deepseek-v4-pro officially ended on 2026-05-31 15:59 UTC. The prices above reflect the standard post-promotional pricing. If you have legacy configurations using the discounted prices ($0.435/$0.87/$0.003625), update them to the current rates for accurate cost tracking.

The legacy model names deepseek-chat and deepseek-reasoner are scheduled for deprecation by DeepSeek on 2026-07-24. Existing user configurations referencing the legacy names continue to work until then; the defaults above use the current V4 names. deepseek-chat maps to deepseek-v4-flash non-thinking mode; deepseek-reasoner maps to deepseek-v4-flash thinking mode.

Default Agent Configuration:

Strategy: prefer deepseek-v4-flash (12x cheaper input, 12x cheaper output) as the workhorse for utility/lightweight agents; reserve deepseek-v4-pro for complex multi-step reasoning. The installer agent runs on Flash with thinking enabled because environment setup tasks (shell commands, config edits) rarely require pro-level reasoning. Run A/B tests on your own workloads before promoting more agents to Pro.

Agent Role	Default Model	Thinking	Reasoning Effort	Max Output	Temperature	Top P
Generator / Refiner	`deepseek-v4-pro`	Enabled	High	32768	(auto)	(auto)
Coder	`deepseek-v4-pro`	Enabled	High	20480	(auto)	(auto)
Primary Agent / Assistant / Pentester	`deepseek-v4-pro`	Enabled	High	16384	(auto)	(auto)
Adviser (mentor/planner)	`deepseek-v4-pro`	Enabled	High	8192	(auto)	(auto)
Installer	`deepseek-v4-flash`	Enabled	High	12288	(auto)	(auto)
Reflector / Searcher / Enricher	`deepseek-v4-flash`	Disabled	—	4096	0.5	0.9
Simple / Simple JSON	`deepseek-v4-flash`	Disabled	—	2048	0.3	0.9

Note: When thinking mode is enabled, DeepSeek silently ignores temperature, top_p, presence_penalty, and frequency_penalty. The langchaingo client automatically nullifies temperature/top_p when reasoning_effort is set, so they appear as "(auto)" in the table above. All thinking-enabled agents also explicitly pass extra_body.thinking.type: enabled as defensive coding against future provider default changes.

Key Features:

Hybrid Thinking Modes: Switch between thinking (deep reasoning) and non-thinking (fast) modes via extra_body.thinking.type
Automatic Prompt Caching: Significant cost reduction on repeated context via cache-hit pricing (1/10 of launch price)
Extended Thinking: Reinforcement learning CoT for complex security analysis (both V4 models)
Strong Coding: Optimized for code generation and exploit development
Long Context: 1M token context window with up to 384K output tokens
Tool Calling: Seamless integration with 20+ pentesting tools via function calling
Streaming: Real-time response streaming for interactive workflows
Multilingual: Strong Chinese and English support
Additional Features: JSON Output, Chat Prefix Completion (beta), FIM/Fill-in-the-Middle Completion (non-thinking mode only)

Concurrency Limits: deepseek-v4-flash: 2500 concurrent requests; deepseek-v4-pro: 500 concurrent requests.

LiteLLM Integration: Set DEEPSEEK_PROVIDER=deepseek to enable model name prefixing when using default PentAGI configurations with LiteLLM proxy. Leave empty for direct API usage.

GLM Provider Configuration

PentAGI integrates with GLM from Zhipu AI (Z.AI), providing advanced language models with MoE architecture, strong reasoning, and agentic capabilities developed by Tsinghua University.

Configuration Variables

Variable	Default Value	Description
`GLM_API_KEY`		GLM API key for authentication
`GLM_SERVER_URL`	`https://api.z.ai/api/paas/v4`	GLM API endpoint URL (international)
`GLM_PROVIDER`		Provider prefix for LiteLLM integration (optional)

Configuration Examples

# Direct API usage (international endpoint)
GLM_API_KEY=your_glm_api_key
GLM_SERVER_URL=https://api.z.ai/api/paas/v4

# Alternative endpoints
GLM_SERVER_URL=https://open.bigmodel.cn/api/paas/v4  # China
GLM_SERVER_URL=https://api.z.ai/api/coding/paas/v4   # Coding-specific

# With LiteLLM proxy
GLM_API_KEY=your_litellm_key
GLM_SERVER_URL=http://litellm-proxy:4000
GLM_PROVIDER=zai  # Adds prefix to model names (zai/glm-4) for LiteLLM

Supported Models

PentAGI supports 14 GLM models with tool calling, streaming, hybrid thinking modes, and prompt caching. Models marked with * are used in default configuration. Thinking is controlled via extra_body.thinking.type ("enabled"/"disabled"); unlike Kimi, GLM is permissive about temperature in either mode.

GLM-5.x Series - Latest Generation (200K context, 128K max output)

Model ID	Thinking	Context	Max Output	Price (Input/Output/Cache)	Use Case
`glm-5.2`*	✅ Hybrid	200K	128K	$1.40/$4.40/$0.26	Newest flagship, improves on GLM-5.1. Supports explicit `reasoning_effort` (high/max) (generator/refiner/adviser/coder/pentester default)
`glm-5.1`	✅ Hybrid	200K	128K	$1.40/$4.40/$0.26	Long-horizon tasks: 8h sustained autonomous execution, Claude Opus 4.6-aligned coding
`glm-5`	✅ Hybrid	200K	128K	$1.00/$3.20/$0.20	Foundation for Agentic Engineering, MoE 744B/40B active, Claude Opus 4.5-level coding
`glm-5-turbo`*	✅ Hybrid	200K	128K	$1.20/$4.00/$0.24	OpenClaw-native: optimized for tool invocation, persistent tasks, long-chain execution (primary_agent/assistant default)

GLM-4.7 Series - Premium with Interleaved Thinking

Model ID	Thinking	Context	Max Output	Price (Input/Output/Cache)	Use Case
`glm-4.7`	✅ Hybrid	200K	128K	$0.60/$2.20/$0.11	Enhanced programming, stable multi-step reasoning
`glm-4.7-flashx`	✅ Hybrid	200K	128K	$0.07/$0.40/$0.01	Ultra-cheap with priority GPU, but lower RPM limits (avoid for high-frequency use)
`glm-4.7-flash`	✅ Hybrid	200K	128K	Free/Free/Free	Free ~30B SOTA model, 1 concurrent request

GLM-4.6 Series - Balanced with Auto-Thinking

Model ID	Thinking	Context	Max Output	Price (Input/Output/Cache)	Use Case
`glm-4.6`	✅ Auto	200K	128K	$0.60/$2.20/$0.11	Balanced, streaming tool calls, token-efficient

GLM-4.5 Series - Unified Reasoning/Coding/Agents

Model ID	Thinking	Context	Max Output	Price (Input/Output/Cache)	Use Case
`glm-4.5`	✅ Auto	128K	96K	$0.60/$2.20/$0.11	Unified, MoE 355B/32B active
`glm-4.5-x`	✅ Auto	128K	96K	$2.20/$8.90/$0.45	Ultra-fast premium, lowest latency
`glm-4.5-air`*	✅ Auto	128K	96K	$0.20/$1.10/$0.03	Cost-effective MoE 106B/12B (simple/simple_json/reflector/searcher/enricher/installer default)
`glm-4.5-airx`	✅ Auto	128K	96K	$1.10/$4.50/$0.22	Accelerated Air with priority GPU
`glm-4.5-flash`	✅ Auto	128K	96K	Free/Free/Free	Free with reasoning/coding/agents support

GLM-4 Legacy - Dense Architecture

Model ID	Thinking	Context	Max Output	Price (Input/Output)	Use Case
`glm-4-32b-0414-128k`	❌	128K	16K	$0.10/$0.10	Ultra-budget dense 32B, parsing without reasoning

Prices: Per 1M tokens. Cache pricing is for prompt cache hit; cache storage is currently free per Z.AI promotion. GLM-4-32B has no cache support.

Default Agent Configuration:

Strategy: glm-5.2 (newest flagship, $1.40 input) for critical reasoning, glm-5-turbo (OpenClaw-native, agent-optimized) for orchestration, glm-4.5-air (cheap MoE with hybrid thinking and reliable RPM) for all utility/installer agents. glm-4.7-flashx is avoided as default due to lower RPM limits causing frequent 429 errors at high frequency.

Agent Role	Default Model	Thinking	Temperature	Top P	Max Output
Generator / Refiner	`glm-5.2`	Enabled	1.0	0.95	32768
Coder	`glm-5.2`	Enabled	1.0	0.95	20480
Adviser / Pentester	`glm-5.2`	Enabled	1.0	0.95	16384
Primary Agent / Assistant	`glm-5-turbo`	Enabled	1.0	0.95	16384
Installer	`glm-4.5-air`	Enabled	1.0	0.95	16384
Simple / Reflector	`glm-4.5-air`	Disabled	0.6	0.9	8192
Searcher / Enricher / Simple JSON	`glm-4.5-air`	Disabled	0.6	0.9	4096

Generator, refiner, and adviser additionally set reasoning.effort: max, which layers llms.WithReasoning(ReasoningMax, 0) on top of extra_body.thinking.type=enabled — this reasoning_effort parameter is only supported by glm-5.2.

Note on temperature: GLM accepts both 1.0 and 0.6 in either thinking/non-thinking mode (per Z.AI docs). langchaingo's IsReasoningModel matches glm-4.5*/glm-4.6*/glm-4.7* prefixes and force-overrides temperature to 1.0 in createChatRequest — this is harmless for GLM (unlike Kimi) but means temperature values for those models in YAML are advisory. glm-5/glm-5.1/glm-5.2/glm-5-turbo are not matched, so explicit values pass through unchanged.

Thinking Modes:

Hybrid (GLM-5.x, GLM-4.7): Explicit toggle via extra_body.thinking.type
Auto (GLM-4.6, GLM-4.5 series): Model automatically determines when reasoning is needed
Reasoning Effort (GLM-5.2 only): supports an explicit reasoning_effort parameter (high/max) on top of hybrid thinking, for finer control over reasoning depth than the other GLM-5.x models
Preserved Thinking (Z.AI Coding capability): all thinking-enabled agents in PentAGI also pass extra_body.thinking.clear_thinking: false so that reasoning_content from previous assistant turns is retained across the conversation. This is required on the standard API endpoint (/api/paas/v4) — on the Coding Plan endpoint it would be enabled by default. Improves reasoning continuity and cache hit rates in multi-turn tool call chains.
All thinking-enabled agents also pass extra_body.tool_choice: auto defensively

Key Features:

Long-Horizon Tasks: GLM-5.1 supports 8-hour sustained autonomous execution, ideal for complex multi-stage agentic workflows
OpenClaw-Native Orchestration: GLM-5-Turbo is specifically optimized for tool invocation, instruction following, and long-chain execution
Prompt Caching: Significant cost reduction on repeated context (cached input pricing shown)
Ultra-Long Context: 200K tokens for GLM-5.x/4.7/4.6 series
MoE Architecture: Efficient 744B/40B active (GLM-5/5.1), 355B/32B (GLM-4.5), 106B/12B (GLM-4.5-Air)
Tool Calling: Seamless integration with 20+ pentesting tools via function calling
Streaming: Real-time streaming with streaming tool calls support (GLM-4.6+)
Multilingual: Exceptional Chinese and English NLP capabilities
Free Options: GLM-4.7-Flash and GLM-4.5-Flash for prototyping and experimentation

LiteLLM Integration: Set GLM_PROVIDER=zai to enable model name prefixing when using default PentAGI configurations with LiteLLM proxy. Leave empty for direct API usage.

Kimi Provider Configuration

PentAGI integrates with Kimi from Moonshot AI, providing ultra-long context models with multimodal capabilities perfect for analyzing extensive codebases and documentation.

Configuration Variables

Variable	Default Value	Description
`KIMI_API_KEY`		Kimi API key for authentication
`KIMI_SERVER_URL`	`https://api.moonshot.ai/v1`	Kimi API endpoint URL (international)
`KIMI_PROVIDER`		Provider prefix for LiteLLM integration (optional)

Configuration Examples

# Direct API usage (international endpoint)
KIMI_API_KEY=your_kimi_api_key
KIMI_SERVER_URL=https://api.moonshot.ai/v1

# Alternative endpoint
KIMI_SERVER_URL=https://api.moonshot.cn/v1  # China

# With LiteLLM proxy
KIMI_API_KEY=your_litellm_key
KIMI_SERVER_URL=http://litellm-proxy:4000
KIMI_PROVIDER=moonshot  # Adds prefix to model names (moonshot/kimi-k2.5) for LiteLLM

Supported Models

PentAGI supports 11 Kimi/Moonshot models with tool calling, streaming, hybrid thinking modes, and multimodal capabilities (text/image/video for K2.x). All kimi-k2-* legacy models (turbo-preview, 0905-preview, 0711-preview, thinking, thinking-turbo) were deprecated by Moonshot on 2026-05-25 and are NOT included. Models marked with * are used in default configuration.

Kimi K3 - Flagship (Always Reasoning)

Model ID	Thinking	Multimodal	Context	Price (Input Miss / Output / Cache Hit)	Use Case
`kimi-k3`*	✅ always	❌	1M	$3.00 / $15.00 / $0.30	Flagship for long-horizon coding and end-to-end knowledge work. Always reasons — no `thinking` toggle, depth set via top-level `reasoning_effort` (currently `max` only) (generator/refiner/adviser default)

Kimi K2.7 Code Series - Coding-Focused

Model ID	Thinking	Multimodal	Context	Price (Input Miss / Output / Cache Hit)	Use Case
`kimi-k2.7-code`*	✅ hybrid	✅	256K	$0.95 / $4.00 / $0.19	Coding-focused, higher success rates on long-context programming tasks (coder/pentester default)
`kimi-k2.7-code-highspeed`*	✅ hybrid	✅	256K	$1.90 / $8.00 / $0.38	Same model as `kimi-k2.7-code` with higher output throughput (~180-260 tokens/s) (primary_agent/assistant default)

Kimi K2.x Series - Multimodal Flagship

Model ID	Thinking	Multimodal	Context	Price (Input Miss / Output / Cache Hit)	Use Case
`kimi-k2.6`	✅ hybrid	✅	256K	$0.95 / $4.00 / $0.16	Latest multimodal flagship: native architecture, stronger code, improved instruction compliance (not used by default config)
`kimi-k2.5`*	✅ hybrid	✅	256K	$0.60 / $3.00 / $0.10	Previous-gen: 36% cheaper input than K2.6 (simple/simple_json/reflector/searcher/enricher/installer default)

Moonshot V1 Series - Generation Models (Flexible Parameters)

Model ID	Thinking	Multimodal	Context	Price (Input / Output)	Use Case
`moonshot-v1-8k`	❌	❌	8K	$0.20 / $2.00	Short text generation, ultra-cheap
`moonshot-v1-32k`	❌	❌	32K	$1.00 / $3.00	Long text generation
`moonshot-v1-128k`	❌	❌	128K	$2.00 / $5.00	Very long context

Moonshot V1 Vision Series - Image Understanding

Model ID	Thinking	Multimodal	Context	Price (Input / Output)	Use Case
`moonshot-v1-8k-vision-preview`	❌	✅	8K	$0.20 / $2.00	Vision + short context
`moonshot-v1-32k-vision-preview`	❌	✅	32K	$1.00 / $3.00	Vision + medium context
`moonshot-v1-128k-vision-preview`	❌	✅	128K	$2.00 / $5.00	Vision + long context

Prices: Per 1M tokens. Cache pricing applies to prompt tokens served from automatic context cache (only Kimi K3/K2.7/K2.x models support cache).

CRITICAL — Kimi parameter constraints per model family: API returns invalid_request_error for any deviation:

kimi-k3: always reasons, no thinking param at all; reasoning depth is set via the top-level reasoning_effort field (low/high/max, default max) — PentAGI pins it to max for all agents using this model. temperature MUST be 1.0, top_p MUST be 0.95, n MUST be 1, presence_penalty/frequency_penalty MUST be 0. Do not switch effort per call — it invalidates the prefix cache.

kimi-k2.7-code / kimi-k2.7-code-highspeed: thinking may be omitted; if set explicitly, only {"type":"enabled","keep":"all"} is accepted (type: disabled is rejected). reasoning_effort is not supported. temperature MUST be 1.0, top_p MUST be 0.95, n MUST be 1; tool_choice: required is not supported (use auto).

kimi-k2.6: thinking mode needs temperature=1.0, top_p=0.95, n=1, thinking.keep="all"; non-thinking mode needs temperature=0.6, top_p=0.95, n=1.

kimi-k2.5: thinking mode needs temperature=1.0, top_p=0.95, n=1 (no keep support); non-thinking mode needs temperature=0.6, top_p=0.95, n=1.

All Kimi models: presence_penalty=0, frequency_penalty=0, tool_choice in {auto, none}.

Moonshot V1 models use standard OpenAI-compatible parameters with no such constraints.

Default Agent Configuration:

Strategy: kimi-k2.5 for utility/orchestration, kimi-k2.7-code-highspeed for the primary/assistant loop, kimi-k3 (always-thinking flagship) for critical reasoning (generator/refiner/adviser), kimi-k2.7-code for coder/pentester. kimi-k2.6 is not used in the default configuration. All kimi-k2.x/k2.7 agents are configured with the API-required fixed parameters (temp/top_p/n) and explicit extra_body.thinking.type. For thinking-enabled agents, extra_body.thinking.keep: "all" is set (where supported) to preserve historical reasoning_content in multi-turn tool call chains (without it Moonshot returns "thinking is enabled but reasoning_content is missing").

Agent Role	Default Model	Thinking	Temperature	Top P	Max Output
Generator / Refiner	`kimi-k3`	Always (effort=max)	1.0	0.95	32768
Adviser (mentor/planner)	`kimi-k3`	Always (effort=max)	1.0	0.95	8192
Coder	`kimi-k2.7-code`	Enabled (keep=all)	1.0	0.95	20480
Pentester	`kimi-k2.7-code`	Enabled (keep=all)	1.0	0.95	16384
Primary Agent / Assistant	`kimi-k2.7-code-highspeed`	Enabled (keep=all)	1.0	0.95	16384
Installer	`kimi-k2.5`	Enabled	1.0	0.95	16384
Reflector / Searcher / Enricher	`kimi-k2.5`	Disabled	0.6	0.95	4096
Simple	`kimi-k2.5`	Disabled	0.6	0.95	8192
Simple JSON	`kimi-k2.5`	Disabled	0.6	0.95	4096

Note: for kimi-k2.5 non-thinking agents, PentAGI also duplicates temperature: 0.6 into extra_body as a workaround — langchaingo's IsReasoningModel matches the substring 2.5 and force-overrides temperature to 1.0, and extra_body bypasses that override.

Key Features:

Always-On Reasoning Flagship: kimi-k3 never disables thinking and offers a 1M token context for the most demanding long-horizon coding and knowledge work
Ultra-Long Context: Up to 256K tokens (K2.7/K2.x) or 1M tokens (K3) for comprehensive codebase/documentation analysis
Native Multimodal: K2.7/K2.6/K2.5 support text + image + video input out of the box
Hybrid Thinking: K2.7/K2.6/K2.5 toggle between thinking and non-thinking via extra_body.thinking.type
Preserved Thinking (K2.7, K2.6): thinking.keep: "all" preserves historical reasoning_content across turns — required for multi-turn tool call chains
Automatic Context Caching: K3/K2.7/K2.x models cache repeated prefixes
Tool Calling: Full function-calling support for K3, K2.7, K2.x, and Moonshot V1
Coding-Optimized Variants: kimi-k2.7-code/kimi-k2.7-code-highspeed target higher success rates on long-context programming tasks, with the highspeed variant tuned for throughput
Multilingual: Strong Chinese, English, and multi-language support

Multi-turn with thinking + tool calls: PentAGI's universal reasoning preservation pattern (TextPartWithReasoning + WithPreserveReasoningContent) automatically ensures reasoning_content is sent back in the required TextContent → ToolCall order, satisfying Moonshot's "thinking is enabled but reasoning_content is missing in assistant tool call message" requirement.

LiteLLM Integration: Set KIMI_PROVIDER=moonshot to enable model name prefixing when using default PentAGI configurations with LiteLLM proxy. Leave empty for direct API usage.

Qwen Provider Configuration

PentAGI integrates with Qwen from Alibaba Cloud Model Studio (DashScope), providing powerful multilingual models with reasoning capabilities and context caching support.

Configuration Variables

Variable	Default Value	Description
`QWEN_API_KEY`		Qwen API key for authentication
`QWEN_SERVER_URL`	`https://dashscope-us.aliyuncs.com/compatible-mode/v1`	Qwen API endpoint URL (international)
`QWEN_PROVIDER`		Provider prefix for LiteLLM integration (optional)

Configuration Examples

# Direct API usage (Global/US endpoint)
QWEN_API_KEY=your_qwen_api_key
QWEN_SERVER_URL=https://dashscope-us.aliyuncs.com/compatible-mode/v1

# Alternative endpoints
QWEN_SERVER_URL=https://dashscope-intl.aliyuncs.com/compatible-mode/v1  # International (Singapore)
QWEN_SERVER_URL=https://dashscope.aliyuncs.com/compatible-mode/v1       # Chinese Mainland (Beijing)

# With LiteLLM proxy
QWEN_API_KEY=your_litellm_key
QWEN_SERVER_URL=http://litellm-proxy:4000
QWEN_PROVIDER=dashscope  # Adds prefix to model names (dashscope/qwen-plus) for LiteLLM

Supported Models

PentAGI supports 33 Qwen models curated for agent workflows: text reasoning, code generation, and vision-language (browser screenshots). All models are non-snapshot main aliases with tool calling, streaming, thinking modes, and context caching. Models marked with * are used in default configuration.

Flagship Models (Top-tier Reasoning)

Model ID	Thinking	Intl	Global/US	China	Price (Input/Output/Cache)	Use Case
`qwen3.7-max`*	✅	✅	✅	✅	$2.50/$7.50/$0.50	Next-gen flagship for agent-centric era (generator/refiner/adviser default)
`qwen3.6-max-preview`	✅	✅	✅	✅	$1.30/$7.80/$0.13	Preview Max with enhanced vibe coding & front-end skills
`qwen3-max`	✅	✅	✅	✅	$1.20/$6.00/$0.24	Previous-gen flagship with agent programming upgrades
`qwen-plus`	✅	✅	✅	✅	$0.40/$4.00/$0.08	Qwen3-backbone Plus with switchable thinking modes

Balanced Plus Models (Mid-tier)

Model ID	Thinking	Intl	Global/US	China	Price (Input/Output/Cache)	Use Case
`qwen3.6-plus`*	✅	✅	✅	✅	$0.50/$3.00/$0.05	Native VL Plus with agentic coding (primary/assistant/pentester default)
`qwen3.5-plus`	✅	✅	✅	✅	$0.40/$2.40/$0.04	Previous-gen native VL with strong multimodal capabilities

Fast Flash Models (Cost-optimized)

Model ID	Thinking	Intl	Global/US	China	Price (Input/Output/Cache)	Use Case
`qwen3.6-flash`	✅	✅	✅	✅	$0.25/$1.50/$0.025	Latest Flash with significant agentic-coding boost
`qwen3.5-flash`*	✅	✅	✅	✅	$0.10/$0.40/$0.01	Ultra-fast lightweight (simple/reflector/searcher/enricher default)
`qwen-flash`	✅	✅	✅	✅	$0.05/$0.40/$0.01	Qwen3-series Flash with 1M context, tiered pricing

Code-Specialized Models

Model ID	Thinking	Intl	Global/US	China	Price (Input/Output/Cache)	Use Case
`qwen3-coder-plus`*	❌	✅	✅	✅	$1.00/$5.00/$0.20	Strong coding agent with autonomous programming (coder default)
`qwen3-coder-flash`*	❌	✅	✅	✅	$0.30/$1.50/$0.06	Fast code-gen with multi-turn tool stability (installer default)
`qwen3-coder-next`	❌	✅	✅	✅	$0.30/$1.50/—	Open-source code generation, SOTA at same scale

Vision-Language Models (Browser & Screenshot Analysis)

Model ID	Thinking	Intl	Global/US	China	Price (Input/Output/Cache)	Use Case
`qwen3-vl-plus`	✅	✅	✅	✅	$0.20/$1.60/$0.04	VL with visual agent capabilities, ultra-long video understanding
`qwen3-vl-flash`	✅	✅	✅	✅	$0.05/$0.40/$0.01	Small VL with 2D/3D localization for browser triage
`qvq-max`	✅	✅	✅	✅	$1.20/$4.80/—	Visual reasoning with chain-of-thought

Open-Source Qwen3.6 Series

Model ID	Thinking	Intl	Global/US	China	Price (Input/Output/Cache)	Use Case
`qwen3.6-27b`	✅	✅	✅	✅	$0.60/$3.60/—	Native VL on hybrid architecture, on-premises ready
`qwen3.6-35b-a3b`	✅	✅	✅	✅	$0.25/$1.49/—	Efficient 35B MoE (~3B active) for continuous monitoring

Open-Source Qwen3.5 Series

Model ID	Thinking	Intl	Global/US	China	Price (Input/Output/Cache)	Use Case
`qwen3.5-397b-a17b`	✅	✅	✅	✅	$0.60/$3.60/—	Largest 397B params (~17B active), exceptional reasoning
`qwen3.5-122b-a10b`	✅	✅	✅	✅	$0.40/$3.20/—	Large 122B params (~10B active), strong balance
`qwen3.5-35b-a3b`	✅	✅	✅	✅	$0.25/$2.00/—	Efficient 35B MoE (~3B active), cost-effective
`qwen3.5-27b`	✅	✅	✅	✅	$0.30/$2.40/—	Medium 27B with hybrid linear attention + sparse MoE

Open-Source Qwen3 Coder Series

Model ID	Thinking	Intl	Global/US	China	Price (Input/Output/Cache)	Use Case
`qwen3-coder-480b-a35b-instruct`	❌	✅	✅	✅	$1.50/$7.50/—	Largest open coder MoE (480B/~35B active)
`qwen3-coder-30b-a3b-instruct`	❌	✅	✅	✅	$0.45/$2.25/—	Efficient 30B MoE (~3B active), repository-scale

Open-Source Qwen3 Dense & MoE Series

Model ID	Thinking	Intl	Global/US	China	Price (Input/Output/Cache)	Use Case
`qwen3-next-80b-a3b-thinking`	✅	✅	✅	✅	$0.15/$1.20/—	Next-gen 80B MoE (~3B active) thinking-only
`qwen3-next-80b-a3b-instruct`	❌	✅	✅	✅	$0.15/$1.20/—	Next-gen 80B MoE instruction-following
`qwen3-235b-a22b`	✅	✅	✅	✅	$0.70/$8.40/—	Dual-mode 235B MoE (~22B active)
`qwen3-32b`	✅	✅	✅	✅	$0.16/$0.64/—	Versatile 32B dense dual-mode
`qwen3-30b-a3b`	✅	✅	✅	✅	$0.20/$2.40/—	Efficient 30B MoE (~3B active)
`qwen3-14b`	✅	✅	✅	✅	$0.35/$4.20/—	Medium 14B dense performance-cost balance
`qwen3-8b`	✅	✅	✅	✅	$0.18/$2.10/—	Compact 8B dense efficiency
`qwen3-4b`	✅	✅	✅	✅	$0.11/$1.26/—	Lightweight 4B dense for simple tasks
`qwen3-1.7b`	✅	✅	✅	✅	$0.11/$1.26/—	Ultra-compact 1.7B basic checks
`qwen3-0.6b`	✅	✅	✅	✅	$0.11/$1.26/—	Smallest 0.6B for edge monitoring

Prices: Per 1M tokens. Cache pricing reflects implicit cache hit (when available); MoE/dense open-source models do not expose cache pricing. Tiered models (Max/Plus) show lowest-tier pricing (typically ≤32k or ≤256k input); larger contexts incur higher rates per Alibaba Cloud pricing.

Region Availability:

Intl (International): Singapore region (dashscope-intl.aliyuncs.com)
Global/US: US Virginia region (dashscope-us.aliyuncs.com)
China: Chinese Mainland Beijing region (dashscope.aliyuncs.com)

Default Agent Configuration:

Agent Role	Default Model	Tier
Generator / Refiner / Adviser (planning, mentor)	`qwen3.7-max`	Flagship
Primary / Assistant / Pentester	`qwen3.6-plus`	Balanced
Coder (exploit development)	`qwen3-coder-plus`	Code+
Installer (env setup)	`qwen3-coder-flash`	Code Fast
Simple / Reflector / Searcher / Enricher	`qwen3.5-flash`	Fast

Key Features:

Agent-Centric Design: Qwen3.7-Max is purpose-built for long-horizon autonomous execution and tool invocation
Automatic Context Caching: 30-50% cost reduction on repeated context with implicit cache
Extended Thinking: Chain-of-thought reasoning for complex security analysis (Qwen3.7/3.6/3.5/3-Max, QVQ-Max)
Code Specialization: Qwen3-Coder series with multi-turn tool interaction and repository-level understanding
Vision-Language: Qwen3-VL series for browser screenshot triage, 2D/3D localization, OCR-level analysis
Tool Calling: Seamless integration with 20+ pentesting tools via function calling
Streaming: Real-time response streaming for interactive workflows
Multilingual: Strong Chinese, English, and multi-language support
Open-Source Variants: Dense and MoE models from 0.6B to 480B for on-premises/air-gapped deployments

LiteLLM Integration: Set QWEN_PROVIDER=dashscope to enable model name prefixing when using default PentAGI configurations with LiteLLM proxy. Leave empty for direct API usage.

Alternative Integrations

DashScope is fully OpenAI-compatible, so Qwen can also power two other PentAGI subsystems through the standard OpenAI client.

As embedding provider (text-embedding-v4, see Alibaba Cloud Model Studio pricing):

EMBEDDING_PROVIDER=openai
EMBEDDING_URL=https://dashscope-intl.aliyuncs.com/compatible-mode/v1  # International (Singapore)
# EMBEDDING_URL=https://dashscope.aliyuncs.com/compatible-mode/v1     # Chinese Mainland
EMBEDDING_KEY=sk-*******
EMBEDDING_MODEL=text-embedding-v4
EMBEDDING_BATCH_SIZE=         # optional, default applies
EMBEDDING_STRIP_NEW_LINES=    # optional, default applies

Note: the Global/US DashScope endpoint (dashscope-us.aliyuncs.com) does not expose embedding APIs — use the International or China endpoints for text-embedding-v4.

As OpenAI-typed custom LLM provider: instead of the dedicated QWEN_* variables, you can wire any Qwen chat model through PentAGI's custom OpenAI-compatible provider by pointing OPENAI_SERVER_URL (or a custom provider entry) to the DashScope /compatible-mode/v1 endpoint and selecting the desired Qwen model name. Useful when you already manage all model traffic through a single OpenAI-shaped client (e.g. shared with LiteLLM/OneAPI proxies).

MiniMax Provider Configuration

PentAGI integrates with MiniMax's M-series through the OpenAI-compatible https://api.minimax.io/v1 endpoint: large-context agentic models with tool calling, JSON output, and streaming.

Configuration Variables

Variable	Default Value	Description
`MINIMAX_API_KEY`		MiniMax API key for authentication
`MINIMAX_SERVER_URL`	`https://api.minimax.io/v1`	MiniMax API endpoint URL
`MINIMAX_PROVIDER`		Provider prefix for LiteLLM integration (optional)

Configuration Examples

# Direct API usage
MINIMAX_API_KEY=your_minimax_api_key
MINIMAX_SERVER_URL=https://api.minimax.io/v1

# With LiteLLM proxy
MINIMAX_API_KEY=your_litellm_key
MINIMAX_SERVER_URL=http://litellm-proxy:4000
MINIMAX_PROVIDER=minimax  # Adds prefix to model names (minimax/MiniMax-M3) for LiteLLM

Supported Models

PentAGI ships 3 MiniMax models with tool calling, JSON output, and streaming. MiniMax-M3 is the default for all agent types.

Model ID	Context	Price (Input/Output, ≤512K context)	Use Case
`MiniMax-M3`*	~1M	$0.30/$1.20 (2x above 512K tokens)	Latest flagship for agentic reasoning, tool use, code generation, and long-context tasks (default)
`MiniMax-M2.7`	204K	$0.30/$1.20	Previous-generation model with strong reasoning and coding
`MiniMax-M2.7-highspeed`	204K	$0.60/$2.40	Low-latency variant of M2.7 for fast-response scenarios

LiteLLM Integration: Set MINIMAX_PROVIDER=minimax to enable model name prefixing when using default PentAGI configurations with LiteLLM proxy. Leave empty for direct API usage.

Advanced Setup

Langfuse Integration

Langfuse provides advanced capabilities for monitoring and analyzing AI agent operations.

Configure Langfuse environment variables in existing .env file.

Langfuse valuable environment variables

Database Credentials

LANGFUSE_POSTGRES_USER and LANGFUSE_POSTGRES_PASSWORD - Langfuse PostgreSQL credentials
LANGFUSE_CLICKHOUSE_USER and LANGFUSE_CLICKHOUSE_PASSWORD - ClickHouse credentials
LANGFUSE_REDIS_AUTH - Redis password

Encryption and Security Keys

LANGFUSE_SALT - Salt for hashing in Langfuse Web UI
LANGFUSE_ENCRYPTION_KEY - Encryption key (32 bytes in hex)
LANGFUSE_NEXTAUTH_SECRET - Secret key for NextAuth

Admin Credentials

LANGFUSE_INIT_USER_EMAIL - Admin email
LANGFUSE_INIT_USER_PASSWORD - Admin password
LANGFUSE_INIT_USER_NAME - Admin username

API Keys and Tokens

LANGFUSE_INIT_PROJECT_PUBLIC_KEY - Project public key (used from PentAGI side too)
LANGFUSE_INIT_PROJECT_SECRET_KEY - Project secret key (used from PentAGI side too)

S3 Storage

LANGFUSE_S3_ACCESS_KEY_ID - S3 access key ID
LANGFUSE_S3_SECRET_ACCESS_KEY - S3 secret access key

Enable integration with Langfuse for PentAGI service in .env file.

LANGFUSE_BASE_URL=http://langfuse-web:3000
LANGFUSE_PROJECT_ID= # default: value from ${LANGFUSE_INIT_PROJECT_ID}
LANGFUSE_PUBLIC_KEY= # default: value from ${LANGFUSE_INIT_PROJECT_PUBLIC_KEY}
LANGFUSE_SECRET_KEY= # default: value from ${LANGFUSE_INIT_PROJECT_SECRET_KEY}

Run the Langfuse stack:

curl -O https://raw.githubusercontent.com/vxcontrol/pentagi/master/docker-compose-langfuse.yml
docker compose -f docker-compose.yml -f docker-compose-langfuse.yml up -d

Visit localhost:4000 to access Langfuse Web UI with credentials from .env file:

LANGFUSE_INIT_USER_EMAIL - Admin email
LANGFUSE_INIT_USER_PASSWORD - Admin password

Monitoring and Observability

For detailed system operation tracking, integration with monitoring tools is available.

Enable integration with OpenTelemetry and all observability services for PentAGI in .env file.

OTEL_HOST=otelcol:8148

Run the observability stack:

curl -O https://raw.githubusercontent.com/vxcontrol/pentagi/master/docker-compose-observability.yml
docker compose -f docker-compose.yml -f docker-compose-observability.yml up -d

Visit localhost:3000 to access Grafana Web UI.

Note

If you want to use Observability stack with Langfuse, you need to enable integration in .env file to set LANGFUSE_OTEL_EXPORTER_OTLP_ENDPOINT to http://otelcol:4318.

To run all available stacks together (Langfuse, Graphiti, and Observability):

docker compose -f docker-compose.yml -f docker-compose-langfuse.yml -f docker-compose-graphiti.yml -f docker-compose-observability.yml up -d

You can also register aliases for these commands in your shell to run it faster:

alias pentagi="docker compose -f docker-compose.yml -f docker-compose-langfuse.yml -f docker-compose-graphiti.yml -f docker-compose-observability.yml"
alias pentagi-up="docker compose -f docker-compose.yml -f docker-compose-langfuse.yml -f docker-compose-graphiti.yml -f docker-compose-observability.yml up -d"
alias pentagi-down="docker compose -f docker-compose.yml -f docker-compose-langfuse.yml -f docker-compose-graphiti.yml -f docker-compose-observability.yml down"

Knowledge Graph Integration (Graphiti)

Important

Graphiti is an optional beta integration and is disabled by default. Review Limitations and Security before enabling it in production.

PentAGI integrates with Graphiti, a temporal knowledge graph system powered by Neo4j, to provide advanced semantic understanding and relationship tracking for AI agent operations. The vxcontrol fork provides custom entity and edge types that are specific to pentesting purposes.

What is Graphiti?

Graphiti asynchronously extracts structured knowledge from agent interactions and builds a graph of entities, relationships, evidence, and temporal context. PentAGI sends agent responses and tool executions to Graphiti and exposes the graphiti_search tool to enabled agents. Graphiti complements the primary pgvector memory; it does not replace it.

Semantic Memory: Store and recall relationships between tools, targets, vulnerabilities, and techniques
Contextual Understanding: Track how different pentesting actions relate to each other over time
Flow-Scoped Recall: Reuse knowledge within the active flow without exposing data from other engagements by default
Advanced Querying: Search temporal context, relationships, successful tools, recent episodes, and entities by type

When enabled, PentAGI captures agent responses, tool execution details, and flow/task/subtask context. Ingestion is asynchronous, so newly submitted events can take time to become searchable.

Deployment Modes and Enabling

Graphiti can run as the bundled Neo4j + Graphiti stack, as an external service, or remain disabled.

For the bundled stack, configure .env:

GRAPHITI_ENABLED=true
GRAPHITI_TIMEOUT=30
GRAPHITI_URL=http://graphiti:8000
GRAPHITI_LLM_CLIENT_TYPE=openai

# Reused by the Graphiti OpenAI preset
OPEN_AI_KEY=your_openai_api_key
OPEN_AI_SERVER_URL=https://api.openai.com/v1

# Bundled Neo4j
NEO4J_USER=neo4j
NEO4J_DATABASE=neo4j
NEO4J_PASSWORD=replace_with_a_strong_password
NEO4J_URI=bolt://neo4j:7687

Download the optional compose file when installing manually, then start both stacks:

curl -O https://raw.githubusercontent.com/vxcontrol/pentagi/master/docker-compose-graphiti.yml
docker compose -f docker-compose.yml -f docker-compose-graphiti.yml up -d

The base stack must create the external pentagi-network before the Graphiti stack can start. The installer handles stack ordering automatically.

For an external Graphiti deployment, set GRAPHITI_ENABLED=true and point GRAPHITI_URL to its API. Do not start docker-compose-graphiti.yml; configure providers, embeddings, the graph database, and ingest tuning on the external service itself.

PentAGI enables its client only when both GRAPHITI_ENABLED=true and GRAPHITI_URL is non-empty. At startup it performs three health-check attempts with a two-second backoff. If they all fail, PentAGI logs a warning and continues with Graphiti disabled.

LLM Provider and Model Presets

GRAPHITI_LLM_CLIENT_TYPE selects one deployment-wide preset. Model names and call parameters are not environment variables; they live in graphiti/<provider>.yaml.

Preset	Credentials and endpoint	Shipped main model
`openai`	`OPEN_AI_KEY`, `OPEN_AI_SERVER_URL`	`openai/gpt-5-mini`
`gemini`	`GEMINI_API_KEY`, `GEMINI_SERVER_URL`	`gemini/gemini-2.5-flash-lite`
`custom`	`LLM_SERVER_KEY`, `LLM_SERVER_URL`	`Qwen/Qwen3.6-27B-FP8`
`litellm`	`GRAPHITI_LITELLM_API_KEY`, `GRAPHITI_LITELLM_BASE_URL`	`openrouter/openai/gpt-oss-20b`

The Gemini preset uses Graphiti's LiteLLM/OpenAI-compatible client path. Point GEMINI_SERVER_URL at a compatible gateway if the native Gemini endpoint does not provide the required OpenAI-compatible API.

Each preset file must contain a matching provider plus MODEL_NAME and SMALL_MODEL_NAME mappings. The small model is used for reranking and lighter calls. Supported call settings include temperature, token limits, sampling and penalty parameters, JSON mode, reasoning effort, verbosity, pricing metadata, and provider-specific extra_body.

The installer copies examples/graphiti beside the installation as ./graphiti. The compose mount is controlled by:

GRAPHITI_CONFIG_PATH=./graphiti
GRAPHITI_CONFIG_DIR=llm_configs

GRAPHITI_CONFIG_PATH may point directly to ./examples/graphiti for development. GRAPHITI_CONFIG_DIR=llm_configs activates the mounted presets. If an older .env omits that variable, a newer compose file mounts an empty host directory at the unused configs path instead of hiding the presets built into the image.

Note

GRAPHITI_MODEL_NAME is obsolete and ignored. Edit the active YAML preset instead, then restart the Graphiti container.

Graphiti Embedding Configuration

By default, Graphiti uses the active LLM preset's credentials and its default OpenAI embedding model. To use PentAGI's shared embedding endpoint explicitly:

GRAPHITI_SEPARATE_EMBEDDING=true
EMBEDDING_URL=https://embedding.example.com/v1
EMBEDDING_KEY=your_embedding_api_key
EMBEDDING_MODEL=openai/text-embedding-3-large

Graphiti's embedder is OpenAI-compatible. EMBEDDING_PROVIDER is used by PentAGI but is not passed to Graphiti, so a non-OpenAI-compatible embedding provider cannot be shared directly.

Ingestion and Extraction Tuning

The supplied defaults prioritize flow isolation and limit expensive extraction to useful events.

Ingest policy actions:

REJECT: do not store the episode.
SKIP_LLM: store the episode for retrieval but do not extract nodes or edges.
PROCESS: store the episode and run full LLM extraction.

Variable	Default	When to change it
`GRAPHITI_INGEST_POLICY_RULES`	`{"graphiti_search":"REJECT","tool_execution_terminal":"PROCESS","tool_execution_file":"PROCESS"}`	Add narrow, case-insensitive name/source patterns when specific events need different handling
`GRAPHITI_INGEST_POLICY_FIELD`	`both`	Restrict matching to `name` or `source_description` only when event naming is controlled
`GRAPHITI_INGEST_POLICY_DEFAULT_ACTION`	`SKIP_LLM`	Use `PROCESS` only when every unmatched event justifies extraction cost
`GRAPHITI_INGEST_USE_GROUP_ACTORS`	`true`	Keep enabled to preserve FIFO ordering independently for each flow
`GRAPHITI_INGEST_WORKER_COUNT`	`16`	Raise for more concurrent flows when the LLM and database have capacity; lower to control load
`GRAPHITI_INGEST_LOCK_BY_GROUP_ID`	`true`	Used only in shared-pool mode; ignored when group actors are enabled
`GRAPHITI_INGEST_TASK_MAX_RETRIES`	`1` (`0`-`5`)	Increase for transient LLM/network failures
`GRAPHITI_INGEST_TASK_RETRY_DELAY_SEC`	`2.0` (`0.5`-`60`)	Increase when an upstream service needs more recovery time
`GRAPHITI_INGEST_TASK_TIMEOUT_SEC`	`0` (`0`-`3600`)	Set a finite value to prevent one stalled request from blocking a flow; `0` disables the timeout
`GRAPHITI_INGEST_QUEUE_MAX_SIZE`	`0`	Set a bound to return HTTP 429 instead of allowing an unlimited backlog
`GRAPHITI_INGEST_DEAD_LETTER_ENABLED`	`false`	Enable when failed episodes must be retained for operational review

Extraction uses the following fallback order: full combined extraction (nodes, attributes, summaries, and edges in one call), regular combined extraction (nodes and edges), then separate node/edge extraction. Empty or failed combined results automatically fall back; these log messages are expected during normal operation.

Variable	Default	Effect
`GRAPHITI_TAXONOMY_LAYER_PROFILE`	`STRUCTURAL,EVIDENCE,PROGRESS,ATTEMPT`	Controls which edge classes appear in prompts and pass validation; `full`/`all` enables every class and `minimal` selects the core attack graph
`GRAPHITI_USE_COMBINED_FULL_EXTRACTION`	`true`	Enables the most compact single-call extraction path
`GRAPHITI_USE_COMBINED_EXTRACTION`	`true`	Enables the regular combined fallback
`GRAPHITI_COMBINED_FULL_GATING_ENABLED`	`true`	Skips expensive full extraction for low-signal administrative/search events
`GRAPHITI_COMBINED_DIAGNOSTIC_SAMPLES`	`false`	Includes content samples in diagnostics; keep disabled because pentest output can contain credentials
`GRAPHITI_ANCHOR_NODE_MODE`	`smart`	`smart` loads all key entities plus limited high-volume types; `limit` applies one total cap
`GRAPHITI_ANCHOR_NODE_LIMIT`	`25` (`1`-`500`)	Total anchor cap in `limit` mode
`GRAPHITI_ANCHOR_MASS_TYPE_LIMIT`	`10` (`1`-`100`)	Per-type cap in `smart` mode; `0` is invalid
`GRAPHITI_ANCHOR_QUERY_TIMEOUT`	`10` (`1`-`60`)	Bounds anchor lookup; timeout degrades gracefully to no anchors

Anchors connect entities across episodes and are used by the separate extraction path. Combined extraction has already produced its edges and does not perform this anchor lookup.

These flags are passed as process environment variables by the bundled compose file. This is important for combined extraction because Graphiti reads those flags when Python modules are imported.

Runtime, Logging, and Neo4j

The values below are PentAGI's recommended .env.example/compose defaults, not the raw Graphiti image fallbacks. Running a freshly pulled image behind an old compose file can instead enable telemetry and global search, use one shared-pool worker with PROCESS as the unmatched ingest action, enable the full taxonomy, and disable combined extraction. Keep the image, compose file, .env, and presets in sync.

Variable	Default	Guidance
`GRAPHITI_CPUS`, `GRAPHITI_MEMORY`	`2.0`, `2G`	Container limits; raise together with concurrency only after observing CPU and memory pressure
`GRAPHITI_SEMAPHORE_LIMIT`	`20`	Limits parallel Graphiti coroutines; it is separate from ingest worker concurrency
`GRAPHITI_TELEMETRY_ENABLED`	`false`	Enables anonymous Graphiti telemetry when set to `true`
`GRAPHITI_LOG_LEVEL`	`INFO`	Use `DEBUG` temporarily; it can produce sensitive and high-volume output
`GRAPHITI_LOG_STDOUT`	`events`	`off`, `events`, or `full`; `events` is recommended for containers
`GRAPHITI_FLOW_LOGGER_WARN_COUNT`	`256`	Warns about growth of cached per-flow loggers; `0` disables the warning
`GRAPHITI_DEBUG_RUNTIME_RESOURCES`	`false`	Enables `/debug/runtime-resources`; expose only to trusted operators
`GRAPHITI_SEARCH_SCOPE`	`flowid`	Keep for flow/tenant isolation; `all` enables global searches and can expose other engagements
`GRAPHITI_LOG_FORMAT`	`json`	Reserved by the current deployment contract; the Graphiti logger does not yet apply it
`NEO4J_CPUS`, `NEO4J_MEMORY`	`4.0`, `4G`	Neo4j container limits; use `neo4j-admin server memory-recommendation --docker` for production sizing
`NEO4J_SHM_SIZE`	`4g`	`/dev/shm` limit; actual use counts toward the container memory limit
`NEO4J_NOFILE`	`65536`	Open-file soft/hard limit, suitable for many indexes and concurrent connections
`NEO4J_HEAP_INITIAL_SIZE`, `NEO4J_HEAP_MAX_SIZE`, `NEO4J_PAGECACHE_SIZE`	`2G`, `2G`, `1G`	JVM heap/page cache sizing; `NEO4J_CPUS`/`NEO4J_MEMORY` only cap the container, the JVM does not reliably size itself to fit inside that cap on its own. Defaults favor heap over page cache — suited to a low write-throughput deployment with occasional wide reads, since query execution/result materialization lives in heap while a small dataset is already comfortably held by 1G of page cache; re-run `neo4j-admin server memory-recommendation --docker` once real data volume is known
`NEO4J_TRANSACTION_MAX`	`1G`	Caps a single transaction's memory (`db.memory.transaction.max`) so one runaway/unbounded query (e.g. a Cartesian product or an unbounded variable-length path before a `LIMIT`) fails cleanly with an out-of-memory Cypher error instead of exhausting the whole heap and taking down every other query on the server
`NEO4J_BOLT_ADVERTISED_ADDRESS`	empty	Set only when Neo4j Browser and the Bolt connector are reverse-proxied on different public domains (e.g. behind Guarder with CORS/cookie-group support for cross-origin bolt access); format `host:port`. Left empty, Neo4j's discovery endpoint advertises whatever `Host` header the request arrived with, which is correct only when both share one domain
`NEO4J_HTTP_ADVERTISED_ADDRESS`	empty	Set only when a reverse proxy in front of Neo4j Browser strips or rewrites the `Host` header, making the dynamic Host-header-based advertised address incorrect; format `host:port`. Leave empty in the common case (proxy forwards `Host` unchanged)

NEO4J_USER, NEO4J_PASSWORD, NEO4J_URI, and NEO4J_DATABASE configure the bundled connection. Neo4j Community Edition supports only its default database; do not configure a separate database name that requires Enterprise multi-database support.

The installer copies examples/neo4j beside the installation as ./neo4j. It contains static, non-.env-tunable settings that don't have a NEO4J_* variable: conf/neo4j.conf and conf/apoc.conf, plus a version-pinned plugins/apoc-*-core.jar. The compose mount is controlled by:

NEO4J_DIR=./neo4j

NEO4J_DIR may point directly to ./examples/neo4j for development. Both conf/ and plugins/ are mounted read-only; the stack still starts on Neo4j's built-in defaults (without APOC) if the directory is absent, since Docker creates an empty one automatically. Do not duplicate any NEO4J_* variable from the table above inside conf/neo4j.conf — the Neo4j Docker entrypoint always strips a matching line from the mounted file and re-appends the environment variable's value, so a duplicated setting in the file would be silently ignored.

The bundled stack currently wires Neo4j only. The Graphiti image contains FalkorDB support, but using it requires a separately configured deployment because the stock compose file does not expose GRAPHITI_GRAPH_BACKEND or FALKORDB_*.

Verification and Troubleshooting

Check service health, queue state, and logs:

docker compose -f docker-compose.yml -f docker-compose-graphiti.yml ps graphiti neo4j
docker compose -f docker-compose.yml -f docker-compose-graphiti.yml logs -f graphiti
curl -fsS http://localhost:8000/healthcheck
curl -fsS http://localhost:8000/queue-size

Neo4j Browser is available at http://localhost:7474; the Graphiti OpenAPI UI is at http://localhost:8000/docs. Both are bound to localhost by the stock compose file.

Common failures:

A missing API key or base URL for the selected preset, a missing YAML file, or a YAML provider mismatch causes the Graphiti container to fail startup validation.
LLM_CLIENT_TYPE=openai rejects local/custom model prefixes; use the custom preset for an OpenAI-compatible local server.
In flowid search mode, requests without a group ID are rejected. PentAGI supplies the flow-derived group ID automatically.
A bounded full queue returns HTTP 429. /queue-size reports waiting, processing, active-group, and dropped counters.
Invalid retry, timeout, or anchor ranges fail startup rather than being silently normalized.

Update .env, docker-compose-graphiti.yml, the Graphiti image, and the graphiti preset directory together. Pulling only a new image can retain older compose defaults and silently change extraction behavior.

Limitations and Security

Graphiti is beta and has no in-app graph explorer.
One provider preset is active for the entire Graphiti deployment; it is not selected per PentAGI agent or flow.
Graphiti extraction, reranking, and embeddings incur billing independently of the model used by the main PentAGI flow.
Search is flow-scoped by default. Cross-flow reuse requires an explicit global-search design and must not be enabled on shared or multi-tenant deployments without additional isolation.
The Graphiti HTTP API has no authentication layer in the bundled service. The stock compose binds it and Neo4j to 127.0.0.1; secure external deployments with network controls and authentication at a trusted reverse proxy.
Agent and tool output may contain credentials and exploitation evidence. Protect Neo4j data, logs, dead letters, diagnostics, and backups accordingly.
If Graphiti is unavailable, PentAGI continues with its primary memory and vector store after logging the failed startup health check. Set GRAPHITI_ENABLED=false to disable the integration explicitly.

GitHub and Google OAuth Integration

OAuth integration with GitHub and Google allows users to authenticate using their existing accounts on these platforms. This provides several benefits:

Simplified login process without need to create separate credentials
Enhanced security through trusted identity providers
Access to user profile information from GitHub/Google accounts
Seamless integration with existing development workflows

PentAGI uses PUBLIC_URL as the public origin/base URL for OAuth redirects. In the default deployment, both GitHub and Google callbacks are handled by:

${PUBLIC_URL}/api/v1/auth/login-callback

For GitHub OAuth:

Create a new OAuth App in your GitHub account.
Set Homepage URL to your PUBLIC_URL.
Set Authorization callback URL to ${PUBLIC_URL}/api/v1/auth/login-callback.
Add the client credentials to your .env file:

PUBLIC_URL=https://pentagi.example.com
OAUTH_GITHUB_CLIENT_ID=your_github_client_id
OAUTH_GITHUB_CLIENT_SECRET=your_github_client_secret

For Google OAuth:

Create OAuth credentials in your Google Cloud project.
Use the same callback endpoint: ${PUBLIC_URL}/api/v1/auth/login-callback.
Add the client credentials to your .env file:

PUBLIC_URL=https://pentagi.example.com
OAUTH_GOOGLE_CLIENT_ID=your_google_client_id
OAUTH_GOOGLE_CLIENT_SECRET=your_google_client_secret

Make sure PUBLIC_URL matches the externally accessible HTTPS address of your PentAGI instance and does not include the callback path itself. If the URL configured in the OAuth provider does not exactly match the callback generated by PentAGI, the provider will reject the login attempt with a redirect URI mismatch error.

Docker Image Configuration

PentAGI allows you to configure Docker image selection for executing various tasks. The system automatically chooses the most appropriate image based on the task type, but you can constrain this selection by specifying your preferred images:

Variable	Default	Description
`PENTAGI_IMAGE`	`vxcontrol/pentagi:latest`	Docker image used for the main PentAGI application service
`DOCKER_DEFAULT_IMAGE`	`debian:latest`	Default Docker image for general tasks and ambiguous cases
`DOCKER_DEFAULT_IMAGE_FOR_PENTEST`	`vxcontrol/kali-linux`	Default Docker image for security/penetration testing tasks

PENTAGI_IMAGE changes the image used by the main pentagi service in docker-compose.yml. The DOCKER_DEFAULT_IMAGE and DOCKER_DEFAULT_IMAGE_FOR_PENTEST variables only affect automatic worker image selection for task execution inside PentAGI. They do not rewrite the rest of the Compose stack, so services such as pgvector, scraper, and the optional graphiti stack still use the image references defined in the compose files.

When DOCKER_DEFAULT_IMAGE and DOCKER_DEFAULT_IMAGE_FOR_PENTEST are set, AI agents will be limited to the image choices you specify. This is particularly useful for:

Security Enforcement: Restricting usage to only verified and trusted images
Environment Standardization: Using corporate or customized images across all operations
Performance Optimization: Utilizing pre-built images with necessary tools already installed

Configuration examples:

# Using a custom PentAGI application image
PENTAGI_IMAGE=registry.example.com/security/pentagi:latest

# Using a custom image for general tasks
DOCKER_DEFAULT_IMAGE=mycompany/custom-debian:latest

# Using a specialized image for penetration testing
DOCKER_DEFAULT_IMAGE_FOR_PENTEST=mycompany/pentest-tools:v2.0

Note

If a user explicitly specifies a particular Docker image in their task, the system will try to use that exact image, ignoring these settings. These variables only affect the system's automatic image selection process.

For an advanced OpenVAS/GVM experiment that uses a custom pentest image, see OpenVAS via a Custom Pentest Image.

Restricted Networks, Docker Mirrors, and Proxies

If your environment cannot reach Docker Hub (docker.io) directly, changing PentAGI environment variables is usually not enough to fix image download failures. PentAGI still relies on Docker's own registry access for Compose-managed services, and the installer network checks also validate Docker Hub reachability.

For restricted networks:

Confirm that the host can resolve and reach docker.io.
If your environment requires an outbound proxy for PentAGI or installer HTTP traffic, set the PROXY_URL environment variable. To route Docker image pulls through a proxy, configure the Docker daemon or Docker Desktop proxy separately — Docker does not use PentAGI's PROXY_URL for registry access.
If Docker Hub is blocked or heavily rate-limited, configure an organization-approved registry mirror or registry proxy before running the installer or docker compose up.
Restart Docker after changing the daemon configuration, then rerun the installer checks or Compose startup.

Example Docker daemon mirror configuration:

{
  "registry-mirrors": ["https://mirror.example.com"]
}

On Linux, this is typically configured in /etc/docker/daemon.json. On Docker Desktop, use the equivalent Docker Engine or proxy settings. A Docker Hub mirror covers Docker Hub-hosted images such as vxcontrol/*, but the main Compose stack already includes quay.io/prometheuscommunity/postgres-exporter, and the optional observability stack includes gcr.io/cadvisor/cadvisor. Those registries still need direct access or individually approved proxy/mirror paths.

See the official Docker documentation for registry mirrors and daemon proxy configuration.

Troubleshooting: "failed to select primary docker image via llm call"

A flow that fails immediately with failed to select primary docker image via llm call usually indicates a problem with the configured LLM backend, not with Docker or the image registry. Older PentAGI versions reported the same failure as failed to get primary docker image, which led users to debug Docker even though the registry was healthy.

When a flow starts, PentAGI makes its first LLM call to choose the primary Docker image for the task. This image-selection call runs through the simple agent type, so a failure here points at the model assigned to that agent type rather than at Docker. A message such as API returned unexpected status code: 502 or 404 in this context is returned by the LLM backend, not by Docker Hub.

This is distinct from the registry reachability problems described above: if Docker pulls succeed and the Compose stack starts, but flow creation still fails at image selection, investigate the LLM backend rather than Docker.

To diagnose:

Check PentAGI logs first: docker logs pentagi.
Check the logs of your configured LLM backend (the server behind your provider or LLM_SERVER_URL).
Verify that the base URL, API key, and model name in Custom LLM Provider Configuration are correct and reachable from the container. If you assign different models per agent type, check the model used by the simple agent type, since image selection runs through it.
For custom, OpenAI-compatible, vLLM, or SGLang backends, confirm that the model supports tool calling (function calling) and that the matching tool-call parser is enabled. A missing or mismatched tool-call parser is a known cause of this failure.

Development

Development Requirements

golang
nodejs
docker
postgres

Environment Setup

Backend Setup

Run once cd backend && go mod download to install needed packages.

For generating swagger files have to run

swag init -g ../../pkg/server/router.go -o pkg/server/docs/ --parseDependency --parseInternal --parseDepth 2 -d cmd/pentagi

before installing swag package via

go install github.com/swaggo/swag/cmd/swag@v1.8.7

For generating graphql resolver files have to run

go run github.com/99designs/gqlgen --config ./gqlgen/gqlgen.yml

after that you can see the generated files in pkg/graph folder.

For generating ORM methods (database package) from sqlc configuration

docker run --rm -v $(pwd):/src -w /src --network pentagi-network -e DATABASE_URL="{URL}" sqlc/sqlc:1.27.0 generate -f sqlc/sqlc.yml

For generating Langfuse SDK from OpenAPI specification

fern generate --local

and to install fern-cli

pnpm add -g fern-api

Testing

For running tests cd backend && go test -v ./...

Frontend Setup

Run once cd frontend && pnpm install to install needed packages.

For generating graphql files have to run pnpm run graphql:generate which using graphql-codegen.ts file.

Be sure that you have graphql-codegen installed globally:

pnpm add -g graphql-codegen

After that you can run:

pnpm run prettier to check if your code is formatted correctly
pnpm run prettier:fix to fix it
pnpm run lint to check if your code is linted correctly
pnpm run lint:fix to fix it

For generating SSL certificates you need to run pnpm run ssl:generate which using generate-ssl.ts file or it will be generated automatically when you run pnpm run dev.

Backend Configuration

Edit the configuration for backend in .vscode/launch.json file:

DATABASE_URL - PostgreSQL database URL (eg. postgres://postgres:postgres@localhost:5432/pentagidb?sslmode=disable)
DOCKER_HOST - Docker SDK API (eg. for macOS DOCKER_HOST=unix:///Users/<my-user>/Library/Containers/com.docker.docker/Data/docker.raw.sock) more info

Optional:

SERVER_PORT - Port to run the server (default: 8443)
SERVER_USE_SSL - Enable SSL for the server (default: false)

PostgreSQL / pgvector connection pool sizing

PentAGI opens two independent connection pools to the same Postgres instance:

Pool	Env var	Default	Used by
Shared `sql.DB`	`DATABASE_MAX_OPEN_CONNS`	`25`	All sqlc queries and GORM handlers share a single `*sql.DB`
Shared `pgxpool`	`DATABASE_VECTOR_MAX_CONNS`	`10`	All pgvector stores (agent memory + knowledge API) share a single pool

Additional tuning knob:

DATABASE_MAX_IDLE_CONNS — maximum idle connections kept open in the sql.DB pool between requests (default: 5).

Budget for the stock vxcontrol/pgvector image (max_connections = 100, superuser_reserved_connections = 3):

Available for client connections  = 97
  pentagi sql.DB  (DATABASE_MAX_OPEN_CONNS)   = 25
  pentagi pgxpool (DATABASE_VECTOR_MAX_CONNS) = 10
  pgexporter                                  =  3
  autovacuum workers                          =  3
  ─────────────────────────────────────────
  Total consumed                              = 41
  Free buffer                                 = 56  (≈ 58 %)

The defaults are sized for 10 parallel flows with concurrent API requests. If you run more flows or deploy multiple PentAGI instances against the same Postgres, raise max_connections via the command override in docker-compose.yml and increase the pool sizes proportionally:

pgvector:
  image: vxcontrol/pgvector:latest
  command: postgres -c max_connections=200

To inspect the live connection budget on a running deployment:

# Postgres limits
docker exec pgvector sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c \
  "SELECT name, setting FROM pg_settings
   WHERE name IN ('"'"'max_connections'"'"', '"'"'superuser_reserved_connections'"'"');"'

# Current usage vs. available
docker exec pgvector sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c \
  "SELECT max_conn, used, max_conn - used AS available
   FROM (SELECT current_setting('"'"'max_connections'"'"')::int AS max_conn,
                count(*) AS used FROM pg_stat_activity) t;"'

# Breakdown by client
docker exec pgvector sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c \
  "SELECT application_name, client_addr, state, count(*)
   FROM pg_stat_activity
   WHERE pid <> pg_backend_pid()
   GROUP BY 1, 2, 3 ORDER BY count DESC;"'

External PostgreSQL and schema handling

DATABASE_URL may point at any PostgreSQL instance, not only the bundled pgvector container. Two extra knobs apply when — and only when — TENANT_ID is set, because that is when PentAGI creates its own schema and rewrites the connection's search_path:

Env var	Default	Purpose
`DATABASE_EXTENSIONS_SCHEMA`	`public`	Schema holding the shared `vector` and `pg_trgm` extensions that every tenant's `search_path` must reach
`DATABASE_SEARCH_PATH_VIA_OPTIONS`	`false`	Send the tenant `search_path` inside the `options` startup parameter instead of as a bare connection parameter

Supabase (cloud or self-hosted) needs both of them considered, and is the reason they exist:

Supabase installs its bundled extensions into an extensions schema instead of public, so set DATABASE_EXTENSIONS_SCHEMA=extensions. Without it, startup aborts with an error naming the schema where vector was actually found — no need to move a provider-managed extension with ALTER EXTENSION.
Supabase's pooler (Supavisor) does not reliably forward a bare search_path connection parameter. Prefer a direct PostgreSQL connection: self-hosted, expose the db service port and bypass the supavisor service; cloud, use the "Direct connection" string (or the IPv4 add-on on IPv4-only networks). If the pooler cannot be bypassed, use its session mode and try DATABASE_SEARCH_PATH_VIA_OPTIONS=true — PentAGI verifies the effective schema on boot and refuses to start if it did not take effect, so a silent cross-tenant data mix-up is not possible.

Both settings are managed by the installer under Server Settings, next to TENANT_ID — see Running Several Instances for that scenario, and Multi-Instance Deployment for the full matrix, including the PgBouncer recipe (pool_mode = session, ignore_startup_parameters, per-tenant connect_query).

Frontend Configuration

Edit the configuration for frontend in .vscode/launch.json file:

VITE_API_URL - Backend API URL. Omit the URL scheme (e.g., localhost:8080 NOT http://localhost:8080)
VITE_USE_HTTPS - Enable SSL for the server (default: false)
VITE_PORT - Port to run the server (default: 8000)
VITE_HOST - Host to run the server (default: 0.0.0.0)

Running the Application

Backend

Run the command(s) in backend folder:

Use .env file to set environment variables like a source .env
Run go run cmd/pentagi/main.go to start the server

Note

The first run can take a while as dependencies and docker images need to be downloaded to setup the backend environment.

Frontend

Run the command(s) in frontend folder:

Run pnpm install to install the dependencies
Run pnpm run dev to run the web app
Run pnpm run build to build the web app

Open your browser and visit the web app URL.

Testing LLM Agents

PentAGI includes a powerful utility called ctester for testing and validating LLM agent capabilities. This tool helps ensure your LLM provider configurations work correctly with different agent types, allowing you to optimize model selection for each specific agent role.

The utility features parallel testing of multiple agents, detailed reporting, and flexible configuration options.

Key Features

Parallel Testing: Tests multiple agents simultaneously for faster results
Comprehensive Test Suite: Evaluates basic completion, JSON responses, function calling, and penetration testing knowledge
Detailed Reporting: Generates markdown reports with success rates and performance metrics
Flexible Configuration: Test specific agents or test groups as needed
Specialized Test Groups: Includes domain-specific tests for cybersecurity and penetration testing scenarios

Usage Scenarios

For Developers (with local Go environment)

If you've cloned the repository and have Go installed:

# Default configuration with .env file
cd backend
go run cmd/ctester/*.go -verbose

# Custom provider configuration
go run cmd/ctester/*.go -config ../examples/configs/openrouter.provider.yml -verbose

# Generate a report file
go run cmd/ctester/*.go -config ../examples/configs/deepinfra.provider.yml -report ../test-report.md

# Test specific agent types only
go run cmd/ctester/*.go -agents simple,simple_json,primary_agent -verbose

# Test specific test groups only
go run cmd/ctester/*.go -groups basic,advanced -verbose

For Users (using Docker image)

If you prefer to use the pre-built Docker image without setting up a development environment:

# Using Docker to test with default environment
docker run --rm -v $(pwd)/.env:/opt/pentagi/.env vxcontrol/pentagi /opt/pentagi/bin/ctester -verbose

# Test with your custom provider configuration
docker run --rm \
  -v $(pwd)/.env:/opt/pentagi/.env \
  -v $(pwd)/my-config.yml:/opt/pentagi/config.yml \
  vxcontrol/pentagi /opt/pentagi/bin/ctester -config /opt/pentagi/config.yml -agents simple,primary_agent,coder -verbose

# Generate a detailed report
docker run --rm \
  -v $(pwd)/.env:/opt/pentagi/.env \
  -v $(pwd):/opt/pentagi/output \
  vxcontrol/pentagi /opt/pentagi/bin/ctester -report /opt/pentagi/output/report.md

Using Pre-configured Providers

The Docker image comes with built-in support for major providers (OpenAI, Anthropic, Gemini, Ollama) and pre-configured provider files for additional services (OpenRouter, OpenCode, Atlas, OrcaRouter, DeepInfra, DeepSeek, Moonshot, Novita, xAI):

# Test with OpenRouter configuration
docker exec -it pentagi /opt/pentagi/bin/ctester -config /opt/pentagi/conf/openrouter.provider.yml

# Test with OpenCode Go plan configuration
docker exec -it pentagi /opt/pentagi/bin/ctester -config /opt/pentagi/conf/opencode.provider.yml

# Test with DeepInfra configuration
docker exec -it pentagi /opt/pentagi/bin/ctester -config /opt/pentagi/conf/deepinfra.provider.yml

# Test with DeepSeek configuration
docker exec -it pentagi /opt/pentagi/bin/ctester -provider deepseek

# Test with GLM configuration
docker exec -it pentagi /opt/pentagi/bin/ctester -provider glm

# Test with Kimi configuration
docker exec -it pentagi /opt/pentagi/bin/ctester -provider kimi

# Test with Qwen configuration
docker exec -it pentagi /opt/pentagi/bin/ctester -provider qwen

# Test with DeepSeek configuration file for custom provider
docker exec -it pentagi /opt/pentagi/bin/ctester -config /opt/pentagi/conf/deepseek.provider.yml

# Test with Moonshot configuration file for custom provider
docker exec -it pentagi /opt/pentagi/bin/ctester -config /opt/pentagi/conf/moonshot.provider.yml

# Test with Novita configuration
docker exec -it pentagi /opt/pentagi/bin/ctester -config /opt/pentagi/conf/novita.provider.yml

# Test with xAI configuration
docker exec -it pentagi /opt/pentagi/bin/ctester -config /opt/pentagi/conf/xai.provider.yml

# Test with OpenAI configuration
docker exec -it pentagi /opt/pentagi/bin/ctester -type openai

# Test with Anthropic configuration
docker exec -it pentagi /opt/pentagi/bin/ctester -type anthropic

# Test with Gemini configuration
docker exec -it pentagi /opt/pentagi/bin/ctester -type gemini

# Test with AWS Bedrock configuration
docker exec -it pentagi /opt/pentagi/bin/ctester -type bedrock

# Test with Custom OpenAI configuration
docker exec -it pentagi /opt/pentagi/bin/ctester -config /opt/pentagi/conf/custom-openai.provider.yml

# Test with Ollama configuration (local inference)
docker exec -it pentagi /opt/pentagi/bin/ctester -config /opt/pentagi/conf/ollama-llama318b.provider.yml

# Test with Ollama Qwen3 32B configuration (requires custom model creation)
docker exec -it pentagi /opt/pentagi/bin/ctester -config /opt/pentagi/conf/ollama-qwen332b-fp16-tc.provider.yml

# Test with Ollama QwQ 32B configuration (requires custom model creation and 71.3GB VRAM)
docker exec -it pentagi /opt/pentagi/bin/ctester -config /opt/pentagi/conf/ollama-qwq32b-fp16-tc.provider.yml

To use these configurations, your .env file only needs to contain:

LLM_SERVER_URL=https://openrouter.ai/api/v1      # or https://api.deepinfra.com/v1/openai or https://api.openai.com/v1 or https://opencode.ai/zen/go/v1 or https://api.novita.ai/openai or https://api.atlascloud.ai/v1 or https://api.orcarouter.ai/v1 or https://api.x.ai/v1
LLM_SERVER_KEY=your_api_key
LLM_SERVER_MODEL=                                # Leave empty, as models are specified in the config
LLM_SERVER_CONFIG_PATH=/opt/pentagi/conf/openrouter.provider.yml  # or deepinfra.provider.ymll or opencode.provider.ymll or custom-openai.provider.yml or novita.provider.yml or atlas.provider.yml or orcarouter.provider.yml or xai.provider.yml
LLM_SERVER_PROVIDER=                             # Provider name for LiteLLM proxy (e.g., openrouter, deepseek, moonshot, novita, opencode, orcarouter, xai)
LLM_SERVER_LEGACY_REASONING=false                # Controls reasoning format, for OpenAI must be true (default: false)
LLM_SERVER_PRESERVE_REASONING=false              # Preserve reasoning content in multi-turn conversations (required by Moonshot, default: false)

# For OpenAI (official API)
OPEN_AI_KEY=your_openai_api_key                  # Your OpenAI API key
OPEN_AI_SERVER_URL=https://api.openai.com/v1     # OpenAI API endpoint

# For Anthropic (Claude models)
ANTHROPIC_API_KEY=your_anthropic_api_key         # Your Anthropic API key
ANTHROPIC_SERVER_URL=https://api.anthropic.com/v1  # Anthropic API endpoint

# For Gemini (Google AI)
GEMINI_API_KEY=your_gemini_api_key               # Your Google AI API key
GEMINI_SERVER_URL=https://generativelanguage.googleapis.com  # Google AI API endpoint

# For AWS Bedrock (enterprise foundation models)
BEDROCK_REGION=us-east-1                         # AWS region for Bedrock service
# Authentication (choose one method, priority: DefaultAuth > BearerToken > AccessKey):
BEDROCK_DEFAULT_AUTH=false                       # Use AWS SDK credential chain (env vars, EC2 role, ~/.aws/credentials)
BEDROCK_BEARER_TOKEN=                            # Bearer token authentication (takes priority over static credentials)
BEDROCK_ACCESS_KEY_ID=your_aws_access_key        # AWS access key ID (static credentials)
BEDROCK_SECRET_ACCESS_KEY=your_aws_secret_key    # AWS secret access key (static credentials)
BEDROCK_SESSION_TOKEN=                           # AWS session token (optional, for temporary credentials with static auth)
BEDROCK_SERVER_URL=                              # Optional custom Bedrock endpoint (VPC endpoints, local testing)
BEDROCK_CONFIG_PATH=                             # Optional path to a custom YAML provider config (overrides built-in model/pricing definitions)

# For Ollama (local server or cloud)
OLLAMA_SERVER_URL=                               # Local: http://ollama-server:11434, Cloud: https://ollama.com
OLLAMA_SERVER_API_KEY=                           # Required for Ollama Cloud (https://ollama.com/settings/keys), leave empty for local
OLLAMA_SERVER_MODEL=
OLLAMA_SERVER_CONFIG_PATH=
OLLAMA_SERVER_PULL_MODELS_TIMEOUT=
OLLAMA_SERVER_PULL_MODELS_ENABLED=
OLLAMA_SERVER_LOAD_MODELS_ENABLED=

# For DeepSeek (Chinese AI with strong reasoning)
DEEPSEEK_API_KEY=                                # DeepSeek API key
DEEPSEEK_SERVER_URL=https://api.deepseek.com     # DeepSeek API endpoint
DEEPSEEK_PROVIDER=                               # Optional: LiteLLM prefix (e.g., 'deepseek')

# For GLM (Zhipu AI)
GLM_API_KEY=                                     # GLM API key
GLM_SERVER_URL=https://api.z.ai/api/paas/v4      # GLM API endpoint (international)
GLM_PROVIDER=                                    # Optional: LiteLLM prefix (e.g., 'zai')

# For Kimi (Moonshot AI)
KIMI_API_KEY=                                    # Kimi API key
KIMI_SERVER_URL=https://api.moonshot.ai/v1       # Kimi API endpoint (international)
KIMI_PROVIDER=                                   # Optional: LiteLLM prefix (e.g., 'moonshot')

# For Qwen (Alibaba Cloud DashScope)
QWEN_API_KEY=                                    # Qwen API key
QWEN_SERVER_URL=https://dashscope-us.aliyuncs.com/compatible-mode/v1  # Qwen API endpoint (US)
QWEN_PROVIDER=                                   # Optional: LiteLLM prefix (e.g., 'dashscope')

# For Ollama (local inference) use variables above
OLLAMA_SERVER_URL=http://localhost:11434
OLLAMA_SERVER_MODEL=llama3.1:8b-instruct-q8_0
OLLAMA_SERVER_CONFIG_PATH=/opt/pentagi/conf/ollama-llama318b.provider.yml
OLLAMA_SERVER_PULL_MODELS_ENABLED=false
OLLAMA_SERVER_LOAD_MODELS_ENABLED=false

Using OpenAI with Unverified Organizations

For OpenAI accounts with unverified organizations that don't have access to the latest reasoning models (o1, o3, o4-mini), you need to use a custom configuration.

To use OpenAI with unverified organization accounts, configure your .env file as follows:

LLM_SERVER_URL=https://api.openai.com/v1
LLM_SERVER_KEY=your_openai_api_key
LLM_SERVER_MODEL=                                # Leave empty, models are specified in config
LLM_SERVER_CONFIG_PATH=/opt/pentagi/conf/custom-openai.provider.yml
LLM_SERVER_LEGACY_REASONING=true                 # Required for OpenAI reasoning format

This configuration uses the pre-built custom-openai.provider.yml file that maps all agent types to models available for unverified organizations, using o3-mini instead of models like o1, o3, and o4-mini.

You can test this configuration using:

# Test with custom OpenAI configuration for unverified accounts
docker exec -it pentagi /opt/pentagi/bin/ctester -config /opt/pentagi/conf/custom-openai.provider.yml

Note

The LLM_SERVER_LEGACY_REASONING=true setting is crucial for OpenAI compatibility as it ensures reasoning parameters are sent in the format expected by OpenAI's API.

Using LiteLLM Proxy

When using LiteLLM proxy to access various LLM providers, model names are prefixed with the provider name (e.g., moonshot/kimi-2.5 instead of kimi-2.5). To use the same provider configuration files with both direct API access and LiteLLM proxy, set the LLM_SERVER_PROVIDER variable:

# Direct access to Moonshot API
LLM_SERVER_URL=https://api.moonshot.ai/v1
LLM_SERVER_KEY=your_moonshot_api_key
LLM_SERVER_CONFIG_PATH=/opt/pentagi/conf/moonshot.provider.yml
LLM_SERVER_PROVIDER=                             # Empty for direct access

# Access via LiteLLM proxy
LLM_SERVER_URL=http://litellm-proxy:4000
LLM_SERVER_KEY=your_litellm_api_key
LLM_SERVER_CONFIG_PATH=/opt/pentagi/conf/moonshot.provider.yml
LLM_SERVER_PROVIDER=moonshot                     # Provider prefix for LiteLLM

With LLM_SERVER_PROVIDER=moonshot, the system automatically prefixes all model names from the configuration file with moonshot/, making them compatible with LiteLLM's model naming convention.

LiteLLM Provider Name Mapping:

When using LiteLLM proxy, set the corresponding *_PROVIDER variable to enable model prefixing:

deepseek - for DeepSeek models (DEEPSEEK_PROVIDER=deepseek → deepseek/deepseek-v4-flash)
zai - for GLM models (GLM_PROVIDER=zai → zai/glm-4)
moonshot - for Kimi models (KIMI_PROVIDER=moonshot → moonshot/kimi-k2.5)
dashscope - for Qwen models (QWEN_PROVIDER=dashscope → dashscope/qwen-plus)
openai, anthropic, gemini - for major cloud providers
opencode - for OpenCode Go plan
openrouter - for OpenRouter aggregator
orcarouter - for OrcaRouter aggregator
deepinfra - for DeepInfra hosting
novita - for Novita AI
xai - for xAI (grok-* models)
Any other provider name configured in your LiteLLM instance

Example with LiteLLM:

# Use DeepSeek models via LiteLLM proxy with model prefixing
DEEPSEEK_API_KEY=your_litellm_proxy_key
DEEPSEEK_SERVER_URL=http://litellm-proxy:4000
DEEPSEEK_PROVIDER=deepseek  # Models become deepseek/deepseek-v4-flash, deepseek/deepseek-v4-pro for LiteLLM

# Direct DeepSeek API usage (no prefix needed)
DEEPSEEK_API_KEY=your_deepseek_api_key
DEEPSEEK_SERVER_URL=https://api.deepseek.com
# Leave DEEPSEEK_PROVIDER empty

This approach allows you to:

Use the same configuration files for both direct and proxied access
Switch between providers without modifying configuration files
Easily test different routing strategies with LiteLLM

Running Tests in a Production Environment

If you already have a running PentAGI container and want to test the current configuration:

# Run ctester in an existing container using current environment variables
docker exec -it pentagi /opt/pentagi/bin/ctester -verbose

# Test specific agent types with deterministic ordering
docker exec -it pentagi /opt/pentagi/bin/ctester -agents simple,primary_agent,pentester -groups basic,knowledge -verbose

# Generate a report file inside the container
docker exec -it pentagi /opt/pentagi/bin/ctester -report /opt/pentagi/data/agent-test-report.md

# Access the report from the host
docker cp pentagi:/opt/pentagi/data/agent-test-report.md ./

Command-line Options

The utility accepts several options:

-env <path> - Path to environment file (default: .env)
-type <provider> - Provider type: custom, openai, anthropic, ollama, bedrock, gemini (default: custom)
-config <path> - Path to custom provider config (default: from LLM_SERVER_CONFIG_PATH env variable)
-tests <path> - Path to custom tests YAML file (optional)
-report <path> - Path to write the report file (optional)
-agents <list> - Comma-separated list of agent types to test (default: all)
-groups <list> - Comma-separated list of test groups to run (default: all)
-verbose - Enable verbose output with detailed test results for each agent

Available Agent Types

Agents are tested in the following deterministic order:

simple - Basic completion tasks
simple_json - JSON-structured responses
primary_agent - Main reasoning agent
assistant - Interactive assistant mode
generator - Content generation
refiner - Content refinement and improvement
adviser - Expert advice and consultation
reflector - Self-reflection and analysis
searcher - Information gathering and search
enricher - Data enrichment and expansion
coder - Code generation and analysis
installer - Installation and setup tasks
pentester - Penetration testing and security assessment

Available Test Groups

basic - Fundamental completion and prompt response tests
advanced - Complex reasoning and function calling tests
json - JSON format validation and structure tests (specifically designed for simple_json agent)
knowledge - Domain-specific cybersecurity and penetration testing knowledge tests

Note: The json test group is specifically designed for the simple_json agent type, while all other agents are tested with basic, advanced, and knowledge groups. This specialization ensures optimal testing coverage for each agent's intended purpose.

Example Provider Configuration

Provider configuration defines which models to use for different agent types:

simple:
  model: "provider/model-name"
  temperature: 0.7
  top_p: 0.95
  n: 1
  max_tokens: 4000

simple_json:
  model: "provider/model-name"
  temperature: 0.7
  top_p: 1.0
  n: 1
  max_tokens: 4000
  json: true

# ... other agent types ...

Optimization Workflow

Create a baseline: Run tests with default configuration to establish benchmark performance
Analyze agent-specific performance: Review the deterministic agent ordering to identify underperforming agents
Test specialized configurations: Experiment with different models for each agent type using provider-specific configs
Focus on domain knowledge: Pay special attention to knowledge group tests for cybersecurity expertise
Validate function calling: Ensure tool-based tests pass consistently for critical agent types
Compare results: Look for the best success rate and performance across all test groups
Deploy optimal configuration: Use in production with your optimized setup

This tool helps ensure your AI agents are using the most effective models for their specific tasks, improving reliability while optimizing costs.

Embedding Configuration and Testing

PentAGI uses vector embeddings for semantic search, knowledge storage, and memory management. The system supports multiple embedding providers that can be configured according to your needs and preferences.

Supported Embedding Providers

PentAGI supports the following embedding providers:

OpenAI (default): Uses OpenAI's text embedding models
Ollama: Local embedding model through Ollama
Mistral: Mistral AI's embedding models
Jina: Jina AI's embedding service
HuggingFace: Models from HuggingFace
GoogleAI: Google's embedding models
VoyageAI: VoyageAI's embedding models

OpenAI-compatible third parties: any provider exposing OpenAI's /embeddings API can be plugged in via EMBEDDING_PROVIDER=openai with a custom EMBEDDING_URL. For example, Qwen DashScope offers text-embedding-v4 through the /compatible-mode/v1 endpoint (International and Chinese Mainland regions only — the US region does not expose embeddings). See the Qwen Alternative Integrations subsection for the full configuration snippet.

Embedding Provider Configuration (click to expand)

Environment Variables

To configure the embedding provider, set the following environment variables in your .env file:

# Primary embedding configuration
EMBEDDING_PROVIDER=openai       # Provider type (openai, ollama, mistral, jina, huggingface, googleai, voyageai)
EMBEDDING_MODEL=text-embedding-3-small  # Model name to use
EMBEDDING_URL=                  # Optional custom API endpoint
EMBEDDING_KEY=                  # API key for the provider (if required)
EMBEDDING_BATCH_SIZE=100        # Number of documents to process in a batch
EMBEDDING_STRIP_NEW_LINES=true  # Whether to remove new lines from text before embedding
EMBEDDING_MAX_TEXT_BYTES=8192   # Max bytes of text sent to embedding model per document (byte proxy for token limit)

# Advanced settings
PROXY_URL=                      # Optional proxy for all API calls
HTTP_CLIENT_TIMEOUT=600         # Timeout in seconds for external API calls (default: 600, 0 = no timeout)
TERMINAL_TOOL_TIMEOUT=1200      # Default timeout in seconds for terminal tool commands when timeout=0 or negative (range: 1–10800; values <= 0 or above 10800 are clamped to 10800 = 3 hours)

# SSL/TLS Certificate Configuration (for external communication with LLM backends and tool servers)
EXTERNAL_SSL_CA_PATH=           # Path to custom CA certificate file (PEM format) inside the container
                                # Must point to /opt/pentagi/ssl/ directory (e.g., /opt/pentagi/ssl/ca-bundle.pem)
EXTERNAL_SSL_INSECURE=false     # Skip certificate verification (use only for testing)

How to Add Custom CA Certificates (click to expand)

If you see this error: tls: failed to verify certificate: x509: certificate signed by unknown authority

Step 1: Get your CA certificate bundle in PEM format (can contain multiple certificates)

Step 2: Place the file in the SSL directory on your host machine:

# Default location (if PENTAGI_SSL_DIR is not set)
cp ca-bundle.pem ./pentagi-ssl/

# Or custom location (if using PENTAGI_SSL_DIR in docker-compose.yml)
cp ca-bundle.pem /path/to/your/ssl/dir/

Step 3: Set the path in .env file (path must be inside the container):

# The volume pentagi-ssl is mounted to /opt/pentagi/ssl inside the container
EXTERNAL_SSL_CA_PATH=/opt/pentagi/ssl/ca-bundle.pem
EXTERNAL_SSL_INSECURE=false

Step 4: Restart PentAGI:

docker compose restart pentagi

Notes:

The pentagi-ssl volume is mounted to /opt/pentagi/ssl inside the container
You can change host directory using PENTAGI_SSL_DIR variable in docker-compose.yml
File supports multiple certificates and intermediate CAs in one PEM file
Use EXTERNAL_SSL_INSECURE=true only for testing (not recommended for production)

Provider-Specific Limitations

Each provider has specific limitations and supported features:

OpenAI: Supports all configuration options
Ollama: Does not support EMBEDDING_KEY as it uses local models
Mistral: Does not support EMBEDDING_MODEL or custom HTTP client
Jina: Does not support custom HTTP client
HuggingFace: Requires EMBEDDING_KEY and supports all other options
GoogleAI: Does not support EMBEDDING_URL, requires EMBEDDING_KEY
VoyageAI: Supports all configuration options

If EMBEDDING_URL and EMBEDDING_KEY are not specified, the system will attempt to use the corresponding LLM provider settings (e.g., OPEN_AI_KEY when EMBEDDING_PROVIDER=openai).

Why Consistent Embedding Providers Matter

It's crucial to use the same embedding provider consistently because:

Vector Compatibility: Different providers produce vectors with different dimensions and mathematical properties
Semantic Consistency: Changing providers can break semantic similarity between previously embedded documents
Memory Corruption: Mixed embeddings can lead to poor search results and broken knowledge base functionality

If you change your embedding provider, you should flush and reindex your entire knowledge base (see etester utility below).

Embedding Tester Utility (etester)

PentAGI includes a specialized etester utility for testing, managing, and debugging embedding functionality. This tool is essential for diagnosing and resolving issues related to vector embeddings and knowledge storage.

Etester Commands (click to expand)

# Test embedding provider and database connection
cd backend
go run cmd/etester/main.go test -verbose

# Show statistics about the embedding database
go run cmd/etester/main.go info

# Delete all documents from the embedding database (use with caution!)
go run cmd/etester/main.go flush

# Recalculate embeddings for all documents (after changing provider)
go run cmd/etester/main.go reindex

# Search for documents in the embedding database
go run cmd/etester/main.go search -query "How to install PostgreSQL" -limit 5

Using Docker

If you're running PentAGI in Docker, you can use etester from within the container:

# Test embedding provider
docker exec -it pentagi /opt/pentagi/bin/etester test

# Show detailed database information
docker exec -it pentagi /opt/pentagi/bin/etester info -verbose

Advanced Search Options

The search command supports various filters to narrow down results:

# Filter by document type
docker exec -it pentagi /opt/pentagi/bin/etester search -query "Security vulnerability" -doc_type guide -threshold 0.8

# Filter by flow ID
docker exec -it pentagi /opt/pentagi/bin/etester search -query "Code examples" -doc_type code -flow_id 42

# All available search options
docker exec -it pentagi /opt/pentagi/bin/etester search -help

Available search parameters:

-query STRING: Search query text (required)
-doc_type STRING: Filter by document type (answer, memory, guide, code)
-flow_id NUMBER: Filter by flow ID (positive number)
-answer_type STRING: Filter by answer type (guide, vulnerability, code, tool, other)
-guide_type STRING: Filter by guide type (install, configure, use, pentest, development, other)
-limit NUMBER: Maximum number of results (default: 3)
-threshold NUMBER: Similarity threshold (0.0-1.0, default: 0.7)

Memory Lifecycle Across Flows

PentAGI stores several kinds of vector documents, and they serve different purposes:

memory captures flow-specific execution history such as tool results and agent observations
guide, answer, and code are intended for reusable knowledge that can help future runs

If you want to inspect what happened in one engagement, search the vector store with the related flow_id. If you want knowledge to survive beyond a single run, store the durable result explicitly as a guide, answer, or code document instead of relying on execution memory alone.

For example, if a target has recurring setup notes, authentication quirks, or target-specific testing methodology, instruct the agent to save that information as a guide and search for it at the beginning of the next engagement. This is the safest current workflow when you want a new flow to start with reusable context.

Flow deletion removes the flow from normal queries through PentAGI's soft-delete mechanism, so reusable knowledge should be treated as a separate concern from per-flow execution history. If you enable the optional Graphiti knowledge graph described earlier in this README, treat its current search context as scoped to the active flow or engagement unless you explicitly build a separate cross-flow reuse workflow.

Common Troubleshooting Scenarios

After changing embedding provider: Always run flush or reindex to ensure consistency
Poor search results: Try adjusting the similarity threshold or check if embeddings are correctly generated
Database connection issues: Verify PostgreSQL is running with pgvector extension installed
Missing API keys: Check environment variables for your chosen embedding provider

Troubleshooting: Flow Stalls or Hangs Without Progress

If a flow starts but then appears to wait indefinitely with no subtasks progressing, a common cause is an embedding provider that is misconfigured or unreachable. PentAGI uses the embedding provider to store and search vector memory while a flow runs, so embedding calls that fail or hang can leave a flow waiting instead of advancing.

1. Check the container logs first. Embedding errors surface in the PentAGI logs:

docker logs pentagi

Look for embedding-related failures such as authentication errors (401/403), wrong-model or not-found errors (404), connection timeouts, or TLS certificate errors. These point at the embedding provider configuration rather than at the flow itself.

2. Validate the provider with etester. The Embedding Tester Utility (etester) checks both the embedding provider and the database connection without starting a flow:

docker exec -it pentagi /opt/pentagi/bin/etester test -verbose

A failing test confirms the problem is in the embedding configuration rather than in the flow.

3. Verify the configuration. Check the following in your .env file against the Supported Embedding Providers list and each provider's documented limitations:

EMBEDDING_PROVIDER is one of the supported providers (default openai).
EMBEDDING_MODEL is a valid model name for that provider.
EMBEDDING_URL and EMBEDDING_KEY are correct for the provider. If both are left empty, PentAGI falls back to the matching LLM provider settings (for example OPEN_AI_KEY and OPEN_AI_SERVER_URL when EMBEDDING_PROVIDER=openai), so a missing or wrong key there can break embeddings too.
The endpoint is reachable from inside the container. If outbound calls go through a proxy, confirm PROXY_URL is set; if calls hang rather than fail quickly, HTTP_CLIENT_TIMEOUT controls how long PentAGI waits on the provider before giving up.

Changing provider? If you switch embedding providers after data has already been indexed, run flush or reindex with etester so old and new vectors are not mixed. See Why Consistent Embedding Providers Matter above.

Function Testing with ftester

PentAGI includes a versatile utility called ftester for debugging, testing, and developing specific functions and AI agent behaviors. While ctester focuses on testing LLM model capabilities, ftester allows you to directly invoke individual system functions and AI agent components with precise control over execution context.

Key Features

Direct Function Access: Test individual functions without running the entire system
Mock Mode: Test functions without a live PentAGI deployment using built-in mocks
Interactive Input: Fill function arguments interactively for exploratory testing
Detailed Output: Color-coded terminal output with formatted responses and errors
Context-Aware Testing: Debug AI agents within the context of specific flows, tasks, and subtasks
Observability Integration: All function calls are logged to Langfuse and Observability stack

Usage Modes

Command Line Arguments

Run ftester with specific function and arguments directly from the command line:

# Basic usage with mock mode
cd backend
go run cmd/ftester/main.go [function_name] -[arg1] [value1] -[arg2] [value2]

# Example: Test terminal command in mock mode
go run cmd/ftester/main.go terminal -command "ls -la" -message "List files"

# Using a real flow context
go run cmd/ftester/main.go -flow 123 terminal -command "whoami" -message "Check user"

# Testing AI agent in specific task/subtask context
go run cmd/ftester/main.go -flow 123 -task 456 -subtask 789 pentester -message "Find vulnerabilities"

Interactive Mode

Run ftester without arguments for a guided interactive experience:

# Start interactive mode
go run cmd/ftester/main.go [function_name]

# For example, to interactively fill browser tool arguments
go run cmd/ftester/main.go browser

Available Functions (click to expand)

Environment Functions

terminal: Execute commands in a container and return the output
file: Perform file operations (read, write, list) in a container

Search Functions

browser: Access websites and capture screenshots
web_search: Unified search orchestrator that agents actually call — pass a query and a mode (links, answer, research, exploit) and it auto-selects, retries, and falls back across the engines below, so you never name an engine explicitly
google: Search the web using Google Custom Search
duckduckgo: Search the web using DuckDuckGo
tavily: Search using Tavily AI search engine
firecrawl: Search using Firecrawl with main-content markdown scraping
traversaal: Search using Traversaal AI search engine
perplexity: Search using Perplexity AI
sploitus: Search for security exploits, vulnerabilities (CVEs), and pentesting tools
searxng: Search using Searxng meta search engine (aggregates results from multiple engines)
internal (ftester-only debug function, not an agent tool): Opt-in browser-analytics fallback engine that discovers links, scrapes each page, and summarizes the result; requires WEB_SEARCH_INTERNAL_ENABLED=true, a configured scraper, and at least one available link engine

Vector Database Functions

search_in_memory: Search for information in vector database
search_guide: Find guidance documents in vector database
search_answer: Find answers to questions in vector database
search_code: Find code examples in vector database

AI Agent Functions

advice: Get expert advice from an AI agent
coder: Request code generation or modification
maintenance: Run system maintenance tasks
memorist: Store and organize information in vector database
pentester: Perform security tests and vulnerability analysis
search: Complex search across multiple sources

Utility Functions

describe: Show information about flows, tasks, and subtasks

Debugging Flow Context (click to expand)

The describe function provides detailed information about tasks and subtasks within a flow. This is particularly useful for diagnosing issues when PentAGI encounters problems or gets stuck.

# List all flows in the system
go run cmd/ftester/main.go describe

# Show all tasks and subtasks for a specific flow
go run cmd/ftester/main.go -flow 123 describe

# Show detailed information for a specific task
go run cmd/ftester/main.go -flow 123 -task 456 describe

# Show detailed information for a specific subtask
go run cmd/ftester/main.go -flow 123 -task 456 -subtask 789 describe

# Show verbose output with full descriptions and results
go run cmd/ftester/main.go -flow 123 describe -verbose

This function allows you to identify the exact point where a flow might be stuck and resume processing by directly invoking the appropriate agent function.

Function Help and Discovery (click to expand)

Each function has a help mode that shows available parameters:

# Get help for a specific function
go run cmd/ftester/main.go [function_name] -help

# Examples:
go run cmd/ftester/main.go terminal -help
go run cmd/ftester/main.go browser -help
go run cmd/ftester/main.go describe -help

You can also run ftester without arguments to see a list of all available functions:

go run cmd/ftester/main.go

Output Format (click to expand)

The ftester utility uses color-coded output to make interpretation easier:

Blue headers: Section titles and key names
Cyan [INFO]: General information messages
Green [SUCCESS]: Successful operations
Red [ERROR]: Error messages
Yellow [WARNING]: Warning messages
Yellow [MOCK]: Indicates mock mode operation
Magenta values: Function arguments and results

JSON and Markdown responses are automatically formatted for readability.

Advanced Usage Scenarios (click to expand)

Debugging Stuck AI Flows

When PentAGI gets stuck in a flow:

Pause the flow through the UI
Use describe to identify the current task and subtask
Directly invoke the agent function with the same task/subtask IDs
Examine the detailed output to identify the issue
Resume the flow or manually intervene as needed

Testing Environment Variables

Verify that API keys and external services are configured correctly:

# Test Google search API configuration
go run cmd/ftester/main.go google -query "pentesting tools"

# Test browser access to external websites
go run cmd/ftester/main.go browser -url "https://example.com"

Developing New AI Agent Behaviors

When developing new prompt templates or agent behaviors:

Create a test flow in the UI
Use ftester to directly invoke the agent with different prompts
Observe responses and adjust prompts accordingly
Check Langfuse for detailed traces of all function calls

Verifying Docker Container Setup

Ensure containers are properly configured:

go run cmd/ftester/main.go -flow 123 terminal -command "env | grep -i proxy" -message "Check proxy settings"

Docker Container Usage (click to expand)

If you have PentAGI running in Docker, you can use ftester from within the container:

# Run ftester inside the running PentAGI container
docker exec -it pentagi /opt/pentagi/bin/ftester [arguments]

# Examples:
docker exec -it pentagi /opt/pentagi/bin/ftester -flow 123 describe
docker exec -it pentagi /opt/pentagi/bin/ftester -flow 123 terminal -command "ps aux" -message "List processes"

This is particularly useful for production deployments where you don't have a local development environment.

Integration with Observability Tools (click to expand)

All function calls made through ftester are logged to:

Langfuse: Captures the entire AI agent interaction chain, including prompts, responses, and function calls
OpenTelemetry: Records metrics, traces, and logs for system performance analysis
Terminal Output: Provides immediate feedback on function execution

To access detailed logs:

Check Langfuse UI for AI agent traces (typically at http://localhost:4000)
Use Grafana dashboards for system metrics (typically at http://localhost:3000)
Examine terminal output for immediate function results and errors

Command-line Options

The main utility accepts several options:

-env <path> - Path to environment file (optional, default: .env)
-provider <type> - Provider type to use (default: custom, options: openai, anthropic, gemini, bedrock, ollama, deepseek, glm, kimi, qwen, minimax, custom)
-flow <id> - Flow ID for testing functions that require it (0 means using mocks, default: 0)
-user <id> - User ID for testing functions that require it (default: 0; 1 is the default admin user)
-task <id> - Task ID for agent context (optional)
-subtask <id> - Subtask ID for agent context (optional)

Function-specific arguments are passed after the function name using -name value format.

Pentesting Prompt Methodology

When refining prompts for offensive security work, give the agent a clear methodology instead of a flat list of payloads:

Start with explicit scope, authorization, and success criteria
Map the application first: roles, routes, parameters, uploads, integrations, and trust boundaries
Prioritize attack surfaces systematically instead of testing everything at once
Validate findings with reproducible evidence before escalating to deeper exploitation
Finish with report-ready notes that capture impact, prerequisites, and next steps

For PentAGI-specific prompt guidance, see backend/docs/prompt_engineering_pentagi.md. For a practical starting point, reuse and adapt examples/prompts/base_web_pentest.md to match the target application, technology stack, and engagement scope.

Building

Building Docker Image

The Docker build process automatically embeds version information from git tags. To properly version your build, use the provided scripts:

Linux/macOS

# Load version variables
source ./scripts/version.sh

# Standard build
docker build \
  --build-arg PACKAGE_VER=$PACKAGE_VER \
  --build-arg PACKAGE_REV=$PACKAGE_REV \
  -t pentagi:$PACKAGE_VER .

# Multi-platform build
docker buildx build \
  --platform linux/amd64,linux/arm64 \
  --build-arg PACKAGE_VER=$PACKAGE_VER \
  --build-arg PACKAGE_REV=$PACKAGE_REV \
  -t pentagi:$PACKAGE_VER .

# Build and push
docker buildx build \
  --platform linux/amd64,linux/arm64 \
  --build-arg PACKAGE_VER=$PACKAGE_VER \
  --build-arg PACKAGE_REV=$PACKAGE_REV \
  -t myregistry/pentagi:$PACKAGE_VER \
  --push .

Windows (PowerShell)

# Load version variables
. .\scripts\version.ps1

# Standard build
docker build `
  --build-arg PACKAGE_VER=$env:PACKAGE_VER `
  --build-arg PACKAGE_REV=$env:PACKAGE_REV `
  -t pentagi:$env:PACKAGE_VER .

# Multi-platform build
docker buildx build `
  --platform linux/amd64,linux/arm64 `
  --build-arg PACKAGE_VER=$env:PACKAGE_VER `
  --build-arg PACKAGE_REV=$env:PACKAGE_REV `
  -t pentagi:$env:PACKAGE_VER .

Quick build without version

For development builds without version tracking:

docker build -t pentagi:dev .

Note

The build scripts automatically determine version from git tags
Release builds (on tag commit) have no revision suffix
Development builds (after tag) include commit hash as revision (e.g., 1.1.0-bc6e800)
To use the built image locally, update the image name in docker-compose.yml or use the build option

Credits

This project is made possible thanks to the following research and developments:

Emerging Architectures for LLM Applications
A Survey of Autonomous LLM Agents
Codel by Andriy Semenets - initial architectural inspiration for agent-based automation

License

PentAGI is licensed under the MIT License.

Third-Party Dependencies

All third-party dependencies use MIT-compatible licenses. See licenses/ directory for detailed license reports.

VXControl Cloud Services

⚠️ Note: While the VXControl Cloud SDK code is MIT licensed, accessing VXControl Cloud Services (threat intelligence, AI support, premium features) requires a separate License Key and compliance with Terms of Service.

The SDK code itself is free to use - service access requires registration.

For questions contact: info@pentagi.com or info@vxcontrol.com

multimodal-art-projection/YuE

YuE2: frontier music generation with symbolic planning, zero-shot covers, and agentic music editing.

https://map-yue2.github.io/

Looking for the original YuE? Its code, documentation, and license are preserved on the YuE-v1 branch.

YuE2: Unifying Symbolic and Audio Music Generation at Frontier Quality

Compose in symbols. Create in sound.

🎧 Demos · 🤗 YuE2 · 🚀 Quick start · 🤖 Agent skill · 📊 Benchmarks · 🤗 MERT2 · 🤗 SheetSage2 · 🤗 WSB · 📦 Release ·

YuE2 brings frontier song quality to music generation with an editable composition. Give it lyrics and a style prompt: it writes a melody-and-chord plan, then realizes that plan as a complete song with vocals and accompaniment.

Frontier quality. YuE2 is competitive with Suno v5/v6 on WildSongBench. YuE2 (best-of-8) achieves 6.9632 SongBench Avg, the highest observed mean among all evaluated settings.
White-box music generation through symbolic planning. Read, play, and change the composition before rendering it. Melody and chords become explicit controls that a person or an agent can inspect and edit.
Zero-shot covers and agentic editing. Reimagine a transcribed song in a new style, or refine a song through a conversation about its score, arrangement, and lyrics—all with the same generation checkpoint.

192 WildSongBench prompts. Both YuE2 settings use symbolic planning. Bo8 = best-of-8. The axes are normalized comparison indices; bubble area represents AudioBox production quality. Scores and evaluation protocol. Vector PDF · SVG.

Hear what you can make

Create	Cover	Edit with an agent
Lyrics + style → score → full song	Source recording → melody score → a new interpretation	Musical feedback → score, style, or lyric revisions → a new recording
Listen and inspect the score	Hear zero-shot covers	Follow an editing conversation

The agentic demo follows The Last Train through 9 steps and 14 versions, from Mandarin pop to English jazz with new harmony and a saxophone solo. Listen to each version and inspect its conversation, score, prompt, and lyrics.

How it works

One AR–NAR Mixture-of-Transformers backbone predicts the score and semantic tokens autoregressively, then generates acoustic latents with flow matching. A VAE decodes those latents into stereo audio. Creation, covering, and editing differ in where the score comes from: YuE2, a transcribed recording, or an edited composition.

The staged Python API exposes plan() → generate_semantic() → synthesize() → decode(). See the generation guide for exact-plan reuse and decoder selection.

Quick start

Linux · Python 3.12 · NVIDIA GPU with BF16 support and 24 GB VRAM. YuE2 produces 48 kHz stereo audio without quantization. Model files download from Hugging Face on first use.

git clone https://github.com/multimodal-art-projection/YuE.git
cd YuE
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install .
python examples/generate.py --output outputs/first-song

Open outputs/first-song/audio.flac. The output directory also retains the score, semantic tokens, acoustic latents, generation settings, and model identities.

The Python interface is equally short:

import json
from pathlib import Path
from yue2 import YuE2Pipeline

request = json.loads(Path("examples/song.json").read_text(encoding="utf-8"))
with YuE2Pipeline.from_pretrained("m-a-p/YuE2-3B", device="cuda") as pipe:
    song = pipe(**request)
    song.save_artifacts("outputs/my-song")
    print(song.truncated)

Setting	Behavior
`cot="full"`	Generate an editable melody-and-chord plan; the default for new songs
`cot="melody"`	Use a melody plan with free accompaniment; recommended for covers
`cot="off"`	Generate directly from lyrics and style
`abc=...`	Supply your own score in `full` or `melody` mode

Generation guide · Original example inputs · v0.1.6 wheel archive

Cover a song

Transcribe a source recording with 🤗 SheetSage2, review its melody ABC, and provide new lyrics or a target style. For covers, use cot="melody" and a score without chord symbols so the accompaniment can adapt to the new style.

from pathlib import Path
from yue2 import YuE2Pipeline

with YuE2Pipeline.from_pretrained("m-a-p/YuE2-3B", device="cuda") as pipe:
    cover = pipe(
        style="English, jazz-funk, warm lead vocal, Rhodes, bass and drums",
        lyrics=Path("cover-lyrics.txt").read_text(encoding="utf-8"),
        abc=Path("cover-score/score.abc").read_text(encoding="utf-8"),
        cot="melody",
        seed=42,
    )
    cover.save_artifacts("outputs/cover")

SheetSage2 runs in a separate environment and loads its MERT2 encoder automatically. The cover guide gives the complete transcription and generation commands. An included original melody example also lets you try score-conditioned generation immediately.

Edit a composition

Export a plan, revise the musical details, and render the edited score:

import json
from pathlib import Path
from yue2 import YuE2Pipeline

request = json.loads(Path("examples/song.json").read_text(encoding="utf-8"))
with YuE2Pipeline.from_pretrained("m-a-p/YuE2-3B", device="cuda") as pipe:
    plan = pipe.plan(**request)
    plan.save("outputs/plan")

Copy outputs/plan/score.abc to edited.abc, then ask an agent to change its harmony, melody, tempo, or form. Supply the edited file as a new score:

python examples/generate.py --request examples/song.json \
  --abc-file edited.abc --cot full --output outputs/edited

The editable score is the white-box interface: you can inspect the intended composition and intervene on it. Editing generates a new complete recording; it does not preserve the original waveform outside an edit. Editing guide and a reproducible harmony example.

Agent skill

The yue2-music skill teaches an agent how to generate songs, transcribe and cover recordings, edit ABC scores, check musical invariants, and organize listening comparisons. It includes portable helpers and references to the released model interfaces.

Use skills/yue2-music/ from this repository with an agent that supports SKILL.md packages. Install it using your agent's skill-directory or import mechanism; the Python runtime is installed separately with pip install .. The earlier v0.1.6 skill ZIP remains available under its bundled license.

Try a concrete request:

Use the yue2-music skill to create an English piano-pop song. Keep the original audio and score. Make a second version with jazz harmony, preserve the vocal melody and lyric order, and give me both versions to compare.

Benchmarks

WildSongBench: 192 prompts, automatic evaluation, September 12, 2026.

System / setting	SongBench Avg ↑	AudioBox PQ ↑	MuLan ↑	PER ↓
YuE2 (best-of-8) †	6.9632	8.2714	0.5051	9.79%
Mureka 9	6.9377	8.0226	0.4394	11.69%
Suno v5	6.8721	8.1698	0.5428	8.10%
YuE2 †	6.7316	8.2598	0.5068	8.44%
Suno v5.5	6.7150	8.1955	0.5089	5.96%
Suno v4.5	6.6995	8.2541	0.5022	5.80%
Suno v6	6.5562	8.1296	0.4916	7.58%
Suno v6 Wild	6.4195	8.1785	0.4999	7.45%
LeVo 2 †	6.3247	8.3966	0.3542	26.12%
MiniMax Music 2.6	6.3222	8.1711	0.4251	24.55%
MiniMax Music 3 †	6.2830	8.2825	0.3928	6.27%
HeartMuLa †	6.2483	8.2933	0.3823	10.71%
Muse †	6.0349	8.0517	0.3937	33.42%
ACE-Step 1.5 †	6.0118	8.0518	0.4372	7.46%
DiffRhythm 2 †	5.2428	7.9782	0.3782	18.41%
YuE 1 †	4.9165	7.8683	0.2623	36.38%
SongBloom †	4.2350	8.1539	0.2697	19.19%

† Publicly available model weights. All 17 evaluated settings are shown, sorted by SongBench Avg; bold values mark the best result in each column.

Both YuE2 settings use symbolic planning and the benchmark decoder, YuE2-Vae-legacy. Standard YuE2 selects from two candidates; best-of-8 selects from eight. Rankings vary by metric; the small gap between the highest means does not establish statistical significance. Full results and selection protocols.

Zero-shot covers. On 948 works, full-score YuE2 reaches 0.647 CLEWS mAP, compared with 0.006 without a score, while using the general generator without cover-specific fine-tuning. Source-identity preservation and target-style quality are measured separately; melody-only covers offer more freedom to change the arrangement. Cover evaluation.

Reproduce the benchmarks

To reproduce the reported benchmark scores, follow the instructions on 🤗 WildSongBench (WSB).

MERT2

State-of-the-art music understanding: SOTA on 14 of 15 MARBLE metrics, with 91.72% genre accuracy on GTZAN.

Demo and results · 🤗 MERT2-30s · 🤗 MERT2-FS

SheetSage2

State-of-the-art audio-to-score transcription: SOTA on 10 of 13 benchmark metrics, with 82.51% vocal melody pitch-class F1 on RWC-Pop.

Demo and results · 🤗 Model and inference

Models and resources

Resource	Purpose
🤗 YuE2-3B	Song generation, symbolic planning, covering, and editing
🤗 YuE2-Vae	Default generation and listening decoder
🤗 YuE2-Vae-legacy	Decoder for the reported benchmark protocol
🤗 SheetSage2	Audio-to-score transcription for covers and editing
🤗 MERT-v2-FullSong	Full-song music representations; SheetSage2's encoder
🤗 MERT-v2-30s	Music representations for short recordings
🤗 WildSongBench	Evaluation prompts and benchmark resources

MERT2 feature extraction is optional for generation. YuE2's pipeline does not require a separate MERT2 model download. Demos and interactive results · Release downloads.

License

Model weights are separately licensed under CC BY-NC 4.0. Third-party components retain their original licenses. The archived YuE-v1 branch retains its original license.

Apache 2.0 applies to the current repository source; the earlier yue2-v0.1.6 download archives retain their bundled licenses.

Citation

The YuE2 technical report is coming soon. For now, please cite MERT and YuE:

@article{li2023mert,
  title = {{MERT}: Acoustic Music Understanding Model with Large-Scale Self-supervised Training},
  author = {Li, Yizhi and Yuan, Ruibin and Zhang, Ge and Ma, Yinghao and Chen, Xingran and Yin, Hanzhi and Xiao, Chenghao and Lin, Chenghua and Ragni, Anton and Benetos, Emmanouil and Gyenge, Norbert and Dannenberg, Roger and Liu, Ruibo and Chen, Wenhu and Xia, Gus and Shi, Yemin and Huang, Wenhao and Wang, Zili and Guo, Yike and Fu, Jie},
  journal = {arXiv preprint arXiv:2306.00107},
  year = {2023},
  eprint = {2306.00107},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.00107}
}

@article{yuan2025yue,
  title = {{YuE}: Scaling Open Foundation Models for Long-Form Music Generation},
  author = {Yuan, Ruibin and Lin, Hanfeng and Guo, Shuyue and Zhang, Ge and Pan, Jiahao and Zang, Yongyi and Liu, Haohe and Liang, Yiming and Ma, Wenye and Du, Xingjian and Du, Xinrun and Ye, Zhen and Zheng, Tianyu and Jiang, Zhengxuan and Ma, Yinghao and Liu, Minghao and Tian, Zeyue and Zhou, Ziya and Xue, Liumeng and Qu, Xingwei and Li, Yizhi and Wu, Shangda and Shen, Tianhao and Ma, Ziyang and Zhan, Jun and Wang, Chunhui and Wang, Yatian and Chi, Xiaowei and Zhang, Xinyue and Yang, Zhenzhu and Wang, Xiangzhou and Liu, Shansong and Mei, Lingrui and Li, Peng and Wang, Junjie and Yu, Jianwei and Pang, Guojian and Li, Xu and Wang, Zihao and Zhou, Xiaohuan and Yu, Lijun and Benetos, Emmanouil and Chen, Yong and Lin, Chenghua and Chen, Xie and Xia, Gus and Zhang, Zhaoxiang and Zhang, Chao and Chen, Wenhu and Zhou, Xinyu and Qiu, Xipeng and Dannenberg, Roger and Liu, Jiaheng and Yang, Jian and Huang, Wenhao and Xue, Wei and Tan, Xu and Guo, Yike},
  journal = {arXiv preprint arXiv:2503.08638},
  year = {2025},
  eprint = {2503.08638},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.08638}
}

Contact

For collaborations, licensing, and data partnerships, please contact gezhang@umich.edu.

yuliskov/SmartTube

Browse media content with your own rules on Android TV

https://smarttubeapp.github.io

Important announcement about the app

My development environment was infected by unknown malicious software, as a result of which a few builds may have been affected. Once the issue was detected, I secured everything with a full disk wipe, restored a clean setup, and now all builds are scanned with VirusTotal. The F-Droid version will also be verified before release.

Public keys may have been compromised, which is why I am sharing this issue. You can download the new version and the new public key below, and instructions for restoring backups are provided.

No extra actions are required since the app uses one-time connection codes. These codes have very limited permissions (for example, they cannot change your password). Still, you can revoke them if you want full peace of mind.

How to revoke access:

Open myaccount.google.com/security
Find “Your connections to third-party apps & services”
Tap “See all connection” and locate YouTube TV or Google Drive
Select the app → “Remove access”

Please keep built-in security features enabled to stay protected.

SmartTube

SmartTube is a free and open-source media client for Android TVs and TV boxes. It allows you to browse and play content from various public sources in a TV-optimized interface.

✅ Features

Clean interface
SponsorBlock integration
Adjustable playback speed
8K resolution support
60fps playback
HDR compatibility
View live chat
Customizable buttons
Does not require Google Services
Helpful international community

❌ Limitations

Not supported on phones and tablets
Comment functionality is unstable
Voice search and casting performance may be inferior to official apps, depending on your device

Give it a try!

Do you have any question? Ctrl+F or ⌘F this readme first!

Device support

Important

Starting in October 2025 new Amazon FireTV devices no longer run Android under the hood. SmartTube will not be compatible with the Fire Stick 4k Select and newer devices which run Amazon's own VegaOS.

Supported: all Android TVs and TV boxes (incl. All FireTV devices released before Oct. 2025, NVIDIA Shield & Chromecast with Google TV), even older ones with Android 4.3 (Kitkat).
Not supported: Smartphones, non-Android platforms like Samsung Tizen, LG webOS, Apple TV, etc.

Installation

video of the installation (note: download url changed to kutt.to/stn_beta or kutt.to/stn_stable)

Do not download SmartTube from any app store, APK websites or blogs; these were uploaded by other people and may contain malware or ads. SmartTube is not officially published on any app store.

There is a beta release (recommended) and a stable release. Beta gets new features and bugfixes faster than the stable release.

You can use either of the following methods to install the app:

(Easiest) Install Downloader by AFTVnews on your Android TV, open it and enter kutt.to/stn_beta or kutt.to/stn_stable, then read, understand and confirm the security prompts. (You can also enter 79015 (for beta) or 28544 (for stable), but this requires an extra step to install the AFTVnews Downloader browser addon if you haven't already.)
Install a file transfer app on your Android TV, download the APK on your phone or computer and transfer it to your TV (e.g. Send Files to TV from the Google Play Store / Amazon AppStore)
Download the APK onto a USB stick, put the USB stick into your TV and use a file manager app from the Google Play Store / Amazon AppStore (e.g. FX File Explorer or X-plore). Android's preinstalled file manager does not work! Do not get the ad-infested FileCommander.
If you are an advanced user, you can install it using ADB. guide | alternative guide

Troubleshooting: See device specific notes below. If installation fails, either your disk space is full or the APK file didn't download correctly; clear up space and try downloading again. If the app installed, but crashes when opening, make sure to install it to internal memory, not to an SD card / external storage.

The app has a built-in updater with changelog. You can also find all releases and the changelog on the Telegram channel @SmartTubeNewsEN (readable without account) or on Github.

latest beta download

latest stable download

Installation (Chromecast with Google TV)

On Chromecast with Google TV, installation of apps is blocked by default, so an extra step is required:

4.1. Enable Developer Options

On your Chromecast, open the side menu and go to Settings > System > About. Scroll down to the Android TV OS build section and click that repeatedly. A toast message will appear, explaining that you are a few steps away from being a developer. Continue clicking until you trigger it.

4.2. Turn on the "unknown sources" setting

Go back to the main Settings page and select Apps > Security & Restrictions > Unknown sources. Turn on the toggle for [Downloader by AFTVnews or] whichever file browser you decided to use [...].

[source & picture guide]

After this, you can follow the general installation guide above.

Installation (Xiaomi devices with Chinese firmware)

Xiaomi's Chinese firmware might block the installation of the beta version. The international firmware is not affected. Solutions:

Use SmartTube's stable version instead (recommended)
Use the international firmware for your device
(if your device is from 2020 or before) You can do a factory reset and then install SmartTube beta before doing any system updates. You can then safely update your system, SmartTube should continue working.

Updating

The app has a built-in updater. You only need to follow the installation procedure once. A few seconds after launching SmartTube, it will notify you if there is any update and also show a changelog. You can disable automatic update checks or manually update in the settings under "about".

If the installation fails, either your disk space is full or the update didn't download correctly; clear up space and try updating again (Settings > About > Check for updates).

Compatibility

SmartTube requires Android 4.3 or above. It does not work on non-Android devices (incl. LG or Samsung TVs). On unsupported TVs, you can use a TV stick or TV box. Though this app technically runs on smartphones and tablets, it is not optimized for such and offers no official support!

It has been successfully tested on TVs, TV boxes and TV sticks that are based on Android, including:

Android TVs & Google TVs (e.g. Philips, Sony)
Chromecast with Google TV & TVs with Chromecast built-in
Amazon FireTV stick (all generations)
NVIDIA Shield
TV boxes running Android (many cheap chinese no-name boxes)
Xiaomi Mi Box

Features

Interface

SmartTube focuses on a clean and distraction-free interface for watching media content. The app does not include advertising UI elements and focuses on playback and browsing experience instead.

SponsorBlock

SmartTube includes optional SponsorBlock integration. From the SponsorBlock website:

SponsorBlock is an open-source crowdsourced browser extension and open API for skipping sponsor segments in YouTube videos. [...] the extension automatically skips sponsors it knows about using a privacy preserving query system. It also supports skipping other categories, such as intros, outros and reminders to subscribe [and non-music parts in music videos].

You can select which categories you want to skip in the settings. Unlike the browser addon, in SmartTube you cannot submit new segments (TVs and TV remotes aren't great devices for such precise operations). Note that SponsorBlock is a free and voluntary project based on user submissions, so don't expect it to 100% work every time. Sometimes, sponsor segments are not yet submitted to the database, sometimes the SponsorBlock servers are offline/overloaded.

Casting

To cast videos from your phone (or other devices), you must link that device to your TV. Unlike the original YouTube app, SmartTube does not automatically show up when you are in the same wifi network. How to link your smartphone and TV:

open SmartTube and go to settings
go to "Remote control" (2nd option)
open your YouTube app on your phone, go to settings > General > watch on TV
click on connect using TV-code and enter the code from your TV

Screenshot guide

Due to technical limitations, you need to open the app on the TV before casting; SmartTube cannot automatically wake up the TV.

Picture-in-Picture (PiP)

SmartTube supports playing videos in PiP mode. This needs to be enabled under Settings > General > Background playback > Picture in picture. The video will go into PiP mode when you press home while playing a video, and also when you press back if enabled in Settings > General > Background playback (activation).

Adjust Speed

You can adjust the playback speed pressing the speed-indicator icon (gauge) in the top row of the player. This is remembered across videos. Some speeds may cause frame drops, this is a known issue.

Voice Search

To enable global voice search, an additional app must be installed alongside SmartTube. This bridge app allows the system voice search to open SmartTube instead of the default YouTube app. For this to work, you must uninstall the original YouTube app. We know this sucks, but you can always reinstall it if you change your mind. The bridge app will not show up in your launcher and you cannot launch it directly; it is only used internally by the system's voice search. On some devices, you need to explicitly say "Youtube" when searching (e.g. say "youtube cute cats" instead of just "cute cats").

On Amazon Fire TV:

Uninstall the original YouTube app (no root required)
Download and install the Amazon Bridge SmartTube app: https://kutt.to/stn_bridge_amazon (e.g. via Downloader by AFTVnews)

On Google Chromecast with Google TV:

Uninstall the original YouTube app (no root required)
Download and install the ATV Bridge SmartTube app: https://kutt.to/stn_bridge_atv (e.g. via Downloader by AFTVnews)

On all other Android devices, sadly root is required to enable this:

Root your device (search for a guide for your specific device)
Uninstall the official YouTube app using root (adb shell pm uninstall com.google.android.youtube.tv)
Download and install the ATV Bridge SmartTube app: https://kutt.to/stn_bridge_atv (e.g. via Downloader by AFTVnews)

Donation

If you want to support my developments you are welcome to buy me a cup of coffee 😃

Patreon (Visa, Mastercard, PayPal)
PayPal: firsth ash@gmai l.com

BTC: 1JAT5VVWarVBkpVbNDn8UA8HXNdrukuBSx
LTC: ltc1qgc24eq9jl9cq78qnd5jpqhemkajg9vudwyd8pw
ETH: 0xe455E21a085ae195a097cd4F456051A9916A5064
ETC: 0x209eCd33Fa61fA92167595eB3Aea92EE1905c815
TRX: TJNPY794aSGZf3WGHTna2VCWm2G5Yua7E8
USDT (TRC20): TJNPY794aSGZf3WGHTna2VCWm2G5Yua7E8
USDT (BEP20): 0x64B28da787BE6ac5889D276A5638d4f077840eC5
USDT (ERC20): 0xe455e21a085ae195a097cd4f456051a9916a5064
TON: UQAc9zgnnzwS8yb5wxAu5CB0RddmjPBjWI-n46oQ7XfCQrgI
XMR: 48QsMjqfkeW54vkgKyRnjodtYxdmLk6HXfTWPSZoaFPEDpoHDwFUciGCe1QC9VAeGrgGw4PKNAksX9RW7myFqYJQDN5cHGT

Support

Please check the FAQ first! Also at least have a short look at the recent chat history.

You can report in our Telegram group or via issue tracker on Github (account required).

Telegram group (international): @SmartTubeEN
Discord group (international): SmartTube Official
Telegram group (RU/UA): @SmartTubeUA
Email: firsth ash@gmai l.com

The international group is in English only. But don't worry if your English is not perfect, we have a friendly international community.

Team

SmartTube is developed single-handedly; there is no larger team or company behind this. This is an open source, hobby project. Several others have helped with translations, some of which can be seen on Github, some have sent their translations directly to Yurii. There are also helpful people in the support chat.

Build

NOTE: OpenJDK 14 or older (!) is required. Newer JDK could cause app crash!
To build and install debug version, run these commands:

git clone https://github.com/yuliskov/SmartTube.git
cd SmartTube
git submodule update --init
adb connect <device_ip_address>
gradlew clean installStstableDebug

Video codecs

Video codecs are the algorithms used for video compression.

Which codec to choose / overview

	recommendation	hardware support	compression, bitrate*	quality
AV01 aka. AV1	best choice, if your device supports	first devices started coming in 2020	best (e.g. 1.6 Mbps)	same
VP9	best choice on most devices	most devices since 2015	better (e.g. 2.1 Mbps)	same
AVC	only for old or slow hardware	all devices	good (e.g. 2.7 Mbps)	same

* Examples taken from the video-only track at 1080p @ 25fps for this video: Dua Lipa - New Rules (Official Music Video)

At the same resolution, a lower bitrate is better! YouTube explicitly targets the same quality regardless of the codec. Older codecs have a higher bitrate only because they are less efficient. On Youtube, you do not get better quality by simply choosing a higher bitrate. Newer codecs have a better compression = lower bitrate = use less bandwidth = save the environment. This is a feature, not a bug. You should use the newest codec that works smoothly on your device, not the least efficient one. AVC usually has the highest bitrate. This is bad, not good.

Which quality to choose?

Currently, there is no automatic mode based on your bandwidth. But you can configure a default video preset yourself under settings > video player > video presets. The first option ("none") will remember your last selection within the video player. Any other preset is used initially for each video; if the selected profile is not available, the next best available option is used. You can still override the profile on each video individually within the player.

To decide the optional resolution / video quality for you, you need to consider a few limiting factors:

Your bandwidth (choose only up to the bitrate that your bandwidth can handle; you can do a speedtest using fast.com by Netflix)
Your TV's display resolution (the quality might slightly improve, if you select the next higher resolution, e.g. 1080p on a 720p display; but don't expect a big difference)
Your TV's capabilities (e.g. HDR, 60fps)

Generally 60fps is an improvement, but if you personally don't notice (or mind) the difference, you can save bandwidth (and the environment) by not choosing 60fps.

HDR

HDR works only if your hardware supports it. It's a complicated mess:

Your TV must support it
If you use a TV box, that TV box and your TV cable and the TV must support HDR
Yes, there truly are different HDMI cable versions with different HDR-support, it's complicated
some devices (like the NVIDIA Shield) generally support HDR, but not the specific HDR format that is used on YouTube 😢

If HDR videos look dim or washed out, then check this article. If HDR is not working, it's probably not this app's fault. You might need to search on the web for "HDR" and your device name for any help.

Liability

We take no responsibility for the use of our tool, or external instances provided by third parties. We strongly recommend you abide by the valid official regulations in your country. Furthermore, we refuse liability for any inappropriate use of third-party services. This tool is provided to you in the spirit of free, open software.

You may view the LICENSE in which this software is provided to you here.

Limitation of Liability.

IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

FAQ

Q: Videos buffer a LOT

A: Try to switch to encrypted DNS like NextDNS. You can set-up such DNS either automatically or manually. To automatic set-up you can use the Intra apk at fdroid and the "AutoStart - No root" apk to make it autolaunch after every TV restart. For manual set-up use this guide.

Q: There is no result for the search that I say (Android 11)

A: They're some reports that the latest update for "Google app for Android TV" could cause this bug. Deleting the update should fix the problem.

Q: AV01 does not play / Why is VP9 slow on my device?

A: Because AV01 is very new, most TVs and TV boxes do not offer hardware support and cannot play AV01 at all.

If your device has hardware support for a codec, videos using that codec should play smoothly. High resolutions might also be slow in VP9 on cheap TV boxes that don't officially support 4k. Your device probably can play VP9 videos even without hardware support, however this requires a powerful CPU to run smoothly. Fixing AV01 without hardware support is technically possible, but currently not planned and probably not efficient enough.

Q: Can you make SmartTube look like the original app?

A: Compared to SmartTube's UI, Stock Youtube and YT Kids are far ahead. However, we'd need someone who's skilled and willing to dedicate enough time and energy into making it. And into maintaining it longterm (incl. new features, bug fixes). All of this for free. If you are / got someone like that, please help. Not to mention that SmartTube follows Google's official template & recommendations for Android TV apps. It's Google's fault that the template is somewhat ugly. 😂

Q: Can the search page be improved?

A: It can be, but it takes someone to do it, similar to the above FAQ-entry. SmartTube is following Google's officially recommended design/template for TV apps and is using the official, preinstalled Android TV keyboard. Sadly, Google did a really bad job regarding the search page and keyboard. Maybe a future SmartTube update can add an embedded keyboard, similar to the original YouTube or other major Android TV apps. Maybe it can improve the looks to be as good or better than in the official YouTube app. But for now, it is the way it is due to lack of time and due to Google's official recommendations being bad.

Q: Can I install this on a Samsung Tizen TV / LG webOS TV / Roku / iOS / toaster?

A: No, this only works on Android devices. If you look at an Android TV's product page, it usually says clearly that it's based on Android. The app cannot easily be ported over to other platforms and we have no plans to even try. Please do not ask. Instead, you can connect a separate TV stick or TV box to your TV.

Q: Can I install this on a smartphone? / Can you add portrait mode? / Scrolling doesn't work.

A: Big No. This app is not for smartphones, we offer zero support for that.

You can cast videos from your smartphone to a TV / TV box running SmartTube, though. Just use the official YouTube app, Morphe or ReVanced, see the casting section for more information.

There will not be a phone version. You can use Morphe, ReVanced, Pure Tuber, LibreTube, NewPipe, or NewPipe x SponsorBlock instead. Please go to their respective support chats for help.

Q: Can I install this on a tablet / car screen / smartphone with docking station?

Yes... maybe.. Requirements:

It is an Android device
It has a large screen
It has a TV remote, controller, or keyboard Touch input is not supported. Mouse/touchpad scrolling neither. You cannot properly use SmartTube with only touch or mouse input.

Some users reported great success (incl. on a car entertainment system). Please share your success stories with us.

Q: I get "unknown codec" / "can't download video" errors

A: please wait 5 seconds for the video to play. If that doesn't help, press the play button. Some users reported, that this issue only appears when they have a USB audio device attached or if their disk storage is full.

Q: I get "the video profile is not supported"

A. Press the "HQ"-button in the bottom-left, select video formats and select anything other than AV01. AV01 is not supported on most devices (apparently including yours), so select VP9 instead. See the section on video codecs for more information.

Q: I get "video unavailable" when watching unlisted videos / my own videos

A: Right, that's currently a bug.

Q: It doesn't show up on my casting list

A: Please read the Casting section.

Q: I get an error saying "Sign in to confirm you're not a bot"

A: Your IP address range might be temporarily/permanently blocked by YouTube from watching videos if you not signed in to your account.

Q: The video is buffering a lot

A: The issue might not be specific to SmartTube, as other unofficial YouTube apps also report this issue. It seems uncommon nowadays, but was very present in the 2nd quarter of 2021. Some users or devices seem to be more affected than others. The official YouTube app & website are apparently only rarely affected. The root cause of the issue is currently unclear, but it appears to be a server-side thing on YouTube's end. Possibly, YouTube is discriminating 3rd party apps.

For now, try to see if it helps to:

Reduce the resolution (or change it back)
Change the video format to AVC
Increase the buffer in the settings
Hit the back button and try playing the video again

Q: The debug information says my display is 1080p, but I have a 4k/UHD display!

A: Do not worry, the debug information is incorrect. SmartTube works fine even above 1080p and you should be able to see that, when you play a video in 4k or UHD. Also do not worry if it says "720p" and you have a 1080p display.

Q: Why does it not autoselect highest quality?

A: It does (by default). If you set a video profile under settings, that acts as a maximum for automatic selection. Check if you configured a video profile, you can unset it by choosing "none".

Please do not confuse quality with bitrate. See the section on video codecs for more information.

Q: Can I set a (maximum) resolution by default?

A: SmartTube automatically select the highest available quality for your video, up to a maximum resolution that you can set in the settings under "video profile". If available, SmartTube will pick the selected video profile, or otherwise the next best one available will be used. You can still always change the video profile while watching videos.

Q: Can it set the resolution to "auto", depending on my available bandwidth?

A: This is planned, but not available yet (sorry 🙇‍♀️). However, you can set a maximum resolution to something that should work for your bandwidth. See above for details.

Q: Why does it skip video segments?

A: SmartTube has a feature called SponsorBlock. You can select which categories should be skipped, if any. See the SponsorBlock section for more details.

Q: How to start the next video automatically / stop after every video?

A: You can switch between different autoplay-modes using the loop-button 🔁

Q: How to remove recommended videos (e.g. news) that are unrelated to me?

A: Recommended videos are defined by YouTube and not by the app, we cannot change the algorithm. They are based on your country, which you can change in the settings. If you are logged in, they are based on your watch history, user profile data, and whatever else Google might use. If you are not logged in, you are like in "incognito mode", so your watch history does not influence your recommendations. Maybe a future version will add optional user profiling without logging in.

Q: Does HDR work?

A: Yes, HDR works if your hardware supports it. The NVIDIA Shield does not. See the section on HDR for more information.

Q: Why do some updates say "don't update if satisfied with the current version" in the changelog?

A: These updates change a lot of code, trying to fix bugs that only affect a few users/devices. Only the affected users should update. For anyone else, there is nothing to gain from updating; however there is the chance of causing new bugs. Do not worry if you updated anyways.

Q: When playing at other speeds, frames are skipped!

A: We currently cannot fix this, sorry.

Q: What is AFR?

A: "Auto Frame Rate". It adjusts the refresh rate of your TV to match the content you're watching. It can slightly improve the smoothness, but the difference is very small; most people barely notice it. It does not work well on every hardware. If you don't know what it does and don't want to test it out yourself, you can safely keep it off.

Recommendation: You can turn it on to see if it works on your device; if it causes issues (or if you don't care to test), turn it off.

Q: Should I choose high or low buffer?

A: The higher your buffer, the more of a video will be preloaded ahead of your current position. A low buffer might minimally reduce your bandwidth usage, if you often close videos before they end. A high buffer can smooth out network issues and prevent the video from pausing to buffer. A higher buffer increases RAM usage, however this shouldn't be an issue.

Recommendation: high.

Q: Can I retain the buffer when seeking back?

A: No, when you seek back (e.g. jump back 5 seconds), SmartTube will have to rebuffer. This might be improved in a future update.

Q: My device freezes when watching YouTube

A: That's a firmware or Android issue. If you are using a custom rom, maybe that rom is buggy. Because this issue is nearly impossible for the developer to debug, we cannot help you, sorry. You can try the usual workarounds: rebooting, clearing cache, reinstalling the app, or factory resetting the device.

Q: Can I download videos?

A: Not with SmartTube

Q: Can updates be installed automatically?

A: No, this is technically not possible. Only the preinstalled app manager (usually Google PlayStore, Amazon AppStore, etc) has the required permission. All other apps, incl. SmartTube can only show open installation prompt. A workaround using root would be possible, but hasn't been implemented yet.

Q: Can I whitelist ads on some channels?

A: No, this is not possible. SmartTube does not have any code to display ads. Adding this functionality would actually take time and effort, which is instead spent on adding useful features and fixing bugs.

alphaXiv/OpenResearch

Turn your coding agents into research agents

https://openresearch.sh/

OpenResearch

The local-first workspace for research agents and autoresearch.

Turn Claude Code, Codex, OpenCode, or Cursor into research agents that can review literature, develop hypotheses, run experiments, and produce research artifacts.

_{macOS 11+ · Windows beta requires Git for Windows}

Get started

Install the CLI on macOS or Linux, then launch OpenResearch:

curl -LsSf https://openresearch.sh/install.sh | sh
orx up

On Windows, use the beta download above after installing Git for Windows.

orx up opens the local dashboard at http://127.0.0.1:4791.

Connect a local model to use LM Studio, oMLX, Ollama, or a custom endpoint with OpenCode.

Create an account at openresearch.sh to receive email updates and use managed OpenResearch compute.

Built for research agents

	OpenResearch gives you
Parallel exploration	Give each research direction an independent agent session and isolated git worktree.
Reproducible experiments	Track variants in a git-native experiment tree; every run receives an immutable archive of its recorded commit.
Evidence in context	Keep logs, diffs, files, results, and artifacts tied to the work that produced them.
Your choice of agent	Use Claude Code, Codex, OpenCode, or Cursor, with the harness and model selected per session.
Your choice of compute	Run locally, on your own infrastructure, or with managed OpenResearch compute.
Local ownership	Keep projects, conversations, experiments, runs, logs, code, and artifacts on your machine.

Autoresearch

OpenResearch can run the full loop autonomously: propose an idea, change the code, launch an experiment, inspect the evidence, and decide what to try next. Multiple agents can explore different directions in parallel while the experiment tree preserves their lineage.

Run anywhere

The same committed source snapshot can run locally, over SSH, or on Slurm, Kubernetes, Ray, Hugging Face Jobs, Modal, Tinker, and managed OpenResearch compute. Publishing the repository is not required.

Run the workspace next to remote GPUs while using the browser on your laptop:

orx up --remote user@host

SSH config aliases and custom ports are supported. The remote service binds to loopback and has no application-level authentication, so other users on that host can reach it.

CLI and agent integration

Install the OpenResearch skill into supported coding agents:

orx install-skills

Common commands:

orx projects
orx project view <project-id>
orx runs <project-id>
orx logs <run-id>
orx exp run <experiment-id>
orx discover keyword <query>
orx paper <arxiv-id-or-doi>

Run orx --help or orx <command> --help for the complete interface.

Local by default

OpenResearch runs on 127.0.0.1 with a local SQLite store. Creating a project or launching a run does not publish your code. An openresearch.sh account is only used for service-owned capabilities such as organizations and managed compute.

Usage analytics

Official release builds send opt-out, coarse usage events tied to a random installation ID. They do not include code, prompts, file contents or paths, repository names, tokens, emails, or project and experiment identifiers.

orx telemetry off
orx telemetry status
orx <command> --no-telemetry

Source and development builds do not send analytics.

debpalash/VoiceStudio

VoiceStudio is the open-source, fully-local ElevenLabs alternative — voice cloning, voice design, video dubbing, dictation, transcription & audiobook creation in 646 languages.

https://voicestudio.sh

NOTE: Electron Rewrite Ongoing: Please dont't create desktop app related issues and pr

VoiceStudio

_{Previously OmniVoice-Studio}

Clone voices, dub video, dictate, and produce long-form audio on your own hardware.

16 TTS engines · 11 ASR engines · 646-language catalogue · macOS, Windows, Linux, and Docker

No account, API key, subscription, or usage meter for the local workflow.

Install · Features · Compare · Requirements · Hardware · Engines · Architecture · API · Docs · FAQ · 简体中文

Warning

Active beta. Use the latest release for stable work. main contains the newest fixes and may change between releases. Report problems through GitHub Issues.

At a glance

	VoiceStudio
Workflows	Voice cloning and design, video dubbing, dictation, stories, audiobooks, batch generation
Language catalogue	646 TTS languages; actual coverage and quality depend on the selected engine
Engines	16 TTS · 11 ASR · switch in Model Catalogue or with `Ctrl`/`Cmd`+`E`
Platforms	macOS 13.3+ on Apple Silicon · Windows 10/11 x64 · Linux x86_64 with glibc 2.39+
Compute	CUDA · Apple Silicon MPS/MLX · ROCm on Linux · CPU · optional remote workers
Interfaces	Desktop app · local REST/SSE/WebSocket API · OpenAI-compatible audio API · MCP Server
Storage	Voices, projects, settings, and outputs stay on the machine by default
License	AGPL-3.0 application; downloaded models keep their upstream terms

The Voice workspace starts with three tabs: From audio for cloning, By design for creating a voice, and Convert for speech-to-speech conversion. Each tab displays its own workflow, with Synthesize Audio or Convert pinned below the scrolling form. The top-bar Engines panel combines engine selection, loaded models, and unload/flush controls; Ctrl/Cmd+E opens it. The searchable language picker shares Dubbing’s flags and language list layout, selects one output language, and retains Auto and the full cloning catalogue. Language options flow into multiple columns when space allows. Expand Workspaces in the sidebar to reveal navigation labels; Escape collapses it.

Dubbing starts with file upload or URL import and nearby language choices. Its Projects panel lists previous dubs so they can be reopened by clicking anywhere on a card; action buttons operate independently. Advanced import options include captions and optional YouTube sign-in. Dubbing places playback controls over the video with background blur and combines the waveform and timed transcript in one compact editing surface. Drag the zoomed waveform left or right to pan; click to seek. Translation language and ISO-code controls stay synchronized; Auto clears any previous language code and dialect. Transcript items group editable text, timing and status, and voice controls into three readable rows that wrap with the panel width. Output Options stays compact with the active settings shown in its summary; expand it to change output, timing, or voice matching. Transcript, glossary, and paste controls share a toolbar above the segment editor. Project details, workflow steps, and Generate/Verify/Export actions use an unfilled header.

The Audiobook Script editor fills the available workspace beneath its markup toolbar; Voices and Book settings stay in their own tabs.

Output settings use aligned rows; review status appears before the collapsible transcript and glossary. Glossary terms have labelled entry fields and an explicit edit action. Launchpad arranges recent files and saved voices side by side when space allows, with responsive card grids and visible Open actions.

The casting board shows icon-based voice cards and searchable selectors for each speaker. Drag a card onto a speaker or choose a voice from that speaker’s menu.

Install

Download a package from the latest release, then follow the platform guide.

Platform	Package	Guide
macOS 13.3+	Apple Silicon DMG	Install on macOS
Windows 10/11	x64 MSI; choose the current-user build when listed to install without admin access	Install on Windows
Linux	AppImage, x86_64 with glibc 2.39+	Install on Linux
Docker	Linux/AMD64 images; CUDA, ROCm, CPU, and worker-only GPU profiles	Run with Docker

First launch creates a managed Python environment and downloads the default model. Later launches reuse both.

Note

On macOS, first launch needs a one-time right-click, then Open approval. Intel Macs cannot run the local Python backend; use a remote backend instead.

Quick Docker run

The published images are linux/amd64 only. On Apple Silicon, use the native macOS app for GPU acceleration. ARM64 hosts should read the architecture requirements before pulling an image.

docker run -d -p 127.0.0.1:3900:3900 -v omnivoice-data:/app/omnivoice_data --name voicestudio palashdeb/omnivoice-studio:stable

First voice

Launch VoiceStudio and open Voice Cloning.
Add a clean voice sample. Three seconds works; 5 to 15 seconds usually gives a better prompt.
Enter text, choose a language, then select Generate.

Tip

Try without installing: Run VoiceStudio in the cloud via the Google Colab notebook. Explore audio quality comparisons in benchmarks and prompt design tips in expressive speech.

Audio samples

Listen to sample outputs produced locally with VoiceStudio:

Workflow	Prompt / Reference Audio	Generated Audio
Voice Cloning	demo_voice.wav	demo_clone_output.wav
Voice Design (US News Anchor)	"Clear, authoritative American broadcast tone"	demo_voice_design_us_news_anchor.wav
Voice Design (UK Audiobook)	"Warm, expressive British storytelling voice"	demo_voice_design_audiobook_uk_narrator.wav
Video Dubbing (Multilingual)	source.src.wav	Spanish · French · Japanese · Chinese

Run from source

Install the development prerequisites (Node 20+/Bun and Python 3.11+), then:

git clone https://github.com/debpalash/VoiceStudio.git
cd VoiceStudio
bun install
bun run desktop

The desktop launcher configures Python dependencies on first run via uv automatically. Use bun run dev for the browser UI. See Contributing for services, tests, and platform packages.

If setup fails

Run Settings → About → Run self-check or uv run python backend/main.py --diagnose --deep.
Check install troubleshooting.
Save a scrubbed diagnostic bundle from the app when opening an issue.
For slow generation, compare measured benchmarks and performance settings.

Features

Area	Included
Voice Cloning	Zero-shot synthesis from a short reference clip (guide)
Voice Design	Create a voice from age, accent, pitch, style, and delivery instructions (expressive speech)
Video Dubbing	Transcribe, translate, preserve speakers, synthesize, and export video; compact translation settings include track selection, and completed dubs flag timing issues for review (export guide)
Stories and audiobooks	Multi-voice scripts · EPUB/PDF import · chapter rendering · `.m4b` export
Dictation Widget	System-wide shortcut, live transcription, optional local-LLM cleanup
Vocal Isolation	Demucs speech/background separation
Speaker Diarization	Pyannote and WhisperX speaker assignment (guide)
Batch Queue	Queue large sets of audio and video jobs with per-job progress, or watch a local folder for new videos
Model Catalogue	Install, remove, select, and route TTS, ASR, and LLM models (catalogue)
Remote Model Downloads	Install models on enrolled remote workers with live progress (guide)
GPU Auto-Detect	CUDA, MPS, ROCm, and CPU routing with per-engine checks (performance)
AI Watermark	AudioSeal embedding and detection
MCP Server	Synthesis and transcription tools for MCP clients (guide)
Diagnostics	Self-checks, error journal, logs, and scrubbed support bundles (troubleshooting)
Local-first	Core creation stays local; network-backed features are explicit opt-ins
Extensible	Registry-based TTS, ASR, and plugin interfaces (acceptance)


_{Model Catalogue: engine, device, and install state}	_{Gallery: save a shared voice as a local profile}

Comparison

VoiceStudio trades managed cloud compute for local control. This is the practical difference:

	VoiceStudio	Typical hosted voice service
Best fit	Private, offline, self-hosted, or high-volume work	Fast setup without local model management
Data path	Local by default; remote features are opt-in	Audio and text are processed by the provider
Cost model	Free software; you supply the hardware	Subscription, credits, or metered API use
Setup	Install the app and model weights	Create an account and use the web app or API
Performance	Depends on your engine and hardware	Provider manages compute and scaling
Offline use	Yes, after required models are installed	Usually requires a network connection
Customization	Source, engines, models, API, and routing are open	Limited to provider options
Maintenance	You manage updates, disk, and compute	Provider manages infrastructure

Requirements

Requirements vary by engine. These values cover the default local workflow.

	Minimum	Recommended
OS	Windows 10 x64 · macOS 13.3 Apple Silicon · Linux x86_64 with glibc 2.39+	Current supported OS release
RAM	8 GB	16 GB+
Disk	10 GB free	20 GB+ SSD
GPU	Optional; CPU mode is supported	NVIDIA CUDA or Apple Silicon
VRAM	4 GB when using a GPU	8 GB+; large optional engines need more
Python from source	3.11+	3.11 or 3.12

ROCm is Linux-only and opt-in. Windows AMD/Ryzen AI uses CPU. Systems with limited VRAM offload work to CPU when required. See performance, benchmarks, and engine disk usage.

Recommended stack by hardware

Hardware	Recommended TTS	Recommended ASR	Why
Apple Silicon (M1–M4)	MLX-Audio · OmniVoice (MPS)	MLX Whisper · Parakeet MLX	Native unified memory, lowest latency on macOS
NVIDIA GPU (8 GB+ VRAM)	OmniVoice · CosyVoice 3	WhisperX	High-fidelity zero-shot cloning, word timestamps, diarization
Low VRAM / CPU-only	PocketTTS · Sherpa-ONNX · KittenTTS	Moonshine · Faster-Whisper (`int8`)	Low memory footprint, optimized CPU inference

Engines

Engine support is capability-specific. Check cloning, language, platform, memory, and license before choosing one. Full setup guides: docs/engines.

Text to speech

Engine	Languages	Clone	Instruct	Linux	macOS ARM	Windows	License
VoiceStudio (default, powered by k2-fsa/OmniVoice)	600+	Yes	Yes	CUDA/CPU	MPS	CUDA/CPU	AGPL-3.0 app · Apache-2.0 code, CC-BY-NC weights³
CosyVoice 3	9 + 18 dialects	Yes	Yes	CUDA/CPU	CPU	CUDA/CPU	Apache-2.0
GPT-SoVITS	5	Yes	No	CUDA/CPU	No	CUDA/CPU	MIT
VoxCPM2	30	Yes	Yes	CUDA/CPU	MPS	CUDA/CPU	Apache-2.0
MOSS-TTS-Nano	20	Yes	No	CUDA/CPU	CPU	CUDA/CPU	Apache-2.0
KittenTTS	English	No	No	CPU	CPU	CPU	MIT
MLX-Audio	Model-dependent	Varies	Varies	No	MLX	No	Varies
Sherpa-ONNX	20+	No	No	CUDA/CPU	CPU	CUDA/CPU	Apache-2.0
IndexTTS 2.5 ⚡	ZH · EN · JA · ES · AR	Yes	No	CUDA/CPU	CPU	CUDA/CPU	Bilibili model license¹
OmniVoice GGUF ⚡	600+	Yes	Yes	CUDA/CPU	MPS/CPU	CUDA/CPU	AGPL-3.0 app · review the derivative model terms³
OmniVoice (subprocess; opt-in off MPS) ⚡	600+	Yes	Yes	CUDA/CPU	MPS via default OmniVoice	CUDA/CPU	AGPL-3.0 app · Apache-2.0 code, CC-BY-NC weights³
PocketTTS ⚡	EN · FR · DE · PT · IT · ES	Yes	No	CPU	CPU	CPU	CC-BY-4.0, gated²
Supertonic 3 ⚡	31	No	No	CPU	CPU	CPU	OpenRAIL-M
MOSS-TTS-v1.5 ⚡	31	Yes	No	CUDA/CPU	CPU	CUDA/CPU	Apache-2.0
dots.tts ⚡	24	Yes	No	CUDA/CPU	CPU	No	Apache-2.0
Confucius4-TTS ⚡	14	Yes	No	CUDA/CPU	CPU	CUDA/CPU	Apache-2.0

⚡ Installed or registered on demand.

¹ IndexTTS 2.5 requires a separate written Bilibili license above 100 million monthly active users or RMB 1 billion annual revenue. Review the model license.

² PocketTTS shows its gated-access and CC-BY-4.0 terms before first use.

³ The OmniVoice snapshot also includes an audio tokenizer under separate Boson Higgs Audio 2 and Meta Llama community terms. VoiceStudio's application license does not replace model or tokenizer terms.

Clone-less engines cannot preserve a reference speaker in dubbing or pinned-voice batch jobs. VoiceStudio rejects those jobs instead of silently changing engines. Heavy engines have separate memory and platform limits; check their engine guide first.

Speech to text

Engine	ID	Languages	Best fit
WhisperX (default)	`whisperx`	~100	Dubbing, subtitles, word-level timing
Faster-Whisper	`faster-whisper`	~100	General cross-platform transcription
Faster-Whisper (isolated)	`faster-whisper-isolated`	~100	Crash-isolated batch transcription
MLX Whisper	`mlx-whisper`	~100	Apple Silicon
PyTorch Whisper	`pytorch-whisper`	~100	CUDA, MPS, and CPU fallback
Parakeet TDT	`nemo-parakeet`	English + 25 EU	Fast CPU/CUDA transcription
Parakeet TDT v3 (MLX)	`parakeet-mlx`	25 EU	Apple Silicon dictation and word timestamps
Moonshine	`moonshine`	English	Low-power, low-latency ONNX
FunASR	`funasr`	50+	VAD and inline diarization
sherpa-onnx (live dictation)	`sherpa-onnx-asr`	Model-dependent	Streaming CPU dictation
OpenAI-compatible ⚠️ configured server	`openai-compat-asr`	Server-dependent	Local gigastt/Qwen3-ASR or a remote endpoint; audio goes only to that server

WhisperX and Faster-Whisper retry with int8 when efficient float16 is unavailable. Pin ASR_COMPUTE_TYPE=int8 or float32 only if automatic selection still fails.

Architecture

Tauri v2 desktop shell (Rust)
        │ IPC
React + Vite UI
        │ HTTP · SSE · WebSocket on localhost:3900
FastAPI backend
        ├── TTS / ASR engine registries
        ├── dubbing / audio / long-form pipelines
        ├── OpenAI-compatible API and MCP server
        └── SQLite + Alembic → omnivoice_data/

Layer	Path	Responsibility
Desktop shell	`frontend/src-tauri/`	Window lifecycle, tray, shortcuts, updater, sidecar bootstrap
Frontend	`frontend/src/`	React UI, Zustand state, API and event clients, i18n
API	`backend/api/`	REST routes, schemas, auth boundaries, streaming
Core services	`backend/services/`	Generation, dubbing, audio processing, persistence
Engines	`backend/engines/`	Isolated and optional engine adapters
Worker system	`backend/worker/`	Authenticated remote compute and job transport
Data	`omnivoice_data/`	Projects, voices, settings, logs, and SQLite state
Delivery	`scripts/`, `deploy/`, `.github/workflows/`	Development, packaging, containers, releases, CI

Network boundary

The desktop talks to a loopback-only backend on localhost:3900.
Loopback API calls need no server key. Remote access requires a share PIN or API key.
Remote workers and OpenAI-compatible ASR are opt-in. Loopback ASR may use HTTP and keeps audio on the machine; non-loopback endpoints require HTTPS, and redirects are not followed.
Analytics is off until consent. If enabled, it sends allowlisted, content-free usage metadata. It never sends text, audio, file names, or projects.

Local speech platform and OpenAI-compatible API

Point an OpenAI-compatible audio client at the local backend:

- base_url="https://api.openai.com/v1"
+ base_url="http://localhost:3900/v1"

Endpoint	Purpose
`POST /v1/audio/speech`	TTS to `mp3`, `opus`, `aac`, `flac`, `wav`, or `pcm`; select a profile with `voice` and an engine with `model`
`POST /v1/audio/transcriptions`	STT to `json`, `text`, `verbose_json`, `srt`, or `vtt`
`WS /v1/audio/transcriptions/stream`	Live PCM/WebM transcription with partial, utterance, and session-final events
`GET /.well-known/voicestudio-speech`	Discover HTTP, WebSocket, MCP, and native dictation-control transports
`GET /v1/audio/voices`	List local voice profiles and engines

from openai import OpenAI

client = OpenAI(base_url="http://localhost:3900/v1", api_key="local")

with client.audio.speech.with_streaming_response.create(
    model="tts-1",
    voice="<profile-id>",
    input="Made on my own hardware.",
    response_format="wav",
) as response:
    response.stream_to_file("speech.wav")

# Quick test via cURL
curl http://localhost:3900/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"model": "tts-1", "input": "Made on my own hardware.", "voice": "default", "response_format": "wav"}' \
  --output speech.wav

The bundled Rust control sidecar lets Herdr, coding agents, VS Code, desktop apps, and TUIs trigger the system-wide dictation flow or reuse its native text insertion. See the speech platform guide. The full API reference is in Settings → OpenAPI Reference. For LAN, Tailscale, or proxy access, read API authentication before exposing the backend.

Agent skills

Install the VoiceStudio skills for Claude Code, Codex, Cursor, and other skills.sh-compatible agents:

npx skills add debpalash/VoiceStudio

omnivoice: synthesize speech and transcribe audio through local VoiceStudio.
oss-maintainer: the repository's open-source maintenance workflow.

Model Context Protocol (MCP)

VoiceStudio mounts an MCP server at http://localhost:3900/mcp for Claude Desktop, Cursor, and AI agents:

{
  "mcpServers": {
    "voicestudio": {
      "url": "http://localhost:3900/mcp"
    }
  }
}

For clients requiring stdio transport, use the bundled local shim (docs/mcp.json):

{
  "mcpServers": {
    "voicestudio": {
      "command": "python",
      "args": ["-m", "backend.mcp_shim"],
      "cwd": "/path/to/VoiceStudio"
    }
  }
}

See the MCP guide for tools (generate_speech, clone_voice, transcribe), file streaming modes, and client bindings.

Google Colab

The notebook runs the app and web UI on a Colab GPU. Colab is remote compute, so uploaded audio and project data do not remain local to your machine.

Documentation

Need	Read
Install	macOS · Windows · Linux · Docker
Fix setup	Troubleshooting · model downloads · Hugging Face token
Choose an engine	Engine guides · benchmarks · expressive speech
Tune hardware	Performance · remote workers
Build integrations	Speech platform · Private production API · API auth · MCP · examples
Build VoiceStudio	Contributing · engine acceptance
Track changes	Changelog · roadmap · latest release
Remove everything	Uninstall guide

FAQ

Does it work on Apple Silicon and Intel Macs?

Apple Silicon is supported with MPS and MLX options. Intel Macs cannot run the local backend because current PyTorch wheels are unavailable; they can connect to a remote backend. See macOS installation.

How much VRAM do I need?

A GPU is optional. Use 4 GB VRAM as the minimum for accelerated work and 8 GB+ for the default multi-stage workflow. Large optional engines can require 12 to 16 GB or more. Check the benchmarks and engine guide.

Why does a longer reference clip not always improve the clone?

Cloning is zero-shot: the clip is a prompt, not training data. Use 5 to 15 seconds of one speaker, close to the microphone, without music, noise, or reverb. Match the tone and pace you want in the output. For training, see data preparation and training.

Can I use generated audio commercially?

VoiceStudio's application license does not restrict generated audio, but it does not grant rights under a model's separate terms. The default OmniVoice repository labels its pretrained weights CC-BY-NC and includes a tokenizer under separate community terms. Review the selected model terms before commercial use.

Does VoiceStudio collect data?

Not unless you opt in. Analytics is off by default and skipping consent keeps it off. When enabled, the app sends allowlisted, content-free usage metadata. Text, audio, file names, voices, and projects are excluded. Change this at Settings → Privacy.

How do I remove VoiceStudio and its data?

Use scripts/uninstall.sh on macOS/Linux or scripts\uninstall.ps1 on Windows. Both show a dry run before deletion. See the uninstall guide for every path.

Community and contributing

GitHub Issues for reproducible bugs and feature requests.
Discord for setup help and project discussion.
Good first issues for a scoped starting point.
Contributing guide for setup, tests, and pull requests.

Support development

VoiceStudio is free and has no paid tier. Donations fund development and infrastructure.

Ko-fi · PayPal · Sponsorship details

Responsible use and safety

VoiceStudio enables zero-shot voice cloning and speech generation on personal hardware. Please use it responsibly:

Consent: Only clone or synthesize voices with explicit permission from the speaker.
Audio provenance: VoiceStudio integrates AudioSeal imperceptible watermarking by default to detect and identify synthetic speech without altering sound quality.
Local privacy: For the default local workflow, audio recordings, transcripts, voices, and projects remain strictly on your local disk; data leaves your device only when you explicitly configure remote workers or external ASR endpoints.

License

VoiceStudio is licensed under AGPL-3.0. You may run it, modify it, and use it internally. The application license itself does not restrict selling generated audio, but downloaded model and tokenizer terms may. If you modify VoiceStudio and provide that modified version as a network service, AGPL requires you to offer the corresponding source under the same license. A commercial license for VoiceStudio-owned code is available for proprietary embedding; it does not relicense third-party models. Contact VoiceStudio@palash.dev. See LICENSE-NOTICE.md for the plain-language scope.

Optional engines and downloaded models retain their own licenses. The bundled omnivoice/ Python code is Apache-2.0 upstream; the default downloaded weights and audio tokenizer use separate terms.

Acknowledgments

VoiceStudio builds on OmniVoice, WhisperX, Demucs, Pyannote, CTranslate2, AudioSeal, Tauri, Supertonic, Sherpa-ONNX, GPT-SoVITS, and PocketTTS.

Download VoiceStudio · Star the project · Join Discord

SnailSploit/Claude-Red

claude-red is a curated library of offensive security skills designed for the Claude skills system. Each skill is a structured SKILL.md file that primes Claude with expert-level methodology for a specific attack surface — from SQLi to shellcode, EDR evasion to exploit development.

claude-red

Offensive security skills for Claude — drop-in SKILL.md files that turn Claude into a context-aware red team operator.

Overview • Quickstart • Categories • Skill Index • Roadmap • Contributing

Overview

claude-red is a curated library of offensive security skills for the Claude Skills system. Each skill is a structured SKILL.md file that primes Claude with expert-level methodology for a specific attack surface — from SQL injection to shellcode, EDR evasion to ADCS abuse.

Drop a skill into your Claude environment and it behaves like a domain specialist: it knows the techniques, the tooling, the edge cases, and the escalation paths. Skills load on demand based on conversational triggers — you don't pay context for skills you aren't using.

Use cases: authorized red team engagements, bug bounty triage, security research, CTF preparation, operator training, and methodical attack surface exploration.

Quickstart

Claude Skills System (Recommended)

git clone https://github.com/SnailSploit/claude-red ~/.claude/skills/claude-red

Claude auto-loads matching skills based on conversational triggers (e.g., mentioning SQL injection loads offensive-sqli).

To install a single category:

git clone --filter=blob:none --sparse https://github.com/SnailSploit/claude-red
cd claude-red && git sparse-checkout set Skills/web Skills/active-directory

Claude Code

cat Skills/web/offensive-sqli/SKILL.md | claude --system-file -

cat Skills/active-directory/**/SKILL.md | claude --system-file -

Claude.ai (Manual)

Paste the contents of a SKILL.md into a Project's system prompt or prepend it to your conversation.

Install Script

./install.sh                           # interactive
./install.sh --target ~/.claude/skills # explicit target
./install.sh --category web            # single category

Skill Index

Web Application

Skills/web/

Skill	Description
`offensive-sqli`	SQL injection — error-based, blind, OOB, DB-specific payloads, ORM CVEs
`offensive-xss`	Cross-site scripting — stored, reflected, DOM-based, mutation XSS
`offensive-ssrf`	Server-side request forgery — cloud metadata pivots, filter bypass
`offensive-ssti`	Server-side template injection — engine fingerprinting, RCE chains
`offensive-xxe`	XML external entity — OOB exfiltration, blind XXE techniques
`offensive-idor`	Insecure direct object references — enumeration, authorization bypass
`offensive-file-upload`	File upload — extension bypass, polyglot files, webshell deployment
`offensive-rce`	Remote code execution — command injection, deserialization chains
`offensive-deserialization`	Insecure deserialization — Java, PHP, .NET gadget chains
`offensive-race-condition`	Race conditions — TOCTOU, single-packet attacks, limit bypass
`offensive-request-smuggling`	HTTP request smuggling — CL.TE, TE.CL, H2 desync
`offensive-open-redirect`	Open redirect — OAuth token theft, phishing, SSRF pivots
`offensive-parameter-pollution`	HTTP parameter pollution — WAF bypass, logic confusion
`offensive-graphql`	GraphQL — introspection abuse, batching attacks, alias-based IDOR
`offensive-waf-bypass`	WAF bypass — encoding tricks, chunked transfer, case mutation
`offensive-business-logic`	Business logic — workflow bypass, pricing abuse, multi-step chains

Auth & Identity

Skills/auth/

Skill	Description
`offensive-jwt`	JWT attacks — alg:none, key confusion, secret cracking, claim tampering
`offensive-oauth`	OAuth/OIDC — redirect URI abuse, token leakage, PKCE bypass

Active Directory

Skills/active-directory/

Skill	Description
`offensive-active-directory`	AD methodology — Kerberoast, ASREProast, ACL abuse, ADCS ESC1-15, delegation, hybrid AAD

Wireless

Skills/wireless/

Skill	Description
`offensive-wifi`	802.11 overview — entrypoint for wireless assessments
`offensive-wifi-recon`	Adapter configuration, monitor mode, multi-band airspace mapping
`offensive-wpa2-psk`	WPA2-PSK — handshake capture, PMKID extraction, hashcat cracking
`offensive-wpa3-sae`	WPA3-SAE — transition-mode downgrade, Dragonblood, side-channel attacks
`offensive-wpa-enterprise`	802.1X/EAP — credential relay, evil-twin RADIUS, certificate abuse
`offensive-wps`	WPS — Pixie Dust offline attack, online PIN brute force, vendor PIN prediction
`offensive-evil-twin`	Evil twin — KARMA, Mana, captive portal credential capture, MITM
`offensive-krack-fragattacks`	KRACK and FragAttacks — supplicant vulnerability testing
`offensive-deauth-disassoc`	Deauthentication — targeted/broadcast frames, PMF awareness
`offensive-bluetooth-ble`	Bluetooth LE — GATT enumeration, pairing downgrade, sniffing, MITM
`offensive-bluetooth-classic`	Bluetooth BR/EDR — SDP probing, KNOB attack, BlueBorne, HID spoofing
`offensive-zigbee-thread-matter`	802.15.4 mesh — KillerBee, Touchlink commissioning abuse, ZCL injection
`offensive-z-wave`	Z-Wave — S0 key derivation, S2 commissioning attacks, hub pivots
`offensive-lorawan-sub-ghz`	LoRaWAN and sub-GHz — ABP/OTAA attacks, KeeLoq, fixed-code replay, TPMS

Cloud

Skills/cloud/

Skill	Description
`offensive-cloud`	Multi-cloud — privilege escalation, IMDS abuse, cross-account pivots, CSPM evasion

Mobile

Skills/mobile/

Skill	Description
`offensive-mobile`	Android and iOS — Frida hooking, certificate pinning bypass, storage, biometric flaws

IoT & Embedded

Skills/iot/

Skill	Description
`offensive-iot`	IoT/OT — hardware interfaces, firmware extraction, RTOS, ICS protocols, MQTT/CoAP

Infrastructure & Red Team

Skills/infrastructure/

Skill	Description
`offensive-initial-access`	Initial access — phishing payloads, drive-by delivery, supply chain vectors (TA0001)
`offensive-advanced-redteam`	Full kill chain — C2 infrastructure, OPSEC, lateral movement, persistence
`offensive-edr-evasion`	EDR evasion — userland unhooking, indirect syscalls, PPID spoofing
`offensive-shellcode`	Shellcode — writing, encoding, injection techniques, position-independent code
`offensive-keylogger-arch`	Input capture — keylogger architecture, hooking mechanisms
`offensive-windows-mitigations`	Windows mitigations — ACG, CIG, CFG, CET bypass techniques
`offensive-windows-boundaries`	Windows boundary defeat — sandbox escape, integrity level bypass

Exploit Development

Skills/exploit-dev/

Skill	Description
`offensive-exploit-development`	Exploit development — stack/heap corruption, ROP chains, mitigation bypass
`offensive-exploit-dev-course`	Structured exploit development curriculum
`offensive-basic-exploitation`	Linux binary exploitation — beginner to intermediate, mitigations disabled
`offensive-crash-analysis`	Crash triage — exploitability assessment, root-cause analysis
`offensive-mitigations`	Modern mitigations — ASLR, CFG, CET, PAC analysis and bypass
`offensive-toctou`	TOCTOU race conditions — binary, kernel, web, and container contexts

Fuzzing & Vulnerability Research

Skills/fuzzing/

Skill	Description
`offensive-fuzzing`	Fuzzing — libFuzzer, AFL++, coverage-guided strategies, mutation engines
`offensive-fuzzing-course`	Vulnerability discovery through fuzzing — structured curriculum
`offensive-bug-identification`	Bug identification — code review patterns, static analysis triggers
`offensive-vuln-classes`	Vulnerability taxonomy — real-world examples, classification frameworks

Reconnaissance

Skills/recon/

Skill	Description
`offensive-osint`	OSINT tooling — recon-ng, theHarvester, Maltego, Spiderfoot
`offensive-osint-methodology`	OSINT methodology — structured intelligence collection and analysis

API Security

Skills/api/

Skill	Description
`offensive-api-security`	API testing — OWASP API Top 10, BOLA, BFLA, mass assignment, rate limiting
`offensive-api-abuse`	API business logic — endpoint chaining, batching abuse, webhook hijacking

Container & Kubernetes

Skills/container/

Skill	Description
`offensive-container-escape`	Container breakout — privileged mode, Docker socket, capabilities, runc CVEs
`offensive-k8s-attacks`	Kubernetes attacks — RBAC abuse, etcd access, kubelet API, pod escape, CRD exploitation

CI/CD & Pipeline

Skills/cicd/

Skill	Description
`offensive-cicd-pipeline`	CI/CD exploitation — GitHub Actions injection, Jenkins RCE, GitLab CI abuse
`offensive-cicd-secrets`	CI/CD secrets — environment variable extraction, vault misconfigs, runner token abuse

Cryptography

Skills/crypto/

Skill	Description
`offensive-crypto-attacks`	Cryptographic attacks — padding oracle, ECB manipulation, hash extension, weak PRNG
`offensive-tls-attacks`	TLS/SSL attacks — POODLE, DROWN, Heartbleed, pinning bypass, 0-RTT replay

Privilege Escalation

Skills/privesc/

Skill	Description
`offensive-linux-privesc`	Linux privilege escalation — SUID, capabilities, sudo, cron, kernel exploits
`offensive-windows-privesc`	Windows privilege escalation — Potato family, service misconfigs, DLL hijacking, UAC bypass

Post-Exploitation

Skills/post-exploitation/

Skill	Description
`offensive-lateral-movement`	Lateral movement — PTH, PTT, NTLM relay, WMI/WinRM/DCOM, tunneling
`offensive-persistence`	Persistence — registry, scheduled tasks, WMI subscriptions, ticket forgery, PAM backdoors
`offensive-data-exfiltration`	Data exfiltration — DNS/HTTPS/ICMP tunneling, cloud staging, steganography

Forensics & C2

Skills/forensics/

Skill	Description
`offensive-anti-forensics`	Anti-forensics — log manipulation, timestomping, ADS abuse, memory cleanup
`offensive-c2-frameworks`	C2 tradecraft — Cobalt Strike, Sliver, Mythic, Havoc, redirectors, domain fronting

Supply Chain

Skills/supply-chain/

Skill	Description
`offensive-supply-chain`	Supply chain attacks — dependency confusion, typosquatting, build system compromise
`offensive-dependency-confusion`	Dependency confusion — npm/PyPI/NuGet/Maven namespace attacks, safe PoC methodology

Social Engineering

Skills/social-engineering/

Skill	Description
`offensive-phishing`	Phishing — GoPhish, EvilGinx2, payload delivery, email authentication bypass
`offensive-social-engineering`	Social engineering — pretexting, vishing, smishing, physical SE, USB drops

Network Attacks

Skills/network/

Skill	Description
`offensive-network-attacks`	Network layer attacks — ARP spoofing, LLMNR/NBT-NS poisoning, VLAN hopping, MITM

AI Security

Skills/ai/

Skill	Description
`offensive-ai-security`	AI/ML security — prompt injection, jailbreaking, RAG poisoning, model extraction

Utility

Skills/utility/

Skill	Description
`offensive-fast-checking`	Fast triage — quick-win identification checklists
`offensive-reporting`	Professional reporting — CVSS scoring, evidence standards, executive summaries

Roadmap

The library is being expanded across multiple phases. See CHANGELOG.md for release history.

Phase	Focus	Skills	Status
1	Internal AD/Windows — split into focused skills	+16	Planned
2	Cloud Identity — Entra, ADFS, Okta, M365	+10	Planned
3	Wireless — WPA2/3, EAP, BLE, Zigbee, Z-Wave, LoRa, sub-GHz	+12	Complete
4	IoT — UART/JTAG, flash extraction, fault injection, RTOS, ICS	+10	Planned
5	Web Fundamentals — recon, auth bypass, access control, CSRF, CORS	+8	Planned
6	Web Advanced — proto pollution, SAML, OIDC, WebSocket, SSI/ESI	+10	Planned
7	Documentation and tooling polish	—	Complete
8	New categories — 10 new domains with 20 skills	+20	Complete
9	Deep rewrites — deserialization, GraphQL, advanced red team, SSTI	—	Complete

Target: ~130 skills across 23+ categories.

Contributing

Contributions welcome. See CONTRIBUTING.md for the skill template, frontmatter standard, and review process. Focused, single-surface skills are preferred over monolithic overviews.

License

MIT — use freely, attribution appreciated.

Acknowledgements

Author: Kai Aizen (SnailSploit) — GenAI security research
Original Checklists: Sahar Shlichov — the offensive checklist collection that many of these skills build on
Community: Pull requests and feedback that keep the library aligned with the evolving threat landscape

Give Claude the right skill and it stops being a chatbot — it becomes an operator.

snailsploit.com • GitHub • Research • X

alibaba/open-code-review

Fast, efficient, battle-tested at Alibaba's scale. Hybrid architecture code review tool: deterministic pipelines + LLM Agent, precise line-level comments, built-in multi-language ruleset (NPE, thread-safety, XSS, SQL injection), OpenAI & Anthropic compatible.

https://open-codereview.ai

OpenCodeReview

English | 简体中文 | 日本語 | 한국어 | Русский

What is Open Code Review?

Open Code Review is an AI-powered code review CLI tool. It originated as Alibaba Group's internal official AI code review assistant — over the past two years, it has served tens of thousands of developers and identified millions of code defects. After thorough validation at massive scale, we incubated it into an open source project for the community. Simply configure a model endpoint to get started.

It reads Git diffs, sends changed files to a configurable LLM via an agent with tool-use capabilities, and generates structured review comments with line-level precision. The agent can read full file contents, search the codebase, inspect other changed files for context, and produce deep reviews — not just surface-level diff feedback. Beyond diff review, ocr scan reviews entire files for auditing unfamiliar codebases or directories that have no meaningful diff.

Visit the official website for more details.

Benchmark

Compared to general-purpose agents (Claude Code), Open Code Review achieves significantly higher Precision and F1 with the same underlying model, while consuming only ~1/9 of the tokens and completing reviews faster. Note that its Recall is lower than general-purpose agents — a deliberate trade-off favoring precision over noise.

A real-world code review benchmark built from 50 popular open-source repositories, 200 real Pull Requests, and 10 programming languages — cross-validated by 80+ senior engineers (1,505 annotated ground-truth issues).

Explore the AACR-Bench dataset on Hugging Face.

Metric	What it measures	Why it matters
F1	Harmonic mean of precision and recall	Best single number for overall review quality
Precision	Proportion of reported issues that are real defects	Higher = fewer false alarms to triage
Recall	Proportion of real defects that are found	Higher = fewer issues slip through review
Avg Time	Wall-clock time per review	Matters for CI pipeline latency
Avg Token	Total tokens consumed per review	Directly impacts API cost

Why Open Code Review?

The Problem with General-Purpose Agents

If you've used general-purpose agents like Claude Code with Skills for code review, you've likely encountered these pain points:

Incomplete coverage — On larger changesets, agents tend to "cut corners," selectively reviewing only some files and missing others.
Position drift — Reported issues frequently don't match the actual code location, with line numbers or file references drifting off target.
Unstable quality — Natural-language-driven Skills are hard to debug, and review quality fluctuates significantly with minor prompt variations.

The root cause: a purely language-driven architecture lacks hard constraints on the review process.

Core Design: Deterministic Engineering × Agent Hybrid

Open Code Review's core philosophy is to combine deterministic engineering with an agent, each handling what it does best.

Deterministic Engineering — Hard Constraints

For review steps that must not go wrong, engineering logic — not the language model — guarantees correctness:

Precise file selection — Determines exactly which files need review and which should be filtered, ensuring no important change is missed.
Smart file bundling — Groups related files into a single review unit (e.g., message_en.properties and message_zh.properties are bundled together). Each bundle runs as a sub-agent with isolated context — a divide-and-conquer strategy that stays stable on very large changesets and naturally supports concurrent review.
Fine-grained rule matching — Matches review rules to each file's characteristics, keeping the model's attention sharply focused and eliminating information noise at the source. Compared to purely language-driven rule guidance, template-engine-based rule matching is more stable and predictable.
External positioning and reflection modules — Independent comment-positioning and comment-reflection modules systematically improve both the location accuracy and content accuracy of AI feedback.

Agent — Dynamic Decision-Making

The agent's strengths are concentrated where they matter most — dynamic decisions and dynamic context retrieval:

Scenario-tuned prompts — Prompt templates deeply optimized for code review, improving effectiveness while reducing token consumption.
Scenario-tuned toolset — Distilled from deep analysis of tool-call traces in large-scale production data — including call frequency distributions, per-tool repetition rates, and the impact of new tools on the overall call chain — resulting in a purpose-built toolset that is more stable and predictable for code review than a generic agent toolkit.

How to Use

Prerequisites

Git >= 2.41 — Open Code Review relies on Git for diff generation, code search, and repository operations.

CLI

Install

npm install -g @alibaba-group/open-code-review

After installation, the ocr command is available globally.

For other installation methods (install script, GitHub Release binary, from source), see Installation.

Quick Start

1. Configure LLM

You must configure an LLM before reviewing code, unless you use Delegation Mode.

ocr config provider          # Select a built-in provider or add a custom one
ocr config model             # Pick a model for the active provider

The interactive UI guides you through provider selection, API key entry, and model configuration, then automatically tests connectivity.

For CLI setup, environment variables, custom providers, and other advanced configuration, see Configuration.

2. Review

cd your-project

# Workspace mode — review all staged, unstaged, and untracked changes
ocr review

# Branch range — reviews feature-branch's changes since it diverged from main (merge-base mode)
ocr review --from main --to feature-branch

# Single commit
ocr review --commit abc123

# Resume an interrupted range or commit review
ocr session list
ocr review --from main --to feature-branch --resume <session-id>

# Full-file scan — review whole files instead of a diff (no git history needed)
ocr scan                          # scan the entire repository
ocr scan --path internal/agent    # scan a directory or specific files
ocr scan --resume <session-id>   # resume an interrupted full-file scan

# Save results to a file (recommended for AI host agents)
ocr review --format json --output result.json

# Delegation mode — let your AI coding agent perform the review itself
# OCR handles file selection and rule resolution; no LLM configuration needed
ocr delegate preview
ocr delegate rule src/main.go src/handler.go

Documentation

Full documentation lives at open-codereview.ai/docs:

Quickstart — install and run your first review
Installation — all platforms and package managers
CLI Reference — every command and flag
Review Rules — customize review rules with path filtering and targeting
Configuration — config keys and environment variables
MCP Server — extend the review agent with external tools
Coding Agent Integrations — choose the platform you use
- Claude Code — install a plugin with review slash commands
- Codex — install a plugin with callable review skills
- Cursor — install a plugin with portable review skills
- OpenCode — install native review tools and slash commands
- QCA Forward — run delegation mode with the QCA host model and a ready-to-publish template
- Skill-compatible agents — install the portable agent skill
Review Execution Modes — after integration, choose which LLM performs the review
- Default (OCR-managed) — OCR runs the review using its configured LLM
- Delegation Mode — your coding agent runs the review using its own LLM; no OCR API key required
CI/CD Integration — GitHub Actions, GitLab CI, GitFlic CI, and Gerrit integration
Session Viewer — browse and replay review sessions in browser, mark comments as fixed or ignored and hide them while you work through the findings
Telemetry — OpenTelemetry integration for observability
FAQ — common questions and troubleshooting

Contributing

This project exists thanks to all the people who contribute. See CONTRIBUTING.md for development setup, coding guidelines, and how to submit pull requests.

License

jihe520/MathModelAgent

🤖📐专为数学建模设计的 Agent & skills ,自动完成数学建模，生成一份完整的可以直接提交的论文。 An Agent Designed for Mathematical Modeling ,Automatically complete mathmodel and generate a complete paper ready for submission.

https://mathmodel.top/home

🤖 MathModelAgent 📐

专为数学建模设计的 Agent
自动完成数学建模，生成一份完整的可以直接提交的论文。

简体中文 | English

⬇️ 下载最新桌面版（推荐）

🎨 姊妹项目：sci-box —— 科研图表 & 流程图 SKILL 合集

🖥️ 桌面版（推荐使用方式）

不想折腾环境？直接下载桌面版，开箱即用。

👉 前往 Releases 下载最新版本

桌面版已内置 Claude Code 与全套 MathModelAgent SKILLS，无需安装 Python / Node.js / Redis，也无需手动配置 SKILL，装好填一个模型 API Key 即可开始建模。

系统	下载文件
macOS（Apple 芯片 M 系列）	`mathmodel-<version>-arm64.dmg`
macOS（Intel 芯片）	`mathmodel-<version>-x64.dmg`
Windows 64 位	`mathmodel-<version>-x64.exe`

macOS 安装包已 Developer ID 签名并通过 Apple 公证。

Tip

不确定自己的 Mac 是哪种芯片？点击左上角 → 关于本机，看「芯片」一栏：显示 Apple M 系列选 arm64，显示 Intel 选 x64。

Warning

Windows 安装包当前未签名，首次安装或运行时可能出现 Microsoft Defender SmartScreen 提示，请选择「更多信息」→「仍要运行」，并务必从官方 Releases 页面下载。

安装后应用会自动检查更新（macOS 支持自动更新，Windows 待代码签名证书配置完成后启用）。

如果你是开发者，想自行部署或参与贡献，请继续阅读下方的 SKILLS 与使用教程。

🌟 愿景：

3 天的比赛时间变为 1 小时自动完整一份可以获奖级别的建模论文

✨ 功能特性

🔍 自动分析问题，数学建模，编写代码，纠正错误，撰写论文
💻 Code Interpreter
- local Interpreter: 基于 jupyter , 代码保存为 notebook 方便再编辑
- 云端 code interpreter: E2B 和 daytona
📝 生成一份编排好格式的论文
🤝 multi-agents: 建模手，代码手，论文手等
🔄 multi-llms: 每个 agent 设置不同的、合适的模型
🤖 支持所有模型: litellm
💰 成本低：workflow agentless，不依赖 agent 框架
🧩 自定义模板：prompt inject 为每个 subtask 单独设置需求
🌐 Web Search: Agent 自主搜索互联网获取真实数据（Tavily API）
📚 RAG 知识库: 从本地知识库检索建模方法、代码模板、论文写作参考（ChromaDB + Rerank）
🤝 HIL 人机协作: 关键节点暂停等待用户审批，支持 6 种决策动作（confirm / edit / regenerate / ask / skip / abort）
🛡️ 四层容错: 有限重试 → Fallback Hand Off → Evaluator Shadow Mode → Feedback Rerun

我在平台中托管了一个在线版本，方便使用，欢迎体验：

https://mathmodel.top/home

SKILLS

项目蒸馏成完全由 SKILLS 驱动不再做 Harness 层

Intro

MathModelAgent SKILL —— 直接在 Harness 中驱动的数学建模自动化方案.

💰 开源免费，接入任意模型 完全开源免费，可接入任何模型。

🧠 端到端自动化 从问题分析、建模、编码、绘图到论文排版和验收，一条 /1start-mathmodel 命令全自动完成，中间阶段自动串联，无需人工干预。

📄 17 套 Typst 论文模板 内置中英文主流赛事模板（国赛、华数杯、华为杯、MCM/ICM 等），自动匹配赛事类型，生成排版精良、可直接提交的 PDF 论文。

📐 内置建模知识库 包含完整的建模规范、模型选择决策树（AHP、TOPSIS、ARIMA、GA 等）、常见易错模式和 MCM/ICM 评分标准，每个阶段自动参考，降低模型幻觉。

✅ 9 步自动验收 文本泄漏检测 → 数值一致性校验 → Typst 编译 → PDF 可视化检查，确保论文零低级错误。

🔧 可组合、可扩展 每个阶段是独立 Skill，可单独调用（如只跑分析、只写论文）；模板和知识库可自由扩展；支持 Typst 生态排版。

🎨 姊妹项目：sci-box（科研图表 & 流程图）

科研绘图和流程图模板已独立成仓库 jihe520/sci-box，可单独安装、单独使用：

SKILL	内容
`scibox-figure`	SHAP / ROC / Taylor / 云雨图 / 和弦图 / 环形热图等科研图表复刻模板（Python + Matplotlib，导出 PNG / PDF / SVG）
`scibox-diagram`	可编辑的 draw.io 模板：五层技术路线图、三栏研究框架、三栏流程图、横向任务流水线

npx skills add jihe520/sci-box

Install & Usage

安装 SKILL

npx skills add jihe520/MathModelAgent --all

运行

// claude
claude --dangerously-skip-permissions
claude: /1start-mathmodel 完成这个数学建模任务

// codex
codex --yolo
codex: $start-mathmodel 完成这个数学建模任务

其他命令

/doctor:  检查环境配置
/typst-author: typst 知识

What Can You Contribute?

项目以后只会做 SKLLS 层的迭代和优化，不会再做其他部分。

如果你希望寻找 Agent 开发岗位，你可以研究该项目 Agent 设计并贡献，我会尽量合并.

你能做什么：

优化贡献比赛 typst Template , 你可以找一些 LaTeX 转成 typst
优化 SKILL Workflow
在不同的 Harness 上测试不同的 LLM, 提供反馈和案例放在 example 仓库

Harness SKILL 的优化需要大量黑盒测试和调优.

Thinking

两年前，我做了一个 Mulit-Agent 的数学建模项目并开源出来，收到了社区的欢迎和很多 star, 感谢大家支持。
感谢开源的 latex 模板，我在此基础上转化为 typst 模板
此 SKILL 是一个基础模板，你可以基于此构建更适合你自己的 MathModel SKILL
For Agent DEVs : 两年前，我都是自己实现一套 Agent 框架，现在和以后更多的 Agent 产品直接基于 Harness 如 Codex / Claude Code / Pi + SKILLS 来构建

🚀 后期计划

视频demo

Caution

项目处于实验探索迭代demo阶段，有许多需要改进优化改进地方，我(项目作者)很忙，有时间会优化更新欢迎贡献

📖 使用教程

提供三种部署方式，请选择最适合你的方案：

下载项目

git clone https://github.com/jihe520/MathModelAgent.git # 克隆项目

如果你想运行命令行版本 cli 切换到 master 分支,部署更简单，但未来不会更新

🐳 方案一：Docker 部署（推荐：安全简单）

确保电脑安装了 docker 环境

启动服务

在项目文件夹下运行:

docker-compose up

访问

现在你可以访问：

前端界面：http://localhost:5173
后端API：http://localhost:8000

配置

侧边栏 -> 头像 -> API Key

💻 方案二: 本地部署（推荐项目开发者部署）

确保电脑中安装好 Python, Nodejs, Redis 环境

step1:安装依赖

下载Redis(记得设置环境变量redis_path)

windows 下载地址：https://github.com/tporadowski/redis/releases
linux or mac 下载地址：https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/

安装后端依赖

# ============ 安装依赖 ============
# 1. 切换到 backend 目录
cd backend
# 2. 安装 uv 包管理器（推荐）
pip install uv
# 3. 同步项目依赖
uv sync

# ============ MacOS / Linux 安装命令 ============
# 1. 设置环境变量
export ENV=DEV
export REDIS_URL=redis://localhost:6379/0

# ============ Windows PowerShell 安装命令 ============
# 1. 设置环境变量
$env:ENV="DEV"
$env:REDIS_URL="redis://localhost:6379/0"
# 2. 设置 PowerShell 执行策略策略为 RemoteSigned
Set-ExecutionPolicy RemoteSigned -Scope CurrentUser
# 3. 创建虚拟环境
python -m venv venv

3.安装前端依赖

cd frontend # 切换到 frontend 目录下
npm install -g pnpm
pnpm i

step2:启动项目

windows用户直接双击运行项目中的win_start.bat 即可启动项目

1.启动 Redis

redis-server

2.启动后端

# ============ MacOS / Linux 安装命令 ============
# 1. 激活虚拟环境
source .venv/bin/activate
# 2. 启动后端服务（激活后可直接使用 uvicorn 命令）
uvicorn app.main:app --host 0.0.0.0 --port 8000 --ws-ping-interval 60 --ws-ping-timeout 120 --reload

# ============ Windows PowerShell 安装命令 ============
# 1. 切换到 backend 目录
cd .\backend\
# 2. 激活虚拟环境
.\venv\Scripts\Activate.ps1
# 3. 启动后端服务
uvicorn app.main:app --host 0.0.0.0 --port 8000 --ws-ping-interval 60 --ws-ping-timeout 120 --reload

3.启动前端

cd .\frontend\
pnpm run dev

修改 backend/.env.dev 的环境变量 REDIS_URL

配置API Key

使用 WebUI 侧边栏 -> 头像 -> API Key
修改 backend/.env.dev 文件先将.env.example文件改为.env.dev 然后在.env.dev中修改各 Agent API 配置

🚀 方案三：自动脚本部署（来自社区）

有没有自动部署的脚本？ mmaAutoSetupRun

教程

运行的结果和产生在backend/project/work_dir/xxx/*目录下

notebook.ipynb: 保存运行过程中产生的代码
res.md: 保存最后运行产生的结果为 markdown 格式

需要自定义自定义提示词模板 template ？ Prompt Inject : prompt

网络状况太差难以配置Docker等设置？网络不畅时的配置过程示例：网络环境极差时的MathModelAgent配置过程

⚙️ 新功能配置

MathModelAgent 支持以下可选功能，默认已关闭，开启后未配置外部依赖时自动降级跳过。详见升级说明。

功能	配置开关	说明
Web Search	`SEARCH_ENABLED` + `TAVILY_API_KEY`	Agent 自主联网搜索真实数据（Tavily API）
RAG 知识库	`RAG_ENABLED`	从本地知识库检索建模方法和代码模板（ChromaDB + Rerank）
HIL 人机协作	`HIL_ENABLED`	关键节点暂停等待用户审批，支持 6 种决策动作
Fallback Hand Off	`FALLBACK_*` 系列	主模型故障自动切换备用模型
Evaluator + Feedback	`EVALUATOR_*` 系列	输出质量评估 + 反馈重跑

快速启用 Web Search：注册 Tavily 获取 API Key，在 backend/.env.dev 中设置 TAVILY_API_KEY=tvly-xxx。

🤝 贡献和开发

DeepWiki | Zread

Tip

如果你有跑出来好的案例可以提交 PR 在该仓库下: MathModelAgent-Example

项目处于开发实验阶段（我有时间就会更新），变更较多，还存在许多 Bug，我正着手修复。
希望大家一起参与，让这个项目变得更好
非常欢迎使用和提交 PRs 和 issues
需求参考后期计划

clone 项目后，下载 Todo Tree 插件，可以查看代码中所有具体位置的 todo

.cursor/* 有项目整体架构、rules、mcp 可以方便开发使用

📄 版权License

个人免费使用，请勿商业用途，商业用途联系我（作者）

License

🙏 Reference

Thanks to the following projects:

其他

💖 Sponsor

☕️ 给作者买一杯咖啡

https://linux.do/

企业

302.AI 是一个按用量付费的企业级AI资源平台，提供市场上最新、最全面的AI模型和API，以及多种开箱即用的在线AI应用

用户

danmo-tyc

👥 GROUP

有问题可以进群问

点击链接加入腾讯频道【MathModelAgent】：https://pd.qq.com/s/7rfbai3au

点击链接加入群聊 779159301【MathModelAgent】：https://qm.qq.com/q/Fw2cCJPoki

Discord

Caution

免责声明: 注意，AI 生成仅供参考，目前水平直接参加国赛获奖是不可能的，但我相信 AI 和该项目未来的成长。

tonhowtf/omniget

Download Udemy and Hotmart courses, YouTube videos, music and books — 1,800+ sites, no terminal. Free open-source desktop app for Windows, macOS and Linux, with a built-in course player, PDF/EPUB reader and music library. Powered by yt-dlp. Your files stay on your computer.

https://tonho.wtf

OmniGet

English · Português (BR) · Русский · 简体中文

Download Udemy and Hotmart courses, YouTube, Instagram, X, Pinterest, TikTok and 1,800+ other sites.
Then transcribe, convert, read and study what you saved. One free desktop app, no terminal.

_{Free. Open source under GPL-3.0. No account, no ads, no telemetry on what you download. Your files stay on your computer.}
_{9,500+ GitHub stars. The most starred repository in the udemy-downloader, hotmart-downloader and course-downloader topics.}

Why OmniGet

You bought a course and want it on your disk before the platform pulls it. You keep a yt-dlp cheat sheet because the flags never stick. You have one site for Instagram stories, another for X videos, a Chrome extension for Pinterest, a Python script for subtitles, and none of them remember your login.

OmniGet puts all of that behind one text box. Paste a link, see a preview with quality options, click download. The same window then plays the course, reads the PDF, transcribes the audio and backs up the Pinterest board. yt-dlp and FFmpeg install themselves and stay updated, so there is nothing to configure and no terminal to open.

How it compares

	OmniGet	yt-dlp alone	Single-site web downloaders	Paid course downloaders
Sites	Courses, Instagram, X, Pinterest, Bilibili, Telegram, torrents natively, plus 1,800+ through yt-dlp	1,800+	One	One or two platforms
Setup	Download one file, open it	Python, PATH, FFmpeg, flags	None	Installer, license key
Logged-in content	Cookies from your browser through the extension	Manual `--cookies` export	Rarely	Sometimes
Queue	Resume, retry with backoff, rules, followed channels	One command at a time	No	Varies
After the download	Player, reader, flashcards, notes, 108 tools	Files	Files, often re-encoded	Files
Price and license	Free, GPL-3.0	Free, Unlicense	Free with ads	Subscription

yt-dlp is the engine OmniGet runs on, and OmniGet would not exist without it. If you live in a terminal and only want files, yt-dlp alone is the right tool.

Download and install

Pick your system. Every build is published on the Releases page. Updates arrive inside the app.

System	What to download	Other ways
Windows 10 / 11	`omniget_x.y.z_x64-setup.exe` (installer) `omniget_x.y.z_x64-portable.exe` (no install, runs from anywhere) `omniget_x.y.z_x64_en-US.msi` (for IT deployments)	`winget install -e --id tonhowtf.OmniGet`
macOS 10.15+	`omniget_x.y.z_aarch64.dmg` for Apple Silicon (M1 and later) `omniget_x.y.z_x64.dmg` for Intel Macs	`brew install --cask tonhowtf/tap/omniget`
Linux	`.deb` for Debian and Ubuntu (amd64 and arm64) `.rpm` for Fedora, openSUSE and RHEL family (x86_64 and aarch64) `.AppImage` for everything else (amd64 and aarch64)	AppImage self-updates through the `.zsync` files

The first launch warning, and how to clear it

OmniGet is not signed with a paid certificate, so each system shows a warning the first time. This is normal for open source desktop apps and you handle it once.

Windows. SmartScreen shows a blue box. Click More info, then Run anyway.

macOS. Gatekeeper refuses to open the app and may say it is "damaged". After you drag OmniGet into Applications, open Terminal (Spotlight, type "Terminal") and paste these two lines:

xattr -cr /Applications/omniget.app
codesign --force --deep --sign - /Applications/omniget.app

Then open OmniGet from Launchpad as usual.

Linux, AppImage on Debian 12+ or Ubuntu 24.04+. Those releases ship without FUSE 2, which AppImage needs. If the file fails with a libfuse error, run sudo apt install libfuse2, or launch it with ./omniget.AppImage --appimage-extract-and-run. The .deb avoids this entirely.

Portable mode

Create an empty file named portable.txt (or .portable) next to the Windows .exe and relaunch. Settings, the database, cookies, plugins, caches, yt-dlp and FFmpeg all move to a data folder next to the executable. Nothing touches AppData, so the whole install fits on a USB stick.

Your first download in one minute

Open OmniGet. The setup screen asks for your language and theme, then installs yt-dlp and FFmpeg with one click. yt-dlp is checked against its SHA-256 before it runs.
Copy any link: a YouTube video, an Instagram reel, an X post, a Pinterest board, a magnet, a direct file URL.
Paste it in the box on the home screen. OmniGet detects the site and shows the title, thumbnail and available qualities. Pick one and press Enter.

The Downloads page shows speed, phase and ETA read straight from the downloader, so a stalled download looks stalled instead of frozen at "3 seconds left". Interrupted downloads resume where they stopped. Rate-limited sites get retried with backoff, and connections per site adapt on their own, so YouTube gets up to 16 parallel fragments while a site that answers 429 gets fewer. When a Python 3.10 or newer is present, yt-dlp runs as a zipapp on it and starts in under a second instead of unpacking the bundled binary on every launch.

Skip the window entirely

Copy a link anywhere on your system and press Ctrl+Shift+D (Cmd+Shift+D on macOS). OmniGet reads the clipboard and starts the download in the background. A second hotkey, Ctrl+Shift+M, grabs audio only, so a YouTube link becomes an MP3 without opening anything. It is off until you enable it, and both shortcuts can be rebound in Settings → Downloads → Clipboard & hotkeys.

What OmniGet downloads

OmniGet has native extractors for the platforms people use most, and hands everything else to yt-dlp, which covers roughly 1,800 sites.

Category	Sites and formats
Online courses	Hotmart, Udemy, Kiwify, Rocketseat and Meta-Analysis Academy through the Courses plugin. Every lesson, section selection, attachments, resume where you stopped.
Video and audio	YouTube (videos, playlists, channels, live from start, chapters, SponsorBlock), Instagram, TikTok, X/Twitter, Reddit, Twitch (VODs, clips, live), Vimeo, Bluesky, Threads, Pinterest, Douyin
Bilibili, signed in	4K, HDR, Dolby Vision, Hi-Res lossless and Dolby Atmos according to your subscription. Danmaku comments as XML, ASS or JSON, NFO files for Kodi and Jellyfin, custom naming templates, 11 URL types including bangumi, courses, favorites, watch later and history
Image galleries	Whole galleries and profiles from 250+ sites through gallery-dl (DeviantArt, Pixiv, ArtStation, Flickr, Tumblr, Imgur, Kemono and more)
Bulk	Paste many links or load a `.txt`, download whole subreddits, Reddit and X profiles, Instagram and Pinterest profiles
Files and transfer	`.torrent` files and magnet links with a built-in BitTorrent client, direct HTTP files, HLS and DASH manifests, and person-to-person transfer between two OmniGet installs with a short word code
Telegram	Photos, videos, files and audio from any channel or group you belong to, through the Telegram plugin

Options you set once and forget: default quality, audio-only format (MP3, M4A, Opus, FLAC or WAV), subtitle languages and format (SRT, VTT, ASS, embedded or sidecar), thumbnail and metadata embedding, filename template, organize by platform, skip existing files, split by chapters, speed limit, concurrent downloads, proxy. Rules send a given channel or host to a folder and quality of your choice without asking again. Followed channels are checked in the background and can download new uploads automatically with a tray notification.

The browser extension, step by step

The extension does two jobs. On sites it recognizes (YouTube, Instagram, TikTok, X, Reddit, Twitch, Pinterest, Bluesky, Telegram, Vimeo, Udemy, Hotmart, Rocketseat, Bilibili, SoundCloud) it sends the page to OmniGet with one click or with Alt+O. On any other site it watches network traffic for MP4, HLS, DASH, WebM and audio streams and lists them in its popup. In both cases it forwards your cookies and referer, which is what lets OmniGet download private content you are logged into, such as Instagram stories, a paid course, or a members-only video. Cookies are grouped by real site, so a Brazilian .com.br domain gets its own entry instead of sharing one with every other .com.br site. The popup also has a Force H.264 switch for YouTube, for computers that stutter on VP9 and AV1.

Pick the level that matches how comfortable you are.

Level 1: from inside the app (no downloads, no zip files)

Open OmniGet and install it if you have not. Launch it once.
Go to Settings → Plugins → Browser extension. Click Update / Install next to Chrome. OmniGet copies the extension that ships inside it to a folder and opens that folder for you.
Open Chrome (Edge, Brave and other Chromium browsers work the same way) and type chrome://extensions in the address bar.
Turn on Developer mode with the switch in the top right corner.
Click Load unpacked and pick the folder OmniGet just opened.
The OmniGet icon appears in the toolbar. An options page opens on its own and says it is looking for the app.
Back in OmniGet, still in Settings → Plugins → Browser extension, click Pair extension. Within a few seconds the app says "Extension connected" and the options page turns green. Done.

From now on, visit any supported page and click the icon. The page, its cookies and its title go to OmniGet and the download starts. Your cookies also show up in Settings → Cookies, where the Courses plugin and the Instagram, X and Pinterest tools reuse them.

Level 2: from the release zip

Every release ships omniget-chrome-extension-vX.Y.Z.zip. Download it from the latest release, unzip it, then follow steps 3 to 7 above pointing Load unpacked at the unzipped folder. Use this if you keep the app on one machine and the browser on another, or if you are installing for someone else.

Level 3: Firefox, other browsers and manual pairing

Firefox: Settings → Plugins → Browser extension → Update / Install next to Firefox, then open about:debugging#/runtime/this-firefox, click Load Temporary Add-on and pick manifest.json in the exported folder. Firefox drops temporary add-ons on restart, so repeat this until the extension is published on AMO. Safari is not supported yet because Safari extensions must ship through the App Store.

Manual pairing: if Pair extension times out, open the extension's options page (right-click the icon → Options), then in OmniGet reveal and copy the Pairing token and paste it into the options page. The endpoint URL is detected automatically. The app listens on 127.0.0.1 ports 47720 to 47729 and the token is generated per install, so nothing leaves your machine.

If the extension is installed but OmniGet is closed, clicks fall back to the omniget:// link scheme, which still queues the URL but cannot carry cookies. Tick "Always allow" the first time Chrome asks.

The Tools section: 108 tools in 16 categories

Tools is the part of OmniGet that grew beyond downloading. Each tile is one job: an isolated Rust command with JSON in and JSON out, which is also what lets AI agents drive them through the built-in MCP server. The hub has a search box that understands English and Portuguese ("legenda" finds subtitle tools) and a platform filter, and tools that only run on Windows say so on the tile and stay hidden elsewhere.

Status legend: no mark means ready, beta means it works but has not been tested against every account type, planned means the tile exists so you can see where things are going and does nothing yet.

YouTube (11)

Download video. Paste a link and pick quality, format and subtitles. Same engine as the home screen.
Metadata. Save the info, description and thumbnail without the video.
Thumbnails. Browse every cover image and save it at any resolution.
Subtitles. Download subtitles, or merge two languages into one bilingual file.
Comments and chapters. Fetch comments or chapter markers, filter them, export JSON or CSV.
Live chat. Save the chat replay of a stream as JSON or CSV.
Subtitle workshop. Edit, translate and re-time SRT, VTT and ASS files with a waveform, two-point sync, find and replace, an auto fix, and AI grammar and translation.
SponsorBlock. See sponsor, intro and outro segments and get the yt-dlp flags to skip them.
Dislikes. Likes, dislikes and rating from Return YouTube Dislike.
Real thumbnail. The frames the CDN already has at 25, 50 and 75 percent, instead of the clickbait cover.
Force H.264. A switch in the browser extension that keeps YouTube on H.264 instead of VP9 and AV1, for machines that stutter on newer codecs.

Speech and subtitles (8)

Transcribe. Audio or video to subtitles with whisper.cpp, offline. Models download on demand, Metal acceleration on macOS.
Text to speech. Natural voices from Microsoft Edge, free, with a synced subtitle file.
Translate subtitles. Translate an SRT with your AI provider or a LibreTranslate server, keeping the timing.
Dub from subtitles. Turn an SRT into a voice track that fits each line and optionally replace the video's audio. beta
Clone a voice, Design a voice and Isolate vocals through a VoiceStudio install running on your machine. beta
Dictation. Press a global shortcut, speak, and whisper types the text where your cursor is. beta

Video editing (6)

Cut a clip. Pick a video on disk and cut out a section. The result lands in the downloads queue.
Convert. Change container, codec or resolution, or compress, through the Convert plugin.
Auto captions and Text to speech open the speech tools above.
Record screen. Screen and system audio through FFmpeg, with a replay buffer that saves what just happened. beta
Timeline editor. planned

Instagram (24)

All of these run on your own Instagram session captured by the browser extension, so stories, close friends and your own lists work. Reads are paced and write actions stop on the first sign of a rate limit.

Download post. Photo, video, reel, IGTV or carousel from a link, best quality.
Download many links. Paste a list or a .txt and get everything.
Reel audio. Keep only the sound, as M4A or MP3.
Stories. Download stories, including close friends, without marking them as seen.
Highlights. One highlight or every highlight of a profile.
Who viewed my story. List and export viewers of each active story.
Profile viewer. Bio, counts, HD photo and whether the account follows you.
Profile picture in HD.
Download a profile. All posts, reels, tagged or saved posts, with a limit you choose.
Who doesn't follow back. Compare followers and following, protect accounts with a whitelist, unfollow at a safe pace.
Fans. Accounts that follow you but you don't follow back, with the option to remove them.
Mutuals.
Who unfollowed me. Snapshots of your lists over time show who left and who arrived.
Ghost followers. Followers who never like or comment, and the ones who engage the most.
Whitelist. Accounts never suggested for unfollowing.
Data export. Read Meta's "Download your information" zip offline: pending requests, close friends, blocked and more.
Profile analytics. Engagement rate, cadence, best days and hours, hashtags and top posts of any public profile.
Compare profiles. Up to six profiles side by side.
Hashtag explorer. Post count, recent and top posts, related hashtags, download.
Export comments. All comments of a post as CSV, with filter.
Who liked. List and export the accounts that liked a post.
Giveaway picker. Draw winners among the comments with rules for mentions, keyword and one entry per person.
Publish. Photo, carousel, reel, video or story through your session or the official Graph API. beta
Schedule posts. Queue posts for a date and time. OmniGet publishes them while it is open. beta

X / Twitter (10)

Public data comes through the FxTwitter API without login. Anything private (bookmarks, your follows, Grok on X) uses your X session from the cookie manager.

Download post. Videos, images and GIFs from any post.
Unroll thread. The whole thread on one page, exported as Markdown, HTML or text.
Post to image. A clean PNG card of a post for sharing anywhere.
Profile X-ray. Engagement, best time to post, top posts and hashtags of any account.
Profile media. Every photo and video from a profile, original quality, in one go.
Advanced search. Build queries with X operators, see trends, export results.
Export bookmarks. All bookmarks with folders, to JSON, CSV, Markdown or HTML. beta
Who doesn't follow back. Audit following vs. followers and unfollow safely with a whitelist. beta
Your X archive. Open the data zip offline: stats, top posts, likes and follow lists.
Grok. Ask Grok with live X search or summarize a thread, through the xAI API or your X session. beta

Pinterest (10)

Works without login for anything public. Cookies are only needed for secret boards and for unsaving.

Download pin. Image in original quality, video as MP4, GIF, carousel or story pages.
Board backup. Every pin of a board or section with originals, videos, CSV/JSON and incremental sync.
Profile backup. All public boards of a profile, one folder per board, plus created pins.
Search without AI or ads. Filters that hide AI images, promoted pins and videos, then download.
Similar pins. The "More like this" of any pin, filterable and downloadable.
Find the source. Destination link, creator, dead-link check, Wayback Machine and reverse image search.
Duplicates in a board. Identical and near-identical pins, with optional unsave.
Color palette. Palette of a board or pin as hex, CSS or JSON.
Offline gallery, PDF, CSV. A board as a searchable HTML gallery, a PDF moodboard or a spreadsheet.
Keyword ideas. Search suggestions, refinements and the words top pins use.

Spotify (2)

Themes and colors. Customize the Spotify client with Spicetify themes. beta
Extensions. Install Spicetify extensions and custom apps from its Marketplace. beta

PDF (6)

Merge. Join several PDFs into one, in the order you choose.
Split. Extract pages or break a PDF into parts.
Compress. Shrink a PDF while keeping it readable.
Convert. PDF to images or Word, and back.
OCR. Make scanned PDFs searchable. beta
Safe PDF. Rebuild a PDF from pixels to strip scripts and forms.

Documents (5)

SlideShare to PDF. Every slide at the largest size, assembled into one PDF.
Google Docs export. Public Docs, Slides and Sheets as PDF, DOCX, PPTX or XLSX.
Calameo pages. Save the pages of a Calameo publication as SVG or JPG. beta
Image galleries. Whole galleries and profiles from 250+ sites with gallery-dl.
Scribd. Save readable books as PDF using your own session. planned

Images (3)

Upscale. Real-ESRGAN on any Vulkan GPU, 2x, 3x or 4x. beta
Resize images. Batch resize by width, height, fit or percent, converting the format if you want.
OCR. Copy the text out of images and slides. beta

Files (4)

Duplicates. Find identical files by hash and free space safely.
Bulk rename. Regex, counters and case changes with a preview before applying.
Find files. Instant search with Everything on Windows, Spotlight on macOS or fd on Linux.
Keep awake. Stop the computer from sleeping during long jobs.

Downloads (2)

Accelerated download. Big files with 16 connections, resume and checksum via aria2.
HLS / DASH manifest. Paste a .m3u8 or .mpd with Referer and cookie. FFmpeg saves an MP4.

Phone (1)

Send to phone. Files, links and text to a paired KDE Connect device.

System (9, Windows-only items marked)

Clean caches. Temp files, logs and app caches with rules per operating system. You review the list before anything is deleted.
Disk analyzer. What takes space, as a treemap plus the largest files, with a send-to-trash button.
Startup manager. See what launches with the system and switch items off. beta
Uninstaller. Remove apps and the leftovers they leave behind. beta
Privacy shield. Control Windows telemetry, ad ID and tracking settings. Windows. beta
Harden Windows. Macros, AutoRun, script host, UAC and Defender settings from hardentools, reversible. Windows. beta
Debloat Windows. Remove preinstalled Store apps. Windows. beta
Registry cleaner. Orphaned entries, with a .reg backup before removal. Windows. beta
Software updater. Update programs in bulk through winget, Chocolatey and Scoop. Windows. beta

Automation (1)

Auto clicker. Click at the exact speed you set, with a global hotkey, limits and random ranges. Windows, macOS and Linux. beta

AI (6)

Compare prices. The cost of the same model across providers, with prices from LiteLLM and models.dev.
AI spending. How much OmniGet spent on AI, by day, model and task, from a local ledger.
Local models (Ollama). See, download and remove local models and use them as a free provider.
Humanize text. Rewrite AI-sounding text so it reads like a person wrote it, without changing what it says. Runs on the AI provider you configured. beta
API keys. A local vault for keys and accounts, with a connection test, balance for OpenRouter, DeepSeek, SiliconFlow and New API, and export to Claude Code, Codex, Cherry Studio, opencode or a .env file.
MCP server. OmniGet's tools exposed over the Model Context Protocol on the local bridge, 31 tools behind the same token the extension uses, with ready-made config snippets for Claude Code, Claude Desktop, Cursor, VS Code, Goose and Codex. beta

Every tool that talks to an AI uses the provider you set in Settings → AI: OpenAI, Anthropic, or any OpenAI-compatible local endpoint such as Ollama or LM Studio. The key is stored locally and never logged. The auto clicker, dictation and the replay buffer can each get a global shortcut of their own.

Plugins: Courses, Study, Telegram, Convert

Plugins are separate Rust libraries loaded at startup. OmniGet installs its official set on first launch and updates them by itself. The Marketplace page shows what is installed, what each plugin is allowed to do (events, notifications, settings, download folders, proxy, managed tools, download queue), and lets you hide, disable or uninstall any of them.

Courses

Sign in to Hotmart, Udemy, Kiwify, Rocketseat or Meta-Analysis Academy through a browser window inside the app, with saved cookies from the extension, or with email and password where the platform allows it. OmniGet lists your purchases, opens the course outline so you can tick the sections you want (it tells you how many lectures are DRM-protected and will be skipped), and downloads every lesson and attachment with continuous lecture numbers if you want them. Hotmart uses the current OIDC login flow, so it keeps working after Hotmart's 2026 auth change, and free courses and courses delivered outside Hotmart Club are listed too. Downloaded courses appear in Study automatically.

Study

Study turns the folder of files you downloaded into something you can actually finish.

Library and player. Point Study at your course folders (nothing is copied or moved). The player resumes to the second, and pressing N captures a note at the current timestamp that jumps back there when clicked.
Reader. PDF, EPUB, DJVU, MOBI, AZW3, FB2, CBZ, CBR, TXT, RTF and HTML, with highlights, bookmarks, collections, a focus mode and a paper-like theme. Covers, titles and authors are pulled from the files.
Notes. A Markdown and LaTeX editor with links between pages, a daily journal, templates, tags, a knowledge graph, and export to .md or PDF. Any note can become a flashcard.
Anki. Spaced repetition decks with import from .apkg, .txt and CSV, filtered decks, presets, note types, tags, media, stats and a review log.
Focus. Pomodoro and deep-work timers with daily and weekly targets that pause the player when the session ends.
Progress and achievements. Streaks, daily goals, a year heatmap and local XP with no leaderboard.
Music. Your local library with covers, artists and albums, synced lyrics, favorites, history, playlists, genres, transcoding, and browsers for Spotify, SoundCloud and YouTube Music so playlists and likes sit next to your files.

Sign in with a QR code or your phone number. Browse every channel and group you belong to, filter by photo, video, document or audio, search files, and download one item or the whole chat with a progress list. Videos from channels can be imported straight into the Study library.

Convert

FFmpeg conversions with GPU acceleration where the machine has it: container, codec, resolution, bitrate and compression for video and audio, no internet required.

Built-in chat, off by default

OmniGet ships a Discord-style chat called OmniDisc for servers you host yourself with omnidisc-server. Text channels, direct messages, friends, roles and permissions, pins, search, voice, video and screen sharing. Direct messages and the files sent in them are end-to-end encrypted with MLS, and the key for an encrypted call is derived from the same group, so the server operator cannot listen in. Voice runs in Rust rather than the web view and screen sharing uses the machine's hardware encoder. Files sent through chat are encrypted at rest and deleted from the server after thirty minutes.

It is experimental and does nothing until you turn it on in Settings → Advanced → Chat (OmniDisc) and add a server.

For League of Legends players

A League menu sits in the sidebar. It reads your running League client locally, with no account and no third-party build site, and does nothing until the client is open. If you never play, switch it off in Settings → Advanced → League of Legends and the menu disappears.

Match scouting for both teams with rank, recent form, KDA and the champions each player actually plays. Win probability that shrinks win rates toward the baseline by sample size and always shows a range. Live gold, CS and level for all ten players. Goals per role you can edit. Runes and summoner spells recommended by the client itself, applied in one click and only ever replacing the page OmniGet created. Champion tiers by role. Player search by Riot ID. Opt-in automation: accept matches, pick and ban from your priority list, grab a champion off the ARAM bench. Every automation has its own switch.

New and marked beta or experimental: a Profile tab that edits what other players see (rank shown in chat, challenge medals and title, banner and crest, chat icon, bulk friend management); a skin, chroma and ward roulette that rolls an owned skin the moment you lock in, with rerolls; a champion and lane raffle for when you want the queue to decide, plus an optional random pick in champion select; full match history and ranked stats for any player through the client's own backend gateway, with replay download; and an AI coach that reviews a game, spots trends over your last matches or answers a question about the current champ select, using your configured AI provider and OP.GG's public data.

Everything else in the box

Command palette (Ctrl+K or Cmd+K) that jumps to any page, setting or tool.
Clipboard detection that offers to download a copied link with one click on a toast.
Cookie manager that keeps sessions per site, captured by the extension or imported from a cookies.txt, with a test button per domain.
Video summaries: paste a URL in Settings → AI, OmniGet fetches the subtitles and summarizes them in the length and language you choose.
Send a file to someone: pick a file, share the word code, the other person pastes it in their OmniGet.
Discord Rich Presence showing what you are listening to, watching or reading. Downloads stay private.
Tray icon, start with system, start minimized, prevent sleep during downloads.
Every download keeps the exact yt-dlp command it ran. Open it, edit a flag, retry.
14 themes, including Catppuccin (four flavors), Dracula, One Dark Pro, three e-ink variants and three Nyxvamp variants.
11 languages: English, Portuguese, Spanish, French, Italian, Greek, Russian, Japanese, Persian, Simplified and Traditional Chinese.
Runs on Windows, macOS (Apple Silicon and Intel) and Linux (x86_64 and ARM64).

Privacy and what OmniGet refuses to do

Everything runs on your computer. There is no account, no server of ours in the middle, and no telemetry about what you download. Cookies and API keys live in your local profile. The only network calls OmniGet makes on its own are to the sites you asked it to download from, to GitHub for updates and plugins, and to the AI provider you configured, when you use an AI tool.

OmniGet downloads what your own logged-in session can already open. It does not bypass DRM, break paywalls, or share credentials, and DRM-protected lectures are skipped and reported. You are responsible for respecting copyright and each platform's terms of service. The full text is in the app under About → Terms and ethics.

Frequently asked questions

Is OmniGet free? Yes. GPL-3.0, no paid tier, no ads, no account.

Is OmniGet a yt-dlp GUI? Partly. yt-dlp handles the long tail of sites and OmniGet bundles it, verifies it and updates it. On top of that sit native extractors for courses, Instagram, X, Pinterest, Bilibili, Telegram and torrents, a queue with resume and retry, the Tools section, and the Study library.

Can OmniGet download a Udemy or Hotmart course I bought? Yes. Install the Courses plugin (it comes preinstalled), sign in through the app, pick the course and sections, and download. Lessons and attachments land in a folder per course and appear in Study. Kiwify, Rocketseat and Meta-Analysis Academy work the same way.

Can it download Instagram stories, close friends or highlights? Yes, using your own session captured by the browser extension. Stories are downloaded without being marked as seen.

Can it download an X video, a whole thread or all media from a profile? Yes. Public posts need no login. Bookmarks and your own follow lists need your X session.

Can it back up a Pinterest board in original quality? Yes, including videos, sections, secret boards with cookies, and incremental sync so you only fetch what is new.

Does it resume interrupted downloads? Yes. Partial files are kept and continued, and rate limits trigger retries with backoff.

Which formats can it save? Video as MP4, MKV or WebM. Audio as MP3, M4A, Opus, FLAC or WAV. Subtitles as SRT, VTT or ASS, embedded or beside the file.

Does it need Python, Node or a terminal? No. Download the app, open it, paste a link. The only terminal step is the one-time macOS Gatekeeper fix above.

macOS says the app is damaged. Run the two commands in the first launch section. It happens because the app is not notarized, and it happens once.

Can I transcribe a video to subtitles offline? Yes. Tools → Speech and subtitles → Transcribe uses whisper.cpp locally. Models download on demand.

Can I run it from a USB stick? Yes, on Windows, with a portable.txt file next to the executable.

Which Linux package should I pick? Debian and Ubuntu: .deb. Fedora, openSUSE, RHEL family: .rpm. Anything else: .AppImage. Both x86_64 and ARM64 are published. OmniGet is not on Flathub.

Command line

omniget-cli ships with every release for Windows, macOS (Intel and Apple Silicon) and Linux. Grab omniget-cli-<version>-<target> from the latest release.

omniget info <url>                     # title, formats and size, downloads nothing
omniget download <url> -q 1080 -o ~/Videos
omniget download <url> --audio-only --subs en,pt
omniget batch links.txt -m 3           # one URL per line, 3 at a time
omniget import-cookies cookies.txt     # Netscape format

Build from source

If you only want to use OmniGet, grab a release. To build it you need Rust (the exact toolchain is pinned in rust-toolchain.toml because the plugin ABI depends on it), Node.js 18+ and pnpm.

git clone https://github.com/tonhowtf/omniget.git
cd omniget
pnpm install
pnpm tauri dev

Linux build dependencies

sudo apt-get install -y libwebkit2gtk-4.1-dev build-essential curl wget file libxdo-dev libssl-dev \
  libayatana-appindicator3-dev librsvg2-dev patchelf libasound2-dev libpipewire-0.3-dev clang libclang-dev

Production build:

pnpm tauri build --config '{"bundle":{"createUpdaterArtifacts":false}}'

Releases sign their updater artifacts with a private key only the maintainer holds, so a plain pnpm tauri build stops with "A public key has been found, but no private key". The flag above turns those artifacts off for a local build and changes nothing else.

The plugins live in their own repositories: omniget-plugin-courses, omniget-plugin-telegram, omniget-plugin-convert and omniget-study-release. The registry is omniget-plugins. pnpm plugins:deploy builds the sibling plugin checkouts and copies them into your local data folder.

Stack: Tauri 2, Rust, SvelteKit with Svelte 5, SQLite, yt-dlp, FFmpeg, librqbit for torrents, whisper.cpp, aria2, gallery-dl.

Contributing and translations

Bug reports and pull requests go to Issues and Pull requests. Questions and quick help live on Discord.

Translations are managed on Weblate. Pick your language and translate in the browser. New strings appear there a few hours after they land in main.

OmniGet is built on yt-dlp, FFmpeg, gallery-dl, whisper.cpp, aria2, SponsorBlock, Return YouTube Dislike, FxTwitter, Spicetify and Tauri. Thank you to everyone who maintains them.

Loop, the creature on the home screen, is OmniGet's mascot. Fan art is welcome. The original artwork may not be used commercially or redistributed modified.

Download OmniGet · GPL-3.0

jiji262/douyin-downloader

A practical Douyin downloader for both single-item and profile batch downloads, with progress display, retries, SQLite deduplication, and browser fallback support. 抖音批量下载工具，去水印，支持视频、图集、合集、音乐(原声)。

Douyin Downloader V2.0

中文文档 (Chinese): [README.zh-CN.md](./README.zh-CN.md)

A practical Douyin downloader supporting videos, image-notes, collections, music, favorites collections, and profile batch downloads, with progress display, retries, SQLite deduplication, download integrity checks, and browser fallback support.

Desktop App (Douzy)

A desktop GUI built on the same backend, with dedicated workspaces for Douyin, TikTok, and YouTube. Paste a link to start, sync account content, follow every task, and manage downloaded works in a local archive.

Three platforms: Douyin videos, galleries, profiles, and collections; TikTok videos, photos, and profiles; YouTube videos, Shorts, channels, and playlists
Account content: sync Douyin following, favorites collections, collected series, and likes
Visual workflow: multi-link queue, task status and retry controls, local download archive, filters, and quick re-download

Beta: The desktop app is currently in closed beta. To try it, download the build from the Releases page.

Douyin link download	TikTok download workspace	YouTube workbench

Paste a video, gallery, profile, or collection link and start in one click.	Download public videos, photo posts, and profiles without signing in.	Scan videos, Shorts, channels, and playlists, then configure video, MP3, or subtitle downloads.
Following management	Favorites and likes	Task Center

Sync creators, filter new works, add notes, and download directly from the list.	Browse collected videos, series, and liked works from the current Douyin account.	Track job results, retry failures, and open output folders.

Screenshots were captured from the current desktop main build. Demonstration data is used for privacy.

Feature Overview

Supported

Feature	Description
Single video download	`/video/{aweme_id}`
Single image-note download	`/note/{note_id}` and `/gallery/{note_id}`
Single collection download	`/collection/{mix_id}` and `/mix/{mix_id}`
Single music download	`/music/{music_id}` (prefers direct audio, fallback to first related aweme)
Short link parsing	`https://v.douyin.com/...`, `v.iesdouyin.com`, bare hosts
Profile batch download	`/user/{sec_uid}` + `mode: [post, like, mix, music]`
Logged-in favorites collections	`/user/self?showTab=favorite_collection` + `mode: [collect, collectmix]`
No-watermark preferred	Automatically selects watermark-free video source
Highest-quality selection	Auto-picks highest bitrate from `video.bit_rate` ladder (video + live-photo)
Live stream recording	`live.douyin.com/{room_id}` → FLV/HLS, preserves partial data on stream end
Comments collection	Per-aweme comments (+ optional replies) saved as `*_comments.json`
Hot search + keyword search	`--hot-board [N]` / `--search "keyword"` dumps to JSONL
REST API server mode	`--serve --serve-port 8000` (optional `fastapi + uvicorn`)
Notification push	Bark / Telegram / Webhook on download completion
Extra assets	Cover, music, avatar, JSON metadata
Video transcription	Optional, using OpenAI Transcriptions API
Concurrent downloads	Configurable concurrency, default 5
Retry with backoff	Exponential backoff (1s, 2s, 5s)
Rate limiting	Default 2 req/s
SQLite history	Records download metadata; does not decide incremental skips
Incremental downloads	Disk-based skip/redownload via `increase.post/like/mix/music`
Time filters	`start_time` / `end_time`
Browser fallback	Launches browser when pagination is blocked, manual CAPTCHA supported
Download integrity check	Content-Length validation, auto-cleanup of incomplete files
Progress display	Rich progress bars, supports `progress.quiet_logs` quiet mode
Docker deployment	Dockerfile included
CI/CD	GitHub Actions for testing and linting

Current Limitations

Browser fallback is fully validated for post; like/mix/music currently relies on API pagination
number.allmix / increase.allmix are retained as compatibility aliases and normalized to mix
collect / collectmix currently work for the account represented by the logged-in cookies only
collect / collectmix must be used alone and cannot be combined with post / like / mix / music
increase currently applies to post / like / mix / music; favorites collection modes do not support incremental stop
Live stream recording saves FLV natively; HLS sources only save the playlist (use ffmpeg for playable output)
The webcast room endpoint is not verified against every live scenario — treat as experimental

Quick Start

1) Requirements

Python 3.8+
macOS / Linux / Windows

2) Install dependencies

pip install -r requirements.txt

For browser fallback and automatic cookie capture:

pip install playwright
python -m playwright install chromium

3) Copy config file

cp config.example.yml config.yml

4) Get cookies (recommended: automatic)

python -m tools.cookie_fetcher --config config.yml

After logging into Douyin, return to the terminal and press Enter. Cookies will be written to your config automatically.

5) Docker deployment (optional)

docker build -t douyin-downloader .
docker run -v $(pwd)/config.yml:/app/config.yml -v $(pwd)/Downloaded:/app/Downloaded douyin-downloader

Minimal Working Config

link:
  - https://www.douyin.com/user/MS4wLjABAAAAxxxx

path: ./Downloaded/
mode:
  - post

number:
  post: 0
  collect: 0
  collectmix: 0

thread: 5
retry_times: 3
proxy: ""
database: true
database_path: dy_downloader.db

progress:
  quiet_logs: true

cookies:
  msToken: ""
  ttwid: YOUR_TTWID
  odin_tt: YOUR_ODIN_TT
  passport_csrf_token: YOUR_CSRF_TOKEN
  sid_guard: ""

browser_fallback:
  enabled: true
  headless: false
  max_scrolls: 240
  idle_rounds: 8
  wait_timeout_seconds: 600

transcript:
  enabled: false
  model: gpt-4o-mini-transcribe
  output_dir: ""
  response_formats: ["txt", "json"]
  api_url: https://api.openai.com/v1/audio/transcriptions
  api_key_env: OPENAI_API_KEY
  api_key: ""

Usage

Run with a config file

python run.py -c config.yml

Append CLI arguments

python run.py -c config.yml \
  -u "https://www.douyin.com/video/7604129988555574538" \
  -t 8 \
  -p ./Downloaded

Arguments

Argument	Description
`-u, --url`	Append download link(s), can be repeated
`-c, --config`	Specify config file (default: `config.yml`)
`-p, --path`	Specify download directory
`-t, --thread`	Specify concurrency
`--show-warnings`	Show warning/error logs
`-v, --verbose`	Show info/warning/error logs
`--hot-board [N]`	Fetch Douyin hot search board and write JSONL; optional top-N
`--search KEYWORD`	Search videos by keyword, write JSONL
`--search-max N`	Max items for `--search` (default 50)
`--serve`	Run as REST API server (requires `pip install fastapi uvicorn`)
`--serve-host HOST`	REST server listen host (default 127.0.0.1)
`--serve-port PORT`	REST server listen port (default 8000)
`--version`	Show version number

Typical Scenarios

Download one video

link:
  - https://www.douyin.com/video/7604129988555574538

Download one image-note

link:
  - https://www.douyin.com/note/7341234567890123456

Download a collection

link:
  - https://www.douyin.com/collection/7341234567890123456

Download a music track

link:
  - https://www.douyin.com/music/7341234567890123456

Batch download a creator's posts

link:
  - https://www.douyin.com/user/MS4wLjABAAAAxxxx
mode:
  - post
number:
  post: 50

Batch download a creator's liked posts

link:
  - https://www.douyin.com/user/MS4wLjABAAAAxxxx
mode:
  - like
number:
  like: 0    # 0 means download all

Download multiple modes at once

link:
  - https://www.douyin.com/user/MS4wLjABAAAAxxxx
mode:
  - post
  - like
  - mix
  - music

Cross-mode deduplication: the same aweme_id won't be downloaded twice across different modes.

Download logged-in favorites collection items

link:
  - https://www.douyin.com/user/self?showTab=favorite_collection
mode:
  - collect
number:
  collect: 0

Download logged-in collected mixes

link:
  - https://www.douyin.com/user/self?showTab=favorite_collection
mode:
  - collectmix
number:
  collectmix: 0

Record a live stream (experimental)

link:
  - https://live.douyin.com/123456789   # or /follow/live/{room_id}
live:
  max_duration_seconds: 3600   # 0 = record until broadcaster ends
  chunk_size: 65536
  idle_timeout_seconds: 30

The recorder saves an FLV file under Downloaded/{author}/live/ plus a *_room.json metadata snapshot. If the broadcaster ends the stream, network goes idle, or you Ctrl+C, any already-recorded bytes are preserved (the .tmp file is promoted to the final file).

Collect comments per aweme

comments:
  enabled: true
  include_replies: false   # true will fetch each comment's second-level replies (extra API calls)
  max_comments: 500        # 0 = no cap
  page_size: 20

Generates a {date}_{title}_{aweme_id}_comments.json next to the media file.

Dump the hot search board

python run.py --hot-board 30 -p ./Downloaded
# Output: ./Downloaded/hot_board/20260424_221530.jsonl

Search by keyword

python run.py --search "猫咪" --search-max 100 -p ./Downloaded
# Output: ./Downloaded/search/猫咪_20260424_221530.jsonl

Run as REST API server

pip install fastapi uvicorn       # one-time optional dep
python run.py --serve --serve-port 8000

Endpoints:

Method	Path	Description
POST	`/api/v1/download`	Submit `{"url": "..."}`, returns `{job_id, status}`
GET	`/api/v1/jobs/{job_id}`	Get a specific job's status/counts
GET	`/api/v1/jobs`	List recent jobs (TTL + capacity capped)
GET	`/api/v1/health`	Health probe

Finished jobs are pruned by TTL (default 24h) and max-jobs (default 500) — in-flight jobs are never pruned. Configure via server.max_jobs / server.job_ttl_seconds.

Send a notification on completion

notifications:
  enabled: true
  on_success: true
  on_failure: true
  providers:
    - type: bark
      url: https://api.day.app/YOUR_DEVICE_KEY
      sound: bell
    - type: telegram
      bot_token: "123456:ABC..."
      chat_id: "987654321"
    - type: webhook                 # works with 企业微信/飞书/钉钉 bot URLs too
      url: https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=xxx
      extra_body:
        msgtype: text

All enabled providers are notified in parallel; a failing provider never blocks the download flow.

Incremental download (disk-based)

increase:
  post: true

With true, the downloader skips an item only when its non-empty primary media already exists under the current download directory. Deleting the media file makes the next run download it again; SQLite history does not affect this decision.

Set a mode to false to redownload and atomically replace existing files within the current number/date/media filters.

Full crawl (no item limit)

number:
  post: 0

Optional Feature: Video Transcription (`transcript`)

Current behavior applies to video items only (image-note items do not generate transcripts).

1) Enable in config

transcript:
  enabled: true
  model: gpt-4o-mini-transcribe
  output_dir: ""        # empty: same folder as video; non-empty: mirrored to target dir
  response_formats:
    - txt
    - json
  api_key_env: OPENAI_API_KEY
  api_key: ""           # can be set directly, or via environment variable

Recommended to provide key through environment variable:

export OPENAI_API_KEY="sk-xxxx"

2) Output files

When enabled, it generates:

xxx.transcript.txt
xxx.transcript.json

If database: true, job status is also recorded in SQLite table transcript_job (success/failed/skipped).

Testing

Recommended:

python3 -m pytest -q

Plain pytest is also supported now:

pytest -q

Key Config Fields

Field	Description
`mode`	Supports `post`/`like`/`mix`/`music`; logged-in favorites mode additionally supports standalone `collect`/`collectmix`
`number.post/like/mix/music/collect/collectmix`	Per-mode download limit, 0 = unlimited
`increase.post/like/mix/music`	`true`: skip existing primary media on disk; `false`: redownload and overwrite current scope
`start_time` / `end_time`	Time filter (format: `YYYY-MM-DD`)
`folderstyle`	Create per-item subdirectories
`browser_fallback.*`	Browser fallback for `post` when pagination is restricted
`progress.quiet_logs`	Quiet logs during progress stage
`transcript.*`	Optional transcription after video download
`comments.*`	Per-aweme comments collection (opt-in)
`live.*`	Live stream recording options (max_duration_seconds / chunk_size / idle_timeout_seconds)
`notifications.*`	Bark/Telegram/Webhook push on completion
`server.*`	REST API server tuning (max_jobs, job_ttl_seconds)
`proxy`	Optional HTTP/HTTPS proxy setting
`database`	Enable SQLite deduplication and history
`database_path`	SQLite path, default is `dy_downloader.db` in the current working directory
`thread`	Concurrent download count
`retry_times`	Retry count on failure

Output Structure

Default with folderstyle: true and database_path: dy_downloader.db:

workspace/
├── config.yml
├── dy_downloader.db          # default location when database: true
└── Downloaded/
    ├── download_manifest.jsonl
    ├── hot_board/                # when --hot-board is used
    │   └── 20260424_221530.jsonl
    ├── search/                   # when --search is used
    │   └── 猫咪_20260424_221530.jsonl
    └── AuthorName/
        ├── post/
        │   └── 2024-02-07_Title_aweme_id/
        │       ├── ...mp4
        │       ├── ..._cover.jpg
        │       ├── ..._music.mp3
        │       ├── ..._data.json
        │       ├── ..._avatar.jpg
        │       ├── ..._comments.json    # when comments.enabled
        │       ├── ...transcript.txt
        │       └── ...transcript.json
        ├── like/
        │   └── ...
        ├── mix/
        │   └── ...
        ├── music/
        │   └── ...
        ├── collect/
        │   └── ...
        ├── collectmix/
        │   └── ...
        └── live/                 # when recording live streams
            └── 2026-04-24_2215_LiveTitle_RoomId/
                ├── ...flv
                └── ..._room.json

Re-downloading Content

The program uses a database record + local file dual check to decide whether to skip already-downloaded content. To force re-download, you need to clean up accordingly:

Re-download a specific item

# Delete local files (folder name contains the aweme_id)
rm -rf Downloaded/AuthorName/post/*_<aweme_id>/

# Delete database record
sqlite3 dy_downloader.db "DELETE FROM aweme WHERE aweme_id = '<aweme_id>';"

Re-download all items from a specific author

rm -rf Downloaded/AuthorName/
sqlite3 dy_downloader.db "DELETE FROM aweme WHERE author_name = 'AuthorName';"

Full reset (re-download everything)

rm -rf Downloaded/
rm dy_downloader.db

Note: Deleting only the database but keeping files will NOT trigger re-download — the program scans local filenames for aweme_id to detect existing downloads. Deleting only files but keeping the database WILL trigger re-download (the program treats "in DB but missing locally" as needing retry).

FAQ

1) Why do I only get around 20 posts?

This is a common pagination risk-control behavior. Make sure:

browser_fallback.enabled: true
browser_fallback.headless: false
complete verification manually in the browser popup, and do not close it too early

2) Why is the progress output noisy/repeated?

By default, progress.quiet_logs: true suppresses logs during progress stage.
Use --show-warnings or -v temporarily when debugging.

3) What if cookies are expired?

Run:

python -m tools.cookie_fetcher --config config.yml

4) Why are transcript files not generated?

Check in order:

whether transcript.enabled is true
whether downloaded items are videos (image-notes are not transcribed)
whether OPENAI_API_KEY (or transcript.api_key) is valid
whether response_formats includes txt or json

5) How to view download history?

sqlite3 dy_downloader.db "SELECT aweme_id, title, author_name, datetime(download_time, 'unixepoch', 'localtime') FROM aweme ORDER BY download_time DESC LIMIT 20;"

Community Group

点击链接加入群聊【QQ群】：https://qm.qq.com/q/9xoNt8Wzv4

Disclaimer

This project is for technical research, learning, and personal data management only. Please use it legally and responsibly:

Do not use it to infringe others' privacy, copyright, or other legal rights
Do not use it for any illegal purpose
Users are solely responsible for all risks and liabilities arising from usage
If platform policies or interfaces change and features break, this is a normal technical risk

By continuing to use this project, you acknowledge and accept the statements above.

License

This project is licensed under the MIT License. See LICENSE for details.

Friendly Links

LINUX DO

Swordfish90/cool-retro-term

A good looking terminal emulator which mimics the old cathode display...

cool-retro-term

> Default Amber	C:\ IBM DOS	$ Default Green

Description

cool-retro-term is a terminal emulator which mimics the look and feel of the old cathode tube screens. It has been designed to be eye-candy, customizable, and reasonably lightweight.

It uses the QML port of qtermwidget (Konsole): https://github.com/Swordfish90/qmltermwidget.

This terminal emulator works under Linux and macOS and requires Qt6.

Settings such as colors, fonts, and effects can be accessed via context menu.

Screenshots

Install

If you want to get a hold of the latest version, just go to the Releases page and grab the latest AppImage (Linux) or dmg (macOS).

Alternatively, most distributions such as Ubuntu, Fedora or Arch already package cool-retro-term in their official repositories.

Building

Check out the wiki and follow the instructions on how to build it on Linux and macOS.

huggingface/transformers

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

https://huggingface.co/transformers

English | 简体中文 | 繁體中文 | 한국어 | Español | 日本語 | हिन्दी | Русский | Português | తెలుగు | Français | Deutsch | Italiano | Tiếng Việt | العربية | اردو | বাংলা | فارسی | Română | Türkçe

State-of-the-art pretrained models for inference and training

Transformers acts as the model-definition framework for state-of-the-art machine learning with text, computer vision, audio, video, and multimodal models, for both inference and training.

It centralizes the model definition so that this definition is agreed upon across the ecosystem. transformers is the pivot across frameworks: if a model definition is supported, it will be compatible with the majority of training frameworks (Axolotl, Unsloth, DeepSpeed, FSDP, PyTorch-Lightning, ...), inference engines (vLLM, SGLang, TGI, ...), and adjacent modeling libraries (llama.cpp, mlx, ...) which leverage the model definition from transformers.

We pledge to help support new state-of-the-art models and democratize their usage by having their model definition be simple, customizable, and efficient.

There are over 1M+ Transformers model checkpoints on the Hugging Face Hub you can use.

Explore the Hub today to find a model and use Transformers to help you get started right away.

Installation

Transformers works with Python 3.10+, and PyTorch 2.5+.

Create and activate a virtual environment with venv or uv, a fast Rust-based Python package and project manager.

# venv
python -m venv .my-env
source .my-env/bin/activate
# uv
uv venv .my-env
source .my-env/bin/activate

Install Transformers in your virtual environment.

# pip
pip install "transformers[torch]"

# uv
uv pip install "transformers[torch]"

Install Transformers from source if you want the latest changes in the library or are interested in contributing. However, the latest version may not be stable. Feel free to open an issue if you encounter an error.

git clone https://github.com/huggingface/transformers.git
cd transformers

# pip
pip install '.[torch]'

# uv
uv pip install '.[torch]'

Quickstart

Get started with Transformers right away with the Pipeline API. The Pipeline is a high-level inference class that supports text, audio, vision, and multimodal tasks. It handles preprocessing the input and returns the appropriate output.

Instantiate a pipeline and specify model to use for text generation. The model is downloaded and cached so you can easily reuse it again. Finally, pass some text to prompt the model.

from transformers import pipeline

pipeline = pipeline(task="text-generation", model="Qwen/Qwen2.5-1.5B")
pipeline("the secret to baking a really good cake is ")
[{'generated_text': 'the secret to baking a really good cake is 1) to use the right ingredients and 2) to follow the recipe exactly. the recipe for the cake is as follows: 1 cup of sugar, 1 cup of flour, 1 cup of milk, 1 cup of butter, 1 cup of eggs, 1 cup of chocolate chips. if you want to make 2 cakes, how much sugar do you need? To make 2 cakes, you will need 2 cups of sugar.'}]

To chat with a model, the usage pattern is the same. The only difference is you need to construct a chat history (the input to Pipeline) between you and the system.

Tip

You can also chat with a model directly from the command line, as long as transformers serve is running.

transformers chat Qwen/Qwen2.5-0.5B-Instruct

import torch
from transformers import pipeline

chat = [
    {"role": "system", "content": "You are a sassy, wise-cracking robot as imagined by Hollywood circa 1986."},
    {"role": "user", "content": "Hey, can you tell me any fun things to do in New York?"}
]

pipeline = pipeline(task="text-generation", model="meta-llama/Meta-Llama-3-8B-Instruct", dtype=torch.bfloat16, device_map="auto")
response = pipeline(chat, max_new_tokens=512)
print(response[0]["generated_text"][-1]["content"])

Expand the examples below to see how Pipeline works for different modalities and tasks.

Automatic speech recognition

from transformers import pipeline

pipeline = pipeline(task="automatic-speech-recognition", model="openai/whisper-large-v3")
pipeline("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac")
{'text': ' I have a dream that one day this nation will rise up and live out the true meaning of its creed.'}

Image classification

from transformers import pipeline

pipeline = pipeline(task="image-classification", model="facebook/dinov2-small-imagenet1k-1-layer")
pipeline("https://huggingface.co/datasets/Narsil/image_dummy/raw/main/parrots.png")
[{'label': 'macaw', 'score': 0.997848391532898},
 {'label': 'sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita',
  'score': 0.0016551691805943847},
 {'label': 'lorikeet', 'score': 0.00018523589824326336},
 {'label': 'African grey, African gray, Psittacus erithacus',
  'score': 7.85409429227002e-05},
 {'label': 'quail', 'score': 5.502637941390276e-05}]

Visual question answering

from transformers import pipeline

pipeline = pipeline(task="visual-question-answering", model="Salesforce/blip-vqa-base")
pipeline(
    image="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-few-shot.jpg",
    question="What is in the image?",
)
[{'answer': 'statue of liberty'}]

Why should I use Transformers?

Easy-to-use state-of-the-art models:
- High performance on natural language understanding & generation, computer vision, audio, video, and multimodal tasks.
- Low barrier to entry for researchers, engineers, and developers.
- Few user-facing abstractions with just three classes to learn.
- A unified API for using all our pretrained models.
Lower compute costs, smaller carbon footprint:
- Share trained models instead of training from scratch.
- Reduce compute time and production costs.
- Hundreds of model architectures with 1M+ pretrained checkpoints across all modalities.
Choose the right framework for every part of a model's lifetime:
- Train state-of-the-art models in 3 lines of code.
- Move a single model between PyTorch/JAX/TF2.0 frameworks at will.
- Pick the right framework for training, evaluation, and production.
Easily customize a model or an example to your needs:
- We provide examples for each architecture to reproduce the results published by its original authors.
- Model internals are exposed as consistently as possible.
- Model files can be used independently of the library for quick experiments.

When shouldn't I use Transformers?

This library is not a modular toolbox of building blocks for neural nets. The code in the model files is not refactored with additional abstractions on purpose, so that researchers can quickly iterate on each of the models without diving into additional abstractions/files.
The training API is optimized to work with PyTorch models provided by Transformers. For generic machine learning loops, you should use another library like Accelerate.
The example scripts are only examples. They may not necessarily work out-of-the-box on your specific use case and you'll need to adapt the code for it to work.

100 projects using Transformers

Transformers is more than a toolkit to use pretrained models, it's a community of projects built around it and the Hugging Face Hub. We want Transformers to enable developers, researchers, students, professors, engineers, and anyone else to build their dream projects.

In order to celebrate Transformers 100,000 stars, we wanted to put the spotlight on the community with the awesome-transformers page which lists 100 incredible projects built with Transformers.

If you own or use a project that you believe should be part of the list, please open a PR to add it!

Example models

You can test most of our models directly on their Hub model pages.

Expand each modality below to see a few example models for various use cases.

Audio

Audio classification with CLAP
Automatic speech recognition with Parakeet, Whisper, GLM-ASR and Moonshine-Streaming
Keyword spotting with Wav2Vec2
Speech to speech generation with Moshi
Text to audio with MusicGen
Text to speech with CSM

Computer vision

Automatic mask generation with SAM
Depth estimation with DepthPro
Image classification with DINO v2
Keypoint detection with SuperPoint
Keypoint matching with SuperGlue
Object detection with RT-DETRv2
Pose Estimation with VitPose
Universal segmentation with OneFormer
Video classification with VideoMAE

Multimodal

Audio or text to text with Voxtral, Audio Flamingo
Document question answering with LayoutLMv3
Image or text to text with Qwen-VL
Image captioning BLIP-2
OCR-based document understanding with GOT-OCR2
Table question answering with TAPAS
Unified multimodal understanding and generation with Emu3
Vision to text with Llava-OneVision
Visual question answering with Llava
Visual referring expression segmentation with Kosmos-2

NLP

Masked word completion with ModernBERT
Named entity recognition with Gemma
Question answering with Mixtral
Summarization with BART
Translation with T5
Text generation with Llama
Text classification with Qwen

Citation

We now have a paper you can cite for the 🤗 Transformers library:

@inproceedings{wolf-etal-2020-transformers,
    title = "Transformers: State-of-the-Art Natural Language Processing",
    author = "Thomas Wolf and Lysandre Debut and Victor Sanh and Julien Chaumond and Clement Delangue and Anthony Moi and Pierric Cistac and Tim Rault and Rémi Louf and Morgan Funtowicz and Joe Davison and Sam Shleifer and Patrick von Platen and Clara Ma and Yacine Jernite and Julien Plu and Canwen Xu and Teven Le Scao and Sylvain Gugger and Mariama Drame and Quentin Lhoest and Alexander M. Rush",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations",
    month = oct,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2020.emnlp-demos.6/",
    pages = "38--45"
}

	Bloome lets multiple AI agents (Claude, ChatGPT, DeepSeek, and more) collaborate in one conversation for agentic video pipelines. It has zero setup, runs in the cloud, works on web and mobile, and lets you share a configured agent with your whole team. Try Bloome.
	Atlas Cloud is a full-modal AI inference platform that gives developers a single AI API for video generation, image generation, and LLM APIs. Instead of managing multiple vendor integrations, you connect once and get unified access to 300+ curated models across all modalities. Check out Atlas Cloud's new coding plan promotion for more budget-friendly API access.

Category	Skills	Focus
Web Application	16	OWASP Top 10, business logic, advanced web vulnerability classes
Auth & Identity	2	JWT exploitation, OAuth/OIDC abuse
Active Directory	1	On-prem AD attack methodology
Wireless	14	802.11, WPA2/3, EAP, WPS, evil-twin, BLE, Zigbee, Z-Wave, LoRa, sub-GHz
Cloud	1	AWS, Azure, GCP attack paths
Mobile	1	Android and iOS application testing
IoT & Embedded	1	Hardware, firmware, RTOS, ICS/OT
Infrastructure & Red Team	7	Initial access, EDR evasion, advanced red team operations, Windows internals
Exploit Development	6	Stack/heap corruption, ROP, mitigations, crash analysis, TOCTOU
Fuzzing & Vulnerability Research	4	libFuzzer, AFL++, coverage-guided fuzzing, vulnerability taxonomy
Reconnaissance	2	OSINT tooling and structured intelligence collection
API Security	2	REST/gRPC/WebSocket testing, business logic abuse
Container & Kubernetes	2	Container escape, Kubernetes cluster exploitation
CI/CD & Pipeline	2	Pipeline exploitation, secrets extraction
Cryptography	2	Cryptographic implementation attacks, TLS/SSL
Privilege Escalation	2	Linux and Windows privilege escalation
Post-Exploitation	3	Lateral movement, persistence mechanisms, data exfiltration
Forensics & C2	2	Anti-forensics tradecraft, C2 framework operations
Supply Chain	2	Supply chain attacks, dependency confusion
Social Engineering	2	Phishing campaigns, physical/vishing/smishing
Network Attacks	1	Layer 2/3 attacks, MITM, protocol poisoning
AI Security	1	Prompt injection, jailbreaking, RAG poisoning
Utility	2	Fast triage checklists, professional reporting

GitHub Daily Trending Repositories

JustVugg/colibri

See it running

The research mission

Core techniques and measured findings

Open hypotheses, experiments, and how to help

The idea

Local cluster mode

How it works

The per-token path

One memory hierarchy instead of one memory requirement

Dual-SSD: two copies of the model, twice the read bandwidth

Never wait for the disk twice

Faithful model, compressed state

Speculative decoding, honestly

What it achieves

Get started

1. Get colibri

2. Get the model

Other supported models

3. Run it

The same commands run any of the models

4. Go deeper

DeepSeek V4

What's next

Supporting the project

Repo layout

Why "colibrì"

Acknowledgements

License

ever-co/ever-gauzy

Ever Gauzy Platform

💡 What's New

🌟 What is it

✨ Features

🌼 Screenshots

Web UI

Desktop Timer UI (Standard)

Desktop Timer UI (Expanded)

🔗 Links

📊 Activity

💻 Demo, Downloads, Testing and Production

Demo

Downloads

Production (SaaS)

Staging

Server & Desktop Apps

🧱 Technology Stack and Requirements

See also README.md and CREDITS.md files in relevant folders for lists of libraries and software included in the Platform, information about licenses, and other details

📄 Documentation

🚀 Quick Start

Super Quick Start

Run with Docker Compose

Demo

Production

Build

Quick Start to manually build & run

Required

Optional / Recommended for Production

🏭 Production

General information

Kubernetes

DigitalOcean App Platform

Virtual Instances / Droplets (via SSH)

Pulumi

🚗 Other self-hosting options

Deploy on RepoCloud

💌 Contact Us

🔐 Security

🛡️ License

The default Ever® Gauzy™ Platform license, without a valid Ever® Gauzy™ Platform Enterprise or Ever® Gauzy™ Platform Small Business License agreement, is the Ever® Gauzy™ Platform Community Edition License

Please see LICENSE for more information on licenses. You can also compare our offering

™️ Trademarks

🍺 Contribute

💪 Thanks to our Contributors

⭐ Star History

❤️ Powered By

©️ Copyright

Copyright © 2019-present, Ever Co. LTD. All rights reserved

🔥 P.S