GitHub Python Weekly Trending Repositories

semantica-agi/semantica

Graph-Native Infrastructure for Context and Accountable AI Systems

The Open Source Palantir for AI Agents

Ingest your enterprise data, extract what matters, build a Context Graph and knowledge graph (KG), and run graph analytics and causal reasoning over all of it, with full decision provenance baked in. Explainable, traceable, and trustworthy by design.

Decision Intelligence · Context Management · Deterministic Reasoning · Ontology Management · Knowledge Modeling · End-to-End Traceability

Open Source · Self-Hostable · Auditable · Governed · Zero Vendor Lock-In

Polyglot Graph Storage · RDF & LPG Support · W3C Standards · Interoperable

Built for High-Stakes, Regulated Domains

pip install semantica

Knowledge Explorer · Context Graphs · Reasoning Engine · Decision Intelligence · Ontology Hub

▶ Watch the full platform walkthrough

Most AI agents act without a trail. They store embeddings, not meaning: context that can't be explained, decisions that can't be audited. In lending, that gap is a compliance exposure, not an inconvenience: an underwriting agent's approval has to survive a regulator's "why" months later.

Semantica sits underneath your LLM, vector store, and agent framework as a deterministic infrastructure layer: no LLM required for graph construction, reasoning, or provenance.

Who it's for:

AI/ML platform teams shipping agents that make consequential decisions and need structured, queryable context built from fragmented raw data, not just a vector index
Data platform teams on Databricks or Snowflake who need to turn tables already sitting in Unity Catalog or a Snowflake warehouse into a governed, lineage-tracked knowledge graph, without exporting that data to a third-party SaaS first
Compliance, risk, and audit teams who need a straight answer to "why did the AI do that?" in a format a regulator will actually accept
Regulated enterprises (finance, healthcare, legal, government, defense) that can't ship a black box, and can't send their data to someone else's SaaS to get one
Platform and infra engineers who want the KG, reasoning, and provenance stack self-hosted and swappable, not locked to one vendor's backend
Data and knowledge engineers building a KG from messy, multi-source data: entities and relationships get extracted, conflicting or contradictory facts are flagged instead of silently overwritten, and duplicates are merged before they turn into noise

Quick Start · Architecture · What You Get · Why Semantica · Decision Intelligence · Context Graphs · Recipe: Audit Trail · Module Reference · Integrations · CLI · Performance · Install

What Semantica Gives You

Context Graphs: A structured, queryable graph of everything your agent knows, decides, and reasons about
Decision Intelligence: Every decision is a first-class object: traceable, searchable by precedent, and causally linked
AI Governance & Ontology: SHACL constraints, conflict detection, compliance rules, OWL generation, and SKOS vocabulary management with a visual editor
Full Auditability: W3C PROV-O provenance on every fact, with audit trails exportable to JSON, CSV, or RDF
Deterministic Reasoning: Forward chaining, Rete network, Datalog, and SPARQL with fully explainable paths, not black boxes
Knowledge Pipeline: Multi-source ingestion, entity-aware chunking, NER/relation/event extraction, and knowledge graph construction, with semantic deduplication and provenance-preserving merges throughout
Enterprise Data Platforms: Native connectors for Databricks (Unity Catalog + Delta Lake, PAT/OAuth M2M auth, catalog/schema/table/lineage introspection) and Snowflake (warehouse/database/schema, key-pair and OAuth auth), so tables already living in your lakehouse or warehouse become graph nodes with provenance, not another export/import hop
Graph Analytics: Centrality, community detection, link prediction, and shortest-path queries over the graph you just built
Polyglot Graph Storage: Native RDF (embedded Oxigraph, Blazegraph, Apache Jena, Eclipse RDF4J via SPARQL) and Labeled Property Graphs (Neo4j, FalkorDB, Apache AGE, AWS Neptune via Cypher), plus vector stores, all swappable without touching your code
Visualization: Explore any graph, ontology, or timeline in an interactive browser workbench
Drop-in Integrations: Native Agno support, a full-featured MCP server, a comprehensive CLI, a REST API, and plugins across major editors

Why Semantica

	Vector DB + RAG	Plain LLM Memory	Semantica
Recall method	Embedding similarity	Token window	Graph traversal + semantic search
Decision history	Not stored	Not stored	First-class queryable objects
Provenance	None	None	W3C PROV-O, source-linked
Reasoning	None	Black box	Forward chain, Rete, Datalog, SPARQL
Conflict detection	Silent overwrite	Silent overwrite	Detected, flagged, resolved
Time travel	No	No	Point-in-time graph snapshots
Compliance export	None	None	PROV-O, SHACL, OWL, RDF
Policy enforcement	None	None	Built-in rule engine + SHACL
Entity resolution	No	No	Blocking + semantic deduplication
Multi-agent context	Separate per agent	Separate per agent	Single shared intelligence layer

Semantica complements your existing stack rather than replacing it. Keep your LLM, vector store, and agent framework exactly as they are; Semantica adds the decision records, causal reasoning, provenance, ontology governance, conflict detection, and audit trails on top. The reasoning engines, KG construction, and provenance layer are fully deterministic; no LLM is required to use them.

Quick Start

pip install semantica

from semantica.context import ContextGraph

graph = ContextGraph(advanced_analytics=True)

# Every agent decision becomes a queryable, auditable knowledge node
decision_id = graph.record_decision(
    category="vendor_selection",
    scenario="Choose cloud provider for HIPAA workload",
    reasoning="AWS offers BAA, mature HIPAA tooling, and existing team expertise",
    outcome="selected_aws",
    confidence=0.93,
)

# Ask "why did this happen?" and get a real, structured answer
chain     = graph.trace_decision_chain(decision_id)       # full causal ancestry
similar   = graph.find_similar_decisions("cloud vendor", max_results=5)  # precedents
impact    = graph.analyze_decision_impact(decision_id)    # downstream influence map
compliant = graph.check_decision_rules({"category": "vendor_selection"})  # policy gate

Verify your install in 5 seconds:

semantica doctor
# Python 3.11.9         pass
# semantica 0.6.5       pass
# faiss vector store    pass
# Config file           pass    ~/.semantica/config.yaml

If Semantica solves a real problem for you, a star helps others find it.

⭐ Star on GitHub · Join Discord

Architecture

Semantica is a real end-to-end pipeline, not a single library with a marketing name. Every stage below is a shipping module, independently importable:

Sources → Ingest → Parse → Normalize → Split → Extract → Conflict Detection → Deduplication
   → Knowledge Graph → [ Ontology · Reasoning · Provenance · Decisions ] → Enriched KG
   → Vector Store + Polyglot Graph Store (RDF & LPG) → Export / Visualize / REST · MCP · CLI

Ingest: files, web, databases, enterprise data platforms (Databricks, Snowflake), cloud (Google Drive, Elasticsearch), streams (Kafka, Kinesis), Git, email, MCP
Parse → Normalize → Split: document parsing, text/entity/date normalization, GraphRAG-native entity-aware chunking
Extract → Conflict Detection → Deduplication: NER, relations, events, triplets; conflicting facts flagged and resolved before they merge
Knowledge Graph: GraphBuilder constructs the graph; bi-temporal facts and full graph analytics (centrality, communities, link prediction) run on top of it
Ontology · Reasoning · Provenance · Decisions: the intelligence layer sitting on the KG, with SHACL/OWL governance, Rete/Datalog/SPARQL inference, W3C PROV-O lineage, and first-class decision records
Storage: polyglot by design, with RDF triple stores (embedded Oxigraph, Blazegraph, Apache Jena, Eclipse RDF4J), Labeled Property Graphs (Neo4j, FalkorDB, Apache AGE, AWS Neptune), and vector stores, all swappable without touching your code
Outputs: export (RDF, OWL, Parquet, Cypher, JSON-LD), interactive visualization, and access via REST API, MCP server, or CLI

→ Full Mermaid diagrams for the pipeline and the decision intelligence lifecycle

Decision Intelligence

Decision Intelligence turns every AI choice from an ephemeral inference into a permanent, auditable, queryable record. It answers "what did your AI decide, why, and what happened next?": the question regulators and enterprise risk teams ask with increasing urgency.

In Semantica, a decision is not a log line. It is a first-class graph node with a full lifecycle. In regulated domains, every AI decision must be traceable to a source and defensible to an auditor: record_decision() creates a permanent, structured record exportable as W3C PROV-O, the format most compliance frameworks accept for regulator submission.

record_decision()             → stored as a graph node with full structured context
add_causal_relationship()     → linked to upstream causes and downstream effects
find_similar_decisions()      → semantic precedent search across all past decisions
trace_decision_chain()        → full causal ancestry back to root causes
analyze_decision_impact()     → downstream influence map - everything this decision affected
check_decision_rules()        → policy compliance gate against configurable rule sets
export / audit trail          → W3C PROV-O, CSV, or JSON for regulator submission

from semantica.context import ContextGraph

graph = ContextGraph(advanced_analytics=True)

# Record decisions with full structured context
app_id = graph.record_decision(
    category="credit_application",
    scenario="Personal loan, $85k income, 31% DTI, 3yr employment",
    reasoning="Income meets threshold; employment stable; no adverse credit events",
    outcome="proceed_to_underwriting",
    confidence=0.88,
    metadata={"applicant_id": "A-7291"},
)
uw_id = graph.record_decision(
    category="loan_underwriting",
    scenario="Underwriting review for A-7291",
    reasoning="DTI within policy; clean 36-month credit history",
    outcome="approved",
    confidence=0.94,
)
rate_id = graph.record_decision(
    category="interest_rate",
    scenario="Rate assignment for approved loan A-7291",
    outcome="rate_set_8.9pct",
    reasoning="Prime + 2.4% based on risk tier B2",
    confidence=0.99,
)

# Build the auditable causal chain - relationship_type must be one of
# CAUSED, INFLUENCED, or PRECEDENT_FOR
graph.add_causal_relationship(app_id, uw_id,   relationship_type="CAUSED")
graph.add_causal_relationship(uw_id,  rate_id, relationship_type="INFLUENCED")

# Query the intelligence
chain     = graph.trace_decision_chain(rate_id)
similar   = graph.find_similar_decisions("personal loan approval, 31% DTI", max_results=5)
impact    = graph.analyze_decision_impact(uw_id)
compliant = graph.check_decision_rules({"category": "loan_underwriting", "confidence": 0.94})
insights  = graph.get_decision_insights()

Context Graphs

A Context Graph is the structured memory layer that traditional RAG is missing. Instead of flat embeddings that answer "what is similar?", a Context Graph answers "what is connected, why, and how?" Every entity, relationship, decision, and fact is a first-class node, queryable by graph traversal. Entities link to source documents, decisions link to evidence and consequences, facts carry full provenance, and conflicts are detected, not silently overwritten.

from semantica.context import ContextGraph, AgentContext
from semantica.vector_store import VectorStore

graph = ContextGraph(advanced_analytics=True)

# Add nodes with typed properties
graph.add_node("acme_corp",    "Organization", name="Acme Corp", industry="SaaS")
graph.add_node("alice_chen",   "Person",       name="Alice Chen", role="CTO")
graph.add_node("contract_001", "Contract",     value=2_400_000, currency="USD")

# Add typed, weighted edges (extra kwargs become edge metadata)
graph.add_edge("alice_chen", "acme_corp",    edge_type="works_for",  since="2019-03-01")
graph.add_edge("acme_corp",  "contract_001", edge_type="party_to",   signed="2024-01-15")

# BFS traversal - hop through the graph from any node
neighbors = graph.get_neighbors("acme_corp", hops=2)

# Point-in-time snapshot - the graph as it existed on any past date
snapshot  = graph.state_at("2024-01-01")

# AgentContext - high-level API for agent memory workflows
vs  = VectorStore(backend="faiss")
ctx = AgentContext(vector_store=vs, knowledge_graph=graph)
ctx.store("Alice approved the Acme renewal in Q1 2024", conversation_id="conv_001")
retrieved = ctx.retrieve("who approved the Acme contract?")

Why graph over embeddings: traversal finds connections embeddings miss (a person 3 hops from a contract); every node carries provenance so you can always ask "where did this come from?"; conflicts are flagged before they corrupt your knowledge base; point-in-time snapshots let you replay history without reprocessing.

Recipe: Audit Trail for a Regulated Decision

The flagship pattern: record a causally-linked decision chain, attach provenance to every entity, and export a regulator-ready audit trail.

from semantica.context import ContextGraph
from semantica.provenance import ProvenanceManager
from semantica.export import RDFExporter

graph = ContextGraph(advanced_analytics=True)
prov  = ProvenanceManager(storage_path="./audit.db")

# Record the decision chain
d1 = graph.record_decision(
    category="drug_interaction_check", scenario="Patient P-4821: warfarin + amiodarone co-prescribed",
    reasoning="Amiodarone potentiates warfarin's anticoagulant effect", outcome="flag_for_review", confidence=0.91,
)
d2 = graph.record_decision(
    category="dosage_adjustment", scenario="INR monitoring plan for P-4821",
    reasoning="Reduce warfarin dose per interaction severity; recheck INR in 5 days", outcome="dose_reduced_30pct", confidence=0.87,
)
# relationship_type must be one of CAUSED, INFLUENCED, or PRECEDENT_FOR
graph.add_causal_relationship(d1, d2, relationship_type="CAUSED")

# Track provenance for every entity
prov.track_entity("patient_P4821", source="ehr/medication_orders_2024.json",
                  metadata={"extractor": "NamedEntityRecognizer"})

# Export W3C PROV-O for regulator submission - RDFExporter expects
# {"entities": [...], "relationships": [...]}, so map ContextGraph.to_dict()'s
# {"nodes": [...], "edges": [...]} shape onto it first
graph_dict = graph.to_dict()
kg = {
    "entities": [{"id": n["id"], "type": n["type"], "text": n["content"]} for n in graph_dict["nodes"]],
    "relationships": [
        {"source_id": e["source"], "target_id": e["target"], "type": e["type"]}
        for e in graph_dict["edges"]
    ],
}
RDFExporter().export(kg, "audit_trail.ttl", format="turtle")

More recipes (GraphRAG pipelines, an AML rules engine, ontology-to-KG in one pass) are in More Recipes below.

Explore the Platform

Every module below is independently importable, with working code samples verified against the current source tree; use one or all of them.

Module	What it does
`semantica.ingest`	Files, web, databases, APIs, streams, email, Git, Parquet, Databricks, Snowflake, MCP
`semantica.semantic_extract`	NER, relation extraction, event detection, triplet generation
`semantica.kg`	Graph construction, centrality, communities, link prediction
`semantica.reasoning`	Forward chaining, Rete, Datalog, SPARQL, fully explainable
`semantica.vector_store`	FAISS, Qdrant, Weaviate, Milvus, Pinecone, PgVector, hybrid search
`semantica.split`	Entity-aware, relation-aware, ontology-aware chunking for GraphRAG
`semantica.provenance`	W3C PROV-O lineage on every fact
`semantica.ontology`	OWL generation, SHACL validation, SKOS vocabularies
`semantica.conflicts`	Detect and resolve conflicting facts across sources
`semantica.deduplication`	Entity resolution at scale
`semantica.normalize`	Text, entity, date, and number normalization; dataset cleaning
`semantica.pipeline`	Declarative, parallel pipeline DSL for ingest → extract → build → export
`semantica.export`	RDF, OWL, Parquet, Cypher, JSON-LD
`semantica.visualization`	Force-directed graphs, ontology hierarchies, temporal dashboards
Temporal Intelligence	Bi-temporal facts, Allen interval algebra, time travel
Multi-Agent (Agno)	One shared context graph across every agent on a team

↓ Expand Module Reference below for every module's working example, or jump to More Recipes, the full Integrations matrix, MCP tool list, and REST endpoints.

Module Reference

Expand any module below for its runnable example.

semantica.ingest: Multi-Source Ingestion

Ingest from files, web, databases, APIs, streams, email, Git repos, Parquet, Databricks, Snowflake, or MCP servers, all through a unified interface.

from semantica.ingest import FileIngestor, WebIngestor, ParquetIngestor, DBIngestor

# Ingest an entire directory of contracts (PDF, DOCX, HTML, TXT)
docs = FileIngestor().ingest_directory("./contracts/", recursive=True)

# Ingest live web content with robots.txt compliance
pages = WebIngestor().ingest_url("https://example.com/reports/annual-2024.html")

# Ingest structured data from Parquet with Snappy compression
records = ParquetIngestor().ingest("./data/transactions.parquet")

# Ingest from a SQL database - specify which tables to pull
rows = DBIngestor().ingest_database(
    connection_string="postgresql://user:pass@localhost/mydb",
    include_tables=["customer_events"],
    max_rows_per_table=50_000,
)

# Enterprise data platforms - pull tables straight out of your lakehouse
# or warehouse, with lineage, instead of exporting to CSV first
from semantica.ingest import DatabricksIngestor, SnowflakeIngestor

# pip install "semantica[db-databricks]"
databricks = DatabricksIngestor(
    host="https://adb-xxx.azuredatabricks.net",
    token="dapi-xxxxxxxx",              # or client_id/client_secret for OAuth M2M
    http_path="/sql/1.0/warehouses/xxxxxxxx",
    catalog="main",
)
customers    = databricks.ingest_table("customers", limit=10_000)
sales        = databricks.ingest_query("SELECT * FROM sales WHERE region = 'EMEA'")
table_lineage = databricks.get_table_lineage("customers", catalog="main", schema="default")  # Unity Catalog lineage

# pip install semantica[db-snowflake]
snowflake = SnowflakeIngestor(
    account="myaccount",
    user="myuser",
    password="mypassword",              # or private_key=... for key-pair; use authenticator="oauth", token=... for OAuth
    warehouse="COMPUTE_WH",
    database="MYDB",
)
orders = snowflake.ingest_table("ORDERS", limit=10_000)

Security Note: Never hardcode credentials (token, password, private_key) in production code; pass them via environment variables (e.g., DATABRICKS_TOKEN, SNOWFLAKE_PASSWORD) or a secrets manager.

Supported sources: Local files (PDF, DOCX, PPTX, HTML, TXT, CSV, JSON, YAML, Excel, XML) · Web pages · RSS/Atom feeds · REST APIs · Databases (PostgreSQL, MySQL, SQLite, Oracle, SQL Server) · Parquet datasets · Databricks (Unity Catalog + Delta Lake) · Snowflake · Git repositories · Email (IMAP/POP3) · Message streams (Kafka, RabbitMQ, Kinesis, Pulsar) · MCP resources · Apache Arrow/Feather/IPC (ArrowIngestor)

DuckDB, Elasticsearch, Google Drive, HuggingFace, MongoDB, and Pandas ingestion also ship (DuckDBIngestor, ElasticIngestor, GDriveIngestor, HuggingFaceIngestor, MongoIngestor, PandasIngestor) but aren't re-exported from the top-level semantica.ingest namespace yet — import them directly: from semantica.ingest.duckdb_ingestor import DuckDBIngestor.

semantica.semantic_extract: NER, Relations, Events, Triplets

Extract structured knowledge from raw text in one pass.

from semantica.semantic_extract import (
    NamedEntityRecognizer,
    RelationExtractor,
    EventDetector,
    TripletExtractor,
)

text = """
Anthropic CEO Dario Amodei announced a $7.3B Series E funding round in partnership
with Google and Spark Capital, valuing the company at $61.5B as of Q4 2024.
"""

# Named entity recognition with confidence thresholding
ner = NamedEntityRecognizer(confidence_threshold=0.7)
entities = ner.extract_entities(text)
# → [Entity(name="Dario Amodei", type="PERSON"), Entity(name="Anthropic", type="ORG"),
#    Entity(name="Google", type="ORG"), Entity(name="$7.3B", type="MONEY"), ...]

# Relationship extraction - bidirectional support
rel_extractor = RelationExtractor(confidence_threshold=0.6, bidirectional=True)
relations = rel_extractor.extract_relations(text, entities=entities)
# → [Relation(subject="Dario Amodei", predicate="ceo_of", object="Anthropic"),
#    Relation(subject="Anthropic", predicate="raised", object="$7.3B Series E"), ...]

# Event detection with temporal processing
events = EventDetector(extract_participants=True, extract_time=True).detect_events(text)
# → [Event(type="FUNDING", participants=["Anthropic","Google","Spark Capital"],
#          amount="$7.3B", date="Q4 2024")]

# RDF triplets with optional provenance metadata
triplets = TripletExtractor(include_temporal=True, include_provenance=True).extract_triplets(text)
# → [("Anthropic", "valuation", "$61.5B"), ("Dario Amodei", "is_ceo_of", "Anthropic"), ...]

Batch processing across many documents uses ner.process_batch([...]), not a per-call extract_entities_batch on the facade class.

semantica.kg: Knowledge Graph Construction & Analysis

Build a production knowledge graph from documents and run graph algorithms over it.

from semantica.ingest import FileIngestor
from semantica.kg import (
    GraphBuilder,
    GraphAnalyzer,
    CentralityCalculator,
    CommunityDetector,
    PathFinder,
    LinkPredictor,
    BiTemporalFact,
)
from datetime import datetime

# Build KG - merge duplicate entities, track temporal edges
sources = FileIngestor().ingest_directory("./contracts/", recursive=True)
kg = GraphBuilder(merge_entities=True, enable_temporal=True).build(sources)

# Graph analytics
analyzer    = GraphAnalyzer()
analysis    = analyzer.analyze_graph(kg)             # full graph metrics

centrality  = CentralityCalculator()
degree      = centrality.calculate_degree_centrality(kg)    # most-connected entities
betweenness = centrality.calculate_betweenness_centrality(kg)

communities = CommunityDetector().detect_communities(kg, method="louvain")  # natural clusters
path        = PathFinder().find_shortest_path(kg, "alice_chen", "contract_001")
predictions = LinkPredictor().predict_links(kg, top_k=10)   # relationship predictions

# Bi-temporal facts - track valid time vs. recorded time independently
fact = BiTemporalFact(
    valid_from=datetime(2024, 3, 1),
    valid_until=datetime(2025, 1, 1),
    recorded_at=datetime(2024, 3, 5),
)

semantica.reasoning: Forward Chaining, Rete, Datalog, SPARQL

Run explainable rule-based inference, not a black box.

from semantica.reasoning import ReteEngine, Rule, Fact, RuleType

rete = ReteEngine()
rete.build_network([
    Rule(
        rule_id="aml_flag",
        name="Flag high-risk transactions",
        conditions=[
            {"field": "amount",  "operator": ">",  "value": 10_000},
            {"field": "country", "operator": "in", "value": ["IR", "KP", "SY"]},
        ],
        conclusion="flag_for_compliance_review",
        rule_type=RuleType.IMPLICATION,
    ),
    Rule(
        rule_id="velocity_check",
        name="Flag rapid sequential transfers",
        conditions=[
            {"field": "transfers_in_1h", "operator": ">", "value": 5},
            {"field": "total_amount",    "operator": ">", "value": 50_000},
        ],
        conclusion="flag_velocity_breach",
        rule_type=RuleType.IMPLICATION,
    ),
])

rete.add_fact(Fact("tx_001", "transaction", [{"amount": 15_000, "country": "IR"}]))
flagged = rete.match_patterns()
# → [{"rule": "aml_flag", "matched_facts": ["tx_001"], "conclusion": "flag_for_compliance_review"}]

Current limitation: ReteEngine's alpha-node condition matcher is intentionally simple in this release — validate match_patterns() output against your actual rule set before wiring it into a production compliance gate; more selective condition evaluation is on the roadmap.

# Recursive Datalog - natural language for graph queries
from semantica.reasoning import DatalogReasoner

engine = DatalogReasoner()
engine.add_fact("parent(tom, bob)")
engine.add_fact("parent(bob, ann)")
engine.add_fact("parent(ann, pat)")
engine.add_rule("ancestor(X, Y) :- parent(X, Y).")
engine.add_rule("ancestor(X, Z) :- parent(X, Y), ancestor(Y, Z).")
ancestors = engine.query("ancestor(tom, ?X)")
# → [{"X": "bob"}, {"X": "ann"}, {"X": "pat"}]

# Explainable reasoning - trace the path, not just the answer
from semantica.reasoning import ExplanationGenerator, Reasoner

reasoner = Reasoner()
reasoner.add_fact("parent(tom, bob)")
reasoner.add_rule("ancestor(X, Y) :- parent(X, Y)")
result = reasoner.forward_chain()

explainer = ExplanationGenerator()
explanation = explainer.generate_explanation(result)
# → Explanation(conclusion="...", steps=[ReasoningStep(...)], justification=Justification(...))

semantica.vector_store: Hybrid & Filtered Semantic Search

Drop-in vector store with multiple backends, hybrid search, and decision-aware retrieval.

from semantica.vector_store import VectorStore, HybridSearch

# In-memory backend shown here: HybridSearch and explain_decision() work out of the box.
# Swap backend="qdrant" / "weaviate" / "milvus" / "pinecone" / "pgvector" / "faiss" once you
# scale past a single process — search() and store_decision() work identically on all of them.
vs = VectorStore(backend="inmemory", dimension=1536)

# Store a decision with scenario description and outcome
vs.store_decision(
    scenario="Personal loan A-7291, $85k income, 31% DTI, 3yr employment",
    outcome="approved",
    confidence=0.94,
    category="loan_underwriting",
)

# Semantic similarity search
results = vs.search(
    query="personal loan approval with low DTI",
    limit=10,
)

# Hybrid search - dense + sparse retrieval in one pass with RRF fusion
hs   = HybridSearch(vector_store=vs)
hits = hs.search("high-risk transactions 2024")

# Explain why a decision was retrieved
explanation = vs.explain_decision(results[0]["id"])

Backends: faiss · qdrant · weaviate · milvus · pinecone · pgvector · sqlite · inmemory

semantica.split: GraphRAG-Native Document Chunking

KG-aware splitting that preserves entity boundaries, relation triplets, and ontology concepts, essential for GraphRAG pipelines.

from semantica.split import TextSplitter, EntityAwareChunker, RelationAwareChunker

text = open("contracts/master_agreement.txt").read()

# Standard recursive chunking
chunks = TextSplitter(method="recursive", chunk_size=1000, chunk_overlap=200).split(text)

# Entity-aware chunking - never splits a named entity across chunks (GraphRAG)
chunks = TextSplitter(method="entity_aware", ner_method="llm", chunk_size=1000).split(text)

# Relation-aware chunking - preserves (subject, predicate, object) triplets intact
chunks = RelationAwareChunker(chunk_size=1000, preserve_triplets=True).chunk(text)

# Graph-based chunking - uses centrality to find natural community boundaries
chunks = TextSplitter(method="graph_based", chunk_size=1000).split(text)

# Hierarchical chunking - multi-level (section → paragraph → sentence)
chunks = TextSplitter(method="hierarchical", levels=["section", "paragraph"]).split(text)

Supported methods: recursive · token · sentence · paragraph · semantic_transformer · entity_aware · relation_aware · graph_based · ontology_aware · hierarchical · community_detection · centrality_based · llm

semantica.provenance: W3C PROV-O Lineage

Every fact is linked to its source. No black boxes, no mystery outputs.

from semantica.provenance import ProvenanceManager

prov = ProvenanceManager(storage_path="./provenance.db")

# Track where every entity came from
prov.track_entity(
    entity_id="acme_corp",
    source="contracts/acme_master_agreement_2024.pdf",
    metadata={"page": 1, "confidence": 0.97, "extractor": "NamedEntityRecognizer"},
)

# Track a relationship's provenance - entity linkage travels in metadata
prov.track_relationship(
    relationship_id="alice_works_for_acme",
    source="hr_records/employees_q1_2024.csv",
    metadata={"source_entity_id": "alice_chen", "target_entity_id": "acme_corp"},
)

# Answer "where did this come from?"
lineage = prov.get_lineage("acme_corp")
trail   = prov.trace_lineage("alice_chen")   # full ancestor chain
entry   = prov.get_provenance("acme_corp")

semantica.ontology: OWL Generation, SHACL Validation

Generate ontologies from data, validate shapes, and manage your vocabulary.

from semantica.ontology import OntologyGenerator, OntologyValidator

data = {
    "entities": [
        {"id": "acme_corp",  "type": "Organization", "industry": "SaaS", "founded": 2012},
        {"id": "alice_chen", "type": "Person",        "role": "CTO",     "since": 2019},
    ],
    "relationships": [
        {"source": "alice_chen", "target": "acme_corp", "type": "works_for"},
    ],
}

gen       = OntologyGenerator(base_uri="https://semantica.dev/ontology/")
ontology  = gen.generate_ontology(data)
classes   = gen.infer_classes(data)
props     = gen.infer_properties(data, classes)
optimized = gen.optimize_ontology(ontology)

# Validate against SHACL shapes
validator = OntologyValidator()
report    = validator.validate(ontology)
# → ValidationResult(valid=True, consistent=True, satisfiable=True, errors=[], warnings=[])

semantica.conflicts: Conflict Detection & Resolution

Detect and resolve conflicting facts from multiple sources before they corrupt your knowledge base.

from semantica.conflicts import ConflictDetector, ConflictResolver, SourceTracker

entities_from_source_a = [
    {"id": "alice_chen", "role": "CTO",   "salary": 250_000, "start_date": "2019-03-01"},
]
entities_from_source_b = [
    {"id": "alice_chen", "role": "VP Eng", "salary": 275_000, "start_date": "2019-03-01"},
]

# Detect all conflict types: value, type, relationship, temporal, logical
detector   = ConflictDetector()
conflicts  = detector.detect_conflicts(entities_from_source_a + entities_from_source_b)
# → [Conflict(entity="alice_chen", field="role",   values=["CTO","VP Eng"], severity="HIGH"),
#    Conflict(entity="alice_chen", field="salary",  values=[250000,275000],   severity="MEDIUM")]

# Resolve using multiple strategies
resolver = ConflictResolver()
resolved = resolver.resolve_conflicts(conflicts, strategy="credibility_weighted")  # weighted by source trust
resolved = resolver.resolve_conflicts(conflicts, strategy="most_recent")          # prefer most recent
resolved = resolver.resolve_conflicts(conflicts, strategy="voting")               # majority wins

# Track source credibility over time
tracker = SourceTracker()
tracker.register_source("source_a", source_type="document", credibility_score=0.85)
tracker.register_source("source_b", source_type="document", credibility_score=0.72)

semantica.deduplication: Entity Resolution at Scale

Block, cluster, and merge duplicates with semantic similarity.

from semantica.deduplication import DuplicateDetector, EntityMerger

entities = [
    {"id": "e1", "name": "Acme Corporation",  "domain": "acme.com"},
    {"id": "e2", "name": "Acme Corp.",         "domain": "acme.com"},
    {"id": "e3", "name": "ACME Corp",          "domain": "acme.co"},
    {"id": "e4", "name": "Globex Industries",  "domain": "globex.com"},
]

detector   = DuplicateDetector(similarity_threshold=0.75, use_clustering=True)
candidates = detector.detect_duplicates(entities)
groups     = detector.detect_duplicate_groups(entities)
# → DuplicateGroup(entities=["e1","e2","e3"], confidence=0.91, strategy="semantic+blocking")

merger  = EntityMerger(preserve_provenance=True)
ops     = merger.merge_duplicates(entities, strategy="keep_most_complete")
history = merger.get_merge_history()

semantica.normalize: Data Normalization & Cleaning

Standardize text, entities, dates, numbers, and encodings before building your knowledge graph.

from semantica.normalize import (
    TextNormalizer,
    EntityNormalizer,
    DateNormalizer,
    NumberNormalizer,
    DataCleaner,
)

# Unicode, whitespace, casing, HTML tags, smart quotes
text  = TextNormalizer().normalize("  Acme Corp.'s Q4 report...  ")
# → "Acme Corp.'s Q4 report..."

# Alias resolution + entity disambiguation with confidence scores
canonical = EntityNormalizer().normalize_entity("ACME Corp.")
# → NormalizedEntity(canonical="Acme Corporation", type="Organization", confidence=0.91)

# Natural language date parsing with timezone conversion
dt    = DateNormalizer().normalize_date("3 weeks ago")
# → datetime(2026, 7, 1, tzinfo=UTC)

# Unit conversion and currency normalization
price = NumberNormalizer().normalize_number("$1.25M USD")
# → NormalizedNumber(value=1_250_000, currency="USD")

# Deduplicate, validate, and impute missing values across a dataset
clean = DataCleaner().clean_data(records, remove_duplicates=True, handle_missing=True)

semantica.pipeline: Pipeline DSL

Compose ingestion, extraction, and graph-building into a declarative, parallel pipeline.

from semantica.pipeline import PipelineBuilder, ExecutionEngine

builder = PipelineBuilder()

# add_step() returns the created PipelineStep, not the builder, so these don't chain
builder.add_step("ingest",      step_type="ingest",           source="./contracts/", recursive=True)
builder.add_step("extract",     step_type="ner_extract")
builder.add_step("relations",   step_type="relation_extract")
builder.add_step("build_kg",    step_type="kg_build",         merge_entities=True)
builder.add_step("deduplicate", step_type="deduplicate",      threshold=0.75)
builder.add_step("export",      step_type="export",           format="turtle", output="kg.ttl")

# connect_steps() and set_parallelism() return the builder, so these do chain
pipeline = (
    builder
    .connect_steps("ingest",      "extract")
    .connect_steps("extract",     "relations")
    .connect_steps("relations",   "build_kg")
    .connect_steps("build_kg",    "deduplicate")
    .connect_steps("deduplicate", "export")
    .set_parallelism(4)
    .build(name="contracts_pipeline")
)

engine   = ExecutionEngine()
result   = engine.execute_pipeline(pipeline)
status   = engine.get_pipeline_status(pipeline.name)
progress = engine.get_progress(pipeline.name)

Temporal Intelligence: Bi-Temporal Graphs & Time Travel

Track when facts were true in the world vs. when they were recorded, and query either axis.

from semantica.context import ContextGraph
from semantica.kg import (
    BiTemporalFact,
    TemporalGraphQuery,
    TemporalNormalizer,
)
from datetime import datetime

graph = ContextGraph(advanced_analytics=True)
graph.add_node("alice_chen", "Person",       role="VP Engineering")
graph.add_node("acme_corp",  "Organization", valuation=1_200_000_000)

# A temporally-bounded edge - valid_from/valid_until define when it held true
graph.add_edge(
    "alice_chen", "acme_corp", edge_type="works_for",
    valid_from="2024-03-01T00:00:00", valid_until="2025-01-01T00:00:00",
)

# Point-in-time snapshots - replay history without reprocessing
snapshot_2023 = graph.state_at("2023-06-01")
snapshot_2024 = graph.state_at("2024-01-01")

# Bi-temporal facts - valid_time is when true in the world;
# recorded_at is when you learned about it
fact = BiTemporalFact(
    valid_from=datetime(2024, 3, 1),
    valid_until=datetime(2025, 1, 1),
    recorded_at=datetime(2024, 3, 5),
)

# Query facts valid within a time window - query_time_range() expects
# {"relationships": [...]} with source_id/target_id keys, which differs from
# ContextGraph.to_dict()'s {"nodes", "edges"} shape, so map it first
graph_dict = graph.to_dict()
kg_relationships = {
    "relationships": [
        {**e, "source_id": e["source"], "target_id": e["target"]}
        for e in graph_dict["edges"]
    ]
}

tq = TemporalGraphQuery()
facts_in_window = tq.query_time_range(
    kg_relationships, query="valid_facts", start_time="2024-01-01", end_time="2024-12-31"
)

# Normalize natural language temporal expressions - returns a (start, end) range
norm = TemporalNormalizer()
start, end = norm.normalize("last quarter")

semantica.export: RDF, OWL, Parquet, Cypher, JSON-LD

Export to any format required by regulators, graph databases, or downstream systems.

from semantica.export import (
    RDFExporter,
    JSONExporter,
    ParquetExporter,
    LPGExporter,
    ReportGenerator,
)

kg = {"entities": [...], "relationships": [...]}

rdf = RDFExporter()
turtle_str = rdf.export_to_rdf(kg, format="turtle")     # returns string
jsonld_str = rdf.export_to_rdf(kg, format="json-ld")

rdf.export(kg, "kg_audit.ttl",    format="turtle")
rdf.export(kg, "kg_audit.jsonld", format="json-ld")
rdf.export(kg, "kg_audit.nt",     format="n-triples")

# Columnar analytics - Snappy-compressed Parquet (writes kg_snapshot_entities.parquet
# and kg_snapshot_relationships.parquet)
ParquetExporter(compression="snappy").export_knowledge_graph(kg, "kg_snapshot")

# JSON knowledge graph
JSONExporter().export_knowledge_graph(kg, "kg.json")

# Neo4j / Memgraph Cypher statements for graph database import
LPGExporter().export(kg, "kg_import.cypher")

# Human-readable HTML report
ReportGenerator().generate_report(
    {"title": "KG Audit Report", "summary": "Weekly ingestion summary", "metrics": {"entities": len(kg["entities"])}},
    file_path="audit_report.html",
    format="html",
)

semantica.visualization: Interactive Graph Workbench

Render force-directed graphs, community maps, ontology hierarchies, and temporal dashboards.

from semantica.visualization import (
    KGVisualizer,
    OntologyVisualizer,
    EmbeddingVisualizer,
    TemporalVisualizer,
)
import numpy as np

kg = {"entities": [...], "relationships": [...]}

# Interactive force-directed graph (opens in browser)
viz = KGVisualizer(layout="force", color_scheme="default")
viz.visualize_network(kg, output="interactive", file_path="kg.html")
viz.visualize_communities(kg, communities, output="interactive")
viz.visualize_centrality(kg, centrality, centrality_type="degree")
viz.visualize_entity_types(kg, output="html", file_path="entity_types.html")

# Ontology class hierarchy
OntologyVisualizer().visualize_hierarchy(ontology, output="interactive")

# 2D embedding projection (UMAP / t-SNE / PCA)
EmbeddingVisualizer().visualize_2d_projection(
    embeddings=np.array([...]),
    labels=["entity_a", "entity_b"],
    method="umap",
)

# Timeline scrubber - watch the graph evolve
TemporalVisualizer().visualize_timeline(kg, output="interactive")

Multi-Agent Shared Context with Agno

One shared intelligence layer. All agents read and write to the same context graph.

# pip install semantica[agno]
from agno.agent import Agent
from agno.team import Team
from agno.models.anthropic import Claude
from semantica.context import ContextGraph
from semantica.vector_store import VectorStore
from integrations.agno import AgnoSharedContext, AgnoDecisionKit, AgnoKGToolkit

shared = AgnoSharedContext(
    vector_store=VectorStore(backend="faiss"),
    knowledge_graph=ContextGraph(advanced_analytics=True),
    decision_tracking=True,
)

researcher = Agent(
    name="Researcher",
    model=Claude(id="claude-sonnet-4-5"),
    memory=shared.bind_agent("researcher"),
    tools=[AgnoKGToolkit(context=shared)],
)
analyst = Agent(
    name="Analyst",
    model=Claude(id="claude-sonnet-4-5"),
    memory=shared.bind_agent("analyst"),
    tools=[AgnoDecisionKit(context=shared)],
)

team = Team(agents=[researcher, analyst], mode="coordinate")
# Researcher's findings are instantly available to the Analyst - no copy, no sync

→ runnable notebooks in the cookbook, each self-contained and runnable in under 5 minutes

More Recipes

The flagship audit-trail recipe is above. Here are three more common patterns.

End-to-End GraphRAG Pipeline

from semantica.ingest import FileIngestor
from semantica.split import TextSplitter
from semantica.semantic_extract import NamedEntityRecognizer, RelationExtractor
from semantica.kg import GraphBuilder
from semantica.vector_store import VectorStore, HybridSearch
from semantica.context import AgentContext

# 1. Ingest
docs = FileIngestor().ingest_directory("./docs/", recursive=True)

# 2. Entity-aware chunking - never splits an entity across a chunk boundary
splitter = TextSplitter(method="entity_aware", chunk_size=1000)
chunks   = [splitter.split(doc["text"]) for doc in docs]

# 3. Extract entities and relations
ner      = NamedEntityRecognizer(confidence_threshold=0.7)
rel_ext  = RelationExtractor(confidence_threshold=0.6)
entities = [ner.extract_entities(chunk) for chunk_group in chunks for chunk in chunk_group]

# 4. Build KG
kg = GraphBuilder(merge_entities=True, enable_temporal=True).build(docs)

# 5. Hybrid retrieval
vs  = VectorStore(backend="inmemory")
ctx = AgentContext(vector_store=vs, knowledge_graph=kg)
ctx.store("Alice approved the Acme renewal in Q1 2024", conversation_id="c1")

results = HybridSearch(vector_store=vs).search("who approved the renewal?")

AML Rules Engine

from semantica.reasoning import ReteEngine, Rule, Fact, RuleType

rete = ReteEngine()
rete.build_network([
    Rule(
        rule_id="sanctions_check",
        name="Flag sanctioned-country transactions",
        conditions=[
            {"field": "amount",  "operator": ">",  "value": 10_000},
            {"field": "country", "operator": "in", "value": ["IR", "KP", "SY", "CU"]},
        ],
        conclusion="flag_for_compliance_review",
        rule_type=RuleType.IMPLICATION,
    ),
])

# Run the rule across a batch of incoming transactions, not just one
for tx in [
    Fact("tx_101", "transaction", [{"amount": 25_000, "country": "IR"}]),
    Fact("tx_102", "transaction", [{"amount": 4_500,  "country": "DE"}]),
    Fact("tx_103", "transaction", [{"amount": 60_000, "country": "KP"}]),
]:
    rete.add_fact(tx)

flagged = rete.match_patterns()

Same condition-matcher caveat as above applies — validate against your rule set before production use.

Ontology-to-Knowledge-Graph in One Pass

from semantica.ingest import FileIngestor
from semantica.semantic_extract import NamedEntityRecognizer, RelationExtractor
from semantica.kg import GraphBuilder
from semantica.ontology import OntologyGenerator, OntologyValidator
from semantica.export import RDFExporter

sources   = FileIngestor().ingest_directory("./contracts/")
ner       = NamedEntityRecognizer(confidence_threshold=0.7)
entities  = ner.process_batch([s["text"] for s in sources])

kg  = GraphBuilder(merge_entities=True).build(sources)
gen = OntologyGenerator(base_uri="https://myco.dev/ontology/")
ont = gen.generate_ontology({"entities": entities[0], "relationships": []})

report = OntologyValidator().validate(ont)
if report.valid:
    RDFExporter().export({"entities": entities[0]}, "ontology.ttl", format="turtle")

Features at a Glance

Capability	Highlights
Context Graphs	Queryable graph of entities, decisions, relationships; causal links; cross-graph navigation
Decision Intelligence	`record_decision` · `trace_decision_chain` · `find_similar_decisions` · `analyze_decision_impact` · `check_decision_rules`
Temporal Intelligence	Point-in-time snapshots · Allen interval algebra (13 relations) · `TemporalNormalizer` · bi-temporal provenance
Distance Intelligence	N×N semantic distance matrices · ego-mode visualization · distance bands · embedding cache
Semantic Extraction	NER · relation extraction · event detection · triplet generation · coreference
Reasoning Engines	Forward chaining · Rete · deductive · abductive · SPARQL · Datalog with explainable output
GraphRAG Chunking	Entity-aware · relation-aware · graph-based · ontology-aware · community-detection chunking
Conflict Detection	Value / type / relationship / temporal / logical conflicts · multiple resolution strategies
Provenance	W3C PROV-O · every fact traced to source · audit log export JSON/CSV/RDF
Ontology Hub	SHACL Studio · visual editor · cross-ontology alignments · health dashboard
Vector Store	FAISS · Pinecone · Weaviate · Qdrant · Milvus · PgVector · hybrid + filtered search
Graph Databases (LPG)	Neo4j · FalkorDB · Apache AGE · AWS Neptune
Triple Stores (RDF)	Oxigraph (embedded) · Blazegraph · Apache Jena · Eclipse RDF4J · unified `TripletStore` interface · SPARQL query & bulk load
Enterprise Data Platforms	Databricks (`DatabricksIngestor`: Unity Catalog + Delta Lake, PAT/OAuth M2M, table/query ingestion, catalog/schema/table/lineage introspection) · Snowflake (`SnowflakeIngestor`: warehouse/database/schema, password/key-pair/OAuth auth)
LLM Providers	All already supported today: OpenAI (GPT-4o, o1, o3) · Anthropic (Claude) · Google Gemini · Mistral · Meta Llama · Groq · Cohere · Azure OpenAI · AWS Bedrock · Ollama · DeepSeek · Perplexity · Together AI · Fireworks AI · Replicate · HuggingFace · via `semantica.llms` and LiteLLM

Performance

Benchmarks from v0.5.0 on a 118,000-node production graph:

Operation	Before	After	Improvement
Node search (118k nodes)	24 ms	0.004 ms	6,000× faster
Embedding cache hit	cold load	revision-based cache	10× throughput
Semantic deduplication	baseline	optimized candidate gen	6.98× faster
Candidate generation	baseline	blocking strategy	63.6% faster

Measured on a 118,000-node production graph (AMD EPYC, 64 GB RAM); the deduplication/candidate-generation figures are historical measurements recorded in CHANGELOG.md rather than an automated tests/ assertion. Results vary by hardware, dataset topology, and backend selection — run pytest tests/vector_store/test_performance_benchmarks.py -s to measure your own data.

CLI

Every capability is available from the terminal. The CLI ships with the package, no separate install required.

pip install semantica
semantica        # startup dashboard
semantica doctor # health check
semantica --help # full grouped command reference

Start with semantica, verify with doctor, build a graph, and explore the command groups from one terminal.

Command groups: ingest · parse · extract · kg · reason · decision · temporal · provenance · ontology · embed · deduplicate · validate · export · visualize · pipeline · server · explorer · mcp · doctor · shell · init · watch

→ Full CLI reference

Integrations

Native plugin bundles for Claude Code, Cursor, Codex, Windsurf, Cline, Continue, VS Code, and OpenClaw; a full-featured MCP server for any MCP-compatible client; a comprehensive REST API; and first-class Agno support for multi-agent shared context. Every major LLM provider is already supported via semantica.llms and LiteLLM: OpenAI, Anthropic, Gemini, Mistral, Llama, Groq, Cohere, Azure, Bedrock, Ollama, DeepSeek, HuggingFace, and more.

MCP setup takes 30 seconds — see MCP Server below.

Full integrations matrix (editors, MCP clients, REST clients, agentic frameworks)

Native Plugin Bundle			MCP Server + Plugin
Claude Code _{Skills · agents · hooks}	Cursor _{Skills · agents}	Codex CLI _{Skills · agents}	Windsurf _plugin	Cline _plugin	Continue _plugin	VS Code _plugin	OpenClaw _{MCP + plugin}
MCP Server	REST API
Claude Desktop _{MCP server}	GitHub Copilot _{REST API}	Roo Code _{REST API}	Goose _{REST API}	Kilo Code _{REST API}	Aider _{REST API}	Amazon Q _{REST API}	Zed _{REST API}

Agentic Frameworks

Native Integration
Agno _{First-class · pip install semantica[agno]}
Already Supported via REST API & MCP
LangChain _{REST API · MCP}	LangGraph _{REST API · MCP}	CrewAI _{REST API · MCP}	LlamaIndex _{REST API · MCP}	AutoGen _{REST API · MCP}	OpenAI Agents _{REST API · MCP}	Google ADK _{REST API · MCP}
Native SDK Integration (Coming Soon)
LangChain _{Dedicated toolkit}	CrewAI _{Dedicated toolkit}	LlamaIndex _{Dedicated toolkit}	AutoGen _{Dedicated toolkit}	OpenAI Agents _{Dedicated toolkit}	Google ADK _{Dedicated toolkit}

MCP Server

Connect any MCP-compatible client (Claude Desktop, Windsurf, Cline, VS Code) in 30 seconds:

python -m semantica.mcp_server
# or via the installed entry point
semantica-mcp

{
  "mcpServers": {
    "semantica": { "command": "python", "args": ["-m", "semantica.mcp_server"] }
  }
}

Tools exposed over MCP:

Tool	What it does
`extract_entities`	NER on any text
`extract_relations`	Relation extraction
`record_decision`	Persist a decision node
`query_decisions`	Search decision history
`find_precedents`	Semantic precedent lookup
`get_causal_chain`	Full causal ancestry
`add_entity`	Add a KG node
`add_relationship`	Add a KG edge
`run_reasoning`	Execute rule set
`get_graph_analytics`	Centrality, communities
`export_graph`	Export to RDF/JSON/Parquet
`get_graph_summary`	Graph statistics

REST API

# Start the backend
python -m semantica.server   # port 8000

# Extract entities & relations via REST
curl -X POST http://localhost:8000/api/enrich/extract \
  -H "Content-Type: application/json" \
  -d '{"text": "Apple CEO Tim Cook announced record earnings."}'

# List recorded decisions
curl "http://localhost:8000/api/decisions?category=vendor_selection"

# Query the knowledge graph
curl "http://localhost:8000/api/graph/node/acme_corp/neighbors?depth=2"

REST endpoints span: enrich (extract) · graph · decisions · reasoning · provenance · ontology · embeddings · search · export · pipeline · temporal · deduplication

Plugin Bundles

Domain skills: extract · ingest · query · ontology · validate · deduplicate · embed · reason · decision · causal · temporal · provenance · policy · explain · export · change · visualize

Specialized agents: kg-assistant · decision-advisor · explainability

Bundles for Claude Code, Cursor, Codex, Windsurf, Cline, Continue, VS Code, and OpenClaw in plugins/.

Knowledge Explorer

A browser-based graph workbench. Pan and zoom live graphs, scrub the timeline, review every decision's causal chain, resolve duplicates, and author your ontology visually. Built on React 19 + Sigma.js.

Workspace	What you can do
Knowledge Graph	Live Sigma.js canvas with ForceAtlas2 layout, Ego Mode, semantic distance heatmap
Timeline	Scrub through temporal events and watch the graph evolve
Decisions	Browse the causal chain behind every recorded decision
Registry	Live audit log of every graph mutation
Entity Resolution	Review and merge duplicates
Ontology Hub	SHACL Studio, visual editor, cross-ontology alignments, SKOS browser
Lineage	W3C PROV-O provenance visualization for any entity

Quickest way to start (no Node.js required):

pip install "semantica[explorer]"
semantica-explorer --graph my_graph.json
# Dashboard opens at http://127.0.0.1:8000

For contributor / dev-server setup: explorer/README.md: Local Setup Guide

What's New in v0.6.5

Security release — upgrading is strongly recommended. Fixes for 5 externally-reported vulnerabilities in the Explorer API and graph/triplet store backends, plus a CodeQL-flagged ReDoS:

Missing authentication on all Explorer API routes (GHSA-j4mq-hprp-987v, Critical): every route now requires SEMANTICA_API_KEY, fails closed (503) rather than open when unconfigured
SSRF via redirect bypass in ontology URL fetching (GHSA-8c7v-62gr-hj6g, High): redirect targets are now re-validated at every hop and the connection is pinned to the validated address, closing a DNS check-then-use race
Cypher injection via unvalidated node labels and property keys (GHSA-482h-hw99-h62p, Critical): Neptune, Neo4j, and FalkorDB now sanitize every label/relationship-type/property-key interpolation site
SPARQL injection via unvalidated triplet IRIs (GHSA-8vgg-8mr4-r236, Critical): Blazegraph, RDF4J, and Jena now validate subject/predicate/object IRIs before interpolation
Missing Origin validation on the WebSocket handshake (GHSA-4643-wpgq-w329, Moderate, anonymous-mode only): /ws/graph-updates now checks Origin against the same allowlist CORSMiddleware enforces for HTTP
Polynomial ReDoS in SPARQL query validation (CodeQL py/polynomial-redos): fixed a backtracking regex in the Explorer's SPARQL route

Also includes: embedded Oxigraph backend for TripletStore, PROV-O trust/spec completeness for ProvenanceManager, and the Altair Anzo triplet store backend.

→ Full release notes · Changelog

Built for High-Stakes Domains

Semantica is designed for environments where AI outputs must be explainable, auditable, and defensible, and where the data itself can't leave your infrastructure. Self-hostable with zero vendor lock-in, it's built as much for organizations handling confidential or classified data as for regulated industries chasing an audit trail:

Finance: Loan underwriting audit trails, fraud detection, AML compliance, regulatory risk knowledge graphs
Healthcare: Clinical decision support, drug interaction graphs, and patient safety audit trails
Legal: Evidence-backed research, contract analysis, case law reasoning, and privilege tracking
Government & Defense: Policy decision records, classified information governance, and regulatory reporting, fully self-hosted with no data leaving your perimeter
Law Enforcement: Case linkage, evidence provenance chains, and investigative knowledge graphs that hold up under legal scrutiny
Cybersecurity: Threat attribution, incident response timelines, and IOC provenance tracking
Autonomous Systems: Decision logs, safety validation, and explainable AI for certification

Installation

pip install semantica           # core
pip install semantica[all]      # everything

pip install semantica[agno]                 # Agno multi-agent integration
pip install semantica[llm-litellm]          # OpenAI, Anthropic, Gemini, Mistral, Llama, Groq, Cohere, Bedrock, Ollama, DeepSeek, and more
pip install semantica[graph-neo4j]          # Neo4j graph store (LPG)
pip install semantica[graph-falkordb]       # FalkorDB graph store (LPG)
pip install semantica[graph-apache-age]     # Apache AGE graph store (LPG)
pip install semantica[graph-amazon-neptune] # AWS Neptune graph store (LPG)
pip install semantica[tripletstore-oxigraph] # Embedded in-memory/on-disk RDF store
# RDF triple stores (Blazegraph, Apache Jena, Eclipse RDF4J) need no extra:
# semantica.triplet_store talks SPARQL over HTTP using the core `requests` dependency
pip install semantica[vectorstore-qdrant]   # Qdrant vector store
pip install semantica[vectorstore-pinecone] # Pinecone vector store
pip install semantica[db-snowflake]         # Snowflake
pip install semantica[db-databricks]        # Databricks (SDK + SQL connector)
pip install semantica[ingest-parquet]       # Parquet / PyArrow
pip install semantica[ingest-arrow]        # Apache Arrow, Feather, IPC
pip install semantica[viz]                  # HTML interactive visualization
pip install semantica[watch]                # Directory file watcher
pip install semantica[explorer]             # Knowledge Explorer dashboard

For production deployments, use Docker or Kubernetes rather than a local pip install. Set SEMANTICA_SECRET_KEY, configure a persistent LPG graph store (Neo4j / FalkorDB / Apache AGE / AWS Neptune) and/or RDF triple store (Blazegraph / Apache Jena / Eclipse RDF4J), and point the vector store at a hosted backend (Qdrant / Pinecone). See ARCHITECTURE.md for the full deployment topology.

# From source
git clone https://github.com/semantica-agi/semantica.git
cd semantica && pip install -e ".[dev]" && pytest tests/

Enterprise

On-premises deployment · Private cloud · Custom domain implementations · SLA-backed support · Professional services for regulated industries (finance, healthcare, legal, government).

getsemantica.ai for enterprise solutions and pricing.

Community & Support


Discord	discord.gg/sV34vps5hH: real-time help, showcases, and announcements
GitHub Discussions	Q&A and feature requests
GitHub Issues	Bug reports
Documentation	docs.getsemantica.ai
Cookbook	Runnable Jupyter notebooks
Changelog	CHANGELOG.md · Release Notes

Star History

Contributors

Contributing

All contributions are welcome: bug fixes, features, tests, and documentation.

Fork the repo and create a branch
pip install -e ".[dev]"
Write tests alongside your changes (pytest tests/)
Open a PR and tag @KaifAhmad1 for review

See CONTRIBUTING.md for full guidelines.

MIT License · Built by Semantica

GitHub · Discord · Twitter/X · Website · Docs · PyPI

If this project helps you build better AI, a star means a lot.

⭐ Star on GitHub →

English · Deutsch · Français · Español · Italiano · Português · العربية · اردو · हिन्दी · 中文 · 日本語 · 한국어

huangruiteng/loopx

Lightweight loop engineering state kernel for long-running AI agent teams. Agent-loop agnostic across Codex, Claude Code, and other coding agents, with durable goals, quota-aware auto-wake, executable todos, evidence logs, and verifiable handoffs.

LoopX

The open, provider-neutral, stateful control plane for long-running agents.

_{Keep objectives, gates, todos, evidence, quota, and handoffs stable while Codex, Claude Code, Cursor, or your own runtime executes bounded turns.}

Public website · Docs · Developer Book · Try LoopX · See real loops · How it works · User manual · 简体中文

把会干活的 Agent，接成可管理、可复盘、可持续改进的数字员工。

Open and provider-neutral, LoopX is a lightweight state kernel and local-first control plane for loop engineering. It keeps long-running work reviewable, restartable, and easier to hand off across turns, tools, and agents without replacing the runtime that performs the work.

Loop engineering for long-running AI agents and peer agent teams.

Keep the loop moving. Keep the judgment human.

Learn LoopX

Developer Book - the curated bilingual path from control-plane foundations to project onboarding and developer contributions. 中文版 · English
Getting started - install, connect a project, and run your first governed loop. Guide
Docs - the full reference and operations site. LoopX Docs

Why LoopX

An agent can finish a task in one session. Long-running work is harder: objectives change, owner decisions appear, evidence goes stale, agents hand work to peers, and a scheduler can keep spending after no useful transition remains. Chat memory and a timer are not enough to govern that.

LoopX keeps the durable control state in one compact layer:

objective / issue / project
   │
   ▼
LoopX state: objective + gates + todos + scope + evidence + quota
   │
   ├─ human judgment needed? ── yes ─▶ ask a concrete question and wait
   │
   ├─ safe fallback available? ──────▶ run one bounded agent slice
   │
   ▼
Codex / Claude Code / Cursor / shell agent executes one turn
   │
   ▼
write evidence + handoff + next todo ─▶ quota decides the next tick

Agent runtimes execute the work. LoopX governs the state that lets engineering, research, discovery, and operations loops continue across runs. It is not another agent framework or a provider-specific orchestration runtime.

A useful mental model is an agent-native Kanban for long-running work. Cards carry identity, authority, evidence, and continuation. Moves are validated operators such as claim, gate, monitor, and writeback. The board is a projection; LoopX state remains the source of truth.

Registered agents are peers. Claims, leases, task boundaries, capabilities, and typed continuation decide who acts next; no durable leader identity is required.

LoopX is useful when you run:

multi-day engineering, research, benchmark, or experiment objectives;
issue and PR loops that must preserve scope, evidence, and review state;
recurring heartbeat or monitor work;
projects with owner, safety, publication, or private-data gates;
peer-agent teams where ownership, leases, and handoff matter;
creator, research, or operations workflows whose progress must remain legible to a non-engineering operator.

LoopX is not an autonomous production controller. Dangerous permissions, publishing, production writes, and final ownership stay with the human.

Evidence

These are not one-turn demos. The public OpenViking contribution sequence and the redacted, owner-run Auto ML showcase each span 200+ hours of elapsed loop lifetime across many bounded turns, decisions, and evidence updates. Elapsed lifetime is wall-clock project time. It is not 200 hours of continuous model execution or a claim of unattended production autonomy. Open each visual to inspect the public-safe graph, evidence branches, and decisions preserved across turns.

Open-Source Issue Fix

200+ hour public contribution arc: PR delivery and reusable fix knowledge evolve together.

LoopX's creator uses this path as an OpenViking contributor. The represented public contribution sequence spans more than 200 elapsed hours from its first PR creation to the latest represented review or update. The Issue-Fix capability keeps rolling repository context, revision-stamped fix knowledge, and reviewer-facing preferences separate; linked PRs plus current checkout source and tests remain authoritative.

Auto ML Experiment

Redacted owner-run showcase: a 200+ hour experiment arc keeps hypotheses, matched evidence, invalid lineages, running replicates, and promote/stop gates visible in one graph.

The redacted public-safe graph preserves decision lineage across that 200+ hour elapsed window. It is an owner-run showcase, not a claim of continuous compute, independent reproduction, a production result, or company or employer endorsement. The redacted image is not sufficient to reproduce the underlying experiment independently.

Auto Research

Reproducible public KNN demo: proposer, executor, and evaluator/promoter agents iterate in parallel while todo, quota, evidence, and targeted wake remain visible.

This screenshot comes from LoopX's built-in exact-KNN demo. The public task, editable and protected files, deterministic CPU evaluator, and dev/held-out commands all live in this repository. Follow the showcase walkthrough or the command path to reproduce the workflow; it is a demo result, not a production research claim.

Used In Real Projects

Independent user · >13h C++ accuracy run. The user reported that a multi-stage task stayed aligned, triggered public research, adopted a public code-memory tool, and improved final precision. Read the evidence boundary.
Independent user · 4d unattended run. The user reported four days without human intervention, useful ongoing work, and a periodic report surface. Read the redacted case.
Independent user · 7 merged PRs. A LoopX-attributed Engine refactor is visible in a public issue and seven merged PRs; attribution and the reported 1B+ token scale remain user reports. Inspect the case.

These are the three strongest current cases, not the full inventory. Browse the complete Showcase catalog for contributor cases, creator dogfooding, reproducible demos, and explicit evidence-strength labels.

More inspectable surfaces:

the public homepage for the product narrative, quick start, and long-running evidence;
the complete Showcase catalog and its bilingual hosted index;
the cross-runtime implementation review demo;
the public user manual.

Try LoopX

Requirements: Python 3.11+, curl, tar, and a macOS or Linux shell. Git is only needed for contributor clone/canary workflows. The Python package has no runtime dependencies outside the standard library.

Install without cloning:

curl -fsSL https://huangruiteng.github.io/loopx/install.sh | bash
export PATH="$HOME/.local/bin:$PATH"
loopx doctor

Then connect from your project root:

cd /path/to/your-project
loopx connect
loopx status

If the project has not been initialized and connect tells you state is missing, use the guided path:

loopx start-goal --guided --project . --goal-text "Your long-running objective"

LoopX should reuse existing state rather than overwrite it. Keep .loopx/, .codex/goals/, and .local/ ignored.

Start From Your Agent

Host	Recommended start	Loop driver
Codex App	Ask the agent to connect this project to LoopX, run `loopx doctor`, preserve existing state, and report the current gate and next todo. Then use `$loopx <complex task>` or choose `loopx` from `/skills`.	Codex App heartbeat automation, refreshed from `quota should-run.scheduler_hint`
Codex App over SSH	`loopx agent-onboard --agent-type codex-app-ssh --project .`	The returned visible `/goal <task_body>`
Codex CLI	Start `codex` in the project, ask it to connect and diagnose LoopX, then use `$loopx <complex task>` or `/skills`.	Visible `/goal <task_body>`; no hidden headless execution by default
Claude Code	Install the opt-in adapter, then run `/loopx <task>` followed by `/loop`.	Native Claude Code `/loop` gated by LoopX
OpenCode	Install the static command facade; opt in to `--with-goal-bridge` for recurring goals.	OpenCode command facade and explicit goal bridge
Pi	Install the opt-in goal extension with `loopx slash-commands --install --surface pi`, then use `/loopx <task>` from a trusted Pi session.	Visible Pi goal extension gated by LoopX quota (`loopx_goal_activate` + `agent_settled` continuation)
Cursor, shell, or custom runner	Use the installer and `loopx doctor`; connect manually or call LoopX from your runner.	Your shell, scheduler, or runner

The exact, copy-ready setup messages and host recovery paths live in Getting Started. Host integrations can inspect the Codex App host command registry contract, the Codex CLI packaged install path, or the Claude Code adapter.

For custom runners, start with the minimal custom runtime example (python3 examples/custom-runtime-minimal-cli-turn-smoke.py), then the full Embed LoopX in Your Agent Runner guide and the worker bridge install contract. The core tick is deliberately small:

loopx quota should-run      # should this registered agent act now?
loopx todo claim            # who owns this slice?
loopx todo update           # what changed?
loopx refresh-state         # what should the next turn see?
loopx quota spend-slot      # account for a completed, validated slice

First-Run Feedback

If LoopX works for you, a one-minute public issue helps us learn what a real first run looks like. It is optional, contains no telemetry, and should not include logs, paths, credentials, internal project names, or goal contents:

loopx first-run-report prints the same prefilled link locally without sending anything.

A successful connection has:

loopx doctor passing;
.loopx/registry.json and a projected active goal state;
loopx status showing the current objective, concrete user gate, and next agent todo;
a visible loop driver or an exact activation instruction;
local runtime state ignored rather than committed.

Clone-based install is only for contributors who want the live canary wrapper:

git clone https://github.com/huangruiteng/loopx ~/loopx
~/loopx/scripts/install-local.sh
loopx doctor

Capabilities

LoopX folds its control-plane mechanics into five questions:

Question	What LoopX keeps visible
What is the objective?	The active goal, explicit scope, and current authority.
What happens next?	Ordered user and agent todos, ownership, claims, and leases.
What needs human judgment?	Concrete user gates instead of a vague "waiting for owner."
What evidence changed?	Compact run history, validation, blockers, and accepted writeback.
May the loop continue?	Quota, capabilities, safe fallback, scheduler hints, and stop conditions.

Control-Plane Surface

Surface	What it does	Start with
Goal state and status	Tracks active state, todos, claims, gates, evidence, run history, and first-screen attention.	`loopx status`, `loopx diagnose`, `loopx review-packet`
Quota and interaction contract	Decides whether a turn should deliver, ask, wait, self-repair, or stay quiet.	`loopx quota should-run`, quota allocation
Agent runtime bridges	Keeps Codex App, Codex CLI, Claude Code, and generic workers aligned with the same guard.	`loopx heartbeat-prompt`, `loopx codex-cli-bootstrap-message`, `loopx worker-bridge`
Operator surfaces	Renders compact status without making the browser the state authority.	`loopx serve-status`, dashboard
External projections	Projects todos and gates into collaboration surfaces while LoopX remains authoritative.	`loopx lark-kanban`, Lark Kanban adapter
Domain capabilities	Packages repeatable work lanes such as issue fixing, content operations, value connector planning, ML experiment advice, benchmark evidence, and Explore.	`loopx issue-fix`, `loopx content-ops`, `loopx value-connectors`, `loopx ml-experiment`, `loopx benchmark`, Explore
Experimental context learning	Lets named registered agents trial provider-neutral Reward Memory through ignored, default-off project configuration. OpenViking is one provider option, not a global dependency.	`loopx reward-memory experiment-status`, Reward Memory architecture
Governance patterns	Captures reusable routing, gate, evidence, projection, and planning shapes.	interaction patterns, state model

The shipped primitives include lifetime goals, concrete user gates, audited safe fallbacks, peer todo ownership, quota and steering, compact run history, evidence-backed handoff, a read-first management surface, project-level value signals, and public/private boundary checks.

Runtime Responsibilities

Role	Responsibility
Agent	Plans, analyzes, uses tools, and performs one bounded action through a host/runtime.
Provider	Calls external systems and returns observations, effect results, and readback.
Capability	Defines the caller outcome, normalizes provider output, validates it, and proposes a typed transition.
Kernel	Owns durable todos, gates, monitors, accepted writeback, quota, recovery, and scheduling.

The execution path is Agent -> Capability -> Provider; the control path returns Provider readback -> Capability transition -> Kernel. An extension is how an optional provider is packaged and managed, not another control-plane owner. See Architecture and Extensions and Capabilities.

Advanced Paths

The first useful loop does not require every optional surface. Add these only when the work needs them.

Inspect the current goal's read-only capability catalog before enabling an advanced path:

loopx configure-goal --goal-id <goal-id>

Without --execute, this reports current/default state, fit, boundaries, and copyable commands without changing project state.

Presets and Auto Research

Safe presets cover daily triage, changelog drafts, and PR watching. The one-command research path coordinates proposer, executor, and evaluator/promoter roles while keeping quota and evidence visible. See the beginner preset guide and Auto Research command path.

loopx preset list
loopx preset show daily-triage

Preset inspection is read-only. For a connected recurring goal, loopx ready-score --goal-id <goal-id> --agent-id <agent-id> reports whether the loop is ready to run repeatedly.

Governed Turns

LoopX can generate one pure, bounded turn decision from a validated receipt, fresh quota state, and a provider-neutral budget. The current Codex CLI quickstart and activation contract are documented in LoopX Turn for Codex CLI.

Explore Graph and Harness

Explore is supported, optional, and default-off. It works best when a task has a measurable offline evaluation, baseline, treatment, and guardrails; it is not a substitute for production approval. Start with the Explore capability and its Lark presentation mapping.

Review Agent Work

Use loopx review-packet for a compact owner-facing view of decisions, evidence, validation, and unresolved gates. The intelligent management surface describes the operator model; the project-level reward model describes conservative value signals across output quantity, quality, token cost, and user attention cost.

For one concrete peer workflow, see the cross-runtime implementation review demo: Claude implements and Codex reviews while LoopX keeps ownership, evidence, quota, and handoff explicit.

App and Projection Paths

Local read-first UI: dashboard guide
Public product overview: public homepage
Documentation portal: hosted docs
Feishu/Lark projection: Lark Kanban adapter
Generic host integration: integration guide
Custom multi-agent runner: minimal custom runtime example, then custom runner integration

Optional projections make state easier to inspect; they do not become the source of truth.

Operating and Recovery

Start daily inspection with:

loopx status
loopx history --goal-id your-project-goal
loopx quota should-run --goal-id your-project-goal

Automatic turns must check quota first and append spend only after validated writeback. Quiet skips, preflight failures, and dry-run previews do not spend. When a user gate blocks one lane, a separately audited safe fallback may continue, but it must not bypass the gate.

Peer agents use loopx todo claim before delivery and loopx todo update after validation so ownership and evidence remain visible.

Scheduler cadence follows quota should-run.scheduler_hint; installed Codex App automations acknowledge the current hint through the returned ack_hint.cli_args. Collision recovery, monitor semantics, self-repair, and the exact operator commands are maintained in Getting Started, Quota Allocation, and Long-Task Cadence Policy.

Before publishing public docs or examples:

loopx check \
  --scan-path README.md \
  --scan-path docs/ \
  --scan-path examples/

Advanced Documentation

Start with the path that matches your current task. Use the hosted documentation portal for the published docs site; the documentation index remains the complete source map. This list stays selective; each category index owns its deeper documents and versioned protocols.

Use and Operate

Getting Started: install, connect, diagnose, daily workflow, heartbeats, dashboard, development, and commands.
User Manual: public onboarding, concepts, FAQ, and selected cases.
Operations: goal continuation, todo, cadence, attention, and authority workflows.
Quota Allocation and Heartbeat Automation Prompt: scheduler eligibility, spend, and scheduled continuation.
Dashboard and Status Data Contract: operator-facing state and projection contracts.
Release Readiness: install/update paths, compatibility gates, release notes, and safe-to-depend-on surfaces.

Understand the Control Plane

Architecture: lifetime-goal invariant and kernel.
State Interaction Model: actors, stores, interaction contract, and writeback.
Concepts: reusable routing, gate, evidence, projection, and planning patterns.
Product Foundations: Loop Engineering principles, project-level reward, and reward-style replanning.
Product Vision: the broader Loop Agent direction.

Integrate and Extend

Integration Guide
Minimal Custom Runtime Example
Custom Agent Runner Integration
Integrations: runtime, host, collaboration, and external-system adapters, including worker bridge and Lark.
Extensions and Capabilities

Build and Review LoopX

Developer Guide: contributor workflows, benchmark development, documentation layout, and quality gates.
Reference and Protocols: stable contracts and versioned implementation protocols, including host command and reward memory architecture.
Control-Plane Developer Course: nine Chinese, code-led lectures.
Testing and Quality: validation layers and risk-based checks.
Public/Private Boundary: safe fixtures, examples, evidence, and publication.

Inspect Outcomes

Showcase Catalog: public-safe cases and evidence labels.
Research and Evidence: benchmark investigations and source-backed findings.
Update Notes: public-safe progress notes.

Project and Community

Partner Projects

LoopX welcomes collaboration with other open-source projects to build the long-running agent ecosystem. Our confirmed partners include:

OpenViking - Self-evolving context database for AI agents
NoKV - AI native distributed file system

Community and Feedback

LoopX is already running real long-running agent goals and is under active development. The most useful feedback comes from real long-running agent projects: where the control plane helped, where it felt heavy, and which gates or handoffs disappeared from view.

Use GitHub Issues for reproducible bugs, install problems, and feature requests.
Open PRs for docs fixes, showcase writeups, and small public-safe examples.
Join the Discord community, or use Lark or WeChat below.

See Support for channel routing and service boundaries, and Communications for official publication sources.

_{Lark: scan to join directly
WeChat: huangrt00 · mention LoopX in the friend request}

Contributing

External contributors should start with Contributor Tasks for public, claimable work and Contributing for setup, validation, and boundary rules. Project roles and public history are recorded in Governance, Authors and Contributors, and Project History.

LoopX keeps local active state separate from the public repository. Do not commit .loopx/, .codex/goals/, live ACTIVE_GOAL_STATE.md, raw benchmark traces, credentials, private logs, or operator artifacts.

Current Status

The v0.4.x line is a usable local control plane for long-running agent work and is entering broader adoption. It is not a full agent platform, an agent runtime, or an autonomous production controller.

Today LoopX ships a durable state kernel for goals, typed todos and decision scopes, peer claims and leases, evidence and writeback, quota-aware scheduling, and cross-turn continuation. Guided start, recurring heartbeat, isolated Codex CLI turns, evidence-backed Issue-Fix admission, optional Explore and auto research paths, public validation canaries, and a read-first multi-project dashboard build on that shared control state.

Support levels remain explicit. The state and CLI contracts are the stable center; several host integrations and advanced paths are optional, default-off, or experimental. LoopX does not grant credentials, approve destructive or production actions, publish on a user's behalf without authorization, or turn an unverified run into evidence of success.

The next milestones are simpler installation and host packaging, broader typed runtime adapters, stronger terminal acceptance across repeated public loops, independent adoption and outcome evidence, and a more polished management surface.

Star History

_{Generated every six hours from GitHub's official stargazer timestamps using a repository-authorized workflow. A snapshot is published only when the fetched rows match GitHub's current star count; GitHub's image cache may delay refreshes.}

License

MIT. See LICENSE.

google/skills

Agent Skills for Google products and technologies

Agent Skills

This repository contains Agent Skills for Google products and technologies, including Google Cloud.

Note

This repository is under active development.

Installation

npx skills add google/skills

From the npx install command, you can select the specific skills from this repo to install.

Available Skills

Additional Google skills

Plugins

This repo also bundles Google product plugins (Skills + MCP servers) for agent harnesses.

Agent harness	Install
Claude Code	`claude plugin marketplace add google/skills`, then `claude plugin install <plugin>@google-plugins`
Codex	`codex plugin marketplace add google/skills`, then install from the `/plugins` browser
Antigravity CLI	`agy plugin install https://github.com/google/skills/<plugin-path>`

Support

If you need help or encounter issues with these skills, search for existing issues or open a new one in the GitHub Issue Tracker.

Contributing

We welcome contributions to improve our skills. You can help by:

Reporting bugs or inaccuracies in the skill Markdown files.
Suggesting new skills to add to this repository (for example, Google technologies or recipes) by filing a feature request.

License

You are free to copy, modify, and distribute these skills under the terms of the Apache 2.0 license. See the LICENSE file for details.

vitali87/code-graph-rag

The ultimate RAG for your monorepo. Query, understand, and edit multi-language codebases with the power of AI and knowledge graphs

Code-Graph-RAG

Code-Graph-RAG parses a multi-language codebase with Tree-sitter, builds a knowledge graph of its structure in Memgraph, and lets you query, edit, and optimise that code in plain English. It works across a monorepo of mixed languages under one unified graph schema.

Latest News 🔥

Ruby Support: Ruby joins the graph through a new pluggable ast-grep tier that adds a language from a single YAML pattern file, emitting Module, Function, and Class nodes plus import edges without a hand-written parser.
Structural Search & Replace: Find and rewrite code by AST pattern with ast-grep, exposed as agent tools so you can match and transform structure across the whole codebase instead of relying on text or regex.
Data-Flow Tracing: New FLOWS_TO taint edges follow values through assignments, function calls, and I/O sinks. This release adds C#, Java, C, and Go, bringing tracing to 10 languages (Python, JavaScript, TypeScript/TSX, Go, Java, Rust, C++, C, and C#).

See NEWS.md for the full history.

What It Does

Point Code-Graph-RAG at a repository and it reads every source file, extracts functions, classes, methods, modules, and the relationships between them, and stores the result as an interconnected graph. Once the graph exists you can:

Ask questions about the codebase in natural language and get answers grounded in the real structure.
Retrieve the actual source of any function, class, or method by name or by intent.
Edit code through the agent with AST-based surgical patching and a diff preview before anything changes.
Optimise code against language best practices or your own coding standards.
Find dead code by walking call and reference edges from entry points.
Search and rewrite structurally by AST pattern with ast-grep.

How It Works

The system has two components:

Multi-language parser. A Tree-sitter based parser reads the codebase and ingests functions, classes, methods, modules, and their relationships into Memgraph under a single language-agnostic schema.
RAG system (codebase_rag/). An interactive CLI that turns natural language into Cypher queries, retrieves matching code, and drives AI-powered editing and optimisation.

Source Code -> Tree-sitter Parser -> AST Analysis -> Memgraph Knowledge Graph
                                                             |
User Query -> AI Model (Cypher Gen) -> Cypher Query -> Graph Results -> Response

See the Architecture Overview and Graph Schema for the full picture.

Supported Languages

Python, TypeScript, TSX, JavaScript, Rust, Go, Java, C, C++, C#, PHP, Lua, and Dart are fully supported. Scala is in development, and Ruby has structural support (modules, functions, classes, and imports) through the pluggable ast-grep tier. See the Language Support matrix for per-language capabilities.

Installation

cgr is published to PyPI. Install it system-wide with the treesitter-full (all languages) and semantic (vector search) extras:

# with uv (recommended)
uv tool install "code-graph-rag[treesitter-full,semantic]"

# or with pipx
pipx install "code-graph-rag[treesitter-full,semantic]"

You also need Docker (for Memgraph), cmake, and ripgrep. Full prerequisites, source installs, and environment setup are in the Installation guide.

Quick Start

# Start the packaged Memgraph + Qdrant stack (no compose file needed)
cgr daemon up

# Parse a repository into the graph, then query it
cgr start --repo-path /path/to/repo --update-graph
cgr start --repo-path /path/to/repo

Repeat the first command for each repository you want indexed; the graph is shared, and syncing one project leaves the others alone. To start over from an empty graph, add --clean — it deletes every project in the shared graph, not just this one, and asks for confirmation first when other projects would be destroyed.

The Quick Start guide walks through parsing, querying, and exporting in five minutes.

MCP Server

Code-Graph-RAG runs as an MCP server so Claude Code and other MCP clients can query and edit your codebase directly. See the MCP Server guide for setup.

Documentation

Getting Started

User Guide

Architecture

Python SDK

Advanced

Enterprise Services

Code-Graph-RAG is open source and free to use. For organisations that need more, we offer fully managed cloud-hosted solutions and on-premise deployments:

Cloud-Hosted Deployment: Managed cloud infrastructure for both the graph database and the AI agent connection. Zero infrastructure overhead, so we handle scaling, updates, and availability while your team focuses on building.
On-Premise & Air-Gapped Deployment: Deploy Code-Graph-RAG entirely within your own environment, including air-gapped networks. Full data sovereignty for regulated industries and security-sensitive organisations.

We also offer custom development, integration consulting, technical support contracts, and team training.

View plans & pricing at code-graph-rag.com

Contributing

Please see CONTRIBUTING.md for contribution guidelines. Good first PRs come from the TODO issues.

Support

For issues or questions, check the Troubleshooting guide first, then open an issue.

License

MIT. See LICENSE.

virgiliojr94/book-to-skill

Turn any technical book PDF into a Claude Code skill — ready to study, reference, and use while you work.

book-to-skill

Turn any technical book, document folder, or collection of sources into a unified agent skill — ready to study, reference, and use while you work in GitHub Copilot CLI, Amp, or Claude Code.

Why · What it generates · Beyond books · How it works · Usage · Install · FAQ · Performance · Architecture · Changelog

24×–51× fewer tokens than dumping the book into context to answer one question, measured on real books (how it's measured).

How it works, in 3 steps:

Point it at a file, folder, or glob — /book-to-skill ./my-book.pdf
It distills the book into a skill — frameworks, decision rules, anti-patterns, and per-chapter files. Structure, not a summary.
Your agent loads it on demand — ask /my-book replication and it reads the right chapter and answers from the real content, no hallucination.

🤔 Why

You buy a great technical book. You read it once. Three months later you can't remember chapter 7 existed.

The usual workarounds don't help:

📄 "Let me just search the PDF" → you get a list of pages, not answers
🧠 "I'll ask the agent about this book" → it either hallucinates or says it doesn't have the content
📝 "I'll take notes as I read" → you end up with a 200-line doc you never open again

book-to-skill solves this by turning the book into a structured skill your agent loads on demand.

Once installed, you just type /your-book-slug replication and the agent reads the right chapter and answers from the actual content. No hallucination. No digging through PDFs. The book becomes part of your workflow.

Works with any host that supports the open Agent Skills standard — GitHub Copilot CLI, Amp, and Claude Code all read the same SKILL.md format.

📦 What it generates

Running /book-to-skill your-book.pdf (or a folder, glob, or list of files) creates a full skill in your agent's skills directory (~/.copilot/skills/<slug>/ for Copilot CLI, ~/.agents/skills/<slug>/ for Amp or cross-agent, ~/.claude/skills/<slug>/ for Claude Code):

File	Purpose	Size
`SKILL.md`	Core mental models + chapter index	~4,000 tokens
`chapters/ch01-*.md` …	One file per chapter, loaded on-demand	~1,000 tokens each
`glossary.md`	Every key term, alphabetically sorted with chapter refs	~1,500 tokens
`patterns.md`	All techniques, algorithms, and design patterns	~2,000 tokens
`cheatsheet.md`	Decision tables and quick-reference rules	~1,000 tokens

Chapter files are loaded on-demand — they don't count against the skill budget until you ask about that topic.

🏢 Beyond books

The name says "book", but the input is any structured prose. The same extraction works on knowledge you own and re-read constantly:

Internal documentation — architecture decision records, runbooks, onboarding guides. Fold a whole docs/ folder into one skill and ask it while you code.
Brand & design systems — voice guidelines, tone-of-voice docs, component principles. Turn a brand book into a skill your team queries instead of skimming a 60-page PDF.
Research clusters — a stack of papers plus your own notes, merged into a single unified skill and updated as new material lands (see Update / fold-in).
Specs & standards — RFCs, API contracts, compliance docs you reference but never memorize.

If you re-open a document often enough to wish you'd memorized it, it's a candidate.

🧾 The Discovery Loop Tax

A PDF-reading agent doesn't just read — it navigates: it re-fetches the ToC, backtracks, and re-processes all of it on every turn. book-to-skill pays that structuring cost once, at conversion, so queries stay proportional to the answer — 24×–51× fewer tokens than dumping the book into context, measured on real books.

📊 Full methodology, numbers, and per-book tables → docs/performance.md

⚙️ How it works

Two halves: a deterministic Python extractor (document → clean text + metadata) and a spec-driven generator (your agent follows SKILL.md to turn that into a structured skill). On-demand chapter files keep the loaded skill small.

🔧 Full walkthrough (Steps 0–10, extraction modes, token budgets) → docs/how-it-works.md

🚀 Usage

/book-to-skill <path|folder|glob> [skill-name] — plus analyze-only, generate-from-analysis, and update/fold-in modes.

▶️ All modes and examples → docs/usage.md

💬 In practice → use cases — a DevEx book became a survey of 300+ engineers; a scanned PDF that stalled became #130. Add yours: the account lives in your own Gist, the index takes a one-line PR.

📥 Install

# One command, any host — via the cross-agent skills CLI:
npx skills add virgiliojr94/book-to-skill

# Or manually — clone into your skills folder (registers /book-to-skill):
git clone https://github.com/virgiliojr94/book-to-skill.git ~/.claude/skills/book-to-skill
# (Copilot CLI: ~/.copilot/skills/ · Amp/cross-agent: ~/.agents/skills/)

📥 All hosts, optional extractors, and the standalone CLI → docs/install.md

❓ FAQ

Common questions — "why not just dump the PDF?", cost, privacy, non-book inputs, multi-file books.

❓ Answers → docs/faq.md

🔧 Requirements

The extractor tries tools in order per format and uses the first available. If nothing is installed, it tells you which command to run. Plain text, Markdown, reStructuredText and AsciiDoc need no extra deps.

Check your setup in one command: python3 scripts/extract.py --check prints which extractors are installed for every format and the exact command to install anything missing — no file needed.

PDF — choose by book type:

Book type	Tool	Install	Speed
Text-heavy (prose, few tables)	`pdftotext` (poppler)	`sudo apt install poppler-utils`	⚡ instant
Text-heavy fallback	`pypdf`	`pip3 install pypdf`	⚡ instant
Text-heavy fallback	`pdfminer.six`	`pip3 install pdfminer.six`	⚡ instant
Technical (code, tables, formulas)	`docling`	`pip3 install docling`	~1.5s/page

Before extraction begins, the skill asks you whether the book is technical or text-heavy and picks the right tool automatically. Docling preserves markdown tables and code blocks; pdftotext is faster for prose-only books.

Scanned PDFs need OCR first. A PDF that is page images with no text layer — a photographed or scanned book — has nothing for these tools to extract. The extractor checks the first pages and stops immediately with an explanation, rather than working through the whole book to produce an empty skill. Run OCR yourself, then convert the result:
ocrmypdf input.pdf output.pdf

EPUB:

Tool	Install	Quality
`ebooklib` + `beautifulsoup4`	`pip3 install ebooklib beautifulsoup4`	⭐⭐⭐ Best
stdlib `zipfile`	built-in — no install needed	⭐⭐ Always available

Other formats:

Format	Tool	Install
DOCX	`python-docx` (fallback: stdlib ZIP/XML)	`pip3 install python-docx`
HTML	`beautifulsoup4` (fallback: stdlib `html.parser`)	`pip3 install beautifulsoup4`
RTF	`striprtf` (fallback: regex)	`pip3 install striprtf`
MOBI / AZW / AZW3	Calibre `ebook-convert` (external app, not pip)	https://calibre-ebook.com/download
TXT / Markdown / reStructuredText / AsciiDoc	built-in	—

📁 Repository structure

book-to-skill/
├── SKILL.md              # Skill definition + step-by-step instructions (the generator spec)
├── scripts/
│   ├── extract.py        # Thin entrypoint wrapper
│   └── extractor/        # Modular extraction package
│       ├── config.py     # Extensions, paths, dependency constants
│       ├── dependencies.py  # optional-dep probing + --check
│       ├── exceptions.py # ExtractionError (per-source failures, batch-safe)
│       ├── utils.py      # CLI parsing, multi-source resolution, chapter detection, runner
│       └── parsers/      # Format-specific parsers (pdf, epub, docx, html, rtf, calibre, text)
├── tools/
│   ├── discovery_tax.py  # measures token cost vs context-dump / discovery loop
│   └── validate_skill.py # checks a generated SKILL.md against host rules (--lens claude|copilot|amp)
├── tests/                # pytest suite (extraction, detection, discovery tax)
├── docs/
│   ├── performance.md    # measured benchmarks, discovery tax, cost
│   └── architecture.md   # pipeline + component map
├── CHANGELOG.md          # release history (semver)
├── CONTRIBUTING.md       # dev setup, PR conventions, release process
├── SECURITY.md           # vulnerability reporting
└── README.md             # This file

⚖️ Copyright & fair use

book-to-skill ships no book content — not a single page. It's a converter you point at files you already own.

Processing is local. Extraction and analysis run on your machine. Your files are never uploaded by this tool. (If your agent's model runs in the cloud, the text you feed it follows that provider's normal data terms — same as any prompt.)
You use your own copy. Bring a book you bought, docs your company owns, or papers you have the right to read.
The output is your notes. A generated skill is a structured, synthesized derivative — framework names, definitions, takeaways — not a reproduction of the text. The skill explicitly never copies raw passages (see Quality Rule #7). Treat it like handwritten study notes: yours, for personal use.
Don't redistribute. Publishing or sharing a generated skill of a copyrighted work can infringe the rights holder. Keep skills of third-party books private. Internal docs, your own writing, and openly-licensed material are fine to share within the bounds of their license.

When in doubt, follow the license or terms of the source document. This project is a tool; how you use it is on you.

💖 Sponsors

book-to-skill is free and MIT-licensed, maintained on personal time. If it saves you tokens or study hours, consider sponsoring its upkeep: PR reviews, multilingual fixes, releases, and docs.

Become a sponsor → github.com/sponsors/virgiliojr94

Every sponsor is listed in BACKERS.md. Thank you for keeping open, privacy-first tooling alive. ✨

License

MIT — applies to the converter (code + skill definition) in this repository, not to any book or document you process with it.

Star History

Comfy-Org/ComfyUI

The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface.

ComfyUI

The most powerful and modular AI engine for content creation.

ComfyUI is the AI creation engine for visual professionals who demand control over every model, every parameter, and every output. Its powerful and modular node graph interface empowers creatives to generate images, videos, 3D models, audio, and more...

ComfyUI natively supports the latest open-source state of the art models.
Partner nodes provide access to the best closed source models such as Nano Banana, Seedance, Hunyuan3D, etc.
It is available on Windows, Linux, and macOS, locally with our desktop application, our portable install or on our cloud.
The most sophisticated workflows can be exposed through a simple UI thanks to App Mode.
It integrates seamlessly into production pipelines with our API endpoints.

Get Started

Local

Desktop Application

The easiest way to get started.
Available on Windows & macOS.

Windows Portable Package

Get the latest commits and completely portable.
Available on Windows.

Manual Install

Supports all operating systems and GPU types (NVIDIA, AMD, Intel, Apple Silicon, Ascend).

Cloud

Comfy Cloud

Our official paid cloud version for those who can't afford local hardware.

Examples

See what ComfyUI can do with the newer template workflows or old example workflows.

Features

A visual node graph for building and reusing image, video, audio, 3D, and text workflows without code.
Reusable subgraphs, workflow templates, App Mode, and a local API for integrating workflows into applications.
Efficient local execution with asynchronous queueing, partial graph re-execution, smart VRAM and RAM management, model offloading, and support for quantized models.
Broad native model support. This is a representative list; browse the workflow library for maintained, ready-to-run templates.
- Image generation: Stable Diffusion 1.5, SDXL, SD3.5, Flux.1, Flux.2, Qwen Image, Z-Image, Hunyuan Image 2.1, HiDream, Lumina Image 2.0, Chroma, Anima, LongCat Image, Ideogram 4, Krea 2, MageFlow, Microsoft Lens, PixelDiT, Kandinsky 5, and Ernie Image.
- Image editing: Flux Kontext, Flux.2 Klein, Qwen Image Edit, HiDream E1.1 and O1, OmniGen2, Boogu, JoyImage Edit, MageFlow Edit, and LongCat Image Edit.
- Video generation: Wan 2.1 and 2.2, LTX-Video 2 and 2.3, HunyuanVideo 1.5, Kandinsky 5 Video, CogVideoX, Cosmos Predict2, Bernini-R, SCAIL 2, and Mochi.
- Audio and video generation: MiniMax H3 and LTX-AV.
- Audio generation: ACE-Step 1.5 and Stable Audio 3.
- 3D and vision: Hunyuan3D 2.1, TripoSplat, SeedVR2, SUPIR, Depth Anything 3, MoGe, SAM 3 and 3.1, RT-DETRv4, and BiRefNet.
- Text generation: Gemma 3 and 4, Qwen3, Qwen3.5, and Qwen3-VL, including multimodal inputs.
Load complete checkpoints or separate diffusion models, VAEs, text encoders, LoRAs, ControlNets, adapters, and upscalers from supported model formats.
Built-in tools for inpainting, outpainting, reference conditioning, masks and compositing, model merging, upscaling, frame interpolation, segmentation, depth estimation, and media processing.
Save and load workflows as JSON, or recover complete workflows and seeds from supported generated media.
Runs fully offline: core does not download anything unless you request it. Use --disable-api-nodes to disable the optional paid Comfy API nodes and force all built-in functionality to stay offline.
Extend ComfyUI with custom nodes
Configure additional model locations with extra_model_paths.yaml.

Release Process

ComfyUI follows a weekly release cycle targeting Monday but this regularly changes because of model releases or large changes to the codebase. There are three interconnected repositories:

ComfyUI Core
- Releases a new major stable version (e.g., v0.7.0) roughly every 2 weeks.
- Starting from v0.4.0 patch versions will be used for fixes backported onto the current stable release.
- Minor versions will be used for releases off the master branch.
- Patch versions may still be used for releases on the master branch in cases where a backport would not make sense.
- Commits outside of the stable release tags may be very unstable and break many custom nodes.
- Serves as the foundation for the desktop release
Comfy Desktop
- Builds a new release using the latest stable core version
ComfyUI Frontend
- Every 2+ weeks frontend updates are merged into the core repository
- Features are frozen for the upcoming core release
- Development continues for the next release cycle

Shortcuts

Keybind	Explanation
`Ctrl` + `Enter`	Queue up current graph for generation
`Ctrl` + `Shift` + `Enter`	Queue up current graph as first for generation
`Ctrl` + `Alt` + `Enter`	Cancel current generation
`Ctrl` + `Z`/`Ctrl` + `Y`	Undo/Redo
`Ctrl` + `S`	Save workflow
`Ctrl` + `O`	Load workflow
`Ctrl` + `A`	Select all nodes
`Alt` + `C`	Collapse/uncollapse selected nodes
`Ctrl` + `M`	Mute/unmute selected nodes
`Ctrl` + `B`	Bypass selected nodes (acts like the node was removed from the graph and the wires reconnected through)
`Delete`/`Backspace`	Delete selected nodes
`Ctrl` + `Backspace`	Delete the current graph
`Space`	Move the canvas around when held and moving the cursor
`Ctrl`/`Shift` + `Click`	Add clicked node to selection
`Ctrl` + `C`/`Ctrl` + `V`	Copy and paste selected nodes (without maintaining connections to outputs of unselected nodes)
`Ctrl` + `C`/`Ctrl` + `Shift` + `V`	Copy and paste selected nodes (maintaining connections from outputs of unselected nodes to inputs of pasted nodes)
`Shift` + `Drag`	Move multiple selected nodes at the same time
`Ctrl` + `D`	Load default graph
`Alt` + `+`	Canvas Zoom in
`Alt` + `-`	Canvas Zoom out
`Ctrl` + `Shift` + LMB + Vertical drag	Canvas Zoom in/out
`P`	Pin/Unpin selected nodes
`Ctrl` + `G`	Group selected nodes
`Q`	Toggle visibility of the queue
`H`	Toggle visibility of history
`R`	Refresh graph
`F`	Show/Hide menu
`.`	Fit view to selection (Whole graph when nothing is selected)
Double-Click LMB	Open node quick search palette
`Shift` + Drag	Move multiple wires at once
`Ctrl` + `Alt` + LMB	Disconnect all wires from clicked slot

Ctrl can also be replaced with Cmd instead for macOS users

Installing

Windows and Mac

We highly recommend using the desktop app:

Link to Download

The desktop app is the easiest and best way to use ComfyUI for new users.

Windows Portable

There is a portable standalone build for Windows that should work for running on Nvidia GPUs or for running on your CPU only. It is not recommended for regular users. Regular users should use the desktop app above.

Direct link to download (nvidia)

Simply download, extract with 7-Zip or with the windows explorer on recent windows versions and run. For smaller models you normally only need to put the checkpoints (the huge ckpt/safetensors files) in: ComfyUI\models\checkpoints but many of the larger models have multiple files. Make sure to follow the instructions to know which subfolder to put them in ComfyUI\models\

If you have trouble extracting it, right click the file -> properties -> unblock

The portable above currently comes with python 3.13 and pytorch cuda 13.0. Update your Nvidia drivers if it doesn't start.

All Official Portable Downloads:

Portable for AMD GPUs

Portable for Intel GPUs

Portable for Nvidia GPUs (supports 20 series and above).

Portable for Nvidia GPUs with pytorch cuda 12.6 and python 3.12 (Supports Nvidia 10 series and older GPUs, DO NOT USE THIS ON NEWER 20 SERIES AND ABOVE GPUS).

How do I share models between another UI and ComfyUI?

See the Config file to set the search paths for models. In the standalone windows build you can find this file in the ComfyUI directory. Rename this file to extra_model_paths.yaml and edit it with your favorite text editor.

comfy-cli

You can install and start ComfyUI using comfy-cli:

pip install comfy-cli
comfy install

Manual Install (Windows, Linux)

Python 3.14 works but some custom nodes may have issues. The free threaded variant works but some dependencies will enable the GIL so it's not fully supported.

Python 3.13 is very well supported. If you have trouble with some custom node dependencies on 3.13 you can try 3.12

torch 2.7 is minimally supported but using a newer version is extremely recommended. Using a cu130 or above version of pytorch is required on Nvidia 20 series and above. Some features and optimizations might only work on newer versions. We generally recommend using the latest major version of pytorch with the latest cuda version unless it is less than 2 weeks old. If your pytorch is more than 6 months old, please update it.

Instructions:

Git clone this repo.

Put your SD checkpoints (the huge ckpt/safetensors files) in: models/checkpoints

Put your VAE in: models/vae

AMD GPUs (Linux)

AMD users can install rocm and pytorch with pip if you don't have it already installed, this is the command to install the stable version:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm7.2

This is the command to install the nightly with ROCm 7.2 which might have some performance improvements:

pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/rocm7.2

AMD GPUs (Experimental: Windows and Linux), RDNA 3, 3.5 and 4 only.

These have less hardware support than the builds above but they work on windows. You also need to install the pytorch version specific to your hardware.

RDNA 3 (RX 7000 series):

pip install --pre torch torchvision torchaudio --index-url https://rocm.nightlies.amd.com/v2/gfx110X-all/

RDNA 3.5 (Strix halo/Ryzen AI Max+ 365):

pip install --pre torch torchvision torchaudio --index-url https://rocm.nightlies.amd.com/v2/gfx1151/

RDNA 4 (RX 9000 series):

pip install --pre torch torchvision torchaudio --index-url https://rocm.nightlies.amd.com/v2/gfx120X-all/

Intel GPUs (Windows and Linux)

Intel Arc GPU users can install native PyTorch with torch.xpu support using pip. More information can be found here

To install PyTorch xpu, use the following command:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/xpu

This is the command to install the Pytorch xpu nightly which might have some performance improvements:

pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/xpu

NVIDIA

Nvidia users should install stable pytorch using this command:

pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu130

This is the command to install pytorch nightly instead which might have performance improvements.

pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu132

Troubleshooting

If you get the "Torch not compiled with CUDA enabled" error, uninstall torch with:

pip uninstall torch

And install it again with the command above.

Dependencies

Install the dependencies by opening your terminal inside the ComfyUI folder and:

pip install -r requirements.txt

After this you should have everything installed and can proceed to running ComfyUI.

Others:

Apple Mac silicon

You can install ComfyUI in Apple Mac silicon (M1, M2, M3 or M4) with any recent macOS version.

Install pytorch nightly. For instructions, read the Accelerated PyTorch training on Mac Apple Developer guide (make sure to install the latest pytorch nightly).
Follow the ComfyUI manual installation instructions for Windows and Linux.
Install the ComfyUI dependencies. If you have another Stable Diffusion UI you might be able to reuse the dependencies.
Launch ComfyUI by running python main.py

Note: Remember to add your models, VAE, LoRAs etc. to the corresponding Comfy folders, as discussed in ComfyUI manual installation.

Ascend NPUs

For models compatible with Ascend Extension for PyTorch (torch_npu). To get started, ensure your environment meets the prerequisites outlined on the installation page. Here's a step-by-step guide tailored to your platform and installation method:

Begin by installing the recommended or newer kernel version for Linux as specified in the Installation page of torch-npu, if necessary.
Proceed with the installation of Ascend Basekit, which includes the driver, firmware, and CANN, following the instructions provided for your specific platform.
Next, install the necessary packages for torch-npu by adhering to the platform-specific instructions on the Installation page.
Finally, adhere to the ComfyUI manual installation guide for Linux. Once all components are installed, you can run ComfyUI as described earlier.

Cambricon MLUs

For models compatible with Cambricon Extension for PyTorch (torch_mlu). Here's a step-by-step guide tailored to your platform and installation method:

Install the Cambricon CNToolkit by adhering to the platform-specific instructions on the Installation
Next, install the PyTorch(torch_mlu) following the instructions on the Installation
Launch ComfyUI by running python main.py

Iluvatar Corex

For models compatible with Iluvatar Extension for PyTorch. Here's a step-by-step guide tailored to your platform and installation method:

Install the Iluvatar Corex Toolkit by adhering to the platform-specific instructions on the Installation
Launch ComfyUI by running python main.py

ComfyUI-Manager

ComfyUI-Manager is an extension that allows you to easily install, update, and manage custom nodes for ComfyUI.

Setup

Install the manager dependencies:

pip install -r manager_requirements.txt

Enable the manager with the --enable-manager flag when running ComfyUI:
```
python main.py --enable-manager
```

Command Line Options

Flag	Description
`--enable-manager`	Enable ComfyUI-Manager
`--enable-manager-legacy-ui`	Use the legacy manager UI instead of the new UI (implies `--enable-manager`)
`--disable-manager-ui`	Disable the manager UI and endpoints while keeping background features like security checks and scheduled installation completion (requires `--enable-manager`)

Running

python main.py

For AMD cards not officially supported by ROCm

Try running it with this command if you have issues:

For 6700, 6600 and maybe other RDNA2 or older: HSA_OVERRIDE_GFX_VERSION=10.3.0 python main.py

For AMD 7600 and maybe other RDNA3 cards: HSA_OVERRIDE_GFX_VERSION=11.0.0 python main.py

AMD ROCm Tips

You can try setting this env variable PYTORCH_TUNABLEOP_ENABLED=1 which might speed things up at the cost of a very slow initial run.

Notes

Only parts of the graph that have an output with all the correct inputs will be executed.

Only parts of the graph that change from each execution to the next will be executed, if you submit the same graph twice only the first will be executed. If you change the last part of the graph only the part you changed and the part that depends on it will be executed.

Dragging a generated png on the webpage or loading one will give you the full workflow including seeds that were used to create it.

You can use () to change emphasis of a word or phrase like: (good code:1.2) or (bad code:0.8). The default emphasis for () is 1.1. To use () characters in your actual prompt escape them like $ or $.

You can use {day|night}, for wildcard/dynamic prompts. With this syntax "{wild|card|test}" will be randomly replaced by either "wild", "card" or "test" by the frontend every time you queue the prompt. To use {} characters in your actual prompt escape them like: \{ or \}.

Dynamic prompts also support C-style comments, like // comment or /* comment */.

To use a textual inversion concepts/embeddings in a text prompt put them in the models/embeddings directory and use them in the CLIPTextEncode node like this (you can omit the .pt extension):

embedding:embedding_filename.pt

How to show high-quality previews?

Use --preview-method auto to enable previews.

The default installation includes a fast latent preview method that's low-resolution. To enable higher-quality previews with TAESD, download the taesd_decoder.pth, taesdxl_decoder.pth, taesd3_decoder.pth and taef1_decoder.pth and place them in the models/vae_approx folder. Once they're installed, restart ComfyUI and launch it with --preview-method taesd to enable high-quality previews.

How to use TLS/SSL?

Generate a self-signed certificate (not appropriate for shared/production use) and key by running the command: openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -sha256 -days 3650 -nodes -subj "/C=XX/ST=StateName/L=CityName/O=CompanyName/OU=CompanySectionName/CN=CommonNameOrHostname"

Use --tls-keyfile key.pem --tls-certfile cert.pem to enable TLS/SSL, the app will now be accessible with https://... instead of http://....

Note: Windows users can use alexisrolland/docker-openssl or one of the 3rd party binary distributions to run the command example above.

If you use a container, note that the volume mount -v can be a relative path so ... -v ".\:/openssl-certs" ... would create the key & cert files in the current directory of your command prompt or powershell terminal.

Support and dev channel

Discord: Try the #help or #feedback channels.

Matrix space: #comfyui_space:matrix.org (it's like discord but open source).

Frontend Development

As of August 15, 2024, we have transitioned to a new frontend, which is now hosted in a separate repository: ComfyUI Frontend. The compiled JS files (from TS/Vue) are published to pypi and installed as a dependency in ComfyUI.

Reporting Issues and Requesting Features

For any bugs, issues, or feature requests related to the frontend, please use the ComfyUI Frontend repository. This will help us manage and address frontend-specific concerns more efficiently.

Using the Latest Frontend

The new frontend is now the default for ComfyUI. However, please note:

The frontend in the main ComfyUI repository is updated fortnightly.
Daily releases are available in the separate frontend repository.

To use the most up-to-date frontend version:

For the latest daily release, launch ComfyUI with this command line argument:
```
--front-end-version Comfy-Org/ComfyUI_frontend@latest
```
For a specific version, replace latest with the desired version number:
```
--front-end-version Comfy-Org/ComfyUI_frontend@1.2.2
```

This approach allows you to easily switch between the stable fortnightly release and the cutting-edge daily updates, or even specific versions for testing purposes.

QA

Which GPU should I buy for this?

See this page for some recommendations

goauthentik/authentik

The authentication glue you need.

What is authentik?

authentik is an open-source Identity Provider (IdP) for modern SSO. It supports SAML, OAuth2/OIDC, LDAP, RADIUS, and more, designed for self-hosting from small labs to large production clusters.

Our enterprise offering is available for organizations to securely replace existing IdPs such as Okta, Auth0, Entra ID, and Ping Identity for robust, large-scale identity management.

Installation

Docker Compose: recommended for small/test setups. See the documentation.
Kubernetes (Helm Chart): recommended for larger setups. See the documentation and the Helm chart repository.
AWS CloudFormation: deploy on AWS using our official templates. See the documentation.
DigitalOcean Marketplace: one-click deployment via the official Marketplace app. See the app listing.

Screenshots

Light	Dark

Development and contributions

See the Developer Documentation for information about setting up local build environments, testing your contributions, and our contribution process.

When you contribute documentation, either to accompany a code change or as a standalone contribution, please be sure to follow our documentation Style Guide.

Security

Please see SECURITY.md.

Adoption

Using authentik? We'd love to hear your story and feature your logo. Email us at hello@goauthentik.io or open a GitHub Issue/PR!

License

uber/ADR

ADR secures enterprise AI agents through observability, security benchmarking, and threat detection. Deployed at Uber.

ADR: Agentic AI Detection and Response

ADR (Agentic AI Detection and Response) is an enterprise security system for AI agents. It helps organizations secure employee-facing agents such as Cursor, Claude Code, and Codex, as well as customer-facing agents such as AI support agents.

ADR is deployed in production at Uber, and the accompanying paper was accepted to MLSys 2026: Paper PDF · Slides PDF

How ADR secures enterprise AI agents

ADR secures enterprise AI agents through four complementary capabilities: observing agent activity, evaluating defenses, detecting threats, and preventing unsafe actions.

ADR Observability: Understand what AI agents are doing and why. In production, ADR captures agent intent, tool use, and execution traces across 7+ AI coding tools on macOS, Linux, and Windows, as well as internal automation and customer-facing support agents.
ADR Benchmark: Test agent security under realistic enterprise conditions. ADR-Bench includes 300+ tasks, 133 MCP servers, and coverage of all 17 agent attack techniques.
ADR Detection: Detect risky agent behavior efficiently. Its two-tier architecture combines high-recall triage with deeper agentic reasoning for suspicious sessions.
ADR Prevention: Stop unsafe actions before they cause harm. This component is not included in the current open-source release. Stay tuned.

Repository layout

This repository contains the open-source ADR Sensor, ADR-Bench, and ADR Detector described in the paper. The offline ADR Explorer engine, which hardens ADR Detection through pre-deployment red teaming, is not included here.

Path	ADR component	Description
Sensor/	ADR Observability	Collect and normalize agent telemetry from Claude Code, Cursor, Codex, opencode, Claude Desktop, and others
Detection/	ADR Benchmark + Detection	Dual-agent detector, 133 MCP servers, 303 benchmark tasks, baselines, figure scripts
docs/REPRODUCIBILITY.md	Evaluation	Step-by-step workflow to reproduce benchmark detection and paper figures

Quick start: ADR Detection

git clone https://github.com/uber/ADR
cd ADR/Detection
uv sync
export ANTHROPIC_API_KEY="..." OPENAI_API_KEY="..."

Default detector is adr (ADR dual-agent). For keyless smoke tests, use --detector llamafirewall (see Detection/README.md).

See docs/REPRODUCIBILITY.md for the full evaluation workflow (inflate packed benchmark → run detectors → plot figures).

Component documentation:

Sensor/README.md: telemetry collection and unified schema
Detection/README.md: ADR-Bench, detector baselines, MCP infrastructure

Citation

@inproceedings{li2026adr,
  title={ADR: An Agentic Detection System for Enterprise Agentic AI Security},
  author={Li, Chenning and Hu, Pan and Xu, Justin and Ozbas, Baris and Liu, Olivia and Van, Caroline and Li, Manxue and Zhou, Wei and Alizadeh, Mohammad and Zhang, Pengyu and Sriramadhesikan, KK and Zhang, Ming},
  booktitle={Proceedings of the Ninth Conference on Machine Learning and Systems},
  year={2026}
}

Or use CITATION.cff.

License

Apache License 2.0. See LICENSE. Detection/benchmark/agentdojo/ is vendored third-party code under its own LICENSE (MIT).

Data notice

Detection/ includes synthetic benchmark fixtures (fake credentials, emulated environments, prompt-injection scenarios) for defensive security research only. Details: docs/OPEN_SOURCE_REVIEW.md.

android/skills

Android skills

Android skills are a dedicated repository of AI-optimized, modular instructions and resources, to help agents better understand and execute specific patterns that follow the best practices and guidance on Android development from developer.android.com.

Android skills follow the open-standard agent skills - markdown files (SKILL.md) that provide a technical specification of a task, and ground LLMs with information on specialized domains and workflows.

Our Android skill development focuses on use cases and workflows where evaluations show LLMs underperform. We aren't prioritizing well-established areas where LLMs are already proficient, such as basic Jetpack Compose best practices.

To learn more, read the official documentation:

Install Android skills

Use Android CLI to install a specific skill into the current directory:

android skills add --skill=r8-analyzer --project=.

Use Android CLI to install all Android skills to directories for all detected agents:

android skills add --all

If you don't have any existing agent directories and don't specify particular agents, the skills will be installed for Gemini and Antigravity at ~/.gemini/antigravity/skills.

Options:

--all - Add all Android skills. If omitted (and --skill isn't specified), only the android-cli skill will be installed.
--agent - A comma-separated list of agents to install the skill for. If omitted, the skill will be installed for all detected agents.
--skill - Specific skill that you want to install. If omitted (and --all isn't specified), only the android-cli skill will be installed.
--project - Path to a project root in which to install the skills.

Activate skills

Your agent automatically activates skills that are relevant to your task. To use a skill, prompt your agent to complete a task that is related to the skill, for example "Make my app UI edge-to-edge." The agent should automatically find and use the skill if it has been installed.

Disclaimer

AI can make mistakes, so always double-check the results.

Contributing

Submit a GitHub issue to provide feedback, report issues, or make new skill requests and changes.

Public contributions are not accepted at this time.

License

Android Skills is licensed under the Apache License 2.0. See the LICENSE.txt file for details.

Review our community guidelines

This project follows Google's Open Source Community Guidelines.

Significant-Gravitas/AutoGPT

AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.

AutoGPT — AI agents that finish the work

Get 10 hours back every week.
Describe what you want done. AutoGPT builds the agent, runs it, and reports back.

Get started · Tour · Pricing · Docs · Discord · Self-host

The open-source platform for AI agents

AutoGPT lets you build, deploy, and run AI agents that carry out complete workflows. Describe an outcome in plain English or shape every step in the visual builder, then run the agent on demand, on a schedule, or from a trigger.

185,000+ GitHub stars. Cited by:


“Next frontier of prompt engineering imo: ‘AutoGPTs’.”	Andrej Karpathy, founding member of OpenAI
“If you have a phone you can run AutoGPT. You don't even need to learn how to code.”	Amjad Masad, co-founder & CEO of Replit
“AutoGPT might be the next big step in AI.”	Lior Alexander, CEO of AlphaSignal

Four surfaces, one platform

AutoPilot Describe the job in plain English and turn the conversation into a working agent.	Agents See every agent, run, cost, and action that needs your attention.
Marketplace Start from proven agents, add one to your library, and customize it for your work.	Build Drag, connect, branch, and inspect blocks for exact control over every step.

Get started

AutoGPT Platform — public, hosted, and managed

The hosted Platform is publicly available. We manage the infrastructure, model access, credentials, reliability, and updates so you can focus on the work your agents perform.

Get started on AutoGPT Platform →

Take the interactive tour →

AutoPilot, Agents, Marketplace, and Build
45+ connected platforms and hundreds of AI models
No model API keys or infrastructure setup
Agents that run on demand, on schedules, and from triggers

The hosted Platform is a paid service with usage-based agent runs. Compare plans and pricing →

Self-host AutoGPT

Note

Self-hosting is the free path. You provide the infrastructure and model API keys, and you maintain the deployment. If you want zero setup, use the managed Platform.

macOS and Linux:

curl -fsSL https://setup.agpt.co/install.sh -o install.sh && bash install.sh

Windows PowerShell:

powershell -c "iwr https://setup.agpt.co/install.bat -o install.bat; ./install.bat"

Read the self-hosting guide →

Managed Platform vs. self-hosting

	AutoGPT Platform	Self-hosted
Access	Public signup	Clone and install
Cost	Paid plan plus agent usage	No license fee; pay your own infrastructure and model providers
Setup	Managed	Docker and configuration required
Model access	Built in	Bring your own API keys
Updates and operations	Managed by AutoGPT	Managed by you
Core builder and agent runtime	Included	Included
Data and infrastructure control	Hosted by AutoGPT	Runs on your infrastructure
Support	Plan-dependent	Community support

Both paths use the same repository. Choose the managed Platform when you want agents running immediately; self-host when infrastructure control matters more than operational convenience.

Why the hosted Platform is paid

Every agent run consumes real model usage, compute, storage, secrets management, and operational support. The managed Platform covers that infrastructure and funds continued development of the open-source project.

Self-hosting remains available without a license fee for people and teams that want to provide and operate those resources themselves.

What you can automate

Area	Example
Executive operations	Prepare a daily brief from internal and external signals
Sales	Research every account before tomorrow's meetings
Marketing	Turn a launch brief into campaign drafts across channels
Engineering	Triage incidents and start with a likely cause
Customer support	Draft replies, collect context, and flag escalations
Research	Monitor sources and return structured reports when something changes

Integrations

Connect the apps that are yours. AutoGPT provides access to hundreds of AI models and connects agents to 45+ platforms, including:

Gmail · Google Calendar · Google Docs · Google Sheets · GitHub · Slack · Discord · Notion · HubSpot · Linear · Airtable · Jira · Salesforce · Stripe · Webflow

Explore the integrations →

Community and support

Resource	Link
Discord	Join the AutoGPT community
Documentation	docs.agpt.co
Bug reports	GitHub Issues
Feature requests	GitHub Discussions
Contributing	CONTRIBUTING.md

License

Component	License	What it means
`autogpt_platform/`	Polyform Shield	Free for personal and internal business use; cannot be sold as a competing hosted service
`classic/` and everything else	MIT	Permissive open-source use

AutoGPT Classic

Looking for the original standalone AutoGPT agent? It remains available in classic/ under the MIT License.

Contributors

Get started with AutoGPT →

SimplifyJobs/Summer2027-Internships

Summer 2027 software engineering, data science, AI, quant, product management, and hardware internship postings. Updated daily by Simplify and Pitt CSC.

Summer 2027 Tech Internships by Pitt CSC & Simplify

Use this repo to share and keep track of Summer 2027 tech internships across software engineering, data science, quant, hardware engineering, AI/ML and more. The list is updated and maintained daily by the Pitt Computer Science Club and Simplify!

🙏 Contribute by submitting an issue! See the contribution guidelines here! 🙏

Browse 349 Internship Roles by Category

💻 Software Engineering (119)

📱 Product Management (32)

🤖 Data Science, AI & Machine Learning (111)

📈 Quantitative Finance (72)

🔧 Hardware Engineering (15)

😫 Hate retyping your resume for every application?

Autofill all your applications in a single click

_{Simplify helps you find internships early and autofill your job applications on millions of sites.}

😮‍💨 Tired of checking every day for new internships?

A community member made ⬇️ SWEList ⬇️ that sends email alerts for new roles in this repo.

_{Share any other cool projects you make using the repo, and we might give you a shoutout!}

Legend

🛂 Does NOT offer sponsorship
🇺🇸 Requires U.S. Citizenship
🔒 Internship application is closed
🔥 FAANG+ company
🎓 Advanced degree required (Master's, PhD, MBA)

Looking for something else?

🔒 For closed/inactive internship listings, check out Inactive Listings
🌍 For off-season internships please see the Off-Season 2027 README
🎓 For new-grad tech jobs, check out New Grad Repo
☀️ For older summer internships, check out Archived Summer 2026 README

FAQs

What kinds of internships are listed here and where do they come from?

This repo focuses on Summer 2027 internships in tech, including software engineering, data science, AI/ML, product management, quantitative finance, hardware engineering, and related roles.

Opportunities come from community submissions and Simplify's automated internship monitoring. Community members add roles as soon as they find them, and Simplify scrapes career pages at top tech companies and startups every hour for new positions.

Where should I start if I'm looking for specific software engineering internships?

Start with the Software Engineering section and use Ctrl+F to search for specific roles, companies, locations, or keywords like backend, frontend, full-stack, mobile, or cybersecurity. New roles are added daily, so we recommend checking back often.

Many SWE internships only stay open for a few days, so applying early can give you a better shot over waiting to batch applications later. You can also browse the full list on Simplify, where you can search and filter over these roles in more depth.

For more tips on navigating the internship process, check out our guide: Zero to Offer

⬇️ Jump to bottom ⬇️

💻 Software Engineering Internship Roles

Company	Role	Location	Age
RTX	Software Engineer Intern	Huntsville, AL	1d
Northrop Grumman	Software Engineer Intern - Aeronautics Systems	Melbourne, FL	1d
GE Vernova	Engineering Intern	Niskayuna, NY	1d
↳	Engineering Intern	Niskayuna, NY	1d
Sentry	Software Engineer Intern - Summer 2027	SF	1d
Kimley-Horn	Software Developer Intern	Cary, NC	1d
🔥 ByteDance	Software Engineer Intern - AML-Engine-Orchestration	Seattle, WA	1d
DV Trading	Software Engineer Intern	NYC	2d
↳	Software Engineer Intern - DV Commodities	London, UK	2d
RTX	Software Engineer Intern	Burnsville, MN	2d
↳	Software Development Intern - Summer 2027	Tewksbury, MA	2d
🔥 ByteDance	Software Engineer Intern - AML-Engine-Orchestration	San Jose, CA	2d
Altamira Technologies	Software Development Intern	Fairborn, OH	2d
Atoms	Machine Learning PhD Software Engineer Intern 🎓	4 locations Seattle, WA LA NYC Mountain View, CA	2d
Gulfstream	Computer Science Intern - Advanced Structures and Materials	Savannah, GA	2d
🔥 TikTok	Software Engineer Intern - Global E-commerce-Search	Seattle, WA	2d
Truist Bank	Technology and Operations Intern Conversion - Data	Charlotte, NC Atlanta, GA	3d
RTX	Software Engineer Intern	Largo, FL	3d
↳	Software Engineer Intern - Summer 2027	Marlborough, MA	3d
McKesson	Technology Intern	Columbus, OH	3d
LPL Financial Holdings	Software Engineer Intern	Austin, TX Fort Mill, SC Charlotte, NC	3d
🔥 ByteDance	Software Development Engineer Intern - Network Automation - 2027 Summer	San Jose, CA	5d
Trillium	Software Engineer Intern - Summer 2027	NYC	5d
Axon	Software Engineer Intern	Boston, MA Seattle, WA	5d
The Nuclear Company	Software Engineering Intern - Product - Platform	Washington, DC	5d
🔥 TikTok	Frontend Engineer Intern - Ads Interface	San Jose, CA	5d
Atoms	Software Engineer Intern - Summer 2027	5 locations Seattle, WA SF LA NYC Mountain View, CA	5d
Belvedere Trading	Software Engineer Intern - Summer 2027	Chicago, IL	5d
🔥 TikTok	Fullstack Software Engineer Intern - Global E-Commerce	Seattle, WA	6d
↳	Fullstack Software Engineer Intern - Global E-Commerce - 2027 Summer	San Jose, CA	6d
🔥 ByteDance	Network Software Engineer Intern - Network Switch	San Jose, CA	6d
↳	Network Software Engineer Intern - Network Observation	San Jose, CA	6d
Ameren	DERMS Co-op/Intern	St. Louis, MO Illinois	6d
↳	Digital Co-op	St. Louis, MO	6d
🔥 ByteDance	AI Network Automation Engineer Intern - Global Physical Network Infrastructure	San Jose, CA	6d
🔥 TikTok	Software Engineer Intern - Foundation Platform	San Jose, CA	7d
↳	Software Engineer Intern - Media Engine	San Jose, CA	7d
🔥 ByteDance	GPU/AI Application System Software Engineer Intern - System Technologies and Engineering	San Jose, CA	7d
↳	Video/Image AI/ML Software Engineer Intern - Multimedia 🎓	San Diego, CA	7d
Northrop Grumman	Software Engineer Intern - Aeronautics Systems	Melbourne, FL	7d
🔥 Roblox	Software Engineer Intern	San Mateo, CA	7d
Vertiv	Thermal Application Engineer Intern - Summer 2027	Westerville, OH	7d
🔥 TikTok	Backend Software Engineer Intern - Creation Platform	San Jose, CA	7d
GE Appliances	Engineering Co-op	LaFayette, GA	8d
↳	Software Engineer Co-op - Software Engineering	Louisville, KY	8d
🔥 TikTok	Backend Software Engineer Intern - LIVE Foundation Governance Engineering	San Jose, CA	8d
↳	Frontend Engineer Intern - Global CRM	San Jose, CA	8d
↳	Software Engineer Intern	San Jose, CA	8d
↳	Software Engineer Intern - Intelligent Creation-Camera	San Jose, CA	8d
↳	Software Engineer Intern - Transaction Platform	San Jose, CA	8d
🔥 ByteDance	Software Engineer Intern - AI Platform	San Jose, CA	8d
🔥 TikTok	Backend Software Engineer Intern - Product Infrastructure	San Jose, CA	8d
↳	Software Engineer Intern - Effects Performance	San Jose, CA	8d
↳	Backend Software Engineer Intern - Global E-Commerce	San Jose, CA	8d
↳	Backend Software Engineer Intern - Global E-Commerce	Seattle, WA	8d
↳	Software Engineer Intern - Ads Interface	San Jose, CA	8d
↳	Software Engineer Intern	San Jose, CA	8d
↳	Software Engineer Intern - Global CRM	San Jose, CA	8d
General Dynamics Information Technology	Software Developer Intern	Annapolis Junction, MD	9d
🔥 SpaceX	Software Engineering Intern/Co-op	8 locations Palo Alto, CA Bastrop, TX Irvine, CA Cape Canaveral, FL Brownsville, TX Redmond, WA McGregor, TX Hawthorne, CA	9d
Chicago Trading Company	Software Engineer Intern	Chicago, IL	9d
🔥 ByteDance	Software Engineer Intern - AI Infra Compute 🎓	Seattle, WA	9d
↳	Software Engineer Intern - AI Infrastructure Compute 🎓	San Jose, CA	9d
↳	Software Engineer Intern - Distributed NoSQL Database Systems	San Jose, CA	9d
↳	Software Engineer Intern	Seattle, WA	9d
JP Morgan Chase	Software Engineer Intern - Software Engineer Program	9 locations Palo Alto, CA Houston, TX Tampa, FL Plano, TX Chicago, IL Jersey City, NJ Columbus, OH NYC Wilmington, DE	9d
🔥 TikTok	Software Engineer Intern	San Jose, CA	9d
↳	Software Engineer Intern - Trust and Safety Engineering	Seattle, WA	9d
↳	Backend Software Engineer Intern	San Jose, CA	9d
↳	Backend Software Engineer Intern - Innovative Growth	San Jose, CA	9d
↳	Backend Software Engineer Intern - Digital Content Center	San Jose, CA	9d
↳	Software Engineer Intern - Social Friending	San Jose, CA	9d
↳	Backend Software Engineer Intern	San Jose, CA	9d
↳	Backend Software Engineer Intern	San Jose, CA	9d
↳	Backend Software Engineer Intern	San Jose, CA	9d
↳	Frontend Engineer Intern - Open Platform	San Jose, CA	9d
↳	Backend Software Engineer Intern	San Jose, CA	9d
↳	Software Engineer Intern	San Jose, CA	9d
↳	Software Engineer Intern - Data Architecture - E-commerce	Seattle, WA	9d
↳	Software Engineer Intern - Agentic Creation-Effect Platform	San Jose, CA	9d
↳	Software Engineer Intern - Business Integrity	San Jose, CA	9d
↳	Software Engineer Intern - Creative Intelligence and Brand Innovation	San Jose, CA	9d
↳	Software Engineer Intern	San Jose, CA	9d
↳	Software Engineer Intern - Ads Measurement Signal and Privacy	San Jose, CA	9d
↳	Software Engineer Intern	Seattle, WA	9d
↳	Frontend Software Engineer Intern - Ads Measurement Signal and Privacy	San Jose, CA	9d
↳	Software Engineer Intern	San Jose, CA	9d
Uline	Software Development Intern	5 locations Waukegan, IL Milwaukee, WI Glenview, IL Pleasant Prairie, WI Kenosha, WI	10d
HPR (Hyannis Port Research)	Software Engineer Intern	Needham, MA	11d
Medtronic	Software Engineer Intern	21 locations Boston, MA Santa Ana, CA Fridley, MN Santa Rosa, CA Irvine, CA Mansfield, MA Mounds View, MN Tempe, AZ Memphis, TN Brooklyn Center, MN Plymouth, MN Brooklyn Park, MN Fort Worth, TX Dexter, MI North Haven, CT Lafayette, CO Minneapolis, MN Newton, MA Jacksonville, FL Eatontown, NJ Boulder, CO	12d
Melius	Software Engineer Intern	NYC	12d
Heliux	Software Engineer Intern	SF	12d
Genworth Financial	IT Development Program Intern - IT Development Program	Richmond, VA	13d
GoDaddy	Software Development Engineer Intern	Tempe, AZ	13d
Virtu Financial	Frontend Developer Internship	NYC	14d
PDT Partners	Software Engineer Intern	NYC	19d
Quadrillion	Software Engineering Intern	NYC	19d
Appian	Software Engineer Intern	McLean, VA	20d
Virtu Financial	Software Engineer Intern - Software Engineer	Austin, TX NYC	22d
Axon	Software Engineering Intern	Boston, MA Seattle, WA	22d
Bloxd	Software Engineer Intern	London, UK	23d
Castleton Commodities International	Front Office Software Engineer Intern	London, UK	23d
↳	Full-Stack Software Engineer Intern	Houston, TX Stamford, CT	23d
Chicago Trading Company	Software Engineering Intern	Chicago, IL	23d
Western Digital	Software Engineer Intern	6 locations Fremont, CA Rochester, MN San Jose, CA Irvine, CA Colorado Springs, CO Longmont, CO	23d
Cybernetic Labs	Software Engineer Intern - Agent Platform	SF	27d
Cybernetic Labs	Full-Stack Software Engineer Intern - Product	SF	28d
↳	Forward Deployed Engineer Intern	SF	28d
Circleback	Software Engineer Intern	SF	28d
Hudson River Trading	Software Engineer Intern - C++ or Python	Austin, TX Chicago, IL NYC	30d
Pylon	Software Engineer Intern	SF	1mo
Marshall Wace	Technology Intern	London, UK	1mo
Marshall Wace	Technology Intern	NYC	1mo
CTGT	Software Engineer Intern	SF	1mo
Optiver	Software Engineer Intern	Austin, TX	1mo
↳	Software Engineer Intern	Chicago, IL	1mo
🔥 Palantir	Forward Deployed Software Engineer Intern - Commercial	Chicago, IL	1mo
🔥 Palantir	Privacy and Civil Liberties Software Engineer Intern	NYC	1mo
SimonComputing	Software Development Internship	Alexandria, VA	1mo

🔒 See 48 more closed roles →

📱 Product Management Internship Roles

📅 Curious when Big Tech product internships typically open? Simplify put together an openings tracker based on historical data for those companies.

Company	Role	Location	Age
🔥 TikTok	Product Operations Intern - TikTok Shop Apps and API 🎓	Seattle, WA	0d
Vertiv	IT Systems Product Management Intern - Summer 2027	Westerville, OH	0d
↳	Product Management Intern - MBA - Power Solutions & Distribution 🎓	Delaware, OH	0d
↳	Product Management Intern	Delaware, OH	0d
🔥 TikTok	Product Manager Intern - Signal and Identity Product	San Jose, CA	0d
↳	Product Manager Intern - Commerce Ads	San Jose, CA	0d
↳	Product Manager Intern - Ads Interface and Platform	San Jose, CA	0d
↳	Product Manager Intern - Ads Interface and Platform	Seattle, WA	0d
🔥 TikTok	AI Safety Evaluation & Governance Product Manager Intern - Multiple Teams	San Jose, CA	1d
Vertiv	Product Management Intern	Westerville, OH	2d
LPL Financial Holdings	Product Intern - Product	Austin, TX Fort Mill, SC	3d
Chamberlain Group	Community Product Management Intern - Summer 2027	Oak Brook, IL	3d
🔥 TikTok	AI Agent Product Manager Intern - Product Infrastructure-Customer Service Platform	San Jose, CA	5d
Medline	MBA Intern - Product Management 🎓	Chicago, IL Northfield, IL	6d
🔥 TikTok	Product Operations Intern - TikTok Shop Apps and API - MBA 🎓	Seattle, WA	7d
↳	Product Manager Intern - Product Infrastructure - Account	San Jose, CA	7d
↳	Strategy Product Manager Intern - Platform Responsibility	San Jose, CA	7d
American Express	Product Management Intern - Global Merchant & Network Services	NYC	7d
↳	Product Management Intern - Global Merchant & Network Services	Phoenix, AZ	7d
🔥 TikTok	Product Operations Intern - Apps and API 🎓	Seattle, WA	8d
Winland Foods	Product Development Intern	Oak Brook, IL	8d
American Express	Product Development Intern - US Consumer Services	NYC	8d
↳	Product Innovation Intern - Credit & Fraud Risk	Phoenix, AZ	8d
🔥 TikTok	Recommendation Product Manager Intern - Content Ecosystem	San Jose, CA	8d
↳	AI Product Manager Intern - Content Ecosystem	San Jose, CA	8d
American Express	Product Management Intern	NYC	9d
↳	Digital Product Analyst Intern	NYC	9d
↳	Product Management Intern - Product Innovation - Credit & Fraud Risk	NYC	9d
Uline	Product Management Intern	5 locations Waukegan, IL Milwaukee, WI Glenview, IL Pleasant Prairie, WI Kenosha, WI	10d
Pentair	Product Management Internship	6 locations Apex, NC Brookfield, WI Delavan, WI North Aurora, IL New Brighton, MN Golden Valley, MN	10d
Appian	Product Manager Intern	McLean, VA	20d
🔥 Databricks	Product Management Intern	SF Bellevue, WA Mountain View, CA	27d

🔒 See 9 more closed roles →

🤖 Data Science, AI & Machine Learning Internship Roles

📄 Here's the resume template used by Stanford CS and Pitt CSC for internship prep.

🧠 Want to know what keywords your resume is missing for a job? Use the blue Simplify application link to instantly compare your resume to any job description.

Company	Role	Location	Age
🔥 ByteDance	Self-Built Engineer Intern - CDN Platform	San Jose, CA	0d
TMEIC Corporation Americas	Applications Intern - AI and Machine Learning	Roanoke, VA	0d
🔥 TikTok	Machine Learning Engineer Intern - E-Commerce Recommendation Foundation 🎓	San Jose, CA	0d
↳	Machine Learning Engineer Intern - E-Commerce Recommendation Foundation 🎓	Seattle, WA	0d
↳	Machine Learning Engineer Intern - E-Commerce Recommendation Mall 🎓	Seattle, WA	0d
Deloitte	AI and Data Engineering Summer Scholar Intern - Government & Public Services	Austin, TX	1d
Quantbot Technologies	Machine Learning Research Engineer Intern	NYC	1d
🔥 ByteDance	Data Lake Infrastructure & Data Analytics Research Engineer Intern - Applied Machine Learning Ark	Seattle, WA	1d
↳	Machine Learning Engineer Intern - AML-Engine-Orchestration	Seattle, WA	1d
↳	Agent Evaluation & Evolution Machine Learning Engineer Intern - Applied Machine Learning Ark	Seattle, WA	1d
↳	Visual Generation & Multimodal Evaluation Machine Learning Engineer Intern - AML-ARK	Seattle, WA	1d
Regions Bank	Technology, Operations, Digital and Data Development Program Associate - Operations - Digital	Hoover, AL	2d
Motorola	Supply Chain Data Analyst Intern	Chicago, IL	2d
Booz Allen	AI RAN Telecommunications Engineer Intern	McLean, VA	2d
🔥 TikTok	Issue Discovery Operations Intern - Content Ecosystem	San Jose, CA	2d
Citadel	Sector Data Scientist Intern	NYC	2d
Altamira Technologies	Research and Development Intern	Fairborn, OH	2d
🔥 ByteDance	Machine Learning Engineer Intern	San Jose, CA	2d
↳	Data Lake Infrastructure and Data Analytics Research Engineer Intern - Applied Machine Learning Ark	San Jose, CA	2d
↳	Visual Generation & Multimodal Evaluation Machine Learning Engineer Intern - Aml-Ark	San Jose, CA	2d
↳	Agent Evaluation & Evolution Machine Learning Engineer Intern - Applied Machine Learning Ark	San Jose, CA	2d
Montenson	Sustainability Engineer Intern	Denver, CO	2d
↳	Artificial Intelligence Intern	Minnesota	2d
↳	System Administrator Intern	Minnesota	2d
↳	Data Analytics Intern - Insights	Minnesota	2d
🔥 TikTok	Data Engineer Intern - Monetization Data	San Jose, CA	2d
PSP Investments	AI Solutions Intern - External Manager Selection & Monitoring 🎓	Montreal, QC, Canada	3d
LPL Financial Holdings	Data Engineer Intern - Data	Austin, TX Fort Mill, SC Charlotte, NC	3d
HNTB	Strategic Technology Intern - SED Division	Tampa, FL Tallahassee, FL	4d
↳	Strategic Technology Intern - SED Division	Tampa, FL Tallahassee, FL	4d
Epic Games	Machine Learning Intern - Special Projects - Epic Research Group 🎓	London, UK	5d
↳	Machine Learning Intern 🎓	Canada United Kingdom United States	5d
↳	Machine Learning Intern - Special Projects - Epic Research Group 🎓	Montreal, QC, Canada	5d
The Nuclear Company	Data Science Intern	Washington, DC	5d
↳	AI/ML Engineer Intern	Washington, DC	5d
🔥 ByteDance	Machine Learning Research Scientist Intern - Seed AI for Science - Atomistic AI 🎓	Seattle, WA	6d
Royal Bank of Canada	Capital Markets Quant Summer Associate - Quantitative Technology Services	NYC	6d
🔥 ByteDance	Applied Machine Learning Production Engineer Intern - AML Production Engineer	San Jose, CA	6d
🔥 TikTok	LLM Post-training Engineer Intern - Research & Product	San Jose, CA	7d
🔥 ByteDance	Research Scientist Intern 🎓	San Jose, CA	7d
↳	Research Scientist Intern - Multimedia Streaming 🎓	San Diego, CA	7d
🔥 TikTok	Machine Learning Engineer Intern - Search	Seattle, WA	7d
↳	Data Scientist Intern - VOD Data	San Jose, CA	7d
↳	Data Engineer Intern - Ads Targeting	San Jose, CA	7d
🔥 TikTok	Machine Learning Engineer Intern - Data-Global E-Commerce-Search 🎓	Seattle, WA	8d
↳	Data Science Intern - TikTok Product	San Jose, CA	8d
↳	Machine Learning Engineer Intern - Commerce Ads - Summer 2027	San Jose, CA	8d
↳	Machine Learning Engineer Intern - Lead Ads	San Jose, CA	8d
↳	Machine Learning Engineer Intern - Data Search Recommendation Global E-Commerce	Seattle, WA	8d
↳	Data Science Intern - TikTok Live	San Jose, CA	8d
↳	Data Science Intern - Integrity and Safety	San Jose, CA	8d
↳	Machine Learning Engineer Intern - Brand Ads	San Jose, CA	8d
↳	Machine Learning Engineer Intern - Performance Monetization 🎓	San Jose, CA	8d
↳	Machine Learning Engineer Intern - App Ads and Gaming	San Jose, CA	8d
↳	Machine Learning Engineer Intern - Search Ads	San Jose, CA	8d
↳	Machine Learning Engineer Intern - Performance Monetization	San Jose, CA	8d
↳	Machine Learning Engineer Intern	San Jose, CA	8d
↳	Machine Learning Engineer Intern - Ads Signal & Measurement	San Jose, CA	8d
↳	Machine Learning Engineer Intern - Commercial AI-CRM and Transaction 🎓	San Jose, CA	8d
GE Vernova	Digital Technology Intern	Greenville, SC Schenectady, NY Atlanta, GA	9d
↳	Digital Technology Intern	Greenville, SC Schenectady, NY Atlanta, GA	9d
StepStone Group	AI Intern 🎓	La Jolla, San Diego, CA	9d
🔥 TikTok	Research Engineer Intern - Monetization Technology - Business Integrity	San Jose, CA	9d
↳	Research Scientist Intern - Monetization Technology - Business Integrity	San Jose, CA	9d
↳	Research Scientist Intern - Monetization Technology - Business Integrity 🎓	San Jose, CA	9d
🔥 ByteDance	AI Perception Intern - Pico 🎓	San Jose, CA	9d
↳	Research Scientist Intern - Multi-modal Agentic Databases 🎓	Seattle, WA	9d
JP Morgan Chase	Data Internship - Data & AI Program	8 locations Palo Alto, CA Newark, DE Plano, TX Chicago, IL Jersey City, NJ Columbus, OH NYC Wilmington, DE	9d
↳	Data & AI Intern - Analyst	8 locations Palo Alto, CA Newark, DE Plano, TX Chicago, IL Jersey City, NJ Columbus, OH NYC Wilmington, DE	9d
🔥 TikTok	Machine Learning Engineer Intern - Recommendation 🎓	San Jose, CA	9d
↳	Machine Learning Engineer Intern - Trust and Safety 🎓	San Jose, CA	9d
↳	Research Scientist Intern - Trust and Safety 🎓	Seattle, WA	9d
↳	Research Scientist Intern - Foundations & Intelligence Service 🎓	San Jose, CA	9d
↳	Machine Learning Scientist Intern - Recommendation 🎓	San Jose, CA	9d
↳	Machine Learning Engineer Intern - Data Search Search Quality 🎓	San Jose, CA	9d
↳	Machine Learning Engineer Intern	San Jose, CA	9d
↳	Machine Learning Engineer Intern - Search Quality	San Jose, CA	9d
↳	Machine Learning Engineer Intern - Basic Ranking 🎓	San Jose, CA	9d
↳	Machine Learning Engineer Intern - Data Search - Visual Search 🎓	San Jose, CA	9d
↳	Machine Learning Engineer Intern	San Jose, CA	9d
↳	Machine Learning Engineer Intern	San Jose, CA	9d
↳	Research Engineer Intern - Agentic Systems & AI Infrastructure - Generalized Architecture	Seattle, WA	9d
↳	Artificial Intelligence Engineer Intern - Client Architecture	San Jose, CA	9d
↳	Machine Learning Scientist Intern - Content Ecology - LLM Application 🎓	San Jose, CA	9d
↳	Machine Learning Scientist Intern 🎓	San Jose, CA	9d
↳	Research Scientist Intern - Neural Graphics and World Models 🎓	San Jose, CA	9d
↳	Generative AI Researcher Intern - Intelligent Creation-Vision and Graphics 🎓	San Jose, CA	9d
↳	Research Scientist Intern - Recommendation 🎓	San Jose, CA	9d
American Express	Campus Undergraduate Summer Internship - Strategy & Analytics - Credit & Fraud Risk	NYC	9d
↳	Undergraduate Intern - Strategy & Analytics	Phoenix, AZ	9d
Uline	Business Intelligence Developer Intern	5 locations Waukegan, IL Milwaukee, WI Glenview, IL Pleasant Prairie, WI Kenosha, WI	10d
↳	Database Administrator Intern	5 locations Waukegan, IL Milwaukee, WI Glenview, IL Pleasant Prairie, WI Kenosha, WI	10d
↳	Financial Data Analyst Intern	5 locations Waukegan, IL Milwaukee, WI Glenview, IL Pleasant Prairie, WI Kenosha, WI	10d
↳	Operations Analyst Intern	5 locations Waukegan, IL Milwaukee, WI Glenview, IL Pleasant Prairie, WI Kenosha, WI	10d
The Boeing Company	Data Analytics Intern	30 locations Ridley Park, PA Seattle, WA Long Beach, CA Mesa, AZ Colorado Springs, CO Renton, WA Dallas, TX Fairfax, VA Plano, TX Chicago, IL Seal Beach, CA Tukwila, WA St Charles, MO San Antonio, TX Arlington County, Arlington, VA Everett, WA Berkeley, MO Auburn, WA Oklahoma City, OK Kent, WA Charleston, SC Bellevue, WA Herndon, VA Atlanta, GA Huntsville, AL North Charleston, SC Huntington Beach, CA Hazelwood, MO Mukilteo, WA El Segundo, CA	10d
↳	Data Analytics Intern	30 locations Ridley Park, PA Seattle, WA Long Beach, CA Mesa, AZ Colorado Springs, CO Renton, WA Dallas, TX Fairfax, VA Plano, TX Chicago, IL Seal Beach, CA Tukwila, WA St Charles, MO San Antonio, TX Arlington County, Arlington, VA Everett, WA Berkeley, MO Auburn, WA Oklahoma City, OK Kent, WA Charleston, SC Bellevue, WA Herndon, VA Atlanta, GA Huntsville, AL North Charleston, SC Huntington Beach, CA Hazelwood, MO Mukilteo, WA El Segundo, CA	10d
Two Sigma	AI Research Scientist Intern - 2027 Summer	NYC	11d
FTI Consulting	Technology Intern - Technology	7 locations Houston, TX Washington, DC SF LA Chicago, IL NYC Atlanta, GA	17d
↳	Technology Intern	7 locations Houston, TX Washington, DC SF LA Chicago, IL NYC Atlanta, GA	17d
G-Research	Data Science Intern	London, UK	21d
↳	Machine Learning Research Intern	London, UK	21d
↳	Natural Language Processing Intern	London, UK	21d
Castleton Commodities International	Data Science Machine Learning Intern	Houston, TX Stamford, CT	23d
↳	Data Engineering Intern	Houston, TX Stamford, CT NYC	23d
↳	Data Science Machine Learning Intern	London, UK	23d
↳	Data Engineering Intern	London, UK	23d
Capital One	Data Science Intern 🎓	8 locations Cambridge, MA SF San Jose, CA Plano, TX McLean, VA Richmond, VA Chicago, IL NYC	24d
↳	Master's Data Science Internship 🎓	8 locations Cambridge, MA SF San Jose, CA Plano, TX McLean, VA Richmond, VA Chicago, IL NYC	24d
AlixPartners	Data Scientist Intern 🎓	NYC	1mo
IMC Trading	Machine Learning Research Intern 🎓	London, UK	1mo
IMC Trading	Machine Learning Research Intern 🎓	Chicago, IL	1mo

🔒 See 32 more closed roles →

📈 Quantitative Finance Internship Roles

Company	Role	Location	Age
DV Trading	Trading Intern	London, UK	0d
Northwestern Mutual	Quantitative Analyst Intern - Public Investments	Milwaukee, WI	1d
Quantbot Technologies	Quantitative Researcher Intern 🎓	London, UK	1d
↳	Data Trading Analyst Intern - Data Trading Lab	London, UK	1d
↳	Quantitative Developer Intern	NYC	1d
↳	Quantitative Researcher Intern	NYC	1d
↳	Data Trading Analyst Intern	NYC	1d
Royal Bank of Canada	Quantitative Analyst Associate - Quantitative Technology Services	NYC	2d
ConocoPhillips	Trading Analytics Intern	Houston, TX	2d
Montenson	Investment Analyst Intern	Minnesota	2d
InfiniteQuant	Quantitative Researcher Intern - Summer 2027 🎓	Dubai - United Arab Emirates NYC	2d
Capula	Trading and Research Intern	London, UK	5d
Belvedere Trading	Quantitative Trading Intern - Summer 2027	Chicago, IL	5d
Royal Bank of Canada	Global Markets Quantitative Trading Summer Analyst - Capital Markets	NYC	6d
↳	Global Markets Sales & Trading Summer Analyst - Capital Markets	NYC	6d
Optiver	Quantitative Intern - Summer 2027	Austin, TX	7d
Maven Securities	Trader Intern - Summer	Chicago, IL	7d
↳	Trader Intern	Chicago, IL	7d
DV Trading	Quantitative Risk Intern	Chicago, IL	8d
Chicago Trading Company	Quant Trading Intern	London, UK	9d
↳	Quantitative Trading Intern	Chicago, IL	9d
Jane Street	Sales and Trading Intern	London, UK	9d
JP Morgan Chase	Quantitative Research Intern 🎓	NYC	9d
↳	Quantitative Research Intern - Risk and Treasury 🎓	Plano, TX Chicago, IL NYC	9d
↳	Quantitative Research Summer Analyst Intern - Asset Management	NYC	9d
↳	Quantitative Research Intern - Markets - Analyst	NYC	9d
Two Sigma	Quantitative Researcher Intern	NYC	11d
Maven Securities	Quant Trader Intern	London, UK	12d
University of Virginia Investment Management Company (UVIMCO)	Investment Analyst Intern	Charlottesville, VA	14d
Trillium	Equity Trader Intern	Miami, FL	19d
↳	Equity Trader Intern	Chicago, IL	19d
↳	Equity Trader Intern	Miami, FL Chicago, IL NYC	19d
Northwestern Mutual	Investment Analyst Intern - Private Debt & Equity	Milwaukee, WI	20d
Anthelion Capital	Quant Developer / Quant Research Intern	NYC	20d
Optiver	Institutional Trader Intern	Chicago, IL	20d
G-Research	Quantitative Research Internship 🎓	London, UK	21d
Virtu Financial	Quantitative Researcher Internship 🎓	NYC	21d
Virtu Financial	Quantitative Trading Intern	Austin, TX Chicago, IL NYC	22d
Chicago Trading Company	Quant Trading Intern	London, UK	22d
AQR Capital Management	Trading Analyst Intern	Greenwich, CT	22d
Castleton Commodities International	Commodities Trading Analyst Internship	Houston, TX Stamford, CT	23d
↳	Commodities Trading Analyst Intern	London, UK	23d
Chicago Trading Company	Quant Trading Intern	Chicago, IL	23d
Prudential Financial	Private Credit – Investment Analyst Program	8 locations SF LA Dallas, TX Newark, NJ Chicago, IL NYC Minneapolis, MN Atlanta, GA	27d
PGIM	Private Credit – Investment Analyst Program	8 locations SF LA Dallas, TX Newark, NJ Chicago, IL NYC Minneapolis, MN Atlanta, GA	27d
Susquehanna International Group (SIG)	Quantitative Trading Internship	London, UK	28d
↳	Quantitative Research Intern 🎓	London, UK	28d
↳	Quantitative Systematic Trading Intern 🎓	London, UK	28d
Voloridge	Quantitative Research Intern	Jupiter, FL	29d
Hudson River Trading	Algorithm Development Intern - Quant Research 🎓	NYC	30d
↳	Algorithm Development Intern - Quant Research	NYC	30d
DRW	Quantitative Trading Analyst Intern	London, UK	30d
Arrowstreet Capital	Quantitative Developer Intern	Boston, MA	1mo
↳	Quantitative Researcher Intern	Boston, MA	1mo
Jump Trading	Campus Quantitative Researcher Intern - PhD 🎓	Chicago, IL NYC	1mo
Flow Traders	Quantitative Trading Intern	NYC	1mo
Susquehanna International Group (SIG)	Trading System Engineer Intern	United States	1mo
↳	Quantitative Strategy Developer Intern	Philadelphia, PA	1mo
Tower Research Capital	Quantitative Developer Intern	Chicago, IL NYC	1mo
Blackstone	Blackstone Credit and Insurance – Quant and Portfolio Analytics Summer Analyst Intern	London, UK	1mo
IMC Trading	Quantitative Trader Intern	Chicago, IL	1mo
↳	Quantitative Research Intern	Chicago, IL	1mo
Optiver	Quantitative Research Intern 🎓	Austin, TX Chicago, IL NYC	1mo
↳	Quantitative Research Intern 🎓	Chicago, IL	1mo
↳	Quantitative Intern	Austin, TX Chicago, IL	1mo
Susquehanna International Group (SIG)	Quantitative Trader Intern	NYC	1mo
↳	Quantitative Trader Intern	Chicago, IL	1mo
Susquehanna International Group (SIG)	Quantitative Systematic Trading Intern 🎓	Philadelphia, PA NYC	1mo
↳	Quantitative Research Intern 🎓	Philadelphia, PA NYC	1mo
↳	Quantitative Research Intern 🎓	Philadelphia, PA NYC	1mo
↳	Quantitative Systematic Trading Intern - PhD 🎓	Philadelphia, PA NYC	1mo
Virtu Financial	Quantitative Research Intern - Quant Research - Undergrad	NYC	1mo

🔒 See 39 more closed roles →

🔧 Hardware Engineering Internship Roles

Company	Role	Location	Age
TMEIC Corporation Americas	Engineer Intern - Spring 2027	Houston, TX	0d
RTX	Software Engineering Intern - Summer 2027	Cedar Rapids, IA	1d
RTX	Software Engineering Intern	El Segundo, CA	2d
Vertiv	Thermal Engineer Intern - Summer 2027	Westerville, OH	2d
RTX	Software Engineer Intern - Summer 2027	Woburn, MA	3d
🔥 ByteDance	ASIC Design Engineer Intern - Video Silicon IP 🎓	San Jose, CA	7d
RTX	Software Engineer Intern - Summer 2027	Cedar Rapids, IA	9d
🔥 SpaceX	Silicon Engineer Intern/Co-op	Palo Alto, CA Irvine, CA Redmond, WA	9d
HPR (Hyannis Port Research)	FPGA Engineer Intern	Needham, MA	11d
Virtu Financial	Hardware Engineer Intern - FPGA 🎓	Austin, TX	12d
Western Digital	Hardware Engineering Intern	6 locations Fremont, CA Rochester, MN San Jose, CA Irvine, CA Colorado Springs, CO Longmont, CO	12d
Optiver	FPGA Engineer Intern	Chicago, IL	22d
↳	FPGA Engineer Intern	Austin, TX	22d
Ciena	Software Engineer Intern	Edinburgh, UK	23d
IMC Trading	Hardware Engineer Intern	Chicago, IL	1mo

🔒 See 16 more closed roles →

⬆️ Back to Top ⬆️

We love our contributors ❤️❤️

Contribute by submitting an issue!

Made with contrib.rocks.

3b1b/manim

Animation engine for explanatory math videos

Manim is an engine for precise programmatic animations, designed for creating explanatory math videos.

Note, there are two versions of manim. This repository began as a personal project by the author of 3Blue1Brown for the purpose of animating those videos, with video-specific code available here. In 2020 a group of developers forked it into what is now the community edition, with a goal of being more stable, better tested, quicker to respond to community contributions, and all around friendlier to get started with. See this page for more details.

Installation

Warning

WARNING: These instructions are for ManimGL only. Trying to use these instructions to install Manim Community/manim or instructions there to install this version will cause problems. You should first decide which version you wish to install, then only follow the instructions for your desired version.

Note

Note: To install manim directly through pip, please pay attention to the name of the installed package. This repository is ManimGL of 3b1b. The package name is manimgl instead of manim or manimlib. Please use pip install manimgl to install the version in this repository.

Manim runs on Python 3.10 or higher.

System requirements are FFmpeg, OpenGL and LaTeX (optional, if you want to use LaTeX). For Linux, Pango along with its development headers are required. See instruction here.

Directly

# Install manimgl
pip install manimgl

# Try it out
manimgl

For more options, take a look at the Using manim sections further below.

If you want to hack on manimlib itself, clone this repository and in that directory execute:

# Install manimgl
pip install -e .

# Try it out
manimgl example_scenes.py OpeningManimExample
# or
manim-render example_scenes.py OpeningManimExample

Linux (Ubuntu/Debian)

Install system dependencies.

sudo apt update

sudo apt install ffmpeg
sudo apt install python3-pip
sudo apt install libpango1.0-dev

Install a lightweight LaTeX distribution (optional, for LaTeX rendering).

sudo apt install texlive-science texlive-fonts-extra texlive-latex-extra

This lightweight setup is significantly smaller than installing texlive-full while still supporting most Manim projects.

Clone and install ManimGL.

git clone https://github.com/3b1b/manim.git
cd manim

python3 -m pip install -e .

manimgl example_scenes.py OpeningManimExample

💡 Optional: Using a virtual environment (venv)

It is recommended to use a virtual environment to avoid conflicts with system packages.

sudo apt install python3-venv

python3 -m venv venv
source venv/bin/activate

python3 -m pip install -e .

If python3-venv is unavailable on your system, try installing the version-specific package instead:

sudo apt install python3.12-venv

Directly (Windows)

Install FFmpeg.
Install a LaTeX distribution. MiKTeX is recommended.

Install the remaining Python packages.

git clone https://github.com/3b1b/manim.git
cd manim
pip install -e .
manimgl example_scenes.py OpeningManimExample

Mac OSX

Install FFmpeg, LaTeX in terminal using homebrew.
```
brew install ffmpeg mactex
```
💡 An alternative to heavyweight MacTeX bundle.

To avoid installing the full MacTeX bundle, which is ~6GB, you can alternatively install the lightweight BasicTeX and then gradually add only the LaTeX packages you actually need. A list of packages sufficient to run examples can be found here. For an overview of the MacTeX installer bundles, see https://www.tug.org/mactex/.
If you are using an ARM-based processor, install Cairo.
```
arch -arm64 brew install pkg-config cairo
```

Install latest version of manim using these command.

git clone https://github.com/3b1b/manim.git
cd manim
pip install -e .
manimgl example_scenes.py OpeningManimExample (make sure to add manimgl to path first.)

Anaconda Install

Install LaTeX as above.
Create a conda environment using conda create -n manim python=3.10.
Activate the environment using conda activate manim.
Install manimgl using pip install -e ..

Using manim

Try running the following:

manimgl example_scenes.py OpeningManimExample

This should pop up a window playing a simple scene.

Look through the example scenes to see examples of the library's syntax, animation types and object types. In the 3b1b/videos repo, you can see all the code for 3blue1brown videos, though code from older videos may not be compatible with the most recent version of manim. The readme of that repo also outlines some details for how to set up a more interactive workflow, as shown in this manim demo video for example.

When running in the CLI, some useful flags include:

-w to write the scene to a file
-o to write the scene to a file and open the result
-s to skip to the end and just show the final frame.
- -so will save the final frame to an image and show it
-n <number> to skip ahead to the n'th animation of a scene.
-f to make the playback window fullscreen

Take a look at custom_config.yml for further configuration. To add your customization, you can either edit this file, or add another file by the same name "custom_config.yml" to whatever directory you are running manim from. For example this is the one for 3blue1brown videos. There you can specify where videos should be output to, where manim should look for image files and sounds you want to read in, and other defaults regarding style and video quality.

Documentation

Documentation is in progress at 3b1b.github.io/manim. And there is also a Chinese version maintained by @manim-kindergarten: docs.manim.org.cn (in Chinese).

manim-kindergarten wrote and collected some useful extra classes and some codes of videos in manim_sandbox repo.

Contributing

Is always welcome. As mentioned above, the community edition has the most active ecosystem for contributions, with testing and continuous integration, but pull requests are welcome here too. Please explain the motivation for a given change and examples of its effect.

License

This project falls under the MIT license.

harveyai/harvey-labs

A benchmark built to evaluate and improve agent capabilities for supporting legal work.

Legal Agent Benchmark (LAB): An open-source benchmark for evaluating agents on real legal work.

Harvey LAB is an open-source project aimed at benchmarking LLM agents' abilities to perform legal work in realistic environments.

LAB consists of two parts: a dataset of tasks containing agent instructions, documents, and rubrics as well as an execution harness for running and evaluating agents against those tasks.

LAB is an ongoing project and we expect to consistently add to and refine the task set and execution harness.

Read the announcement post: Introducing Harvey's Legal Agent Benchmark

Getting Started

Start with the full walkthrough in docs/tutorial.md — it takes one realistic M&A data-room assignment end to end: setup, task inspection, agent run, scoring, report review, and comparison dashboards.

Additional Documentation

Guide	Description
Architecture	Task model, harness, tools, adapters, reports, and sweeps
Evaluation Methodology	All-pass rubric scoring and LLM judge behavior
Contributing	Add tasks, model adapters, evaluation improvements, and docs

Citation

If you use Harvey LAB in your research, please cite it as:

@misc{harveylab2026,
  title   = {Harvey LAB: The Legal Agent Benchmark},
  author  = {{Harvey AI}},
  year    = {2026},
  version = {v1.0},
  url     = {https://github.com/harveyai/harvey-labs/tree/v1.0},
  note    = {Announcement: \url{https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark}}
}

beenuar/AiSOC

Open-source AI-powered Security Operations Center — alert fusion, purple-team drills, agent-assisted triage, MITRE ATT&CK investigation. MIT-licensed, self-hostable.

AiSOC

An open-source, self-hostable AI SOC. The agent's prompts, tool calls, and rationale are logged step-by-step and replayable. MIT-licensed.

_{The community-maintained demo at tryaisoc.com runs on Fly.io and can go offline; see docs/operations/live-demo-runbook.md and use Codespaces as the always-on fallback.}

_{90-second walkthrough — agent investigates the seeded LockBit 3.0 case end-to-end. The rendered .mp4 + hero.gif land with the v8.0 launch; the brief is in docs/demo/SCREENCAST_SHOTLIST.md.}

Try AiSOC in 60 seconds

One command — no clone, no Docker, no keys (npx aisoc lands on npm with the v8.0 launch; today it builds from packages/aisoc-lite/):

npx aisoc triage --demo
# ✓ AiSOC triaged 200 alerts: 12 TP, 171 FP suppressed (85.5% noise), 17 need review — in 0.1s

The wedge CLI scores a batch of alerts to verdicts (escalate / review / suppress) with a deterministic engine ported from the production triage scorer — zero LLM key required. Or pick whichever path matches what you already have on your machine:

If you have…	Run this	What you get
Python 3.10+ (no Docker)	`pip install -e packages/aisoc-sandbox && aisoc-sandbox demo`	Offline agent investigation walked through Detect → Triage → Hunt → Respond and printed to stdout. < 5 s. No API key, no network.
A browser (zero install)	Open in Codespaces	Browser IDE → `pnpm aisoc:demo --no-open` → click forwarded port `3000`. ~5 min cold.
Docker + pnpm	`git clone https://github.com/beenuar/AiSOC && cd AiSOC && pnpm aisoc:demo`	Local stack on Postgres + Redis + Kafka + api + agents + web. Browser opens at `INC-RT-001`.
Nothing (clean Linux/macOS/Win)	`curl -fsSL https://raw.githubusercontent.com/beenuar/AiSOC/main/install.sh \| bash`	Bootstraps Docker, Node, pnpm, git for you; then runs `pnpm aisoc:demo`.

The first row is new: aisoc-sandbox is a zero-dependency, in-memory simulator of the agent funnel. Pick a bundled scenario (lateral-movement, aws-credential-exfil, phishing-payload, kubernetes-privesc, github-token-theft) or feed in your own JSON via --file. The other three rows boot the real stack and land you on /cases/INC-RT-001?tab=ledger — a LockBit 3.0 ransomware case mid-investigation, with the AI agent's prompts, tool calls, and rationale streaming into the Investigation Ledger. Stop the real stack with pnpm aisoc:demo:down.

Does the demo still boot on main? Every push runs compose-smoke (the same pnpm aisoc:demo path you'd run locally) and e2e against the seeded console; nightly compose-smoke-nightly repeats it with cold caches. A red badge below is a release-blocker.

Full multi-platform deploy guide is in apps/docs/docs/installation.md (Render, Fly.io, Docker Compose, Kubernetes, Terraform). Production-grade install with full storage tier: infra/helm/ or infra/terraform/.

What AiSOC is

AiSOC is a single self-hostable stack that ingests security events, correlates them, runs AI-driven investigation, and surfaces the result in a SOC console. The agent and the substrate are MIT-licensed, so you can read, fork, or replace either of them.

Three properties distinguish it from closed-source AI SOC vendors:

Agent decisions are logged. The Investigation Ledger stores the LLM prompt, the response, the evidence cited, and the downstream tool calls for every step of every run. Replays are available later.
The substrate has a public eval harness in CI. Five suites gate every PR targeting main / develop — alert reduction is a real measurement against a fixed 1 000-alert stream; three rubric-based suites are substrate self-consistency gates over a deterministic 200-incident dataset (55 templates) with per-template macros; a fifth gate validates the backing telemetry corpus. The benchmark page documents exactly what each suite measures and what it does not.
You control what leaves your perimeter. No callbacks to a vendor cloud and no "model improvement" telemetry. With a hosted LLM, evidence is pseudonymized by default (internal IPs, hostnames, emails, paths, secrets, usernames become opaque tokens); run a local model (Ollama/vLLM) for a fully air-gapped path. Exactly what leaves under each mode: docs/trust/data-flows.md.

The orchestrator is a ~600-line LangGraph in services/agents/. It is small enough to read end-to-end, swap models in, and patch.

How AiSOC compares

Capability	AiSOC	Wazuh	Splunk ES	Closed-source AI SOC
Open-source license	MIT	GPL-2	proprietary	proprietary
Self-hostable	yes	yes	enterprise-only	cloud-only
Autonomous AI investigation	LangGraph	no	partial (Splunk AI)	yes
Agent decision audit trail	public Investigation Ledger	n/a	n/a	not published
Public substrate eval harness	CI-gated, reproducible, with synthetic telemetry corpus + per-template macros	n/a	n/a	not published
Detection content	947 executable (869 native) firing on the live stream + 6 000-rule provenance-tracked imported library (truth table)	1 200+ rules	1 000+ apps	curated
Plugin SDK	Python / TypeScript / Go	YAML rules only	apps	proprietary
Data residency	your infra	your infra	partial	vendor cloud
Pricing	$0 (self-host)	$0 (self-host)	per ingest GB	enterprise

Closed-source AI SOC vendors ship working products. AiSOC's contribution is making the agent itself open, the per-step decision trail readable, and the substrate gated by a public eval harness on every PR targeting main / develop.

What you'll see in the console


Alerts queue — server-anchored SLA countdowns, atomic claim, one-click triage. Docs	Investigation Rail — narrative, pivot-path entity chips, 6-event timeline, recommended actions. Docs

`/hunt` workbench — type a hypothesis in English, get ES\|QL / SPL / KQL back, save + schedule. Docs	Marketplace — plugins, playbooks, detections with one-click tenant install. Docs

_{The four tiles above are SVG placeholders. Real PNG screenshots ship with the next Phase 2 visuals rollup; the walkthrough video at the top of this README is the canonical reference until then.}

Architecture

flowchart LR
    subgraph Sources["Sources"]
        EDR["EDR / XDR"]
        SIEM["SIEM"]
        Cloud["Cloud APIs"]
        IDP["Identity"]
        Net["Network"]
    end

    subgraph Ingest["Ingest & Normalize"]
        Connectors["Connectors\n(Python · 78 vendors)"]
        OsqueryTLS["osquery-tls\n(Python · host telemetry)"]
        IngestSvc["Ingest worker\n(Go · OCSF)"]
        Enrich["Enrichment\n(Go · IOC + Shodan)"]
    end

    subgraph Spine["Event Spine"]
        Kafka[("Apache Kafka")]
    end

    subgraph Detect["Detect & Reason"]
        Fusion["Fusion\n(Python · ML)"]
        UEBA["UEBA\n(Python · baseline)"]
        Rules["Rule engine\n(Sigma · YARA · KQL)"]
        Agents["AI Agents\n(LangGraph)"]
    end

    subgraph Storage["Storage Tier"]
        PG[("PostgreSQL")]
        CH[("ClickHouse")]
        OS[("OpenSearch")]
        QD[("Qdrant")]
        N4[("Neo4j")]
        RD[("Redis")]
    end

    subgraph Surface["Surface"]
        API["Core API\n(FastAPI)"]
        Web["Web Console + Responder PWA\n(Next.js)"]
        MCP["MCP Server\n(TS · stdio)"]
    end

    Sources --> Connectors --> IngestSvc --> Kafka
    OsqueryTLS --> IngestSvc
    IngestSvc --> Enrich --> Kafka
    Kafka --> Fusion --> Storage
    Kafka --> UEBA --> Kafka
    Kafka --> Rules --> Kafka
    Agents --> Storage
    API --> Storage
    Web --> API
    MCP --> API

Full architecture (every service, every storage role, the v1.5 console workbench, and the Investigation Ledger contract) is in apps/docs/docs/architecture.md. The deeper system-design write-up — including ML fusion, the Neo4j-at-ingest schema, and the threat-intel pipeline — lives at docs/architecture/SYSTEM_DESIGN.md. The full monorepo layout is at apps/docs/docs/architecture/overview.md.

What's in the box

A handful of headline capabilities — the rest are catalogued in apps/docs/docs/features/ and indexed at the top of apps/docs/docs/intro.md:

Maturity (v7.7.0 — Fully-Operational release). The end-to-end spine is wired and CI-gated: ingest → ClickHouse lake → live detection → fused alert → auto-triage → governed response. Connectors, Investigation Rail + Ledger, Hunt-as-Code, live-stream detection, and copilot auto-triage are GA. Autonomous response defaults to copilot/dry-run (an autonomy policy governs every real execution). The live-agent LLM benchmark is preview (the deterministic-tier scoreboard is CI-gated per PR); substrate eval suites are GA. Every product claim is backed by a failing test — claim-to-gate matrix: 46 GATED / 9 PARTIAL / 0 NO GATE. Full per-claim status: docs/audit/REALITY_REPORT.md. v7.7.0 adds three detection-authoring modes (Python framework + AI builder + no-code), least-privilege invoking-identity scoping for response actions, self-service data lifecycle (retention + a ReDoS-proof transform DSL + custom parsers), an agentless CSPM scanner with compliance auto-evidence and Opsgenie/email/SOAR destinations, and a customizable report builder — all tested, all landed on main.

83 click-and-connect data connectors (EDR/XDR, SIEM, NDR, cloud, CNAPP, identity, SaaS, VCS, K8s audit, network) with schema-driven config, live Test connection, and vault-encrypted secrets — recently adding Qualys, GreyNoise, JumpCloud, Darktrace, and Imperva alongside IBM QRadar, Netskope, Zeek/Suricata NDR, and more. One query runs SIEM-agnostic federated search across Splunk SPL / Sentinel KQL / Elastic ES|QL / QRadar AQL. Walkthrough: apps/docs/docs/connectors/index.md.
End-to-end SIEM spine — a cold docker compose up ingests connector data → lands it in the ClickHouse event lake → the executable detection corpus (947 rules) fires on the live stream → a fused alert is created, all asserted by an extended integration gate. Fuse-time threat-intel + CISA-KEV enrichment now feeds the confidence score and exploit-in-wild boost, and stateful/windowed detections (brute-force, password-spray, port-scan) run alongside the corpus. apps/docs/docs/architecture.md.
Autonomous triage + governed response — every fused alert is auto-triaged by the agent (copilot/read-only by default) with a prompt-injection guard that demotes tampered evidence to manual review; a unified confidence × blast-radius × reversibility policy authorizes auto-execution only for reversible, low-blast actions at high confidence (everything else stays gated to a human), with real rollback + post-action verification. apps/docs/docs/concepts/automation-maturity.md.
Advanced Data Explorer — one investigation surface (NL + SQL over the lake, plus pivots to identity/graph/intel), replacing the SIEM context-switch. /explore.
Investigation Rail + replayable Investigation Ledger — every prompt, tool call, evidence chip, and rationale stored against a case, replayable in the UI and shareable as a redacted public permalink (live demo replay). apps/docs/docs/console/investigation-rail.md.
Detection-as-Code lifecycle — propose → review → eval-gate → promote; CI rejects any candidate that fails its own positive/negative fixtures (the non-circular gate) or regresses MITRE accuracy. Analyst false-positive feedback now feeds a self-improving tuner that proposes scoped rule exceptions / severity changes (human-approved, never auto-applied). apps/docs/docs/concepts/detections.md — and the 869 native rules live in detections/.
Three-model AI + tool-using agents — Semantic (graph-at-ingest), Behavioral (UEBA fused into alert scoring), and Knowledge (LLM), with fuse-time attack-chain grouping. The agent calls real tools (IOC enrichment, MITRE lookup, graph blast-radius) through an LLM tool-calling loop, and a scored planner routes each alert to the right specialist instead of fanning out to all four.
Cost-governed LLM routing — per-tenant budgets + circuit breaker, token/cost telemetry, a content-addressed response cache, a cheap-first cost cascade (escalate to the strong model only on low confidence), multi-model gateway fallbacks, and per-tenant BYOK keys. services/agents/app/routing/.
Hunt-as-Code — YAML hypotheses with MITRE tags, cron schedules, and natural-language /hunt workbench. hunts/ + apps/docs/docs/console/rule-tuning.md. Plus free, login-free browser tools: a Sigma/SPL/KQL/ES|QL rule translator, an ATT&CK coverage grader, NL→Sigma, and a noise calculator.
Public weekly benchmark scoreboard — the same harness that gates PRs; the deterministic-tier row is CI-gated for freshness on every PR, and the funded weekly job appends live-LLM rows. A new groundedness/hallucination axis flags any indicator the agent asserts that isn't in the evidence it was given. apps/docs/docs/benchmark-scoreboard.mdx.

Use it from Claude, Cursor, or Cody

AiSOC ships an MCP server (services/mcp/) so analysts can query alerts, run agent investigations, and replay every step the agent took without leaving the IDE or chat. The server exposes 13 tools — discovery, deep-dive, governed lake query, and the action / replay set that walks the agent decision ledger step-by-step.

Status — monorepo source build today; npm publish lands in v8.0. Full setup is in apps/docs/docs/integrations/mcp.md, which shows the today-vs-v8.0 invocations side by side.

Extend it

Three contribution surfaces; each is one file plus optional fixtures, and CI validates every PR.

Detection rule. Drop a Sigma YAML under detections/ with a positive / negative fixture in detections/fixtures/. The validate-detections workflow tests it on every PR. Spec: docs/connectors/.
Connector. Subclass BaseConnector in services/connectors/app/connectors/, register it in _CONNECTOR_CLASSES, and add a plugins/<id>/plugin.yaml manifest. The marketplace picks it up automatically. Walkthrough: apps/docs/docs/connectors/.
Playbook. Drop a YAML under playbooks/; validate-playbooks gates the PR. Schema: playbook.schema.json.

Plugin and detection SDK (Python · TypeScript · Go) — see apps/docs/docs/plugins/overview.md. The CLI (aisoc-cli) is in packages/aisoc-cli/; PyPI publish lands in v8.0.

In your CI: add - uses: beenuar/aisoc-action@v1 to triage your repo's Dependabot / CodeQL / secret-scanning alerts on every PR (deterministic, nothing leaves your runner; dogfooded on this repo, Marketplace publish lands with v8.0). Docs.

Roadmap & releases

Latest GitHub release with downloads: https://github.com/beenuar/AiSOC/releases/latest
Per-release narrative: RELEASES.md (mirrors what used to live in this README)
Machine-readable inventory with file paths, env-var diffs, test counts: CHANGELOG.md
v8.0 wave-2 in flight ([~] items): docs/roadmap/v8-progress.md
Bigger-picture roadmap (BYOC multi-cloud, MSSP rollups, federated search): ROADMAP.md

Contributing

PRs of every size are welcome. Read CONTRIBUTING.md for the workflow and the Code of Conduct before opening a PR.

First-time contributors: pick a good first issue. Need help? Open a Q&A discussion.

Credits

AiSOC is built and improved by a growing community of contributors, security researchers, and operators. The full attribution — including bug reporters and security researchers — lives in .github/CREDITS.md. The always-up-to-date code-contribution graph is on the GitHub contributors page.

Security

For security issues, please do not open a public issue. Use GitHub's private vulnerability reporting. Full policy in SECURITY.md. AiSOC follows coordinated disclosure.

License

Report a bug · Request a feature · Contribute · Read the docs · Reproduce the benchmark

666ghj/MiroFish

A Simple and Universal Swarm Intelligence Engine, Predicting Anything. 简洁通用的群体智能引擎，预测万物

简洁通用的群体智能引擎，预测万物
A Simple and Universal Swarm Intelligence Engine, Predicting Anything

English | 中文文档

⚡ Overview

MiroFish is a next-generation AI prediction engine powered by multi-agent technology. By extracting seed information from the real world (such as breaking news, policy drafts, or financial signals), it automatically constructs a high-fidelity parallel digital world. Within this space, thousands of intelligent agents with independent personalities, long-term memory, and behavioral logic freely interact and undergo social evolution. You can inject variables dynamically from a "God's-eye view" to precisely deduce future trajectories — rehearse the future in a digital sandbox, and win decisions after countless simulations.

You only need to: Upload seed materials (data analysis reports or interesting novel stories) and describe your prediction requirements in natural language
MiroFish will return: A detailed prediction report and a deeply interactive high-fidelity digital world

Our Vision

MiroFish is dedicated to creating a swarm intelligence mirror that maps reality. By capturing the collective emergence triggered by individual interactions, we break through the limitations of traditional prediction:

At the Macro Level: We are a rehearsal laboratory for decision-makers, allowing policies and public relations to be tested at zero risk
At the Micro Level: We are a creative sandbox for individual users — whether deducing novel endings or exploring imaginative scenarios, everything can be fun, playful, and accessible

From serious predictions to playful simulations, we let every "what if" see its outcome, making it possible to predict anything.

🌐 Live Demo

Welcome to visit our online demo environment and experience a prediction simulation on trending public opinion events we've prepared for you: mirofish-live-demo

📸 Screenshots

🎬 Demo Videos

1. Wuhan University Public Opinion Simulation + MiroFish Project Introduction

Click the image to watch the complete demo video for prediction using BettaFish-generated "Wuhan University Public Opinion Report"

2. Dream of the Red Chamber Lost Ending Simulation

Click the image to watch MiroFish's deep prediction of the lost ending based on hundreds of thousands of words from the first 80 chapters of "Dream of the Red Chamber"

Financial Prediction, Political News Prediction and more examples coming soon...

🔄 Workflow

Graph Building: Seed extraction & Individual/collective memory injection & GraphRAG construction
Environment Setup: Entity relationship extraction & Persona generation & Agent configuration injection
Simulation: Dual-platform parallel simulation & Auto-parse prediction requirements & Dynamic temporal memory updates
Report Generation: ReportAgent with rich toolset for deep interaction with post-simulation environment
Deep Interaction: Chat with any agent in the simulated world & Interact with ReportAgent

🚀 Quick Start

Option 1: Source Code Deployment (Recommended)

Prerequisites

Tool	Version	Description	Check Installation
Node.js	18+	Frontend runtime, includes npm	`node -v`
Python	≥3.11, ≤3.12	Backend runtime	`python --version`
uv	Latest	Python package manager	`uv --version`

1. Configure Environment Variables

# Copy the example configuration file
cp .env.example .env

# Edit the .env file and fill in the required API keys

Required Environment Variables:

# LLM API Configuration (supports any LLM API with OpenAI SDK format)
# Recommended: Alibaba Qwen-plus model via Bailian Platform: https://bailian.console.aliyun.com/
# High consumption, try simulations with fewer than 40 rounds first
LLM_API_KEY=your_api_key
LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
LLM_MODEL_NAME=qwen-plus

# Zep Cloud Configuration
# Free monthly quota is sufficient for simple usage: https://app.getzep.com/
ZEP_API_KEY=your_zep_api_key

2. Install Dependencies

# One-click installation of all dependencies (root + frontend + backend)
npm run setup:all

Or install step by step:

# Install Node dependencies (root + frontend)
npm run setup

# Install Python dependencies (backend, auto-creates virtual environment)
npm run setup:backend

3. Start Services

# Start both frontend and backend (run from project root)
npm run dev

Service URLs:

Frontend: http://localhost:3000
Backend API: http://localhost:5001

Start Individually:

npm run backend   # Start backend only
npm run frontend  # Start frontend only

Option 2: Docker Deployment

# 1. Configure environment variables (same as source deployment)
cp .env.example .env

# 2. Pull image and start
docker compose up -d

Reads .env from root directory by default, maps ports 3000 (frontend) / 5001 (backend)

Mirror address for faster pulling is provided as comments in docker-compose.yml, replace if needed.

📬 Join the Conversation

The MiroFish team is recruiting full-time/internship positions. If you're interested in multi-agent simulation and LLM applications, feel free to send your resume to: mirofish@shanda.com

📄 Acknowledgments

MiroFish has received strategic support and incubation from Shanda Group!

MiroFish's simulation engine is powered by OASIS (Open Agent Social Interaction Simulations), We sincerely thank the CAMEL-AI team for their open-source contributions!

📈 Project Statistics

stanfordnlp/dspy

DSPy: The framework for programming—not prompting—language models

DSPy: Programming—not prompting—Foundation Models

Documentation: DSPy Docs

DSPy is the framework for programming—rather than prompting—language models. It allows you to iterate fast on building modular AI systems and offers algorithms for optimizing their prompts and weights, whether you're building simple classifiers, sophisticated RAG pipelines, or Agent loops.

DSPy stands for Declarative Self-improving Python. Instead of brittle prompts, you write compositional Python code and use DSPy to teach your LM to deliver high-quality outputs. Learn more via our official documentation site or meet the community, seek help, or start contributing via this GitHub repo and our Discord server.

Documentation: dspy.ai

Please go to the DSPy Docs at dspy.ai

Installation

pip install dspy

To install the very latest from main:

pip install git+https://github.com/stanfordnlp/dspy.git

📜 Citation & Reading More

If you're looking to understand the framework, please go to the DSPy Docs at dspy.ai.

If you're looking to understand the underlying research, this is a set of our papers:

[Jul'25] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning
[Jun'24] Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs
[Oct'23] DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
[Jul'24] Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Together
[Jun'24] Prompts as Auto-Optimized Training Hyperparameters
[Feb'24] Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models
[Jan'24] In-Context Learning for Extreme Multi-Label Classification
[Dec'23] DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines
[Dec'22] Demonstrate-Search-Predict: Composing Retrieval & Language Models for Knowledge-Intensive NLP

To stay up to date or learn more, follow @DSPyOSS on Twitter or the DSPy page on LinkedIn.

The DSPy logo is designed by Chuyi Zhang.

If you use DSPy or DSP in a research paper, please cite our work as follows:

@inproceedings{khattab2024dspy,
  title={DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines},
  author={Khattab, Omar and Singhvi, Arnav and Maheshwari, Paridhi and Zhang, Zhiyuan and Santhanam, Keshav and Vardhamanan, Sri and Haq, Saiful and Sharma, Ashutosh and Joshi, Thomas T. and Moazam, Hanna and Miller, Heather and Zaharia, Matei and Potts, Christopher},
  journal={The Twelfth International Conference on Learning Representations},
  year={2024}
}
@article{khattab2022demonstrate,
  title={Demonstrate-Search-Predict: Composing Retrieval and Language Models for Knowledge-Intensive {NLP}},
  author={Khattab, Omar and Santhanam, Keshav and Li, Xiang Lisa and Hall, David and Liang, Percy and Potts, Christopher and Zaharia, Matei},
  journal={arXiv preprint arXiv:2212.14024},
  year={2022}
}

ZhuLinsen/daily_stock_analysis

LLM 驱动的多市场股票智能分析系统：多源行情、实时新闻、决策看板与自动推送，支持零成本定时运行。 LLM-powered multi-market stock analysis system with multi-source market data, real-time news, decision dashboard, automated notifications, and cost-free scheduled runs.

📈 股票智能分析系统

🤖 基于 AI 大模型的 A股/港股/美股/日股/韩股/台股自选股智能分析系统，每日自动分析并推送「决策仪表盘」到企业微信/飞书/Telegram/Discord/Slack/邮箱

产品预览 · 功能特性 · 快速开始 · 推送效果 · 文档中心 · 完整指南

简体中文 | English | 繁體中文

💖 赞助商 (Sponsors)

🖥️ 产品预览

✨ 功能特性

能力	覆盖内容
AI 决策报告	核心结论、评分、趋势、买卖点位、风险警报、催化因素、操作检查清单
多市场数据聚合	覆盖 A股、港股、美股、日股、韩股、台股和 ETF，支持行情、K 线、技术指标、新闻、公告、基本面与报告辅助数据；不同市场的数据源和能力边界见市场支持边界
Web / 桌面工作台	手动分析、任务进度、历史报告、完整 Markdown、回测、持仓、配置管理、浅色 / 深色主题
Agent 策略问股	多轮追问，支持均线、缠论、波浪、趋势、热点、事件、成长、预期等 15 种内置策略，覆盖 Web/Bot/API
智能导入与补全	图片、CSV/Excel、剪贴板导入；股票代码/名称/拼音/别名补全
自动化与推送	GitHub Actions、Docker、本地定时任务、FastAPI 服务和企业微信/飞书/Telegram/Discord/Slack/邮件推送

功能细节、字段契约、基本面 P0 超时语义、交易纪律、数据源优先级、Web/API 行为请看完整配置与部署指南。

技术栈与数据来源

类型	支持
AI 模型	Anspire、AIHubMix、Gemini、OpenAI 兼容、DeepSeek、通义千问、Claude、Ollama 本地模型等
行情数据	TickFlow、AkShare、Tushare、Pytdx、Baostock、YFinance、Longbridge
新闻搜索	Anspire、SerpAPI、Tavily、Bocha、Brave、MiniMax、SearXNG
社交舆情	Stock Sentiment API（Reddit / X / Polymarket，仅美股，可选）

项目默认内置 AkShare、Baostock、YFinance 等免费行情源，可零配置运行；免费源受上游限流、接口变动和网络波动影响，稳定性不保证。长期定时、批量分析或更稳定行情建议配置 TickFlow、Tushare、Longbridge 等 token 型数据源，适用市场、Actions 映射和 fallback 规则见数据源配置。

🚀 快速开始

方式一：GitHub Actions（推荐）

5 分钟完成部署，零成本，无需服务器。

1. Fork 本仓库

点击右上角 Fork 按钮（顺便点个 Star⭐ 支持一下）

2. 配置 Secrets

Settings → Secrets and variables → Actions → New repository secret

AI 模型配置（至少配置一个）

默认先选一个模型服务商并填写 API Key；需要多模型、图片识别、本地模型或高级路由时，再参考 LLM 配置指南。

Secret 名称	说明	必填
`ANSPIRE_API_KEYS`	Anspire API Key，一Key同时启用全球热门大模型和联网搜索，本项目新用户提供30元等额的免费额度（GLM5.2、GPT等模型特惠中）	推荐
`AIHUBMIX_KEY`	AIHubMix API Key，一Key切换使用全系模型，无需科学上网，本项目可享 10% 优惠	推荐
`GEMINI_API_KEY`	Google Gemini API Key	可选
`ANTHROPIC_API_KEY`	Anthropic Claude API Key	可选
`OPENAI_API_KEY`	OpenAI 兼容 API Key（支持 DeepSeek、通义千问等）	可选
`OPENAI_BASE_URL` / `OPENAI_MODEL`	使用 OpenAI 兼容服务时填写	可选

Ollama 更适合本地 / Docker 部署，GitHub Actions 推荐使用云端 API。

通知渠道配置（至少配置一个）

Secret 名称	说明
`WECHAT_WEBHOOK_URL`	企业微信机器人
`FEISHU_WEBHOOK_URL`	飞书机器人
`TELEGRAM_BOT_TOKEN` + `TELEGRAM_CHAT_ID`	Telegram
`DISCORD_WEBHOOK_URL`	Discord Webhook
`SLACK_BOT_TOKEN` + `SLACK_CHANNEL_ID`	Slack Bot
`EMAIL_SENDER` + `EMAIL_PASSWORD`	邮件推送

更多渠道、签名校验、分组邮件、Markdown 转图片等配置见通知渠道详细配置。

自选股配置（必填）

Secret 名称	说明	必填
`STOCK_LIST`	自选股代码，如 `600519,hk00700,AAPL,7203.T,005930.KS,2330.TW`	✅

新闻源配置（推荐）

新闻源会显著影响舆情、公告、事件和催化因素质量，建议至少配置一个搜索服务。

Secret 名称	说明	必填
`ANSPIRE_API_KEYS`	Anspire AI Search：汇聚全球舆情信息，适配A股、美股、港股等新闻和舆情检索；同一Key可复用大模型服务，本项目新用户提供免费30元等额的免费点数	推荐
`SERPAPI_API_KEYS`	SerpAPI：搜索引擎结果补强，适合实时金融新闻	推荐
`TAVILY_API_KEYS`	Tavily：通用新闻搜索 API	可选
`BOCHA_API_KEYS`	博查搜索：中文搜索优化，支持 AI 摘要	可选
`BRAVE_API_KEYS`	Brave Search：隐私优先，美股资讯补强	可选
`MINIMAX_API_KEYS`	MiniMax：结构化搜索结果	可选
`SEARXNG_BASE_URLS`	SearXNG 自建实例：无配额兜底，适合私有部署	可选

更多搜索源、社交舆情和降级规则见搜索服务配置。

行情数据源配置（可选）

默认使用 AkShare、Baostock、YFinance 等免费数据源，日志中"未配置"的提示不影响运行。如需更稳定的行情，可按市场配置以下 Secret：

Secret 名称	适用市场	说明
`TUSHARE_TOKEN`	A 股	提升历史行情稳定性
`LONGBRIDGE_OAUTH_CLIENT_ID` + `LONGBRIDGE_OAUTH_TOKEN_CACHE_B64`	港股/美股	补齐量比、换手率、PE 等字段

详见数据源配置。

3. 启用 Actions

Actions 标签 → I understand my workflows, go ahead and enable them

4. 手动测试

Actions → 每日股票分析 → Run workflow → Run workflow

完成

默认每个工作日 18:00（北京时间）自动执行，也可手动触发。默认非交易日（含 A/H/US 节假日）不执行；强制运行、交易日检查、断点续传等规则见完整指南。

方式二：客户端配置教程 / 本地运行 / Docker 部署

# 克隆项目
git clone https://github.com/ZhuLinsen/daily_stock_analysis.git && cd daily_stock_analysis

# 安装依赖
pip install -r requirements.txt

# 配置环境变量
cp .env.example .env && vim .env

# 运行分析
python main.py

常用命令：

python main.py --debug
python main.py --dry-run
python main.py --stocks 600519,hk00700,AAPL,2330.TW
python main.py --market-review
python main.py --schedule
python main.py --serve-only

Docker 部署、定时任务、云服务器访问请参考完整指南；桌面客户端打包请参考桌面端打包说明。

📱 推送效果

决策仪表盘

🎯 2026-02-08 决策仪表盘
共分析3只股票 | 🟢买入:0 🟡观望:2 🔴卖出:1

📊 分析结果摘要
⚪ 中钨高新(000657): 观望 | 评分 65 | 看多
⚪ 永鼎股份(600105): 观望 | 评分 48 | 震荡
🟡 新莱应材(300260): 卖出 | 评分 35 | 看空

⚪ 中钨高新 (000657)
📰 重要信息速览
💭 舆情情绪: 市场关注其AI属性与业绩高增长，情绪偏积极，但需消化短期获利盘和主力流出压力。
📊 业绩预期: 基于舆情信息，公司2025年前三季度业绩同比大幅增长，基本面强劲，为股价提供支撑。

🚨 风险警报:

风险点1：2月5日主力资金大幅净卖出3.63亿元，需警惕短期抛压。
风险点2：筹码集中度高达35.15%，表明筹码分散，拉升阻力可能较大。
风险点3：舆情中提及公司历史违规记录及重组相关风险提示，需保持关注。
✨ 利好催化:

利好1：公司被市场定位为AI服务器HDI核心供应商，受益于AI产业发展。
利好2：2025年前三季度扣非净利润同比暴涨407.52%，业绩表现强劲。
📢 最新动态: 【最新消息】舆情显示公司是AI PCB微钻领域龙头，深度绑定全球头部PCB/载板厂。2月5日主力资金净卖出3.63亿元，需关注后续资金流向。

---
生成时间: 18:00

大盘复盘

🎯 2026-01-10 大盘复盘

📊 主要指数
- 上证指数: 3250.12 (🟢+0.85%)
- 深证成指: 10521.36 (🟢+1.02%)
- 创业板指: 2156.78 (🟢+1.35%)

📈 市场概况
上涨: 3920 | 下跌: 1349 | 涨停: 155 | 跌停: 3

🔥 板块表现
领涨: 互联网服务、文化传媒、小金属
领跌: 保险、航空机场、光伏设备

⚙️ 配置说明

完整环境变量、模型渠道、通知渠道、数据源优先级、交易纪律、基本面 P0 语义和部署说明请参考完整配置指南。

🖥️ Web 界面

Web 工作台提供配置管理、任务监控、手动分析、历史报告、完整 Markdown 报告、Agent 问股、回测、持仓管理、智能导入和浅色 / 深色主题。启动方式：

python main.py --webui
python main.py --webui-only

访问 http://127.0.0.1:8000 即可使用。认证、智能导入、搜索补全、历史报告复制、云服务器访问等细节见本地 WebUI 管理界面。

🤖 Agent 策略问股

配置任意可用 AI API Key 后，Web /chat 页面即可使用策略问股；如需显式关闭可设置 AGENT_MODE=false。

支持均线金叉、缠论、波浪理论、多头趋势、热点题材、事件驱动、成长质量、预期重估等内置策略
支持实时行情、K 线、技术指标、新闻和风险信息调用
支持多轮追问、会话导出、发送到通知渠道和后台执行
支持自定义策略文件与多 Agent 编排（实验性）

Agent 具体参数、skill 命名兼容、多 Agent 模式和预算护栏见完整指南与 LLM 配置指南。

🧩 相关项目 (Related Projects)

DSA 聚焦日常分析报告；选股实现参考 AlphaSift，AlphaEvo 用于策略验证与进化。

项目	定位
AlphaSift	DSA 选股实现的参考项目
AlphaEvo	策略回测与自我进化，用于验证策略规则，并通过迭代探索策略参数与组合

📬 联系与合作

合作邮箱	zhuls345@gmail.com 项目咨询、部署支持与功能扩展	_{扫码关注小红书}
小红书	欢迎关注小红书
问题反馈	提交 Issue

📄 License

欢迎在二次开发或引用时注明本仓库来源，感谢支持项目持续维护。

⚠️ 免责声明

本项目仅供学习和研究使用，不构成任何投资建议。股市有风险，投资需谨慎。作者不对使用本项目产生的任何损失负责。

practical-tutorials/project-based-learning

Curated list of project-based tutorials

Project Based Learning

A list of programming tutorials in which aspiring software developers learn how to build an application from scratch. These tutorials are divided into different primary programming languages. Tutorials may involve multiple technologies and languages.

To get started, simply fork this repo. Please refer to CONTRIBUTING.md for contribution guidelines.

C/C++:

Build an Interpreter (Chapter 14 on is written in C)
Memory Allocators 101 - Write a simple memory allocator
Write a Shell in C
Write a FUSE Filesystem
Build Your Own Text Editor
Build Your Own Lisp
How to Program an NES Game in C
Write an OS from scratch
How to create an OS from scratch
Building a CHIP-8 Emulator
Beginning Game Programming with C++ and SDL
Implementing a Key-Value Store
Tiny 3D graphics projects
- Tiny Renderer or how OpenGL works: software rendering in 500 lines of code
- Understandable RayTracing in 256 lines of bare C++
- KABOOM! in 180 lines of bare C++
- 486 lines of C++: old-school FPS in a weekend
Writing a minimal x86-64 JIT compiler in C++
- Part 1
- Part 2
Build a Live Code-reloader Library for C++
Write a hash table in C
Let's Build a Simple Database
Let's Write a Kernel
Write a Bootloader in C
Linux Container in 500 Lines of Code
Write Your Own Virtual Machine
Learning KVM - Implement Your Own Linux Kernel
Build Your Own Redis with C/C++
Write a C compiler
- Part 1: Integers, Lexing and Code Generation
- Part 2: Unary Operators
- Part 3: Binary Operators
- Part 4: Even More Binary Operators
- Part 5: Local Variables
- Part 6: Conditionals
- Part 7: Compound Statements
- Part 8: Loops
- Part 9: Functions
- Part 10: Global Variables
Implementing a Language with LLVM
Meta Crush Saga: a C++17 compile-time game
High-Performance Matrix Multiplication
Space Invaders from Scratch
- Part 1
- Part 2
- Part 3
- Part 4
- Part 5
Tetris Tutorial in C++ Platform Independent
Writing a Linux Debugger
- Part 1: Setup
- Part 2: Breakpoints
- Part 3: Registers and memory
- Part 4: Elves and dwarves
- Part 5: Source and signals
- Part 6: Source-level stepping
- Part 7: Source-level breakpoints
- Part 8: Stack unwinding
- Part 9: Handling variables
- Part 10: Advanced topics
Let's write a compiler
- Part 1: Introduction, selecting a language, and doing some planning
- Part 2: A lexer
- Part 3: A parser
- Part 4: Testing
- Part 5: A code generator
- Part 6: Input and output
- Part 7: Arrays
- Part 8: Strings, forward references, and conclusion

Network programming

OpenGL:

Creating 2D Breakout game clone in C++ with OpenGL
- Breakout
- Setting up
- Rendering Sprites
- Levels
- Collisions
- Particles
- Postprocessing
- Powerups
- Audio
- Render text
- Final thoughts
Handmade Hero
How to Make Minecraft in C++/OpenGL (video)

C#:

Learn C# By Building a Simple RPG Game
Create a Rogue-like game in C#
Create a Blank App with C# and Xamarin (work in progress)
Build iOS Photo Library App with Xamarin and Visual Studio
Building the CoreWiki This is a Wiki-style content management system that has been completely written in C# with ASP.NET Core and Razor Pages. You can find the source code here.

Clojure:

Dart:

Flutter:

Elixir

Erlang

F#:

Write your own Excel in 100 lines of F#

Java:

Build an Interpreter (Chapter 4-13 is written in Java)
Build a Simple HTTP Server with Java
Build an Android Flashlight App (video)
Build a Spring Boot App with User Authentication

JavaScript:

HTML and CSS:

Mobile Application:

Web Applications:

React:

Others (Hapi, Express...):

Build a Progressive Web Application (PWA)
- Part 1
- Part 2
- Part 3
Build A Native Desktop App with JS
Build a Powerful API with NodeJs,GraphQL and Hapi
- Part I

D3.js

Game Development:

Make 2D Breakout Game using Phaser
Make Flappy Bird in HTML5 and JavaScript with Phaser
- Part 1
- Part 2

Desktop Application:

Build A Desktop Chat App with React and Electron

Miscellaneous:

Kotlin:

Keddit - Learn Kotlin While Developing an Android Application

Lua:

LÖVE:

BYTEPATH: Creation of a Complete Game with Lua and LÖVE

Python:

Web Scraping:

Web Applications:

Build a Microblog with Flask
Create a Blog Web App In Django
Choose Your Own Adventure Presentations
Build a Todo List with Flask and RethinkDB
Build a Todo List with Django and Test-Driven Development
Build a RESTful Microservice in Python
Microservices with Docker, Flask, and React
Build A Simple Web App With Flask
Create A Django API in under 20 minutes
Build a Community-driven delivery application with Django, Postgres and JavaScript
- Part 1
- Part 2
Realtime Chat application with Vue, django-notifs, RabbitMQ and uWSGI
- Part 1
- Part 2
- Part 3
- Part 4
- Part 5
- Part 6
Creating APIs with Python - Comprehensive Course for Beginners (video)
Build Your First Streamlit Data App

Bots:

Build a Reddit Bot
How to Make a Reddit Bot - YouTube (video)
Build a Facebook Messenger Bot
Making a Reddit + Facebook Messenger Bot
How To Create a Telegram Bot Using Python
- Part 1
- Part 2
Create a Twitter Bot In Python

Data Science:

Learn Python For Data Science by Doing Several Projects (video):

Machine Learning:

OpenCV:

Deep Learning:

Miscellaneous:

Build a Simple Interpreter
Build a Simple Blockchain in Python
Building a Gas Pump Scanner with OpenCV/Python/iOS
Build a Distributed Streaming System with Python and Kafka
Writing a basic x86-64 JIT compiler from scratch in stock Python
Making a low level (Linux) debugger
- Part 1
- Part 2: C
Implementing a Search Engine
- Part 1
- Part 2
- Part 3
Build the Game of Life
Create terminal ASCII art
Write a Tic-Tac-Toe AI
Create photomosaic art
Build the game "Snake" in the terminal
Write yourself a Git
A Python implementation of a Python bytecode runner
Create a Voice assistant using Python

Go:

Create a Real Time Chat App with Golang, Angular 2, and WebSocket
Building Go Web Applications and Microservices Using Gin
How to Use Godog for Behavior-driven Development in Go
Building Blockchain in Go
Building a container from scratch in Go - Liz Rice (Microscaling Systems)(video)
Build Web Application with GoLang
Building a Chat Application in Go with ReactJS
Go WebAssembly Tutorial - Building a Calculator Tutorial
REST Servers in Go
Let's build a URL shortener in Go - with Gin & Redis
Building a TCP Chat in Go(video)
Building a BitTorrent client from the ground up in Go
REST API masterclass with Go, PostgreSQL and Docker(video playlist)in progress

PHP:

How To Build A Blog With Laravel (video)
Make Your Own Blog (in Pure PHP)
Build A Real Estate Website Example with SilverStripe
Building Realtime Chat App with Laravel 5.4 and VueJS (video)
Build A Social Network: Laravel 5 - Youtube (video)
Build a full-featured multi-tenant app with Laravel
Build a Laravel CRUD Application From Scratch

OCaml:

Ruby:

Ruby on Rails:

Haskell:

R:

Rust:

A Simple Web App in Rust
- Part 1
- Part 2a
- Part 2b
Write an OS in pure Rust
Build a browser engine in Rust
Write a Microservice in Rust
Learning Rust with Too Many Linked Lists
Rust in Detail: Writing Scalable Chat Service from Scratch
- Part 1: Implementing WebSocket. Introduction.
- Part 2: Sending and Receiving Messages
Writing a Rust Roguelike for the Desktop and the Web
Single Page Applications using Rust
Writing NES Emulator in Rust
Create a simulation of evolution using neural network and genetic algorithm, and compile the application to WebAssembly
- Part 1
- Part 2
- Part 3
- Part 4
hecto: Build your own text editor in Rust

Scala:

Swift:

Additional Resources

donnemartin/system-design-primer

Learn how to design large-scale systems. Prep for the system design interview. Includes Anki flashcards.

English ∙ 日本語 ∙ 简体中文 ∙ 繁體中文 | العَرَبِيَّة‎ ∙ বাংলা ∙ Português do Brasil ∙ Deutsch ∙ ελληνικά ∙ עברית ∙ Italiano ∙ 한국어 ∙ فارسی ∙ Polski ∙ русский язык ∙ Español ∙ ภาษาไทย ∙ Türkçe ∙ tiếng Việt ∙ Français | Add Translation

Help translate this guide!

The System Design Primer

Motivation

Learn how to design large-scale systems.

Prep for the system design interview.

Learn how to design large-scale systems

Learning how to design scalable systems will help you become a better engineer.

System design is a broad topic. There are a vast number of resources scattered throughout the web on system design principles.

This repo is an organized collection of resources to help you learn how to build systems at scale.

Learn from the open source community

This is a continually updated, open source project.

Contributions are welcome!

Prep for the system design interview

In addition to coding interviews, system design is a required component of the technical interview process at many tech companies.

Practice common system design interview questions and compare your results with sample solutions: discussions, code, and diagrams.

Additional topics for interview prep:

Anki flashcards

The provided Anki flashcard decks use spaced repetition to help you retain key system design concepts.

Great for use while on-the-go.

Coding Resource: Interactive Coding Challenges

Looking for resources to help you prep for the Coding Interview?

Check out the sister repo Interactive Coding Challenges, which contains an additional Anki deck:

Coding deck

Contributing

Learn from the community.

Feel free to submit pull requests to help:

Fix errors
Improve sections
Add new sections
Translate

Content that needs some polishing is placed under development.

Review the Contributing Guidelines.

Index of system design topics

Summaries of various system design topics, including pros and cons. Everything is a trade-off.

Each section contains links to more in-depth resources.

Study guide

Suggested topics to review based on your interview timeline (short, medium, long).

Q: For interviews, do I need to know everything here?

A: No, you don't need to know everything here to prepare for the interview.

What you are asked in an interview depends on variables such as:

How much experience you have
What your technical background is
What positions you are interviewing for
Which companies you are interviewing with
Luck

More experienced candidates are generally expected to know more about system design. Architects or team leads might be expected to know more than individual contributors. Top tech companies are likely to have one or more design interview rounds.

Start broad and go deeper in a few areas. It helps to know a little about various key system design topics. Adjust the following guide based on your timeline, experience, what positions you are interviewing for, and which companies you are interviewing with.

Short timeline - Aim for breadth with system design topics. Practice by solving some interview questions.
Medium timeline - Aim for breadth and some depth with system design topics. Practice by solving many interview questions.
Long timeline - Aim for breadth and more depth with system design topics. Practice by solving most interview questions.

	Short	Medium	Long
Read through the System design topics to get a broad understanding of how systems work	👍	👍	👍
Read through a few articles in the Company engineering blogs for the companies you are interviewing with	👍	👍	👍
Read through a few Real world architectures	👍	👍	👍
Review How to approach a system design interview question	👍	👍	👍
Work through System design interview questions with solutions	Some	Many	Most
Work through Object-oriented design interview questions with solutions	Some	Many	Most
Review Additional system design interview questions	Some	Many	Most

How to approach a system design interview question

How to tackle a system design interview question.

The system design interview is an open-ended conversation. You are expected to lead it.

You can use the following steps to guide the discussion. To help solidify this process, work through the System design interview questions with solutions section using the following steps.

Step 1: Outline use cases, constraints, and assumptions

Gather requirements and scope the problem. Ask questions to clarify use cases and constraints. Discuss assumptions.

Who is going to use it?
How are they going to use it?
How many users are there?
What does the system do?
What are the inputs and outputs of the system?
How much data do we expect to handle?
How many requests per second do we expect?
What is the expected read to write ratio?

Step 2: Create a high level design

Outline a high level design with all important components.

Sketch the main components and connections
Justify your ideas

Step 3: Design core components

Dive into details for each core component. For example, if you were asked to design a url shortening service, discuss:

Generating and storing a hash of the full url
- MD5 and Base62
- Hash collisions
- SQL or NoSQL
- Database schema
Translating a hashed url to the full url
- Database lookup
API and object-oriented design

Step 4: Scale the design

Identify and address bottlenecks, given the constraints. For example, do you need the following to address scalability issues?

Load balancer
Horizontal scaling
Caching
Database sharding

Discuss potential solutions and trade-offs. Everything is a trade-off. Address bottlenecks using principles of scalable system design.

Back-of-the-envelope calculations

You might be asked to do some estimates by hand. Refer to the Appendix for the following resources:

Source(s) and further reading

Check out the following links to get a better idea of what to expect:

System design interview questions with solutions

Common system design interview questions with sample discussions, code, and diagrams.

Solutions linked to content in the solutions/ folder.

Question
Design Pastebin.com (or Bit.ly)	Solution
Design the Twitter timeline and search (or Facebook feed and search)	Solution
Design a web crawler	Solution
Design Mint.com	Solution
Design the data structures for a social network	Solution
Design a key-value store for a search engine	Solution
Design Amazon's sales ranking by category feature	Solution
Design a system that scales to millions of users on AWS	Solution
Add a system design question	Contribute

Design Pastebin.com (or Bit.ly)

View exercise and solution

Design the Twitter timeline and search (or Facebook feed and search)

View exercise and solution

Design a web crawler

View exercise and solution

Design Mint.com

View exercise and solution

Design the data structures for a social network

View exercise and solution

Design a key-value store for a search engine

View exercise and solution

Design Amazon's sales ranking by category feature

View exercise and solution

Design a system that scales to millions of users on AWS

View exercise and solution

Object-oriented design interview questions with solutions

Common object-oriented design interview questions with sample discussions, code, and diagrams.

Solutions linked to content in the solutions/ folder.

Note: This section is under development

Question
Design a hash map	Solution
Design a least recently used cache	Solution
Design a call center	Solution
Design a deck of cards	Solution
Design a parking lot	Solution
Design a chat server	Solution
Design a circular array	Contribute
Add an object-oriented design question	Contribute

System design topics: start here

New to system design?

First, you'll need a basic understanding of common principles, learning about what they are, how they are used, and their pros and cons.

Step 1: Review the scalability video lecture

Scalability Lecture at Harvard

Topics covered:
- Vertical scaling
- Horizontal scaling
- Caching
- Load balancing
- Database replication
- Database partitioning

Step 2: Review the scalability article

Scalability

Topics covered:
- Clones
- Databases
- Caches
- Asynchronism

Next steps

Next, we'll look at high-level trade-offs:

Performance vs scalability
Latency vs throughput
Availability vs consistency

Keep in mind that everything is a trade-off.

Then we'll dive into more specific topics such as DNS, CDNs, and load balancers.

Performance vs scalability

A service is scalable if it results in increased performance in a manner proportional to resources added. Generally, increasing performance means serving more units of work, but it can also be to handle larger units of work, such as when datasets grow.¹

Another way to look at performance vs scalability:

If you have a performance problem, your system is slow for a single user.
If you have a scalability problem, your system is fast for a single user but slow under heavy load.

Source(s) and further reading

Latency vs throughput

Latency is the time to perform some action or to produce some result.

Throughput is the number of such actions or results per unit of time.

Generally, you should aim for maximal throughput with acceptable latency.

Source(s) and further reading

Understanding latency vs throughput

Availability vs consistency

CAP theorem

Source: CAP theorem revisited

In a distributed computer system, you can only support two of the following guarantees:

Consistency - Every read receives the most recent write or an error
Availability - Every request receives a response, without guarantee that it contains the most recent version of the information
Partition Tolerance - The system continues to operate despite arbitrary partitioning due to network failures

Networks aren't reliable, so you'll need to support partition tolerance. You'll need to make a software tradeoff between consistency and availability.

CP - consistency and partition tolerance

Waiting for a response from the partitioned node might result in a timeout error. CP is a good choice if your business needs require atomic reads and writes.

AP - availability and partition tolerance

Responses return the most readily available version of the data available on any node, which might not be the latest. Writes might take some time to propagate when the partition is resolved.

AP is a good choice if the business needs to allow for eventual consistency or when the system needs to continue working despite external errors.

Source(s) and further reading

Consistency patterns

With multiple copies of the same data, we are faced with options on how to synchronize them so clients have a consistent view of the data. Recall the definition of consistency from the CAP theorem - Every read receives the most recent write or an error.

Weak consistency

After a write, reads may or may not see it. A best effort approach is taken.

This approach is seen in systems such as memcached. Weak consistency works well in real time use cases such as VoIP, video chat, and realtime multiplayer games. For example, if you are on a phone call and lose reception for a few seconds, when you regain connection you do not hear what was spoken during connection loss.

Eventual consistency

After a write, reads will eventually see it (typically within milliseconds). Data is replicated asynchronously.

This approach is seen in systems such as DNS and email. Eventual consistency works well in highly available systems.

Strong consistency

After a write, reads will see it. Data is replicated synchronously.

This approach is seen in file systems and RDBMSes. Strong consistency works well in systems that need transactions.

Source(s) and further reading

Transactions across data centers

Availability patterns

There are two complementary patterns to support high availability: fail-over and replication.

Fail-over

Active-passive

With active-passive fail-over, heartbeats are sent between the active and the passive server on standby. If the heartbeat is interrupted, the passive server takes over the active's IP address and resumes service.

The length of downtime is determined by whether the passive server is already running in 'hot' standby or whether it needs to start up from 'cold' standby. Only the active server handles traffic.

Active-passive failover can also be referred to as master-slave failover.

Active-active

In active-active, both servers are managing traffic, spreading the load between them.

If the servers are public-facing, the DNS would need to know about the public IPs of both servers. If the servers are internal-facing, application logic would need to know about both servers.

Active-active failover can also be referred to as master-master failover.

Disadvantage(s): failover

Fail-over adds more hardware and additional complexity.
There is a potential for loss of data if the active system fails before any newly written data can be replicated to the passive.

Replication

Master-slave and master-master

This topic is further discussed in the Database section:

Availability in numbers

Availability is often quantified by uptime (or downtime) as a percentage of time the service is available. Availability is generally measured in number of 9s--a service with 99.99% availability is described as having four 9s.

99.9% availability - three 9s

Duration	Acceptable downtime
Downtime per year	8h 45min 57s
Downtime per month	43m 49.7s
Downtime per week	10m 4.8s
Downtime per day	1m 26.4s

99.99% availability - four 9s

Duration	Acceptable downtime
Downtime per year	52min 35.7s
Downtime per month	4m 23s
Downtime per week	1m 5s
Downtime per day	8.6s

Availability in parallel vs in sequence

If a service consists of multiple components prone to failure, the service's overall availability depends on whether the components are in sequence or in parallel.

In sequence

Overall availability decreases when two components with availability < 100% are in sequence:

Availability (Total) = Availability (Foo) * Availability (Bar)

If both Foo and Bar each had 99.9% availability, their total availability in sequence would be 99.8%.

In parallel

Overall availability increases when two components with availability < 100% are in parallel:

Availability (Total) = 1 - (1 - Availability (Foo)) * (1 - Availability (Bar))

If both Foo and Bar each had 99.9% availability, their total availability in parallel would be 99.9999%.

Domain name system

Source: DNS security presentation

A Domain Name System (DNS) translates a domain name such as www.example.com to an IP address.

DNS is hierarchical, with a few authoritative servers at the top level. Your router or ISP provides information about which DNS server(s) to contact when doing a lookup. Lower level DNS servers cache mappings, which could become stale due to DNS propagation delays. DNS results can also be cached by your browser or OS for a certain period of time, determined by the time to live (TTL).

NS record (name server) - Specifies the DNS servers for your domain/subdomain.
MX record (mail exchange) - Specifies the mail servers for accepting messages.
A record (address) - Points a name to an IP address.
CNAME (canonical) - Points a name to another name or CNAME (example.com to www.example.com) or to an A record.

Services such as CloudFlare and Route 53 provide managed DNS services. Some DNS services can route traffic through various methods:

Weighted round robin
- Prevent traffic from going to servers under maintenance
- Balance between varying cluster sizes
- A/B testing
Latency-based
Geolocation-based

Disadvantage(s): DNS

Accessing a DNS server introduces a slight delay, although mitigated by caching described above.
DNS server management could be complex and is generally managed by governments, ISPs, and large companies.
DNS services have recently come under DDoS attack, preventing users from accessing websites such as Twitter without knowing Twitter's IP address(es).

Source(s) and further reading

Content delivery network

Source: Why use a CDN

A content delivery network (CDN) is a globally distributed network of proxy servers, serving content from locations closer to the user. Generally, static files such as HTML/CSS/JS, photos, and videos are served from CDN, although some CDNs such as Amazon's CloudFront support dynamic content. The site's DNS resolution will tell clients which server to contact.

Serving content from CDNs can significantly improve performance in two ways:

Users receive content from data centers close to them
Your servers do not have to serve requests that the CDN fulfills

Push CDNs

Push CDNs receive new content whenever changes occur on your server. You take full responsibility for providing content, uploading directly to the CDN and rewriting URLs to point to the CDN. You can configure when content expires and when it is updated. Content is uploaded only when it is new or changed, minimizing traffic, but maximizing storage.

Sites with a small amount of traffic or sites with content that isn't often updated work well with push CDNs. Content is placed on the CDNs once, instead of being re-pulled at regular intervals.

Pull CDNs

Pull CDNs grab new content from your server when the first user requests the content. You leave the content on your server and rewrite URLs to point to the CDN. This results in a slower request until the content is cached on the CDN.

A time-to-live (TTL) determines how long content is cached. Pull CDNs minimize storage space on the CDN, but can create redundant traffic if files expire and are pulled before they have actually changed.

Sites with heavy traffic work well with pull CDNs, as traffic is spread out more evenly with only recently-requested content remaining on the CDN.

Disadvantage(s): CDN

CDN costs could be significant depending on traffic, although this should be weighed with additional costs you would incur not using a CDN.
Content might be stale if it is updated before the TTL expires it.
CDNs require changing URLs for static content to point to the CDN.

Source(s) and further reading

Load balancer

Source: Scalable system design patterns

Load balancers distribute incoming client requests to computing resources such as application servers and databases. In each case, the load balancer returns the response from the computing resource to the appropriate client. Load balancers are effective at:

Preventing requests from going to unhealthy servers
Preventing overloading resources
Helping to eliminate a single point of failure

Load balancers can be implemented with hardware (expensive) or with software such as HAProxy.

Additional benefits include:

SSL termination - Decrypt incoming requests and encrypt server responses so backend servers do not have to perform these potentially expensive operations
- Removes the need to install X.509 certificates on each server
Session persistence - Issue cookies and route a specific client's requests to same instance if the web apps do not keep track of sessions

To protect against failures, it's common to set up multiple load balancers, either in active-passive or active-active mode.

Load balancers can route traffic based on various metrics, including:

Random
Least loaded
Session/cookies
Round robin or weighted round robin
Layer 4
Layer 7

Layer 4 load balancing

Layer 4 load balancers look at info at the transport layer to decide how to distribute requests. Generally, this involves the source, destination IP addresses, and ports in the header, but not the contents of the packet. Layer 4 load balancers forward network packets to and from the upstream server, performing Network Address Translation (NAT).

Layer 7 load balancing

Layer 7 load balancers look at the application layer to decide how to distribute requests. This can involve contents of the header, message, and cookies. Layer 7 load balancers terminate network traffic, reads the message, makes a load-balancing decision, then opens a connection to the selected server. For example, a layer 7 load balancer can direct video traffic to servers that host videos while directing more sensitive user billing traffic to security-hardened servers.

At the cost of flexibility, layer 4 load balancing requires less time and computing resources than Layer 7, although the performance impact can be minimal on modern commodity hardware.

Horizontal scaling

Load balancers can also help with horizontal scaling, improving performance and availability. Scaling out using commodity machines is more cost efficient and results in higher availability than scaling up a single server on more expensive hardware, called Vertical Scaling. It is also easier to hire for talent working on commodity hardware than it is for specialized enterprise systems.

Disadvantage(s): horizontal scaling

Scaling horizontally introduces complexity and involves cloning servers
- Servers should be stateless: they should not contain any user-related data like sessions or profile pictures
- Sessions can be stored in a centralized data store such as a database (SQL, NoSQL) or a persistent cache (Redis, Memcached)
Downstream servers such as caches and databases need to handle more simultaneous connections as upstream servers scale out

Disadvantage(s): load balancer

The load balancer can become a performance bottleneck if it does not have enough resources or if it is not configured properly.
Introducing a load balancer to help eliminate a single point of failure results in increased complexity.
A single load balancer is a single point of failure, configuring multiple load balancers further increases complexity.

Source(s) and further reading

Reverse proxy (web server)

Source: Wikipedia

A reverse proxy is a web server that centralizes internal services and provides unified interfaces to the public. Requests from clients are forwarded to a server that can fulfill it before the reverse proxy returns the server's response to the client.

Additional benefits include:

Increased security - Hide information about backend servers, blacklist IPs, limit number of connections per client
Increased scalability and flexibility - Clients only see the reverse proxy's IP, allowing you to scale servers or change their configuration
SSL termination - Decrypt incoming requests and encrypt server responses so backend servers do not have to perform these potentially expensive operations
- Removes the need to install X.509 certificates on each server
Compression - Compress server responses
Caching - Return the response for cached requests
Static content - Serve static content directly
- HTML/CSS/JS
- Photos
- Videos
- Etc

Load balancer vs reverse proxy

Deploying a load balancer is useful when you have multiple servers. Often, load balancers route traffic to a set of servers serving the same function.
Reverse proxies can be useful even with just one web server or application server, opening up the benefits described in the previous section.
Solutions such as NGINX and HAProxy can support both layer 7 reverse proxying and load balancing.

Disadvantage(s): reverse proxy

Introducing a reverse proxy results in increased complexity.
A single reverse proxy is a single point of failure, configuring multiple reverse proxies (ie a failover) further increases complexity.

Source(s) and further reading

Application layer

Source: Intro to architecting systems for scale

Separating out the web layer from the application layer (also known as platform layer) allows you to scale and configure both layers independently. Adding a new API results in adding application servers without necessarily adding additional web servers. The single responsibility principle advocates for small and autonomous services that work together. Small teams with small services can plan more aggressively for rapid growth.

Workers in the application layer also help enable asynchronism.

Microservices

Related to this discussion are microservices, which can be described as a suite of independently deployable, small, modular services. Each service runs a unique process and communicates through a well-defined, lightweight mechanism to serve a business goal. ¹

Pinterest, for example, could have the following microservices: user profile, follower, feed, search, photo upload, etc.

Service Discovery

Systems such as Consul, Etcd, and Zookeeper can help services find each other by keeping track of registered names, addresses, and ports. Health checks help verify service integrity and are often done using an HTTP endpoint. Both Consul and Etcd have a built in key-value store that can be useful for storing config values and other shared data.

Disadvantage(s): application layer

Adding an application layer with loosely coupled services requires a different approach from an architectural, operations, and process viewpoint (vs a monolithic system).
Microservices can add complexity in terms of deployments and operations.

Source(s) and further reading

Database

Source: Scaling up to your first 10 million users

Relational database management system (RDBMS)

A relational database like SQL is a collection of data items organized in tables.

ACID is a set of properties of relational database transactions.

Atomicity - Each transaction is all or nothing
Consistency - Any transaction will bring the database from one valid state to another
Isolation - Executing transactions concurrently has the same results as if the transactions were executed serially
Durability - Once a transaction has been committed, it will remain so

There are many techniques to scale a relational database: master-slave replication, master-master replication, federation, sharding, denormalization, and SQL tuning.

Master-slave replication

The master serves reads and writes, replicating writes to one or more slaves, which serve only reads. Slaves can also replicate to additional slaves in a tree-like fashion. If the master goes offline, the system can continue to operate in read-only mode until a slave is promoted to a master or a new master is provisioned.

Source: Scalability, availability, stability, patterns

Disadvantage(s): master-slave replication

Additional logic is needed to promote a slave to a master.
See Disadvantage(s): replication for points related to both master-slave and master-master.

Master-master replication

Both masters serve reads and writes and coordinate with each other on writes. If either master goes down, the system can continue to operate with both reads and writes.

Source: Scalability, availability, stability, patterns

Disadvantage(s): master-master replication

You'll need a load balancer or you'll need to make changes to your application logic to determine where to write.
Most master-master systems are either loosely consistent (violating ACID) or have increased write latency due to synchronization.
Conflict resolution comes more into play as more write nodes are added and as latency increases.
See Disadvantage(s): replication for points related to both master-slave and master-master.

Disadvantage(s): replication

There is a potential for loss of data if the master fails before any newly written data can be replicated to other nodes.
Writes are replayed to the read replicas. If there are a lot of writes, the read replicas can get bogged down with replaying writes and can't do as many reads.
The more read slaves, the more you have to replicate, which leads to greater replication lag.
On some systems, writing to the master can spawn multiple threads to write in parallel, whereas read replicas only support writing sequentially with a single thread.
Replication adds more hardware and additional complexity.

Source(s) and further reading: replication

Federation

Source: Scaling up to your first 10 million users

Federation (or functional partitioning) splits up databases by function. For example, instead of a single, monolithic database, you could have three databases: forums, users, and products, resulting in less read and write traffic to each database and therefore less replication lag. Smaller databases result in more data that can fit in memory, which in turn results in more cache hits due to improved cache locality. With no single central master serializing writes you can write in parallel, increasing throughput.

Disadvantage(s): federation

Federation is not effective if your schema requires huge functions or tables.
You'll need to update your application logic to determine which database to read and write.
Joining data from two databases is more complex with a server link.
Federation adds more hardware and additional complexity.

Source(s) and further reading: federation

Scaling up to your first 10 million users

Sharding

Source: Scalability, availability, stability, patterns

Sharding distributes data across different databases such that each database can only manage a subset of the data. Taking a users database as an example, as the number of users increases, more shards are added to the cluster.

Similar to the advantages of federation, sharding results in less read and write traffic, less replication, and more cache hits. Index size is also reduced, which generally improves performance with faster queries. If one shard goes down, the other shards are still operational, although you'll want to add some form of replication to avoid data loss. Like federation, there is no single central master serializing writes, allowing you to write in parallel with increased throughput.

Common ways to shard a table of users is either through the user's last name initial or the user's geographic location.

Disadvantage(s): sharding

You'll need to update your application logic to work with shards, which could result in complex SQL queries.
Data distribution can become lopsided in a shard. For example, a set of power users on a shard could result in increased load to that shard compared to others.
- Rebalancing adds additional complexity. A sharding function based on consistent hashing can reduce the amount of transferred data.
Joining data from multiple shards is more complex.
Sharding adds more hardware and additional complexity.

Source(s) and further reading: sharding

Denormalization

Denormalization attempts to improve read performance at the expense of some write performance. Redundant copies of the data are written in multiple tables to avoid expensive joins. Some RDBMS such as PostgreSQL and Oracle support materialized views which handle the work of storing redundant information and keeping redundant copies consistent.

Once data becomes distributed with techniques such as federation and sharding, managing joins across data centers further increases complexity. Denormalization might circumvent the need for such complex joins.

In most systems, reads can heavily outnumber writes 100:1 or even 1000:1. A read resulting in a complex database join can be very expensive, spending a significant amount of time on disk operations.

Disadvantage(s): denormalization

Data is duplicated.
Constraints can help redundant copies of information stay in sync, which increases complexity of the database design.
A denormalized database under heavy write load might perform worse than its normalized counterpart.

Source(s) and further reading: denormalization

Denormalization

SQL tuning

SQL tuning is a broad topic and many books have been written as reference.

It's important to benchmark and profile to simulate and uncover bottlenecks.

Benchmark - Simulate high-load situations with tools such as ab.
Profile - Enable tools such as the slow query log to help track performance issues.

Benchmarking and profiling might point you to the following optimizations.

Tighten up the schema

MySQL dumps to disk in contiguous blocks for fast access.
Use CHAR instead of VARCHAR for fixed-length fields.
- CHAR effectively allows for fast, random access, whereas with VARCHAR, you must find the end of a string before moving onto the next one.
Use TEXT for large blocks of text such as blog posts. TEXT also allows for boolean searches. Using a TEXT field results in storing a pointer on disk that is used to locate the text block.
Use INT for larger numbers up to 2^32 or 4 billion.
Use DECIMAL for currency to avoid floating point representation errors.
Avoid storing large BLOBS, store the location of where to get the object instead.
VARCHAR(255) is the largest number of characters that can be counted in an 8 bit number, often maximizing the use of a byte in some RDBMS.
Set the NOT NULL constraint where applicable to improve search performance.

Use good indices

Columns that you are querying (SELECT, GROUP BY, ORDER BY, JOIN) could be faster with indices.
Indices are usually represented as self-balancing B-tree that keeps data sorted and allows searches, sequential access, insertions, and deletions in logarithmic time.
Placing an index can keep the data in memory, requiring more space.
Writes could also be slower since the index also needs to be updated.
When loading large amounts of data, it might be faster to disable indices, load the data, then rebuild the indices.

Avoid expensive joins

Denormalize where performance demands it.

Partition tables

Break up a table by putting hot spots in a separate table to help keep it in memory.

Tune the query cache

In some cases, the query cache could lead to performance issues.

Source(s) and further reading: SQL tuning

NoSQL

NoSQL is a collection of data items represented in a key-value store, document store, wide column store, or a graph database. Data is denormalized, and joins are generally done in the application code. Most NoSQL stores lack true ACID transactions and favor eventual consistency.

BASE is often used to describe the properties of NoSQL databases. In comparison with the CAP Theorem, BASE chooses availability over consistency.

Basically available - the system guarantees availability.
Soft state - the state of the system may change over time, even without input.
Eventual consistency - the system will become consistent over a period of time, given that the system doesn't receive input during that period.

In addition to choosing between SQL or NoSQL, it is helpful to understand which type of NoSQL database best fits your use case(s). We'll review key-value stores, document stores, wide column stores, and graph databases in the next section.

Key-value store

Abstraction: hash table

A key-value store generally allows for O(1) reads and writes and is often backed by memory or SSD. Data stores can maintain keys in lexicographic order, allowing efficient retrieval of key ranges. Key-value stores can allow for storing of metadata with a value.

Key-value stores provide high performance and are often used for simple data models or for rapidly-changing data, such as an in-memory cache layer. Since they offer only a limited set of operations, complexity is shifted to the application layer if additional operations are needed.

A key-value store is the basis for more complex systems such as a document store, and in some cases, a graph database.

Source(s) and further reading: key-value store

Document store

Abstraction: key-value store with documents stored as values

A document store is centered around documents (XML, JSON, binary, etc), where a document stores all information for a given object. Document stores provide APIs or a query language to query based on the internal structure of the document itself. Note, many key-value stores include features for working with a value's metadata, blurring the lines between these two storage types.

Based on the underlying implementation, documents are organized by collections, tags, metadata, or directories. Although documents can be organized or grouped together, documents may have fields that are completely different from each other.

Some document stores like MongoDB and CouchDB also provide a SQL-like language to perform complex queries. DynamoDB supports both key-values and documents.

Document stores provide high flexibility and are often used for working with occasionally changing data.

Source(s) and further reading: document store

Wide column store

Source: SQL & NoSQL, a brief history

Abstraction: nested map ColumnFamily<RowKey, Columns<ColKey, Value, Timestamp>>

A wide column store's basic unit of data is a column (name/value pair). A column can be grouped in column families (analogous to a SQL table). Super column families further group column families. You can access each column independently with a row key, and columns with the same row key form a row. Each value contains a timestamp for versioning and for conflict resolution.

Google introduced Bigtable as the first wide column store, which influenced the open-source HBase often-used in the Hadoop ecosystem, and Cassandra from Facebook. Stores such as BigTable, HBase, and Cassandra maintain keys in lexicographic order, allowing efficient retrieval of selective key ranges.

Wide column stores offer high availability and high scalability. They are often used for very large data sets.

Source(s) and further reading: wide column store

Graph database

Source: Graph database

Abstraction: graph

In a graph database, each node is a record and each arc is a relationship between two nodes. Graph databases are optimized to represent complex relationships with many foreign keys or many-to-many relationships.

Graphs databases offer high performance for data models with complex relationships, such as a social network. They are relatively new and are not yet widely-used; it might be more difficult to find development tools and resources. Many graphs can only be accessed with REST APIs.

Source(s) and further reading: graph

Source(s) and further reading: NoSQL

SQL or NoSQL

Source: Transitioning from RDBMS to NoSQL

Reasons for SQL:

Structured data
Strict schema
Relational data
Need for complex joins
Transactions
Clear patterns for scaling
More established: developers, community, code, tools, etc
Lookups by index are very fast

Reasons for NoSQL:

Semi-structured data
Dynamic or flexible schema
Non-relational data
No need for complex joins
Store many TB (or PB) of data
Very data intensive workload
Very high throughput for IOPS

Sample data well-suited for NoSQL:

Rapid ingest of clickstream and log data
Leaderboard or scoring data
Temporary data, such as a shopping cart
Frequently accessed ('hot') tables
Metadata/lookup tables

Source(s) and further reading: SQL or NoSQL

Cache

Source: Scalable system design patterns

Caching improves page load times and can reduce the load on your servers and databases. In this model, the dispatcher will first lookup if the request has been made before and try to find the previous result to return, in order to save the actual execution.

Databases often benefit from a uniform distribution of reads and writes across its partitions. Popular items can skew the distribution, causing bottlenecks. Putting a cache in front of a database can help absorb uneven loads and spikes in traffic.

Client caching

Caches can be located on the client side (OS or browser), server side, or in a distinct cache layer.

CDN caching

CDNs are considered a type of cache.

Web server caching

Reverse proxies and caches such as Varnish can serve static and dynamic content directly. Web servers can also cache requests, returning responses without having to contact application servers.

Database caching

Your database usually includes some level of caching in a default configuration, optimized for a generic use case. Tweaking these settings for specific usage patterns can further boost performance.

Application caching

In-memory caches such as Memcached and Redis are key-value stores between your application and your data storage. Since the data is held in RAM, it is much faster than typical databases where data is stored on disk. RAM is more limited than disk, so cache invalidation algorithms such as least recently used (LRU) can help invalidate 'cold' entries and keep 'hot' data in RAM.

Redis has the following additional features:

Persistence option
Built-in data structures such as sorted sets and lists

There are multiple levels you can cache that fall into two general categories: database queries and objects:

Row level
Query-level
Fully-formed serializable objects
Fully-rendered HTML

Generally, you should try to avoid file-based caching, as it makes cloning and auto-scaling more difficult.

Caching at the database query level

Whenever you query the database, hash the query as a key and store the result to the cache. This approach suffers from expiration issues:

Hard to delete a cached result with complex queries
If one piece of data changes such as a table cell, you need to delete all cached queries that might include the changed cell

Caching at the object level

See your data as an object, similar to what you do with your application code. Have your application assemble the dataset from the database into a class instance or a data structure(s):

Remove the object from cache if its underlying data has changed
Allows for asynchronous processing: workers assemble objects by consuming the latest cached object

Suggestions of what to cache:

User sessions
Fully rendered web pages
Activity streams
User graph data

When to update the cache

Since you can only store a limited amount of data in cache, you'll need to determine which cache update strategy works best for your use case.

Cache-aside

Source: From cache to in-memory data grid

The application is responsible for reading and writing from storage. The cache does not interact with storage directly. The application does the following:

Look for entry in cache, resulting in a cache miss
Load entry from the database
Add entry to cache
Return entry

def get_user(self, user_id):
    user = cache.get("user.{0}", user_id)
    if user is None:
        user = db.query("SELECT * FROM users WHERE user_id = {0}", user_id)
        if user is not None:
            key = "user.{0}".format(user_id)
            cache.set(key, json.dumps(user))
    return user

Memcached is generally used in this manner.

Subsequent reads of data added to cache are fast. Cache-aside is also referred to as lazy loading. Only requested data is cached, which avoids filling up the cache with data that isn't requested.

Disadvantage(s): cache-aside

Each cache miss results in three trips, which can cause a noticeable delay.
Data can become stale if it is updated in the database. This issue is mitigated by setting a time-to-live (TTL) which forces an update of the cache entry, or by using write-through.
When a node fails, it is replaced by a new, empty node, increasing latency.

Write-through

Source: Scalability, availability, stability, patterns

The application uses the cache as the main data store, reading and writing data to it, while the cache is responsible for reading and writing to the database:

Application adds/updates entry in cache
Cache synchronously writes entry to data store
Return

Application code:

set_user(12345, {"foo":"bar"})

Cache code:

def set_user(user_id, values):
    user = db.query("UPDATE Users WHERE id = {0}", user_id, values)
    cache.set(user_id, user)

Write-through is a slow overall operation due to the write operation, but subsequent reads of just written data are fast. Users are generally more tolerant of latency when updating data than reading data. Data in the cache is not stale.

Disadvantage(s): write through

When a new node is created due to failure or scaling, the new node will not cache entries until the entry is updated in the database. Cache-aside in conjunction with write through can mitigate this issue.
Most data written might never be read, which can be minimized with a TTL.

Write-behind (write-back)

Source: Scalability, availability, stability, patterns

In write-behind, the application does the following:

Add/update entry in cache
Asynchronously write entry to the data store, improving write performance

Disadvantage(s): write-behind

There could be data loss if the cache goes down prior to its contents hitting the data store.
It is more complex to implement write-behind than it is to implement cache-aside or write-through.

Refresh-ahead

Source: From cache to in-memory data grid

You can configure the cache to automatically refresh any recently accessed cache entry prior to its expiration.

Refresh-ahead can result in reduced latency vs read-through if the cache can accurately predict which items are likely to be needed in the future.

Disadvantage(s): refresh-ahead

Not accurately predicting which items are likely to be needed in the future can result in reduced performance than without refresh-ahead.

Disadvantage(s): cache

Need to maintain consistency between caches and the source of truth such as the database through cache invalidation.
Cache invalidation is a difficult problem, there is additional complexity associated with when to update the cache.
Need to make application changes such as adding Redis or memcached.

Source(s) and further reading

Asynchronism

Source: Intro to architecting systems for scale

Asynchronous workflows help reduce request times for expensive operations that would otherwise be performed in-line. They can also help by doing time-consuming work in advance, such as periodic aggregation of data.

Message queues

Message queues receive, hold, and deliver messages. If an operation is too slow to perform inline, you can use a message queue with the following workflow:

An application publishes a job to the queue, then notifies the user of job status
A worker picks up the job from the queue, processes it, then signals the job is complete

The user is not blocked and the job is processed in the background. During this time, the client might optionally do a small amount of processing to make it seem like the task has completed. For example, if posting a tweet, the tweet could be instantly posted to your timeline, but it could take some time before your tweet is actually delivered to all of your followers.

Redis is useful as a simple message broker but messages can be lost.

RabbitMQ is popular but requires you to adapt to the 'AMQP' protocol and manage your own nodes.

Amazon SQS is hosted but can have high latency and has the possibility of messages being delivered twice.

Task queues

Tasks queues receive tasks and their related data, runs them, then delivers their results. They can support scheduling and can be used to run computationally-intensive jobs in the background.

Celery has support for scheduling and primarily has python support.

Back pressure

If queues start to grow significantly, the queue size can become larger than memory, resulting in cache misses, disk reads, and even slower performance. Back pressure can help by limiting the queue size, thereby maintaining a high throughput rate and good response times for jobs already in the queue. Once the queue fills up, clients get a server busy or HTTP 503 status code to try again later. Clients can retry the request at a later time, perhaps with exponential backoff.

Disadvantage(s): asynchronism

Use cases such as inexpensive calculations and realtime workflows might be better suited for synchronous operations, as introducing queues can add delays and complexity.

Source(s) and further reading

Communication

Source: OSI 7 layer model

Hypertext transfer protocol (HTTP)

HTTP is a method for encoding and transporting data between a client and a server. It is a request/response protocol: clients issue requests and servers issue responses with relevant content and completion status info about the request. HTTP is self-contained, allowing requests and responses to flow through many intermediate routers and servers that perform load balancing, caching, encryption, and compression.

A basic HTTP request consists of a verb (method) and a resource (endpoint). Below are common HTTP verbs:

Verb	Description	Idempotent*	Safe	Cacheable
GET	Reads a resource	Yes	Yes	Yes
POST	Creates a resource or trigger a process that handles data	No	No	Yes if response contains freshness info
PUT	Creates or replace a resource	Yes	No	No
PATCH	Partially updates a resource	No	No	Yes if response contains freshness info
DELETE	Deletes a resource	Yes	No	No

*Can be called many times without different outcomes.

HTTP is an application layer protocol relying on lower-level protocols such as TCP and UDP.

Source(s) and further reading: HTTP

Transmission control protocol (TCP)

Source: How to make a multiplayer game

TCP is a connection-oriented protocol over an IP network. Connection is established and terminated using a handshake. All packets sent are guaranteed to reach the destination in the original order and without corruption through:

Sequence numbers and checksum fields for each packet
Acknowledgement packets and automatic retransmission

If the sender does not receive a correct response, it will resend the packets. If there are multiple timeouts, the connection is dropped. TCP also implements flow control and congestion control. These guarantees cause delays and generally result in less efficient transmission than UDP.

To ensure high throughput, web servers can keep a large number of TCP connections open, resulting in high memory usage. It can be expensive to have a large number of open connections between web server threads and say, a memcached server. Connection pooling can help in addition to switching to UDP where applicable.

TCP is useful for applications that require high reliability but are less time critical. Some examples include web servers, database info, SMTP, FTP, and SSH.

Use TCP over UDP when:

You need all of the data to arrive intact
You want to automatically make a best estimate use of the network throughput

User datagram protocol (UDP)

Source: How to make a multiplayer game

UDP is connectionless. Datagrams (analogous to packets) are guaranteed only at the datagram level. Datagrams might reach their destination out of order or not at all. UDP does not support congestion control. Without the guarantees that TCP support, UDP is generally more efficient.

UDP can broadcast, sending datagrams to all devices on the subnet. This is useful with DHCP because the client has not yet received an IP address, thus preventing a way for TCP to stream without the IP address.

UDP is less reliable but works well in real time use cases such as VoIP, video chat, streaming, and realtime multiplayer games.

Use UDP over TCP when:

You need the lowest latency
Late data is worse than loss of data
You want to implement your own error correction

Source(s) and further reading: TCP and UDP

Remote procedure call (RPC)

Source: Crack the system design interview

In an RPC, a client causes a procedure to execute on a different address space, usually a remote server. The procedure is coded as if it were a local procedure call, abstracting away the details of how to communicate with the server from the client program. Remote calls are usually slower and less reliable than local calls so it is helpful to distinguish RPC calls from local calls. Popular RPC frameworks include Protobuf, Thrift, and Avro.

RPC is a request-response protocol:

Client program - Calls the client stub procedure. The parameters are pushed onto the stack like a local procedure call.
Client stub procedure - Marshals (packs) procedure id and arguments into a request message.
Client communication module - OS sends the message from the client to the server.
Server communication module - OS passes the incoming packets to the server stub procedure.
Server stub procedure - Unmarshalls the results, calls the server procedure matching the procedure id and passes the given arguments.
The server response repeats the steps above in reverse order.

Sample RPC calls:

GET /someoperation?data=anId

POST /anotheroperation
{
  "data":"anId";
  "anotherdata": "another value"
}

RPC is focused on exposing behaviors. RPCs are often used for performance reasons with internal communications, as you can hand-craft native calls to better fit your use cases.

Choose a native library (aka SDK) when:

You know your target platform.
You want to control how your "logic" is accessed.
You want to control how error control happens off your library.
Performance and end user experience is your primary concern.

HTTP APIs following REST tend to be used more often for public APIs.

Disadvantage(s): RPC

RPC clients become tightly coupled to the service implementation.
A new API must be defined for every new operation or use case.
It can be difficult to debug RPC.
You might not be able to leverage existing technologies out of the box. For example, it might require additional effort to ensure RPC calls are properly cached on caching servers such as Squid.

Representational state transfer (REST)

REST is an architectural style enforcing a client/server model where the client acts on a set of resources managed by the server. The server provides a representation of resources and actions that can either manipulate or get a new representation of resources. All communication must be stateless and cacheable.

There are four qualities of a RESTful interface:

Identify resources (URI in HTTP) - use the same URI regardless of any operation.
Change with representations (Verbs in HTTP) - use verbs, headers, and body.
Self-descriptive error message (status response in HTTP) - Use status codes, don't reinvent the wheel.
HATEOAS (HTML interface for HTTP) - your web service should be fully accessible in a browser.

Sample REST calls:

GET /someresources/anId

PUT /someresources/anId
{"anotherdata": "another value"}

REST is focused on exposing data. It minimizes the coupling between client/server and is often used for public HTTP APIs. REST uses a more generic and uniform method of exposing resources through URIs, representation through headers, and actions through verbs such as GET, POST, PUT, DELETE, and PATCH. Being stateless, REST is great for horizontal scaling and partitioning.

Disadvantage(s): REST

With REST being focused on exposing data, it might not be a good fit if resources are not naturally organized or accessed in a simple hierarchy. For example, returning all updated records from the past hour matching a particular set of events is not easily expressed as a path. With REST, it is likely to be implemented with a combination of URI path, query parameters, and possibly the request body.
REST typically relies on a few verbs (GET, POST, PUT, DELETE, and PATCH) which sometimes doesn't fit your use case. For example, moving expired documents to the archive folder might not cleanly fit within these verbs.
Fetching complicated resources with nested hierarchies requires multiple round trips between the client and server to render single views, e.g. fetching content of a blog entry and the comments on that entry. For mobile applications operating in variable network conditions, these multiple roundtrips are highly undesirable.
Over time, more fields might be added to an API response and older clients will receive all new data fields, even those that they do not need, as a result, it bloats the payload size and leads to larger latencies.

RPC and REST calls comparison

Operation	RPC	REST
Signup	POST /signup	POST /persons
Resign	POST /resign { "personid": "1234" }	DELETE /persons/1234
Read a person	GET /readPerson?personid=1234	GET /persons/1234
Read a person’s items list	GET /readUsersItemsList?personid=1234	GET /persons/1234/items
Add an item to a person’s items	POST /addItemToUsersItemsList { "personid": "1234"; "itemid": "456" }	POST /persons/1234/items { "itemid": "456" }
Update an item	POST /modifyItem { "itemid": "456"; "key": "value" }	PUT /items/456 { "key": "value" }
Delete an item	POST /removeItem { "itemid": "456" }	DELETE /items/456

Source: Do you really know why you prefer REST over RPC

Source(s) and further reading: REST and RPC

Security

This section could use some updates. Consider contributing!

Security is a broad topic. Unless you have considerable experience, a security background, or are applying for a position that requires knowledge of security, you probably won't need to know more than the basics:

Encrypt in transit and at rest.
Sanitize all user inputs or any input parameters exposed to user to prevent XSS and SQL injection.
Use parameterized queries to prevent SQL injection.
Use the principle of least privilege.

Source(s) and further reading

Appendix

You'll sometimes be asked to do 'back-of-the-envelope' estimates. For example, you might need to determine how long it will take to generate 100 image thumbnails from disk or how much memory a data structure will take. The Powers of two table and Latency numbers every programmer should know are handy references.

Powers of two table

Power           Exact Value         Approx Value        Bytes
---------------------------------------------------------------
7                             128
8                             256
10                           1024   1 thousand           1 KB
16                         65,536                       64 KB
20                      1,048,576   1 million            1 MB
30                  1,073,741,824   1 billion            1 GB
32                  4,294,967,296                        4 GB
40              1,099,511,627,776   1 trillion           1 TB

Source(s) and further reading

Powers of two

Latency numbers every programmer should know

Latency Comparison Numbers
--------------------------
L1 cache reference                           0.5 ns
Branch mispredict                            5   ns
L2 cache reference                           7   ns                      14x L1 cache
Mutex lock/unlock                           25   ns
Main memory reference                      100   ns                      20x L2 cache, 200x L1 cache
Compress 1K bytes with Zippy            10,000   ns       10 us
Send 1 KB bytes over 1 Gbps network     10,000   ns       10 us
Read 4 KB randomly from SSD*           150,000   ns      150 us          ~1GB/sec SSD
Read 1 MB sequentially from memory     250,000   ns      250 us
Round trip within same datacenter      500,000   ns      500 us
Read 1 MB sequentially from SSD*     1,000,000   ns    1,000 us    1 ms  ~1GB/sec SSD, 4X memory
HDD seek                            10,000,000   ns   10,000 us   10 ms  20x datacenter roundtrip
Read 1 MB sequentially from 1 Gbps  10,000,000   ns   10,000 us   10 ms  40x memory, 10X SSD
Read 1 MB sequentially from HDD     30,000,000   ns   30,000 us   30 ms 120x memory, 30X SSD
Send packet CA->Netherlands->CA    150,000,000   ns  150,000 us  150 ms

Notes
-----
1 ns = 10^-9 seconds
1 us = 10^-6 seconds = 1,000 ns
1 ms = 10^-3 seconds = 1,000 us = 1,000,000 ns

Handy metrics based on numbers above:

Read sequentially from HDD at 30 MB/s
Read sequentially from 1 Gbps Ethernet at 100 MB/s
Read sequentially from SSD at 1 GB/s
Read sequentially from main memory at 4 GB/s
6-7 world-wide round trips per second
2,000 round trips per second within a data center

Latency numbers visualized

Source(s) and further reading

Additional system design interview questions

Common system design interview questions, with links to resources on how to solve each.

Question	Reference(s)
Design a file sync service like Dropbox	youtube.com
Design a search engine like Google	queue.acm.org stackexchange.com ardendertat.com stanford.edu
Design a scalable web crawler like Google	quora.com
Design Google docs	code.google.com neil.fraser.name
Design a key-value store like Redis	codecapsule.com allthingsdistributed.com
Design a cache system like Memcached	slideshare.net
Design a recommendation system like Amazon's	hulu.com ijcai13.org
Design a tinyurl system like Bitly	n00tc0d3r.blogspot.com
Design a chat app like WhatsApp	highscalability.com
Design a picture sharing system like Instagram	highscalability.com highscalability.com
Design the Facebook news feed function	quora.com quora.com slideshare.net
Design the Facebook timeline function	facebook.com highscalability.com
Design the Facebook chat function	erlang-factory.com facebook.com
Design a graph search function like Facebook's	facebook.com facebook.com facebook.com
Design a content delivery network like CloudFlare	figshare.com
Design a trending topic system like Twitter's	michael-noll.com snikolov .wordpress.com
Design a random ID generation system	blog.twitter.com github.com
Return the top k requests during a time interval	cs.ucsb.edu wpi.edu
Design a system that serves data from multiple data centers	highscalability.com
Design an online multiplayer card game	indieflashblog.com buildnewgames.com
Design a garbage collection system	stuffwithstuff.com washington.edu
Design an API rate limiter	https://stripe.com/blog/
Design a Stock Exchange (like NASDAQ or Binance)	Jane Street Golang Implementation Go Implementation
Add a system design question	Contribute

Real world architectures

Articles on how real world systems are designed.

Source: Twitter timelines at scale

Don't focus on nitty gritty details for the following articles, instead:

Identify shared principles, common technologies, and patterns within these articles
Study what problems are solved by each component, where it works, where it doesn't
Review the lessons learned

Type	System	Reference(s)
Data processing	MapReduce - Distributed data processing from Google	research.google.com
Data processing	Spark - Distributed data processing from Databricks	slideshare.net
Data processing	Storm - Distributed data processing from Twitter	slideshare.net

Data store	Bigtable - Distributed column-oriented database from Google	harvard.edu
Data store	HBase - Open source implementation of Bigtable	slideshare.net
Data store	Cassandra - Distributed column-oriented database from Facebook	slideshare.net
Data store	DynamoDB - Document-oriented database from Amazon	harvard.edu
Data store	MongoDB - Document-oriented database	slideshare.net
Data store	Spanner - Globally-distributed database from Google	research.google.com
Data store	Memcached - Distributed memory caching system	slideshare.net
Data store	Redis - Distributed memory caching system with persistence and value types	slideshare.net

File system	Google File System (GFS) - Distributed file system	research.google.com
File system	Hadoop File System (HDFS) - Open source implementation of GFS	apache.org

Misc	Chubby - Lock service for loosely-coupled distributed systems from Google	research.google.com
Misc	Dapper - Distributed systems tracing infrastructure	research.google.com
Misc	Kafka - Pub/sub message queue from LinkedIn	slideshare.net
Misc	Zookeeper - Centralized infrastructure and services enabling synchronization	slideshare.net
	Add an architecture	Contribute

Company architectures

Company	Reference(s)
Amazon	Amazon architecture
Cinchcast	Producing 1,500 hours of audio every day
DataSift	Realtime datamining At 120,000 tweets per second
Dropbox	How we've scaled Dropbox
ESPN	Operating At 100,000 duh nuh nuhs per second
Google	Google architecture
Instagram	14 million users, terabytes of photos What powers Instagram
Justin.tv	Justin.Tv's live video broadcasting architecture
Facebook	Scaling memcached at Facebook TAO: Facebook’s distributed data store for the social graph Facebook’s photo storage How Facebook Live Streams To 800,000 Simultaneous Viewers
Flickr	Flickr architecture
Mailbox	From 0 to one million users in 6 weeks
Netflix	A 360 Degree View Of The Entire Netflix Stack Netflix: What Happens When You Press Play?
Pinterest	From 0 To 10s of billions of page views a month 18 million visitors, 10x growth, 12 employees
Playfish	50 million monthly users and growing
PlentyOfFish	PlentyOfFish architecture
Salesforce	How they handle 1.3 billion transactions a day
Stack Overflow	Stack Overflow architecture
TripAdvisor	40M visitors, 200M dynamic page views, 30TB data
Tumblr	15 billion page views a month
Twitter	Making Twitter 10000 percent faster Storing 250 million tweets a day using MySQL 150M active users, 300K QPS, a 22 MB/S firehose Timelines at scale Big and small data at Twitter Operations at Twitter: scaling beyond 100 million users How Twitter Handles 3,000 Images Per Second
Uber	How Uber scales their real-time market platform Lessons Learned From Scaling Uber To 2000 Engineers, 1000 Services, And 8000 Git Repositories
WhatsApp	The WhatsApp architecture Facebook bought for $19 billion
YouTube	YouTube scalability YouTube architecture

Company engineering blogs

Architectures for companies you are interviewing with.

Questions you encounter might be from the same domain.

Source(s) and further reading

Looking to add a blog? To avoid duplicating work, consider adding your company blog to the following repo:

kilimchoi/engineering-blogs

Under development

Interested in adding a section or helping complete one in-progress? Contribute!

Distributed computing with MapReduce
Consistent hashing
Scatter gather
Contribute

Credits

Credits and sources are provided throughout this repo.

Special thanks to:

Contact info

Feel free to contact me to discuss any issues, questions, or comments.

My contact info can be found on my GitHub page.

License

I am providing code and resources in this repository to you under an open source license. Because this is my personal repository, the license you receive to my code and resources is from me and not my employer (Facebook).

Copyright 2017 Donne Martin

Creative Commons Attribution 4.0 International License (CC BY 4.0)

http://creativecommons.org/licenses/by/4.0/