GitHub C Weekly Trending

Genymobile/scrcpy

Display and control your Android device

This GitHub repo (https://github.com/Genymobile/scrcpy) is the only official source for the project. Do not download releases from random websites, even if their name contains scrcpy.

scrcpy (v4.0)

pronounced "screen copy"

This application mirrors Android devices (video and audio) connected via USB or TCP/IP and allows control using the computer's keyboard and mouse. It does not require root access or an app installed on the device. It works on Linux, Windows, and macOS.

It focuses on:

lightness: native, displays only the device screen
performance: 30~120fps, depending on the device
quality: 1920×1080 or above
low latency: 35~70ms
low startup time: ~1 second to display the first image
non-intrusiveness: nothing is left installed on the Android device
user benefits: no account, no ads, no internet required
freedom: free and open source software

Its features include:

audio forwarding (Android 11+)
recording
virtual display
mirroring with Android device screen off
copy-paste in both directions
configurable quality
camera mirroring (Android 12+)
mirroring as a webcam (V4L2) (Linux-only)
physical keyboard and mouse simulation (HID)
gamepad support
OTG mode
and more…

Prerequisites

The Android device requires at least API 21 (Android 5.0).

Audio forwarding is supported for API >= 30 (Android 11+).

Make sure you enabled USB debugging on your device(s).

On some devices (especially Xiaomi), you might get the following error:

Injecting input events requires the caller (or the source of the instrumentation, if any) to have the INJECT_EVENTS permission.

In that case, you need to enable an additional option USB debugging (Security Settings) (this is an item different from USB debugging) to control it using a keyboard and mouse. Rebooting the device is necessary once this option is set.

Note that USB debugging is not required to run scrcpy in OTG mode.

Get the app

Must-know tips

Reducing resolution may greatly improve performance (scrcpy -m1024)
Right-click triggers BACK
Middle-click triggers HOME
Alt+f toggles fullscreen
There are many other shortcuts

Usage examples

There are a lot of options, documented in separate pages. Here are just some common examples.

Capture the screen in H.265 (better quality), limit the size to 1920, limit the frame rate to 60fps, disable audio, and control the device by simulating a physical keyboard:
```
scrcpy --video-codec=h265 --max-size=1920 --max-fps=60 --no-audio --keyboard=uhid
scrcpy --video-codec=h265 -m1920 --max-fps=60 --no-audio -K  # short version
```
Start VLC in a new virtual display (separate from the device display):
```
scrcpy --new-display=1920x1080 --start-app=org.videolan.vlc
```
Start VLC in a new flex display using H.265 with a bitrate of 16 Mbps, while keeping the display active so it does not turn off:
```
scrcpy --new-display -x --keep-active --start-app=org.videolan.vlc --video-codec=h265 -b16M
```

Record the device camera in H.265 at 1920x1080 (and microphone) to an MP4 file:

scrcpy --video-source=camera --video-codec=h265 --camera-size=1920x1080 --record=file.mp4

Capture the device front camera and expose it as a webcam on the computer (on Linux):

scrcpy --video-source=camera --camera-size=1920x1080 --camera-facing=front --v4l2-sink=/dev/video2 --no-playback

Control the device without mirroring by simulating a physical keyboard and mouse (USB debugging not required):
```
scrcpy --otg
```
Control the device using gamepads plugged into the computer:
```
scrcpy --gamepad=uhid
scrcpy -G  # short version
```

User documentation

The application provides a lot of features and configuration options. They are documented in the following pages:

Resources

FAQ
Translations (not necessarily up to date)
Build instructions
Developers
Verify release signatures

Articles

Contact

You can open an issue for bug reports, feature requests or general questions.

For bug reports, please read the FAQ first, you might find a solution to your problem immediately.

You can also use:

Reddit: r/scrcpy
BlueSky: @scrcpy.bsky.social
Twitter: @scrcpy_app

Donate

I'm @rom1v, the author and maintainer of scrcpy.

If you appreciate this application, you can support my open source work:

License

Copyright (C) 2018 Genymobile
Copyright (C) 2018-2026 Romain Vimont

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

cactus-compute/cactus

Low-latency AI engine for mobile devices & wearables

Cactus

A low-latency AI engine for mobile devices & wearables. Main features:

Fast: fastest inference on ARM CPU
Low RAM: zero-copy memory mapping ensures 10x lower RAM use than other engines
Multimodal: one SDK for speech, vision, and language models
Cloud fallback: automatically route requests to cloud models if needed
Energy-efficient: NPU-accelerated prefill

┌─────────────────┐
│  Cactus Engine  │ ←── OpenAI-compatible APIs for all major languages
└─────────────────┘     Chat, vision, STT, RAG, tool call, cloud handoff
         │
┌─────────────────┐
│  Cactus Graph   │ ←── Zero-copy computation graph (PyTorch for mobile)
└─────────────────┘     Custom models, optimised for RAM & quantisation
         │
┌─────────────────┐
│ Cactus Kernels  │ ←── ARM SIMD kernels (Apple, Snapdragon, Exynos, etc)
└─────────────────┘     Custom attention, KV-cache quant, chunked prefill

Quick Demo (Mac)

Step 1: brew install cactus-compute/cactus/cactus
Step 2: cactus transcribe or cactus run

Cactus Engine

#include "cactus.h"

cactus_model_t model = cactus_init(
    "path/to/weight/folder",
    "path to txt or dir of txts for auto-rag",
    false
);

const char* messages = R"([
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "My name is Henry Ndubuaku"}
])";

const char* options = R"({
    "max_tokens": 50,
    "stop_sequences": ["<|im_end|>"]
})";

char response[4096];
int result = cactus_complete(
    model,            // model handle
    messages,         // JSON chat messages
    response,         // response buffer
    sizeof(response), // buffer size
    options,          // generation options
    nullptr,          // tools JSON
    nullptr,          // streaming callback
    nullptr,          // user data
    nullptr,          // pcm audio buffer
    0                 // pcm buffer size
);

Example response from Gemma3-270m

{
    "success": true,        // generation succeeded
    "error": null,          // error details if failed
    "cloud_handoff": false, // true if cloud model used
    "response": "Hi there!",
    "function_calls": [],   // parsed tool calls
    "confidence": 0.8193,   // model confidence
    "time_to_first_token_ms": 45.23,
    "total_time_ms": 163.67,
    "prefill_tps": 1621.89,
    "decode_tps": 168.42,
    "ram_usage_mb": 245.67,
    "prefill_tokens": 28,
    "decode_tokens": 50,
    "total_tokens": 78
}

Cactus Graph

#include "cactus.h"

CactusGraph graph;
auto a = graph.input({2, 3}, Precision::FP16);
auto b = graph.input({3, 4}, Precision::INT8);

auto x1 = graph.matmul(a, b, false);
auto x2 = graph.transpose(x1);
auto result = graph.matmul(b, x2, true);

float a_data[6] = {1.1f, 2.3f, 3.4f, 4.2f, 5.7f, 6.8f};
float b_data[12] = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12};

graph.set_input(a, a_data, Precision::FP16);
graph.set_input(b, b_data, Precision::INT8);

graph.execute();
void* output_data = graph.get_output(result);

graph.hard_reset();

API & SDK References

Reference	Language	Description
Engine API	C	Chat completion, streaming, tool calling, transcription, embeddings, RAG, vision, VAD, vector index, cloud handoff
Graph API	C++	Tensor operations, matrix multiplication, attention, normalization, activation functions
Python SDK	Python	Mac, Linux
Swift SDK	Swift	iOS, macOS, tvOS, watchOS, Android
Kotlin SDK	Kotlin	Android, iOS (via KMP)
Flutter SDK	Dart	iOS, macOS, Android
Rust SDK	Rust	Mac, Linux
React Native	JavaScript	iOS, Android

Model weights: Pre-converted weights for all supported models at huggingface.co/Cactus-Compute.

Benchmarks (CPU-only, no GPU)

All weights INT4 quantised
LFM: 1k-prefill / 100-decode, values are prefill tps / decode tps
LFM-VL: 256px input, values are latency / decode tps
Parakeet: 20s audio input, values are latency / decode tps
Missing latency = no NPU support yet

Device	LFM 1.2B	LFMVL 1.6B	Parakeet 1.1B	RAM
Mac M4 Pro	582/100	0.2s/98	0.1s/900k+	76MB
iPad/Mac M3	350/60	0.3s/69	0.3s/800k+	70MB
iPhone 17 Pro	327/48	0.3s/48	0.3s/300k+	108MB
iPhone 13 Mini	148/34	0.3s/35	0.7s/90k+	1GB
Galaxy S25 Ultra	255/37	-/34	-/250k+	1.5GB
Pixel 6a	70/15	-/15	-/17k+	1GB
Galaxy A17 5G	32/10	-/11	-/40k+	727MB
CMF Phone 2 Pro	-	-	-	-
Raspberry Pi 5	69/11	13.3s/11	4.5s/180k+	869MB

Supported Transcription Model

STT: 20s audio input on Macbook Air M3 chip
Benchmark dataset: internal evals with production users

Model	Params	End2End ms	Latency ms	Decode toks/sec	NPU	RTF	WER
UsefulSensors/moonshine-base	61M	361.35	182	262	yes	0.0180	0.1395
openai/whisper-tiny	39M	232.03	137.38	581	yes	0.0116	0.1860
openai/whisper-base	74M	329.37	178.65	358	yes	0.0164	0.1628
openai/whisper-small	244M	856.79	332.63	108	yes	0.0428	0.0930
openai/whisper-medium	769M	2085.87	923.33	49	yes	0.1041	0.0930
openai/whisper-large-v3	1.55B	5994	2050	15.72	no	0.2992	-
nvidia/parakeet-ctc-0.6b	600M	201.77	201.44	5214285	yes	0.0101	0.0930
nvidia/parakeet-tdt-0.6b-v3	600M	718.91	718.82	3583333	yes	0.0359	0.0465
nvidia/parakeet-ctc-1.1b	1.1B	279.03	278.92	4562500	yes	0.0139	0.1628
snakers4/silero-vad	-	-	-	-	-	-	-
pyannote/segmentation-3.0	-	-	-	-	-	-	-
pyannote/wespeaker-voxceleb-resnet34-LM	-	-	-	-	-	-	-

Supported LLMs

Gemma weights are often gated on HuggingFace, needs tokens
Run huggingface-cli login and input your huggingface token

Model	Features
google/gemma-3-270m-it	completion
google/functiongemma-270m-it	tools
google/gemma-3-1b-it	completion, gated
google/gemma-4-E2B-it	completion, tools, embed, vision, speech
google/gemma-3n-E2B-it	completion, tools
google/gemma-4-E4B-it	completion, tools, embed, vision, speech
google/gemma-3n-E4B-it	completion, tools
google/gemma-4-E2B-it	vision, audio, completion, tools, Apple NPU
google/gemma-4-E4B-it	vision, audio, completion, tools, Apple NPU
Qwen/Qwen3-0.6B	completion, tools, embed
Qwen/Qwen3-Embedding-0.6B	embed
Qwen/Qwen3.5-0.8B	vision, completion, tools, embed
Qwen/Qwen3-1.7B	completion, tools, embed
Qwen/Qwen3.5-2B	vision, completion, tools, embed
LiquidAI/LFM2.5-350M	completion, tools, embed
LiquidAI/LFM2-700M	completion, tools, embed
LiquidAI/LFM2-8B-A1B	completion, tools, embed
LiquidAI/LFM2.5-1.2B-Thinking	completion, tools, embed
LiquidAI/LFM2.5-1.2B-Instruct	completion, tools, embed
LiquidAI/LFM2-2.6B	completion, tools, embed
LiquidAI/LFM2-VL-450M	vision, txt & img embed, Apple NPU
LiquidAI/LFM2.5-VL-450M	vision, txt & img embed, Apple NPU
LiquidAI/LFM2.5-VL-1.6B	vision, txt & img embed, Apple NPU
tencent/Youtu-LLM-2B	completion, tools, embed
nomic-ai/nomic-embed-text-v2-moe	embed

Roadmap

Date	Status	Milestone
Sep 2025	Done	Released v1
Oct 2025	Done	Chunked prefill, KVCache Quant (2x prefill)
Nov 2025	Done	Cactus Attention (10 & 1k prefill = same decode)
Dec 2025	Done	Team grows to +6 Research Engineers
Jan 2026	Done	Apple NPU/RAM, 5-11x faster iOS/Mac
Feb 2026	Done	Hybrid inference, INT4, lossless Quant (1.5x)
Mar 2026	Coming	Qualcomm/Google NPUs, 5-11x faster Android
Apr 2026	Coming	Mediatek/Exynos NPUs, Cactus@ICLR
May 2026	Coming	Kernel→C++, Graph/Engine→Rust, Mac GPU & VR
Jun 2026	Coming	Torch/JAX model transpilers
Jul 2026	Coming	Wearables optimisations, Cactus@ICML
Aug 2026	Coming	Orchestration
Sep 2026	Coming	Full Cactus paper, chip manufacturer partners

Using this repo

┌──────────────────────────────────────────────────────────────────────────────┐
│                                                                              │
│ Step 0: if on Linux (Ubuntu/Debian)                                          │
│ sudo apt-get install python3 python3-venv python3-pip cmake                  │
│   build-essential libcurl4-openssl-dev                                       │
│                                                                              │
│ Step 1: clone and setup                                                      │
│ git clone https://github.com/cactus-compute/cactus && cd cactus              │
│ source ./setup                                                               │
│                                                                              │
│ Step 2: use the commands                                                     │
│──────────────────────────────────────────────────────────────────────────────│
│                                                                              │
│  cactus auth                         manage Cloud API key                    │
│    --status                          show key status                         │
│    --clear                           remove saved key                        │
│                                                                              │
│  cactus run <model>                  opens playground (auto downloads)       │
│    --precision INT4|INT8|FP16        quantization (default: INT4)            │
│    --token <token>                   HF token (gated models)                 │
│    --reconvert                       force reconversion from source          │
│                                                                              │
│  cactus transcribe [model]           live mic transcription (parakeet-tdt-0.6b-v3) │
│    --file <audio.wav>                transcribe file instead of mic          │
│    --precision INT4|INT8|FP16        quantization (default: INT4)            │
│    --token <token>                   HF token (gated models)                 │
│    --reconvert                       force reconversion from source          │
│                                                                              │
│  cactus download <model>             downloads model to ./weights            │
│    --precision INT4|INT8|FP16        quantization (default: INT4)            │
│    --token <token>                   HuggingFace API token                   │
│    --reconvert                       force reconversion from source          │
│                                                                              │
│  cactus convert <model> [dir]        convert model, supports LoRA merge      │
│    --precision INT4|INT8|FP16        quantization (default: INT4)            │
│    --lora <path>                     LoRA adapter to merge                   │
│    --token <token>                   HuggingFace API token                   │
│                                                                              │
│  cactus build                        build for ARM → build/libcactus.a       │
│    --apple                           Apple (iOS/macOS)                       │
│    --android                         Android                                 │
│    --flutter                         Flutter (all platforms)                 │
│    --python                          shared lib for Python FFI               │
│                                                                              │
│  cactus test                         run unit tests and benchmarks           │
│    --model <model>                   default: LFM2-VL-450M                   │
│    --transcribe_model <model>        default: moonshine-base                 │
│    --benchmark                       use larger models                       │
│    --precision INT4|INT8|FP16        regenerate weights with precision       │
│    --reconvert                       force reconversion from source          │
│    --no-rebuild                      skip building library                   │
│    --llm / --stt / --performance     run specific test suite                 │
│    --ios                             run on connected iPhone                 │
│    --android                         run on connected Android                │
│                                                                              │
│  cactus clean                        remove all build artifacts              │
│  cactus --help                       show all commands and flags             │
│                                                                              │
└──────────────────────────────────────────────────────────────────────────────┘

Maintaining Organisations

Citation

If you use Cactus in your research, please cite it as follows:

@software{cactus,
  title        = {Cactus: AI Inference Engine for Phones & Wearables},
  author       = {Ndubuaku, Henry and Cactus Team},
  url          = {https://github.com/cactus-compute/cactus},
  year         = {2025}
}

N/B: Scroll all the way up and click the shields link for resources!

mit-pdos/xv6-riscv

Xv6 for RISC-V

xv6 is a re-implementation of Dennis Ritchie's and Ken Thompson's Unix Version 6 (v6). xv6 loosely follows the structure and style of v6, but is implemented for a modern RISC-V multiprocessor using ANSI C.

ACKNOWLEDGMENTS

xv6 is inspired by John Lions's Commentary on UNIX 6th Edition (Peer to Peer Communications; ISBN: 1-57398-013-7; 1st edition (June 14, 2000)). See also https://pdos.csail.mit.edu/6.1810/, which provides pointers to on-line resources for v6.

The following people have made contributions: Russ Cox (context switching, locking), Cliff Frey (MP), Xiao Yu (MP), Nickolai Zeldovich, and Austin Clements.

We are also grateful for the bug reports and patches contributed by Abhinavpatel00, Takahiro Aoyagi, Marcelo Arroyo, Hirbod Behnam, Silas Boyd-Wickizer, Anton Burtsev, carlclone, Ian Chen, clivezeng, Dan Cross, Cody Cutler, Mike CAT, Tej Chajed, Asami Doi,Wenyang Duan, echtwerner, eyalz800, Nelson Elhage, Saar Ettinger, Alice Ferrazzi, Nathaniel Filardo, flespark, Peter Froehlich, Yakir Goaron, Shivam Handa, Matt Harvey, Bryan Henry, jaichenhengjie, Jim Huang, Matúš Jókay, John Jolly, Alexander Kapshuk, Anders Kaseorg, kehao95, Wolfgang Keller, Jungwoo Kim, Jonathan Kimmitt, Eddie Kohler, Vadim Kolontsov, Austin Liew, l0stman, Pavan Maddamsetti, Imbar Marinescu, Yandong Mao, Matan Shabtay, Hitoshi Mitake, Carmi Merimovich, mes900903, Mark Morrissey, mtasm, Joel Nider, Hayato Ohhashi, OptimisticSide, papparapa, phosphagos, Harry Porter, Greg Price, Zheng qhuo, Quancheng, RayAndrew, Jude Rich, segfault, Ayan Shafqat, Eldar Sehayek, Yongming Shen, Fumiya Shigemitsu, snoire, Taojie, Cam Tenny, tyfkda, Warren Toomey, Stephen Tu, Alissa Tung, Rafael Ubal, unicornx, Amane Uehara, Pablo Ventura, Luc Videau, Xi Wang, WaheedHafez, Keiichi Watanabe, Lucas Wolf, Nicolas Wolovick, wxdao, Grant Wu, x653, Andy Zhang, Jindong Zhang, Icenowy Zheng, ZhUyU1997, and Zou Chang Wei.

ERROR REPORTS

Please send errors and suggestions to Frans Kaashoek and Robert Morris (kaashoek,rtm@mit.edu). The main purpose of xv6 is as a teaching operating system for MIT's 6.1810, so we are more interested in simplifications and clarifications than new features.

BUILDING AND RUNNING XV6

You will need a RISC-V "newlib" tool chain from https://github.com/riscv/riscv-gnu-toolchain, and qemu compiled for riscv64-softmmu. Once they are installed, and in your shell search path, you can run "make qemu".

nginx/nginx

The official NGINX Open Source repository.

NGINX (pronounced "engine x" or "en-jin-eks") is the world's most popular Web Server, high performance Load Balancer, Reverse Proxy, API Gateway and Content Cache.

NGINX is free and open source software, distributed under the terms of a simplified 2-clause BSD-like license.

Enterprise distributions, commercial support and training are available from F5, Inc.

Important

The goal of this README is to provide a basic, structured introduction to NGINX for novice users. Please refer to the full NGINX documentation for detailed information on installing, building, configuring, debugging, and more. These documentation pages also contain a more detailed Beginners Guide, How-Tos, Development guide, and a complete module and directive reference.

How it works

NGINX is installed software with binary packages available for all major operating systems and Linux distributions. See Tested OS and Platforms for a full list of compatible systems.

Important

While nearly all popular Linux-based operating systems are distributed with a community version of nginx, we highly advise installation and usage of official packages or sources from this repository. Doing so ensures that you're using the most recent release or source code, including the latest feature-set, fixes and security patches.

Modules

NGINX is comprised of individual modules, each extending core functionality by providing additional, configurable features. See "Modules reference" at the bottom of nginx documentation for a complete list of official modules.

NGINX modules can be built and distributed as static or dynamic modules. Static modules are defined at build-time, compiled, and distributed in the resulting binaries. See Dynamic Modules for more information on how they work, as well as, how to obtain, install, and configure them.

Tip

You can issue the following command to see which static modules your NGINX binaries were built with:

nginx -V

See Configuring the build for information on how to include specific Static modules into your nginx build.

Configurations

NGINX is highly flexible and configurable. Provisioning the software is achieved via text-based config file(s) accepting parameters called "Directives". See Configuration File's Structure for a comprehensive description of how NGINX configuration files work.

Note

The set of directives available to your distribution of NGINX is dependent on which modules have been made available to it.

Runtime

Rather than running in a single, monolithic process, NGINX is architected to scale beyond Operating System process limitations by operating as a collection of processes. They include:

A "master" process that maintains worker processes, as well as, reads and evaluates configuration files.
One or more "worker" processes that process data (eg. HTTP requests).

The number of worker processes is defined in the configuration file and may be fixed for a given configuration or automatically adjusted to the number of available CPU cores. In most cases, the latter option optimally balances load across available system resources, as NGINX is designed to efficiently distribute work across all worker processes.

Tip

Processes synchronize data through shared memory. For this reason, many NGINX directives require the allocation of shared memory zones. As an example, when configuring rate limiting, connecting clients may need to be tracked in a common memory zone so all worker processes can know how many times a particular client has accessed the server in a span of time.

Downloading and installing

Follow these steps to download and install precompiled NGINX binaries. You may also choose to build NGINX locally from source code.

Stable and Mainline binaries

NGINX binaries are built and distributed in two versions: stable and mainline. Stable binaries are built from stable branches and only contain critical fixes backported from the mainline version. Mainline binaries are built from the master branch and contain the latest features and bugfixes. You'll need to decide which is appropriate for your purposes.

Linux binary installation process

The NGINX binary installation process takes advantage of package managers native to specific Linux distributions. For this reason, first-time installations involve adding the official NGINX package repository to your system's package manager. Follow these steps to download, verify, and install NGINX binaries using the package manager appropriate for your Linux distribution.

Upgrades

Future upgrades to the latest version can be managed using the same package manager without the need to manually download and verify binaries.

FreeBSD installation process

For more information on installing NGINX on FreeBSD system, visit https://nginx.org/en/docs/install.html

Windows executables

Windows executables for mainline and stable releases can be found on the main NGINX download page. Note that the current implementation of NGINX for Windows is at the Proof-of-Concept stage and should only be used for development and testing purposes. For additional information, please see nginx for Windows.

Dynamic modules

NGINX version 1.9.11 added support for Dynamic Modules. Unlike Static modules, dynamically built modules can be downloaded, installed, and configured after the core NGINX binaries have been built. Official dynamic module binaries are available from the same package repository as the core NGINX binaries described in previous steps.

Tip

NGINX JavaScript (njs), is a popular NGINX dynamic module that enables the extension of core NGINX functionality using familiar JavaScript syntax.

Important

If desired, dynamic modules can also be built statically into NGINX at compile time.

Getting started with NGINX

For a gentle introduction to NGINX basics, please see our Beginner’s Guide.

Installing SSL certificates and enabling TLS encryption

See Configuring HTTPS servers for a quick guide on how to enable secure traffic to your NGINX installation.

Load Balancing

For a quick start guide on configuring NGINX as a Load Balancer, please see Using nginx as HTTP load balancer.

Rate limiting

See our Rate Limiting with NGINX blog post for an overview of core concepts for provisioning NGINX as an API Gateway.

Content caching

See A Guide to Caching with NGINX and NGINX Plus blog post for an overview of how to use NGINX as a content cache (e.g. edge server of a content delivery network).

Building from source

The following steps can be used to build NGINX from source code available in this repository.

Installing dependencies

Most Linux distributions will require several dependencies to be installed in order to build NGINX. The following instructions are specific to the apt package manager, widely available on most Ubuntu/Debian distributions and their derivatives.

Tip

It is always a good idea to update your package repository lists prior to installing new packages.

sudo apt update

Installing compiler and make utility

Use the following command to install the GNU C compiler and Make utility.

sudo apt install gcc make

Installing dependency libraries

sudo apt install libpcre3-dev zlib1g-dev

Warning

This is the minimal set of dependency libraries needed to build NGINX with rewriting and gzip capabilities. Other dependencies may be required if you choose to build NGINX with additional modules. Monitor the output of the configure command discussed in the following sections for information on which modules may be missing. For example, if you plan to use SSL certificates to encrypt traffic with TLS, you'll need to install the OpenSSL library. To do so, issue the following command.

sudo apt install libssl-dev

Cloning the NGINX GitHub repository

Using your preferred method, clone the NGINX repository into your development directory. See Cloning a GitHub Repository for additional help.

git clone https://github.com/nginx/nginx.git

Configuring the build

Prior to building NGINX, you must run the configure script with appropriate flags. This will generate a Makefile in your NGINX source root directory that can then be used to compile NGINX with options specified during configuration.

From the NGINX source code repository's root directory:

auto/configure

Important

Configuring the build without any flags will compile NGINX with the default set of options. Please refer to https://nginx.org/en/docs/configure.html for a full list of available build configuration options.

Compiling

The configure script will generate a Makefile in the NGINX source root directory upon successful execution. To compile NGINX into a binary, issue the following command from that same directory:

make

Location of binary and installation

After successful compilation, a binary will be generated at <NGINX_SRC_ROOT_DIR>/objs/nginx. To install this binary, issue the following command from the source root directory:

sudo make install

Important

The binary will be installed into the /usr/local/nginx/ directory.

Running and testing the installed binary

To run the installed binary, issue the following command:

sudo /usr/local/nginx/sbin/nginx

You may test NGINX operation using curl.

curl localhost

The output of which should start with:

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>

Asking questions and reporting issues

See our Support guidelines for information on how discuss the codebase, ask troubleshooting questions, and report issues.

Contributing code

Please see the Contributing guide for information on how to contribute code.

Additional help and resources

See the NGINX Community Blog for more tips, tricks and HOW-TOs related to NGINX and related projects.
Access nginx.org, your go-to source for all documentation, information and software related to the NGINX suite of projects.

Changelog

See our changelog to keep track of updates.

License

2-clause BSD-like license

Additional documentation available at: https://nginx.org/en/docs

DavidXanatos/TaskExplorer

Power full Task Manager

TaskExplorer

Task Explorer is a powerful task management tool designed not only to monitor running applications but to provide deep insight into what those applications are doing. Its user interface prioritizes speed and efficiency, delivering real-time data on processes with minimal interaction. Instead of requiring multiple windows or sub-windows, Task Explorer displays relevant information in accessible panels. When selecting a process, detailed information is displayed in the lower half of the screen, allowing you to navigate through the data seamlessly using the arrow keys. The dynamic data refresh allows users to observe changes in real-time, offering additional clarity and insight into system performance and behavior.

Features

Task Explorer offers an array of advanced features to provide comprehensive visibility into the system. The Thread Panel displays a stack trace for the selected thread, offering immediate insights into the current actions of an application, which is particularly useful for diagnosing deadlocks or performance bottlenecks. The Memory Panel allows users to view and edit process memory, featuring an advanced memory editor with string search capabilities. In the Handles Panel, all open handles are displayed, including essential details such as file names, current file positions, and sizes, giving a clear view of the disk operations a program is performing.

The Socket Panel provides visibility into all open connections or sockets for each process, with additional data rate information. It also has the option to show pseudo UDP connections based on ETW data, allowing users to monitor network communications effectively. The Modules Panel lists all loaded DLLs and memory-mapped files, with the ability to unload or inject DLLs as needed. Additionally, the application includes a variety of other useful panels, including Token, Environment, Windows, GDI, and .NET panels.

By double-clicking a process, you can open the Task Info Panels in a separate window, enabling the simultaneous inspection of multiple processes. The system monitoring capabilities are robust as well, featuring toolbar graphs that show real-time usage of system resources such as CPU, handles, network traffic, and disk access. The System Info Panels display all open files and sockets and allow users to control system services, including drivers. Dedicated performance panels for CPU, Memory, Disk I/O, Network, and GPU resources offer detailed graphs, making it easy to monitor and optimize system performance.

For users who need more screen space, the System Info Panel can be fully collapsed or opened in a separate window, maximizing the available area for the task panels.

Screen Shots

System Requirements

Task Explorer is compatible with Windows 7 or higher, on both 32-bit and 64-bit systems.

Additional Information

Task Explorer is built using the Qt Framework, ensuring a cross-platform user interface with plans to eventually port the tool to Linux, which could make it one of the first advanced, GUI-based task managers for the platform. On Windows, Task Explorer leverages the Process Hacker library and uses a custom-compiled version of the systeminformer.sys driver from the SystemInformer project, ensuring robust performance and system monitoring capabilities.

Support

If you find Task Explorer useful, please consider supporting the project on Patreon: https://www.patreon.com/DavidXanatos

Icons provided by Icons8.

curl/curl

A command line tool and library for transferring data with URL syntax, supporting DICT, FILE, FTP, FTPS, GOPHER, GOPHERS, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, MQTT, MQTTS, POP3, POP3S, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET, TFTP, WS and WSS. libcurl offers a myriad of powerful features

curl is a command-line tool for transferring data from or to a server using URLs. It supports these protocols: DICT, FILE, FTP, FTPS, GOPHER, GOPHERS, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, MQTT, MQTTS, POP3, POP3S, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET, TFTP, WS and WSS.

Learn how to use curl by reading the man page or everything curl.

Find out how to install curl by reading the INSTALL document.

libcurl is the library curl is using to do its job. It is readily available to be used by your software. Read the libcurl man page to learn how.

Open Source

curl is Open Source and is distributed under an MIT-like license.

Contact

All contributors to the project are listed in the THANKS document.

Commercial support

For commercial support, maybe private and dedicated help with your problems or applications using (lib)curl visit the support page.

Website

Visit the curl website for the latest news and downloads.

Source code

Download the latest source from the Git server:

git clone https://github.com/curl/curl

Security problems

Report suspected security problems privately and not in public.

Backers

Thank you to all our backers 🙏 Become a backer.

HansKristian-Work/vkd3d-proton

Fork of VKD3D. Development branches for Proton's Direct3D 12 implementation.

vkd3d-proton

vkd3d-proton is a fork of VKD3D, which aims to implement the full Direct3D 12 API on top of Vulkan. The project serves as the development effort for Direct3D 12 support in Proton.

Upstream

The original project is available at WineHQ.

Priorities

Performance and game compatibility are important targets, at the expense of compatibility with older drivers and systems. Modern Vulkan extensions and features are aggressively made use of to improve game performance and compatibility. It is recommended to use the very latest drivers you can get your hands on for the best experience. Backwards compatibility with the vkd3d standalone API is not a goal of this project.

Drivers

There are some hard requirements on drivers to be able to implement D3D12 in a reasonably performant way.

Vulkan 1.3
Descriptor indexing with at least 1000000 UpdateAfterBind descriptors for all types except UniformBuffer. Essentially all features in VkPhysicalDeviceDescriptorIndexingFeatures must be supported.
Further, the following device features are required:
- samplerMirrorClampToEdge
- shaderDrawParameters
VK_EXT_robustness2
VK_KHR_push_descriptor

Some notable extensions that should be supported for optimal or correct behavior. These extensions will likely become mandatory later.

VK_EXT_image_view_min_lod

VK_EXT_mutable_descriptor_type (or the vendor VALVE alias) and VK_EXT_descriptor_buffer are also highly recommended, but not mandatory.

AMD (RADV)

For AMD, RADV is the recommended driver and the one that sees most testing on AMD GPUs. The minimum requirement at the moment is Mesa 22.0.

NOTE: For older Mesa versions, use the v2.6 release.

NVIDIA

The Vulkan beta drivers generally contain the latest driver fixes that we identify while getting games to work. The latest drivers (stable, beta or Vulkan beta tracks) are always preferred. If you're having problems, always try the latest drivers. At minimum, 535 series drivers are needed, which fixes a bunch of bugs.

Intel

We have not done any testing against Intel GPUs yet.

Cloning the repo

To clone the repo you should run:

git clone --recursive https://github.com/HansKristian-Work/vkd3d-proton

in order to pull in all the submodules which are needed for building.

Building vkd3d-proton

Requirements:

wine (for widl) [for native builds]
- On Windows this may be substituted for Strawberry Perl as it ships widl and is easy to find and install -- although this dependency may be eliminated in the future.
Meson build system (at least version 0.49)
glslang compiler
Mingw-w64 compiler, headers and tools (at least version 7.0) [for cross-builds for d3d12.dll which are default]

Building:

The simple way

Inside the vkd3d-proton directory, run:

./package-release.sh master /your/target/directory --no-package

This will create a folder vkd3d-master in /your/target/directory, which contains both 32-bit and 64-bit versions of vkd3d-proton, which can be set up in the same way as the release versions as noted above.

If you want to build natively (ie. for libvkd3d-proton.so), pass --native to the build script. This option will make it build using your system's compilers.

In order to preserve the build directories for development, pass --dev-build to the script. This option implies --no-package. After making changes to the source code, you can then do the following to rebuild vkd3d-proton:

# change to build.86 for 32-bit
ninja -C /your/target/directory/build.64 install

Compiling manually (cross for d3d12.dll, default)

# 64-bit build.
meson --cross-file build-win64.txt --buildtype release --prefix /your/vkd3d-proton/directory build.64
ninja -C build.64 install

# 32-bit build
meson --cross-file build-win32.txt --buildtype release --prefix /your/vkd3d-proton/directory build.86
ninja -C build.86 install

Compiling manually (native)

# 64-bit build.
meson --buildtype release --prefix /your/vkd3d-proton/directory build.64
ninja -C build.64 install

# 32-bit build
CC="gcc -m32" CXX="g++ -m32" \
PKG_CONFIG_PATH="/usr/lib32/pkgconfig:/usr/lib/i386-linux-gnu/pkgconfig:/usr/lib/pkgconfig" \
meson --buildtype release --prefix /your/vkd3d-proton/directory build.86
ninja -C build.86 install

Cross-compilation build for aarch64

First, setup a distrobox using Steam's runtime for it.

distrobox create --image registry.gitlab.steamos.cloud/steamrt/steamrt4/sdk/arm64-on-amd64 --name aarch64
distrobox enter aarch64
# Inside container
sudo apt install mingw-w64-tools
meson setup build-aarch64 \
  --buildtype release \
  --cross-file /usr/share/meson/cross/aarch64-linux-gnu-gcc.txt \
  --cross-file ./build-widl.txt \
  -Denable_extras=true \
  -Denable_tests=true \
  --prefix /tmp/vkd3d-proton-aarch64
ninja -C build-aarch64 install
cp build-aarch64/tests/d3d12 /tmp/vkd3d-proton-aarch64/bin

Using vkd3d-proton

The intended way to use vkd3d-proton is as native Win32 DLLs (d3d12.dll and d3d12core.dll). These serve as a drop-in replacement for D3D12, and can be used in Wine (Proton or vanilla flavors), or on Windows.

vkd3d-proton does not supply the necessary DXGI components on its own. Instead, DXVK (2.1+) and vkd3d-proton share a DXGI implementation.

A note on using vkd3d-proton on Windows

Native Windows use is mostly relevant for developer testing purposes. Do not expect games running on Windows 7 or 8.1 to magically make use of vkd3d-proton, as many games will only even attempt to load d3d12.dll if they are running on Windows 10.

Native Linux build

A native Linux binary can be built, but it is not intended to be compatible with upstream Wine. A native option is mostly relevant for development purposes for the time being.

Environment variables

Most of the environment variables used by vkd3d-proton are for debugging purposes. The environment variables are not considered a part of API and might be changed or removed in the future versions of vkd3d-proton.

Some of debug variables are lists of elements. Elements must be separated by commas or semicolons.

VKD3D_CONFIG - a list of options that change the behavior of vkd3d-proton.
- vk_debug - enables Vulkan debug extensions and loads validation layer.
- skip_application_workarounds - Skips all application workarounds. For debugging purposes.
- nodxr - Disables DXR support.
- dxr - DXR is normally enabled automatically. This config forces it to be enabled even when considered unsafe.
- dxr12 - Enables experimental support for DXR 1.2 if VK_EXT_opacity_micromap is available.
- force_static_cbv - Unsafe speed hack on NVIDIA. May or may not give a significant performance uplift.
- single_queue - Do not use asynchronous compute or transfer queues.
- no_upload_hvv - Blocks any attempt to use host-visible VRAM (large/resizable BAR) for the UPLOAD heap. May free up vital VRAM in certain critical situations, at cost of lower GPU performance. A fraction of VRAM is reserved for resizable BAR allocations either way, so it should not be a real issue even on lower VRAM cards.
- force_host_cached - Forces all host visible allocations to be CACHED, which greatly accelerates captures.
- no_invariant_position - Avoids workarounds for invariant position. The workaround is enabled by default.
VKD3D_DEBUG - controls the debug level for log messages produced by vkd3d-proton. Accepts the following values: none, err, info, fixme, warn, trace.
VKD3D_SHADER_DEBUG - controls the debug level for log messages produced by the shader compilers. See VKD3D_DEBUG for accepted values.
VKD3D_LOG_FILE - If set, redirects VKD3D_DEBUG logging output to a file instead.
VKD3D_VULKAN_DEVICE - a zero-based device index. Use to force the selected Vulkan device.
VKD3D_FILTER_DEVICE_NAME - skips devices that don't include this substring.
VKD3D_DISABLE_EXTENSIONS - a list of Vulkan extensions that vkd3d-proton should not use even if available.
VKD3D_TEST_DEBUG - enables additional debug messages in tests. Set to 0, 1 or 2.
VKD3D_TEST_MATCH - a match string. Only the tests whose names exactly match the string will be run, e.g. VKD3D_TEST_FILTER=clear_render_target will only match tests named 'clear_render_target'. Useful for debugging or developing new tests.
VKD3D_TEST_FILTER - a filter string. Only the tests whose names matches the filter string will be run, e.g. VKD3D_TEST_FILTER=clear_render will match tests named 'clear_render_target' or 'target_clear_render'. Useful for debugging or developing new tests.
VKD3D_TEST_EXCLUDE - excludes tests of which the name is included in the string, e.g. VKD3D_TEST_EXCLUDE=test_root_signature_priority,test_conservative_rasterization_dxil.
VKD3D_TEST_PLATFORM - can be set to "wine", "windows" or "other". The test platform controls the behavior of todo(), todo_if(), bug_if() and broken() conditions in tests.
VKD3D_TEST_BUG - set to 0 to disable bug_if() conditions in tests.
VKD3D_PROFILE_PATH - If profiling is enabled in the build, a profiling block is emitted to ${VKD3D_PROFILE_PATH}.${pid}.
VKD3D_SWAPCHAIN_PRESENT_MODE - accepts a Vulkan present mode name. Forces the use of the specified present mode if supported. Currently accepts IMMEDIATE, MAILBOX, FIFO, FIFO_RELAXED, FIFO_LATEST_READY.

Frame rate limit

The VKD3D_FRAME_RATE environment variable can be used to limit the frame rate. A value of 0 uncaps the frame rate, while any positive value will limit rendering to the given number of frames per second.

Shader cache

By default, vkd3d-proton manages its own driver cache. This cache is intended to cache DXBC/DXIL -> SPIR-V conversion. This reduces stutter (when pipelines are created last minute and app relies on hot driver cache) and load times (when applications do the right thing of loading PSOs up front).

Behavior is designed to be close to DXVK state cache.

Default behavior

vkd3d-proton.cache (and vkd3d-proton.cache.write) are placed in the current working directory. Generally, this is the game install folder when running in Steam.

Custom directory

VKD3D_SHADER_CACHE_PATH=/path/to/directory overrides the directory where vkd3d-proton.cache is placed.

Disable cache

VKD3D_SHADER_CACHE_PATH=0 disables the internal cache, and any caching would have to be explicitly managed by application.

Behavior of ID3D12PipelineLibrary

When explicit shader cache is used, the need for application managed pipeline libraries is greatly diminished, and the cache applications interact with is a dummy cache. If the vkd3d-proton shader cache is disabled, ID3D12PipelineLibrary stores everything relevant for a full cache, i.e. SPIR-V and PSO driver cache blob. VKD3D_CONFIG=pipeline_library_app_cache is an alternative to VKD3D_SHADER_CACHE_PATH=0 and can be automatically enabled based on app-profiles if relevant in the future if applications manage the caches better than vkd3d-proton can do automagically.

CPU profiling (development)

Pass -Denable_profiling=true to Meson to enable a profiled build. With a profiled build, use VKD3D_PROFILE_PATH environment variable. The profiling dumps out a binary blob which can be analyzed with programs/vkd3d-profile.py. The profile is a trivial system which records number of iterations and total ticks (ns) spent. It is easy to instrument parts of code you are working on optimizing.

Advanced shader debugging

These features are only meant to be used by vkd3d-proton developers. For any builtin RenderDoc related functionality pass -Denable_renderdoc=true to Meson.

VKD3D_SHADER_DUMP_PATH - path where shader bytecode is dumped. Bytecode is dumped in format of $hash.{spv,dxbc,dxil}.
VKD3D_SHADER_OVERRIDE - path to where overridden shaders can be found. If application is creating a pipeline with $hash and $VKD3D_SHADER_OVERRIDE/$hash.spv exists, that SPIR-V file will be used instead.
VKD3D_AUTO_CAPTURE_SHADER - If this is set to a shader hash, and the RenderDoc layer is enabled, vkd3d-proton will automatically make a capture when a specific shader is encountered.
VKD3D_AUTO_CAPTURE_COUNTS - A comma-separated list of indices. This can be used to control which queue submissions to capture. E.g., use VKD3D_AUTO_CAPTURE_COUNTS=0,4,10 to capture the 0th (first submission), 4th and 10th submissions which are candidates for capturing. If VKD3D_AUTO_CAPTURE_COUNTS is -1, the entire app runtime can be turned into one big capture. This is only intended to be used when capturing something like the test suite, or tiny applications with a finite runtime to make it easier to debug cross submission work.

If only VKD3D_AUTO_CAPTURE_COUNTS is set, any queue submission is considered for capturing. If only VKD3D_AUTO_CAPTURE_SHADER is set, VKD3D_AUTO_CAPTURE_COUNTS is considered to be equal to "0", i.e. a capture is only made on first encounter with the target shader. If both are set, the capture counter is only incremented and considered when a submission contains the use of the target shader.

Breadcrumbs debugging

For debugging GPU hangs, it's useful to know where crashes happen. If the build has trace enabled (non-release builds), breadcrumbs support is also enabled.

VKD3D_CONFIG=breadcrumbs will instrument command lists with VK_AMD_buffer_marker or VK_NV_device_checkpoints. On GPU device lost or timeout, crash dumps are written to the log. For best results on RADV, use RADV_DEBUG=syncshaders. The logs will print a digested form of the command lists which were executing at the time, and attempt to narrow down the possible range of commands which could have caused a crash.

Shader logging

It is possible to log the output of replaced shaders, essentially a custom shader printf. To enable this feature, VK_KHR_buffer_device_address must be supported. First, use VKD3D_SHADER_DEBUG_RING_SIZE_LOG2=28 for example to set up a 256 MiB ring buffer in host memory. Since this buffer is allocated in host memory, feel free to make it as large as you want, as it does not consume VRAM. A worker thread will read the data as it comes in and log it. There is potential here to emit more structured information later. The main reason this is implemented instead of the validation layer printf system is run-time performance, and avoids any possible accidental hiding of bugs by introducing validation layers which add locking, etc. Using debugPrintEXT is also possible if that fits better with your debugging scenario. With this shader replacement scheme, we're able to add shader logging as unintrusive as possible.

# Inside folder full of override shaders, build everything with:
make -C /path/to/include/shader-debug M=$PWD

The shader can then include #include "debug_channel.h" and use various functions below.

void DEBUG_CHANNEL_INIT(uvec3 ID);

is used somewhere in your replaced shader. This should be initialized with gl_GlobalInvocationID or similar. This ID will show up in the log. For each subgroup which calls DEBUG_CHANNEL_INIT, an instance counter is generated. This allows you to correlate several messages which all originate from the same instance counter, which is logged alongside the ID. An invocation can be uniquely identified with the instance + DEBUG_CHANNEL_INIT id. DEBUG_CHANNEL_INIT can be called from non-uniform control flow, as it does not use barrier() or similar constructs. It can also be used in vertex and fragment shaders for this reason.

void DEBUG_CHANNEL_MSG();
void DEBUG_CHANNEL_MSG(uint v0);
void DEBUG_CHANNEL_MSG(uint v0, uint v1, ...); // Up to 4 components, can be expanded as needed up to 16.
void DEBUG_CHANNEL_MSG(int v0);
void DEBUG_CHANNEL_MSG(int v0, int v1, ...); // Up to 4 components, ...
void DEBUG_CHANNEL_MSG(float v0);
void DEBUG_CHANNEL_MSG(float v0, float v1, ...); // Up to 4 components, ...

These functions log, formatting is #%x for uint, %d for int and %f for float type.

Descriptor debugging

If -Denable_descriptor_qa=true is enabled in build, you can set the VKD3D_DESCRIPTOR_QA_LOG env-var to a file. All descriptor updates and copies are logged so that it's possible to correlate descriptors with GPU crash dumps. enable_descriptor_qa is not enabled by default, since it adds some flat overhead in an extremely hot code path.

GPU-assisted debugging

If VKD3D_CONFIG=descriptor_qa_checks is set with a build which enables -Denable_descriptor_qa=true, all shaders will be instrumented to check for invalid access. In the log, you will see this to make sure the feature is enabled.

932:info:vkd3d_descriptor_debug_init_once: Enabling descriptor QA checks!

The main motivation is the tight integration and high performance. GPU-assisted debugging can be run at well over playable speeds.

Descriptor heap index out of bounds

============
Fault type: HEAP_OUT_OF_RANGE
Fault type: MISMATCH_DESCRIPTOR_TYPE
CBV_SRV_UAV heap cookie: 1800
Shader hash and instruction: edbaf1b5ed344467 (1)
Accessed resource/view cookie: 0
Shader desired descriptor type: 8 (STORAGE_BUFFER)
Found descriptor type in heap: 0 (NONE)
Failed heap index: 1024000
==========

The instruction (1), is reported as well, and a disassembly of the shader in question can be used to pinpoint exactly where things are going wrong. Dump all shaders with VKD3D_SHADER_DUMP_PATH=/my/folder, and run spirv-cross -V /my/folder/edbaf1b5ed344467.spv. (NOTE: clear out the folder before dumping, existing files are not overwritten). The faulting instruction can be identified by looking at last argument, e.g.:

uint fixup_index = descriptor_qa_check(heap_index, descriptor_type, 1u /* instruction ID */);

Mismatch descriptor type

============
Fault type: MISMATCH_DESCRIPTOR_TYPE
CBV_SRV_UAV heap cookie: 1800 // Refer to VKD3D_DESCRIPTOR_QA_LOG
Shader hash and instruction: edbaf1b5ed344467 (1)
Accessed resource/view cookie: 1802 // Refer to VKD3D_DESCRIPTOR_QA_LOG
Shader desired descriptor type: 8 (STORAGE_BUFFER)
Found descriptor type in heap: 1 (SAMPLED_IMAGE)
Failed heap index: 1025
==========

Accessing destroyed resource

============
Fault type: DESTROYED_RESOURCE
CBV_SRV_UAV heap cookie: 1800
Shader hash and instruction: edbaf1b5ed344467 (2)
Accessed resource/view cookie: 1806
Shader desired descriptor type: 1 (SAMPLED_IMAGE)
Found descriptor type in heap: 1 (SAMPLED_IMAGE)
Failed heap index: 1029
==========

Debugging descriptor crashes with RADV dumps (hardcore ultra nightmare mode)

For when you're absolutely desperate, there is a way to debug GPU hangs. First, install umr and make the binary setsuid.

ACO_DEBUG=force-waitcnt RADV_DEBUG=hang VKD3D_DESCRIPTOR_QA_LOG=/somewhere/desc.txt %command%

It is possible to use RADV_DEBUG=hang,umr as well, but from within Wine, there are weird things happening where UMR dumps do not always succeed. Instead, it is possible to invoke umr manually from an SSH shell when the GPU hangs.

#!/bin/bash

mkdir -p "$HOME/umr-dump"

# For Navi, older GPUs might have different rings. See RADV source.
umr -R gfx_0.0.0 > "$HOME/umr-dump/ring.txt" 2>&1
umr -O halt_waves -wa gfx_0.0.0 > "$HOME/umr-dump/halt-waves-1.txt" 2>&1
umr -O bits,halt_waves -wa gfx_0.0.0 > "$HOME/umr-dump/halt-waves-2.txt" 2>&1

A folder is placed in ~/radv_dumps* by RADV, and the UMR script will place wave dumps in ~/umr-dump.

First, we can study the wave dumps to see where things crash, e.g.:

    pgm[6@0x800120e26c00 + 0x584 ] = 0xf0001108		image_load v47, v[4:5], s[48:55] dmask:0x1 dim:SQ_RSRC_IMG_2D unorm
    pgm[6@0x800120e26c00 + 0x588 ] = 0x000c2f04	;;
    pgm[6@0x800120e26c00 + 0x58c ] = 0xbf8c3f70		s_waitcnt vmcnt(0)
 *  pgm[6@0x800120e26c00 + 0x590 ] = 0x930118c0		s_mul_i32 s1, 64, s24
    pgm[6@0x800120e26c00 + 0x594 ] = 0xf40c0c09		s_load_dwordx8 s[48:55], s[18:19], s1
    pgm[6@0x800120e26c00 + 0x598 ] = 0x02000000	;;

excp: 256 is a memory error (at least on 5700xt).

TRAPSTS[50000100]:
	                excp:      256 |         illegal_inst:        0 |           buffer_oob:        0 |           excp_cycle:        0 |
	       excp_wave64hi:        0 |          xnack_error:        1 |              dp_rate:        2 |      excp_group_mask:        0 |

We can inspect all VGPRs and all SGPRs, here for the image descriptor.

    [  48..  51] = { 0130a000, c0500080, 810dc1df, 93b00204 }
    [  52..  55] = { 00000000, 00400000, 002b0000, 800130c8 }

Decode the VA and study bo_history.log. There is a script in RADV which lets you query history for a VA. This lets us verify that the VA in question was freed at some point. At point of writing, there is no easy way to decode raw descriptor blobs, but when you're desperate enough you can do it by hand 😐

In pipeline.log we have the full SPIR-V (with OpSource reference to the source DXIL/DXBC) and disassembly of the crashed pipeline. Here we can study the code to figure out which descriptor was read.

    // s7 is the descriptor heap index, s1 is the offset (64 bytes per image descriptor),
    // s[18:19] is the descriptor heap.
    s_mul_i32 s1, 64, s7                                        ; 930107c0
    s_load_dwordx8 s[48:55], s[18:19], s1                       ; f40c0c09 02000000
    s_waitcnt lgkmcnt(0)                                        ; bf8cc07f
    image_load v47, v[4:5], s[48:55] dmask:0x1 dim:SQ_RSRC_IMG_2D unorm ; f0001108 000c2f04

    [   4..   7] = { 03200020, ffff8000, 0000002b, 00000103 }

Which is descriptor index #259. Based on this, we can inspect the descriptor QA log and verify that the application did indeed do something invalid, which caused the GPU hang.

microsoft/mimalloc

mimalloc is a compact general purpose allocator with excellent performance.

^v3: ^v2: ^v1: ^v3:

mimalloc

mimalloc (pronounced "me-malloc") is a general purpose allocator with excellent performance characteristics. Initially developed by Daan Leijen for the runtime systems of the Koka and Lean languages.

Latest release : v3.3.2 (2026-04-29) recommended.
Latest v2 release: v2.3.2 (2026-04-29) stable.
Latest v1 release: v1.9.10 (2026-04-29) legacy.

mimalloc is a drop-in replacement for malloc and can be used in other programs without code changes, for example, on dynamically linked ELF-based systems (Linux, BSD, etc.) you can use it as:

> LD_PRELOAD=/usr/lib/libmimalloc.so  myprogram

It also includes a way to dynamically override the default allocator in Windows. Notable aspects of the design include:

small and consistent: the library is about 10k LOC using simple and consistent data structures. This makes it very suitable to integrate and adapt in other projects. For runtime systems it provides hooks for a monotonic heartbeat and deferred freeing (for bounded worst-case times with reference counting). Partly due to its simplicity, mimalloc has been ported to many systems (Windows, macOS, Linux, WASM, various BSD's, Haiku, MUSL, etc) and has excellent support for dynamic overriding. At the same time, it is an industrial strength allocator that runs (very) large scale distributed services on thousands of machines with excellent worst case latencies.
free list sharding: instead of one big free list (per size class) we have many smaller lists per "mimalloc page" which reduces fragmentation and increases locality -- things that are allocated close in time get allocated close in memory. (A mimalloc page contains blocks of one size class and is usually 64KiB on a 64-bit system).
free list multi-sharding: the big idea! Not only do we shard the free list per mimalloc page, but for each page we have multiple free lists. In particular, there is one list for thread-local free operations, and another one for concurrent free operations. Free-ing from another thread can now be a single CAS without needing sophisticated coordination between threads. Since there will be thousands of separate free lists, contention is naturally distributed over the heap, and the chance of contending on a single location will be low -- this is quite similar to randomized algorithms like skip lists where adding a random oracle removes the need for a more complex algorithm.
eager page purging: when a "page" becomes empty (with increased chance due to free list sharding) the memory is marked to the OS as unused (reset or decommitted) reducing (real) memory pressure and fragmentation, especially in long running programs.
secure: mimalloc can be built in secure mode, adding guard pages, randomized allocation, encrypted free lists, etc. to protect against various heap vulnerabilities. The performance penalty is usually around 10% on average over our benchmarks.
first-class heaps: efficiently create and use multiple heaps to allocate across different regions. A heap can be destroyed at once instead of deallocating each object separately. New: v3 has true first-class heaps where one can allocate in a heap from any thread.
bounded: it does not suffer from blowup [1], has bounded worst-case allocation times (wcat) (upto OS primitives), bounded space overhead (~0.2% meta-data, with low internal fragmentation), and has no internal points of contention using only atomic operations.
fast: In our benchmarks (see below), mimalloc outperforms other leading allocators (jemalloc, tcmalloc, Hoard, etc), and often uses less memory. A nice property is that it does consistently well over a wide range of benchmarks. There is also good huge OS page support for larger server programs.

The documentation gives a full overview of the API. You can read more on the design of mimalloc in the technical report which also has detailed benchmark results.

Enjoy!

Versions

There are three maintained versions of mimalloc. These are mostly equal except for how the OS memory is handled. New development is mostly on v3, while v1 and v2 are maintained with security and bug fixes.

v3: recommended: simplifies the lock-free design of previous versions and improves sharing of memory between threads. On certain large workloads this version may use (much) less memory. Also supports true first-class heaps (that can allocate from any thread) and has more efficient heap-walking (for the CPython GC for example). (release tags: v3.x, development branch dev3).
v2: stable mimalloc version. Uses thread-local segments to reduce fragmentation. (release tags: v2.x, development branch dev2 and main)
v1: legacy version: initial design of mimalloc (release tags: v1.9.x, development branch dev). Send PR's against this version if possible.

Releases

2026-04-29, v1.9.10, v2.3.2, v3.3.2: various bug and security fixes through LLM audit (by @Zoxc). Only increase minimal purge size automatically if allow_thp is set to 2. Enable large OS alignment on all platforms (fixing OS large pages on Windows). Fix accounting of committed memory on Linux/macOS. Update MSVC atomics implementation when using C mode. Upstream Emscripten fixes. Proper atomic do-once implementation.
2026-04-20, v1.9.9, v2.3.1, v3.3.1: various bug and security fixes. Special thanks to @jinpzhanAMD, @res2k, and @GoldJohnKing for their help in improving Windows finalization, and @Zoxc for his help in finding various issues.
2026-04-15, v1.9.8, v2.3.0, v3.3.0: initial support for github (binary) releases, fix visiting of full pages during collection (performance), fix THP alignment (performance), fix arm64 cross-compilation on Windows, enable guard pages in debug mode, always use uncommitted areas between arenas (security), enable static overloading of malloc etc. on Windows with the static CRT (by @Noxybot), fix TLS slot leak on Windows (v3), enable clean DLL load/unload with statically linked mimalloc (v3), fix race in mi_heap_destroy (v3), by default put page meta info separate from allocated objects (v3,security), fix C++ overrides for emscripten. Various bugs found by DeepTest include: fix offset for mi_heap_realloc_aligned, fix mi_(w)dupenv_s buffer size, fix potential overflow in size options, and error codes for mi_reallocarr(ay).
2026-02-03, v3.2.8 (rc3): Fix thread reinitialize issue on macOS. Fix SIMD codegen bug on older GCC versions. Extend Windows TLS slot limit from 64 to 1088. Report commit statistics more precise. Fixes issue in free-page search in arenas.
2026-01-15, v1.9.7, v2.2.7, v3.2.7 (rc2): Fix zero initializing blocks that were OS allocated.
For v3 various bug and performance fixes. Fix Debian 32-bit compilation.
2026-01-08, v1.9.6, v2.2.6, v3.2.6 (rc1): Important bug fixes. Many improvements to v3 including true first-class heaps where one can allocate in heap from any thread, and track statistics per heap as well. Added MIMALLOC_ALLOW_THP option. This is by default enabled except on Android. When THP is detected on v3, mimalloc will set the MIMALLOC_MINIMAL_PURGE_SIZE to 2MiB to avoid breaking up potential THP huge pages. v3 uses faster TLS access on Windows, and has improved performance for mi_calloc and aligned allocations. Fixed rare race condition on older v3, fixed potential buffer overflow in debug statistics, add API for returning allocated sizes on allocation and free.
2025-06-13, v3.1.5: Bug fix release where memory was not always correctly committed (issue #1098).
2025-06-09, v1.9.4, v2.2.4, v3.1.4 (beta) : Some important bug fixes, including a case where OS memory was not always fully released. Improved v3 performance, build on XBox, fix build on Android, support interpose for older macOS versions, use MADV_FREE_REUSABLE on macOS, always check commit success, better support for Windows fixed TLS offset, etc.
2025-03-28, v1.9.3, v2.2.3, v3.0.3 (beta) : Various small bug and build fixes, including: fix arm32 pre v7 builds, fix mingw build, get runtime statistics, improve statistic commit counts, fix execution on non BMI1 x64 systems.
2025-03-06, v1.9.2, v2.2.2, v3.0.2-beta: Various small bug and build fixes. Add mi_options_print, mi_arenas_print, and the experimental mi_stat_get and mi_stat_get_json. Add mi_thread_set_in_threadpool and mi_heap_set_numa_affinity (v3 only). Add vcpkg portfile. Upgrade mimalloc-redirect to v1.3.2. MI_OPT_ARCH is off by default now but still assumes armv8.1-a on arm64 for fast atomic operations. Add QNX support.
2025-01-03, v1.8.9, v2.1.9, v3.0.1-alpha: Interim release. Support Windows arm64. New guarded build that can place OS guard pages behind objects to catch buffer overflows as they occur. Many small fixes: build on Windows arm64, cygwin, riscV, and dragonfly; fix Windows static library initialization to account for thread local destructors (in Rust/C++); macOS tag change; macOS TLS slot fix; improve stats; consistent mimalloc.dll on Windows (instead of mimalloc-override.dll); fix mimalloc-redirect on Win11 H2; add 0-byte to canary; upstream CPython fixes; reduce .bss size; allow fixed TLS slot on Windows for improved performance.
Older release notes

Special thanks to:

Sergiy Kuryata for his contributions on reducing memory commit -- especially on Windows with the Windows thread pool (now implemented in v3).
David Carlier (@devnexen) for his many contributions, and making mimalloc work better on many less common operating systems, like Haiku, Dragonfly, etc.
Mary Feofanova (@mary3000), Evgeniy Moiseenko, and Manuel Pöter (@mpoeter) for making mimalloc TSAN checkable, and finding memory model bugs using the genMC model checker.
Weipeng Liu (@pongba), Zhuowei Li, Junhua Wang, and Jakub Szymanski, for their early support of mimalloc and deployment at large scale services, leading to many improvements in the mimalloc algorithms for large workloads.
Jason Gibson (@jasongibson) for exhaustive testing on large scale workloads and server environments, and finding complex bugs in (early versions of) mimalloc.
Manuel Pöter (@mpoeter) and Sam Gross(@colesbury) for finding an ABA concurrency issue in abandoned segment reclamation. Sam also created the no GIL Python fork which uses mimalloc internally.

Usage

mimalloc is used in various large scale low-latency services and programs, for example:

Building

Windows

Open ide/vs2022/mimalloc.sln in Visual Studio 2022 and build. The mimalloc-lib project builds a static library (in out/msvc-x64), while the mimalloc-override-dll project builds a DLL for overriding malloc in the entire program.

Linux, macOS, BSD, etc.

We use cmake as the build system:

> mkdir -p out/release
> cd out/release
> cmake ../..
> make

This builds the library as a shared (dynamic) library (.so or .dylib), a static library (.a), and as a single object file (.o).

> sudo make install (install the library and header files in /usr/local/lib and /usr/local/include)

You can build the debug version which does many internal checks and maintains detailed statistics as:

> mkdir -p out/debug
> cd out/debug
> cmake -DCMAKE_BUILD_TYPE=Debug ../..
> make

This will name the shared library as libmimalloc-debug.so.

Finally, you can build a secure version that uses guard pages, encrypted free lists, etc., as:

> mkdir -p out/secure
> cd out/secure
> cmake -DMI_SECURE=ON ../..
> make

This will name the shared library as libmimalloc-secure.so. Use cmake ../.. -LH to see all the available build options.

The examples use the default compiler. If you like to use another, use:

> CC=clang CXX=clang++ cmake ../..

Cmake with Visual Studio

You can also use cmake on Windows. Open a Visual Studio 2022 development prompt and invoke cmake with the right generator and architecture, like:

> cmake ..\.. -G "Visual Studio 17 2022" -A x64 -DMI_OVERRIDE=ON

The cmake build type is specified when actually building, for example:

> cmake --build . --config=Release

You can also install the LLVM toolset on Windows to build with the clang-cl compiler directly:

> cmake ../.. -G "Visual Studio 17 2022" -T ClangCl

Single Source

You can also directly build the single src/static.c file as part of your project without needing cmake at all. Make sure to also add the mimalloc include directory to the include path.

Using the Library

The preferred usage is including <mimalloc.h>, linking with the shared- or static library, and using the mi_malloc API exclusively for allocation. For example,

> gcc -o myprogram -lmimalloc myfile.c

mimalloc uses only safe OS calls (mmap and VirtualAlloc) and can co-exist with other allocators linked to the same program. If you use cmake, you can simply use:

find_package(mimalloc 1.8 REQUIRED)

in your CMakeLists.txt to find a locally installed mimalloc. Then use either:

target_link_libraries(myapp PUBLIC mimalloc)

to link with the shared (dynamic) library, or:

target_link_libraries(myapp PUBLIC mimalloc-static)

to link with the static library. See test\CMakeLists.txt for an example.

For best performance in C++ programs, it is also recommended to override the global new and delete operators. For convenience, mimalloc provides mimalloc-new-delete.h which does this for you -- just include it in a single(!) source file in your project. In C++, mimalloc also provides the mi_stl_allocator struct which implements the std::allocator interface.

You can pass environment variables to print verbose messages (MIMALLOC_VERBOSE=1) and statistics (MIMALLOC_SHOW_STATS=1) (in the debug version):

> env MIMALLOC_SHOW_STATS=1 ./cfrac 175451865205073170563711388363

175451865205073170563711388363 = 374456281610909315237213 * 468551

subproc 0
 blocks          peak       total     current       block      total#
  bin S    4:    75.3 KiB    55.2 MiB     0          32   B       1.8 M    ok
  bin S    6:    31.0 KiB   180.4 KiB     0          48   B       3.8 K    ok
  bin S    8:    64   B      64   B       0          64   B       1        ok
  bin S    9:   160   B     160   B       0          80   B       2        ok
  bin S   17:     1.2 KiB     1.2 KiB     0         320   B       4        ok
  bin S   21:   640   B       3.1 KiB     0         640   B       5        ok
  bin S   33:     5.0 KiB     5.0 KiB     0           5.0 KiB     1        ok

  binned    :    84.2 Ki     41.5 Mi      0                                ok
  huge      :     0           0           0                                ok
  total     :    84.2 KiB    41.5 MiB     0
  malloc req:                29.7 MiB

 pages           peak       total     current       block      total#
  touched   :   152.8 KiB   152.8 KiB   152.8 KiB
  pages     :     8          14           0                                ok
  abandoned :     1         249           0                                ok
  reclaima  :     0
  reclaimf  :   249
  reabandon :     0
  waits     :     0
  extended  :    38
  retire    :    35
  searches  :     0.7 avg

 arenas          peak       total     current       block      total#
  reserved  :     1.0 GiB     1.0 GiB     1.0 GiB
  committed :     4.8 MiB     4.8 MiB     4.4 MiB
  reset     :     0
  purged    :   385.5 Ki
  arenas    :     1
  rollback  :     0
  mmaps     :     3
  commits   :     0
  resets    :     1
  purges    :     2
  guarded   :     0
  heaps     :     1           1           1

 process         peak       total     current       block      total#
  threads   :     1           1           1
  numa nodes:     1
  elapsed   :     0.553 s
  process   : user: 0.557 s, system: 0.013 s, faults: 29, peak rss: 2.1 MiB, peak commit: 4.8 MiB

The above model of using the mi_ prefixed API is not always possible though in existing programs that already use the standard malloc interface, and another option is to override the standard malloc interface completely and redirect all calls to the mimalloc library instead .

Environment Options

You can set further options either programmatically (using mi_option_set), or via environment variables:

MIMALLOC_SHOW_STATS=1: show statistics when the program terminates.
MIMALLOC_VERBOSE=1: show verbose messages (including statistics).
MIMALLOC_SHOW_ERRORS=1: show error and warning messages.

Advanced options:

MIMALLOC_ARENA_EAGER_COMMIT=2: turns on eager commit for the large arenas (usually 1GiB) from which mimalloc allocates segments and pages. Set this to 2 (default) to only enable this on overcommit systems (e.g. Linux). Set this to 1 to enable explicitly on other systems as well (like Windows or macOS) which may improve performance (as the whole arena is committed at once). Note that eager commit only increases the commit but not the actual the peak resident set (rss) so it is generally ok to enable this.
MIMALLOC_PURGE_DELAY=N: the delay in N milli-seconds (by default 1000 in v3) after which mimalloc will purge OS pages that are not in use. This signals to the OS that the underlying physical memory can be reused which can reduce memory fragmentation especially in long running (server) programs. Setting N to 0 purges immediately when a page becomes unused which can improve memory usage but also decreases performance. Setting it to -1 disables purging completely.
MIMALLOC_PURGE_DECOMMITS=1: By default "purging" memory means unused memory is decommitted (MEM_DECOMMIT on Windows, MADV_DONTNEED (which decresease rss immediately) on mmap systems). Set this to 0 to instead "reset" unused memory on a purge (MEM_RESET on Windows, generally MADV_FREE (which does not decrease rss immediately) on mmap systems). Mimalloc generally does not "free" OS memory but only "purges" OS memory, in other words, it tries to keep virtual address ranges and decommits within those ranges (to make the underlying physical memory available to other processes).

Further options for large workloads and services:

MIMALLOC_ALLOW_THP=1: By default always allow transparent huge pages (THP) on Linux systems. On Android only this is by default off. When set to 0, THP is disabled for the process that mimalloc runs in. If enabled, mimalloc also sets the MIMALLOC_MINIMAL_PURGE_SIZE in v3 to 2MiB to avoid potentially breaking up transparent huge pages when purging memory.
MIMALLOC_USE_NUMA_NODES=N: pretend there are at most N NUMA nodes. If not set, the actual NUMA nodes are detected at runtime. Setting N to 1 may avoid problems in some virtual environments. Also, setting it to a lower number than the actual NUMA nodes is fine and will only cause threads to potentially allocate more memory across actual NUMA nodes (but this can happen in any case as NUMA local allocation is always a best effort but not guaranteed).
MIMALLOC_ALLOW_LARGE_OS_PAGES=0: Set to 1 to use large OS pages (2 or 4MiB) when available; for some workloads this can significantly improve performance. However, large OS pages cannot be purged or shared with other processes so may lead to increased memory usage in some cases. Use MIMALLOC_VERBOSE to check if the large OS pages are enabled -- usually one needs to explicitly give permissions for large OS pages (as on Windows and Linux). However, sometimes the OS is very slow to reserve contiguous physical memory for large OS pages so use with care on systems that can have fragmented memory (for that reason, we generally recommend to use MIMALLOC_RESERVE_HUGE_OS_PAGES instead whenever possible).
MIMALLOC_RESERVE_HUGE_OS_PAGES=N: where N is the number of 1GiB huge OS pages. This reserves the huge pages at startup and sometimes this can give a large (latency) performance improvement on big workloads. Usually it is better to not use MIMALLOC_ALLOW_LARGE_OS_PAGES=1 in combination with this setting. Just like large OS pages, use with care as reserving contiguous physical memory can take a long time when memory is fragmented (but reserving the huge pages is done at startup only once). Note that we usually need to explicitly give permission for huge OS pages (as on Windows and Linux)). The huge pages are usually allocated evenly among NUMA nodes. We can use MIMALLOC_RESERVE_HUGE_OS_PAGES_AT=N where N is the numa node (starting at 0) to allocate all the huge pages at a specific numa node instead.

Use caution when using fork in combination with either large or huge OS pages: on a fork, the OS uses copy-on-write for all pages in the original process including the huge OS pages. When any memory is now written in that area, the OS will copy the entire 1GiB huge page (or 2MiB large page) which can cause the memory usage to grow in large increments.

Secure Mode

mimalloc can be build in secure mode by using the -DMI_SECURE=ON flags in cmake. This build enables various mitigations to make mimalloc more robust against exploits. In particular:

All internal mimalloc page meta-data is surrounded by guard pages (so a buffer overflow exploit cannot reach into the metadata).
All free list pointers are encoded with per-page keys which is used both to prevent overwrites with a known pointer, as well as to detect heap corruption.
Double free's are detected (and ignored).
The free lists are initialized in a random order and allocation randomly chooses between extension and reuse within a page to mitigate against attacks that rely on a predicable allocation order. Similarly, the larger heap blocks allocated by mimalloc from the OS are also address randomized.
If enabling -DMI_SECURE_FULL=ON there will also be guard pages at the end of each (64KiB) mimalloc page (thus interleaving valid block data with inaccessible gaps). This setting is not recommended in general as it is more expensive and can lead to reaching the maximum VMA limit on Linux systems if the heap gets too large.

As always, evaluate with care as part of an overall security strategy as all of the above are mitigations but not guarantees.

Debug Mode

When mimalloc is built using debug mode, (-DCMAKE_BUILD_TYPE=Debug), various checks are done at runtime to catch development errors.

Statistics are maintained in detail for each object size. They can be shown using MIMALLOC_SHOW_STATS=1 at runtime.
All objects have padding at the end to detect (byte precise) heap block overflows.
Double free's, and freeing invalid heap pointers are detected.
Corrupted free-lists and some forms of use-after-free are detected.

Guarded Mode

mimalloc can be build in guarded mode using the -DMI_GUARDED=ON flags in cmake. This is ON by default when building a debug version of mimalloc. Guarded mode enables placing OS guard pages behind certain object allocations to catch buffer overflows as they occur. This can be invaluable to catch buffer-overflow bugs in large programs. However, it also means that any object allocated with a guard page takes at least 8 KiB memory for the guard page and its alignment. As such, allocating a guard page for every allocation may be too expensive both in terms of memory, and in terms of performance with many system calls. Therefore, there are various environment variables (and options) to tune this:

MIMALLOC_GUARDED_SAMPLE_RATE=N: Set the sample rate to N (by default 0). This mode places a guard page behind every N suitable object allocations (per thread). Since the performance in guarded mode without placing guard pages is close to release mode, this can be used to enable guard pages even in production to catch latent buffer overflow bugs. Set the sample rate to 1 to guard every object, and to 0 to place no guard pages at all.
MIMALLOC_GUARDED_SAMPLE_SEED=N: Start sampling at N (by default random). Can be used to reproduce a buffer overflow if needed.
MIMALLOC_GUARDED_MIN=N, MIMALLOC_GUARDED_MAX=N: Minimal and maximal rounded object sizes for which a guard page is considered (0 and 1GiB respectively). If you suspect a buffer overflow occurs with an object of size 141, set the minimum and maximum to 148 and the sample rate to 1 to have all of those guarded.
MIMALLOC_GUARDED_PRECISE=1: If we have an object of size 13, we would usually place it an aligned 16 bytes in front of the guard page. Using MIMALLOC_GUARDED_PRECISE places it exactly 13 bytes before a page so that even a 1 byte overflow is detected. This violates the C/C++ minimal alignment guarantees though so use with care.

Overriding Standard Malloc

Overriding the standard malloc (and new) can be done either dynamically or statically.

Dynamic override

This is the recommended way to override the standard malloc interface.

Dynamic Override on Linux, BSD

On these ELF-based systems we preload the mimalloc shared library so all calls to the standard malloc interface are resolved to the mimalloc library.

> env LD_PRELOAD=/usr/lib/libmimalloc.so myprogram

You can set extra environment variables to check that mimalloc is running, like:

> env MIMALLOC_VERBOSE=1 LD_PRELOAD=/usr/lib/libmimalloc.so myprogram

or run with the debug version to get detailed statistics:

> env MIMALLOC_SHOW_STATS=1 LD_PRELOAD=/usr/lib/libmimalloc-debug.so myprogram

Dynamic Override on MacOS

On macOS we can also preload the mimalloc shared library so all calls to the standard malloc interface are resolved to the mimalloc library.

> env DYLD_INSERT_LIBRARIES=/usr/lib/libmimalloc.dylib myprogram

Note that certain security restrictions may apply when doing this from the shell.

Dynamic Override on Windows

We use a separate redirection DLL to override mimalloc on Windows such that we redirect all malloc/free calls that go through the (dynamic) C runtime allocator, including those from other DLL's or libraries. As it intercepts all allocation calls on a low level, it can be used on large programs that include other 3rd party components. There are four requirements to make the overriding work well:

Use the C-runtime library as a DLL (using the /MD or /MDd switch).
Link your program explicitly with the mimalloc.dll.lib export library for the mimalloc.dll. (which must be compiled with -DMI_OVERRIDE=ON, which is the default though). To ensure the mimalloc.dll is actually loaded at run-time it is easiest to insert some call to the mimalloc API in the main function, like mi_version() (or use the /include:mi_version switch on the linker command, or similarly, #pragma comment(linker, "/include:mi_version") in some source file). See the mimalloc-test-override project for an example on how to use this.
The mimalloc-redirect.dll must be put in the same directory as the main mimalloc.dll at runtime (as it is a dependency of that DLL). The redirection DLL ensures that all calls to the C runtime malloc API get redirected to mimalloc functions (which reside in mimalloc.dll).
Ensure the mimalloc.dll comes as early as possible in the import list of the final executable (so it can intercept all potential allocations). You can use minject -l <exe> to check this if needed.

For best performance on Windows with C++, it is also recommended to also override the new/delete operations (by including mimalloc-new-delete.h a single(!) source file in your project).

The environment variable MIMALLOC_DISABLE_REDIRECT=1 can be used to disable dynamic overriding at run-time. Use MIMALLOC_VERBOSE=1 to check if mimalloc was successfully redirected.

For different platforms than x64, you may need a specific redirection dll. Furthermore, we cannot always re-link an executable or ensure mimalloc.dll comes first in the import table. In such cases the minject tool can be used to patch the executable's import tables.

Static override

On Unix-like systems, you can also statically link with mimalloc to override the standard malloc interface. The recommended way is to link the final program with the mimalloc single object file (mimalloc.o). We use an object file instead of a library file as linkers give preference to that over archives to resolve symbols. To ensure that the standard malloc interface resolves to the mimalloc library, link it as the first object file. For example:

> gcc -o myprogram mimalloc.o  myfile1.c ...

Another way to override statically that works on all platforms, is to link statically to mimalloc (as shown in the introduction) and include a header file in each source file that re-defines malloc etc. to mi_malloc. This is provided by mimalloc-override.h. This only works reliably though if all sources are under your control or otherwise mixing of pointers from different heaps may occur!

Note: recently we also enabled static overloading on Windows. In that case you need to link with the static CRT release runtime (/MT) and link with the static mimalloc(-debug).obj (to take precendence over the definitions in the CRT library).

Tools

Generally, we recommend using the standard allocator with memory tracking tools, but mimalloc can also be build to support the address sanitizer or the excellent Valgrind tool. Moreover, it can be build to support Windows event tracing (ETW). This has a small performance overhead but does allow detecting memory leaks and byte-precise buffer overflows directly on final executables. See also the test/test-wrong.c file to test with various tools.

Valgrind

To build with valgrind support, use the MI_TRACK_VALGRIND=ON cmake option:

> cmake ../.. -DMI_TRACK_VALGRIND=ON

This can also be combined with secure mode or debug mode. You can then run your programs directly under valgrind:

> valgrind <myprogram>

If you rely on overriding malloc/free by mimalloc (instead of using the mi_malloc/mi_free API directly), you also need to tell valgrind to not intercept those calls itself, and use:

> MIMALLOC_SHOW_STATS=1 valgrind  --soname-synonyms=somalloc=*mimalloc* -- <myprogram>

By setting the MIMALLOC_SHOW_STATS environment variable you can check that mimalloc is indeed used and not the standard allocator. Even though the Valgrind option is called --soname-synonyms, this also works when overriding with a static library or object file. To dynamically override mimalloc using LD_PRELOAD together with valgrind, use:

> valgrind --trace-children=yes --soname-synonyms=somalloc=*mimalloc* /usr/bin/env LD_PRELOAD=/usr/lib/libmimalloc.so -- <myprogram>

See also the test/test-wrong.c file to test with valgrind.

Valgrind support is in its initial development -- please report any issues.

ASAN

To build with the address sanitizer, use the -DMI_TRACK_ASAN=ON cmake option:

> cmake ../.. -DMI_TRACK_ASAN=ON

This can also be combined with secure mode or debug mode. You can then run your programs as:'

> ASAN_OPTIONS=verbosity=1 <myprogram>

When you link a program with an address sanitizer build of mimalloc, you should generally compile that program too with the address sanitizer enabled. For example, assuming you build mimalloc in out/debug:

clang -g -o test-wrong -Iinclude test/test-wrong.c out/debug/libmimalloc-asan-debug.a -lpthread -fsanitize=address -fsanitize-recover=address

Since the address sanitizer redirects the standard allocation functions, on some platforms (macOSX for example) it is required to compile mimalloc with -DMI_OVERRIDE=OFF. Address sanitizer support is in its initial development -- please report any issues.

ETW

Event tracing for Windows (ETW) provides a high performance way to capture all allocations though mimalloc and analyze them later. To build with ETW support, use the -DMI_TRACK_ETW=ON cmake option.

You can then capture an allocation trace using the Windows performance recorder (WPR), using the src/prim/windows/etw-mimalloc.wprp profile. In an admin prompt, you can use:

> wpr -start src\prim\windows\etw-mimalloc.wprp -filemode
> <my_mimalloc_program>
> wpr -stop <my_mimalloc_program>.etl

and then open <my_mimalloc_program>.etl in the Windows Performance Analyzer (WPA), or use a tool like TraceControl that is specialized for analyzing mimalloc traces.

Performance

Last update: 2021-01-30

We tested mimalloc against many other top allocators over a wide range of benchmarks, ranging from various real world programs to synthetic benchmarks that see how the allocator behaves under more extreme circumstances. In our benchmark suite, mimalloc outperforms other leading allocators (jemalloc, tcmalloc, Hoard, etc), and has a similar memory footprint. A nice property is that it does consistently well over the wide range of benchmarks.

General memory allocators are interesting as there exists no algorithm that is optimal -- for a given allocator one can usually construct a workload where it does not do so well. The goal is thus to find an allocation strategy that performs well over a wide range of benchmarks without suffering from (too much) underperformance in less common situations.

As always, interpret these results with care since some benchmarks test synthetic or uncommon situations that may never apply to your workloads. For example, most allocators do not do well on xmalloc-testN but that includes even the best industrial allocators like jemalloc and tcmalloc that are used in some of the world's largest systems (like Chrome or FreeBSD).

Also, the benchmarks here do not measure the behaviour on very large and long-running server workloads, or worst-case latencies of allocation. Much work has gone into mimalloc to work well on such workloads (for example, to reduce virtual memory fragmentation on long-running services) but such optimizations are not always reflected in the current benchmark suite.

We show here only an overview -- for more specific details and further benchmarks we refer to the technical report. The benchmark suite is automated and available separately as mimalloc-bench.

Benchmark Results on a 16-core AMD 5950x (Zen3)

Testing on the 16-core AMD 5950x processor at 3.4Ghz (4.9Ghz boost), with 32GiB memory at 3600Mhz, running Ubuntu 20.04 with glibc 2.31 and GCC 9.3.0.

We measure three versions of mimalloc: the main version mi (tag:v1.7.0), the new v2.0 beta version as xmi (tag:v2.0.0), and the main version in secure mode as smi (tag:v1.7.0).

The other allocators are Google's tcmalloc (tc, tag:gperftools-2.8.1) used in Chrome, Facebook's jemalloc (je, tag:5.2.1) by Jason Evans used in Firefox and FreeBSD, the Intel thread building blocks allocator (tbb, tag:v2020.3), rpmalloc (rp,tag:1.4.1) by Mattias Jansson, the original scalable Hoard (git:d880f72) allocator by Emery Berger [1], the memory compacting Mesh (git:67ff31a) allocator by Bobby Powers et al [8], and finally the default system allocator (glibc, 2.31) (based on PtMalloc2).

Any benchmarks ending in N run on all 32 logical cores in parallel. Results are averaged over 10 runs and reported relative to mimalloc (where 1.2 means it took 1.2× longer to run). The legend also contains the overall relative score between the allocators where 100 points is the maximum if an allocator is fastest on all benchmarks.

The single threaded cfrac benchmark by Dave Barrett is an implementation of continued fraction factorization which uses many small short-lived allocations. All allocators do well on such common usage, where mimalloc is just a tad faster than tcmalloc and jemalloc.

The leanN program is interesting as a large realistic and concurrent workload of the Lean theorem prover compiling its own standard library, and there is a 13% speedup over tcmalloc. This is quite significant: if Lean spends 20% of its time in the allocator that means that mimalloc is 1.6× faster than tcmalloc here. (This is surprising as that is not measured in a pure allocation benchmark like alloc-test. We conjecture that we see this outsized improvement here because mimalloc has better locality in the allocation which improves performance for the other computations in a program as well).

The single threaded redis benchmark again show that most allocators do well on such workloads.

The larsonN server benchmark by Larson and Krishnan [2] allocates and frees between threads. They observed this behavior (which they call bleeding) in actual server applications, and the benchmark simulates this. Here, mimalloc is quite a bit faster than tcmalloc and jemalloc probably due to the object migration between different threads.

The mstressN workload performs many allocations and re-allocations, and migrates objects between threads (as in larsonN). However, it also creates and destroys the N worker threads a few times keeping some objects alive beyond the life time of the allocating thread. We observed this behavior in many larger server applications.

The rptestN benchmark by Mattias Jansson is a allocator test originally designed for rpmalloc, and tries to simulate realistic allocation patterns over multiple threads. Here the differences between allocators become more apparent.

The second benchmark set tests specific aspects of the allocators and shows even more extreme differences between them.

The alloc-test, by OLogN Technologies AG, is a very allocation intensive benchmark doing millions of allocations in various size classes. The test is scaled such that when an allocator performs almost identically on alloc-test1 as alloc-testN it means that it scales linearly.

The sh6bench and sh8bench benchmarks are developed by MicroQuill as part of SmartHeap. In sh6bench mimalloc does much better than the others (more than 2.5× faster than jemalloc). We cannot explain this well but believe it is caused in part by the "reverse" free-ing pattern in sh6bench. The sh8bench is a variation with object migration between threads; whereas tcmalloc did well on sh6bench, the addition of object migration causes it to be 10× slower than before.

The xmalloc-testN benchmark by Lever and Boreham [5] and Christian Eder, simulates an asymmetric workload where some threads only allocate, and others only free -- they observed this pattern in larger server applications. Here we see that the mimalloc technique of having non-contended sharded thread free lists pays off as it outperforms others by a very large margin. Only rpmalloc, tbb, and glibc also scale well on this benchmark.

The cache-scratch benchmark by Emery Berger [1], and introduced with the Hoard allocator to test for passive-false sharing of cache lines. With a single thread they all perform the same, but when running with multiple threads the potential allocator induced false sharing of the cache lines can cause large run-time differences. Crundal [6] describes in detail why the false cache line sharing occurs in the tcmalloc design, and also discusses how this can be avoided with some small implementation changes. Only the tbb, rpmalloc and mesh allocators also avoid the cache line sharing completely, while Hoard and glibc seem to mitigate the effects. Kukanov and Voss [7] describe in detail how the design of tbb avoids the false cache line sharing.

On a 36-core Intel Xeon

For completeness, here are the results on a big Amazon c5.18xlarge instance consisting of a 2×18-core Intel Xeon (Cascade Lake) at 3.4GHz (boost 3.5GHz) with 144GiB ECC memory, running Ubuntu 20.04 with glibc 2.31, GCC 9.3.0, and Clang 10.0.0. This time, the mimalloc allocators (mi, xmi, and smi) were compiled with the Clang compiler instead of GCC. The results are similar to the AMD results but it is interesting to see the differences in the larsonN, mstressN, and xmalloc-testN benchmarks.

Peak Working Set

The following figure shows the peak working set (rss) of the allocators on the benchmarks (on the c5.18xlarge instance).

Note that the xmalloc-testN memory usage should be disregarded as it allocates more the faster the program runs. Similarly, memory usage of larsonN, mstressN, rptestN and sh8bench can vary depending on scheduling and speed. Nevertheless, we hope to improve the memory usage on mstressN and rptestN (just as cfrac, larsonN and sh8bench have a small working set which skews the results).

References

[1] Emery D. Berger, Kathryn S. McKinley, Robert D. Blumofe, and Paul R. Wilson. Hoard: A Scalable Memory Allocator for Multithreaded Applications the Ninth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS-IX). Cambridge, MA, November 2000. pdf
[2] P. Larson and M. Krishnan. Memory allocation for long-running server applications. In ISMM, Vancouver, B.C., Canada, 1998. pdf
[3] D. Grunwald, B. Zorn, and R. Henderson. Improving the cache locality of memory allocation. In R. Cartwright, editor, Proceedings of the Conference on Programming Language Design and Implementation, pages 177–186, New York, NY, USA, June 1993. pdf
[4] J. Barnes and P. Hut. A hierarchical O(n*log(n)) force-calculation algorithm. Nature, 324:446-449, 1986.
[5] C. Lever, and D. Boreham. Malloc() Performance in a Multithreaded Linux Environment. In USENIX Annual Technical Conference, Freenix Session. San Diego, CA. Jun. 2000. Available at https://github.com/kuszmaul/SuperMalloc/tree/master/tests
[6] Timothy Crundal. Reducing Active-False Sharing in TCMalloc. 2016. CS16S1 project at the Australian National University. pdf
[7] Alexey Kukanov, and Michael J Voss. The Foundations for Scalable Multi-Core Software in Intel Threading Building Blocks. Intel Technology Journal 11 (4). 2007
[8] Bobby Powers, David Tench, Emery D. Berger, and Andrew McGregor. Mesh: Compacting Memory Management for C/C++ In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI'19), June 2019, pages 333-–346.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

Older Release Notes

2024-05-21, v1.8.7, v2.1.7: Fix build issues on less common platforms. Started upstreaming patches from the CPython integration. Upstream vcpkg patches.
2024-05-13, v1.8.6, v2.1.6: Fix build errors on various (older) platforms. Refactored aligned allocation.
2024-04-22, v1.8.4, v2.1.4: Fixes various bugs and build issues. Add MI_LIBC_MUSL cmake flag for musl builds. Free-ing code is refactored into a separate module (free.c). Mimalloc page info is simplified with the block size directly available (and new block_size_shift to improve aligned block free-ing). New approach to collection of abandoned segments: When a thread terminates the segments it owns are abandoned (containing still live objects) and these can be reclaimed by other threads. We no longer use a list of abandoned segments but this is now done using bitmaps in arena's which is more concurrent (and more aggressive). Abandoned memory can now also be reclaimed if a thread frees an object in an abandoned page (which can be disabled using mi_option_abandoned_reclaim_on_free). The option mi_option_max_segment_reclaim gives a maximum percentage of abandoned segments that can be reclaimed per try (=10%).
2023-04-24, v1.8.2, v2.1.2: Fixes build issues on freeBSD, musl, and C17 (UE 5.1.1). Reduce code size/complexity by removing regions and segment-cache's and only use arenas with improved memory purging -- this may improve memory usage as well for larger services. Renamed options for consistency. Improved Valgrind and ASAN checking.
2023-04-03, v1.8.1, v2.1.1: Fixes build issues on some platforms.
2023-03-29, v1.8.0, v2.1.0: Improved support dynamic overriding on Windows 11. Improved tracing precision with asan and Valgrind, and added Windows event tracing ETW (contributed by Xinglong He). Created an OS abstraction layer to make it easier to port and separate platform dependent code (in src/prim). Fixed C++ STL compilation on older Microsoft C++ compilers, and various small bug fixes.
2022-12-23, v1.7.9, v2.0.9: Supports building with asan and improved Valgrind support. Support arbitrary large alignments (in particular for std::pmr pools). Added C++ STL allocators attached to a specific heap (thanks @vmarkovtsev). Heap walks now visit all object (including huge objects). Support Windows nano server containers (by Johannes Schindelin,@dscho). Various small bug fixes.
2022-11-03, v1.7.7, v2.0.7: Initial support for Valgrind for leak testing and heap block overflow detection. Initial support for attaching heaps to a specific memory area (only in v2). Fix realloc behavior for zero size blocks, remove restriction to integral multiple of the alignment in alloc_align, improved aligned allocation performance, reduced contention with many threads on few processors (thank you @dposluns!), vs2022 support, support pkg-config, .
2022-04-14, v1.7.6, v2.0.6: fix fallback path for aligned OS allocation on Windows, improve Windows aligned allocation even when compiling with older SDK's, fix dynamic overriding on macOS Monterey, fix MSVC C++ dynamic overriding, fix warnings under Clang 14, improve performance if many OS threads are created and destroyed, fix statistics for large object allocations, using MIMALLOC_VERBOSE=1 has no maximum on the number of error messages, various small fixes.
2022-02-14, v1.7.5, v2.0.5 (alpha): fix malloc override on Windows 11, fix compilation with musl, potentially reduced committed memory, add bin/minject for Windows, improved wasm support, faster aligned allocation, various small fixes.
2021-11-14, v1.7.3, v2.0.3 (beta): improved WASM support, improved macOS support and performance (including M1), improved performance for v2 for large objects, Python integration improvements, more standard installation directories, various small fixes.
2021-06-17, v1.7.2, v2.0.2 (beta): support M1, better installation layout on Linux, fix thread_id on Android, prefer 2-6TiB area for aligned allocation to work better on pre-windows 8, various small fixes.
2021-04-06, v1.7.1, v2.0.1 (beta): fix bug in arena allocation for huge pages, improved aslr on large allocations, initial M1 support (still experimental).
2021-01-31, v2.0.0: beta release 2.0: new slice algorithm for managing internal mimalloc pages.
2021-01-31, v1.7.0: stable release 1.7: support explicit user provided memory regions, more precise statistics, improve macOS overriding, initial support for Apple M1, improved DragonFly support, faster memcpy on Windows, various small fixes.
2020-09-24, v1.6.7: stable release 1.6: using standard C atomics, passing tsan testing, improved handling of failing to commit on Windows, add mi_process_info api call.
2020-08-06, v1.6.4: stable release 1.6: improved error recovery in low-memory situations, support for IllumOS and Haiku, NUMA support for Vista/XP, improved NUMA detection for AMD Ryzen, ubsan support.
2020-05-05, v1.6.3: stable release 1.6: improved behavior in out-of-memory situations, improved malloc zones on macOS, build PIC static libraries by default, add option to abort on out-of-memory, line buffered statistics.
2020-04-20, v1.6.2: stable release 1.6: fix compilation on Android, MingW, Raspberry, and Conda, stability fix for Windows 7, fix multiple mimalloc instances in one executable, fix strnlen overload, fix aligned debug padding.
2020-02-17, v1.6.1: stable release 1.6: minor updates (build with clang-cl, fix alignment issue for small objects).
2020-02-09, v1.6.0: stable release 1.6: fixed potential memory leak, improved overriding and thread local support on FreeBSD, NetBSD, DragonFly, and macOSX. New byte-precise heap block overflow detection in debug mode (besides the double-free detection and free-list corruption detection). Add nodiscard attribute to most allocation functions. Enable MIMALLOC_PAGE_RESET by default. New reclamation strategy for abandoned heap pages for better memory footprint.
2020-02-09, v1.5.0: stable release 1.5: improved free performance, small bug fixes.
2020-01-22, v1.4.0: stable release 1.4: improved performance for delayed OS page reset, more eager concurrent free, addition of STL allocator, fixed potential memory leak.
2020-01-15, v1.3.0: stable release 1.3: bug fixes, improved randomness and stronger free list encoding in secure mode.
2019-12-22, v1.2.2: stable release 1.2: minor updates.
2019-11-22, v1.2.0: stable release 1.2: bug fixes, improved secure mode (free list corruption checks, double free mitigation). Improved dynamic overriding on Windows.
2019-10-07, v1.1.0: stable release 1.1.
2019-09-01, v1.0.8: pre-release 8: more robust windows dynamic overriding, initial huge page support.
2019-08-10, v1.0.6: pre-release 6: various performance improvements.

uber/h3

Hexagonal hierarchical geospatial indexing system

H3: A Hexagonal Hierarchical Geospatial Indexing System

H3 is a geospatial indexing system using a hexagonal grid that can be (approximately) subdivided into finer and finer hexagonal grids, combining the benefits of a hexagonal grid with S2's hierarchical subdivisions.

Documentation is available at https://h3geo.org/. Developer documentation in Markdown format is available under the dev-docs directory.

Post bug reports or feature requests to the GitHub Issues page
Ask questions by posting to the H3 tag on StackOverflow
There is also an H3 Slack workspace

Installing

We recommend using prebuilt bindings if they are available for your programming language. Bindings for Java, JavaScript, Python, and others are available.

On macOS, you can install H3 using brew:

brew install h3

Otherwise, to build H3 from source, please see the following instructions.

Building from source

Still here? To build the H3 C library, you'll need a C compiler (tested with gcc and clang), CMake, and Make. If you intend to contribute to H3, you must have clang-format installed and we recommend installing ccmake and LCOV to configure the cmake arguments to build and run the tests and generate the code coverage report. We also recommend using gcc for the code coverage as some versions of clang generate annotations that aren't compatible with lcov. Doxygen is needed to build the API documentation.

Install build-time dependencies

Alpine

# Installing the bare build requirements
apk add cmake make gcc libtool musl-dev

Debian/Ubuntu

# Installing the bare build requirements
sudo apt install cmake make gcc libtool
# Installing useful tools for development
sudo apt install clang-format cmake-curses-gui lcov doxygen

macOS (using brew)

First make sure you have the developer tools installed and then

# Installing the bare build requirements
brew install cmake
# Installing useful tools for development
brew install clang-format lcov doxygen

Windows (Visual Studio)

You will need to install CMake and Visual Studio, including the Visual C++ compiler. For building on Windows, please follow the Windows build instructions.

FreeBSD

# Installing the build requirements
sudo pkg install bash cmake gmake doxygen lcov

Compilation

When checking out the H3 Git repository, by default you will check out the latest development version of H3. When using H3 in an application, you will want to check out the most recently released version:

git checkout v$(<VERSION)

From the repository root, you can compile H3 with:

mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
make

All subsequent make commands should be run from within the build directory.

Note: There are several ways to build H3 with CMake; the method above is just one example that restricts all build artifacts to the build directory.

You can install system-wide with:

sudo make install

If using the method above, from the repository root, you can clean all build artifacts with:

rm -rf build

Testing

After making the project, you can test with make test. You can run a faster test suite that excludes the most expensive tests with make test-fast.

Coverage

You can generate a code coverage report if lcov is installed, and if the project was built with the CMAKE_BUILD_TYPE=Debug and ENABLE_COVERAGE=ON options. For example, from a clean repository, you could run:

mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Debug -DENABLE_COVERAGE=ON ..
make
make coverage

You can then view a detailed HTML coverage report by opening coverage/index.html in your browser.

Benchmarks

You can run timing benchmarks by building with the CMAKE_BUILD_TYPE=Release, and running make benchmarks:

mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
make
make benchmarks

Documentation

You can build developer documentation with make docs if Doxygen was installed when CMake was run. Index of the documentation will be dev-docs/_build/html/index.html.

After making the project, you can build KML files to visualize the hexagon grid with make kml. The files will be placed in KML.

To build the documentation website, see the website/ directory.

Usage

From the command line

To get the H3 index for some location:

./bin/latLngToCell --resolution 10 --latitude 40.689167 --longitude -74.044444

10 is the H3 resolution, between 0 (coarsest) and 15 (finest). The coordinates entered are the latitude and longitude, in degrees, you want the index for (these coordinates are the Statue of Liberty). You should get an H3 index as output, like 8a2a1072b59ffff.

You can then take this index and get some information about it, for example:

./bin/cellToBoundary --index 8a2a1072b59ffff

This will produce the vertices of the hexagon at this location:

8a2a1072b59ffff
{
   40.690058601 -74.044151762
   40.689907695 -74.045061792
   40.689270936 -74.045341418
   40.688785091 -74.044711031
   40.688935993 -74.043801021
   40.689572744 -74.043521377
}

You can get the center coordinate of the hexagon like so:

./bin/cellToLatLng --index 8a2a1072b59ffff

This will produce some coordinate:

40.6894218437 -74.0444313999

From C

The above features of H3 can also be used from C. For example, you can compile and run examples/index.c like so:

cc -lh3 examples/index.c -o example
./example

You should get output like:

The index is: 8a2a1072b59ffff
Boundary vertex #0: 40.690059, -74.044152
Boundary vertex #1: 40.689908, -74.045062
Boundary vertex #2: 40.689271, -74.045341
Boundary vertex #3: 40.688785, -74.044711
Boundary vertex #4: 40.688936, -74.043801
Boundary vertex #5: 40.689573, -74.043521
Center coordinates: 40.689422, -74.044431

Contributing

Pull requests and Github issues are welcome. Please see our contributing guide for more information.

Before we can merge your changes, you must agree to the Uber Contributor License Agreement.

Legal and Licensing

H3 is licensed under the Apache 2.0 License.

telekom-security/tpotce

🍯 T-Pot - The All In One Multi Honeypot Platform 🐝

T-Pot - The All In One Multi Honeypot Platform

T-Pot is the all in one, optionally distributed, multiarch (amd64, arm64) honeypot plattform, supporting 20+ honeypots and countless visualization options using the Elastic Stack, animated live attack maps and lots of security tools to further improve the deception experience.

TL;DR

Meet the system requirements. The T-Pot installation needs at least 8-16 GB RAM, 128 GB free disk space as well as a working (outgoing non-filtered) internet connection.
Download or use a running, supported distribution.
Install the ISO with as minimal packages / services as possible (ssh required)
Install curl: $ sudo [apt, dnf, zypper] install curl if not installed already
Run installer as non-root from $HOME:

env bash -c "$(curl -sL https://github.com/telekom-security/tpotce/raw/master/install.sh)"

Follow instructions, read messages, check for possible port conflicts and reboot

Disclaimer

You install and run T-Pot within your responsibility. Choose your deployment wisely as a system compromise can never be ruled out.
For fast help research the Issues and Discussions.
The software is designed and offered with best effort in mind. As a community and open source project it uses lots of other open source software and may contain bugs and issues. Report responsibly.
Honeypots - by design - should not host any sensitive data. Make sure you don't add any.
By default, your data is submitted to Sicherheitstacho. You can disable this in the config (~/tpotce/docker-compose.yml) by removing the ewsposter section. But in this case sharing really is caring!

Technical Concept

T-Pot's main components have been moved into the tpotinit Docker image allowing T-Pot to now support multiple Linux distributions, even macOS and Windows (although both limited to the feature set of Docker Desktop). T-Pot uses docker and docker compose to reach its goal of running as many honeypots and tools as possible simultaneously and thus utilizing the host's hardware to its maximum.

Honeypots and Tools

T-Pot offers docker images for the following honeypots:
adbhoney, beelzebub, ciscoasa, citrixhoneypot, conpot, cowrie, ddospot, dicompot, dionaea, elasticpot, endlessh, galah, go-pot, glutton, h0neytr4p, hellpot, heralding, honeyaml, honeypots, honeytrap, ipphoney, log4pot, mailoney, medpot, miniprint, redishoneypot, sentrypeer, snare, tanner, wordpot

Alongside the following tools:

Autoheal a tool to automatically restart containers with failed healthchecks.
Cyberchef a web app for encryption, encoding, compression and data analysis.
Elastic Stack to beautifully visualize all the events captured by T-Pot.
Elasticvue a web front end for browsing and interacting with an Elasticsearch cluster.
Fatt a pyshark based script for extracting network metadata and fingerprints from pcap files and live network traffic.
T-Pot-Attack-Map a beautifully animated attack map for T-Pot.
P0f is a tool for purely passive traffic fingerprinting.
Spiderfoot an open source intelligence automation tool.
Suricata a Network Security Monitoring engine.

... to give you the best out-of-the-box experience possible and an easy-to-use multi-honeypot system.

Technical Architecture

The source code and configuration files are fully stored in the T-Pot GitHub repository. The docker images are built and preconfigured for the T-Pot environment.

The individual Dockerfiles and configurations are located in the docker folder.

Services

T-Pot offers a number of services which are basically divided into five groups:

System services provided by the OS
- SSH for secure remote access.
Elastic Stack
- Elasticsearch for storing events.
- Logstash for ingesting, receiving and sending events to Elasticsearch.
- Kibana for displaying events on beautifully rendered dashboards.
Tools
- NGINX provides secure remote access (reverse proxy) to Kibana, CyberChef, Elasticvue, GeoIP AttackMap, Spiderfoot and allows for T-Pot sensors to securely transmit event data to the T-Pot hive.
- CyberChef a web app for encryption, encoding, compression and data analysis.
- Elasticvue a web front end for browsing and interacting with an Elasticsearch cluster.
- T-Pot Attack Map a beautifully animated attack map for T-Pot.
- Spiderfoot an open source intelligence automation tool.
Honeypots
- A selection of the 23 available honeypots based on the selected docker-compose.yml.
Network Security Monitoring (NSM)
- Fatt a pyshark based script for extracting network metadata and fingerprints from pcap files and live network traffic.
- P0f is a tool for purely passive traffic fingerprinting.
- Suricata a Network Security Monitoring engine.

User Types

During the installation and during the usage of T-Pot there are two different types of accounts you will be working with. Make sure you know the differences of the different account types, since it is by far the most common reason for authentication errors.

Service	Account Type	Username / Group	Description
SSH	OS	`<OS_USERNAME>`	The user you chose during the installation of the OS.
Nginx	BasicAuth	`<WEB_USER>`	`<web_user>` you chose during the installation of T-Pot.
CyberChef	BasicAuth	`<WEB_USER>`	`<web_user>` you chose during the installation of T-Pot.
Elasticvue	BasicAuth	`<WEB_USER>`	`<web_user>` you chose during the installation of T-Pot.
Geoip Attack Map	BasicAuth	`<WEB_USER>`	`<web_user>` you chose during the installation of T-Pot.
Spiderfoot	BasicAuth	`<WEB_USER>`	`<web_user>` you chose during the installation of T-Pot.
T-Pot	OS	`tpot`	`tpot` this user / group is always reserved by the T-Pot services.
T-Pot Logs	BasicAuth	`<LS_WEB_USER>`	`LS_WEB_USER` are automatically managed.

System Requirements

Depending on the supported Linux distro images, hive / sensor, installing on real hardware, in a virtual machine or other environments there are different kind of requirements to be met regarding OS, RAM, storage and network for a successful installation of T-Pot (you can always adjust ~/tpotce/docker-compose.yml and ~/tpotce/.envto your needs to overcome these requirements).

T-Pot Type	RAM	Storage	Description
Hive	16GB	256GB SSD	As a rule of thumb, the more honeypots, sensors & data, the more RAM and storage is needed.
Sensor	8GB	128GB SSD	Since honeypot logs are persisted (~/tpotce/data) for 30 days, storage depends on attack volume.

T-Pot does require ...

an IPv4 address via DHCP or statically assigned
a working, non-proxied, internet connection ... for a successful installation and operation.

If you need proxy support or otherwise non-standard features, you should check the docs of the supported Linux distro images and / or the Docker documentation.

Running in a VM

All of the supported Linux distro images will run in a VM which means T-Pot will just run fine. The following were tested / reported to work:

Some configuration / setup hints:

While Intel versions run stable, Apple Silicon (arm64) support has known issues which in UTM may require switching Display to Console Only during initial installation of the OS and afterwards back to Full Graphics.
During configuration you may need to enable promiscuous mode for the network interface in order for fatt, suricata and p0f to work properly.
If you want to use a wifi card as a primary NIC for T-Pot, please be aware that not all network interface drivers support all wireless cards. In VirtualBox e.g. you have to choose the "MT SERVER" model of the NIC.

Running on Hardware

T-Pot is only limited by the hardware support of the supported Linux distro images. It is recommended to check the HCL (hardware compatibility list) and test the supported distros with T-Pot before investing in dedicated hardware.

Running in a Cloud

T-Pot is tested on and known to run on ...

Telekom OTC using the post install method ... others may work, but remain untested.

Some users report working installations on other clouds and hosters, i.e. Azure and GCP. Hardware requirements may be different. If you are unsure you should research issues and discussions and run some functional tests. With T-Pot 24.04.0 and forward we made sure to remove settings that were known to interfere with cloud based installations.

Required Ports

Besides the ports generally needed by the OS, i.e. obtaining a DHCP lease, DNS, etc. T-Pot will require the following ports for incoming / outgoing connections. Review the T-Pot Architecture for a visual representation. Also some ports will show up as duplicates, which is fine since used in different editions.

Port	Protocol	Direction	Description
80, 443	tcp	outgoing	T-Pot Management: Install, Updates, Logs (i.e. OS, GitHub, DockerHub, Sicherheitstacho, etc.
11434	tcp	outgoing	LLM based honeypots: Access your Ollama installation
64294	tcp	incoming	T-Pot Management: Sensor data transmission to hive (through NGINX reverse proxy) to 127.0.0.1:64305
64295	tcp	incoming	T-Pot Management: Access to SSH
64297	tcp	incoming	T-Pot Management Access to NGINX reverse proxy
5555	tcp	incoming	Honeypot: ADBHoney
22	tcp	incoming	Honeypot: Beelzebub (LLM required)
5000	udp	incoming	Honeypot: CiscoASA
8443	tcp	incoming	Honeypot: CiscoASA
443	tcp	incoming	Honeypot: CitrixHoneypot
80, 102, 502, 1025, 2404, 10001, 44818, 47808, 50100	tcp	incoming	Honeypot: Conpot
161, 623	udp	incoming	Honeypot: Conpot
22, 23	tcp	incoming	Honeypot: Cowrie
19, 53, 123, 1900	udp	incoming	Honeypot: Ddospot
11112	tcp	incoming	Honeypot: Dicompot
21, 42, 135, 443, 445, 1433, 1723, 1883, 3306, 8081	tcp	incoming	Honeypot: Dionaea
69	udp	incoming	Honeypot: Dionaea
9200	tcp	incoming	Honeypot: Elasticpot
22	tcp	incoming	Honeypot: Endlessh
80, 443, 8080, 8443	tcp	incoming	Honeypot: Galah (LLM required)
8080	tcp	incoming	Honeypot: Go-pot
80, 443	tcp	incoming	Honeypot: H0neytr4p
21, 22, 23, 25, 80, 110, 143, 443, 993, 995, 1080, 5432, 5900	tcp	incoming	Honeypot: Heralding
3000	tcp	incoming	Honeypot: Honeyaml
21, 22, 23, 25, 80, 110, 143, 389, 443, 445, 631, 1080, 1433, 1521, 3306, 3389, 5060, 5432, 5900, 6379, 6667, 8080, 9100, 9200, 11211	tcp	incoming	Honeypot: qHoneypots
53, 123, 161, 5060	udp	incoming	Honeypot: qHoneypots
631	tcp	incoming	Honeypot: IPPHoney
80, 443, 8080, 9200, 25565	tcp	incoming	Honeypot: Log4Pot
25	tcp	incoming	Honeypot: Mailoney
2575	tcp	incoming	Honeypot: Medpot
9100	tcp	incoming	Honeypot: Miniprint
6379	tcp	incoming	Honeypot: Redishoneypot
5060	tcp/udp	incoming	Honeypot: SentryPeer
80	tcp	incoming	Honeypot: Snare (Tanner)
8090	tcp	incoming	Honeypot: Wordpot

Ports and availability of SaaS services may vary based on your geographical location.

For some honeypots to reach full functionality (i.e. Cowrie or Log4Pot) outgoing connections are necessary as well, in order for them to download the attacker's malware. Please see the individual honeypot's documentation to learn more by following the links to their repositories.

LLM-Based Honeypots

We think LLM-Based Honeypots mark the beginning of a game change for the deception / honeypot field. Consequently, starting with the release of T-Pot 24.04.1, two LLM-based honeypots, Beelzebub and Galah, have been introduced. These honeypots require an installation of Ollama, which needs to be configured in the T-Pot configuration file. You can also adjust the settings in this file for ChatGPT support, but note that changes will also be required in the docker compose file (~/tpotce/compose/llm.yml) to accommodate these adjustments.

Follow the links in the Honeypots and Tools section to find out more about Beelzebub and Galah.

Ollama

🚨 CPU-based usage is not recommended, not even for testing.

To set up and run Ollama, refer to the Ollama GitHub repository for instructions. For entry-level or testing purposes, results can be achieved using a Nvidia RTX 4060 Ti 16GB or equivalent (AMD's ROCm is also supported by Ollama), with models like openchat and Llama3. As a general rule with LLM-based systems, the better and more hardware you use, the faster and more accurate the results will be, especially when tasks are offloaded to multiple GPUs and larger models.

ChatGPT

ChatGPT support for these honeypots will remain untested in relation to T-Pot.

System Placement

It is recommended to get yourself familiar with how T-Pot and the honeypots work before you start exposing towards the internet. For a quickstart run a T-Pot installation in a virtual machine.

Once you are familiar with how things work you should choose a network you suspect intruders in or from (i.e. the internet). Otherwise T-Pot will most likely not capture any attacks (unless you want to prove a point)! For starters it is recommended to put T-Pot in an unfiltered zone, where all TCP and UDP traffic is forwarded to T-Pot's network interface. To avoid probing for T-Pot's management ports you should put T-Pot behind a firewall and forward all TCP / UDP traffic in the port range of 1-64000 to T-Pot while allowing access to ports > 64000 only from trusted IPs and / or only expose the ports relevant to your use-case. If you wish to catch malware traffic on unknown ports you should not limit the ports you forward since glutton and honeytrap dynamically bind any TCP port that is not occupied by other honeypot daemons and thus give you a better representation of the risks your setup is exposed to.

Installation

Download one of the supported Linux distro images, follow the TL;DR instructions or git clone the T-Pot repository and run the installer ~/tpotce/install.sh. Running T-Pot on top of a running and supported Linux system is possible, but a clean installation is recommended to avoid port conflicts with running services. The T-Pot installer will require direct access to the internet as described here.

Choose your distro

Steps to Follow:

Download a supported Linux distribution from the list below. (NOTE: Red Hat Enterprise Linux >= 8 is supported, but omitted from the list below due to its subscription-based nature. See Red Hat Enterprise Linux for details).
During installation choose a minimum, netinstall or server version that will only install essential packages.
Never install a graphical desktop environment such as Gnome or KDE. T-Pot will fail to work with it due to port conflicts.
Make sure to install SSH, so you can connect to the machine remotely.

Distribution Name	x64	arm64
Alma Linux OS 9.x Minimal ISO	download	download
Debian 13 Network Install	download	download
Fedora Server 42 Network Install	download	download
OpenSuse Tumbleweed Network Image	download	download
Rocky Linux OS 9.x Minimal ISO	download	download
Ubuntu 24.04.x Live Server	download	download

Raspberry Pi 4 (8GB) Support

Distribution Name	arm64
Raspberry Pi OS (64Bit, Lite)	download

Get and install T-Pot

Clone the GitHub repository: $ git clone https://github.com/telekom-security/tpotce or follow the TL;DR and skip this section.
Change into the tpotce/ folder: $ cd tpotce
Run the installer as non-root: $ ./install.sh:
- ⚠️ Depending on your Linux distribution of choice the installer will:
  - Change the SSH port to tcp/64295
  - Disable the DNS Stub Listener to avoid port conflicts with honeypots
  - Set SELinux to Monitor Mode
  - Set the firewall target for the public zone to ACCEPT
  - Add Docker's repository and install Docker
  - Install recommended packages
  - Remove packages known to cause issues
  - Add the current user to the docker group (allow docker interaction without sudo)
  - Add dps and dpsw aliases (grc docker ps -a, watch -c "grc --colour=on docker ps -a)
  - Add la, ll and ls aliases (for exa, a improved ls command)
  - Add mi (for micro, a great alternative to vi and / or nano)
  - Display open ports on the host (compare with T-Pot required ports)
  - Add and enable tpot.service to /etc/systemd/system so T-Pot can automatically start and stop
Follow the installer instructions, you will have to enter your user (sudo or root) password at least once
Check the installer messages for errors and open ports that might cause port conflicts
Reboot: $ sudo reboot

macOS & Windows

Sometimes it is just nice if you can spin up a T-Pot instance on macOS or Windows, i.e. for development, testing or just the fun of it. As Docker Desktop is rather limited not all honeypot types or T-Pot features are supported. Also remember, by default the macOS and Windows firewall are blocking access from remote, so testing is limited to the host. For production it is recommended to run T-Pot on Linux.
To get things up and running just follow these steps:

Install Docker Desktop for macOS or Windows.
Clone the GitHub repository: git clone https://github.com/telekom-security/tpotce (in Windows make sure the code is checked out with LF instead of CRLF!)
Go to: cd ~/tpotce
Copy cp compose/mac_win.yml ./docker-compose.yml
Create a WEB_USER by running ~/tpotce/genuser.sh (macOS) or ~/tpotce/genuserwin.ps1 (Windows)

Adjust the .env file by changing TPOT_OSTYPE=linux to either mac or win:

# OSType (linux, mac, win)
#  Most docker features are available on linux
TPOT_OSTYPE=mac

You have to ensure on your own there are no port conflicts keeping T-Pot from starting up.
Start T-Pot: docker compose up or docker compose up -d if you want T-Pot to run in the background.
Stop T-Pot: CTRL-C (it if was running in the foreground) and / or docker compose down -v to stop T-Pot entirely.

Red Hat Enterprise Linux

Red Hat Enterprise Linux (RHEL) is a somewhat unique case in that:

Connections to Red Hat repositories depend on a Red Hat subscription. You will not be able to update the OS or install new packages if the targeted machine is not subscribed. If your server is not attached to a Red Hat subscription, installation will fail!
Ansible is installed from a RHEL-specific repository by the installer. Do not attempt to install it from the upstream repositories.
Docker is installed from EPEL, which is installed by the installer script. Do not attempt to install it from the community installer script.
T-Pot will only install successfully on RHEL >= 8. One of the convenience dependencies (grc) depends on Python 2, which was removed after RHEL 7. It is omitted from the RHEL installation of T-Pot.

Installation Types

Standard / Hive

With T-Pot Standard / Hive all services, tools, honeypots, etc. will be installed on to a single host which also serves as a Hive endpoint. Make sure to meet the system requirements. You can adjust ~/tpotce/docker-compose.yml to your personal use-case or create your very own configuration using ~/tpotce/compose/customizer.py for a tailored T-Pot experience to your needs. Once the installation is finished you can proceed to First Start.

Distributed

The distributed version of T-Pot requires at least two hosts

the T-Pot Hive, the standard installation of T-Pot (install this first!),
and a T-Pot Sensor, which will host only the honeypots, some tools and transmit log data to the Hive.
The Sensor will not start before finalizing the Sensor installation as described in Distributed Deployment.

Uninstall T-Pot

Uninstallation of T-Pot is only available on the supported Linux distros.
To uninstall T-Pot run ~/tpotce/uninstall.sh and follow the uninstaller instructions, you will have to enter your password at least once.
Once the uninstall is finished reboot the machine sudo reboot

First Start

Once the T-Pot Installer successfully finishes, the system needs to be rebooted (sudo reboot). Once rebooted you can log into the system using the user you setup during the installation of the system. Logins are according to the User Types:

user: [<OS_USERNAME>]
pass: [password]

You can login via SSH to access the command line: ssh -l <OS_USERNAME> -p 64295 <your.ip>:

user: [<OS_USERNAME>]
pass: [password, ssh key recommended]

You can also login from your browser and access the T-Pot WebUI and tools: https://<your.ip>:64297

user: [<WEB_USER>]
pass: [password]

Standalone First Start

There is not much to do except to login and check via dps if all services and honeypots are starting up correctly and login to Kibana and / or Geoip Attack Map to monitor the attacks.

Distributed Deployment

Planning and Certificates

The distributed deployment involves planning as T-Pot Init will only create a self-signed certificate for the IP of the Hive host which usually is suitable for simple setups. Since logstash will check for a valid certificate upon connection, a distributed setup involving Hive to be reachable on multiple IPs (i.e. RFC 1918 and public NAT IP) and maybe even a domain name will result in a connection error where the certificate cannot be validated as such a setup needs a certificate with a common name and SANs (Subject Alternative Name).
Before deploying any sensors make sure you have planned out domain names and IPs properly to avoid issues with the certificate. For more details see issue #1543.
Adjust the example to your IP / domain setup and follow the commands to change the certificate of Hive:

sudo systemctl stop tpot

sudo openssl req \
    -nodes \
    -x509 \
    -sha512 \
    -newkey rsa:8192 \
    -keyout "$HOME/tpotce/data/nginx/cert/nginx.key" \
    -out "$HOME/tpotce/data/nginx/cert/nginx.crt" \
    -days 3650 \
    -subj '/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd' \
    -addext "subjectAltName = IP:192.168.1.200, IP:1.2.3.4, DNS:my.primary.domain, DNS:my.secondary.domain"
    
sudo chmod 774 $HOME/tpotce/data/nginx/cert/*
sudo chown tpot:tpot $HOME/tpotce/data/nginx/cert/*

sudo systemctl start tpot

The T-Pot configuration file (.env) does allow to disable the SSL verification for logstash connections from Sensor to the Hive by setting LS_SSL_VERIFICATION=none. For security reasons this is only recommended for lab or test environments.

If you choose to use a valid certificate for the Hive signed by a CA (i.e. Let's Encrypt), logstash, and therefore the Sensor, should have no problems to connect and transmit its logs to the Hive.

Deploying Sensors

Once you have rebooted the Sensor as instructed by the installer you can continue with the distributed deployment by logging into Hive and go to cd ~/tpotce folder. Make sure you understood the Planning and Certificates before continuing with the actual deployment.

If you have not done already generate a SSH key to securely login to the Sensor and to allow Ansible to run a playbook on the sensor:

Run ssh-keygen, follow the instructions and leave the passphrase empty:

Generating public/private rsa key pair.
Enter file in which to save the key (/home/<your_user>/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/<your_user>/.ssh/id_rsa
Your public key has been saved in /home/<your_user>/.ssh/id_rsa.pub

Deploy the key to the Sensor by running ssh-copy-id -p 64295 <Sensor_SSH_USER>@<Sensor_IP>):

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/<your_user>/.ssh/id_rsa.pub"
The authenticity of host '[<Sensor_IP>]:64295 ([<Sensor_IP>]:64295)' can't be stablished.
ED25519 key fingerprint is SHA256:naIDxFiw/skPJadTcgmWZQtgt+CdfRbUCoZn5RmkOnQ.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
<your_user>@172.20.254.124's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh -p '64295' '<your_user>@<Sensor_IP>'"
and check to make sure that only the key(s) you wanted were added.

As suggested follow the instructions to test the connection ssh -p '64295' '<your_user>@<Sensor_IP>'.
Once the key is successfully deployed run ./deploy.sh and follow the instructions.

Removing Sensors

Identify the TPOT_HIVE_USER ENV on the Sensor in the $HOME/tpotce/.env config (it is a base64 encoded string). Now identify the same string in the LS_WEB_USER ENV on the Hive in the $HOME/tpotce/.env config. Remove the string and restart T-Pot.
Now you can safely delete the Sensor machine.

Community Data Submission

T-Pot is provided in order to make it accessible to everyone interested in honeypots. By default, the captured data is submitted to a community backend. This community backend uses the data to feed Sicherheitstacho. You may opt out of the submission by removing the # Ewsposter service from ~/tpotce/docker-compose.yml by following these steps:

Stop T-Pot services: systemctl stop tpot
Open ~/tpotce/docker-compose.yml: micro ~/tpotce/docker-compose.yml
Remove the following lines, save and exit micro (CTRL+Q):

# Ewsposter service
  ewsposter:
    container_name: ewsposter
    restart: always
    depends_on:
      tpotinit:
        condition: service_healthy
    networks:
     - ewsposter_local
    environment:
     - EWS_HPFEEDS_ENABLE=false
     - EWS_HPFEEDS_HOST=host
     - EWS_HPFEEDS_PORT=port
     - EWS_HPFEEDS_CHANNELS=channels
     - EWS_HPFEEDS_IDENT=user
     - EWS_HPFEEDS_SECRET=secret
     - EWS_HPFEEDS_TLSCERT=false
     - EWS_HPFEEDS_FORMAT=json
    image: ${TPOT_REPO}/ewsposter:${TPOT_VERSION}
    pull_policy: ${TPOT_PULL_POLICY}
    volumes:
     - ${TPOT_DATA_PATH}:/data
     - ${TPOT_DATA_PATH}/ews/conf/ews.ip:/opt/ewsposter/ews.ip

Start T-Pot services: systemctl start tpot

It is encouraged not to disable the data submission as it is the main purpose of the community approach - as you all know sharing is caring 😍

Opt-In HPFEEDS Data Submission

As an Opt-In it is possible to share T-Pot data with 3rd party HPFEEDS brokers.

Follow the instructions here to stop the T-Pot services and open ~/tpotce/docker-compose.yml.
Scroll down to the ewsposter section and adjust the HPFEEDS settings to your needs.
If you need to add a CA certificate add it to ~/tpotce/data/ews/conf and set EWS_HPFEEDS_TLSCERT=/data/ews/conf/<your_ca.crt>.
Start T-Pot services: systemctl start tpot.

Remote Access and Tools

Remote access to your host / T-Pot is possible with SSH (on tcp/64295) and some services and tools come with T-Pot to make some of your research tasks a lot easier.

SSH

According to the User Types you can login via SSH to access the command line: ssh -l <OS_USERNAME> -p 64295 <your.ip>:

user: [<OS_USERNAME>]
pass: [password]

T-Pot Landing Page

According to the User Types you can open the T-Pot Landing Page from your browser via https://<your.ip>:64297:

user: [<WEB_USER>]
pass: [password]

Kibana Dashboard

On the T-Pot Landing Page just click on Kibana and you will be forwarded to Kibana. You can select from a large variety of dashboards and visualizations all tailored to the T-Pot supported honeypots.

Attack Map

On the T-Pot Landing Page just click on Attack Map and you will be forwarded to the Attack Map. Since the Attack Map utilizes web sockets you may need to re-enter the <WEB_USER> credentials.

Cyberchef

On the T-Pot Landing Page just click on Cyberchef and you will be forwarded to Cyberchef.

Elasticvue

On the T-Pot Landing Page just click on Elasticvue and you will be forwarded to Elasticvue.

Spiderfoot

On the T-Pot Landing Page just click on Spiderfoot and you will be forwarded to Spiderfoot.

Configuration

T-Pot Config File

T-Pot offers a configuration file providing variables not only for the docker services (i.e. honeypots and tools) but also for the docker compose environment. The configuration file is hidden in ~/tpoce/.env. There is also an example file (env.example) which holds the default configuration.
Before the first start run ~/tpotce/genuser.sh or setup the WEB_USER manually as described here.

Customize T-Pot Honeypots and Services

In ~/tpotce/compose you will find everything you need to adjust the T-Pot Standard / Hive installation:

customizer.py
llm.yml
mac_win.yml
mini.yml
mobile.yml
sensor.yml
standard.yml
tarpit.yml
tpot_services.yml

The .yml files are docker compose files, each representing a different set of honeypots and tools with tpot_services.yml being a template for customizer.py to create a customized docker compose file.

To activate a compose file follow these steps:

Stop T-Pot with systemctl stop tpot.
Copy the docker compose file cp ~/tpotce/compose/<dockercompose.yml> ~/tpotce/docker-compose.yml.
Start T-Pot with systemctl start tpot.

To create your customized docker compose file:

Go to cd ~/tpotce/compose.
Run python3 customizer.py.
The script will guide you through the process of creating your own docker-compose.yml. As some honeypots and services occupy the same ports it will check if any port conflicts are present and notify regarding the conflicting services. You then can resolve them manually by adjusting docker-compose-custom.yml or re-run the script.
Stop T-Pot with systemctl stop tpot.
Copy the custom docker compose file: cp docker-compose-custom.yml ~/tpotce and cd ~/tpotce.
Check if everything works by running docker-compose -f docker-compose-custom.yml up. In case of errors follow the Docker Compose Specification for mitigation. Most likely it is just a port conflict you can adjust by editing the docker compose file.
If everything works just fine press CTRL-C to stop the containers and run docker-compose -f docker-compose-custom.yml down -v.
Replace docker compose file with the new and successfully tested customized docker compose file mv ~/tpotce/docker-compose-custom.yml ~/tpotce/docker-compose.yml.
Start T-Pot with systemctl start tpot.

Maintenance

T-Pot is designed to be low maintenance. Since almost everything is provided through docker images there is basically nothing you have to do but let it run. We will upgrade the docker images regularly to reduce the risks of compromise; however you should read this section closely.

Should an update fail, opening an issue or a discussion will help to improve things in the future, but the offered solution will always be to perform a fresh install as we simply cannot provide any support for lost data!

General Updates

T-Pot security depends on the updates provided for the supported Linux distro images. Make sure to review the OS documentation and ensure updates are installed regularly by the OS. By default (~/tpotce/.env) TPOT_PULL_POLICY=always will ensure that at every T-Pot start docker will check for new docker images and download them before creating the containers.

Update Script

T-Pot releases are offered through GitHub and can be pulled using ~/tpotce/update.sh.
If you made any relevant changes to the T-Pot config files make sure to create a backup first!
Updates may have unforeseen consequences. Create a backup of the machine or the files most valuable to your work!

The update script will ...

mercilessly overwrite local changes to be in sync with the T-Pot master branch
create a full backup of the ~/tpotce folder
update all files in ~/tpotce to be in sync with the T-Pot master branch
restore your custom ews.cfg from ~/tpotce/data/ews/conf and the T-Pot configuration (~/tpotce/.env).

Daily Reboot

By default T-Pot will add a daily reboot including some cleaning up. You can adjust this line with sudo crontab -e

#Ansible: T-Pot Daily Reboot
42 2 * * * bash -c 'systemctl stop tpot.service && docker container prune -f; docker image prune -f; docker volume prune -f; /usr/sbin/shutdown -r +1 "T-Pot Daily Reboot"'

Known Issues

The following issues are known, simply follow the described steps to solve them.

Docker Images Fail to Download

Some time ago Docker introduced download rate limits. If you are frequently downloading Docker images via a single or shared IP, the IP address might have exhausted the Docker download rate limit. Login to your Docker account to extend the rate limit.

sudo su -
docker login

T-Pot Networking Fails

T-Pot is designed to only run on machines with a single NIC. T-Pot will try to grab the interface with the default route, however it is not guaranteed that this will always succeed. At best use T-Pot on machines with only a single NIC.

Start T-Pot

The T-Pot service automatically starts and stops on each reboot (which occurs once on a daily basis as setup in sudo crontab -l during installation).
If you want to manually start the T-Pot service you can do so via systemctl start tpot and observe via dpsw the startup of the containers.

Stop T-Pot

The T-Pot service automatically starts and stops on each reboot (which occurs once on a daily basis as setup in sudo crontab -l during installation).
If you want to manually stop the T-Pot service you can do so via systemctl stop tpot and observe via dpsw the shutdown of the containers.

T-Pot Data Folder

All persistent log files from the honeypots, tools and T-Pot related services are stored in ~/tpotce/data. This includes collected artifacts which are not transmitted to the Elastic Stack.

Log Persistence

All log data is stored in the T-Pot Data Folder and will be persisted for the number of cycles set for TPOT_PERSISTENCE_CYCLES=<1-999> in the T-Pot configuration file ~/tpotce/.env. It defaults to 30.
Elasticsearch indices are handled by the tpot Index Lifecycle Policy which can be adjusted directly in Kibana (make sure to "Include managed system policies").

By default the tpot Index Lifecycle Policy keeps the indices for 30 days. This offers a good balance between storage and speed. However you may adjust the policy to your needs.

Factory Reset

All log data stored in the T-Pot Data Folder (except for Elasticsearch indices, of course) can be erased by running clean.sh. Sometimes things might break beyond repair and it has never been easier to reset a T-Pot to factory defaults (make sure to enter cd ~/tpotce).

Stop T-Pot using systemctl stop tpot.
Move / Backup the ~/tpotce/data folder to a safe place (this is optional, just in case).
Delete the ~/tpotce/data folder using sudo rm -rf ~/tpotce/data.
Reset T-Pot to the last fetched commit:

cd ~/tpotce/
git reset --hard

Now you can run ~/tpotce/install.sh.

Show Containers

You can show all T-Pot relevant containers by running dps or dpsw [interval]. The interval (s) will re-run dps periodically.

Blackhole

Blackhole will run T-Pot in kind of a stealth mode manner without permanent visits of publicly known scanners and thus reducing the possibility of being exposed. While this is of course always a cat and mouse game the blackhole feature is null routing all requests from known mass scanners while still catching the events through Suricata.
The feature is activated by setting TPOT_BLACKHOLE=DISABLED in ~/tpotce/.env, then run systemctl stop tpot and systemctl start tpot or sudo reboot.
Enabling this feature will drastically reduce attackers visibility and consequently result in less activity. However as already mentioned it is neither a guarantee for being completely stealth nor will it prevent fingerprinting of some honeypot services.

Add Users to Nginx (T-Pot WebUI)

Nginx (T-Pot WebUI) allows you to add as many <WEB_USER> accounts as you want (according to the User Types).
To add a new user run ~/tpotce/genuser.sh.
To remove users open ~/tpotce/.env, locate WEB_USER and remove the corresponding base64 string (to decode: echo <base64_string> | base64 -d, or open CyberChef and load "From Base64" recipe).
For the changes to take effect you need to restart T-Pot using systemctl stop tpot and systemctl start tpot or sudo reboot.

Import and Export Kibana Objects

Some T-Pot updates will require you to update the Kibana objects. Either to support new honeypots or to improve existing dashboards or visualizations. Make sure to export first so you do not loose any of your adjustments.

Export

Go to Kibana
Click on "Stack Management"
Click on "Saved Objects"
Click on "Export <no.> objects"
Click on "Export all" This will export a NDJSON file with all your objects. Always run a full export to make sure all references are included.

Import

Download the NDJSON file and unzip it.
Go to Kibana
Click on "Stack Management"
Click on "Saved Objects"
Click on "Import" and leave the defaults (check for existing objects and automatically overwrite conflicts) if you did not make personal changes to the Kibana objects.
Browse for NDJSON file When asked: "If any of the objects already exist, do you want to automatically overwrite them?" you answer with "Yes, overwrite all".

Troubleshooting

Generally T-Pot is offered as is without any commitment regarding support. Issues and discussions can be opened, but be prepared to include basic necessary info, so the community is able to help.

Logs

Check if your containers are running correctly: dps
Check if your system resources are not exhausted: htop, docker stats
Check if there is a port conflict:

systemctl stop tpot
grc netstat -tulpen
mi ~/tpotce/docker-compose.yml
docker-compose -f ~/tpotce/docker-compose.yml up
CTRL+C
docker-compose -f ~/tpotce/docker-compose.yml down -v

Check individual container logs: docker logs -f <container_name>
Check tpotinit log: cat ~/tpotce/data/tpotinit.log

RAM and Storage

The Elastic Stack is hungry for RAM, specifically logstash and elasticsearch. If the Elastic Stack is unavailable, does not receive any logs or simply keeps crashing it is most likely a RAM or storage issue.
While T-Pot keeps trying to restart the services / containers run docker logs -f <container_name> (either logstash or elasticsearch) and check if there are any warnings or failures involving RAM.

Storage failures can be identified easier via htop.

Contact

T-Pot is provided as is open source without any commitment regarding support (see the disclaimer).

If you are a security researcher and want to responsibly report an issue please get in touch with our CERT.

Issues

Please report issues (errors) on our GitHub Issues, but troubleshoot first. Issues not providing information to address the error will be closed or converted into discussions.

Use the search function first, it is possible a similar issue has been addressed or discussed already, with the solution just a search away.

Discussions

General questions, ideas, show & tell, etc. can be addressed on our GitHub Discussions.

Use the search function, it is possible a similar discussion has been opened already, with an answer just a search away.

Licenses

The software that T-Pot is built on uses the following licenses.
GPLv2: conpot, galah, dionaea, honeytrap, suricata
GPLv3: adbhoney, elasticpot, ewsposter, log4pot, fatt, heralding, ipphoney, miniprint, redishoneypot, sentrypeer, snare, tanner
Apache 2 License: cyberchef, dicompot, elasticsearch, go-pot, h0neytr4p, logstash, kibana, docker
MIT license: autoheal, beelzebub, ciscoasa, ddospot, elasticvue, glutton, hellpot, honeyaml, maltrail
Unlicense: endlessh
Other: citrixhoneypot, cowrie, mailoney, Elastic License, Wordpot
AGPL-3.0: honeypots
Public Domain (CC): Harvard Dataverse

Credits

Without open source and the development community we are proud to be a part of, T-Pot would not have been possible! Our thanks are extended but not limited to the following people and organizations:

The developers and development communities of

adbhoney, beelzebub, ciscoasa, citrixhoneypot, conpot, cowrie, ddospot, dicompot, dionaea, docker, elasticpot, elasticsearch, elasticvue, endlessh, ewsposter, fatt, galah, glutton, go-pot, h0neytr4p, hellpot, heralding, honeyaml, honeypots, honeytrap, ipphoney, kibana, logstash, log4pot, mailoney, maltrail, medpot, miniprint, p0f, redishoneypot, sentrypeer, spiderfoot, snare, tanner, suricata, wordpot

The following companies and organizations

docker, elastic.io, honeynet project

And of course YOU for joining the community!

Testimonials

One of the greatest feedback we have gotten so far is by one of the Conpot developers:
"[...] I highly recommend T-Pot which is ... it's not exactly a swiss army knife .. it's more like a swiss army soldier, equipped with a swiss army knife. Inside a tank. A swiss tank. [...]"

And from @robcowart (creator of ElastiFlow):
"#TPot is one of the most well put together turnkey honeypot solutions. It is a must-have for anyone wanting to analyze and understand the behavior of malicious actors and the threat they pose to your organization."

Thank you 💖

redis/redis

For developers, who are building real-time data-driven applications, Redis is the preferred, fastest, and most feature-rich cache, data structure server, and document and vector query engine.

This document serves as both a quick start guide to Redis and a detailed resource for building it from source.

New to Redis? Start with What is Redis and Getting Started
Ready to build from source? Jump to Build Redis from Source
Want to contribute? See the Code contributions section and CONTRIBUTING.md
Looking for detailed documentation? Navigate to redis.io/docs

What is Redis?

For developers, who are building real-time data-driven applications, Redis is the preferred, fastest, and most feature-rich cache, data structure server, and document and vector query engine.

Key use cases

Redis excels in various applications, including:

Caching: Supports multiple eviction policies, key expiration, and hash-field expiration.
Distributed Session Store: Offers flexible session data modeling (string, JSON, hash).
Data Structure Server: Provides low-level data structures (strings, lists, sets, hashes, sorted sets, JSON, etc.) with high-level semantics (counters, queues, leaderboards, rate limiters) and supports transactions & scripting.
NoSQL Data Store: Key-value, document, and time series data storage.
Search and Query Engine: Indexing for hash/JSON documents, supporting vector search, full-text search, geospatial queries, ranking, and aggregations via Redis Search.
Event Store & Message Broker: Implements queues (lists), priority queues (sorted sets), event deduplication (sets), streams, and pub/sub with probabilistic stream processing capabilities.
Vector Store for GenAI: Integrates with AI applications (e.g. LangGraph, mem0) for short-term memory, long-term memory, LLM response caching (semantic caching), and retrieval augmented generation (RAG).
Real-Time Analytics: Powers personalization, recommendations, fraud detection, and risk assessment.

Why choose Redis?

Redis is a popular choice for developers worldwide due to its combination of speed, flexibility, and rich feature set. Here's why people choose Redis for:

Performance: Because Redis keeps data primarily in memory and uses efficient data structures, it achieves extremely low latency (often sub-millisecond) for both read and write operations. This makes it ideal for applications demanding real-time responsiveness.
Flexibility: Redis isn't just a key-value store, it provides native support for a wide range of data structures and capabilities listed in What is Redis?
Extensibility: Redis is not limited to the built-in data structures, it has a modules API that makes it possible to extend Redis functionality and rapidly implement new Redis commands
Simplicity: Redis has a simple, text-based protocol and well-documented command set
Ubiquity: Redis is battle tested in production workloads at a massive scale. There is a good chance you indirectly interact with Redis several times daily
Versatility: Redis is the de facto standard for use cases such as:
- Caching: quickly access frequently used data without needing to query your primary database
- Session management: read and write user session data without hurting user experience or slowing down every API call
- Querying, sorting, and analytics: perform deduplication, full text search, and secondary indexing on in-memory data as fast as possible
- Messaging and interservice communication: job queues, message brokering, pub/sub, and streams for communicating between services
- Vector operations: Long-term and short-term LLM memory, RAG content retrieval, semantic caching, semantic routing, and vector similarity search

In summary, Redis provides a powerful, fast, and flexible toolkit for solving a wide variety of data management challenges. If you want to know more, here is a list of starting points:

What is Redis Open Source?

Redis Community Edition (Redis CE) was renamed Redis Open Source with the v8.0 release.

Redis Ltd. also offers Redis Software, a self-managed software with additional compliance, reliability, and resiliency for enterprise scaling, and Redis Cloud, a fully managed service integrated with Google Cloud, Azure, and AWS for production-ready apps.

Read more about the differences between Redis Open Source and Redis here.

Getting started

If you want to get up and running with Redis quickly without needing to build from source, use one of the following methods:

Redis Cloud
Official Redis Docker images (Alpine/Debian)
```
docker run -d -p 6379:6379 redis:latest
```
Redis binary distributions
Redis quick start guides

If you prefer to build Redis from source - see instructions below.

Redis starter projects

To get started as quickly as possible in your language of choice, use one of the following starter projects:

Using Redis with client libraries

To connect your application to Redis, you will need a client library. Redis has documented client libraries in most popular languages, with community-supported client libraries in additional languages.

Using Redis with redis-cli

redis-cli is Redis' command line interface. It is available as part of all the binary distributions and when you build Redis from source.

You can start a redis-server instance, and then, in another terminal try the following:

cd src
./redis-cli

redis> ping
PONG
redis> set foo bar
OK
redis> get foo
"bar"
redis> incr mycounter
(integer) 1
redis> incr mycounter
(integer) 2
redis>

Using Redis with Redis Insight

For a more visual and user-friendly experience, use Redis Insight - a tool that lets you explore data, design, develop, and optimize your applications while also serving as a platform for Redis education and onboarding. Redis Insight integrates Redis Copilot, a natural language AI assistant that improves the experience when working with data and commands.

Redis data types, processing engines, and capabilities

Redis provides a variety of data types, processing engines, and capabilities to support a wide range of use cases:

Important: Features marked with an asterisk (*) require Redis to be compiled with the BUILD_WITH_MODULES=yes flag when building Redis from source

String: Sequences of bytes, including text, serialized objects, and binary arrays used for caching, counters, and bitwise operations.
JSON: Nested JSON documents that are indexed and searchable using JSONPath expressions and with Redis Search
Array: Sparse, index-addressable collection of string values
Hash: Field-value maps used to represent basic objects and store groupings of key-value pairs with support for hash field expiration (TTL)
Redis Search: Use Redis as a document database, a vector database, a secondary index, and a search engine. Define indexes for hash and JSON documents and then use a rich query language for vector search, full-text search, geospatial queries, and aggregations.
List: Linked lists of string values used as stacks, queues, and for queue management.
Set: Unordered collection of unique strings used for tracking unique items, relations, and common set operations (intersections, unions, differences).
Sorted set: Collection of unique strings ordered by an associated score used for leaderboards and rate limiters.
Vector set (beta): Collection of vector embeddings used for semantic similarity search, semantic caching, semantic routing, and Retrieval Augmented Generation (RAG).
Geospatial indexes: Coordinates used for finding nearby points within a given radius or bounding box.
Bitmap: A set of bit-oriented operations defined on the string type used for efficient set representations and object permissions.
Bitfield: Binary-encoded strings that let you set, increment, and get integer values of arbitrary bit length used for limited-range counters, numeric values, and multi-level object permissions such as role-based access control (RBAC)
Hyperloglog: A probabilistic data structure for approximating the cardinality of a set used for analytics such as counting unique visits, form fills, etc.
*Bloom filter: A probabilistic data structure to check if a given value is present in a set. Used for fraud detection, ad placement, and unique column (i.e. username/email/slug) checks.
*Cuckoo filter: A probabilistic data structure for checking if a given value is present in a set while also allowing limited counting and deletions used in targeted advertising and coupon code validation.
*t-digest: A probabilistic data structure used for estimating the percentile of a large dataset without having to store and order all the data points. Used for hardware/software monitoring, online gaming, network traffic monitoring, and predictive maintenance.
*Top-k: A probabilistic data structure for finding the most frequent values in a data stream used for trend discovery.
*Count-min sketch: A probabilistic data structure for estimating how many times a given value appears in a data stream used for sales volume calculations.
Time series: Data points indexed in time order used for monitoring sensor data, asset tracking, and predictive analytics
Pub/sub: A lightweight messaging capability. Publishers send messages to a channel, and subscribers receive messages from that channel.
Stream: An append-only log with random access capabilities and complex consumption strategies such as consumer groups. Used for event sourcing, sensor monitoring, and notifications.
Transaction: Allows the execution of a group of commands in a single step. A request sent by another client will never be served in the middle of the execution of a transaction. This guarantees that the commands are executed as a single isolated operation.
Programmability: Upload and execute Lua scripts on the server. Scripts can employ programmatic control structures and use most of the commands while executing to access the database. Because scripts are executed on the server, reading and writing data from scripts is very efficient.

Cloud hosted Redis

Fully-managed Redis with real-time performance at scale.

Redis Cloud

Community

Redis Community Resources

Build Redis from source

This section refers to building Redis from source. If you want to get up and running with Redis quickly without needing to build from source see the Getting started section.

Build and run Redis with all data structures - Ubuntu 20.04 (Focal)

Tested with the following Docker image:

ubuntu:20.04

Install required dependencies

Update your package lists and install the necessary development tools and libraries:

apt-get update
apt-get install -y sudo
sudo apt-get install -y --no-install-recommends ca-certificates wget dpkg-dev gcc g++ libc6-dev libssl-dev make git python3 python3-pip python3-venv python3-dev unzip rsync clang automake autoconf gcc-10 g++-10 libtool

Use GCC 10 as the default compiler

Update the system's default compiler to GCC 10:

sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 100 --slave /usr/bin/g++ g++ /usr/bin/g++-10

Install CMake

Install CMake using pip3 and link it for system-wide access:
```
pip3 install cmake==3.31.6
sudo ln -sf /usr/local/bin/cmake /usr/bin/cmake
cmake --version
```
Note: CMake version 3.31.6 is the latest supported version. Newer versions cannot be used.
Download the Redis source

Download a specific version of the Redis source code archive from GitHub.

Replace <version> with the Redis version, for example: 8.0.0.
```
cd /usr/src
wget -O redis-<version>.tar.gz https://github.com/redis/redis/archive/refs/tags/<version>.tar.gz
```
Extract the source archive

Create a directory for the source code and extract the contents into it:
```
cd /usr/src
tar xvf redis-<version>.tar.gz
rm redis-<version>.tar.gz
```

Build Redis

Set the necessary environment variables and compile Redis:

cd /usr/src/redis-<version>
export BUILD_TLS=yes BUILD_WITH_MODULES=yes INSTALL_RUST_TOOLCHAIN=yes DISABLE_WERRORS=yes
make -j "$(nproc)" all

Run Redis

cd /usr/src/redis-<version>
./src/redis-server redis-full.conf

Build and run Redis with all data structures - Ubuntu 22.04 (Jammy)

Tested with the following Docker image:

ubuntu:22.04

Install required dependencies

Update your package lists and install the necessary development tools and libraries:

apt-get update
apt-get install -y sudo
sudo apt-get install -y --no-install-recommends ca-certificates wget dpkg-dev gcc g++ libc6-dev libssl-dev make git cmake python3 python3-pip python3-venv python3-dev unzip rsync clang automake autoconf libtool

Install CMake

Install CMake using pip3 and link it for system-wide access:
```
pip3 install cmake==3.31.6
sudo ln -sf /usr/local/bin/cmake /usr/bin/cmake
cmake --version
```
Note: CMake version 3.31.6 is the latest supported version. Newer versions cannot be used.
Download the Redis source

Download a specific version of the Redis source code archive from GitHub.

Replace <version> with the Redis version, for example: 8.0.0.
```
cd /usr/src
wget -O redis-<version>.tar.gz https://github.com/redis/redis/archive/refs/tags/<version>.tar.gz
```
Extract the source archive

Create a directory for the source code and extract the contents into it:
```
cd /usr/src
tar xvf redis-<version>.tar.gz
rm redis-<version>.tar.gz
```

Build Redis

Set the necessary environment variables and build Redis:

cd /usr/src/redis-<version>
export BUILD_TLS=yes BUILD_WITH_MODULES=yes INSTALL_RUST_TOOLCHAIN=yes DISABLE_WERRORS=yes
make -j "$(nproc)" all

Run Redis

cd /usr/src/redis-<version>
./src/redis-server redis-full.conf

Build and run Redis with all data structures - Ubuntu 24.04 (Noble)

Tested with the following Docker image:

ubuntu:24.04

Install required dependencies

Update your package lists and install the necessary development tools and libraries:

apt-get update
apt-get install -y sudo
sudo apt-get install -y --no-install-recommends ca-certificates wget dpkg-dev gcc g++ libc6-dev libssl-dev make git cmake python3 python3-pip python3-venv python3-dev unzip rsync clang automake autoconf libtool

Download the Redis source

Download a specific version of the Redis source code archive from GitHub.

Replace <version> with the Redis version, for example: 8.0.0.
```
cd /usr/src
wget -O redis-<version>.tar.gz https://github.com/redis/redis/archive/refs/tags/<version>.tar.gz
```
Extract the source archive

Create a directory for the source code and extract the contents into it:
```
cd /usr/src
tar xvf redis-<version>.tar.gz
rm redis-<version>.tar.gz
```

Build Redis

Set the necessary environment variables and build Redis:

cd /usr/src/redis-<version>
export BUILD_TLS=yes BUILD_WITH_MODULES=yes INSTALL_RUST_TOOLCHAIN=yes DISABLE_WERRORS=yes
make -j "$(nproc)" all

Run Redis

cd /usr/src/redis-<version>
./src/redis-server redis-full.conf

Build and run Redis with all data structures - Debian 11 (Bullseye) / 12 (Bookworm)

Tested with the following Docker images:

debian:bullseye
debian:bullseye-slim
debian:bookworm
debian:bookworm-slim

Install required dependencies

Update your package lists and install the necessary development tools and libraries:

apt-get update
apt-get install -y sudo
sudo apt-get install -y --no-install-recommends ca-certificates wget dpkg-dev gcc g++ libc6-dev libssl-dev make git cmake python3 python3-pip python3-venv python3-dev unzip rsync clang automake autoconf libtool

Download the Redis source

Download a specific version of the Redis source code archive from GitHub.

Replace <version> with the Redis version, for example: 8.0.0.
```
cd /usr/src
wget -O redis-<version>.tar.gz https://github.com/redis/redis/archive/refs/tags/<version>.tar.gz
```
Extract the source archive

Create a directory for the source code and extract the contents into it:
```
cd /usr/src
tar xvf redis-<version>.tar.gz
rm redis-<version>.tar.gz
```

Build Redis

Set the necessary environment variables and build Redis:

cd /usr/src/redis-<version>
export BUILD_TLS=yes BUILD_WITH_MODULES=yes INSTALL_RUST_TOOLCHAIN=yes DISABLE_WERRORS=yes
make -j "$(nproc)" all

Run Redis

cd /usr/src/redis-<version>
./src/redis-server redis-full.conf

Build and run Redis with all data structures - AlmaLinux 8.10 / Rocky Linux 8.10

Tested with the following Docker images:

almalinux:8.10
almalinux:8.10-minimal
rockylinux/rockylinux:8.10
rockylinux/rockylinux:8.10-minimal

Prepare the system

For 8.10-minimal, install sudo and dnf as follows:

microdnf install dnf sudo -y

For 8.10 (regular), install sudo as follows:

dnf install sudo -y

Clean the package metadata, enable required repositories, and install development tools:

sudo dnf clean all
sudo tee /etc/yum.repos.d/goreleaser.repo > /dev/null <<EOF
[goreleaser]
name=GoReleaser
baseurl=https://repo.goreleaser.com/yum/
enabled=1
gpgcheck=0
EOF
sudo dnf update -y
sudo dnf groupinstall "Development Tools" -y
sudo dnf config-manager --set-enabled powertools
sudo dnf install -y epel-release

Install required dependencies

Update your package lists and install the necessary development tools and libraries:

sudo dnf install -y --nobest --skip-broken pkg-config wget gcc-toolset-13-gcc gcc-toolset-13-gcc-c++ git make openssl openssl-devel python3.11 python3.11-pip python3.11-devel unzip rsync clang curl libtool automake autoconf jq systemd-devel

Create a Python virtual environment:

python3.11 -m venv /opt/venv

Enable the GCC toolset:

sudo cp /opt/rh/gcc-toolset-13/enable /etc/profile.d/gcc-toolset-13.sh
echo "source /etc/profile.d/gcc-toolset-13.sh" | sudo tee -a /etc/bashrc

Install CMake

Install CMake 3.25.1 manually:

CMAKE_VERSION=3.25.1
ARCH=$(uname -m)
if [ "$ARCH" = "x86_64" ]; then
  CMAKE_FILE=cmake-${CMAKE_VERSION}-linux-x86_64.sh
else
  CMAKE_FILE=cmake-${CMAKE_VERSION}-linux-aarch64.sh
fi
wget https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/${CMAKE_FILE}
chmod +x ${CMAKE_FILE}
./${CMAKE_FILE} --skip-license --prefix=/usr/local --exclude-subdir
rm ${CMAKE_FILE}
cmake --version

Download the Redis source

Download a specific version of the Redis source code archive from GitHub.

Replace <version> with the Redis version, for example: 8.0.0.
```
cd /usr/src
wget -O redis-<version>.tar.gz https://github.com/redis/redis/archive/refs/tags/<version>.tar.gz
```
Extract the source archive

Create a directory for the source code and extract the contents into it:
```
cd /usr/src
tar xvf redis-<version>.tar.gz
rm redis-<version>.tar.gz
```

Build Redis

Enable the GCC toolset, set the necessary environment variables, and build Redis:

source /etc/profile.d/gcc-toolset-13.sh
cd /usr/src/redis-<version>
export BUILD_TLS=yes BUILD_WITH_MODULES=yes INSTALL_RUST_TOOLCHAIN=yes DISABLE_WERRORS=yes
make -j "$(nproc)" all

Run Redis

cd /usr/src/redis-<version>
./src/redis-server redis-full.conf

Build and run Redis with all data structures - AlmaLinux 9.5 / Rocky Linux 9.5

Tested with the following Docker images:

almalinux:9.5
almalinux:9.5-minimal
rockylinux/rockylinux:9.5
rockylinux/rockylinux:9.5-minimal

Prepare the system

For 9.5-minimal, install sudo and dnf as follows:

microdnf install dnf sudo -y

For 9.5 (regular), install sudo as follows:

dnf install sudo -y

Clean the package metadata, enable required repositories, and install development tools:

sudo tee /etc/yum.repos.d/goreleaser.repo > /dev/null <<EOF
[goreleaser]
name=GoReleaser
baseurl=https://repo.goreleaser.com/yum/
enabled=1
gpgcheck=0
EOF
sudo dnf clean all
sudo dnf makecache
sudo dnf update -y

Install required dependencies

Update your package lists and install the necessary development tools and libraries:

sudo dnf install -y --nobest --skip-broken pkg-config xz wget which gcc-toolset-13-gcc gcc-toolset-13-gcc-c++ git make openssl openssl-devel python3 python3-pip python3-devel unzip rsync clang curl libtool automake autoconf jq systemd-devel

Create a Python virtual environment:

python3 -m venv /opt/venv

Enable the GCC toolset:

sudo cp /opt/rh/gcc-toolset-13/enable /etc/profile.d/gcc-toolset-13.sh
echo "source /etc/profile.d/gcc-toolset-13.sh" | sudo tee -a /etc/bashrc

Install CMake

Install CMake 3.25.1 manually:

CMAKE_VERSION=3.25.1
ARCH=$(uname -m)
if [ "$ARCH" = "x86_64" ]; then
  CMAKE_FILE=cmake-${CMAKE_VERSION}-linux-x86_64.sh
else
  CMAKE_FILE=cmake-${CMAKE_VERSION}-linux-aarch64.sh
fi
wget https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/${CMAKE_FILE}
chmod +x ${CMAKE_FILE}
./${CMAKE_FILE} --skip-license --prefix=/usr/local --exclude-subdir
rm ${CMAKE_FILE}
cmake --version

Download the Redis source

Download a specific version of the Redis source code archive from GitHub.

Replace <version> with the Redis version, for example: 8.0.0.
```
cd /usr/src
wget -O redis-<version>.tar.gz https://github.com/redis/redis/archive/refs/tags/<version>.tar.gz
```
Extract the source archive

Create a directory for the source code and extract the contents into it:
```
cd /usr/src
tar xvf redis-<version>.tar.gz
rm redis-<version>.tar.gz
```

Build Redis

Enable the GCC toolset, set the necessary environment variables, and build Redis:

source /etc/profile.d/gcc-toolset-13.sh
cd /usr/src/redis-<version>
export BUILD_TLS=yes BUILD_WITH_MODULES=yes INSTALL_RUST_TOOLCHAIN=yes DISABLE_WERRORS=yes
make -j "$(nproc)" all

Run Redis

cd /usr/src/redis-<version>
./src/redis-server redis-full.conf

Build and run Redis with all data structures - macOS 13 (Ventura) and macOS 14 (Sonoma)

Install Homebrew

If Homebrew is not already installed, follow the installation instructions on the Homebrew home page.

Install required packages

export HOMEBREW_NO_AUTO_UPDATE=1
brew update
brew install coreutils
brew install make
brew install openssl
brew install llvm@18
brew install cmake
brew install gnu-sed
brew install automake
brew install libtool
brew install wget

Install Rust

Rust is required to build the JSON package.

RUST_INSTALLER=rust-1.80.1-$(if [ "$(uname -m)" = "arm64" ]; then echo "aarch64"; else echo "x86_64"; fi)-apple-darwin
wget --quiet -O ${RUST_INSTALLER}.tar.xz https://static.rust-lang.org/dist/${RUST_INSTALLER}.tar.xz
tar -xf ${RUST_INSTALLER}.tar.xz
(cd ${RUST_INSTALLER} && sudo ./install.sh)

Download the Redis source

Download a specific version of the Redis source code archive from GitHub.

Replace <version> with the Redis version, for example: 8.0.0.
```
cd ~/src
wget -O redis-<version>.tar.gz https://github.com/redis/redis/archive/refs/tags/<version>.tar.gz
```
Extract the source archive

Create a directory for the source code and extract the contents into it:
```
cd ~/src
tar xvf redis-<version>.tar.gz
rm redis-<version>.tar.gz
```

Build Redis

cd ~/src/redis-<version>
export HOMEBREW_PREFIX="$(brew --prefix)"
export BUILD_WITH_MODULES=yes
export BUILD_TLS=yes
export DISABLE_WERRORS=yes
PATH="$HOMEBREW_PREFIX/opt/libtool/libexec/gnubin:$HOMEBREW_PREFIX/opt/llvm@18/bin:$HOMEBREW_PREFIX/opt/make/libexec/gnubin:$HOMEBREW_PREFIX/opt/gnu-sed/libexec/gnubin:$HOMEBREW_PREFIX/opt/coreutils/libexec/gnubin:$PATH"
export LDFLAGS="-L$HOMEBREW_PREFIX/opt/llvm@18/lib"
export CPPFLAGS="-I$HOMEBREW_PREFIX/opt/llvm@18/include"
mkdir -p build_dir/etc
make -C redis-8.0 -j "$(nproc)" all OS=macos
make -C redis-8.0 install PREFIX=$(pwd)/build_dir OS=macos

Run Redis

export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
build_dir/bin/redis-server redis-full.conf

Build and run Redis with all data structures - macOS 15 (Sequoia)

Support and instructions will be provided at a later date.

Building Redis - flags and general notes

Redis can be compiled and used on Linux, OSX, OpenBSD, NetBSD, FreeBSD. We support big endian and little endian architectures, and both 32 bit and 64-bit systems.

It may compile on Solaris derived systems (for instance SmartOS) but our support for this platform is best effort and Redis is not guaranteed to work as well as on Linux, OSX, and *BSD.

To build Redis with all the data structures (including JSON, time series, Bloom filter, cuckoo filter, count-min sketch, top-k, and t-digest) and with Redis Query Engine, make sure first that all the prerequisites are installed (see build instructions above, per operating system). You need to use the following flag in the make command:

make BUILD_WITH_MODULES=yes

Note: BUILD_WITH_MODULES=yes is not supported on 32 bit systems.

To build Redis with just the core data structures, use:

make

To build with TLS support, you need OpenSSL development libraries (e.g. libssl-dev on Debian/Ubuntu) and the following flag in the make command:

make BUILD_TLS=yes

To build with systemd support, you need systemd development libraries (such as libsystemd-dev on Debian/Ubuntu or systemd-devel on CentOS), and the following flag:

make USE_SYSTEMD=yes

To append a suffix to Redis program names, add the following flag:

make PROG_SUFFIX="-alt"

You can build a 32 bit Redis binary using:

make 32bit

After building Redis, it is a good idea to test it using:

make test

If TLS is built, running the tests with TLS enabled (you will need tcl-tls installed):

./utils/gen-test-certs.sh
./runtest --tls

Fixing build problems with dependencies or cached build options

Redis has some dependencies which are included in the deps directory. make does not automatically rebuild dependencies even if something in the source code of dependencies changes.

When you update the source code with git pull or when code inside the dependencies tree is modified in any other way, make sure to use the following command in order to really clean everything and rebuild from scratch:

make distclean

This will clean: jemalloc, lua, hiredis, linenoise and other dependencies.

Also, if you force certain build options like 32bit target, no C compiler optimizations (for debugging purposes), and other similar build time options, those options are cached indefinitely until you issue a make distclean command.

Fixing problems building 32 bit binaries

If after building Redis with a 32 bit target you need to rebuild it with a 64 bit target, or the other way around, you need to perform a make distclean in the root directory of the Redis distribution.

In case of build errors when trying to build a 32 bit binary of Redis, try the following steps:

Install the package libc6-dev-i386 (also try g++-multilib).
Try using the following command line instead of make 32bit: make CFLAGS="-m32 -march=native" LDFLAGS="-m32"

Allocator

Selecting a non-default memory allocator when building Redis is done by setting the MALLOC environment variable. Redis is compiled and linked against libc malloc by default, except for jemalloc being the default on Linux systems. This default was picked because jemalloc has proven to have fewer fragmentation problems than libc malloc.

To force compiling against libc malloc, use:

make MALLOC=libc

To compile against jemalloc on Mac OS X systems, use:

make MALLOC=jemalloc

Monotonic clock

By default, Redis will build using the POSIX clock_gettime function as the monotonic clock source. On most modern systems, the internal processor clock can be used to improve performance. Cautions can be found here: http://oliveryang.net/2015/09/pitfalls-of-TSC-usage/

On ARM aarch64 systems, the hardware clock is enabled by default because the ARM Generic Timer is architecturally guaranteed to be available and monotonic on all ARMv8-A processors (see the “The Generic Timer in AArch64 state” section of the Arm Architecture Reference Manual for Armv8-A).

To build with support for the processor's internal instruction clock on other architectures, use:

make CFLAGS="-DUSE_PROCESSOR_CLOCK"

Verbose build

Redis will build with a user-friendly colorized output by default. If you want to see a more verbose output, use the following:

make V=1

Running Redis with TLS

Please consult the TLS.md file for more information on how to use Redis with TLS.

Running Redis with the Query Engine and optional proprietary Intel SVS-VAMANA optimisations

License Disclaimer If you are using Redis Open Source under AGPLv3 or SSPLv1, you cannot use it together with the Intel Optimizations (Leanvec and LVQ binaries). The reason is that the Intel SVS license is not compatible with those licenses. The Leanvec and LVQ techniques are closed source and are only available for use with Redis Open Source when distributed under the RSALv2 license. For more details, please refer to the information provided by Intel here.

By default, Redis with the Redis Query Engine supports SVS-VAMANA index with global 8-bit quantisation. To compile Redis with the Intel SVS-VAMANA optimisations, LeanVec and LVQ, use the following:

make BUILD_INTEL_SVS_OPT=yes

Alternatively, you can export the variable before running the build step for your platform:

export BUILD_INTEL_SVS_OPT=yes
make

Code contributions

By contributing code to the Redis project in any form, including sending a pull request via GitHub, a code fragment or patch via private email or public discussion groups, you agree to release your code under the terms of the Redis Software Grant and Contributor License Agreement. Please see the CONTRIBUTING.md file in this source distribution for more information. For security bugs and vulnerabilities, please see SECURITY.md and the description of the ability of users to backport security patches under Redis Open Source 7.4+ under BSDv3. Open Source Redis releases are subject to the following licenses:

Version 7.2.x and prior releases are subject to BSDv3. These contributions to the original Redis core project are owned by their contributors and licensed under the 3BSDv3 license as referenced in the REDISCONTRIBUTIONS.txt file. Any copy of that license in this repository applies only to those contributions;
Versions 7.4.x to 7.8.x are subject to your choice of RSALv2 or SSPLv1; and
Version 8.0.x and subsequent releases are subject to the tri-license RSALv2/SSPLv1/AGPLv3 at your option as referenced in the LICENSE.txt file.

Redis Trademarks

The purpose of a trademark is to identify the goods and services of a person or company without causing confusion. As the registered owner of its name and logo, Redis accepts certain limited uses of its trademarks, but it has requirements that must be followed as described in its Trademark Guidelines available at: https://redis.io/legal/trademark-policy/.

OISF/suricata

Suricata is a network Intrusion Detection System, Intrusion Prevention System and Network Security Monitoring engine developed by the OISF and the Suricata community.

Suricata

Introduction

Suricata is a network IDS, IPS and NSM engine developed by the OISF and the Suricata community.

Resources

Contributing

We're happily taking patches and other contributions. Please see our Contribution Process for how to get started.

Suricata is a complex piece of software dealing with mostly untrusted input. Mishandling this input will have serious consequences:

in IPS mode a crash may knock a network offline
in passive mode a compromise of the IDS may lead to loss of critical and confidential data
missed detection may lead to undetected compromise of the network

In other words, we think the stakes are pretty high, especially since in many common cases the IDS/IPS will be directly reachable by an attacker.

For this reason, we have developed a QA process that is quite extensive. A consequence is that contributing to Suricata can be a somewhat lengthy process.

On a high level, the steps are:

GitHub-CI based checks. This runs automatically when a pull request is made.
Review by devs from the team and community
QA runs from private QA setups. These are private due to the nature of the test traffic.

Overview of Suricata's QA steps

OISF team members are able to submit builds to our private QA setup. It will run a series of build tests and a regression suite to confirm no existing features break.

The final QA runs takes a few hours minimally, and generally runs overnight. It currently runs:

extensive build tests on different OS', compilers, optimization levels, configure features
static code analysis using cppcheck, scan-build
runtime code analysis using valgrind, AddressSanitizer, LeakSanitizer
regression tests for past bugs
output validation of logging
unix socket testing
pcap based fuzz testing using ASAN and LSAN
traffic replay based IDS and IPS tests

Next to these tests, based on the type of code change further tests can be run manually:

traffic replay testing (multi-gigabit)
large pcap collection processing (multi-terabytes)
fuzz testing (might take multiple days or even weeks)
pcap based performance testing
live performance testing
various other manual tests based on evaluation of the proposed changes

It's important to realize that almost all of the tests above are used as acceptance tests. If something fails, it's up to you to address this in your code.

One step of the QA is currently run post-merge. We submit builds to the Coverity Scan program. Due to limitations of this (free) service, we can submit once a day max. Of course it can happen that after the merge the community will find issues. For both cases we request you to help address the issues as they may come up.

FAQ

Q: Will you accept my PR?

A: That depends on a number of things, including the code quality. With new features it also depends on whether the team and/or the community think the feature is useful, how much it affects other code and features, the risk of performance regressions, etc.

Q: When will my PR be merged?

A: It depends, if it's a major feature or considered a high risk change, it will probably go into the next major version.

Q: Why was my PR closed?

A: As documented in the Suricata GitHub workflow, we expect a new pull request for every change.

Normally, the team (or community) will give feedback on a pull request after which it is expected to be replaced by an improved PR. So look at the comments. If you disagree with the comments we can still discuss them in the closed PR.

If the PR was closed without comments it's likely due to QA failure. If the GitHub-CI checks failed, the PR should be fixed right away. No need for a discussion about it, unless you believe the QA failure is incorrect.

Q: The compiler/code analyser/tool is wrong, what now?

A: To assist in the automation of the QA, we're not accepting warnings or errors to stay. In some cases this could mean that we add a suppression if the tool supports that (e.g. valgrind, DrMemory). Some warnings can be disabled. In some exceptional cases the only 'solution' is to refactor the code to work around a static code checker limitation false positive. While frustrating, we prefer this over leaving warnings in the output. Warnings tend to get ignored and then increase risk of hiding other warnings.

Q: I think your QA test is wrong

A: If you really think it is, we can discuss how to improve it. But don't come to this conclusion too quickly, more often it's the code that turns out to be wrong.

Q: Do you require signing of a contributor license agreement?

A: Yes, we do this to keep the ownership of Suricata in one hand: the Open Information Security Foundation. See http://suricata.io/about/open-source/ and http://suricata.io/about/contribution-agreement/

dorianborian/sesame-robot

An open and affordable mini quadruped robot based on ESP32.

The Sesame Robot Project

Greetings, from your new best friend.

Sesame is an accessible Open-Source robotics project based on the ESP32 microcontroller system, with an emphasis on expression and movement. This project is designed for makers and engineers of all skill levels! Sesame offers a dynamic platform designed to start working with walking robots. To build a sesame robot, you will need basic soldering skills, $50-60 in hardware components, access to a 3D printer, and a basic understanding of Arduino IDE.

This repository contains the CAD design files, STL files, build and wiring guides, and the base/expanded firmware for the ESP32-based controller. There is also some included debugging firmware that may be helpful in getting your Sesame up and running.

Features

Quadruped Design: Uses 8 servo motors (2 per leg) to achieve roughly 8 total degrees of freedom.
Emotive Display: Features a 128x64 OLED screen acting as a reactive face that syncs with movement.
Fully Printable: Designed entirely for 3D printing in PLA with minimal supports.
Network Connectivity: Connect to your WiFi network for remote control and API access.
JSON API: RESTful API for programmatic control from Python, JavaScript, and more.
Conversational Faces: Expressive emotion library with talk variants for voice assistant projects.
Sesame Studio: New animation composer software to easily create custom movements.
Sesame Companion App: Python application for voice control and advanced interactions.
Serial CLI: Control the robot and trigger animations via a Serial Command Line Interface or the web UI.
Pre-programmed Emotes: Includes animations for Walking, Waving, Dancing, Pointing, Resting, and more.

Watch the launch video on YouTube

Getting Started

Follow these steps to build your own Sesame Robot:

1. Gather Parts

Check the Bill of Materials (BOM) for a complete list of required electronics and hardware.

Microcontroller: Lolin S2 Mini (recommended for DIY builds), Sesame Distro Board V2 (included in Build Kits, pre-flashed), or ESP32-DevKitC-32E with Distro Board V1 (legacy)
Actuators: 8x MG90 Servos
Power: 5V 3A source (USB-C PD for S2 Mini and V2 Distro Board, or battery + buck converter; see BOM for the 2× 10440 Li-ion + 2× AAA holder option)

2. Print Parts

Download the STLs and follow the Printing Guide.

Designed for PLA
Minimal supports required

3. Build & Wire

Follow the Build Guide and Wiring Guide to assemble the frame and connect the electronics.

4. Flash Firmware

Upload the code from the Firmware Directory.

Requires Arduino IDE
Configure WiFi AP settings

5. Create Animations

Use Sesame Studio to visually design poses and sequences for your robot.

Software & Firmware

Sesame Studio

Sesame Studio is a standalone desktop application included in software/sesame-studio/. It allows you to:

Visually pose the robot using a schematic interface.
Generate C++ code for servo angles automatically.
Sequence frames into complex animations.

> Go to Sesame Studio

Sesame Simulator

The Sesame Simulator, created by Jay Li, is a Rust-based 3D simulation environment for testing Sesame's movements and kinematics in a virtual space. It features:

Physics-based Simulation: Test walking and balance without hardware.
Web-based Interface: Run the simulator directly in your browser.
URDF Integration: Accurate modeling of Sesame's physical properties.

> Go to Sesame Simulator

Sesame Companion App

The Sesame Companion App is a Python-based application that enables advanced control and interaction with your robot over your local network. It leverages the new JSON API and network mode features to provide:

Voice Assistant Integration: Control Sesame with voice commands and see real-time emotional expressions.
Remote Control: Command your robot from anywhere on your local network.
Face Control: Change expressions dynamically based on conversation or context.
API Examples: Reference implementation for building your own integrations.

The Companion App works with robots running the latest firmware with network mode enabled.

> Go to Sesame Companion App Repository

Firmware

The ESP32 firmware (sesame-firmware-main.ino) handles the kinematics, face display, and WiFi control interface.

Web UI: Control the robot from your phone via the built-in Access Point.
Custom Faces: Add your own bitmaps (guide in firmware docs).

> Go to Firmware Docs

Contributing

This robot is a platform for building new features, cosmetics, tools, and ideas. Since the current firmware is a basic implementation, pull requests are very welcome for:

Kinematics improvements
New animations
Improved Web UI/UX
Sensor integration (Ultrasonic, Gyro, etc.)

I would also love to see forks of this project with new hardware, software, faces, etc. Be sure to send me a message if you end up building one, and I might feature you on my website or channel!

Created by Dorian Todd. Need help with your Sesame Robot? Send me a message on Discord, my username is "starphee"

valkey-io/valkey

A flexible distributed key-value database that is optimized for caching and other realtime workloads.

This project was forked from the open source Redis project right before the transition to their new source available licenses.

This README is just a fast quick start document. More details can be found under valkey.io

What is Valkey?

Valkey is a high-performance data structure server that primarily serves key/value workloads. It supports a wide range of native structures and an extensible plugin system for adding new data structures and access patterns.

Building Valkey using `Makefile`

Valkey can be compiled and used on Linux, macOS, OpenBSD, NetBSD, FreeBSD. We support big endian and little endian architectures, and both 32 bit and 64 bit systems.

It may compile on Solaris derived systems (for instance SmartOS) but our support for this platform is best effort and Valkey is not guaranteed to work as well as in Linux, macOS, and *BSD.

It is as simple as:

% make

To build with TLS support, you'll need OpenSSL development libraries (e.g. libssl-dev on Debian/Ubuntu).

To build TLS support as Valkey built-in:

% make BUILD_TLS=yes

To build TLS as Valkey module:

% make BUILD_TLS=module

Note that sentinel mode does not support TLS module.

To build with experimental RDMA support you'll need RDMA development libraries (e.g. librdmacm-dev and libibverbs-dev on Debian/Ubuntu).

To build RDMA support as Valkey built-in:

% make BUILD_RDMA=yes

To build RDMA as Valkey module:

% make BUILD_RDMA=module

To build with systemd support, you'll need systemd development libraries (such as libsystemd-dev on Debian/Ubuntu or systemd-devel on CentOS) and run:

% make USE_SYSTEMD=yes

To build with enhanced stack traces that include file names and line numbers for all functions (including static functions), use libbacktrace:

% make USE_LIBBACKTRACE=yes

To build Valkey without the Lua engine:

% make BUILD_LUA=no

To append a suffix to Valkey program names, use:

% make PROG_SUFFIX="-alt"

You can build a 32 bit Valkey binary using:

% make 32bit

After building Valkey, it is a good idea to test it using:

% make test

The above runs the main integration tests. Additional tests are started using:

% make test-unit     # Unit tests
% make test-modules  # Tests of the module API
% make test-sentinel # Valkey Sentinel integration tests
% make test-cluster  # Valkey Cluster integration tests

More about running the integration tests can be found in tests/README.md and for unit tests, see src/unit/README.md.

Performance monitoring

Valkey Performance Dashboards provide a consolidated view of throughput trends across versions, helping contributors validate improvements and identify regressions quickly.

Performance Overview - Compare throughput across Valkey versions
Unstable Branch Dashboard - Track performance of all commits in the unstable branch

Fixing build problems with dependencies or cached build options

Valkey has some dependencies which are included in the deps directory. make does not automatically rebuild dependencies even if something in the source code of dependencies changes.

% make distclean

This will clean: jemalloc, lua, libvalkey, linenoise and other dependencies.

Also if you force certain build options like 32bit target, no C compiler optimizations (for debugging purposes), and other similar build time options, those options are cached indefinitely until you issue a make distclean command.

Fixing problems building 32 bit binaries

If after building Valkey with a 32 bit target you need to rebuild it with a 64 bit target, or the other way around, you need to perform a make distclean in the root directory of the Valkey distribution.

In case of build errors when trying to build a 32 bit binary of Valkey, try the following steps:

Install the package libc6-dev-i386 (also try g++-multilib).
Try using the following command line instead of make 32bit: make CFLAGS="-m32 -march=native" LDFLAGS="-m32"

Allocator

Selecting a non-default memory allocator when building Valkey is done by setting the MALLOC environment variable. Valkey is compiled and linked against libc malloc by default, with the exception of jemalloc being the default on Linux systems. This default was picked because jemalloc has proven to have fewer fragmentation problems than libc malloc.

To force compiling against libc malloc, use:

% make MALLOC=libc

To compile against jemalloc on Mac OS X systems, use:

% make MALLOC=jemalloc

Monotonic clock

By default, Valkey uses the processor's internal instruction clock (TSC on x86, CNTVCT on ARM) for monotonic time tracking, which provides approximately 3x faster time access compared to POSIX clock_gettime (~10-30ns vs ~100ns). This is enabled by default on supported architectures (x86_64 Linux and aarch64) and automatically falls back to POSIX clock_gettime on unsupported systems.

For more information about processor clock usage, see: http://oliveryang.net/2015/09/pitfalls-of-TSC-usage/

To disable the processor clock and force POSIX clock_gettime, use:

% make CFLAGS="-DNO_PROCESSOR_CLOCK"

Verbose build

Valkey will build with a user-friendly colorized output by default. If you want to see a more verbose output, use the following:

% make V=1

Running Valkey

To run Valkey with the default configuration, just type:

% cd src
% ./valkey-server

If you want to provide your valkey.conf, you have to run it using an additional parameter (the path of the configuration file):

% cd src
% ./valkey-server /path/to/valkey.conf

It is possible to alter the Valkey configuration by passing parameters directly as options using the command line. Examples:

% ./valkey-server --port 9999 --replicaof 127.0.0.1 6379
% ./valkey-server /etc/valkey/6379.conf --loglevel debug

All the options in valkey.conf are also supported as options using the command line, with exactly the same name.

Running Valkey with TLS:

Running manually

To manually run a Valkey server with TLS mode (assuming ./utils/gen-test-certs.sh was invoked so sample certificates/keys are available):

TLS built-in mode:

./src/valkey-server --tls-port 6379 --port 0 \
    --tls-cert-file ./tests/tls/valkey.crt \
    --tls-key-file ./tests/tls/valkey.key \
    --tls-ca-cert-file ./tests/tls/ca.crt

TLS module mode:

./src/valkey-server --tls-port 6379 --port 0 \
    --tls-cert-file ./tests/tls/valkey.crt \
    --tls-key-file ./tests/tls/valkey.key \
    --tls-ca-cert-file ./tests/tls/ca.crt \
    --loadmodule src/valkey-tls.so

Note that you can disable TCP by specifying --port 0 explicitly. It's also possible to have both TCP and TLS available at the same time, but you'll have to assign different ports.

Use valkey-cli to connect to the Valkey server:

./src/valkey-cli --tls \
    --cert ./tests/tls/valkey.crt \
    --key ./tests/tls/valkey.key \
    --cacert ./tests/tls/ca.crt

Specifying --tls-replication yes makes a replica connect to the primary.

Using --tls-cluster yes makes Valkey Cluster use TLS across nodes.

Running Valkey with RDMA:

Note that Valkey Over RDMA is an experimental feature. It may be changed or removed in any minor or major version. Currently, it is only supported on Linux.

RDMA built-in mode:

./src/valkey-server --protected-mode no \
     --rdma-bind 192.168.122.100 --rdma-port 6379

RDMA module mode:

./src/valkey-server --protected-mode no \
     --loadmodule src/valkey-rdma.so --rdma-bind 192.168.122.100 --rdma-port 6379

It's possible to change bind address/port of RDMA by runtime command:

192.168.122.100:6379> CONFIG SET rdma-port 6380

It's also possible to have both RDMA and TCP available, and there is no conflict of TCP(6379) and RDMA(6379), Ex:

% ./src/valkey-server --protected-mode no \
     --loadmodule src/valkey-rdma.so --rdma-bind 192.168.122.100 --rdma-port 6379 \
     --port 6379

Note that the network card (192.168.122.100 of this example) should support RDMA. To test a server supports RDMA or not:

% rdma res show (a new version iproute2 package)

Or:

% ibv_devices

Playing with Valkey

You can use valkey-cli to play with Valkey. Start a valkey-server instance, then in another terminal try the following:

% cd src
% ./valkey-cli
valkey> ping
PONG
valkey> set foo bar
OK
valkey> get foo
"bar"
valkey> incr mycounter
(integer) 1
valkey> incr mycounter
(integer) 2
valkey>

Installing Valkey

In order to install Valkey binaries into /usr/local/bin, just use:

% make install

You can use make PREFIX=/some/other/directory install if you wish to use a different destination.

Note: For compatibility with Redis, we create symlinks from the Redis names (redis-server, redis-cli, etc.) to the Valkey binaries installed by make install. The symlinks are created in same directory as the Valkey binaries. The symlinks are removed when using make uninstall. The creation of the symlinks can be skipped by setting the makefile variable USE_REDIS_SYMLINKS=no.

make install will just install binaries in your system, but will not configure init scripts and configuration files in the appropriate place. This is not needed if you just want to play a bit with Valkey, but if you are installing it the proper way for a production system, we have a script that does this for Ubuntu and Debian systems:

% cd utils
% ./install_server.sh

Note: install_server.sh will not work on macOS; it is built for Linux only.

The script will ask you a few questions and will setup everything you need to run Valkey properly as a background daemon that will start again on system reboots.

You'll be able to stop and start Valkey using the script named /etc/init.d/valkey_<portnumber>, for instance /etc/init.d/valkey_6379.

Building using `CMake`

In addition to the traditional Makefile build, Valkey supports an alternative, experimental, build system using CMake.

To build and install Valkey, in Release mode (an optimized build), type this into your terminal:

mkdir build-release
cd $_
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/opt/valkey
sudo make install
# Valkey is now installed under /opt/valkey

Other options supported by Valkey's CMake build system:

Special build flags

-DBUILD_TLS=<yes|no> enable TLS build for Valkey. Default: no
-DBUILD_RDMA=<no|module> enable RDMA module build (only module mode supported). Default: no
-DBUILD_MALLOC=<libc|jemalloc|tcmalloc|tcmalloc_minimal> choose the allocator to use. Default on Linux: jemalloc, for other OS: libc
-DBUILD_SANITIZER=<address|thread|undefined> build with address sanitizer enabled. Default: disabled (no sanitizer)
-DBUILD_UNIT_GTESTS=[yes|no] when set, the build will produce unit tests executable valkey-unit-gtests. Default: no
-DBUILD_TEST_MODULES=[yes|no] when set, the build will include the modules located under the tests/modules folder. Default: no
-DBUILD_EXAMPLE_MODULES=[yes|no] when set, the build will include the example modules located under the src/modules folder. Default: no

Common flags

-DCMAKE_BUILD_TYPE=<Debug|Release...> define the build type, see CMake manual for more details
-DCMAKE_INSTALL_PREFIX=/installation/path override this value to define a custom install prefix. Default: /usr/local
-G"<Generator Name>" generate build files for "Generator Name". By default, CMake will generate Makefiles.

Verbose build

CMake generates a user-friendly colorized output by default. If you want to see a more verbose output, use the following:

make VERBOSE=1

Troubleshooting

During the CMake stage, CMake caches variables in a local file named CMakeCache.txt. All variables generated by Valkey are removed from the cache once consumed (this is done by calling to unset(VAR-NAME CACHE)). However, some variables, like the compiler path, are kept in cache. To start a fresh build either remove the cache file CMakeCache.txt from the build folder, or delete the build folder completely.

It is important to re-run CMake when adding new source files.

Integration with IDE

During the CMake stage of the build, CMake generates a JSON file named compile_commands.json and places it under the build folder. This file is used by many IDEs and text editors for providing code completion (via clangd).

A small caveat is that these tools will look for compile_commands.json under the Valkey's top folder. A common workaround is to create a symbolic link to it:

cd /path/to/valkey/
# We assume here that your build folder is `build-release`
ln -sf $(pwd)/build-release/compile_commands.json $(pwd)/compile_commands.json

Restart your IDE and voila

Code contributions

Please see the CONTRIBUTING.md. For security bugs and vulnerabilities, please see SECURITY.md.

Valkey is an open community project under LF Projects

Valkey a Series of LF Projects, LLC 2810 N Church St, PMB 57274 Wilmington, Delaware 19802-4447

openzfs/zfs

OpenZFS on Linux and FreeBSD

OpenZFS is an advanced file system and volume manager which was originally developed for Solaris and is now maintained by the OpenZFS community. This repository contains the code for running OpenZFS on Linux and FreeBSD.

Official Resources

Documentation - for using and developing this repo
ZoL site - Linux release info & links
Mailing lists
OpenZFS site - for conference videos and info on other platforms (illumos, OSX, Windows, etc)

Installation

Full documentation for installing OpenZFS on your favorite operating system can be found at the Getting Started Page.

Contribute & Develop

We have a separate document with contribution guidelines.

We have a Code of Conduct.

Release

OpenZFS is released under a CDDL license. For more details see the NOTICE, LICENSE and COPYRIGHT files; UCRL-CODE-235197

Supported Kernels and Distributions

Linux

Given the wide variety of Linux environments, we prioritize development and testing on stable, supported kernels and distributions.

Kernel (kernel.org)

All longterm kernels from kernel.org are supported. stable kernels are usually supported in the next OpenZFS release.

Supported longterm kernels: 6.18, 6.12, 6.6, 6.1, 5.15, 5.10.

Red Hat Enterprise Linux (RHEL)

All RHEL (and compatible systems: AlmaLinux OS, Rocky Linux, etc) on the full or maintenance support tracks are supported.

Supported RHEL releases: 8.10, 9.7, 10.1.

Ubuntu

All Ubuntu LTS releases are supported.

Supported Ubuntu releases: 24.04 “Noble”, 22.04 “Jammy”.

Debian

All Debian stable and LTS releases are supported.

Supported Debian releases: 13 “Trixie”, 12 “Bookworm”, 11 “Bullseye”.

Other Distributions

Generally, if a distribution is following an LTS kernel, it should work well with OpenZFS.

FreeBSD

All FreeBSD releases receiving security support are supported by OpenZFS.

Supported FreeBSD releases: 15.0, 14.4, 13.5.

eclipse-mosquitto/mosquitto

Eclipse Mosquitto - An open source MQTT broker

Eclipse Mosquitto

Mosquitto is an open source implementation of a server for version 5.0, 3.1.1, and 3.1 of the MQTT protocol. It also includes a C and C++ client library, the mosquitto_pub mosquitto_rr, and mosquitto_sub utilities for publishing and subscribing, and the mosquitto_ctrl, mosquitto_signal, and mosquitto_passwd applications for helping administer the broker.

Installing

See https://mosquitto.org/download/ for details on installing binaries for various platforms.

Quick start

If you have installed a binary package the broker should have been started automatically. If not, it can be started with a very basic configuration:

mosquitto

Then use mosquitto_sub to subscribe to a topic:

mosquitto_sub -t 'test/topic' -v

And to publish a message:

mosquitto_pub -t 'test/topic' -m 'hello world'

Note that starting the broker like this allows anonymous/unauthenticated access but only from the local computer, so it's only really useful for initial testing.

If you want to have clients from another computer connect, you will need to provide a configuration file. If you have installed from a binary package, you will probably already have a configuration file at somewhere like /etc/mosquitto/mosquitto.conf. If you've compiled from source, you can write your config file then run as mosquitto -c /path/to/mosquitto.conf.

To start your config file you define a listener and you will need to think about what authentication you require. It is not advised to run your broker with anonymous access when it is publicly available.

For details on how to do this, look at the authentication methods available and the dynamic security plugin.

Documentation

Documentation for the broker, clients and client library API can be found in the man pages, which are available online at https://mosquitto.org/man/. There are also pages with an introduction to the features of MQTT, the mosquitto_passwd utility for dealing with username/passwords, and a description of the configuration file options available for the broker.

Detailed client library API documentation can be found at https://mosquitto.org/api/

Building from source

To build from source the recommended route for end users is to download the archive from https://mosquitto.org/download/.

On Windows and Mac, use cmake to build. On other platforms, just run make to build. For Windows, see also README-windows.md.

If you are building from the git repository then the documentation will not already be built. Use make binary to skip building the man pages, or install docbook-xsl on Debian/Ubuntu systems.

Build Dependencies

cJSON - required
c-ares (libc-ares-dev on Debian based systems) - optional, enable with WITH_SRV=yes
libedit - for mosquitto_ctrl interactive shell - optional, disable with WITH_EDITLINE=no
libmicrohttpd - for broker http api support - optional, disable with WITH_HTTP_API=no
openssl (libssl-dev on Debian based systems) - optional, disable with WITH_TLS=no
pthreads - for client library thread support. This is required to support the mosquitto_loop_start() and mosquitto_loop_stop() functions. If compiled without pthread support, the library isn't guaranteed to be thread safe.
sqlite3 - for persistence support in the broker - optional, disable with WITH_SQLITE=no
uthash / utlist - bundled versions of these headers are provided, disable their use with WITH_BUNDLED_DEPS=no
xsltproc (xsltproc and docbook-xsl on Debian based systems) - only needed when building from git sources - disable with WITH_DOCS=no

Equivalent options for enabling/disabling features are available when using the CMake build. It is also possible to enable/disable building of specific plugins in the CMake build.

Building mosquitto - Using vcpkg

You can download and install mosquitto using the vcpkg dependency manager:

git clone https://github.com/Microsoft/vcpkg.git
cd vcpkg
./bootstrap-vcpkg.sh
./vcpkg integrate install
./vcpkg install mosquitto

The mosquitto port in vcpkg is kept up to date by Microsoft team members and community contributors. If the version is out of date, please create an issue or pull request on the vcpkg repository.

Credits

Mosquitto was written by Roger Light roger@atchoo.org. There have been substantial contributions by other people in the community both in terms of code and other help.

robertdavidgraham/masscan

TCP port scanner, spews SYN packets asynchronously, scanning entire Internet in under 5 minutes.

MASSCAN: Mass IP port scanner

This is an Internet-scale port scanner. It can scan the entire Internet in under 5 minutes, transmitting 10 million packets per second, from a single machine.

Its usage (parameters, output) is similar to nmap, the most famous port scanner. When in doubt, try one of those features -- features that support widespread scanning of many machines are supported, while in-depth scanning of single machines aren't.

Internally, it uses asynchronous transmission, similar to port scanners like scanrand, unicornscan, and ZMap. It's more flexible, allowing arbitrary port and address ranges.

NOTE: masscan uses its own ad hoc TCP/IP stack. Anything other than simple port scans may cause conflict with the local TCP/IP stack. This means you need to use either the --src-ip option to run from a different IP address, or use --src-port to configure which source ports masscan uses, then also configure the internal firewall (like pf or iptables) to firewall those ports from the rest of the operating system.

This tool is free, but consider contributing money to its development: Bitcoin wallet address: 1MASSCANaHUiyTtR3bJ2sLGuMw5kDBaj4T

Building

On Debian/Ubuntu, it goes something like the following. It doesn't really have any dependencies other than a C compiler (such as gcc or clang).

sudo apt-get --assume-yes install git make gcc
git clone https://github.com/robertdavidgraham/masscan
cd masscan
make

This puts the program in the masscan/bin subdirectory. To install it (on Linux) run:

make install

The source consists of a lot of small files, so building goes a lot faster by using the multi-threaded build. This requires more than 2gigs on a Raspberry Pi (and breaks), so you might use a smaller number, like -j4 rather than all possible threads.

make -j

While Linux is the primary target platform, the code runs well on many other systems (Windows, macOS, etc.). Here's some additional build info:

Windows w/ Visual Studio: use the VS10 project
Windows w/ MinGW: just type make
Windows w/ cygwin: won't work
Mac OS X /w XCode: use the XCode4 project
Mac OS X /w cmdline: just type make
FreeBSD: type gmake
other: try just compiling all the files together, cc src/*.c -o bin/masscan

On macOS, the x86 binaries seem to work just as fast under ARM emulation.

Usage

Usage is similar to nmap. To scan a network segment for some ports:

# masscan -p80,8000-8100 10.0.0.0/8 2603:3001:2d00:da00::/112

This will:

scan the 10.x.x.x subnet, and 2603:3001:2d00:da00::x subnets
scans port 80 and the range 8000 to 8100, or 102 ports total, on both subnets
print output to <stdout> that can be redirected to a file

To see the complete list of options, use the --echo feature. This dumps the current configuration and exits. This output can be used as input back into the program:

# masscan -p80,8000-8100 10.0.0.0/8 2603:3001:2d00:da00::/112 --echo > xxx.conf
# masscan -c xxx.conf --rate 1000

Banner checking

Masscan can do more than just detect whether ports are open. It can also complete the TCP connection and interaction with the application at that port in order to grab simple "banner" information.

Masscan supports banner checking on the following protocols:

FTP
HTTP
IMAP4
memcached
POP3
SMTP
SSH
SSL
SMBv1
SMBv2
Telnet
RDP
VNC

The problem with this is that masscan contains its own TCP/IP stack separate from the system you run it on. When the local system receives a SYN-ACK from the probed target, it responds with a RST packet that kills the connection before masscan can grab the banner.

The easiest way to prevent this is to assign masscan a separate IP address. This would look like one of the following examples:

# masscan 10.0.0.0/8 -p80 --banners --source-ip 192.168.1.200
  # masscan 2a00:1450:4007:810::/112 -p80 --banners --source-ip 2603:3001:2d00:da00:91d7:b54:b498:859d

The address you choose has to be on the local subnet and not otherwise be used by another system. Masscan will warn you that you've made a mistake, but you might've messed up the other machine's communications for several minutes, so be careful.

In some cases, such as WiFi, this isn't possible. In those cases, you can firewall the port that masscan uses. This prevents the local TCP/IP stack from seeing the packet, but masscan still sees it since it bypasses the local stack. For Linux, this would look like:

# iptables -A INPUT -p tcp --dport 61000 -j DROP
# masscan 10.0.0.0/8 -p80 --banners --source-port 61000

You probably want to pick ports that don't conflict with ports Linux might otherwise choose for source-ports. You can see the range Linux uses, and reconfigure that range, by looking in the file:

/proc/sys/net/ipv4/ip_local_port_range

On the latest version of Kali Linux (2018-August), that range is 32768 to 60999, so you should choose ports either below 32768 or 61000 and above.

Setting an iptables rule only lasts until the next reboot. You need to lookup how to save the configuration depending upon your distro, such as using iptables-save and/or iptables-persistent.

On Mac OS X and BSD, there are similar steps. To find out the ranges to avoid, use a command like the following:

# sysctl net.inet.ip.portrange.first net.inet.ip.portrange.last

On FreeBSD and older MacOS, use an ipfw command:

# sudo ipfw add 1 deny tcp from any to any 40000 in
# masscan 10.0.0.0/8 -p80 --banners --source-port 40000

On newer MacOS and OpenBSD, use the pf packet-filter utility. Edit the file /etc/pf.conf to add a line like the following:

block in proto tcp from any to any port 40000:40015

Then to enable the firewall, run the command:

# pfctl -E

If the firewall is already running, then either reboot or reload the rules with the following command:

# pfctl -f /etc/pf.conf

Windows doesn't respond with RST packets, so neither of these techniques are necessary. However, masscan is still designed to work best using its own IP address, so you should run that way when possible, even when it is not strictly necessary.

The same thing is needed for other checks, such as the --heartbleed check, which is just a form of banner checking.

How to scan the entire Internet

While useful for smaller, internal networks, the program is really designed with the entire Internet in mind. It might look something like this:

# masscan 0.0.0.0/0 -p0-65535

Scanning the entire Internet is bad. For one thing, parts of the Internet react badly to being scanned. For another thing, some sites track scans and add you to a ban list, which will get you firewalled from useful parts of the Internet. Therefore, you want to exclude a lot of ranges. To blacklist or exclude ranges, you want to use the following syntax:

# masscan 0.0.0.0/0 -p0-65535 --excludefile exclude.txt

This just prints the results to the command-line. You probably want them saved to a file instead. Therefore, you want something like:

# masscan 0.0.0.0/0 -p0-65535 -oX scan.xml

This saves the results in an XML file, allowing you to easily dump the results in a database or something.

But, this only goes at the default rate of 100 packets/second, which will take forever to scan the Internet. You need to speed it up as so:

# masscan 0.0.0.0/0 -p0-65535 --max-rate 100000

This increases the rate to 100,000 packets/second, which will scan the entire Internet (minus excludes) in about 10 hours per port (or 655,360 hours if scanning all ports).

The thing to notice about this command-line is that these are all nmap compatible options. In addition, "invisible" options compatible with nmap are also set for you: -sS -Pn -n --randomize-hosts --send-eth. Likewise, the format of the XML file is inspired by nmap. There are, of course, a lot of differences, because the asynchronous nature of the program leads to a fundamentally different approach to the problem.

The above command-line is a bit cumbersome. Instead of putting everything on the command-line, it can be stored in a file instead. The above settings would look like this:

# My Scan
rate =  100000.00
output-format = xml
output-status = all
output-filename = scan.xml
ports = 0-65535
range = 0.0.0.0-255.255.255.255
excludefile = exclude.txt

To use this configuration file, use the -c:

# masscan -c myscan.conf

This also makes things easier when you repeat a scan.

By default, masscan first loads the configuration file /etc/masscan/masscan.conf. Any later configuration parameters override what's in this default configuration file. That's where I put my "excludefile" parameter so that I don't ever forget it. It just works automatically.

Getting output

By default, masscan produces fairly large text files, but it's easy to convert them into any other format. There are five supported output formats:

xml: Just use the parameter -oX <filename>. Or, use the parameters --output-format xml and --output-filename <filename>.
binary: This is the masscan builtin format. It produces much smaller files so that when I scan the Internet my disk doesn't fill up. They need to be parsed, though. The command-line option --readscan will read binary scan files. Using --readscan with the -oX option will produce an XML version of the results file.
grepable: This is an implementation of the Nmap -oG output that can be easily parsed by command-line tools. Just use the parameter -oG <filename>. Or, use the parameters --output-format grepable and --output-filename <filename>.
json: This saves the results in JSON format. Just use the parameter -oJ <filename>. Or, use the parameters --output-format json and --output-filename <filename>.
list: This is a simple list with one host and port pair per line. Just use the parameter -oL <filename>. Or, use the parameters --output-format list and --output-filename <filename>. The format is:
```
<port state> <protocol> <port number> <IP address> <POSIX timestamp>  
open tcp 80 XXX.XXX.XXX.XXX 1390380064
```

Comparison with Nmap

Where reasonable, every effort has been taken to make the program familiar to nmap users, even though it's fundamentally different. Masscan is tuned for wide range scanning of a lot of machines, whereas nmap is designed for intensive scanning of a single machine or a small range.

Two important differences are:

no default ports to scan, you must specify -p <ports>
target hosts are IP addresses or simple ranges, not DNS names, nor the funky subnet ranges nmap can use (like 10.0.0-255.0-255).

You can think of masscan as having the following settings permanently enabled:

-sS: this does SYN scan only (currently, will change in the future)
-Pn: doesn't ping hosts first, which is fundamental to the async operation
-n: no DNS resolution happens
--randomize-hosts: scan completely randomized, always, you can't change this
--send-eth: sends using raw libpcap

If you want a list of additional nmap compatible settings, use the following command:

# masscan --nmap

Transmit rate (IMPORTANT!!)

This program spews out packets very fast. On Windows, or from VMs, it can do 300,000 packets/second. On Linux (no virtualization) it'll do 1.6 million packets-per-second. That's fast enough to melt most networks.

Note that it'll only melt your own network. It randomizes the target IP addresses so that it shouldn't overwhelm any distant network.

By default, the rate is set to 100 packets/second. To increase the rate to a million use something like --rate 1000000.

When scanning the IPv4 Internet, you'll be scanning lots of subnets, so even though there's a high rate of packets going out, each target subnet will receive a small rate of incoming packets.

However, with IPv6 scanning, you'll tend to focus on a single target subnet with billions of addresses. Thus, your default behavior will overwhelm the target network. Networks often crash under the load that masscan can generate.

Design

This section describes the major design issues of the program.

Code Layout

The file main.c contains the main() function, as you'd expect. It also contains the transmit_thread() and receive_thread() functions. These functions have been deliberately flattened and heavily commented so that you can read the design of the program simply by stepping line-by-line through each of these.

Asynchronous

This is an asynchronous design. In other words, it is to nmap what the nginx web-server is to Apache. It has separate transmit and receive threads that are largely independent from each other. It's the same sort of design found in scanrand, unicornscan, and ZMap.

Because it's asynchronous, it runs as fast as the underlying packet transmit allows.

Randomization

A key difference between Masscan and other scanners is the way it randomizes targets.

The fundamental principle is to have a single index variable that starts at zero and is incremented by one for every probe. In C code, this is expressed as:

for (i = 0; i < range; i++) {
    scan(i);
}

We have to translate the index into an IP address. Let's say that you want to scan all "private" IP addresses. That would be the table of ranges like:

192.168.0.0/16
10.0.0.0/8
172.16.0.0/12

In this example, the first 64k indexes are appended to 192.168.x.x to form the target address. Then, the next 16-million are appended to 10.x.x.x. The remaining indexes in the range are applied to 172.16.x.x.

In this example, we only have three ranges. When scanning the entire Internet, we have in practice more than 100 ranges. That's because you have to blacklist or exclude a lot of sub-ranges. This chops up the desired range into hundreds of smaller ranges.

This leads to one of the slowest parts of the code. We transmit 10 million packets per second and have to convert an index variable to an IP address for each and every probe. We solve this by doing a "binary search" in a small amount of memory. At this packet rate, cache efficiencies start to dominate over algorithm efficiencies. There are a lot of more efficient techniques in theory, but they all require so much memory as to be slower in practice.

We call the function that translates from an index into an IP address the pick() function. In use, it looks like:

for (i = 0; i < range; i++) {
    ip = pick(addresses, i);
    scan(ip);
}

Masscan supports not only IP address ranges, but also port ranges. This means we need to pick from the index variable both an IP address and a port. This is fairly straightforward:

range = ip_count * port_count;
for (i = 0; i < range; i++) {
    ip   = pick(addresses, i / port_count);
    port = pick(ports,     i % port_count);
    scan(ip, port);
}

This leads to another expensive part of the code. The division/modulus instructions are around 90 clock cycles, or 30 nanoseconds, on x86 CPUs. When transmitting at a rate of 10 million packets/second, we have only 100 nanoseconds per packet. I see no way to optimize this any better. Luckily, though, two such operations can be executed simultaneously, so doing two of these, as shown above, is no more expensive than doing one.

There are actually some easy optimizations for the above performance problems, but they all rely upon i++, the fact that the index variable increases one by one through the scan. Actually, we need to randomize this variable. We need to randomize the order of IP addresses that we scan or we'll blast the heck out of target networks that aren't built for this level of speed. We need to spread our traffic evenly over the target.

The way we randomize is simply by encrypting the index variable. By definition, encryption is random and creates a 1-to-1 mapping between the original index variable and the output. This means that while we linearly go through the range, the output IP addresses are completely random. In code, this looks like:

range = ip_count * port_count;
for (i = 0; i < range; i++) {
    x = encrypt(i);
    ip   = pick(addresses, x / port_count);
    port = pick(ports,     x % port_count);
    scan(ip, port);
}

This also has a major cost. Since the range is an unpredictable size instead of a nice even power of 2, we can't use cheap binary techniques like AND (&) and XOR (^). Instead, we have to use expensive operations like MODULUS (%). In my current benchmarks, it's taking 40 nanoseconds to encrypt the variable.

This architecture allows for lots of cool features. For example, it supports "shards". You can set up 5 machines each doing a fifth of the scan or range / shard_count. Shards can be multiple machines, or simply multiple network adapters on the same machine, or even (if you want) multiple IP source addresses on the same network adapter.

Or, you can use a 'seed' or 'key' to the encryption function, so that you get a different order each time you scan, like x = encrypt(seed, i).

We can also pause the scan by exiting out of the program, and simply remembering the current value of i, and restart it later. I do that a lot during development. I see something going wrong with my Internet scan, so I hit to stop the scan, then restart it after I've fixed the bug.

Another feature is retransmits/retries. Packets sometimes get dropped on the Internet, so you can send two packets back-to-back. However, something that drops one packet may drop the immediately following packet. Therefore, you want to send the copy about 1 second apart. This is simple. We already have a 'rate' variable, which is the number of packets-per-second rate we are transmitting at, so the retransmit function is simply to use i + rate as the index. One of these days I'm going to do a study of the Internet, and differentiate "back-to-back", "1 second", "10 second", and "1 minute" retransmits this way in order to see if there is any difference in what gets dropped.

C10 Scalability

The asynchronous technique is known as a solution to the "c10k problem". Masscan is designed for the next level of scalability, the "C10M problem".

The C10M solution is to bypass the kernel. There are three primary kernel bypasses in Masscan:

custom network driver
user-mode TCP stack
user-mode synchronization

Masscan can use the PF_RING DNA driver. This driver DMAs packets directly from user-mode memory to the network driver with zero kernel involvement. That allows software, even with a slow CPU, to transmit packets at the maximum rate the hardware allows. If you put 8 10-gbps network cards in a computer, this means it could transmit at 100-million packets/second.

Masscan has its own built-in TCP stack for grabbing banners from TCP connections. This means it can easily support 10 million concurrent TCP connections, assuming of course that the computer has enough memory.

Masscan has no "mutex". Modern mutexes (aka. futexes) are mostly user-mode, but they have two problems. The first problem is that they cause cache-lines to bounce quickly back-and-forth between CPUs. The second is that when there is contention, they'll do a system call into the kernel, which kills performance. A mutex on the fast path of a program severely limits scalability. Instead, Masscan uses "rings" to synchronize things, such as when the user-mode TCP stack in the receive thread needs to transmit a packet without interfering with the transmit thread.

Portability

The code runs well on Linux, Windows, and Mac OS X. All the important bits are in standard C (C90). Therefore, it compiles on Visual Studio with Microsoft's compiler, the Clang/LLVM compiler on Mac OS X, and GCC on Linux.

Windows and Macs aren't tuned for packet transmit, and get only about 300,000 packets-per-second, whereas Linux can do 1,500,000 packets/second. That's probably faster than you want anyway.

Safe code

A bounty is offered for vulnerabilities, see the VULNINFO.md file for more information.

This project uses safe functions like safe_strcpy() instead of unsafe functions like strcpy().

This project has automated unit regression tests (make regress).

Compatibility

A lot of effort has gone into making the input/output look like nmap, which everyone who does port scans is (or should be) familiar with.

IPv6 and IPv4 coexistence

Masscan supports IPv6, but there is no special mode, both are supported at the same time. (There is no -6 option -- it's always available).

In any example you see of masscan usage, simply put an IPv6 address where you see an IPv4 address. You can include IPv4 and IPv6 addresses simultaneously in the same scan. Output includes the appropriate address at the same location, with no special marking.

Just remember that IPv6 address space is really big. You probably don't want to scan for big ranges, except maybe the first 64k addresses of a subnet that were assigned via DHCPv6.

Instead, you'll probably want to scan large lists of addresses stored in a file (--include-file filename.txt) that you got from other sources. Like everywhere else, this file can contain lists of both IPv4 and IPv6 addresses. The test file I use contains 8 million addresses. Files of that size need a couple extra seconds to be read on startup (masscan sorts the addresses and removes duplicates before scanning).

Remember that masscan contains its own network stack. Thus, the local machine you run masscan from does not need to be IPv6 enabled -- though the local network needs to be able to route IPv6 packets.

PF_RING

To get beyond 2 million packets/second, you need an Intel 10-gbps Ethernet adapter and a special driver known as "PF_RING ZC" from ntop. Masscan doesn't need to be rebuilt in order to use PF_RING. To use PF_RING, you need to build the following components:

libpfring.so (installed in /usr/lib/libpfring.so)
pf_ring.ko (their kernel driver)
ixgbe.ko (their version of the Intel 10-gbps Ethernet driver)

You don't need to build their version of libpcap.so.

When Masscan detects that an adapter is named something like zc:enp1s0 instead of something like enp1s0, it'll automatically switch to PF_RING ZC mode.

A more detail discussion can be found in PoC||GTFO 0x15.

Regression testing

The project contains a built-in unit test:

$ make test
bin/masscan --selftest
selftest: success!

This tests a lot of tricky bits of the code. You should do this after building.

Performance testing

To test performance, run something like the following to a throw-away address, to avoid overloading your local router:

$ bin/masscan 0.0.0.0/4 -p80 --rate 100000000 --router-mac 66-55-44-33-22-11

The bogus --router-mac keeps packets on the local network segments so that they won't go out to the Internet.

You can also test in "offline" mode, which is how fast the program runs without the transmit overhead:

$ bin/masscan 0.0.0.0/4 -p80 --rate 100000000 --offline

This second benchmark shows roughly how fast the program would run if it were using PF_RING, which has near zero overhead.

By the way, the randomization algorithm makes heavy use of "integer arithmetic", a chronically slow operation on CPUs. Modern CPUs have doubled the speed at which they perform this calculation, making masscan much faster.

Authors

This tool created by Robert Graham: email: robert_david_graham@yahoo.com twitter: @ErrataRob

License

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, version 3 of the License.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program. If not, see https://www.gnu.org/licenses/.

GitHub C Weekly Trending

Genymobile/scrcpy

scrcpy (v4.0)

Prerequisites

Get the app

Must-know tips

Usage examples

User documentation

Resources

Articles

Contact

Donate

License

cactus-compute/cactus

Cactus

Quick Demo (Mac)

Cactus Engine

Cactus Graph

API & SDK References

Benchmarks (CPU-only, no GPU)

Supported Transcription Model

Supported LLMs

Roadmap

Using this repo

Maintaining Organisations

Citation

mit-pdos/xv6-riscv

nginx/nginx

Table of contents

How it works

Modules

Configurations

Runtime

Downloading and installing

Stable and Mainline binaries

Linux binary installation process

Upgrades

FreeBSD installation process

Windows executables

Dynamic modules

Getting started with NGINX

Installing SSL certificates and enabling TLS encryption

Load Balancing

Rate limiting

Content caching

Building from source

Installing dependencies

Installing compiler and make utility

Installing dependency libraries

Cloning the NGINX GitHub repository

Configuring the build

Compiling

Location of binary and installation

Running and testing the installed binary

Asking questions and reporting issues

Contributing code

Additional help and resources

Changelog

License

DavidXanatos/TaskExplorer

TaskExplorer

Features

Screen Shots

System Requirements

Additional Information

Support

curl/curl

Open Source

Contact

Commercial support

Website

Source code

Security problems

Backers

Sponsors

HansKristian-Work/vkd3d-proton

vkd3d-proton

Upstream

Priorities

Drivers