GitHub Dockerfile Daily Trending

piyushsachdeva/CKA-2024

This repository contains the notes and code snippets of the CKA 2025 YouTube Playlist. The content is based on the latest 2025 curriculum and includes hands-on demos, assignments, and exam-based scenarios. I will be covering everything from the basics to the advanced level.

Certified Kubernetes Administrator(CKA) 20245

This is the GitHub repository to host all the notes, diagrams, assignments, and resources from the CKA course published on YouTube.

Note: If you found the content helpful, consider giving the repository a star ⭐

Checkout this file for the #40daysofkubernetes challenge details

🔗 Join our community 👇

🏆 #40daysofkubernetes Challenge - COMPLETED!

The #40daysofkubernetes challenge has been successfully completed! 🎉

Congratulations to all the amazing participants who dedicated their time and effort to learn Kubernetes throughout this journey.

🥇 Challenge Winners

🖊️ Blogging Champions

Shivam Gautam
Rahul Vadakkiniyil
Shriram Sahu
Teddy D
Pratik Shinde
Baivab Mukhopadhyay

👥 Community Champions

Baivab Mukhopadhyay - Discord: itzmario7320
Priyanka
Vivek
Dheeman Das
Micheal Choudhary

A huge thanks to everyone who participated, shared their learning journey, and contributed to building this amazing community! 🙏

Day0: Intro to the series

Course Introduction
Discord community server for assistance
Live QnA sessions every weekend
#40daysofkubernetes challenge

Day 1: Docker Fundamentals

Status: Video is live, check out 👉 Day1 👈 folder for notes and useful links ✅

What is Docker?
How is it different from Virtual Machines
Docker Architecture
Docker flow
Docker commands

Day 2 Video: Dockerize an application

Status: Video is live, check out 👉 Day2 👈 folder for notes and useful links ✅

What is Dockerfile, and how do you write it?
docker pull , push, tag etc

Day3 Video: Docker Multi-Stage Builds

Status: Video is live, check out 👉 Day3 👈 folder for notes and useful links ✅

How to write a dockerfile for multistage build
Benefits of multi-stage builds
Other docker best practices

Day 4 Video: Why do We need Kubernetes?

Status: Video is live, check out 👉 Day4 👈 folder for notes and useful links ✅

Day 5 Video: Kubernetes Architecture

Status: Video is live, check out 👉 Day5 👈 folder for notes and useful links ✅

Control plane VS Worker Nodes
Overview of control plane components

Day 6 Video: Install Kubernetes Cluster locally

Status: Video is live, check out 👉 Day6 👈 folder for notes and useful links ✅

Install Kind cluster locally
How to access the cluster

Day7 Video: Pods in Kubernetes

Status: Video is live, check out 👉 Day7 👈 folder for notes and useful links ✅

What are pods in Kubernetes?
Containers VS Pods
Imperative VS Declarative way for creating Kubernetes resources
Create a sample pod using the imperative way
Create a sample pod using the declarative way
Inspect the pods

Day8 Video: Replicasets and Deployments in Kubernetes:

Status: Video is live, check out 👉 Day8 👈 folder for notes and useful links ✅

Replication Controller
ReplicaSet
Deployments
How to perform Rolling updates/rollback
Scale the deployment

Day9 Video: Services in Kubernetes:

Status: Video is live, check out 👉 Day9 👈 folder for notes and useful links ✅

What are services in Kubernetes, and why do we need them?
Node port, ClusterIP, and LoadBalancer

Day 10 Video: Namespaces:

Status: Video is live, check out 👉 Day10 👈 folder for notes and useful links ✅

NameSpaces
Services and namespaces

Day 11 Video: Multi-container Pods

Status: Video is live, check out 👉 Day11 👈 folder for notes and useful links ✅

What are multi-container pods
Multi-container pods pattern - sidecar/init etc
Environment variables in Kubernetes

Day 12 Video: Daemonset, Cronjob, and job

Status: Video is live, check out 👉 Day12 👈 folder for notes and useful links ✅

What are Daemonset, cronjobs and Jobs
Cron fundamentals with examples

Day13 Video: Static Pods

Status: Video is live, check out 👉 Day13 👈 folder for notes and useful links ✅

What are static pods
Labels and selectors
Manual Scheduling

Day14 Video: Taints and Tolerations

Status: Video is live, check out 👉 Day14 👈 folder for notes and useful links ✅

Day15 Video: Node Affinity

Status: Video is live, check out 👉 Day15 👈 folder for notes and useful links ✅

Day16 Video: Resource Requests and Limits

Status: Video is live, check out 👉 Day16 👈 folder for notes and useful links ✅

Day17 Video: Autoscaling in Kubernetes

Status: Video is live, check out 👉 Day17 👈 folder for notes and useful links ✅

Horizontal VS Vertical Autoscaling
HPA, VPA, Cluster autoscaling, NAP
Metrics server

Day18 Video: Probes in Kubernetes

Status: Video is live, check out 👉 Day18 👈 folder for notes and useful links ✅

Liveness VS Readiness Probes
HTTP/TCP/Command-based health checks

Day19 Video: Config maps and Secrets

Status: Video is live, check out 👉 Day19 👈 folder for notes and useful links ✅

concept and demo

Day 20 Video: How SSL/TLS works

Status: Video is live, check out 👉 Day20 👈 folder for notes and useful links ✅

Symmetric VS Asymmetric encryption
SSL certificates and Certificate Authority

Day 21 Video: TLS in Kubernetes

Status: Video is live, check out 👉 Day21 👈 folder for notes and useful links ✅

How TLS works in Kubernetes
Why we need TLS in Kubernetes
Private key and public certificates

Day 22 Video: Authorization in Kubernetes

Status: Video is live, check out 👉 Day22 👈 folder for notes and useful links ✅

Authorization VS Authentication
Authorization types, ABAC, RBAC, Node, Webhook
Kubeconfig

Day 23 Video: Role-based access control (RBAC)

Status: Video is live, check out 👉 Day23 👈 folder for notes and useful links ✅

Role and role binding
Generate and approve the certificate
grant access to the user

Day 24 Video: Cluster role and cluster role binding

Status: Video is live, check out 👉 Day24 👈 folder for notes and useful links ✅

concept and demo

Day 25 Video: Service Account

Status: Video is live, check out 👉 Day25 👈 folder for notes and useful links ✅

What are service accounts, and why do we use them?
Create a service account and grant access to it

Day26 Video: Network Policies

Status: Video is live, check out 👉 Day26 👈 folder for notes and useful links ✅

Network policy concept
CNI installation
enforce network policy by creating the object

Day27 Video: Use Kubeadm to install a Kubernetes cluster

Status: Video is live, check out 👉 Day27 👈 folder for notes and useful links ✅

Provision underlying infrastructure to deploy a Kubernetes cluster
Setup Master Node to deploy Kubernetes components
Setup multiple worker nodes and join the master node

Day28 Video: Docker storage fundamentals

Status: Video is live, check out 👉 Day28 👈 folder for notes and useful links ✅

Why do we need storage in docker containers
persistent docker storage

Day29 Video: Storage in Kubernetes

How storage works in Kubernetes
hostpath volumes in Kubernetes
Persistent volumes and Persistent volume claims
Volume modes, Access modes, and reclaim policies for volumes
Storage classes and provisions

Day30 Video: How does DNS work?

What happens when you type a website address in your browser
different components involved in DNS
End-to-end flow
Important files and resources

Day31 Video: DNS in kubernetes

How DNS works in Kubernetes
Core-DNS

Day32 Video: Kubernetes Networking

CNI , Network Add-on
Containerd vs runc , container runtime

Day 33 Video: Ingress controller and Ingress resources

Day 34 Video: Perform a version upgrade on a Kubernetes cluster using Kubeadm

Day 35 Video: Implement etcd backup and restore

Day 36 Video: Monitoring, Logging and Alerting

Monitor Cluster components, Evaluate cluster and node logging
Understand how to monitor applications, metric server
Manage container stdout & stderr logs

Day 37 Video: Troubleshoot application failure

Day 38 Video: Troubleshoot cluster component failure

Day 39 Video: Network Troubleshooting

Worker node failure
cordon, uncordon and drain (maintenance)\

Day 40 Video: JSONPath, advance kubectl commands

JSON for beginners
JSON v/s YAML
JSONPATH basics
Multiple JSONPATH queries to fetch details

Day 41 Mission CKA

Exam Pattern
Pre-requisites
Last-minute preparation
Tips and Tricks
Sample questions

Day 42 Video: Realtime project: Host your own container registry on Kubernetes

This project will include multiple Kubernetes topics with real-time implementation.

Day 43 Video: Helm

Day 44 Video - Kustomize

Day 45 Video - StatefulSets

Day 46 Video - Priority class and preemption

Frequently asked questions:

I've been overwhelmed by the fantastic response to my new YouTube series on the Certified Kubernetes Administrator (CKA) exam! But with all the excitement came some questions, and I'm here to address them head-on.

1) 𝐈 𝐜𝐚𝐧𝐧𝐨𝐭 𝐚𝐟𝐟𝐨𝐫𝐝 𝐭𝐡𝐞 𝐂𝐊𝐀 𝐞𝐱𝐚𝐦 𝐢𝐭'𝐬 𝐭𝐨𝐨 𝐞𝐱𝐩𝐞𝐧𝐬𝐢𝐯𝐞. 𝐒𝐡𝐨𝐮𝐥𝐝 𝐈 𝐬𝐭𝐢𝐥𝐥 𝐟𝐨𝐥𝐥𝐨𝐰 𝐚𝐥𝐨𝐧𝐠?

The content of this playlist is based on the exam curriculum because it provides a well-organized learning path. You don't have to worry about the roadmap; you can focus on learning. Even if you are not taking the exam now, you are still gaining the hands-on knowledge that a Kubernetes administrator should have; that's good enough!

𝟐) 𝐖𝐡𝐚𝐭 𝐚𝐫𝐞 𝐭𝐡𝐞 𝐩𝐫𝐞𝐫𝐞𝐪𝐮𝐢𝐬𝐢𝐭𝐞𝐬 𝐟𝐨𝐫 𝐭𝐡𝐢𝐬 𝐜𝐨𝐮𝐫𝐬𝐞?

The only prerequisites are basic Linux understanding; all the other prerequisites are already covered in this series. Check out the GitHub repo below for the course content. Yes, it is absolutely free for anyone to join and follow along.

𝟑) 𝐖𝐡𝐢𝐜𝐡 𝐂𝐥𝐨𝐮𝐝 𝐩𝐫𝐨𝐯𝐢𝐝𝐞𝐫 𝐚𝐫𝐞 𝐰𝐞 𝐮𝐬𝐢𝐧𝐠 𝐟𝐨𝐫 𝐭𝐡𝐞 𝐝𝐞𝐦𝐨?

We will not be using any Cloud as the focus is on Kubernetes. Using a cloud service would mean the cloud provider manages the control plane, limiting learning. We will use a local Kubernetes installation on a Kind cluster and a Kubeadm installation on virtual machines.

𝟒) 𝐀𝐫𝐞 𝐰𝐞 𝐝𝐨𝐢𝐧𝐠 𝐚𝐧𝐲 𝐫𝐞𝐚𝐥-𝐭𝐢𝐦𝐞 𝐩𝐫𝐨𝐣𝐞𝐜𝐭𝐬 𝐢𝐧 𝐭𝐡𝐢𝐬 𝐬𝐞𝐫𝐢𝐞𝐬?

Yes, there will be a couple of real-time projects at the end of this series. We cannot do real-time full-fledged projects in Kubernetes without completing all the essential topics. That is why I added the projects at the end and hands-on tasks in almost every video.

5) 𝐇𝐨𝐰 𝐦𝐚𝐧𝐲 𝐯𝐢𝐝𝐞𝐨𝐬 𝐰𝐢𝐥𝐥 𝐛𝐞 𝐩𝐮𝐛𝐥𝐢𝐬𝐡𝐞𝐝 𝐞𝐯𝐞𝐫𝐲 𝐰𝐞𝐞𝐤?

I have finished recording the entire series and am editing the last few videos. I plan to publish 3 to 4 weekly videos every Monday through Thursday. I have chosen not to publish daily so that viewers following the series have time to understand the concepts, practice hands-on, document their learning, and not feel left behind.

6) 𝐖𝐡𝐚𝐭 𝐢𝐟 𝐰𝐞 𝐚𝐫𝐞 𝐬𝐭𝐮𝐜𝐤 𝐬𝐨𝐦𝐞𝐰𝐡𝐞𝐫𝐞 𝐚𝐧𝐝 𝐧𝐞𝐞𝐝 𝐚𝐬𝐬𝐢𝐬𝐭𝐚𝐧𝐜𝐞?

You can comment on the video by providing complete details about the issue/query, or you can open a new thread in the discord channel of help-cka-2024, and someone will help you. I also plan to host a live Q&A session every weekend.

TechHutTV/homelab

Homelab stacks, templates, and more fun resources!

My Homelab Setup

This repo contains Docker Compose stacks, configuration templates, and documentation for everything running in my homelab. It spans two servers - an Unraid NAS for media and a Proxmox node for everything else - tied together with Ubiquiti networking, self-hosted cloud services, and a full monitoring stack.

Whether you're looking for a complete Servarr media setup behind a VPN, a Frigate NVR config with Coral TPU, or just want to see how someone else organizes their self-hosted infrastructure, feel free to dig around and use whatever is helpful. This is a work in progress - I still have a ton to update and add.

Apps - List of all the apps and services.
Automations - n8n workflows and automation.
Cloud - NextCloud, Immich, and self-hosted cloud services.
Home Assistant - Smart home services and automation.
Media Server - Plex, Jellyfin, *arr stack, and more.
Monitoring - Graphs and visualizations for Unraid, Proxmox, and more.
NetBird - Self-hosted zero-trust networking with PocketID.
Proxy Management - NGINX Proxy Manager, DDNS with Cloudflare, local domains, and more.
Storage - Current storage and backup solution.
Surveillance - Frigate NVR solution with Coral TPU.

Hardware

Compute

	OpnNAS SL-R12 Pro	Miniforum AI X1 Pro
OS	Unraid	Proxmox
Role	Media server, NextCloud VM	Grafana, NPM, n8n, and other services
CPU	Intel Core i5-13500T	AMD Ryzen AI 9 HX 370
RAM	Crucial Pro 96GB DDR5	96GB SODIMM DDR5
Storage	5x Seagate Exos 20TB, 7x Seagate IronWolf 4TB	2x WD SN580 1TB NVMe (mirrored boot)
Link	learn more	get it here

The OpnNAS is an early release press model - check their site for availability. The Miniforum handles all non-media workloads as a dedicated Proxmox node.

Networking

Ubiquiti sent all of this over to me. There is no way I'd be able to afford all this stuff on my own dime, so note that. This setup is crazy overpowered for what I need.

Device	Description
UDM Pro Max	Primary router, gateway, and UniFi Protect NVR with 2x 8TB HDDs for camera storage.
USW Pro Max 24 PoE	Main 24-port PoE switch powering cameras and access points.
USW Pro XG 10 PoE	10G aggregation switch for high-bandwidth connections between devices.
2x U7 Pro XG	Wi-Fi 7 access points providing wireless coverage.
USW Lite 16 PoE	Secondary 16-port PoE switch for additional wired devices.

Accessories

1x U-Rack-6U-TL
1x UACC-Cable-Patch-EL-0.15M-W-24
5x UACC-Cable-Patch-EL-3M-W
2x UACC-Cable-Patch-EL-15M-W

Security

Currently running two camera systems - originally fully Reolink, but added UniFi Protect after Ubiquiti sent equipment to try. See the video overview.

Camera	Details
3x UVC-G6-Turret-W	Connected to the UDM Pro Max via UniFi Protect. Planning to slowly transition fully to Ubiquiti as budget allows.
Reolink PoE	RLC-1224A 12MP and 4K Floodlight. Great for the Reolink ecosystem and 3rd party services like Frigate.
Reolink NVR Kit	16CH 12MP (my kit) - simple PoE plug-and-play solution. Kit cameras require the NVR to function. Budget option: 8CH 5MP.

Other Devices

Raspberry Pi 5 (PoE HAT) - Twingate connector and Cloudflare DDNS updater.
Raspberry Pi 4 - OctoPrint for 3D printer management.
VPS (Hostinger KVM) - Self-hosted NetBird management server.

Previous hardware? See old_hardware.md.

Stars

mikeroyal/Self-Hosting-Guide

Self-Hosting Guide. Learn all about locally hosting (on premises & private web servers) and managing software applications by yourself or your organization. Including Cloud, LLMs, WireGuard, Automation, Home Assistant, and Networking.

Self Hosting Guide

A guide for getting started with Self Hosting devices including software and hardware that will make you a better and more efficient Self Hosting.

Note: You can easily convert this markdown file to a PDF in VSCode using this handy extension Markdown PDF.

Note 2: This guide will constantly be updated with new info as becomes available and please feel to make an issue if you think something should be added.

Getting Started with Self-Hosting

Back to the Top

Self-Hosting is the practice of locally hosting(on premises & private web servers) and managing software applications by a person or organization instead of monthly subscriptions from Software as a service (SaaS) providers.

Most self-hosted software can be installed using Docker, a packaging system which allows software to bundle their configuration and dependencies and isolate them from your operating system. Software using docker can be installed using the command line or via graphical interfaces such as Portainer. Software is installed with Docker by downloading an image file containing the application, then creating a copy that sets up its own dependencies and configuration within what is called a container. Without containers you would often need to install different versions of the same programming languages or tools to satisfy the dependencies for the software you want to use which can get complicated.

Tools for Self-Hosting

Back to the Top

Containers

Back to the Top

Container is a standard unit of software that packages up code and all its dependencies(including CPU, memory, file storage, and network connections) so the application runs quickly and reliably from one computing environment to another.

Application Container Security Guide | NIST (PDF)

Container Image is a lightweight, standalone, executable package of software that includes everything needed to run an application such as the code, runtime, system tools, system libraries, and settings.

Best places to get Container Images:

Docker Compose is a tool that was developed to help define and share multi-container applications. With Compose, we can create a YAML file to define the services and with a single command, can spin everything up or tear it all down.

Docker Include is a Compose application can declare dependency on another Compose application. This is useful if you want to reuse other Compose files. Also, if you need to factor out parts of your application model into separate Compose files so they can be managed separately or shared with others.

Kompose is a conversion tool for Docker Compose to container orchestrators such as Kubernetes or OpenShift.

SwarmKit is a toolkit for orchestrating distributed systems at any scale. It includes primitives for node discovery, raft-based consensus, task scheduling and more.

Containerd is a daemon that manages the complete container lifecycle of its host system, from image transfer and storage to container execution and supervision to low-level storage to network attachments and beyond. It is available for Linux and Windows.

ContainersSSH is an SSH Server that Launches Containers in Kubernetes and Docker on demand.

Podman is a daemonless, open source, Linux native tool designed to make it easy to find, run, build, share and deploy applications using Open Containers Initiative (OCI) Containers and Container Images. Podman provides a command line interface (CLI) familiar to anyone who has used the Docker Container Engine.

Lima is a tool that launches Linux virtual machines with automatic file sharing and port forwarding (similar to WSL2), and containerd. It's a great free and open-source alternative for Docker Desktop.

Colima is a container runtimes on macOS (and Linux) with minimal setup.

Portainer Community Edition is a lightweight service delivery platform for containerized applications that can be used to manage Docker, Swarm, Kubernetes and ACI environments. It is designed to be as simple to deploy as it is to use.

Yacht is a container management UI with a focus on templates and 1-click deployments.

Kitematic is a simple application for managing Docker containers on Mac, Linux and Windows letting you control your app containers from a graphical user interface (GUI).

HashiCorp Nomad is a simple and flexible scheduler and orchestrator to deploy and manage containers and non-containerized applications across on-premises and clouds at scale.

Open Container Initiative is an open governance structure for the express purpose of creating open industry standards around container formats and runtimes.

OpenNebula is an open source platform delivering a simple but feature-rich and flexible solution to build and manage enterprise clouds for virtualized services, containerized applications and serverless computing.

Buildah is a command line tool to build Open Container Initiative (OCI) images. It can be used with Docker, Podman, Kubernetes.

Red Hat Universal Base Images (UBI) is a tool that offers a way to build your container images on a foundation of Red Hat Enterprise Linux software. They are OCI-compliant, container-based, operating system images with complementary runtime languages and packages that are freely redistributable. Easily find UBI images in the Red Hat container catalog, and they are buildable and deployable anywhere.

Red Hat Quay is a project that Builds, Stores, and Distributes your Applications and Containers.

ctop is a tool that provides a concise and condensed overview of real-time metrics for multiple containers as well as a single container view for inspecting a specific container. It comes with built-in support for Docker and runC; connectors for other container and cluster systems are planned for future releases.

runc is a CLI tool for spawning and running containers on Linux according to the OCI specification.

container-images is a collection of container images used in CI across various opencontainers projects.

Clair is an open source project for the static analysis of vulnerabilities in application containers (currently including OCI and Docker).

Shipwright is a WebUI to generate templates for Yacht, Portainer, Docker-Compose, and Unraid.

Alnoda Workspaces is an open-source portable containerized browser-based development environments in Docker containers. You can create your own custom workspace or customize any of the workspaces with your preferred stack of applications without knowing much of the Docker.

Autoheal monitors and restarts unhealthy docker containers.

Dozzle is a small lightweight application with a web based interface to monitor Docker logs. It doesn’t store any log files. It is for live monitoring of your container logs only.

Diun is a tool that receive notifications when a Docker image is updated on a Docker registry.

WatchTower is a process for automating Docker container base image updates.

Kasm Workspaces is a a highly configurable container streaming platform that enables you to stream and deliver containerized applications over the web. It offers tools that you can use to create desktop workspaces and provide access to virtual desktops to end users. It also ensures data loss prevention as well as secure and private web browsing.

Nginx Proxy is an automation tool that sets up a container running nginx and docker-gen. Docker-gen generates reverse proxy configs for nginx and reloads nginx when containers are started and stopped.

Visual Studio Code Dev Containers is an extension that lets you use a Docker container as a full-featured development environment. It allows you to open any folder inside (or mounted into) a container and take advantage of Visual Studio Code's full feature set. A devcontainer.json file in your project tells VS Code how to access (or create) a development container with a well-defined tool and runtime stack.

CI/CD

Back to the Top

CI/CD: Continuous Integration and Continuous Delivery

Drone is a Continuous Delivery system built on container technology. Drone uses a simple YAML configuration file, a superset of docker-compose, to define and execute Pipelines inside Docker containers.

Woodpecker is a CI service, a community fork of Drone.

Travis CI is a hosted continuous integration service used to build and test software projects hosted at GitHub.

Circle CI is a continuous integration and continuous delivery platform that helps software teams work smarter, faster.

Buddy is a fully-featured DevOps platform with no learning curve that packs everything you need from a CI/CD tool.

Buildbot is a continuous integration tool which automates the compile or test cycle required to validate changes to the project code base. It queues jobs, executes the jobs when the required resources are available, and reports the results.

Development

Back to the Top

Proxmox VE(Virtual Environment) is an open-source platform for enterprise virtualization. It has a built-in web interface that you can use to easily manage VMs and containers, software-defined storage and networking, high-availability clustering, and multiple out-of-the-box tools on a single solution.

Terraform provider plugin for Proxmox is a Terraform provider for the Proxmox virtualization platform and exposes Terraform resources to provision QEMU VMs and LXC Containers.

OTF is an open source alternative to Terraform Enterprise. Includes SSO, team management, agents, and as many applies as you can throw hardware at.

Semaphore UI is a modern UI for Ansible. It lets you easily run Ansible playbooks, get notifications about fails, control access to deployment system.

APITable is an API-oriented low-code platform for building collaborative apps and better than all other Airtable open-source alternatives.

Chisel Kubernetes Operator is a Kubernetes operator for Chisel. It allows you to use Chisel as a LoadBalancer provider for your Kubernetes cluster, similar to inlets-operator.

Docker-pgautoupgrade is a PostgreSQL Docker container that automatically upgrades your database. It's whole purpose in life is to automatically detect the version of PostgreSQL used in the existing PostgreSQL data directory, and automatically upgrade it (if needed) to the required version of PostgreSQL.

IT-Tools is a collection of handy online tools for developers, with great UX.

Lazygit is a simple terminal UI for git commands, written in Go with the gocui library.

LazyDocker is a simple terminal UI for both docker and docker-compose, written in Go with the gocui library.

Code-Server is Visual Studio Code running on a remote server, accessible through the browser.

Turbopilot is an open source large-language-model based code completion engine that runs locally on your CPU.

Self-Hosted Sentry nightly is an official bootstrap for running your own Sentry with Docker. Sentry, feature-complete and packaged up for low-volume deployments and proofs-of-concept.

Visual Studio Live Share is a service/extension that enables you to collaboratively edit and debug with others in real time, regardless of the programming languages you're using or app types you're building. You can instantly and securely share your current project, start a joint debugging session, share terminal instances, forward localhost web apps, have voice calls, and more.

GistPad is a Visual Studio Code extension that allows you to edit GitHub Gists and repositories from the comfort of your favorite editor. You can open, create, delete, fork and star gists and repositories, and then seamlessly begin editing files as if they were local, without ever cloning, pushing or pulling anything.

Live Server is an extension for Visual Studio Code that launches a development local Server with live reload feature for static & dynamic pages.

Gitea is a community managed painless self-hosted Git service.

Act is a a tool to run your GitHub Actions locally.

Act runner is a runner for Gitea based on act.

GitLab is an open source end-to-end software development platform with built-in version control, issue tracking, code review, CI/CD, and more. Self-host GitLab on your own servers, in a container, or on a cloud provider.

Bonobo Git Server - Set up your own self hosted git server on IIS for Windows. Manage users and have full control over your repositories with a nice user friendly graphical interface.

Fossil - Distributed version control system featuring wiki and bug tracker.

Gerrit - A code review and project management tool for Git based projects.

Gitblit - Pure Java stack for managing, viewing, and serving Git repositories.

gitbucket - Easily installable GitHub clone powered by Scala.

Gitea - Community managed fork of Gogs, lightweight code hosting solution.

Gitlist - Web-based git repository browser - GitList allows you to browse repositories using your favorite browser, viewing files under different revisions, commit history and diffs.

Gitolite - Gitolite allows you to setup git hosting on a central server, with fine-grained access control and many more powerful features.

GitPrep - Portable Github clone.

Gogs - Painless self-hosted Git Service written in Go.

Kallithea - Source code management system that supports two leading version control systems, Mercurial and Git, with a web interface.

Klaus - Simple, easy-to-set-up Git web viewer that Just Works.

Lavagna - Lavagna is an open-source issue/project management tool designed for small teams. Lightweight, pure Java, easy to install, easy to use.

Leantime - Leantime is a lean project management system for small teams and startups helping to manage projects from ideation through delivery.

Taiga is an open-source project management software for cross-functional teams that work agile across both scrum and kanban frameworks.

Planka is a realtime kanban board for workgroups built with React and Redux.

Microgit - Git hosting service made in Crystal and Lucky. MIT Crystal

OneDev - All-In-One DevOps Platform. With Git Management, Issue Tracking, and CI/CD. Simple yet Powerful.

OpenProject - OpenProject is a web-based project management system.

Pagure - A lightweight, powerful, and flexible git-centric forge with features laying the foundation for federated and decentralized development.

Phorge - Phorge is an open source, community driven platform for collaborating, managing, organizing and reviewing software development projects.

Redmine - Redmine is a flexible project management web application. (Demo,

RhodeCode - RhodeCode is an open source platform for software development teams. It unifies and simplifies repository management for Git, Subversion, and Mercurial.

SCM Manager - The easiest way to share and manage your Git, Mercurial and Subversion repositories over http.

Taiga - Agile Project Management Tool based on the Kanban and Scrum methods.

Titra - Time-tracking solution for freelancers and small teams.

Traq - Project management and issue tracking system written in PHP.

Tuleap - Tuleap is a libre suite to plan, track, code and collaborate on software projects.

UVDesk - UVDesk community is a service oriented, event driven extensible opensource helpdesk system that can be used by your organization to provide efficient support to your clients effortlessly whichever way you imagine.

ZenTao - An agile(scrum) project management system/tool.

k3s-ansible is the easiest way to bootstrap a self-hosted High Availability Kubernetes cluster. A fully automated HA k3s etcd install with kube-vip, MetalLB, and more.

Soft Serve is a tasty, self-hostable Git server for the command line.

Coolify is an open-source & self-hostable Heroku/Netlify alternative.

Corosync Cluster Engine is a Group Communication System with additional features for implementing high availability within applications.

Glow is a terminal based markdown reader designed from the ground up to bring out the beauty—and power—of the CLI. It's used to discover markdown files, read documentation directly on the command line and stash markdown files to your own private collection, so you can read them anywhere.

Deep Lake is a data lake for deep learning applications. Our open-source dataset format is optimized for rapid streaming and querying of data while training models at scale, and it includes a simple API for creating, storing, and collaborating on AI datasets of any size. It can be deployed locally or in the cloud, and it enables you to store all of your data in one place, ranging from simple annotations to large videos.

Node-Red is a low-code programming for event-driven applications.

krunvm is a CLI-based utility for creating microVMs from OCI images, using libkrun and buildah.

Zeal is an offline documentation browser for software developers inspired by Dash.

Web servers

Back to The Top

Web servers

Apache - Most popular web server.

OpenResty Manager - The easiest using, powerful and beautiful OpenResty Manager(Nginx Enhanced Version), open source alternative to OpenResty Edge.

Beakon - A self-host location sharing webserver. Beakon aims to leak as little data as possible and uses mostly self-contained libraries and local database files. Where possible, it will reference local files and not reach out over any network.

Caddy - The HTTP/2 Web Server with Fully Managed TLS.

Cherokee - Lightweight, high-performance web server/reverse proxy.

Lighttpd - Web server more optimized for speed-critical environments.

Nginx - Reverse proxy, load balancer, HTTP cache, and web server.

uWSGI - The uWSGI project aims at developing a full stack for building hosting services.

Web Performance

HAProxy - Software based load Balancing, SSL offloading and performance optimization, compression, and general web routing.

Squid - Caching proxy for the web supporting HTTP, HTTPS, FTP, and more.

Traefik - Taefik is a modern HTTP reverse proxy and load balancer made to deploy microservices with ease.

Varnish - HTTP based web application accelerator focusing on optimizing caching and compression.

LLMs

Back to the Top

Large Language Models (LLMs) is a language model that uses artificial neural networks to generate text (AI chatbots/search engines). Some notable ones are GPT-3, GPT-4, BLOOM, and LLaMA.

llama.cpp is a Port of Facebook's LLaMA model in C/C++.

ollama is a tool to get up and running with Llama 2 and other large language models locally.

LocalAI is a self-hosted, community-driven, local OpenAI-compatible API. Drop-in replacement for OpenAI running LLMs on consumer-grade hardware with no GPU required. It's an API to run ggml compatible models: llama, gpt4all, rwkv, whisper, vicuna, koala, gpt4all-j, cerebras, falcon, dolly, starcoder, and many others.

Serge is a web interface for chatting with Alpaca through llama.cpp. Fully self-hosted & dockerized, with an easy to use API.

OpenLLM is an open platform for operating large language models (LLMs) in production. Fine-tune, serve, deploy, and monitor any LLMs with ease.

Llama-gpt is a self-hosted, offline, ChatGPT-like chatbot. Powered by Llama 2. 100% private, with no data leaving your device.

Llama2 webui is a tool to run any Llama 2 locally with gradio UI on GPU or CPU from anywhere (Linux/Windows/Mac). Use llama2-wrapper as your local llama2 backend for Generative Agents/Apps.

Llama2.c is a tool to Train the Llama 2 LLM architecture in PyTorch then inference it with one simple 700-line C file (run.c).

Alpaca.cpp is a fast ChatGPT-like model locally on your device. It combines the LLaMA foundation model with an open reproduction of Stanford Alpaca a fine-tuning of the base model to obey instructions (akin to the RLHF used to train ChatGPT) and a set of modifications to llama.cpp to add a chat interface.

GPT4All is an ecosystem of open-source chatbots trained on a massive collections of clean assistant data including code, stories and dialogue based on LLaMa.

MiniGPT-4 is an enhancing Vision-language Understanding with Advanced Large Language Models

LoLLMS WebUI is a the hub for LLM (Large Language Model) models. It aims to provide a user-friendly interface to access and utilize various LLM models for a wide range of tasks. Whether you need help with writing, coding, organizing data, generating images, or seeking answers to your questions.

LM Studio is a tool to Discover, download, and run local LLMs.

Ava PLS small, all-in-one desktop app to run LLMs locally.

Gradio Web UI is a tool for Large Language Models. Supports transformers, GPTQ, llama.cpp (ggml/gguf), Llama models.

OpenPlayground is a playfround for running ChatGPT-like models locally on your device.

Vicuna is an open source chatbot trained by fine tuning LLaMA. It apparently achieves more than 90% quality of chatgpt and costs $300 to train.

Yeagar ai is a Langchain Agent creator designed to help you build, prototype, and deploy AI-powered agents with ease.

KoboldCpp is an easy-to-use AI text-generation software for GGML models. It's a single self contained distributable from Concedo, that builds off llama.cpp, and adds a versatile Kobold API endpoint, additional format support, backward compatibility, as well as a fancy UI with persistent stories, editing tools, save formats, memory, world info, author's note, characters, and scenarios.

Minima is a configurable conversational RAG system that runs LLM locally and on-premises using containers.

ChatGPT

Back to the Top

Chatbot UI for ChatGPT

Chatbot UI by mckaywrigley is an advanced chatbot kit for OpenAI's chat models built on top of Chatbot UI Lite using Next.js, TypeScript, and Tailwind CSS. This version of ChatBot UI supports both GPT-3.5 and GPT-4 models. Conversations are stored locally within your browser. You can export and import conversations to safeguard against data loss. See a demo.

Chatbot UI Lite by mckaywrigley is a simple chatbot starter kit for OpenAI's chat model using Next.js, TypeScript, and Tailwind CSS. See a demo.

Build locally with Docker:

docker build -t chatgpt-ui .
docker run -e OPENAI_API_KEY=xxxxxxxx -p 3000:3000 chatgpt-ui

Running Locally on Windows, MacOS, and Linux:

1. Clone Project Repo

git clone https://github.com/mckaywrigley/chatbot-ui.git

2. Install Dependencies

npm i

3. Provide OpenAI API Key

Create a .env.local file in the root of the repo with your OpenAI API Key:

OPENAI_API_KEY=YOUR_KEY

You can set OPENAI_API_HOST where access to the official OpenAI host is restricted or unavailable, allowing users to configure an alternative host for their specific needs.
Additionally, if you have multiple OpenAI Organizations, you can set OPENAI_ORGANIZATION to specify one.

4. Run App

npm run dev

You done you should be able to start chatting with ChatGPT!

Chatbot UI

MiniGPT-4 is an enhancing Vision-language Understanding with Advanced Large Language Models

Launching Demo Locally

Try out the demo demo.py on your local machine by running

python demo.py --cfg-path eval_configs/minigpt4_eval.yaml --gpu-id 0

Here, the demo loads Vicuna as 8 bit by default to save some GPU memory usage. Besides, the default beam search width is 1. Under this setting, the demo cost about 23G GPU memory. If you have a more powerful GPU with larger GPU memory, you can run the model in 16 bit by setting low_resource to False in the config file minigpt4_eval.yaml and use a larger beam search width.

MiniGPT-4 Demo

GPT4All is an ecosystem of open-source chatbots trained on a massive collections of clean assistant data including code, stories and dialogue based on LLaMa.

GPT4All UI is a Flask web application that provides a chat UI for interacting with the GPT4All chatbot.

llama.cpp is a Port of Facebook's LLaMA model in C/C++.

Serge is a web interface for chatting with Alpaca through llama.cpp. Fully self-hosted & dockerized, with an easy to use API.

OpenPlayground is a playfround for running ChatGPT-like models locally on your device.

Vicuna is an open source chatbot trained by fine tuning LLaMA. It apparently achieves more than 90% quality of chatgpt and costs $300 to train.

Yeagar ai is a Langchain Agent creator designed to help you build, prototype, and deploy AI-powered agents with ease.

DoctorGPT is a lightweight self-contained binary that monitors your application logs for problems and diagnoses them.

HttpGPT is an Unreal Engine 5 plugin that facilitates integration with OpenAI's GPT based services (ChatGPT and DALL-E) through asynchronous REST requests, making it easy for developers to communicate with these services. It also includes Editor Tools to integrate Chat GPT and DALL-E image generation directly in the Engine.

Automation

Back to the Top

Accelerated Text - Automatically generate multiple natural language descriptions of your data varying in wording and structure.

Activepieces - No-code business automation tool like Zapier or Tray. For example, you can send a Slack notification for each new Trello card.

ActiveWorkflow - An intelligent process and workflow automation platform based on software agents.

Alltube - Web GUI for youtube-dl, a program to download videos and audio from more than 100 websites.

AmIUnique - Learn how identifiable you are on the Internet (browser fingerprinting tool).

Automatisch - Business automation tool that lets you connect different services like Twitter, Slack, and more to automate your business processes (Open source Zapier alternative).

Baserow - Open source online database tool and Airtable alternative. Create your own database without technical experience.

betanin - Music organization man-in-the-middle of your torrent client and music player. Based on beets.io, similar to Sonarr and Radarr.

ChiefOnboarding - Employee onboarding platform that allows you to provision user accounts and create sequences with todo items, resources, text/email/Slack messages, and more! Available as a web portal and Slack bot.

Datasette - An open source multi-tool for exploring and publishing data, easy import and export and database management.

Eonza - Eonza is used to create scripts and automate tasks on servers or VPS hosting. Manage your servers from any browser on any device.

Exadel CompreFace - Face recognition system that provides REST API for face recognition, face detection, and other face services, and is easily deployed with docker. There are SDKs for Python and JavaScript languages. Can be used without prior machine learning skills.

feed2toot - Feed2toot parses a RSS feed, extracts the last entries and sends them to Mastodon.

feedmixer - FeedMixer is a WSGI (Python3) micro web service which takes a list of feed URLs and returns a new feed consisting of the most recent n entries from each given feed(Returns Atom, RSS, or JSON).

Headphones - Automated music downloader for NZB and Torrent, written in Python. It supports SABnzbd, NZBget, Transmission, µTorrent, Deluge and Blackhole.

Healthchecks - Django app which listens for pings and sends alerts when pings are late.

HRConvert2 - Drag-and-drop file conversion server with session based authentication, automatic temporary file maintenance, and logging capability.

Huginn - Allows you to build agents that monitor and act on your behalf.

Kibitzr - Lightweight personal web assistant with powerful integrations.

Krayin - Free and Opensource Laravel CRM Application.

Leon - Open-source personal assistant who can live on your server.

Lidarr - Lidarr is a music collection manager for Usenet and BitTorrent users.

Matchering - A containerized web app for automated music mastering. An open-source alternative to LANDR, eMastered, and MajorDecibel.

Medusa - Automatic Video Library Manager for TV Shows. It watches for new episodes of your favorite shows, and when they are posted it does its magic. (Source Code) GPL-3.0 Python

MeTube - Web GUI for youtube-dl, with playlist support. Allows downloading videos from dozens of websites. AGPL-3.0 Python/Nodejs/Docker

Nautobot is a Network Source of Truth and Network Automation Platform built as a web application atop the Django Python framework with a PostgreSQL or MySQL database.

nefarious - Web application that automates downloading Movies and TV Shows.

NocoDB - No-code platform that turns any database into a smart spreadsheet. It can be considered as an Airtable or Smartsheet alternative.

OliveTin - OliveTin is a web interface for running Linux shell commands.

Patrowl - Open Source, Smart and Scalable Security Operations Orchestration Platform.

Podgrab - Lightweight podcast manager and automatic podcast episode downloader. It will monitor podcasts for your and download them automatically whenever a new episode goes live.

pyLoad - Lightweight, customizable and remotely manageable downloader for 1-click-hosting sites like rapidshare.com or uploaded.to.

Radarr - Radarr is an independent fork of Sonarr reworked for automatically downloading movies via Usenet and BitTorrent, à la Couchpotato.

SickRage - SickRage is an automatic Video Library Manager for TV Shows. Automatic torrent/nzb searching, downloading, and processing at the qualities you want.

SiteInspector - Web-based tool for catching spelling errors, grammatical errors, broken links, and other errors on websites.

Sonarr - Automatic TV Shows downloader and manager for Usenet and BitTorrent. It can grab, sort and rename new episodes and automatically upgrade the quality of files already downloaded when a better quality format becomes available.

StackStorm - StackStorm (aka IFTTT for Ops) is event-driven automation for auto-remediation, security responses, troubleshooting, deployments, and more. Includes rules engine, workflow, 160 integration packs with 6000+ actions and ChatOps.

µTask - Automation engine that models and executes business processes declared in yaml.

Configuration Management

Back to The Top

Ansible - is a tool is a powerful, agentless tool that works everywhere and with everything. When you add in proven enterprise engineering and support from Red Hat that's written in Python.

Ansible.Ai is an AI for Ansible Content Development tool to automate in your IT infrastructure and it will generate syntactically correct playbook to help you get there.

CFEngine - is a Lightweight agent system where the configuration state is specified via a declarative language.

mgmt - is a next generation config management written in Go.

Pallet - is a Infrastructure definition, configuration and management via a Clojure DSL.

Puppet - is an automated administrative engine for your Linux, Unix, and Windows systems, performs administrative tasks (such as adding users, installing packages, and updating server configurations) based on a centralized specification.

Chef - is a powerful automation platform that transforms infrastructure into code automating how infrastructure is configured, deployed and managed across any environment.

(R)?ex - is a friendly automation framework to any combinations of local and remote execution, push and pull style of management, or imperative and declarative approach.

Salt - is an event-driven automation tool and framework to deploy, configure, and manage complex IT systems. It automates common infrastructure administration tasks and ensure that all the components of your infrastructure are operating in a consistent desired state.

Fleek is an all-in-one management system for everything you need to be productive on your computer.

Cloud Storage

Back to The Top

Swift - A highly available, distributed, eventually consistent object/blob store.

Syncthing - Open Source system for private, encrypted and authenticated distribution of data.

git-annex assistant - A synchronized folder on each of your MacOS and Linux computers, Android devices, removable drives, NAS appliances, and cloud services.

NextCloud - Provides access to your files via the web.

ownCloud - Provides universal access to your files via the web, your computer or your mobile devices.

Seafile - Another Open Source Cloud Storage solution.

SparkleShare - Provides cloud storage and file synchronization services. By default, it uses Git as a storage backend.

Cloud

Back to the Top

Linode

Linode is a cloud hosting company that provides virtual private servers and variety of other cloud services.

Linode Tools

Linode Cloud Manager is a user- and mobile-friendly interface to deploy and manage virtual machines, configure networking, and control user accounts.

Linode API is a tool that makes easy to configure, manage, and deploy user management, billing, support tickets, and more with programmatic access to Linode products and services.

Linode CLI is a tool to deploy and manage Linux servers from Linode without leaving the command line.

Linode Images is a service to capture, store, and deploy your custom images across Linodes or data centers. Easily create your own raw disk image and upload a compressed .gz image file (up to 5 GB) using the Cloud Manager or API to easily deploy to the Linode size and data center you need.

Linode Integrations is a collection of integrations lets you connect infrastructure and dev tools to the Linode platform. That let's you manage your Linode resources using the tools you know and love.

StackScripts is a tool to automatically configure new Linode instances using simple scripts. Create your own StackScript or browse the community StackScript library.

Linode Bare Metal is the single-tenant solution for applications and organizations with security, compliance, and performance needs. Bare Metal combines direct hardware access and the flexibility of a virtual machine.

Nextcloud

Back to the Top

Nextcloud is an industry-leading, on-premises content collaboration platform for file sync & share and communication server. It is fully open source and you can host it yourself or pay a company to do it for you. Also checkout the following links below:

Nexcloud login screen

Nextcloud Hub is a tool that allows you to share and collaborate on documents, send and receive email, manage your calendar and have video chats without data leaks. As fully on-premises solution, Nextcloud Hub provides the benefits of online collaboration without the compliance and security risks.

Nexcloud Hub

Nextcloud AIO (All In One) is a tool that provides easy deployment and maintenance with most features included in this one Nextcloud instance.

Nextcloud Desktop Client is a tool to synchronize files from Nextcloud Server with your computer.

Nextcloud Deck is a kanban style organization tool aimed at personal planning and project organization for teams integrated with Nextcloud.

Nextcloud Files is a tool tool that allows your employees have easy access to their files, photos and documents to work and can share and collaborate with team members, customers and partners. So IT knows nobody besides those they shared with has access to those files.

Nextcloud Talk is a tool that protects your communication better than other team collaboration platforms like Microsoft Teams or Slack, making sure your data stays on your servers. It also goes further than other encrypted communication technologies by keeping even metadata from leaking.

Nextcloud Home is a tool that allows you store your documents, calendar, contacts and photos on your server at home, at one of at one Nextcloud's providers or in a data center you trust.

Nextcloud Enterprise is a service that gives professional organizations software optimized and tested for mission critical environments.

Nextcloud Outlook Integration is a tool that automatically upload files to replace large attachments or integrate Calendars and Contacts in Microsoft Outlook.

Collabora Online in Nextcloud is a powerful LibreOffice-based online office suite with collaborative editing, which supports all major document, spreadsheet and presentation file formats and works in all modern browsers.

ONLYOFFICE integration in Nextcloud is a service that empowers your users to collaborate on office documents with team members in real time. It has compatibility with Microsoft Office formats means perfect documents, every time.

Nextcloud VM(virtual machine appliance) is a set of carefully crafted family of *nix scripts, which interactively guide you through a quality-controlled installation of a Nextcloud instance for Home/SME Server and scripts for Raspberry Pi 4. It is Community developed and maintained.

LibreSign is a Libre digital signature app for Nextcloud.

DigitalOcean

Back to the Top

DigitalOcean is a cloud infrastructure provider that provides developers cloud services that help to deploy and scale applications that run simultaneously on multiple computers with data centers worldwide.

DigitalOcean Tools

DigitalOcean API is a service that manages your DigitalOcean infrastructure with our RESTful API.

DigitalOcean Client libraries is a collection of libraries lets you use the DigitalOcean API in a variety of programming languages.

DigitalOcean CLI is a service that manages your DigitalOcean infrastructure through your terminal with our open source Command Line Interface (CLI).

Terraform provider is a service that allows the user treat their DigitalOcean infrastructure like code with Terraform.

DigitalOcean Custom images is a service that quickly builds your environment in the cloud by provisioning servers with your own custom image, or choose from various Linux distributions.

Container Registry is a service that easily stores, manages, and protects private container images.

Back4app Web Deployment

Back4app Web Deployment is a Container as a Service (CaaS) provider platform that allows the dev teams to build and deploy containerized applications with no downtime. You can simply connect it to a GitHub repository and publish the code within seconds.

MinIO Object Storage

Back to the Top

MinIO is a High Performance Object Storage released under GNU Affero General Public License v3.0. It is API compatible with Amazon S3 cloud storage service. Use MinIO to build high performance infrastructure for machine learning, analytics and application data workloads. It's one of the fastest object storage platforms globally, with a read/write speed of 183GB/s-171GB/s if you use standard hardware. It can function as the main storage tier for many workloads like Spark, TensorFlow, Presto, Hadoop HDFS, and H2O.

MinIO UI

Run the following command to run the latest stable image of MinIO as a container using an ephemeral data volume:

Podman

podman run -p 9000:9000 -p 9001:9001 \
  quay.io/minio/minio server /data --console-address ":9001"

Docker

#docker run -p 9000:9000 --name minio -d minio/minio server /export

If you're using an SSD mounted at /mnt/sdd, then we can run the following to use it instead:

# docker run -v /mnt/ssd:/export -p 9000:9000 --name minio -d minio/minio server /export

MacOS

brew install minio/stable/minio
minio server /data

Binary Download for MacOS

wget https://dl.min.io/server/minio/release/darwin-amd64/minio
chmod +x minio
./minio server /data

Linux

wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
./minio server /data

Architecture	URL
64-bit Intel/AMD	https://dl.min.io/server/minio/release/linux-amd64/minio
64-bit ARM	https://dl.min.io/server/minio/release/linux-arm64/minio
64-bit PowerPC LE (ppc64le)	https://dl.min.io/server/minio/release/linux-ppc64le/minio
IBM Z-Series (S390X)	https://dl.min.io/server/minio/release/linux-s390x/minio

Windows

To run MinIO on 64-bit Windows hosts, download the MinIO executable from the following URL:

https://dl.min.io/server/minio/release/windows-amd64/minio.exe

Use the following command to run a standalone MinIO server on the Windows host. Replace D:\ with the path to the drive or directory in which you want MinIO to store data. You must change the terminal or powershell directory to the location of the minio.exe executable, or add the path to that directory to the system $PATH:

minio.exe server D:\

Install from Source

Use the following commands to compile and run a standalone MinIO server from source. Source installation is only intended for developers and advanced users. If you do not have a working Golang environment, please follow How to install Golang. The minimum version required is go1.19.

go install github.com/minio/minio@latest

After you install MinIO:

The MinIO deployment starts using default root credentials minioadmin:minioadmin. You can test the deployment using the MinIO Console, an embedded web-based object browser built into MinIO Server. Point a web browser running on the host machine to http://127.0.0.1:9000 and log in with the root credentials. You can use the Browser to create buckets, upload objects, and browse the contents of the MinIO server.

When you run Minio you will be issued a key and a secret. These are used by the client or the web front-end to connect securely. I found my codes by typing in docker logs minio.

Created minio configuration file at /root/.minio

Endpoint:  http://172.17.0.2:9000  http://127.0.0.1:9000
AccessKey: accessCode
SecretKey: secretCode
Region:    us-west-1
SQS ARNs:  <none>

Browser Access:
   http://172.17.0.2:9000  http://127.0.0.1:9000

Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
   $ mc config host add myminio http://172.17.0.2:9000 accessCode secretCode

Object API (Amazon S3 compatible):
   Go:         https://docs.minio.io/docs/golang-client-quickstart-guide
   Java:       https://docs.minio.io/docs/java-client-quickstart-guide
   Python:     https://docs.minio.io/docs/python-client-quickstart-guide
   JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide

Drive Capacity: 50 GiB Free, 70 GiB Total

If you'd like to learn more then most of the Minio client commands support a help flag or give info on the command line:

NAME:
  mc - Minio Client for cloud storage and filesystems.

USAGE:
  mc [FLAGS] COMMAND [COMMAND FLAGS | -h] [ARGUMENTS...]

COMMANDS:
  ls       List files and folders.
  mb       Make a bucket or a folder.
  cat      Display file and object contents.
  pipe     Redirect STDIN to an object or file or STDOUT.
  share    Generate URL for sharing.
  cp       Copy files and objects.
  mirror   Mirror buckets and folders.
  diff     Show differences between two folders or buckets.
  rm       Remove files and objects.
  events   Manage object notifications.
  watch    Watch for files and objects events.
  policy   Manage anonymous access to objects.
  session  Manage saved sessions for cp and mirror commands.
  config   Manage mc configuration file.
  update   Check for new mc update.
  version  Print version info.
  help, h  Shows a list of commands or help for one command

Advanced options

You can have your client point to multiple Minio servers, which is really neat especially if you're working on a distributed team.

Minio's test-server called "play" is already configured in the default client, you can see all the servers you have configured with mc config host list.

To upload the photo to Minio's "play" S3 server just type in:

# mc mb play/somebucketname

# mc cp ~/Downloads/IMG_2016120-25.jpg play/somebucketname

Recursive uploads:

If you want to test something larger out you could try uploading your entire Downloads photo, and then you should use the --recursive flag to make sure nothing's missed:

# mc cp --recursive ~/Downloads/IMG_2016120-25.jpg myminio/photos

Databases

Back to the Top

SQL

SQL is a standard language for storing, manipulating and retrieving data in relational databases.

Coolify is an open-source & self-hostable Heroku/Netlify alternative.

MySQL is a fully managed database service to deploy cloud-native applications using the world's most popular open source database.

PostgreSQL is a powerful, open source object-relational database system with over 30 years of active development that has earned it a strong reputation for reliability, feature robustness, and performance.

PostgREST is a tool that serves a fully RESTful API from any existing PostgreSQL database. It provides a cleaner, more standards-compliant, faster API than you are likely to write from scratch.

NocoDB is an open source #NoCode platform that turns any database into a smart spreadsheet. It turns any MySQL, PostgreSQL, SQL Server, SQLite & MariaDB into a smart-spreadsheet.

DBeaver is a free multi-platform database tool for developers, database administrators, analysts and all people who need to work with databases. Supports all popular databases: MySQL, PostgreSQL, SQLite, Oracle, DB2, SQL Server, Sybase, MS Access, Teradata, Firebird, Apache Hive, Phoenix, Presto, etc.

OmniDB is a web-based tool for database management.

Navicat is a series of graphical database management and development software produced by CyberTech Ltd. for MySQL, MariaDB, MongoDB, Oracle, SQLite, PostgreSQL and Microsoft SQL Server.

HeidiSQL is free software, and has the aim to be easy to learn. It lets you see and edit data and structures from computers running one of the database systems MariaDB, MySQL, Microsoft SQL, PostgreSQL and SQLite.

Beekeeper Studio is a cross-platform SQL editor and database manager(MySQL, Postgres, SQLite, SQL Server, and more.) available for Linux, Mac, and Windows.

UI Bakery is a web-based low-code internal tool builder. It can visualize the data pulled from PostgreSQL, MongoDB, MySQL, MicrosoftSQL, Redis.

IBM DB2 is a collection of hybrid data management products offering a complete suite of AI-empowered capabilities designed to help you manage both structured and unstructured data on premises as well as in private and public cloud environments. Db2 is built on an intelligent common SQL engine designed for scalability and flexibility.

OracleDB is a powerful fully managed database helps developers manage business-critical data with the highest availability, reliability, and security.

MariaDB is an enterprise open source database solution for modern, mission-critical applications.

EventQL is a distributed, analytical database. It allows you to store massive amounts of structured data and explore it using SQL and other programmatic query facilities.

CockroachDB is the SQL database for building global, scalable cloud services that survive disasters.

SQLite is a C-language library that implements a small, fast, self-contained, high-reliability, full-featured, SQL database engine.SQLite is the most used database engine in the world. SQLite is built into all mobile phones and most computers and comes bundled inside countless other applications that people use every day.

SQLite Database Browser is an open source SQL tool that allows users to create, design and edits SQLite database files. It lets users show a log of all the SQL commands that have been issued by them and by the application itself.

TimescaleDB is an open-source database designed to make SQL scalable for time-series data. It is engineered up from PostgreSQL and packaged as a PostgreSQL extension, providing automatic partitioning across time and space (partitioning key), as well as full SQL support.

InfluxDB is an open source time series platform. This includes APIs for storing and querying data, processing it in the background for ETL or monitoring and alerting purposes, user dashboards, Internet of Things sensor data, and visualizing and exploring the data and more. It also has support for processing data from Graphite.

Atlas is an in-memory dimensional time series database.

dbWatch is a complete database monitoring/management solution for SQL Server, Oracle, PostgreSQL, Sybase, MySQL and Azure. Designed for proactive management and automation of routine maintenance in large scale on-premise, hybrid/cloud database environments.

Adminer is an SQL management client tool for managing databases, tables, relations, indexes, users. Adminer has support for all the popular database management systems such as MySQL, MariaDB, PostgreSQL, SQLite, MS SQL, Oracle, Firebird, SimpleDB, Elasticsearch and MongoDB.

Knex is a query builder for PostgreSQL, MySQL, CockroachDB, SQL Server, SQLite3 and Oracle, designed to be flexible, portable, and fun to use.

rqlite is an easy-to-use, lightweight, distributed relational database, which uses SQLite as its storage engine.

osquery is a SQL powered operating system instrumentation, monitoring, and analytics framework.

SQLModel is a library for interacting with SQL databases from Python code, with Python objects. It is designed to be intuitive, easy to use, highly compatible, and robust.

Citus is a PostgreSQL extension that transforms Postgres into a distributed database—so you can achieve high performance at any scale.

DbVisualizer is a SQL management tool that allows users to manage a wide range of databases such as Oracle, Sybase, SQL Server, MySQL, H3, and SQLite.

AppDynamics Database is a management product for Microsoft SQL Server. With AppDynamics you can monitor and trend key performance metrics such as resource consumption, database objects, schema statistics and more, allowing you to proactively tune and fix issues in a High-Volume Production Environment.

Toad is a SQL Server DBMS toolset developed by Quest. It increases productivity by using extensive automation, intuitive workflows, and built-in expertise. This SQL management tool resolve issues, manage change and promote the highest levels of code quality for both relational and non-relational databases.

Lepide SQL Server is an open source storage manager utility to analyse the performance of SQL Servers. It provides a complete overview of all configuration and permission changes being made to your SQL Server environment through an easy-to-use, graphical user interface.

Sequel Pro is a fast MacOS database management tool for working with MySQL. This SQL management tool helpful for interacting with your database by easily to adding new databases, new tables, and new rows.

ElasticSearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch is developed in Java.

Logstash is a tool for managing events and logs. When used generically, the term encompasses a larger system of log collection, processing, storage and searching activities.

Kibana is an open source data visualization plugin for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. Users can create bar, line and scatter plots, or pie charts and maps on top of large volumes of data.

Trino is a Distributed SQL query engine for big data. It is able to tremendously speed up ETL processes, allow them all to use standard SQL statement, and work with numerous data sources and targets all in the same system.

Tableau is a Data Visualization software used in relational databases, cloud databases, and spreadsheets. Tableau was acquired by Salesforce in August 2019.

DataGrip is a professional DataBase IDE developed by Jet Brains that provides context-sensitive code completion, helping you to write SQL code faster. Completion is aware of the tables structure, foreign keys, and even database objects created in code you're editing.

RStudio is an integrated development environment for R and Python, with a console, syntax-highlighting editor that supports direct code execution, and tools for plotting, history, debugging and workspace management.

NoSQL

Back to the Top

NoSQL is a database that is interchangeably referred to as "nonrelational, or "non-SQL" to highlight that the database can handle huge volumes of rapidly changing, unstructured data in different ways than a relational (SQL-based) database with rows and tables.

Scylla is the real-time big data database that is API-compatible with Apache Cassandra and Amazon DynamoDB.

Apache Cassandra™ is an open source NoSQL distributed database trusted by thousands of companies for scalability and high availability without compromising performance. Cassandra provides linear scalability and proven fault-tolerance on commodity hardware or cloud infrastructure make it the perfect platform for mission-critical data.

Apache HBase™ is an open-source, NoSQL, distributed big data store. It enables random, strictly consistent, real-time access to petabytes of data. HBase is very effective for handling large, sparse datasets. HBase serves as a direct input and output to the Apache MapReduce framework for Hadoop, and works with Apache Phoenix to enable SQL-like queries over HBase tables.

Hadoop Distributed File System (HDFS) is a distributed file system that handles large data sets running on commodity hardware. It is used to scale a single Apache Hadoop cluster to hundreds (and even thousands) of nodes. HDFS is one of the major components of Apache Hadoop, the others being MapReduce and YARN.

Redis(REmote DIctionary Server) is an open source (BSD licensed), in-memory data structure store, used as a database, cache, and message broker. It provides data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes, and streams.

FoundationDB is an open source distributed database designed to handle large volumes of structured data across clusters of commodity servers. It organizes data as an ordered key-value store and employs ACID transactions for all operations. It is especially well-suited for read/write workloads but also has excellent performance for write-intensive workloads. FoundationDB was acquired by Apple in 2015.

CouchbaseDB is an open source distributed multi-model NoSQL document-oriented database. It creates a key-value store with managed cache for sub-millisecond data operations, with purpose-built indexers for efficient queries and a powerful query engine for executing SQL queries.

MongoDB is a document database meaning it stores data in JSON-like documents.

NoSQLBooster is a cross-platform IDE for MongoDB v2.6-5.0, which provides a build-in MongoDB script debugger, SQL query, server monitoring tools, chaining fluent query, query code generator, task scheduling, ES2020 support, and advanced IntelliSense experience.

ClickHouse® is an open-source column-oriented database management system that allows generating analytical data reports in real-time.

Neo4j is a graph database management system that provides an array of tools, libraries, and frameworks to make development faster and easier.

Remote Access

Back to the Top

FreeRDP is a free remote desktop protocol library and clients.

Rustdesk is an open source virtual/remote desktop infrastructure for everyone. Display and control your PC (Windows, macOS, and Linux) and Android devices.

TinyPilot is a tool that enables KVM over IP letting you control any computer remotely.

X2Go is open source remote desktop software for Linux that uses a modified NX 3 protocol. It gives remote access to a Linux system's GUI.

Apache Guacamole is a clientless remote desktop gateway. It supports standard protocols like VNC, RDP, and SSH.

Remmina is a Remote access screen and file sharing to your desktop. It has Remote Access Protocol Plugins for RDP, SSH, SPICE, VNC, X2Go, HTTP/HTTPS.

Remotely is a remote control and remote scripting solution, built with .NET 6, Blazor, SignalR Core, and WebRTC.

P2P Remote Desktop is a portable, no configuration or installation needed remote desktop tool.

Cloudflare Tunnel is a tunneling daemon that proxies traffic from the Cloudflare network to your origins. This daemon sits between Cloudflare network and your origin (a webserver). This attracts client requests and sends them to you via this daemon, without requiring you to poke holes on your firewall and your origin(webserver) can remain as closed as possible.

WireGuard® is a straight-forward, fast and modern VPN that utilizes state-of-the-art cryptography. It aims to be faster, simpler, leaner, and more useful than IPsec while avoiding the massive headache. WireGuard is designed as a general-purpose VPN for running on embedded interfaces and super computers alike, fit for many circumstances. It's cross-platform (Windows, macOS, BSD, iOS, Android) and widely deployable.

NetBird is an open-source VPN management platform built on top of WireGuard® making it easy to create secure private networks for your organization or home.

Tailscale is a WireGuard-based app that makes secure, private networks easy for teams of any scale. It works like an overlay network between the computers of your networks using all kinds of NAT traversal sorcery.

Headscale is an open source, self-hosted implementation of the Tailscale coordination server.

MeshCentral is a full computer management web site. It can run your own web server to remotely manage and control computers on a local network or anywhere on the internet. Once you get the server started, create device group and download and install an agent on each computer you want to manage.

VNC Viewer is a free remote desktop application that use can use on your iPhone, iPad, Mac, Windows and Linux computers from anywhere in the world.

TightVNC is a free remote desktop application. It can see the desktop of a remote machine and control it with your local mouse and keyboard, just like you would do it sitting in the front of that computer.

KRDC is a client application that allows you to view or even control the desktop session on another machine that is running a compatible server. VNC and RDP is supported.

Krfb Desktop Sharing is a server application that allows you to share your current session with a user on another machine, who can use a VNC client to view or even control the desktop.

wayvnc is a VNC server for wlroots-based Wayland compositors (no_entry Gnome, KDE and Weston are not supported). It attaches to a running Wayland session, creates virtual input devices, and exposes a single display via the RFB protocol.

Waypipe is a proxy for Wayland clients. It forwards Wayland messages and serializes changes to shared memory buffers over a single socket.

Virtualization

Back to the Top

HVM (Hardware Virtual Machine) is a virtualization type that provides the ability to run an operating system directly on top of a virtual machine without any modification, as if it were run on the bare-metal hardware.

PV(ParaVirtualization) is an efficient and lightweight virtualization technique introduced by the Xen Project team, later adopted by other virtualization solutions. PV does not require virtualization extensions from the host CPU and thus enables virtualization on hardware architectures that do not support Hardware-assisted virtualization.

Network functions virtualization (NFV) is the replacement of network appliance hardware with virtual machines. The virtual machines use a hypervisor to run networking software and processes such as routing and load balancing. NFV allows for the separation of communication services from dedicated hardware, such as routers and firewalls. This separation means network operations can provide new services dynamically and without installing new hardware. Deploying network components with network functions virtualization only takes hours compared to months like with traditional networking solutions.

Software Defined Networking (SDN) is an approach to networking that uses software-based controllers or application programming interfaces (APIs) to communicate with underlying hardware infrastructure and direct traffic on a network. This model differs from that of traditional networks, which use dedicated hardware devices (routers and switches) to control network traffic.

Virtualized Infrastructure Manager (VIM) is a service delivery and reduce costs with high performance lifecycle management Manage the full lifecycle of the software and hardware comprising your NFV infrastructure (NFVI), and maintaining a live inventory and allocation plan of both physical and virtual resources.

Management and Orchestration(MANO) is an ETSI-hosted initiative to develop an Open Source NFV Management and Orchestration (MANO) software stack aligned with ETSI NFV. Two of the key components of the ETSI NFV architectural framework are the NFV Orchestrator and VNF Manager, known as NFV MANO.

Magma is an open source software platform that gives network operators an open, flexible and extendable mobile core network solution. Their mission is to connect the world to a faster network by enabling service providers to build cost-effective and extensible carrier-grade networks. Magma is 3GPP generation (2G, 3G, 4G or upcoming 5G networks) and access network agnostic (cellular or WiFi). It can flexibly support a radio access network with minimal development and deployment effort.

OpenRAN is an intelligent Radio Access Network(RAN) integrated on general purpose platforms with open interface between software defined functions. Open RANecosystem enables enormous flexibility and interoperability with a complete openess to multi-vendor deployments.

Open vSwitch(OVS)is an open source production quality, multilayer virtual switch licensed under the open source Apache 2.0 license. It is designed to enable massive network automation through programmatic extension, while still supporting standard management interfaces and protocols (NetFlow, sFlow, IPFIX, RSPAN, CLI, LACP, 802.1ag).

Edge is a distributed computing framework that brings enterprise applications closer to data sources such as IoT devices or local edge servers. This proximity to data at its source can deliver strong business benefits, including faster insights, improved response times and better bandwidth availability.

Multi-access edge computing (MEC) is an Industry Specification Group (ISG) within ETSI to create a standardized, open environment which will allow the efficient and seamless integration of applications from vendors, service providers, and third-parties across multi-vendor Multi-access Edge Computing platforms.

Virtualized network functions(VNFs) is a software application used in a Network Functions Virtualization (NFV) implementation that has well defined interfaces, and provides one or more component networking functions in a defined way. For example, a security VNF provides Network Address Translation (NAT) and firewall component functions.

Cloud-Native Network Functions(CNF) is a network function designed and implemented to run inside containers. CNFs inherit all the cloud native architectural and operational principles including Kubernetes(K8s) lifecycle management, agility, resilience, and observability.

Physical Network Function(PNF) is a physical network node which has not undergone virtualization. Both PNFs and VNFs (Virtualized Network Functions) can be used to form an overall Network Service.

Network functions virtualization infrastructure(NFVI) is the foundation of the overall NFV architecture. It provides the physical compute, storage, and networking hardware that hosts the VNFs. Each NFVI block can be thought of as an NFVI node and many nodes can be deployed and controlled geographically.

Virtualization-based Security (VBS) is a hardware virtualization feature to create and isolate a secure region of memory from the normal operating system.

Hypervisor-Enforced Code Integrity (HVCI) is a mechanism whereby a hypervisor, such as Hyper-V, uses hardware virtualization to protect kernel-mode processes against the injection and execution of malicious or unverified code. Code integrity validation is performed in a secure environment that is resistant to attack from malicious software, and page permissions for kernel mode are set and maintained by the hypervisor.

NVIDIA virtual GPU (vGPU) is a software enables powerful GPU performance for workloads ranging from graphics-rich virtual workstations to data science and AI, enabling IT to leverage the management and security benefits of virtualization as well as the performance of NVIDIA GPUs required for modern workloads.

AMD MxGPU is a hardware-based virtualized GPU solution, is built on industry standard SR-IOV (Single-Root I/O Virtualization) technology and allows multiple virtualized users per physical GPU to work remotely.

Proxmox Virtual Environment(VE) is a complete open-source platform for enterprise virtualization. It inlcudes a built-in web interface that you can easily manage VMs and containers, software-defined storage and networking, high-availability clustering, and multiple out-of-the-box tools on a single solution.

KVM (for Kernel-based Virtual Machine) is a full virtualization solution for Linux on x86 hardware containing virtualization extensions (Intel VT or AMD-V). It consists of a loadable kernel module, kvm.ko, that provides the core virtualization infrastructure and a processor specific module, kvm-intel.ko or kvm-amd.ko.

QEMU is a fast processor emulator using a portable dynamic translator. QEMU emulates a full system, including a processor and various peripherals. It can be used to launch a different Operating System without rebooting the PC or to debug system code.

Quickemu is a program that quickly create and run optimised Windows, macOS and Linux desktop virtual machines.

Hyper-V enables running virtualized computer systems on top of a physical host. These virtualized systems can be used and managed just as if they were physical computer systems, however they exist in virtualized and isolated environment. Special software called a hypervisor manages access between the virtual systems and the physical hardware resources. Virtualization enables quick deployment of computer systems, a way to quickly restore systems to a previously known good state, and the ability to migrate systems between physical hosts.

Cloud Hypervisor is an open source Virtual Machine Monitor (VMM) that runs on top of KVM. The project focuses on exclusively running modern, cloud workloads, on top of a limited set of hardware architectures and platforms. Cloud workloads refers to those that are usually run by customers inside a cloud provider. Cloud Hypervisor is implemented in Rust and is based on the rust-vmm crates.

VirtManager is a graphical tool for managing virtual machines via libvirt. Most usage is with QEMU/KVM virtual machines, but Xen and libvirt LXC containers are well supported. Common operations for any libvirt driver should work.

oVirt is an open-source distributed virtualization solution, designed to manage your entire enterprise infrastructure. oVirt uses the trusted KVM hypervisor and is built upon several other community projects, including libvirt, Gluster, PatternFly, and Ansible. Founded by Red Hat as a community project on which Red Hat Enterprise Virtualization is based allowing for centralized management of virtual machines, compute, storage and networking resources, from an easy-to-use web-based front-end with platform independent access.

Firecracker is an open source virtualization technology that is purpose-built for creating and managing secure, multi-tenant container and function-based services that provide serverless operational models. It runs workloads in lightweight virtual machines, called microVMs, which combine the security and isolation properties provided by hardware virtualization technology with the speed and flexibility of containers.

Foreman is a free open source project that gives you the power to easily automate repetitive tasks, quickly deploy applications, and proactively manage your servers life cycle, on-premises or in the cloud.

Harvester is an open source hyper-converged infrastructure (HCI) software built on Kubernetes.

Anthos is a modern application management platform that provides a consistent development and operations experience for cloud and on-premises environments.

HyperKit is a toolkit for embedding hypervisor capabilities in your application. It includes a complete hypervisor, based on xhyve/bhyve, which is optimized for lightweight virtual machines and container deployment. It is designed to be interfaced with higher-level components such as the VPNKit and DataKit. HyperKit currently only supports macOS using the Hypervisor.framework making it a core component of Docker Desktop for Mac.

Intel® Graphics Virtualization Technology (Intel® GVT) is a full GPU virtualization solution with mediated pass-through, starting from 4th generation Intel Core (TM) processors with Intel processor graphics(Broadwell and newer). It can be used to virtualize the GPU for multiple guest virtual machines, effectively providing near-native graphics performance in the virtual machine and still letting your host use the virtualized GPU normally.

Apple Hypervisor is a frameowrk that builds virtualization solutions on top of a lightweight hypervisor, without third-party kernel extensions. Hypervisor provides C APIs so you can interact with virtualization technologies in user space, without writing kernel extensions (KEXTs). As a result, the apps you create using this framework are suitable for distribution on the Mac App Store.

Apple Virtualization Framework is a framework that provides high-level APIs for creating and managing virtual machines on Apple silicon and Intel-based Mac computers. This framework is used to boot and run a Linux-based operating system in a custom environment that you define. It also supports the Virtio specification, which defines standard interfaces for many device types, including network, socket, serial port, storage, entropy, and memory-balloon devices.

Apple Paravirtualized Graphics Framework is a framework that implements hardware-accelerated graphics for macOS running in a virtual machine, hereafter known as the guest. The operating system provides a graphics driver that runs inside the guest, communicating with the framework in the host operating system to take advantage of Metal-accelerated graphics.

Cilicon is a macOS App that leverages Apple's Virtualization Framework to create, provision and run ephemeral virtual machines with minimal setup or maintenance effort. You should be able to get up and running with your self-hosted CI in less than an hour.

Xen is focused on advancing virtualization in a number of different commercial and open source applications, including server virtualization, Infrastructure as a Services (IaaS), desktop virtualization, security applications, embedded and hardware appliances, and automotive/aviation.

Ganeti is a virtual machine cluster management tool built on top of existing virtualization technologies such as Xen or KVM and other open source software. Once installed, the tool assumes management of the virtual instances (Xen DomU).

Packer is an open source tool for creating identical machine images for multiple platforms from a single source configuration. Packer is lightweight, runs on every major operating system, and is highly performant, creating machine images for multiple platforms in parallel. Packer does not replace configuration management like Chef or Puppet. In fact, when building images, Packer is able to use tools like Chef or Puppet to install software onto the image.

Vagrant is a tool for building and managing virtual machine environments in a single workflow. With an easy-to-use workflow and focus on automation, Vagrant lowers development environment setup time, increases production parity, and makes the "works on my machine" excuse a relic of the past. It provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.

Password Management

Back to the Top

Bitwarden is a free and open-source password management service that stores sensitive information such as website credentials in an encrypted vault.

Bitwarden Server is a project contains the APIs, database, and other core infrastructure items needed for the "backend" of all bitwarden client applications. Checkout Bitwarden's self-hosted release repository.

Vaultwarden is an unofficial Bitwarden compatible server written in Rust, formerly known as bitwarden_rs.

Passbolt is an open-source/self-hosted password manager for teams. It allows you to securely share and store credentials. For instance, the wifi password of your office, the administrator password of a router or your organization's social media account passwords, all of them can be secured using passbolt.

KeePassXC is a modern, secure, and open-source password manager that stores and manages your most sensitive information. You can run KeePassXC on Windows, macOS, and Linux systems. It saves many different types of information, such as usernames, passwords, URLs, attachments, and notes in an offline, encrypted file that can be stored in any location, including private and public cloud solutions.

AuthPass.app is an Open-Source Password Manager for mobile and desktop that is Keepass 2.x (kdbx 3.x) compatible.

pass is an open-source unix-based password utilitiy with various gui clients

SSH

Back to The Top

Secure Shell Protocol (SSH) is a cryptographic network protocol for operating network services securely over an unsecured network.

Advanced SSH config is a tool that enhances ssh_config file capabilities, completely transparent.

AutoSSH is a tool that automatically respawn ssh session after network interruption.

ContainersSSH is an SSH Server that Launches Containers in Kubernetes and Docker on demand.

Cluster SSH is a tool that controls a number of xterm windows via a single graphical console.

DSH is a Dancer's shell / distributed shell - Wrapper for executing multiple remote shell commands from one command line.

Flightplan is a node.js library for streamlining application deployment or systems administration tasks (local and remote hosts).

Mosh is a command-line program, like SSH. You can use it inside xterm, gnome-terminal, urxvt, Terminal.app, iTerm, emacs, screen, or tmux.

Parallel SSH is an asynchronous parallel SSH library designed for large scale automation. It differentiates ifself from alternatives, other libraries and higher level frameworks like Ansible or Chef.

SSH Audit is a tool for SSH server & client configuration auditing (banner, key exchange, encryption, mac, compression, compatibility, security, etc).

Sshwifty is a SSH and Telnet connector made for the Web. It can be deployed on your computer or server to provide SSH and Telnet access interface for any compatible (standard) web browser.

SSHrc is a tool that sources ~/.sshrc on your local computer after logging in remotely.

StormSSH is a command line tool to manage SSH connections.

Tailscale SSH is a service that allows Tailscale to manage the authentication and authorization of SSH connections on your tailnet.

VPN

Back to The Top

VPN (Virtual Private Network) is a service that encrypts your internet traffic on unsecured networks to protect your online identity, hide your IP address, and shield your online data from third parties.

Wireguard - New minimal VPN Solution that is very fast.

OpenVPN - Uses a custom security protocol that utilizes SSL/TLS for key exchange.

Pritunl - OpenVPN based solution that's easy to set up.

SoftEther - Multi-protocol software VPN with advanced features.

sshuttle - Poor man's VPN.

strongSwan - Complete IPsec implementation for Linux.

tinc - Distributed p2p VPN.

LDAP

Back to The Top

LDAP Servers

389 Directory Server - Developed by Red Hat.

Apache Directory Server - Apache Software Foundation project written in Java.

Fusion Directory - Improve the Management of the services and the company directory based on OpenLDAP. OpenDJ - Fork of OpenDS.

OpenDS - Another directory server written in Java.

OpenLDAP - Developed by the OpenLDAP Project.

LDAP management

Apache Directory Studio - The Eclipse-based LDAP browser and directory client

Log Management

Back to The Top

Echofish - A web based real-time event log aggregation, analysis, monitoring and management system.

Fluentd - Log Collector and Shipper.

Flume - Distributed log collection and aggregation system.

Graylog2 - Pluggable Log and Event Analysis Server with Alerting options.

Heka - Stream processing system which may be used for log aggregation.

Elasticsearch - A Lucene Based Document store mainly used for log indexing, storage and analysis.

Kibana - Visualize logs and time-stamped data.

Logstash - Tool for managing events and logs.

Octopussy - Log Management Solution (Visualize/Alert/Report).

DNS

Back to The Top

Duckdns - A free service which will point a DNS (sub domains of duckdns.org) to an IP of your choice. The service is completely free, and doesn't require reactivation or forum posts to maintain its existence.

dnsmasq - A lightweight service providing DNS, DHCP and TFTP services to small-scale networks.

MagicDNS is a tool that automatically registers DNS names for devices in your network.

Bind - The most widely used name server software.

djbdns - A collection of DNS applications, including tinydns.

Designate - DNS REST API that support several DNS servers as its backend.

Knot - High performance authoritative-only DNS server.

Lexicon is a tool that provides a way to manipulate DNS records on multiple DNS providers in a standardized way.

NSD - Authoritative only, high performance, simple name server.

PowerDNS - DNS server with a variety of data storage back-ends and load balancing features.

CoreDNS is a DNS server/forwarder, written in Go, that chains plugins. Each plugin performs a (DNS) function.

Unbound - Validating, recursive, and caching DNS resolver.

Yadifa - Lightweight authoritative Name Server with DNSSEC capabilities powering the .eu top-level domain.

Network Tools

Back to the Top

MQTT is an OASIS standard messaging protocol for the Internet of Things (IoT). It is designed as an extremely lightweight publish/subscribe messaging transport that is ideal for connecting remote devices with a small code footprint and minimal network bandwidth.

Mongoose is a networking library for C/C++. It implements event-driven non-blocking APIs for TCP, UDP, HTTP, WebSocket, MQTT. It is designed for connecting devices and bringing them online.

Nautobot is a Network Source of Truth and Network Automation Platform built as a web application atop the Django Python framework with a PostgreSQL or MySQL database.

Eclipse Mosquitto is an open source implementation of a server for version 5.0, 3.1.1, and 3.1 of the MQTT protocol.

Ejabberd is an open-source, robust, scalable and extensible realtime platform built using Erlang/OTP, that includes XMPP Server, MQTT Broker and SIP Service.

Nebula is a scalable overlay networking tool with a focus on performance, simplicity and security. It lets you seamlessly connect computers anywhere in the world. Nebula is portable, and runs on Linux, OSX, Windows, iOS, and Android. It can be used to connect a small number of computers, but is also able to connect tens of thousands of computers.

LibreSpeed is a network speed test tool that can be run on your LAN or hosted in the cloud.

SmokePing is a deluxe latency measurement tool. It can measure, store and display latency, latency distribution and packet loss. It uses RRDtool to maintain a longterm data-store and to draw pretty graphs, giving up to the minute information on the state of each network connection.

Tailnet is your private network. When you log in for the first time to Tailscale on your phone, laptop, desktop, or cloud VM, a tailnet is created. For personal users, you are a tailnet of many devices and one person. Each device gets a private Tailscale IP address in the CGNAT range and every device can talk directly to every other device, wherever they are on the internet.

Tailscale SSH is a service that allows Tailscale to manage the authentication and authorization of SSH connections on your tailnet.

Tailscale Funnel is a feature that allows you to route traffic from the wider internet to one or more of your Tailscale nodes. You can think of this as publicly sharing a node for anyone to access, even if they don’t have Tailscale themselves.

Cockpit is a web-based graphical interface for servers, intended for everyone. It uses your system’s normal user logins and privileges. Network-wide logins are also supported through single-sign-on and other authentication techniques.

NetBox is a leading solution for modeling and documenting modern networks. It combines the traditional disciplines of IP address management (IPAM) and datacenter infrastructure management (DCIM) with powerful APIs and extensions, NetBox provides the ideal "source of truth" to power network automation.

Network UPS Tools (NUT) is a project that provides support for Power Devices, such as Uninterruptible Power Supplies, Power Distribution Units, Automatic Transfer Switches, Power Supply Units and Solar Controllers. NUT provides a common protocol and set of tools to monitor and manage such devices, and to consistently name equivalent features and data points, across a vast range of vendor-specific protocols and connection media types.

Dnsmasq is a tool that provides network infrastructure for small networks: DNS, DHCP, router advertisement and network boot. It is designed to be lightweight and have a small footprint, suitable for resource constrained routers and firewalls. It has also been widely used for tethering on smartphones and portable hotspots, and to support virtual networking in virtualisation frameworks. Supported platforms include Linux (with glibc and uclibc), Android, BSD, and MacOS.

Nginx proxy manager (NPM) is a reverse proxy management system running on Docker. It's easy to set up and does not require users to know how to work with Nginx servers or SSL certificates.

Netdata is high-fidelity infrastructure monitoring and troubleshooting, real-time monitoring Agent collects thousands of metrics from systems, hardware, containers, and applications with zero configuration. It runs permanently on all your physical/virtual servers, containers, cloud deployments, and edge/IoT devices, and is perfectly safe to install on your systems mid-incident without any preparation.

Pi-hole is a DNS sinkhole that protects your devices from unwanted content, without installing any client-side software, intended for use on a private network. It is designed for use on embedded devices with network capability, such as the Raspberry Pi, but it can be used on other machines running Linux and cloud implementations.

OWASP Amass is a tool that performs network mapping of attack surfaces and external asset discovery using open source information gathering and active reconnaissance techniques.

Smap is a port scanner built with shodan.io's free API. It takes same command line arguments as Nmap and produces the same output which makes it a drop-in replacament for Nmap.

ORY Oathkeeper is an Identity & Access Proxy (IAP) and Access Control Decision API that authorizes HTTP requests based on sets of Access Rules.

Ory Kratos is a developer-friendly, security-hardened and battle-test Identity, User Management and Authentication system for the Cloud. The Kratos identity server (similiar to Auth0, Okta, Firebase) with Ory-hardened authentication, MFA, FIDO2, TOTP, WebAuthn, profile management, identity schemas, social sign in, registration, account recovery, passwordless.

Ory Hydra is a hardened, OpenID Certified OAuth 2.0 Server and OpenID Connect Provider optimized for low-latency, high throughput, and low resource consumption. Ory Hydra is not an identity provider (user sign up, user login, password reset flow), but connects to your existing identity provider through a login and consent app.

Ory Keto is an Open Source (Go) implementation of Zanzibar: Google's Consistent, Global Authorization System. It ships gRPC, REST APIs, newSQL, and an easy and granular permission language. Supports ACL, RBAC, and other access models.

AdGuard Home is a DNS relay station with ad/tracker/other blocking, IP address redirections, and DNS-over-HTTPS.

NetBird is an open-source VPN management platform built on top of WireGuard® making it easy to create secure private networks for your organization or home.

Supabase is an open source Firebase alternative. It is building the features of Firebase using enterprise-grade open source tools.

Plik is a scalable & friendly temporary file upload system (Wetransfer like) in golang.

Restify is a framework, utilizing connect style middleware for building REST APIs.

Traefik is an open source Edge Router that makes publishing your services a fun and easy experience. It receives requests on behalf of your system and finds out which components are responsible for handling them. What sets Traefik apart, besides its many features, is that it automatically discovers the right configuration for your services.

Traefik Mesh is a simple, yet full-featured service mesh. It is container-native and fits as your de-facto service mesh in your Kubernetes cluster. It supports the latest Service Mesh Interface specification SMI that facilitates integration with pre-existing solution.

DuckDNS is a free service that allows you to bind your own favorite subdomain under duckdns.org to the public IP address in use from your router, even though such address is dynamically allocated by your internet service provider and therefore changes over time.

Trust-DNS is a Rust based DNS client, server, and Resolver, built to be safe and secure from the ground up.

Hugo is a static HTML and CSS website generator written in Go. It is optimized for speed, ease of use, and configurability. Hugo takes a directory with content and templates and renders them into a full HTML website.

sshuttle is a transparent proxy server that works as a poor man's VPN that forwards connection over ssh. It works with Linux and MacOS and supports DNS tunneling.

NetHopper is a Multi-Cloud Application Network as a Service. The easiest way to visualize, connect, secure, protect, and monitor microservices across any cluster, site, cloud, or network.

Cypress is a tool that makes it fast, easy and reliable testing for anything that runs in a browser.

Kimchi is an HTML5 based management tool for KVM. It is designed to make it as easy as possible to get started with KVM and create your first guest.

ION is a distributed real-time communication system, the goal is to chat anydevice, anytime, anywhere.

FreeRDP is a free remote desktop protocol library and clients.

Pimox is a port of Proxmox to the Raspberry Pi allowing you to build a Proxmox cluster of Rapberry Pi's or even a hybrid cluster of Pis and x86 hardware.

PiKVM is a very simple and fully functional Raspberry Pi-based KVM over IP.

Firezone is a self-hosted WireGuard®-based VPN server and Linux firewall.

Monoid is an open-source suite of tools for automating data privacy.

Pinecone is an experimental overlay routing protocol suite which is the foundation of the current P2P Matrix demos. It is designed to provide end-to-end encrypted connectivity between devices at a global scale over any compatible medium (currently TCP, WebSockets, Bluetooth Low Energy etc), allowing multi-hop peer-to-peer connectivity between devices even in places where there is no Internet connectivity.

Service Discovery

Back to The Top

Consul is a tool for service discovery, monitoring and configuration. Install Consul on Self-Hosted Kubernetes Clusters.

Linkerd is an ultralight, security-first service mesh for Kubernetes. Linkerd adds critical security, observability, and reliability features to your Kubernetes stack with no code change required.

Doozerd is a highly-available, completely consistent store for small amounts of extremely important data.

Admiral is a tool for for service discovery that provides automatic configuration and service discovery for multicluster Istio service mesh.

ScaleCube is a library that simplifies the development of reactive and distributed applications by providing an embeddable microservices library. It connects distributed microservices in a way that resembles a fabric when viewed collectively. It greatly simplifies and streamlines asynchronous programming and provides a tool-set for managing microservices architecture.

DPS(dns-proxy-server) is a lightweight end user (Developers, Server Administrators) DNS server tool for service discovery, which make it easy to develop in systems where one hostname can solve to different IPs based on the configured environment, so you can:

Solve hostnames from local configuration database.
Solve hostnames from docker containers using docker hostname option or HOSTNAMES env.
Solve hostnames from a list of configured remote DNS servers(as a proxy) if no answer of two above .
Graphic interface to Create/List/Update/Delete A/CNAME records.
Solve host machine IP using host.docker hostname.
Access container by its container name / service name.
Specify from which network solve container IP.

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.

Security

Back to The Top

Blackbox - Safely store secrets in Git/Mercurial. Provides tooling to automatically encrypt secrets like passwords.

CrowdSec - Locally scans log files and optionnaly requests, detecting and blocking malicious behaviors. AppSec capabilities to enable virtual-patching and turn your install into a WAF. Share attacks signals and benefit from real time blocklist of the most agressive IPs attacking CrowdSec's network.

Denyhosts - Thwart SSH dictionary based attacks and brute force attacks.

Fail2Ban - Scans log files and takes action on IPs that show malicious behavior.

fwknop - Protects ports via Single Packet Authorization in your firewall.

Glastopf - A low-interaction web application honeypot to emulate vulnerabilities and gather attack data.

Kippo - A medium-interaction SSH honeypot, mostly used as a standalone SSH daemon with a configurable Filesystem sandbox. OSSEC - OSSEC is a HIDS that performs log analysis, FIM, rootkit detection, and much more.

OSQuery - Query your servers status and info using a SQL like interface.

OPNsense is an open source firewall and routing software developed by Deciso. It offers an integrated Netflow analyser without the need for additional plugins or tools, similar to what you may find in high-end commercial products.

pfSense - Firewall and Router FreeBSD distribution.

Snort - Snort is a free and open source network intrusion prevention system (NIPS) and network intrusion detection system (NIDS) created by Martin Roesch in 1998.

SpamAssassin - A powerful and popular email spam filter employing a variety of detection technique.

BounCA - is a personal SSL / Certificate Authority Key management tool. Create self-signed SSL certificates via your browser.

Troubleshooting

Back to The Top

NETworkManager - A powerful tool for managing networks and troubleshoot network problems. It contains features like a WiFi analyzer, IP scanner, port scanner, ping monitor, traceroute, DNS lookup or a LLDP/CDP capture.

Wireshark - The world's foremost network protocol analyzer.

Selfspy is a daemon for Unix/X11, MacOS (thanks to @ljos) and Windows (thanks to @Foxboron), that continuously monitors and stores what you are doing on your computer. This way, you can get all sorts of nifty statistics and reminders on what you have been up to.

Cilium - A networking, observability, and security solution with an eBPF-based dataplane. It provides a simple flat Layer 3 network with the ability to span multiple clusters in either a native routing or overlay mode.

Netshoot - A Docker + Kubernetes network trouble-shooting swiss-army container.

Kubevious - A suite of app-centric assurance, validation, and introspection products for Kubernetes. It helps running modern Kubernetes applications without disasters and costly outages by continuously validating application manifests, cluster state, and configuration.

HOMER - A robust, carrier-grade, scalable Packet and Event capture system and VoiP/RTC Monitoring Application based on the HEP/EEP protocol and ready to process & store insane amounts of signaling, rtc events, logs and statistics with instant search, end-to-end analysis and drill-down capabilities.

mitmproxy - A Python tool used for intercepting, viewing and modifying network traffic. Invaluable in troubleshooting certain problems.

Sysdig - Capture system state and activity from a running Linux instance, then save, filter and analyze.

Sysdig Inspect - A powerful opensource interface for container troubleshooting and security investigation.

Monitoring

Back to the Top

Proxmox Mail Gateway is an open-source email security solution protecting your mail server against all email threats from the moment they emerge.

M2MLabs MainSpring is an application framework for building machine-to-machine applications like vehicle tracking or machine remote monitoring. In such applications typically a remote device equipped with sensors (e.g. gps, temperature, pressure) and actors communicates with a server application that is running the device communication protocol, device configuration, storage of data sent by the devices as well as the application business logic and the presentation layer.

VictoriaMetrics is a fast and scalable open source time series database and monitoring solution which exists in a Single and in a cluster version. It is compatible with Prometheus pull model and supports a wide variety of ingestion protocols: Influx, Graphite, Prometheus remote_write, Prometheus exposion format, OpenTSDB put message, JSON line format, Arbitrary CSV data, native binary formant, DataDog agent or DogStatsD; as way as many ways to query data via PromQL or MetricsQL from Grafana or own VMUI.

Kestra is an infinitely scalable orchestration and scheduling platform, creating, running, scheduling, and monitoring millions of complex pipelines.

InfluxDB is an open source time series database, purpose-built by InfluxData for monitoring metrics and events, provides real-time visibility into stacks, sensors, and systems. Use InfluxDB to capture, analyze, and store millions of points per second, meet demanding SLA's, and chart a path to automation.

Grafana is a tool that allows you to query, visualize, alert on and understand your metrics no matter where they are stored.

Prometheus is a free software application used for event monitoring and alerting. It records real-time metrics in a time series database (allowing for high dimensionality) built using a HTTP pull model, with flexible queries and real-time alerting.

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream.

Thanos is a set of components that can be composed into a highly available metric system with unlimited storage capacity, which can be added seamlessly on top of existing Prometheus deployments.

Wyze is a great security and monitoring application to live stream HD video from the security cameras from anywhere in the world.

Uptime Kuma is a fancy self-hosted monitoring tool.

Gatus is a developer-oriented health dashboard that gives you the ability to monitor your services using HTTP, ICMP, TCP, and even DNS queries as well as evaluate the result of said queries by using a list of conditions on values like the status code, the response time, the certificate expiration, the body and many others.

Upptime is the open-source uptime monitor and status page, powered entirely by GitHub Actions, Issues, and Pages.

HertzBeat is an open-source, real-time monitoring system with custom-monitor and agentless. It supports web service, database, os, middleware and more.

Tautulli is a python based web application for monitoring, analytics and notifications for Plex Media Server.

Flower is a web based tool for monitoring and administrating Celery clusters.

Weave Scope is a tool for Troubleshooting & Monitoring for Docker & Kubernetes. It automatically generates a map of your application, enabling you to intuitively understand, monitor, and control your containerized, microservices-based application.

Statping (Status Page & Monitoring Server) is an easy to use Status Page for your websites and applications. Statping will automatically fetch the application and render a beautiful status page with tons of features for you to build an even better status page.

Vector is a high-performance, end-to-end (agent & aggregator) observability data pipeline that puts you in control of your observability data. Collect, transform, and route all your logs, metrics, and traces to any vendors you want today and any other vendors you may want tomorrow.

Open Service Mesh (OSM) is a lightweight, extensible, cloud native service mesh that allows users to uniformly manage, secure, and get out-of-the-box observability features for highly dynamic microservice environments.

Ciao is a tool that checks HTTP(S) URL endpoints for a HTTP status code (or errors on the lower TCP stack) and sends a notification on status change via E-Mail or Webhooks.

Server is a simple server for sending and receiving messages in real-time per WebSocket.

Ngxtop is a real-time metrics for nginx server (and others).

Blocky is a fast and lightweight DNS proxy as ad-blocker for local network with many features

Dashy is a self-hostable personal dashboard built for you. Includes status-checking, widgets, themes, icon packs, a UI editor and tons more.

Restic is a modern backup program that can back up your files: from Linux, BSD, Mac and Windows. To many different storage types, including self-hosted and online services. easily, being a single executable that you can run without a server or complex setup. effectively, only transferring the parts that actually changed in the files you back up.

Autorestic is a wrapper around the amazing restic. While being amazing the restic cli can be a bit overwhelming and difficult to manage if you have many different locations that you want to backup to multiple locations.

MinIO is a high performance object storage tool that provides the world's fastest object storage server. With READ/WRITE speeds of 325 GiB/s and 165 GiB/s on standard hardware with default parity (EC:4), object storage can operate as the primary storage tier for a diverse set of workloads ranging from Spark, Presto, TensorFlow.

Greyhole is a tool that uses Samba to create a storage pool of all your available hard drives (whatever their size, however they are connected), and allows you to create redundant copies of the files you store, in order to prevent data loss when part of your hardware fails.

Falcon LogScale is purpose-built to help any organization achieve the benefits of large-scale logging and analysis. Falcon LogScale has virtually no latency, even at ingest volumes of 1PB(Petabyte) per day.

Googerteller is a tool that makes an audible sound any time your computer sends a packet to a Google tracker or a Google service, which excludes Google Cloud users.

TeslaMate is a powerful, self-hosted data logger for your Tesla.

OneUptime is an open-source complete SRE and DevOps platform. It monitors your website, dashboards, API's, and more and alerts your team when downtime happens.

Parca is a tool for continuous profiling for analysis of CPU and memory usage, down to the line number and throughout time. Saving infrastructure cost, improving performance, and increasing reliability.

DeviceHive is a free, highly scalable open-source IoT platform for data collection, processing and analysis, visualization, and device management with the broad range of integration options.

Distributed Services Architecture (DSA) is an open source IoT platform that facilitates device inter-communication, logic and applications at every layer of the Internet of Things infrastructure. The objective is to unify the disparate devices, services and applications into a structured and adaptable real-time data model.

IoTivity is an open source software framework enabling seamless device-to-device connectivity to address the emerging needs of the Internet of Things.

Eclipse IoT Project provides open source technology that will be used to build IoT solutions for industry and consumers.

Dashboards

Back to The Top

Adagios is a Web based Nagios configuration interface.

Dash is a low-overhead monitoring web dashboard for a GNU/Linux machine.

Thruk is a Multibackend monitoring web interface with support for Naemon, Nagios, Icinga and Shinken.

Uchiwa is a simple dashboard for the Sensu monitoring framework.

Grafana is a tool that allows you to query, visualize, alert on and understand your metrics no matter where they are stored.

Analytics

Back to the Top

Plausible Analytics - Simple, open-source, lightweight (< 1 KB) and privacy-friendly web analytics.

PostHog - Product analytics, session recording, feature flagging and a/b testing that you can self-host.

Ackee - Self-hosted analytics tool for those who care about privacy.

AWStats - Generate statistics from web, streaming, ftp or mail server logfiles.

Chartbrew - Web application that can connect directly to databases and APIs and use the data to create beautiful charts.

Countly Community Edition - Real time mobile and web analytics, crash reporting and push notifications platform.

Druid - Distributed, column-oriented, real-time analytics data store.

EDA - Web application for data analysis and visualization.

GoAccess - Real-time web log analyzer and interactive viewer that runs in a terminal.

GoatCounter - Easy web statistics without tracking of personal data.

Metabase - Easy, open-source way for everyone in your company to ask questions and learn from data.

Offen - Fair, lightweight and open web analytics tool. Gain insights while your users have full access to their data.

Open Web Analytics - Web analytics framework that lets you stay in control of how you instrument and analyze the use of your websites and applications.

Redash - Connect and query your data sources, build dashboards to visualize data and share them with your company.

RudderStack - Collect, unify, transform, and store your customer data, and route it to a wide range of common, popular marketing, sales, and product tools. alternative to Segment.

Shynet - Modern, privacy-friendly, and detailed web analytics that works without cookies or JS.

Superset - Modern data exploration and visualization platform.

Umami - Simple, fast, privacy-focused alternative to Google Analytics.

Search

Back to the Top

Meilisearch is a lightning-fast search engine that fits effortlessly into your apps, websites, and workflow.

Shodan is the world's first search engine for Internet-connected (IoT) devices.

Whoogle Search is a self-hosted, ad-free, privacy-respecting metasearch engine.

SearX is a Privacy-respecting, hackable metasearch engine.

SearXNG is a free internet metasearch engine which aggregates results from various search services and databases.

Sonic is a fast, lightweight & schema-less search backend. An alternative to Elasticsearch that runs on a few MBs of RAM.

Zinc is a search engine that does full text indexing. It is a lightweight alternative to Elasticsearch and runs using a fraction of the resources.

Cylect.io is the ultimate searching tool that is here to assist anyone looking for specific information through vast amounts of websites, search engines, and data collectors.

Lyra is a fast, in-memory, typo-tolerant, full-text search engine written in TypeScript.

Hugo Lyra is a typescript module for creating LyraSearch indexes for static Hugo sites, it comes with server and client libraries.

Typesense is a fast, typo-tolerant search engine for building delightful search experiences.

Tantivy is a full-text search engine library inspired by Apache Lucene and written in Rust.

Toshi is meant to be a full-text search engine similar to Elasticsearch. Toshi strives to be to Elasticsearch what Tantivy is to Lucene.

FlexSearch is a Next-Generation full text search library for Browser and Node.js.

fd is a program to find entries in your filesystem. It is a simple, fast and user-friendly alternative to find.

k8s at home search is a tool that indexs Flux HelmReleases from Github repositories with the k8s-at-home topic on GitHub.

OpenFind is an app to find text in real life. Easily search your entire photo library in split seconds. This runs 100% offline. No servers, nothing weird going on.

Notifications

Back to the Top

Apprise is a tool that allows you to send a notification to almost all of the most popular notification services available to us today such as: Telegram, Discord, Slack, Amazon SNS, Gotify, etc.

ntfy is a simple HTTP-based pub-sub notification service. It allows you to send notifications to your phone or desktop via scripts from any computer, entirely without signup, cost or setup. It's also open source if you want to run your own.

Countly is a product analytics solution and innovation enabler that helps teams track product performance and customer journey and behavior across mobile, web, and desktop applications. Ensuring privacy by design, Countly allows you to innovate and enhance your products to provide personalized and customized customer experiences, and meet key business and revenue goals.

notifiers is a general wrapper for a variety of 3rd party providers and built in ones (like SMTP) aimed solely at sending notifications.

Pushover is a tool that makes it easy to get real-time notifications on your Android, Android Wear, iPhone, iPad, Apple Watch and Desktop.

Simplepush is a tool to send end-to-end encrypted push notifications to your Android and iPhone.

UnifiedPush is a set of specifications and tools that lets the user choose how push notifications are delivered. All in a free and open source way.

RSS

Back to the Top

RSS Guard is a simple RSS/ATOM feed reader for Windows, Linux, BSD, OS/2 or macOS which can work with RSS/ATOM/JSON feeds as well as many online feed services:

Feedly
Gmail
Google Reader API (Bazqux, FreshRSS, Inoreader, Miniflux, Reedah, The Old Reader and more)
Nextcloud News
Tiny Tiny RSS

Feedly is an RSS tool where you can privately organize and research the topics and trends that matter to you. It offers useful integrations with Facebook, Twitter, Evernote, Buffer, OneNote, Pinterest, LinkedIn, IFTTT, and Zapier so that you can easily share stories with your networks and teammates.

FreshRSS is a self-hosted RSS and Atom feed aggregator. It is lightweight, easy to work with, powerful, and customizable.

ArchiveBox is a powerful, self-hosted internet archiving solution to collect, save, and view sites you want to preserve offline. It takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more.

RSSHub is an open source, easy to use, and extensible RSS feed generator. It's capable of generating RSS feeds from pretty much everything.

Miniflux V2 is a minimalist and opinionated feed reader.

Websites/Blogs

Back to the Top

Lyra is a fast, in-memory, typo-tolerant, full-text search engine written in TypeScript.

Hugo Lyra is a typescript module for creating LyraSearch indexes for static Hugo sites, it comes with server and client libraries.

Kopage is a self-hosted Website Builder. It's compatible with cPanel and other popular hosting control panels. Compatible with cPanel and other popular hosting control panels.

Ghost is a fully-managed PaaS & self-hosted open source software, and can be installed and maintained relatively easily on just about any VPS hosting provider.

Cloudron is a self-hosted immutable infrastructure design allows easy migration of apps across servers. In fact, you can move your entire server along with all its apps to another cloud provider in no time.

Directus is a real-time API and App dashboard for managing SQL database content.

Haven is a Self-hosted private blog instead of using Facebook.

Antville is an open source project aimed at the development of a simple site hosting system with many advanced features.

October is a Self-hosted Content Management System (CMS) and web platform whose sole purpose is to make your development workflow simple again.

Grav is a Fast, Simple, and Flexible, file-based Web-platform. There is Zero installation required. Just extract the ZIP archive, and you are already up and running. It comes with a powerful Package Management System to allow for simple installation and upgrading of plugins and themes, as well as simple updating of Grav itself.

Orchard is a free, open source, community-focused Content Management System built on the ASP.NET MVC platform.

Netlify CMS is a CMS for static site generators. Give users a simple way to edit and add content to any site built with a static site generator.

Zola is a fast static site generator in a single binary with everything built-in.

FlatPress is a lightweight, easy-to-set-up blogging engine.

Chyrp Lite is an ultra-lightweight blogging engine. It provides four beautiful blog themes and a friendly administration console, all fully navigable on a broad range of devices, thanks to the power of responsive HTML5.

WriteFreely is an open source platform for building a writing space on the web.

Sandstorm is an open source project built by a community of volunteers with the goal of making it really easy to run open source web applications.

YunoHost is a Debian-based distribution which strives to make it easy to quickly set up a server and host web applications.

Social

Back to the Top

Mattermost is a secure, open source platform for communication, collaboration, and workflow orchestration across tools and teams.

Mastadon is a a decentralized social media platform that supports audio, video and picture posts, accessibility descriptions, polls, content warnings, animated avatars, custom emojis, thumbnail crop control, and more, to help you express yourself online.

Telegram is a cross-platform, cloud-based instant messaging service. It has an open API and source code free for everyone. Telegram also provides end-to-end encrypted video calling, VoIP, file sharing and several other features.

ActivityPub is a decentralized social networking protocol based on the ActivityStreams 2.0 data format. It provides a client to server API for creating, updating and deleting content, as well as a federated server to server API for delivering notifications and subscribing to content.

Lemmy is similar to sites like Reddit, Lobste.rs, or Hacker News. Where you subscribe to forums you're interested in, post links and discussions, then vote, and comment on them. Behind the scenes, it is very different; anyone can easily run a server, and all these servers are federated, and connected to the same universe, called the Fediverse.

Lemmy-UI is the official web app for Lemmy, written in inferno.

Mlem is a Lemmy client for iOS.

Jerboa is an Android client for Lemmy, a federated reddit alternative.

GoToSocial is an ActivityPub social network server, written in Golang.

Berty is a secure peer-to-peer messaging app that works with or without internet access, cellular data or trust in the network.

Pleroma is a free and open communication for everyone. Pleroma is social networking software compatible with other Fediverse software such as Misskey, Pixelfed, Mastodon and many others.

Matrix is a tool that gives you simple HTTP APIs and SDKs (iOS, Android, Web) to create chatrooms, direct chats and chat bots, complete with end-to-end encryption, file transfer, synchronised conversation history, formatted messages, read receipts and more.

Element is a Matrix web client built using the Matrix React SDK.

Nostr(Notes and Other Stuff Transmitted by Relays) is a truly censorship-resistant alternative to Twitter that has a chance of working.

Fritter is an open source frontend for Twitter on mobile devices, focusing on giving you the best experience and keeping your data private, local and in your hands.

Nitter is a free and open source alternative Twitter front-end focused on privacy and performance. All requests go through the backend, meaning the client never talks to Twitter and prevents Twitter from tracking your IP or JavaScript fingerprint.

Diaspora is a privacy-aware, distributed, open source social network.

Hubzilla is a general purpose communication server integrated with a web publishing system and a decentralised permission system.

Expanse is a fully selfhosted multi-user web app for externally storing Reddit items (saved, created, upvoted, downvoted, hidden) to bypass Reddit's 1000-item listing limits.

Apollo is a beautiful Reddit app built for fast navigation with an incredibly powerful set of features.

Infinity is a Reddit client on Android written in Java. It does not have any ads and it features a clean UI and smooth browsing experience.

RedReader is an unofficial open source Reddit client for Android.

Nostr

Back to the Top

Nostr (Notes and Other Stuff Transmitted by Relays) is a protocol, designed for simplicity, that aims to create a censorship-resistant global social network. The protocol is based on very simple & flexible event objects (which are passed around as plain JSON) and uses standard elliptic-curve cryptography for keys and signing.

nostr - overview and FAQ.
NIPs - the "Nostr Implementation Possibilities" describe the protocol in technical detail.
nostr, a basic tour - an intro to nostr.
UseNostr - A small guide for anyone who wants to learn more about how nostr works and what it can do.
nostr.how - Quick-start to onboard desktop users with Alby & Astral.
nostr.guide - A guide to all things nostr.
nostr address book - A directory of twitter users accounts and their NOSTR addresses.
NNostr - a C# relay.
nostr-rs-relay - a minimalistic relay written in Rust that saves data on SQLite.
Relayer Basic - a simple relay based on relayer backed by Postgres.
nodestr - a Node.js implementation.
sovereign-stack - a tool that helps you deploy nostr relays and create self-hosted (bitcoin-only) Value4Value websites.
expensive relay - a relay that requires payment for registration
me.untethr.nostr-relay - a relay written in Clojure
Minds Nostr Relay - a relay for Minds, an open-source social network
- Minds Engine - Nostr - relevant Minds API code for reading/writing Minds posts using Nostr
NostrPostr Relay - a Kotlin Relay supporting both SQLite and Postgresql
nostrpy - relay, client, and other tooling in python
nostream - a nostr relay written in Typescript backed by PostgreSQL (renamed from nostr-ts-relay)
nostr_relay – a nostr relay written in python, backed by SQLite
søstr – a private nostr relay written in rust, saves all notes from one pubkey and publish them to anyone that requests them
knostr – a nostr relay implemented in Kotlin with support for Postgres and metrics(micrometer).
PyRelay – a python implementation of a nostr relay, using asyncio.
strfry – C++ implementation backed by LMDB with efficient syncing of events using merkle trees
Astro – Elixir based implementation built to be performant and highly distributed.
Nex - A powerful and scalable Nostr relay written in Elixir with Postgres DB.
gnost-relay - nostr relay written in go backed by postgresql database.
nostring - A Nostr relay written in Deno.
Denostr - Deno based, cloud native nostr implemention support by ByteTrade and Revo.
nostr-relay-nestjs- A Nostr relay implemented using the NestJS framework
Ephemerelay - An in-memory Nostr relay that doesn't store data.
Servus - A self-contained, single executable, CMS / blogging engine reminiscent of Jekyll which also acts as a personal Nostr relay for your blog posts. Written in Rust.
nostr relay registry - real-time checking of status of some known relays.
nostr.info - real-time checking of status of some known relays.
nostr.watch - real-time checking of status of some known relays.
Astral - a branle fork with global feed and UI makeover
damus - a twitter-like nostr client for iOS and MacOS.
more-speech - desktop client for nostr written in Clojure.
futr - nostr client desktop app written in Haskell.
Minds - open source social network. Supports reading and creating posts using the Nostr protocol.
Jester - Chess over nostr.
- Jester instance
Sendstr - shared clipboard between devices over nostr.
nosbin - pastebin over nostr.
noscl - a basic command-line client written in Go.
loquaz - a desktop app written in Rust for direct encrypted chat.
nostr console - a nostr command line client written in Dart. Binaries available for Windows, Linux, and MacOS.
ArcadeCity - Public group chats and P2P services (WIP) over nostr.
second exchange - an experiment to work out something of like medium, something of creator economy where users are rewarded for engaging in quality discussion and most importantly engaging in governance-related discussion.
scalastr - A barebones nostr client written in scala.
Nostros - A nostr mobile client for Android.
Nostrify.me - Nostr client built in SvelteKit.
NostrEmitter - Simple E2E encrypted client and EventEmitter object
Lightning.Pub - A nostr daemon for Lightning nodes.
shockwallet - A Lightning wallet that uses nostr and lnurl to connect to nodes.
coracle - A nostr web client.
nostrweb - another nostr web client in vanilla JS.
- nostr.ch - live instance
Bija - A desktop client written in python. Currently Linux only.
Nosky - A native Android client for Nostr. Still in development.
Stackerstan - A decentralised organisation built on Bitcoin and Nostr, implemented as a replicated state machine in Golang.
nostr-java - A nostr client API written in java, for generating, signing and publishing events to relays.
bolt.fun - A bitcoin lightning makers community that supports reading and creating comments using Nostr.
iris - A nostr web client.
- iris.to - live instance
- Android app
gossip - A desktop client in rust presented with egui.
Attached - Open-Source ReactNative Expo app for Nostr (iOS, Android). Currently under app stores review.
Member - Progressive Web App Client. Works on desktop and mobile.
- member.cash - live instance
dispute - A cross-platform (Linux, Android, iOS, MacOs, Windows and Web) client for NOSTR
Snort - Nostr UI written in react
Hamstr - A twitter-style web client built with Vue.js
Nozzle - A Twitter-like native Android client written with Jetpack Compose
electron-nostr - A bare-bones desktop nostr client using electron-react-boilerplate. Goal is to be an easy template for people to experiment with different ideas on decentralized ratings, reputation, and web of trust.
Nostrid - Multi-platform client currently offering binaries for Android, Windows, MacOS and Linux.
- Nostrid.Web - Web version running completely on the browser. It can be installed locally as PWA.
nostr-chat-widget-react - A React component that provides a live-chat widget over nostr that can be embedded into any website.
Blockcore Notes - Progressive Web App that can be installed on mobile and desktop, organize following in circles and have both public and private following lists. Dynamic interface for different uses, such as optimized for photograph viewing.
Noteon- Yet another nostr client focused on private chat and group chat with a simple, elegant and secure interface.
emon - Encrypted DMs over nostr with lightning payments integrated (WIP).
notebin.org - Nostr UI created with NextJS, support for markdown and code highlighting.
Daisy - Mobile client for Android and iOS.
Flycat - A 2000s old-school style web client which support blogging on Nostr.
Amethyst - An Android client for nostr written in Kotlin.
MeShell - Web, iOS and Android blog type client destined to publish articles and researches for independent journalists.
Disgus - A comment widget like Disqus, but for Nostr.
Tamga - An offline first nostr contact & profile manager for iOS!
nostromat- A Twitter-style Nostr web client, written in Clojurescript/React.
nostrom.at - live instance.
blogstack.io - Blogging site for nostr, supports markdown.
Votestr - Poll web app with nostr authentication and blind signature unlinkability.
gnost-deflate-client- A CLI nostr client written in go implementing permessage-deflate websocket compression.
algia - A cli application for nostr.
algia-web - A small resource consumption oriented Nostr web client.
Blowater - A desktop Web client focusing on chat with delightful UX.
Written Self hosted blog using nostr long-form content (NIP-23) and it shows only posts by selected authors.
Nostr Nests - Nostr Nests is an audio space for chatting, brainstorming, debating, jamming, micro-conferences and more.
nblog - a self-host nostr ghost blog
Nostribe.com - Nostr client web app built with Next.js 13 and TypeScript.
- Nostribe.com - Live instance.
Nostrtium - Post to Nostr directly from within WordPress
uBlog - A minimalist blog on nostr that allows anyone to easily create their own personal micro-blog.
Listr - A Nostr Web Client for Making Lists, built with Next.js 13 and TypeScript.
nostr-ruby - a Ruby implementation of the nostr protocol.
nostr - a Ruby Nostr gem for use by clients.
NNostr.Client - a C# Nostr library for use by clients.
nostr-tools - a JavaScript client that abstracts the relay management code for use by clients.
nostr-relaypool-ts - a TypeScript relay pool library on top of nostr-tools that simplifies handling subscriptions to multiple servers.
nostr-react - React Hooks for Nostr.
go-nostr - a Go library that implements relay management, plus event encoding and signing utils.
nostr_rust - Functional Rust implementation of the nostr protocol.
nostr-js - a javascript implementation of the nostr protocol.
nostr-rs - a Rust implementation of the nostr protocol
nostr
- nostr: Rust implementation of Nostr protocol.
- nostr-sdk: High level client library.
- bindings: UniFFI (Kotlin, Swift, Python, Ruby) bindings.
relayer - a server framework for writing custom relays.
NostrPostr - a Kotlin Nostr library for clients or relays.
python-nostr - a python library for making clients.
nostr-bot - a Rust library for writing bots.
NostrKit - a Swift library for interacting with relays.
nostr-relay-inspector - A library that returns useful information about relays based on nostr-js.
schorr_snap - A snap plugin for Metamask Flask, supports nostr.
nostr-deno - a client library for Deno javascript runtime.
nostr-types - a rust library defining types useful for the nostr protocol.
dart-nostr - a Dart library for Flutter.
nostr-connect - Nostr Connect SDK for TypeScript is a library that allows you to easily integrate Nostr Connect into your web application.
pynostr - a python library for nostr.
nostr-php - a PHP library for nostr.
smtp nostr gateway - a bridge that forwards emails to pubkeys as encrypted direct messages.
matrix-nostr-bridge - a simple Matrix-to-Nostr or Nostr-to-Matrix bridge.
Mostr - a bridge between Nostr and the Fediverse (Mastodon, ActivityPub, etc.).
nostrich.fun - A feature-rich directory of nostr projects. A fork of LightningNetworkStores.com
git-nostr-tools - A cli tool for sending code patches over nostr.
nostr-cln-events - A CLN plugin to push clightning node events to nostr.
nostr registry - a database of known relays with their uptime and NIP support tables
nostr-fzf - Nostr Directory; a tool for searching usernames and channels.
nostr-notify - desktop nostr notifications using libnotify.
nostr-launch - A tool for launching a bunch of relays and clients locally for development and testing.
nostr GitHub Action - send events from GitHub Actions.
nostrefresh - A simple refresh function for nostr web pages.
anonroom - anonymous chat room inside nostr.
nostril - A C cli tool for creating nostr events.
nostr-rs-relay-compose - A Docker compose deployment for nostr-rs-relay with SSL support based on Traefik.
nostr.guru - a nostr web gateway for viewing events by their ID.
nostrandom.netlify.app - generate publish-able Nostr event with random keys.
nashboard - A Nostr network dashboard with network statistics, reachable here.
ndxstr - nostr's layer 2 indexing nodes, with more advanced querying capability than currently supported by relays.
nostrillery - A tool for running performance tests against Nostr relays.
nostr-terminal - A SSH-like access to your machine via web terminal, powered by Nostr.
nostreq - A Nostr relay event request generator.
nostr.io - A network statistics with last published notes, top 50 publishers, and top 50 followed users.
nostr-commander - A simple but convenient CLI-based Nostr app for following users, sending DMs, etc.
nostr.directory - A searchable database of nostr users and their other social media links.
nostr-tool - A Rust CLI tool to generate and publish events.
frostr - Create joint nostr identities and require t-of-n signatures to post.
nostr.rest - Mine proof of work public keys with user specified prefixes.
lnpass - A key manager for Lightning and nostr.
sb.nostr.band - Search bots that you can create and follow to receive new posts matching a keyword right into your feed.
rss.nostr.band - Create custom RSS feeds with posts matching your keywords and consume using your favorite RSS reader.
nostrview - A nostr search engine. Search by content, tags, events or pub keys.
nostr-bulk-dms - A tool that allows you to send DMs over nostr to many recipients in bulk.
nostrify - A Core Lightning plugin that sends events (forwards, connect, disconnect, etc.) to nostr.
nip06-web - a website to generate or restore NIP-06 seed phrases
nip06-cli - a Node.js CLI to generate or restore NIP-06 seed phrases.
nostr-broadcast - A tool lets you take your events from some relays and broadcast them to another relay. Could be helpful for backing up your notes to a private relay.
nostr-follow-bundler - A tool lets you create lists of profiles that other users can then see and follow themselves.
nostr-proxy - Push and get events to your Proxy, get results from multiple Nostr relays.
nostrends - Trending on Nostr, like Twitter trends. Live at nostrends.vercel.app.
homebrew-nostr - Homebrew tap for Nostr software.
heyxynip5 - A CLI helper for converting nostr npub/nsec to their hex format for NIP-05 verification.
http-nostr-publisher - A Cloudflare worker to publish Nostr events to relays through a non-blocking HTTP interface .
blastr - A nostr cloudflare workers proxy relay that publishes to all known relays.
keystr-rs - An application for managing Nostr keys. Written in Rust, with simple UI (Iced).
nostr_simple_publish - Drupal module to publish content to Nostr.
nostr-spam-detection - An experiment in building a machine learning model to label Nostr spam content for filtering and relay rejection.
blogsync - Self-host blog articles from long-form notes e.g. via Caddy server.
NostrFlu - A tool to collect and resend following lists from relays. You can also check badges.
nkcli - A CLI tool for nostr key manage and serve NIP-46.
git-nostr- A tool to enhance git cli with nostr communications.
nostr-wtf - A set of nostr tools available and deployed on a web app including a pubkey converter and relay query tool.
strfry policies- A collection of moderation & antispam policies for the strfry relay developed in TypeScript/Deno.
nostrum - Nostrum it's a mobile app that allows you to sign transactions and messages with your Nostr keys. Nostrum is the reference implementation for a remote signer app (ie. Wallet) of the Nostr Connect protocol.
nostr-signing-device - Signing device for Nostr built on ESP32.

iMessage

Back to the Top

iMessage-exporter is a binary exports iMessage data to txt or html formats. It can also run diagnostics to find problems with the iMessage database.

pypush is a POC demo of my recent iMessage reverse-engineering. It can currently register as a new device on an Apple ID, set up encryption keys, and send and receive iMessages!

Self-Host Beeper is a self-hosted universal chat app that can chat with your friends on iMessage from your Android device using your phone number. You can also join iMessage Group Chats with your phone number and blue bubbles, and share full-resolution images, videos, and audio.

Beeper Mini is an Andorid app that can chat with your friends on iMessage from your Android device using your phone number. You can also join iMessage Group Chats with your phone number and blue bubbles, and share full-resolution images, videos, and audio.

Beeper Bridge Manager is a tool for running self-hosted bridges with the Beeper Matrix server.

Matrix Ansible and Docker Deploy is a Matrix (An open network for secure, decentralized communication) server setup using Ansible and Docker.

Communications

Back to the Top

Postmoogle is an actual SMTP server that allows you to send and receive emails on your matrix server. It can't be used with arbitrary email providers, because it acts as an actual email provider itself, so you can use it to send emails from your apps and scripts as well.

SimpleX is a privacy redefined messenger without user IDs. Other apps have user IDs: Signal, Matrix, Session, Briar, Jami, Cwtch, etc. SimpleX does not, not even random numbers.

Element is a Matrix web client built using the Matrix React SDK.

Mattermost is a secure, open source platform for communication, collaboration, and workflow orchestration across tools and teams.

Berty is a secure peer-to-peer messaging app that works with or without internet access, cellular data or trust in the network.

Pleroma is a free and open communication for everyone. Pleroma is social networking software compatible with other Fediverse software such as Misskey, Pixelfed, Mastodon and many others.

ffsend is a easily and securely share files from the command line. A fully featured Firefox Send client.

Nostr(Notes and Other Stuff Transmitted by Relays) is a truly censorship-resistant alternative to Twitter that has a chance of working.

Diaspora is a privacy-aware, distributed, open source social network.

Hubzilla is a general purpose communication server integrated with a web publishing system and a decentralised permission system.

Expanse is a fully selfhosted multi-user web app for externally storing Reddit items (saved, created, upvoted, downvoted, hidden) to bypass Reddit's 1000-item listing limits.

giscus is a comments system powered by GitHub Discussions. Let visitors leave comments and reactions on your website via GitHub.

Mailroute is a great tool that provides the best email filtering & security( CMMC, NIST 800-171, DFARS, DISA, HIPPA). It protects your inbox, stop spam, viruses, ransomware, security threats & more with email filtering services. With an easy setup on Office 365, Google & more.

Docker Mailserver is a production-ready fullstack but simple mail server (SMTP, IMAP, LDAP, Antispam, Antivirus, etc.) running inside a container. Only configuration files, no SQL database.

Diun is a CLI application written in Go and delivered as a single executable (and a Docker image) to receive notifications when a Docker image is updated on a Docker registry.

iRedMail is a self-hosted email server.

iRedMail Easy is a web-based deployment platform, it offers an easy to use web interface to help you deploy iRedMail server, keep your server up to date, also get fast and professional technical support from iRedMail team.

Spider Email Archiver is an On-Premises Email Archiving Software.

MailCow is a self-hosted email server.

Nextcloud Talk is a on-premises, private audio/video conferencing and text chat through browser and mobile interfaces with integrated screen sharing and SIP integration.

Poste.io Email Server is self-hosted SMTP + IMAP + POP3 + Antispam + Antivirus Web administration + Web email. It is easy setup with a DNS guide for protect from spam.

Business Management

Back to the Top

Nextcloud is a suite of enterprise client-server software for creating and using file hosting services. It offers an on-premise Universal File Access and sync platform with powerful collaboration capabilities and desktop, mobile and web interfaces.

Odoo is a suite of open source business apps that cover all your company needs: CRM, eCommerce, accounting, inventory, point of sale, project management, etc.

Kanboard is project management software that focuses on the Kanban methodology.

Eden Workplace is a complete workplace management platform that lets you achieve more. Desk Booking Software to make desk reservations easier for your team, including assigning permanent and hybrid desks, providing wayfinding solutions for employees.

Matomo is an ethical alternative where you won't make privacy sacrifices or compromise your site. Matomo is the Google Analytics alternative that protects your data and your customer's privacy.

Plausible Analytics is a simple, lightweight (< 1 KB), open-source and privacy-friendly alternative to Google Analytics. It doesn’t use cookies and is fully compliant with GDPR, CCPA and PECR. You can self-host Plausible or have us run it for you in the Cloud.

InvoicePlane is a self-hosted open source application for managing your quotes, invoices, clients and payments.

Collaboration & Synchronization

Back to the Top

Syncthing is a continuous file synchronization program. It synchronizes files between two or more computers in real time.

Synology is a tool that allows you to easily access and manage files in your Synology Drive on the go. Apart from common file types, such as documents, images, videos and music, you can also open Synology Office document, spreadsheets and slides in the user-friendly viewer provided by Drive.

Nextcloud is a suite of client-server software for creating and using file hosting services. It offers an on-premise Universal File Access and sync platform with powerful collaboration capabilities and desktop, mobile and web interfaces.

Lsyncd (Live Syncing Mirror Daemon) is a tool used in Linux systems to keep directories synchronized. These directories can be found locally, within the same machine, or remotely, on different machines. For remote synchronization, this article focuses on using SSH to accomplish it.

FileRun is a self-hosted Google Drive alternative. It is a full featured web based file manager with an easy to use user interface.

FileBrowser provides a file managing interface within a specified directory and it can be used to upload, delete, preview, rename and edit your files. It allows the creation of multiple users and each user can have its own directory.

Rsync is a utility in the command line which enables users to transfer and synchronize files efficiently between a computer and an external hard drive in the entire connected network.

Warpinator is a free, open-source tool for sending and receiving files between computers that are on the same network.

LocalSend is a free and open-source tool that allows you to send files and messages over the local LAN network to nearby devices. Everything is sent securely over HTTPS. The TLS/SSL certificate is generated on the fly on each device. It's avilable on Windows, macOS, Linux, iOS, and Android.

FileZilla Client is a fast and reliable cross-platform FTP, FTPS and SFTP client with lots of useful features and an intuitive graphical user interface.

Dragit is an application for intuitive file sharing between devices. It's useful for when you want to send file from one computer to another with minimal effort. Dragit automatically detects devices in the local network with help of mDNS protocol and allows you to send file immediately.

WinFsp is a set of software components for Windows computers that allows the creation of user mode file systems. In this sense it is similar to FUSE (Filesystem in Userspace), which provides the same functionality on UNIX-like computers.

SSHFS-Win is a minimal port of SSHFS to Windows. Looking under the hood it uses Cygwin for the POSIX environment and WinFsp for the FUSE (Filesystem in Userspace) functionality.

RiftShare is a cross platform (Windows, MacOS, Linux) file sharing tool that supports fully encrypted transfers both on the local network and off network using a simple passphrase. RiftShare uses magic-wormhole under the hood and is compatible with other magic-wormhole clients. It is also fully open source and licensed under the GPLv3.

Usermode FTP Server is a tool that let's you start an FTP server as user and transfer files with any FTP client. Allowing you to access your files directly with many file browsers' builtin FTP support: Windows File Explorer, Thunar, Gnome Files, Dolphin and many more.

TagSpaces is a free, no vendor lock-in, open source application for organizing, annotating and managing local files with the help of tags. It features advanced note taking functionalities and some capabilities of to-do apps. It's available for Windows, Linux, Mac OS and Android.

Listmonk is a standalone, self-hosted, newsletter and mailing list manager. It is fast, feature-rich, and packed into a single binary.

Encryption

Back to the Top

VeraCrypt is free open-source disk encryption software for Windows, Mac OS X and Linux. The file encryption, data encryption performed by VeraCrypt is real-time (on-the-fly), automatic, transparent, needs very little memory, and does not involve temporary unencrypted files.
AxCrypt is an inexpensive and effective encryption tool for Windows, macOS, iOS, and Android.

AESCrypt is an advanced file encryption utility that integrates with the Windows shell or runs from the Linux command prompt to provide a simple, yet powerful, tool for encrypting files using the Advanced Encryption Standard (AES). It is available for Windows, MacOS, and Linux.

Linux Unified Key Setup (LUKS) is a disk encryption specification created by Clemens Fruhwirth in 2004 and was originally intended for Linux. It uses device mapper crypt ( dm-crypt) as a kernel module to handle encryption on the block device level.

GNU Privacy Guard (GnuPG) is a complete and free implementation of the OpenPGP standard as defined by RFC4880 (also known as PGP ). It allows you to encrypt and sign your data and communications; it features a versatile key management system, along with access modules for all kinds of public key directories.

Pretty Good Privacy (PGP) is an encryption program that provides cryptographic privacy and authentication for data communication. It's used for signing, encrypting, and decrypting texts, e-mails, files, directories, and whole disk partitions and to increase the security of e-mail communications.

Deadbolt is a Dead-simple file encryption for any OS.

Infisical is an open-source, end-to-end encrypted platform to sync secrets and configs across your team and infrastructure.

Hemmelig.app is a tool that keeps your sensitive information out of chat logs, emails, and more with encrypted secrets.

How Encryption Keys work

Symmetric is a data encryption method whereby the same private key is used to encode and decode information.
Asymmetric is a data encryption method that allows users to encrypt information using shared keys. For example, if you need to send a message across the internet, but you don't want anyone but the intended recipient to see what you've written.

Types of Encryption

Triple DES (Triple Data Encryption Algorithm) is a symmetric-key block cipher, which applies the DES cipher algorithm three times to each data block(contains 64 bits of data).
AES (Advanced Encryption Standard) is an algorithm that encrypts and decrypts data in blocks of 128 bits. It can do this using 128-bit, 192-bit, or 256-bit keys.
RSA (Rivest–Shamir–Adleman) is a type of public-key cryptography used for secure data transmission of e-mail and other digital transactions over the Internet.
Twofish is a symmetric key block cipher with a block size of 128 bits and key sizes up to 256 bits. It is an advanced version of Blowfish encryption.
Format Preserving Encryption (FPE) is a valid encryption algorithm to be used for compliance with NIST standards. It is mostly used in on-premise encryption and tokenization solutions.

Application Level Encryption

Hashes is a function that converts an input of letters and numbers into an encrypted output of a fixed length. For example, algorithms such as MD5 (Message Digest 5) or SHA (Secure Hash Algorithm).
Digital Certificates is a file that verifies the identity of a device or user and enables encrypted connections. A digital signature is a hashing approach that uses a numeric string to provide authenticity and validate identity. Digital certificates are typically issued by a certificate authority (CA), which is a trusted third-party entity that issues digital certificates for use by other parties.

Backups

Back to the Top

Proxmox Backup Server is an enterprise backup solution for backing up and restoring VMs, containers, and physical hosts. The open-source solution supports incremental backups, deduplication, Zstandard compression, and authenticated encryption.

BackupPC is a high-performance, enterprise-grade system for backing up Linux, Windows and macOS PCs and laptops to a server's disk. BackupPC is highly configurable and easy to install and maintain.

BorgWarehouse is a fast and modern WebUI for a BorgBackup's central repository server.

Emborg is a simple command line utility to orchestrate backups. It is built as a front-end to Borg, a powerful and fast de-duplicating backup program.

Borgmatic is a simple, configuration-driven backup software for servers and workstations. It protects your files with client-side encryption. Backup your databases too. Monitor it all with integrated third-party services.

Vorta is a backup client for macOS and Linux desktops. It integrates the mighty Borg Backup with your favorite desktop environment to protect your data from disk failure, ransomware and theft.

UrBackup is an easy to setup Open Source client/server backup system, that through a combination of image and file backups accomplishes both data safety and a fast restoration time. File and image backups are made while the system is running without interrupting current processes. Available for Windows, macOS, and Linux.

Kopia is a user-friendly desktop app for Windows, macOS, and Linux which allows you to create snapshots, define policies, and restore files quickly with Fast and Encrypted Backups.

Clonezilla is a partition and disk imaging/cloning program. It helps you to do system deployment, bare metal backup and recovery. Three types of Clonezilla are available, Clonezilla live, Clonezilla lite server, and Clonezilla SE (server edition).

rsnapshot is a filesystem snapshot utility based on rsync. This makes it easy to make periodic snapshots of local machines, and remote machines over ssh.

Duplicity is a tool that backs directories by producing encrypted tar-format volumes and uploading them to a remote or local file server. Because duplicity uses librsync, the incremental archives are space efficient and only record the parts of files that have changed since the last backup.

ZnapZend is a high performance open source ZFS backup with mbuffer and ssh support. It uses the built-in snapshot functionality of ZFS for fully consistent backups. For each fileset, a pre- and post-snapshot command can be configured to quiet down any software writing to the fileset prior to snapshotting.

SnapRAID is a folder-based backup tool that behaves like a software or hardware RAID5/6 disk raid, but is not a disk raid itself. There is no realtime recovery, free space between disks cannot be combined and manual excution of backup is needed.

rsync.net is a Cloud Storage for Offsite Backup that give you an empty UNIX filesystem to access with any SSH tool. Built on ZFS for data security and fault tolerance with support for rsync/sftp/scp/borg/rclone/restic/git-annex.

Snapshots Management/System Recovery

Back to the Top

rsnapshot is a filesystem snapshot utility based on rsync. This makes it easy to make periodic snapshots of local machines, and remote machines over ssh.

Sanoid is a policy-driven snapshot management tool for ZFS filesystems.

ZFSBootMenu is a Linux bootloader that attempts to provide an experience similar to FreeBSD's. This allows a user to have multiple "boot environments" (with different distributions, for example), manipulate snapshots before booting, and, for the adventurous user, even bootstrap a system installation via zfs recv.

Btrfs maintenance toolbox is a set of scripts supplementing the btrfs filesystem and aims to automate a few maintenance tasks. This means the scrub, balance, snapshots, trim or defragmentation.

Btrbk is a backup tool for btrfs subvolumes, taking advantage of btrfs specific capabilities to create atomic snapshots and transfer them incrementally to your backup locations.

ksync is a toool that sync files between your local system and a kubernetes cluster. It transparently updates containers running on the cluster from your local checkout.

Verify is a snapshot tool that simplifies the assertion of complex data models and documents.

Timeshift is a Linux application for providing functionality to restore your system just like Windows System Restore tool. Timeshift makes snapshots of your system in regular intervals which are further used at the time of restoration or undo all changes in the system.

CRIU (Checkpoint and Restore in Userspace) is a utility to checkpoint/restore Linux tasks. Using this tool, you can freeze a running application (or part of it) and checkpoint it to a hard drive as a collection of files. You can then use the files to restore and run the application from the point it was frozen at.

Rsync time backup is a Time Machine style backup with rsync. It creates incremental backups of files and directories to the destination of your choice. The backups are structured in a way that makes it easy to recover any file at any point in time. It works on Linux, macOS and Windows (via WSL).

rdiff-backup is a simple backup tool which can be used locally and remotely, on Linux and Windows, and even cross-platform between both. Users have reported using it successfully on FreeBSD and MacOS.

Mainframer is a tool that executes a command on a remote machine while syncing files back and forth. The process is known as remote execution (in general) and remote build (in particular cases).

Archiving

Back to the Top

Access to Memory (AtoM) - Web-based, open source application for standards-based archival description and access in a multilingual, multi-repository environment.

ArchiveBox - Self-hosted wayback machine that creates HTML & screenshot archives of sites from your bookmarks, browsing history, RSS feeds, or other sources.

Archivematica - Mature digital preservation system designed to maintain standards-based, long-term access to collections of digital objects.

ArchivesSpace - Archives information management application for managing and providing Web access to archives, manuscripts and digital objects.

CKAN - CKAN is a tool for making open data websites.

Collective Access - Providence - Highly configurable Web-based framework for management, description, and discovery of digital and physical collections supporting a variety of metadata standards, data types, and media formats.

Omeka S - Omeka S is a web publication system for universities, galleries, libraries, archives, and museums. It consists of a local network of independently curated exhibits sharing a collaboratively built pool of items, media, and their metadata.

Wayback - A self-hosted toolkit for archiving webpages to the Internet Archive, archive.today, IPFS, and local file systems.

Home Server

Back to the Top

Home Assistant is an open source home automation that puts local control and privacy first. Home Assistant is powered by a worldwide community of tinkerers and DIY enthusiasts that runs great on Raspberry Pi.

Homebridge is a software framework that allows you to integrate with smart home devices that do not natively support HomeKit. There are over 2,000 Homebridge plugins supporting thousands of different smart accessories.

Homebridge UI is a tool that provides an easy to use interface to manage your Homebridge plugins, configuration and accessories.

Install and configure Homebridge plugins.
Monitor your Homebridge server via a fully customisable widget-based dashboard.
View and control Homebridge accessories.
Backup and Restore your Homebridge instance.

ESPHome is a system to control your ESP8266/ESP32 by simple yet powerful configuration files and control them remotely through Home Automation systems.

Shelly Cloud is a Smart home control tool that has been perfected and provides precise monitoring of your Shelly devices no matter where you are. Shelly devices are compatible with Alexa, Google Home, Android, and iOS.

Zigbee is the full-stack, secure, reliable, and market-proven solution used by a majority of large smart home ecosystem providers, such as Amazon's Echo Plus, Samsung SmartThings, Signify (Philips Hue), and more.

openHAB is a cross-platform software with the aim to integrate all kinds of Smart Home technologies, devices, etc.

Z-Wave is the leading wireless communications protocol behind many of the secure, trusted brands that are working to make everyone's home smarter and safer.

Homey is an applciation to control, automate and monitor your entire smart home from your phone, tablet or desktop.

Caddy is the only web server to use HTTPS automatically and by default. Caddy obtains and renews TLS certificates for your sites automatically.

Bazarr is a companion application to Sonarr and Radarr. It can manage and download subtitles based on your requirements. You define your preferences by TV show or movie and Bazarr takes care of everything for you.

Sonarr is a PVR for Usenet and BitTorrent users. It can monitor multiple RSS feeds for new episodes of your favorite shows and will grab, sort and rename them.

Homarr is a customizable browser's home page to interact with your homeserver's Docker containers (e.g. Sonarr/Radarr)

Midarr is a free and open source (and always will be), Midarr aims to provide a tailored experience for you and your users:

Beautifully crafted user interface.
Real-time online statuses.
Simple and easy invite system.
Integrates with your existing services, Radarr and Sonarr.

Rustdesk is an open source virtual/remote desktop infrastructure for everyone. Display and control your PC (Windows, macOS, and Linux) and Android devices.

TinyPilot is a tool that enables KVM over IP letting you control any computer remotely.

PM2 is a production process manager for Node.js applications with a built-in load balancer. It allows you to keep applications alive forever, to reload them without downtime and to facilitate common system admin tasks.

authentik is an open-source Identity Provider focused on flexibility and versatility. You can use authentik in an existing environment to add support for new protocols. authentik is also a great solution for implementing signup/recovery/etc in your application, so you don't have to deal with it.

ESPHome Remote IS a WI-FI smart home remote with display that runs on ESPHome. It uses Lilygo T-Display or M5Stack Fire.

Tdarr is a distributed transcode automation application using FFmpeg/HandBrake + Audio/Video library analytics + video health checking (Windows, macOS, Linux & Docker). A common use for Tdarr is to simply convert video files from h264 to h265 (hevc), saving 40%-50% in size.

AppFlowy is an open-source alternative to Notion where you're in charge of your data and customizations.

deemix is a barebone deezer downloader library built from the ashes of Deezloader Remix.

Neko is a self hosted virtual browser that runs in docker and uses WebRTC.

QNAP Switch System (QSS) is a configuration interface for QNAP's managed switch series. Enable management functions such as link aggregation, VLAN, and RSTP, to take care of your network topology with ease.

ASUSTOR is a subsidiary of ASUS and a leading provider of network attached storage (NAS). It specializes in the development and integration of storage, backup, multimedia, video surveillance and mobile applications for home and enterprise users.

Seafile is an open-source, cross-platform file-hosting software system. Seafile organize files into libraries stored on a central server. Each library can be synced into any desktop computer(Windows, Mac and Linux) and mobile devices through apps.

FreeNAS is an Open Source Storage Platform and supports sharing across Windows, Apple, and UNIX-like systems. This includes ZFS (high storage capacities and integrates file systems and volume management into a single piece of software). It supports UPS, CIFS/SMB, FTP, NFS, RSYNC, SSH, AFP, Unison, UPnP, Webserver, iSCSI protocols, local and MS AD authentication, and disk enctyption.

Gladys Assistant is a privacy-first, open-source home assistant and runs great on Raspberry Pi.

Audiobookshelf is a self-hosted audiobook and podcast server.

Mistborn is a secure platform for easily standing up and managing your own cloud services: including firewall, ad-blocking, and multi-factor WireGuard VPN access.

Media Server

Back to the Top

Overseerr is a free and open source software application for managing requests for your media library. It integrates with your existing services, such as Sonarr, Radarr, and Plex.

Jellyfin is a Free Software Media System that puts you in control of managing and streaming your media. It is an alternative to the proprietary Emby and Plex, to provide media from a dedicated server to end-user devices via multiple apps.

Swiftfin is a modern video client for the Jellyfin media server. Redesigned in Swift to maximize direct play with the power of VLC and look native on all classes of Apple devices.

Intro Skipper is a tool that analyzes the audio of television episodes to detect and skip over intro sequences in Jellyfin.

Jellyseerr is a free and open source software application for managing requests for your media library. It is a a fork of Overseerr built to bring support for Jellyfin & Emby media servers.

Midarr is a free and open source (and always will be), Midarr aims to provide a tailored experience for you and your users:

Beautifully crafted user interface.
Real-time online statuses.
Simple and easy invite system.
Integrates with your existing services, Radarr and Sonarr.

Kirino Media Server is a lightweight, modular alternative to Plex and Jellyfin.

Emby is a home media server built on top of other popular open source technologies such as Service Stack, jQuery, jQuery mobile, and Mono. It features a REST-based API with built-in documention to facilitate client development.

OpenMediaVault is a next generation network attached storage (NAS) solution based on Debian Linux. It contains services like SSH, (S)FTP, SMB/CIFS, AFS, UPnP media server, DAAP media server, RSync, BitTorrent client and many more.

MediaElch is a MediaManager for Kodi. Information about Movies, TV Shows, Concerts and Music are stored as NFO files.

tinyMediaManager is a media management tool written in Java/Swing. It is written to provide metadata for the Kodi Media Center (formerly known as XBMC), MediaPortal and Plex media server.

FileBot is the ultimate tool for renaming and organizing your movies, TV shows and Anime. Match and rename media files against online databases, download artwork and cover images, fetch subtitles, write metadata, and more, all at once in matter of seconds.

Plex media server is a application that gives you the power to add, access and share all the entertainment that matters to you, on almost any device. With 50,000+ on demand titles and hundreds of channels of live TV, plus your own personal media collection, using one powerful app.

Tautulli is a 3rd party application that you can run alongside your Plex Media Server to monitor activity and track various statistics.

Plex DupeFinder is a python script that finds duplicate versions of media (TV episodes and movies) in your Plex Library and tells Plex to remove the lowest rated files/versions (based on user-specified scoring) to leave behind a single file/version.

Prometheus Exporter for Plex is an expose library playback, storage, and host metrics in a Prometheus format.

Infuse is a Video Player for iOS, Apple TV, and Mac. It plays every video file ever created to avoid wasting hours converting and transcoding files.

InfuseSync is a plugin for Emby and Jellyfin media servers that tracks all media changes to decrease sync times with Infuse clients.

InvidTUI is an invidious client, which fetches data from invidious instances and displays a user interface in the terminal, and allows for selecting and playing Youtube audio and video.

Polaris is a music streaming application, designed to let you enjoy your music collection from any computer or mobile device. Polaris works by streaming music directly from your computer (or cloud server), without uploading it to a third-party.

AirSonic is a free, web-based media streamer, providing ubiquitous access to your music.

TubeSync is a PVR (personal video recorder) for YouTube. Or, like Sonarr but for YouTube (with a built-in download client). It is designed to synchronize channels and playlists from YouTube to local directories and update your media server once media is downloaded.

yt-fts is a simple python script that uses yt-dlp to scrape all of a youtube channels subtitles and load them into an sqlite database that is searchable from the command line. It allows you to query a channel for specific key word or phrase and will generate time stamped youtube urls to the video containing the keyword.

Tube Archivist is a self hosted YouTube media server.

PeerTube is an ActivityPub-federated video streaming platform using P2P directly in your web browser.

Ant Media Server is a streaming engine software that provides adaptive, ultra low latency streaming by using WebRTC technology with ~0.5 seconds latency.

Castopod is an open-source hosting platform made for podcasters who want engage and interact with their audience.

Festival is a music player for local album collections.

HD HomeRun Scribe 4K is a free local live TV with DVR with 4 tuners and 150 hours of recording storage all-in-one amazing box for watching and recording free TV all around your home.

RuneAudio is a free and open source software that turns embedded hardware into Hi-Fi music players.

Volumio (The Audiophile Music Player) is a very powerful and convenient music aggregator, now Volumio can also be used with great results in all sorts of different situations.

Snapcast is a multiroom client-server audio player, where all clients are time synchronized with the server to play perfectly synced audio. It's not a standalone player, but an extension that turns your existing audio player into a Sonos-like multiroom solution.

SonoBus is an easy to use application for streaming high-quality, low-latency peer-to-peer audio between devices over the internet or a local network.

MythTV is a Free Open Source software digital video recorder (DVR) project distributed under the terms of the GNU GPL.

Smart Home Automation

Back to the Top

Smart home is a process that allows homeowners to control appliances, thermostats, lights, and other smart devices remotely using a smartphone or tablet through an internet connection.

Most smart devices have their own Virtual Local Area Network (VLAN) with little to no internet access with broadcasts forwarding to LAN Subnet aka Subnetwork for discovery. Using software such as Home Assistant, Homebridge, ESPHome, etc. help simplify the process of controlling and automating all your smart devices.

Matter is an open standard for smart home technology that lets your device work with any Matter-certified ecosystem using a single protocol. Matter comes from the Connectivity Standards Alliance, an organization of hundreds of companies(Amazon, Apple, Google, Comcast, Zigbee Alliance, and Connectivity Standards Alliance (CSA) creating products for the smart home.

Proprietary Smart Devices

Amazon Alexa is a smart virtual assistant software to manage Alexa-enabled devices, control music playback, view shopping lists on the go, keep track of upcoming reminders, check on active timers and much more.
Google Assistant is a smart virtual assistant software on mobile and home automation devices.
Apple HomeKit is a software framework that enables your app to coordinate and control home automation accessories from multiple vendors to present a coherent, user-focused interface. Using HomeKit, your app can: Discover HomeKit-compatible automation accessories and add them to a persistent, cross-device home configuration database.
Samsung SmartThings is a sofwtare framework that you can connect, monitor and control multiple smart home devices quicker and easier. Connect your Samsung smart TVs, smart appliances, smart speakers and brands like Ring, Nest and Philips Hue all from one app.
Philips Hue is a smart lighting system. The smart lights, Hue Bridge, and smart controls will forever change the way you experience light.
Sonos is the wireless home sound system that fills as many rooms as you want with great-sounding music, movies, and TV.

------------------------------------------------------------------

Add-ons are additional applications and services, that can be run alongside Home Assistant. The Home Assistant OS and Supervised installations types, provide the Supervisor, which is capable of running and managing these add-ons.

Home Assistant Official Add-ons

Addons created and maintained by the Home Assistant team.

DuckDNS - This updates your Duck DNS IP address and generate SSL using Let's Encrypt.
Almond - An Open, Privacy-Preserving Virtual Assistant.
HomeMatic - HomeMatic central based on OCCU.
Let's Encrypt - Get a free SSL certificate from Let's Encrypt; an open and automated certificate authority (CA).
MariaDB - An open source relational database (fork of MySQL).
File editor - Browser-based configuration file editor.
Mosquitto - Fast and reliable MQTT broker.
Terminal & SSH - Allows logging in remotely to using a web terminal or SSH client.
Samba - Access your configuration files using Windows network shares.
NGINX SSL proxy - Reverse proxy with SSL termination.
deCONZ - Control a ZigBee network using ConBee or RaspBee hardware by Dresden Elektronik.
TellStick - Run a TellStick and TellStick Duo service.
Ada - Ada is voice assistant powered by Almond which is open and privacy-preserving.
Fully Kiosk Browser is a powerful kiosk browser for Android devices. It provides a number of features for monitoring and controlling your Android device. This integration gives you access to control your device and view the status in Home Assistant.

Home Assistant Third Party Add-ons

Add-ons created by the community.

Dasshio - Easily use your Amazon Dash Buttons.
InfluxDB - Scalable datastore for metrics, events, and real-time analytics.
Grafana - Open platform for beautiful analytics and monitoring.
Tor - Protect your privacy and access your instance via Tor.
Spotify Connect - Spotify Connect client for playing music on your Home Assistant device.
SSH & Web Terminal - SSH and Web-based terminal with tons of pre-loaded useful tools.
UniFi Controller - The UniFi Controller allows you to manage your UniFi network using a web browser.
Node-RED - Flow-based programming for the Internet of Things.
Plex Media Server - Your recorded media beautifully organized and ready to stream.
IDE - Advanced web-based IDE, based on Cloud9 IDE.
zigbee2mqtt - Zigbee to MQTT bridge, get rid of your proprietary Zigbee bridges.
Matrix - A secure and decentralized communication platform.
AdGuard Home - A network-wide ad-and-tracker blocking DNS server with parental control.
Traccar - Traccar is modern GPS Tracking Platform.
Home Panel - A touch-compatible web frontend for controlling the home.
Hass.io Google Drive Backup - A complete and easy to configure solution for backing up your snapshots to Google Drive.
Grocy - A groceries & household management solution for your home.
EmonCMS - A powerful open-source web app for processing, logging, and visualizing energy, temperature, and other environmental data.
CrowdSec - A next-gen collaborative IPS/IDS to protect you from intrusion.
AppDaemon - Python Apps and HADashboard.
TasmoAdmin - Centrally manage all your Sonoff-Tasmota devices.
Aircast - AirPlay capabilities for your Chromecast players.
AirSonos - AirPlay capabilities for your Sonos players.
Log Viewer - Browser-based live log viewing utility.
Tautulli - Monitor and get statistics from your Plex server.
motionEye - Simple, elegant and feature-rich CCTV/NVR for your cameras.
JupyterLab - Create documents containing live code, equations, visualizations, and explanatory text.
Glances - A cross-platform system monitoring tool written in Python.

Home Assistant Custom Cards

The Home Assistant Dashboards allows people to build custom cards on top of it, which you can easily add to your instance.

Simple Thermostat - A simpler and more flexible thermostat card.
Card Modder - Style your Lovelace cards.
Bar Card - Customizable animated bar card.
forked-daapd Card - Control a forked daapd instance.
Dual Gauge Card - Shows two gauges in one.
Atomic Calendar Revive - Calendar card with advanced settings.
Simple Weather Card - A minimalistic weather card, inspired by Google Material Design.
Auto-Entities Card - Dynamically adds entities.
Canvas Gauge Card - Use awesome gauges from canvas-gauges.com.
Big Number Card - Display big numbers for sensors, including severity level as background.

Animated Weather Card - Nice looking card showing the weather, with subtle animations.
Thermostat Card - Thermostat control card that looks like a Nest Thermostat.

Raspberry Pi Status Card - Show status of your Raspberry Pis.
Mini Media Player - A minimalistic media player card.
Mini Graph Card - A minimalistic sensor graph card.
Button card - Button card for your entities.
Slider Entity Row - Add a slider to adjust, e.g., the brightness of lights in lovelace entity cards.
Power Wheel Card - An intuitive way to represent the power that your home is consuming or producing.
Home Card - A quick glance of the state of your home.
Banner Card - A fluffy linkable banner with interactive glances to spice up your home dashboards.
Spotify Card - List and select from current available devices and users top playlists on Spotify.
Battery Entity - Displaying battery levels for battery entities.
Multiple Entity Row - Show multiple entity states or attributes on entity rows.
Home Feed Card - Display a combination of persistent notifications, calendar events, and entities in the style of a feed.
Config Template Card - Allow using templates in Lovelace.
RGB Light Card - Colorful buttons to control your RGB Lights.
Restriction Card - A card to provide restrictions on Lovelace cards defined within.
Vacuum Map Card - This card provides a user-friendly way to fully control Xiaomi (Roborock/Viomi/Dreame/Roidmi) and Neato (+ possibly other) vacuums.
Vacuum Card - A card to card for controlling a vacuum cleaner robot.
Purifier Card - A card for controlling air purifiers.

Home Assistant Custom Integrations

Additional integrations for Home Assistant created by the community.

Lutron Caseta Pro - Integrates Lutron Caseta Smart Bridge PRO / RA2 Select.
SmartIR - Integrates devices using Broadlink IR.
Alexa Media Player - Allow control of Amazon Alexa devices.
Circadian Lighting - Circadian Lighting slowly synchronizes your color changing lights with the regular naturally occuring color temperature of the sky throughout the day.
Volkswagen Carnet - Integrates Volkswagen Carnet (requires valid Carnet subscription).
Untappd - Connects with your Untappd account.
Elasticsearch - Publishes events to Elasticsearch.
HASS Aarlo - Asynchronous Arlo integration. Similar to the Arlo web site; monitors events and states for all base stations, cameras and doorbells.
Xiaomi Cloud Map Extractor - Presents a live view of a map for Xiaomi (Roborock/Viomi/Roidmi/Dreame) vacuums without a need for rooting.
Xiaomi Hygrothermo - Sensor platform for Xiaomi Mijia BT Hygrothermo temperature and humidity sensor.
WebRTC Camera - View RTSP streams from IP Cameras in real-time through WebRTC or MSE with Pan/Zoom controls.
Sonoff LAN - Control Sonoff devices with eWeLink (original) firmware over LAN and/or Cloud.
Spotcast - Start Spotify playback on an idle Chromecast device as well as control Spotify connect devices.
The Watchman - Keep track of missing entities and services in your config files.

Home Assistant integrations. Credit: Home Assistant

Homebridge UI is a tool that provides an easy to use interface to manage your Homebridge plugins, configuration and accessories.

Install and configure Homebridge plugins.
Monitor your Homebridge server via a fully customisable widget-based dashboard.
View and control Homebridge accessories.
Backup and Restore your Homebridge instance.

Homebridge Raspberry Pi Image is a free Raspbian based Raspberry Pi image with Homebridge and Homebridge Config UI X pre-installed.

Homebridge Config UI X is a web based management tool for Homebridge that allows you to manage all aspects of your Homebridge setup.

Homebridge webOS TV is a plugin for homebridge which allows you to control your LG webOS TV from your Home app! It should work with all TVs that support webOS2 and newer.

Homebridge Unifi Protect is a Homebridge plugin that provides HomeKit support to the UniFi Protect device ecosystem. UniFi Protect is Ubiquiti's next-generation video security platform, with rich camera, doorbell, and NVR controller hardware options for you to choose from, as well as an app which you can use to view, configure and manage your video camera and doorbells.

Homebridge Camera FFmpeg is a Homebridge Plugin Providing FFmpeg-based Camera Support.

Homebridge Mi Aqara is a homebridge plugin for XiaoMi Aqara plugin.

Homebridge Camera UI is a tool that allows you to expose cameras from camera.ui to HomeKit via Homebridge.

HOOBS is a tool that makes smart accessories(over 2000 options) compatible with your favorite ecosystem. Whether you prefer Apple Homekit, Google Home, or Amazon Alexa, you’re unlikely to find compatible accessories and services that all work together nicely under one roof.

ESPHome is a system to control your ESP8266/ESP32 by simple yet powerful configuration files and control them remotely through Home Automation systems.

Homey is an applciation to control, automate and monitor your entire smart home from your phone, tablet or desktop.

Ecobee is a home automation company in Canada that makes thermostats for residential and commercial use.

Lutron Caséta is a smart lighting control system that is a great solution for giving any client smart lighting control. It was purposely built to work in homes of all ages and it works with older wiring as well as new.

Insteon switches is a Hub for controlling and configuring your home's devices is quick, easy and fun. The setup takes a couple of minutes and a few moments per light switch, sensor, etc. It bridges your devices to 3rd party services like Amazon Alexa and Google Assistant, while being controlled from any smartphone or tablet.

Jeedom is an open source software; taht gives you complete access to the software that manages your home automation. It's compatible with various protocols such as, ZigBee, Z-Wave, EnOcean, KNX, LoRaWAN, BACnet, Modbus, etc..

Beestat is a tool that connects with your thermostat and provides you with useful charts and analytics so that you can make informed decisions and see how the changes you make lower your energy footprint.

openHAB is a cross-platform software with the aim to integrate all kinds of Smart Home technologies, devices, etc.

Z-Wave is the leading wireless communications protocol behind many of the secure, trusted brands that are working to make everyone's home smarter and safer.

pfSense is a firewall/router computer software distribution based on FreeBSD.

AdGuard Home is a DNS relay station with ad/tracker/other blocking, IP address redirections, and DNS-over-HTTPS.

OpenWRT is an open-source project for embedded operating systems based on Linux, primarily used on embedded devices to route network traffic.

ZoneMinder is a full-featured, open source, state-of-the-art video surveillance software system. Monitor your home, office, or wherever you want.

Voice Assistants

Back to the Top

$13 voice assistant remote for Home Assistant

Wyoming is a peer-to-peer protocol for voice assistants (basically JSONL + PCM audio). It's used in Rhasspy and the Home Assistant for communication with voice services.

Wyoming Faster Whisper is a Wyoming protocol server for the faster-whisper speech to text system.

Wyoming Porcupine1 is a Wyoming protocol server for the porcupine1 wake word detection system.

Wyoming Snowboy is a Wyoming protocol server for the snowboy wake word detection system.

faster-whisper is a reimplementation of OpenAI's Whisper model using CTranslate2, which is a fast inference engine for Transformer models.

Porcupine is a highly-accurate and lightweight wake word engine. It enables building always-listening voice-enabled applications. It uses deep neural networks trained in real-world environments.

Rhasspy is an open source voice assistant toolkit for many human languages.

openWakeWord is an open-source wakeword library that can be used to create voice-enabled applications and interfaces. It includes pre-trained models for common words & phrases that work well in real-world environments.

Conversation is an integration allows you to converse with Home Assistant. You can either converse by pressing the microphone in the frontend (supported browsers only (no iOS)) or by calling the conversation/process service with the transcribed text.

Piper is a fast, local neural text to speech system that sounds great and is optimized for the Raspberry Pi 4.

Mycroft is an open source voice assistant that is private by default and completely customizable.

DeepSpeech is an open source embedded (offline, on-device) speech-to-text engine which can run in real time on devices ranging from a Raspberry Pi 4 to high power GPU servers.

Leon is your open-source personal assistant.

Olivia is an open-source chatbot built in Golang using Machine Learning technologies. Its goal is to provide a free and open-source alternative to big services like DialogFlow.

Alan SDK is an voice assistant SDK to build a voice interface for websites and web apps (JavaScript, React, Angular, Vue, Ember, Electron).

OpenAssistant is a chat-based assistant that understands tasks, can interact with third-party systems, and retrieve information dynamically to do so.

Video Surveillance

Back to the Top

Frigate is an open source NVR built around real-time AI object detection. All processing is performed locally on your own hardware, and your camera feeds never leave your home.

hkcam is an open-source implementation of an HomeKit IP camera. It uses ffmpeg to access the camera stream and publishes the stream to HomeKit using hap. The camera stream can be viewed in a HomeKit app.

OpenDataCam is an open source tool to quantify the world. It quantifies and tracks moving objects with live video analysis. It is designed to be an accessible, affordable and open-source solution to better understand interactions in urban environments. It never records any photo or video data. The system only saves surveyed meta-data, in particular the path an object moved or number of counted objects at a certain point.

Viseron is a Self-hosted, local only NVR and AI Computer Vision software.

zmninja is a high performance, cross platform ionic app for Home/Commerical Security Surveillance using ZoneMinder.

Moonfire NVR is a security camera network video recorder.

Shinobi Pro is a Next Generation in Open-Source Video Management Software with support for over 6000 IP and USB Cameras.

WyzeHacks is a project contains a set of scripts trying to provide additional features not implemented by the official firmware. Currently, it provides the following functions:

Enable telnetd on your camera.
Customize the default root password for telnet login.
Redirect all the recordings to an NFS share.
Redirect console logs into an NFS share.
Automatically reboot the camera at certain time.
Automatically archive the recordings.

Text-To-Speech Synthesis (TTS)

Back to the Top

whisper.cpp is a high-performance inference of OpenAI's Whisper automatic speech recognition (ASR) model.

WaaS is a Whisper as a Service (GUI and API for OpenAI Whisper).

Web Whisper is a OpenAI's whisper on your web browser. Demo

Vosk is an offline open source speech recognition toolkit. It enables speech recognition for 20+ languages and dialects.

Coqui TTS is a deep learning toolkit for Text-to-Speech, battle-tested in research and production.

Mozilla TTS is a library for advanced Text-to-Speech generation. It's built on the latest research, was designed to achieve the best trade-off among ease-of-training, speed and quality.

NVIDIA NeMo is a conversational AI toolkit built for researchers working on automatic speech recognition (ASR), text-to-speech synthesis (TTS), large language models (LLMs), and natural language processing (NLP).

Video and Audio Processing

Back to the Top

Intel® Quick Sync Video is a tools that uses the dedicated media processing capabilities of Intel® Graphics Technology to decode and encode fast, enabling the processor to complete other tasks and improving system responsiveness.

Intel® QuickAssist Technology (Intel® QAT) is a scalable, flexible, and extendable way to accelerate data encryption/decryption and compression for applications from networking to enterprise, cloud to storage, and content delivery to database.

FFmpeg is a leading multimedia framework that can decode, encode, transcode, mux, demux, stream, filter and play pretty much anything that humans and machines have created. It supports the most obscure ancient formats up to the cutting edge ones on multiple platforms such as Windows, macOS, and Linux.

FFmpeg.guide is a simple GUI tool to create complex FFmpeg filtergraphs quickly and correctly, without having to mess with the cumbersome filter syntax.

HandBrake is a tool for transcoding video from almost any format with a selection of widely supported codecs. It is supported on Window, macOS, and Linux.

Tdarr is a cross-platform conditional based transcoding application for automating media library transcode/remux management in order to process your media files as required. It can set rules for the required codecs, containers, languages etc that your media should have which helps keeps things organized and can increase compatability with your devices. A common use for Tdarr is to simply convert video files from h264 to h265 (hevc), saving 40%-50% in size.

SRS is a simple, high efficiency and realtime video server, supports RTMP, WebRTC, HLS, HTTP-FLV, SRT and GB28181.

obsws-python is a Python SDK for OBS Studio WebSocket v5.0.

Video/Audio Standards

AAC(Advanced Audio Coding) is an audio coding standard for lossy digital audio compression. It's endorsed by ISO and IEC as MPEG-2 and MPEG-4 standards for video streams.

H.264(AVC) is a video compression standard based on block-oriented and motion-compensated integer-DCT coding that defines multiple profiles (tools) and levels (max bitrates and resolutions) with support up to 8K.

H.265(HEVC) is a video compression standard that is the successor to H.264(AVC). It offers a 25% to 50% better data compression at the same level of video quality, or improved video quality at the same bit-rate.

HTTP Live Streaming (HLS) is a communications protocol developed by Apple that sends live and on‐demand audio and video to iPhone, iPad, Mac, Apple Watch, Apple TV, and PC.

Dynamic Adaptive Streaming over HTTP (DASH) is an adaptive streaming protocol that allows for a video stream to switch between bit rates on the basis of network performance, in order to keep a video playing.

OpenMAX™ is a cross-platform API that provides comprehensive streaming media codec and application portability by enabling accelerated multimedia components to be developed, integrated and programmed across multiple operating systems and silicon platforms.

GStreamer is a library for constructing graphs of media-handling components. The applications it supports range from simple Ogg/Vorbis playback, audio/video streaming to complex audio (mixing) and video (non-linear editing) processing. Applications can take advantage of advances in codec and filter technology transparently.

Media Source Extensions (MSE) is a W3C specification that allows JavaScript to send byte streams to media codecs within Web browsers that support HTML5 video and audio. Also, this allows the implementation of client-side prefetching and buffering code for streaming media entirely in JavaScript.

WebRTC is an open-source project that adds real-time communication capabilities to your application that works on top of an open standard. It supports video, voice, and generic data to be sent between peers, allowing developers to build powerful voice- and video-communication solutions.

Podcasting

Back to the Top

Castopod is an open-source hosting platform made for podcasters who want engage and interact with their audience.

Sovereign Feeds is a tool to Search for your podcasts and add them to your favorites.

IPFS Podcasting is a Decentralized Podcast Distribution over IPFS where you can crowd hosting podcast episodes with storage & bandwidth provided by volunteer nodes.

Audiobookshelf is a self-hosted audiobook and podcast server.

Vod2Pod-RSS is a tool that converts a YouTube or Twitch channel into a podcast with ease. It creates a podcast RSS that can be listened to directly inside any podcast client. VODs are transcoded to MP3 on the fly and no server storage is needed.

Podverse is creating an open source podcast app for iOS, Android, F-Droid, and Web.

Alby is a Bitcoin Lightning App for your Browser.

Alby wallet API is an implemented OAuth an open standard that apps use to provide client applications with secure delegated access. Thus Podverse users create an Alby wallet account or use their existing Alby account to grant specific access rights to their Alby wallet.

Blubrry is a podcast hosting service for publishing platform, live customer support, stress-free migration, and impactful statistics.

SATurn is a tool that lets you connect your getalby.com Account and see which content resonates most with your audience and recognize your top contributors.

AntennaPod is a podcast player that is completely open. The app is open-source and you can subscribe to any RSS feed. AntennaPod is built by volunteers without commercial interest, so it respects your privacy while giving you full control.

Podgrab is a self-hosted podcast manager/downloader/archiver tool to download podcast episodes as soon as they become live with an integrated player.

Podify is a self-hosted service that allows you to download videos and audio from any source supported by youtube-dl, sort the downloads into feeds, and subscribe to these feeds using your favorite podcast app.

dir2cast is designed to turn a directory of MP3s into a podcast - automatically. Perfect for, say, radio shows - upload the MP3s to a folder, and use dir2cast.php as your PodCast URL.

Snipd is an AI-powered podcast player. Search for moments in the transcript, get summaries, share clips to social media, and export to your second brain & note taking apps. Discover, save, and share highlights from podcasts.

Wave Share is a serverless, peer-to-peer, local file sharing through sound.

KBD Audio is a collection of command-line and GUI tools for capturing and analyzing audio data.

AudioBooks

Back to the Top

Audioserve is a simple personal server to serve audio files from directories. Intended primarily for audio books, but anything with decent directories structure will do. Focus here is on simplicity and minimalist design.

Audiobookshelf is a self-hosted audiobook and podcast server.

Jellyfin Bookshelf Plugin

Health

Back to the Top

Connect is the swiss army knife of healthcare integration.

Fasten is an open-source, self-hosted, personal/family electronic medical record aggregator, designed to integrate with 1000's of insurances/hospitals/clinics

ERPNext is a Free and Open Source Enterprise Resource Planning (ERP) for managing businesses.

OpenEMR is a Free and Open Source electronic health records and medical practice management application. It features fully integrated electronic health records, practice management, scheduling, electronic billing, internationalization, free support, a vibrant community, and a whole lot more. It runs on Windows, Linux, MacOS, and many other platforms.

Ryot (Roll Your Own Tracker) is a self hosted platform for tracking various facets of your life - media, fitness etc.

Gardening

Back to the Top

OpenSprinkler is an open-source, web-based smart sprinkler controller for lawn and plant watering, drip irrigation, farm irrigation, hydroponics etc. The current version is OS 3.2, with built-in WiFi (based on ESP8266) and OLED display.

Droplet is an ALL-IN-ONE Irrigation and monitoring system for ESPHome and Home Assistant.

9 Valve Sprinkler Controller is a 9 valve sprinkler controller for use with customized firmware such as ESPHOME.

GardenBot is an open source garden monitoring system. Their website is a collection of tutorials for how to build things (like a soil moisture sensor), software for running GardenBot, resources, links, and more.

farmOS is a web-based application for farm management, planning, and record keeping. It is developed by a community of farmers, developers, researchers, and organizations with the aim of providing a standard platform for agricultural data collection and management.

OpenFarm is a free and open database and web application for farming and gardening knowledge. One might think of it as the Wikipedia for growing plants, though it functions more like a cooking recipes site.

Growstuff is an open source/open data project for food gardeners. They crowdsource information on what our members are growing and harvesting, aggregate it, and make it available as open data via their API.

Harvest Helper is a tool that provides growing, harvesting and recipe information for the 45 plants in the database as well as a json api so that people can hopefully use this data to build other apps.

HappyPlants is a mobile web application that's all about collecting, organising, and adding all kinds of information of your plants. Basically, creating your own plant database in a visual way.

Automated irrigation system is an open source application to water plants automatically. Up to now there is almost no free professional software and instructions available to build a DYI irrigation that is scalable, accurate and most importantly, durable.

Pigrow is a garden automation suite designed to help gardeners monitor, log, graph and control their grow space using a raspberry Pi, various sensor and a few relay modules.

Tania is a farm management software for the hobbyist and smallholder farmer.

Maps

Back to the Top

Magic Earth is aTurn-by-turn navigation, OpenStreetMap, Crowd-Sourced Traffic, 3D maps, Satellite maps, Offline maps and Transit.

Organic Maps is a free Android & iOS offline maps app for travelers, tourists, hikers, and cyclists. It uses crowd-sourced OpenStreetMap data and is developed with love by MapsWithMe (MapsMe) founders and our community. No ads, no tracking, no data collection, no crapware.

MapTiler Server is a self-hosted Aerial and satellite imagery maps of the entire world from your own server or laptop.

GPSLogger is a GPS tool that uses the GPS capabilities of your Android phone to log coordinates to GPS format files at regular intervals. This can be particularly useful if you want to geotag your photos after a day out or share your travel route with someone.

KelperJs is a open source full-stack geosocial network platform.

OpenStreetMap(OSM) is a map of the world, created by people like you and free to use under an open license. Hosting is supported by UCL, Fastly, Bytemark Hosting, and other partners.

uMap is a tool that lets you create maps with OpenStreetMap layers in a minute and embed them in your site.

Martin is a tile server able to generate vector tiles from large PostGIS databases on the fly, or serve tiles from PMTile and MBTile files. Martin optimizes for speed and heavy traffic, and is written in Rust.

MapLibre GL JS is an open-source library for publishing maps on your websites or webview based apps. Fast displaying of maps is possible thanks to GPU-accelerated vector tile rendering.

MapLibre Native is an Interactive vector tile maps for iOS, Android and other platforms.

Maplibre-rs is an Experimental Maps for Web, Mobile and Desktop.

Bookmarks

Back to the Top

Linkding is a simple bookmark service that you can host yourself. It's designed be to be minimal, fast, and easy to set up using Docker.

Linkwarden is a fully self-hostable, open-source collaborative bookmark manager to collect, organize and archive webpages. Linkwarden Docker Image

LinkAce is a self-hosted bookmark archive to collect links of your favorite websites.

Eagle is a tool to collect, search and organize your design files in a logical way and all in one place.

Shlink is a self-hosted URL shortener that keeps control over all your shortened URLs, by serving them under your own domains, using this simple yet powerful tool.

Pinry is a tiling image board system for people who want to save, tag, and share images, videos and webpages in an easy to skim through format.

Shaark is a self-hosted platform to keep and share your content: web links, posts, passwords and pictures.

Maglit is an encrypted and privacy respecting Link Shortener service that supports not only your regular website links but also Magnet Links which are extensively used to download and share torrents.

Photos

Back to the Top

PhotoPrism® is an AI-powered app for browsing, organizing & sharing your photo collection. It makes use of the latest technologies to tag and find pictures automatically without getting in your way. You can run it at home, on a private server, or in the cloud.

Immich is a high performance self-hosted photo and video backup solution directly from your mobile phone.

Piwigo is a full featured, self-hosted, and open source photo gallery application for the web. It comes with more than 200 templates, plugins, and configurations that let you personalize how your photos are presented. It allows users to upload photos from digiKam, Shotwell, Lightroom or mobile applications. Lastly, users can create galleries and give viewing permissions to their clients. They can download individual photos or whole albums, post comments, give ratings, mark photos as favorites, and perform searches.

Czkawka is a Multi-functional app to find duplicates, empty folders, similar images, etc.

Phockup is a Media sorting tool to organize photos and videos from your camera in folders by year, month and day.

PiGallery 2 is a fast directory-first photo gallery website, with rich UI, optimized for running on low resource servers (especially on Raspberry Pi).

Photoview is a simple self-hosted and user-friendly photo gallery that's made for photographers and aims to provide an easy and fast way to navigate directories, with thousands of high-resolution photos.

digiKam is a free and open-source Professional Photo Management tool.

ShareX is a free and open source program that lets you capture or record any area of your screen and share it with a single press of a key. It also allows uploading images, text or other types of files to many supported destinations you can choose from.

PhotoSync is a service to wirelessly transfer, backup & share photos/videos to your computer, NAS, other phones and popular cloud/photo services. It's available for Windows, MacOS, Linux, Android, and iOS.

Lychee is a great looking and easy-to-use photo-management-system you can run on your server, to manage and share photos.

Photoview is a simple and user-friendly photo gallery that's made for photographers and aims to provide an easy and fast way to navigate directories, with thousands of high-resolution photos.

Gimme-iPhotos is a tool uses pyicloud to synchronize photos and videos from iCloud to your local machine.

PyiCloud is a module which allows pythonistas to interact with iCloud webservices. It's powered by the fantastic requests HTTP library.

Pixelfed is a fresh take on photo sharing. It decentralized ActivityPub protocol so you can comment, follow, and interact with remote Pixelfed, Mastodon and Pleroma posts and profiles from your Pixelfed account as if you were both on the same website.

Chevereto is an image hosting software that allows you to create a beautiful and full-featured image hosting website on your own server.

Got Your Back (GYB) is a command line tool for backing up your Gmail messages to your computer using Gmail's API over HTTPS.

Upscayl is a free and open source desktop application that lets you upscale your low resolution images using advanced AI Models. Upscayl is a Linux-First Application that prioritizes Linux builds but is also cross-platform.

Librephotos is a self-hosted open source photo management service. This is the repository of the backend.

Librephotos frontend is a self-hosted open source photo management service. This is the repository of the frontend.

Librephotos Mobile is an open-source Android and iOS Mobile Application for self-hosted Librephotos Server.

Librephotos Docker is the Dockerfiles for the automated build process of LibrePhotos.

OneFolder is a Desktop app to sort your images the same way you would in Google Photos, but locally, no need to run a server (and compatible with NAS).

Pastebins

Back to the Top

Bepasty is a pastebin for all kinds of files.

Bin is a paste bin that's actually minimalist.

Dpaste is a Simple pastebin with multiple text and code option, with short url result easy to remember.

Drift is a Self-hosted Github Gist clone.

EdPaste is a Self-hosted pastebin written in Laravel (PHP Framework).

ExBin is a pastebin with public/private snippets and netcat server.

Fiche is a Command line pastebin, all you need is netcat.

Filite is a simple, light and standalone pastebin, URL shortener and file-sharing service.

FlashPaper is a one-time encrypted zero-knowledge password/secret sharing application focused on simplicity and security. No database or complicated set-up required.

Hasty Paste is a place to quickly paste some text and share it. Mostly used for sharing debug logs and such to help developers provide tech support. The project aims to be both fast and minimal.

Lenpaste is a Web service that allows you to share notes anonymously, an alternative to pastebin.

LogPaste is a Minimal pastebin web app that's easy to self-host and persists data to any S3-compatible backend.

MicroBin is a super tiny, feature rich, configurable, self-contained and self-hosted paste bin web application. It is very easy to set up and use, and will only require a few megabytes of memory and disk storage. MicroBin Docker install setup.

Opengist is a Self-hosted pastebin powered by Git.

Paaster is a secure by default end-to-end encrypted pastebin built with the objective of simplicity.

Pastefy - Beautiful, simple and easy to deploy Pastebin with optional Client-Encryption, Multitab-Pastes, an API, a highlighted Editor and more.

Pastila is a Minimalistic paste service. Single page, zero click experience.

Pasty is a fast and lightweight code pasting server.

PrivateBin is a minimalist, opensource online pastebin/discussion board where the server has zero knowledge of hosted data.

Prologic pastebin is a Simple pastebin service with convenient api and CLI.

PurritoBin is a Ultra fast, minimalistic, encrypted command line paste-bin, where the server has no knowledge of the paste data.

Rustypaste is a minimal file upload/pastebin service.

Spacebin is a Reliable Pastebin server in Golang and Fiber.

Sup3rS3cretMes5age is a simple to deploy and use secret message service using Hashicorp Vault as a secrets storage.

Wastebin is a Lightweight, minimal and fast pastebin with an SQLite backend.

YABin is a pastebin that contains plentiful features while remaining simple. Supports optional E2E encryption, a client-side CLI app, syntax highlighting, minimalistic UI, APIs, keyboard shortcuts, and more. It can even be run in serverless environments.

Note-Taking

Back to the Top

Joplin is an open source note-taking app that you can securely access from any device.

HedgeDoc is an open-source, web-based, self-hosted, collaborative markdown editor.

Lapce is a Lightning-fast And Powerful Code Editor written in pure Rust with a UI in Druid (which is also written in Rust).

nb is a CLI and local web plain text note‑taking, bookmarking, and archiving with linking, tagging, filtering, search, Git versioning & syncing, Pandoc conversion, + more, in a single portable script.

Outline is the fastest knowledge base for growing teams. It provides a beautiful, realtime collaborative, feature packed, and markdown compatible.

Rustpad is an open-source collaborative text editor based on the operational transformation algorithm. Share a link to this pad with others, and they can edit from their browser while seeing your changes in real time.

Turtl is a secure, collaborative notebook for bookmarks or passwords, files or shopping lists.

The Everything App is an app where you can do everything: Protect your thoughts & data with end-to-end encryption. Local, on-device encryption. Only you have encryption keys. Offline account creation: control your keys, own your data. No server, no gatekeeper: peer-to-peer sync on local networks. Locally store your data, self-host your backups where you please.

TiddlyWiki is a single-file mode wiki application for todo lists, effective project management tool and of course writing drafts and notes. It has extensions for all the major browsers.

Laverna is a note taking application with Markdown editor and encryption support. Consider it like open source alternative to Evernote.

Notesnook is a fully open source & end-to-end encrypted note taking alternative to Evernote.

Zettlr is an open-source Markdown editor for the 21st century.

Carnet is a complete open source note taking app. It has extensions for all the major browsers.

Frog is a tool that quickly extract text from almost any source: youtube, screencasts, PDFs, webpages, photos, etc. Grab the image and get the text.

Zeal is an offline documentation browser for software developers inspired by Dash.

Time Monitoring

Back to the Top

ActivityWatch is an app that automatically tracks how you spend time on your devices.

Kimai is a free & open source timetracker. It tracks work time and prints out a summary of your activities on demand.

Solidtime is an open source time tracking software for individuals and teams, with a modern user interface and reporting.

TimeTagger is an open source time-tracker based on an interactive timeline and powerful reporting.

Traggo is a tag-based time tracking tool. In Traggo there are no tasks, only tagged time spans.

Wikis

Back to the Top

Archivy is a self-hostable knowledge repository that allows you to learn and retain information in your own personal and extensible wiki.

BookStack - BookStack is a simple, self-hosted, easy-to-use platform for organizing and storing information. It allows for documentation to be stored in a book like fashion.

Cowyo - Cowyo is a feature-rich wiki for minimalists.

django-wiki - Wiki system with complex functionality for simple integration and a superb interface. Store your knowledge with style: Use django models.

Documize - Modern Docs + Wiki software with built-in workflow, single binary executable, just bring MySQL/Percona.

Dokuwiki - Easy to use, lightweight, standards-compliant wiki engine with a simple syntax allowing reading the data outside the wiki. All data is stored in plain text files, therefore no database is required.

Gitit - Wiki program that stores pages and uploaded files in a git repository, which can then be modified using the VCS command line tools or the wiki's web interface.

Gollum - Simple, Git-powered wiki with a sweet API and local frontend.

Instiki - Instiki is a wiki clone so pretty and easy to set up, you'll wonder if it’s really a wiki. Runs on Rails and focuses on portability and stability.

Mediawiki - MediaWiki is a free and open-source wiki software package written in PHP. It serves as the platform for Wikipedia and the other Wikimedia projects, used by hundreds of millions of people each month.

Pepperminty Wiki - Complete markdown-powered wiki contained in a single PHP file.

PineDocs - Simple, fast, customizable and lightweight site for browsing files.

PmWiki - Wiki-based system for collaborative creation and maintenance of websites.

PukiWiki - PukiWiki is a free, simple, open-source wiki management system. Minimalistic and simple design with many tools for collaborative work. Created in Japanese, so translator needed.

Raneto - Raneto is an open source Knowledgebase platform that uses static Markdown files to power your Knowledgebase.

TiddlyWiki - Reusable non-linear personal web notebook.

Tiki - Wiki CMS Groupware with the most built-in features.

TWiki - TWiki is a Perl-based structured wiki application, typically used to run a collaboration platform, knowledge or document management system, a knowledge base, or team portal.

WackoWiki - WackoWiki is a light and easy to install multilingual Wiki-engine.

Gramax - Free, open-source application for creating, editing, and publishing Git-driven documentation sites using Markdown and a visual editor.

Gaming

Back to the Top

Cartridge is a self-hosted game library made with Laravel + Vue.js.

Cartridge Features

Scan for ROM files and match with IGDB game information
Serve ROM download links alongside game details
Manage access to library with user creation and permissions (WIP)
Allow users to request games (Planned)
Play select ROMs in-browser using JS emulators (Planned)
Track played and favorite games (even ones that aren't available for download) (Planned)

Moonlight Game Streaming is a program that let you stream from your PC games over the Internet with no configuration required. Stream from almost any device, whether you're in another room or miles away from your gaming rig. Sunshine is a Game stream host for Moonlight that is a self-hosted, low latency, cloud gaming solution with support for AMD, Intel, and NVIDIA GPUs. It is an open source implementation of NVIDIA's GameStream.

Chiaki is a Free and Open Source Software Client for PlayStation 4 and PlayStation 5 Remote Play for Linux, FreeBSD, OpenBSD, Android, macOS, Windows, Nintendo Switch and potentially even more platforms.

Game Emulators

EmuDeck is a tool that takes care of everything for your retrogaming needs from RetroArch Configuration, Bezels, Gamepad Configuration for GameCube, Wii, Citra, SNES, etc. EmuDeck will even install EmulationStation Desktop Edition and carry over all their custom configurations and no need to configure ROM paths or anything.

EmulationStation Desktop Edition (ES-DE) is a frontend application for browsing and launching games from your multi-platform game collection. It's available for Unix/Linux, macOS(M1 & Intel) and Windows.

RetroPie is a frontend for emulators that allows you to turn your Raspberry Pi, ODroid C1/C2, or PC into a retro-gaming machine. It builds upon Raspbian, EmulationStation, RetroArch and many other projects to enable you to play your favourite Arcade, home-console, and classic PC games with the minimum set-up.

RetroArch is a frontend for emulators, game engines and media players. It enables you to run classic games on a wide range of computers and consoles through its slick graphical interface. Settings are also unified so configuration is done once and for all. RetroArch Flatpak

Pterodactyl is a free, open-source game server management panel built with PHP, React, and Go. Designed with security in mind, Pterodactyl runs all game servers in isolated Docker containers while exposing a beautiful and intuitive UI to end users.

LinuxGSM (Linux Game Server Managers) is a command-line tool for quick, simple deployment and management of Linux dedicated game servers.

Cartridge is a self-hosted game library made with Laravel + Vue.js.

Cartridge Features

Scan for ROM files and match with IGDB game information
Serve ROM download links alongside game details
Manage access to library with user creation and permissions (WIP)
Allow users to request games (Planned)
Play select ROMs in-browser using JS emulators (Planned)
Track played and favorite games (even ones that aren't available for download) (Planned)

Cartridge UI

Dolphin is an emulator for two recent Nintendo video game consoles: the GameCube and the Wii. It allows PC gamers to enjoy games for these two consoles in full HD (1080p) with several enhancements: compatibility with all PC controllers, turbo speed, networked multiplayer, and even more.

Citra is an open-source emulator for the Nintendo 3DS capable of playing many of your favorite games.

yuzu is an experimental open-source emulator for the Nintendo Switch from the creators of Citra.[

m64p is a Nintendo 64 Emulator. It uses mupen64plus-gui, a brand new mupen64plus frontend written in Qt5. It supports all of the things you’d expect from a frontend (savestate management, pausing, screenshots).

DeSmuME is a Nintendo DS emulator.

Snes9x is a portable, freeware Super Nintendo Entertainment System (SNES) emulator.

bsnes is a Super Nintendo (SNES) emulator focused on performance, features, and ease of use.

mGBA is a new emulator for running Game Boy Advance games. It aims to be faster and more accurate than many existing Game Boy Advance emulators, as well as adding features that other emulators lack.

DOSBox is an open-source DOS emulator which primarily focuses on running DOS Games.

DOSBox Staging is a full x86 CPU emulator (independent of host architecture), capable of running DOS programs that require real or protected mode.

Flycast is a multi-platform Sega Dreamcast, Naomi and Atomiswave emulator derived from reicast.

PCSX2 is a PlayStation 2 'emulator', a free program that tries to replicate the PlayStation 2 console to enable you to play PS2 games on your PC.

RPCS3 is an experimental open-source Sony PlayStation 3 emulator and debugger written in C++ for Windows and Linux. RPCS3 started development in May of 2011 by its founders DH and Hykem. The emulator is currently capable of running over 1800 commercial titles powered by Vulkan and OpenGL.

MAME is a Arcade Machine Emulator.

xemu is an original Xbox emulator.

Xenia is an Xbox 360 Emulator.

Also checkout these subreddits for more great Game Emulators recommendations

Foundations/Projects

Back to the Top

Open Source Hardware Association (OSHWA) is a non-profit organization that advocates for open-source hardware. It aims to act as a hub of open source hardware activity of all types while actively cooperating with other initiatives such as the TAPR Open Hardware License, open-source development groups at CERN, and the Open Source Initiative (OSI).

The Open Connectivity Foundation is dedicated to ensuring secure interoperability for consumers, businesses and industries by delivering a standard communications platform, a bridging specification, an open source implementation and a certification program allowing devices to communicate regardless of form factor, operating system, service provider, transport technology or ecosystem.

Raspberry Pi Foundation is a UK-based charity with the mission to enable young people to realise their full potential through the power of computing and digital technologies.

OpenSSF(Open Source Security Foundation) is a cross-industry forum for a collaborative effort to improve open source software security.

OpenJS Foundation is the premier home for critical open source JavaScript projects, including Appium, Dojo, jQuery, Node.js, and webpack, and 27 more.

EdgeX Foundry is a vendor-neutral project under the Linux Foundation. The initiative is aligned around a common goal: the simplification and standardization of the foundation for edge computing architectures in the Industrial IoT market, while still allowing the ecosystem to add significant value.

Eclipse Foundation provides our global community of individuals and organizations with a mature, scalable and commercially-friendly environment for open source software collaboration and innovation.

System Hardware

Back to the Top

CPUs

Intel Processors(x86)

Back to the Top

I recommend using Intel CPUs no older than the second generation of the Intel Core processors (Core i7, i5, i3) AKA Sandy Bridge(Jan. 2011) for those that want to utilize Intel® Quick Sync Video. Though, if you're concerned about power efficiency(~5W idle) I would recommend 7th Generation or newer.

Also, I recommend using Intel® QuickAssist Technology (Intel® QAT) a scalable, flexible, and extendable way to accelerate data encryption/decryption and compression for applications from networking to enterprise, cloud to storage, and content delivery to database. Available in 3rd Gen Intel® Xeon® Scalable Processors and Intel Atom® Processor C Series/P Series.

AMD Processors(x86)

Back to the Top

Devices

Back to the Top

Note: Will be adding more device soon!

Raspberry Pi
Turing Pi 2
Home Assistant Yellow
ZimaBoard
ODROID-H3 and H3+
Intel® NUC Mini PCs
Beelink mini PC
M1 Mac Mini
Nexcom Industrial Computers
Aeotec MultiSensor 7, 6-in-1 Zwave Sensors
reTerminal Raspberry Pi (CM4 module) all-in-one board
KOOLCORE R1 - The smallest mini PC with 4 x 2.5G LANs
Khadas VIM1S
Asustor DriveStor 4 NAS
TRENDnet TEG-S350 (2.5 GbE) Switch
Storinator™ is a line of Ultra-Large, Direct-Wired storage Servers by 45Drives.
HL15 from 45HomeLab is an open-source, open-platform, 15-bay homelab server. The HL15 features enterprise architecture and strength brought to a scale that works for the homelab. The server's direct-wired architecture can provide blazing fast transfer speed of up to 2GB per second.
LattePanda Sigma is a powerful and compact x86 Windows single board computer (SBC). It features the 13th Intel® Core™ i5-1340P Rapter Lake (12-Core, 16-Thread) processor and 16GB Dual-Channel LPDDR5-6400MHz memory.
Apex Storage X21 is a storage solution that gives you have the freedom to choose system hardware thatworks best for you with the following benefits.
- Host 21 x M.2 Gen 4 NVME SSD’s
- 168 TB + Storage Per Card
- 31 GBps Read/Write Speeds
- Industry Leading IOPS
- 100 PCIe 4.0 Lanes
- Full UEFI/Secure Boot Support
GL.iNet is a leading developer of OpenWrt Wi-Fi and IoT Network Solutions. They build Wi-Fi routers, IoT gateways and remote device management platforms for a wide range of scenarios. All their routers include powerful built-in firewall, Shadow also supports OpenVPN, WireGuard® and customized DNS server in order to level up your online security.
Protectli Vault is a series of small computers that have firewalls with advanced firmware protection to keep your network safe. All Vaults are Opertaing System (OS) agnostic and support a variety of popular open source software distributions. Run a firewall like OPNsense, pfSense, or others. Also, run any hypervisor on your system.
Espclicker is a small device that you can use to integrate not IoT devices into your smart home setup, It connects directly to the push buttons of the device, which you want to control, and then simulates pressing them. You can simulate button clicks, hold time, and multiple clicks.
Pockethernet is a smartphone connected Ethernet network analyzer & cable tester that fits into your pocket. It helps any IT admin dealing with Ethernet network installation and maintenance. It lets you check the ethernet link, find cable faults, PoE voltage, VLAN, DHCP results and much more with the press of a button.
$13 USD voice assistant remote for Home Assistant
ATOM ECHO is a Programmable Smart Speaker based on the M5ATOM design. Music can be played using the BT (Bluetooth Technology) capabilities of the ESP32 from a mobile phone or tablet. The device could be programmed to access AWS, Azure, and other cloud platforms, using the built-in microphone and speaker for voice interaction.

Operating Systems

Back to the Top

Creating a bootable media device(USB/MicroSD card)

Rufus is a utility that helps format and create bootable USB flash drives.

Rufus

Etcher is an open source, cross-platform software that makes it easy to flash operating system images to a microSD card or USB device.

Etcher UI

A List of Operating Systems that are great for either settig up a personal Home Server or a Enterprise Server for your Organization/Company.

Home Assistant OS is a container-based system for managing your Home Assistant Core installation and related applications. The system is controlled via Home Assistant which communicates with the Supervisor. The Supervisor provides an API to manage the installation. This includes changing network settings or installing and updating software.

Home Assistant OS

Umbrel is an OS for running a personal server in your home. It can Self-host open source apps like Nextcloud, Bitcoin node, and more.

Umbrel

CasaOS is a simple, easy-to-use, elegant open-source Home Cloud system.

CasaOS

TrueNAS® CORE is the world's most popular storage OS because it gives you the power to build your own professional-grade storage system to use in a variety of data-intensive applications without any software costs. It's based on FreeBSD and Linux, using the OpenZFS file system.

TrueNAS CORE

Alpine Linux is a security-oriented, lightweight Linux distribution based on musl libc and busybox.

Xfce4 Desktop

Enable the Community repository, then execute command:

apk add xfce4

Alpine Linux Xfce

Mate Desktop

Enable the Community repository, then execute command:

apk add mate-desktop-environment

Alpine Linux MATE

Ubuntu is a modern open source operating system on Linux for the enterprise Server, Desktop, Cloud, and IoT developed by Canonical.

Ubuntu Server
Ubuntu for ARM
Ubuntu for Raspberry Pi
Ubuntu Flavours is for those that prefer an alternative desktop environment such as KDE Plasma Desktop, MATE, Xfce, LXQt, Budgie, and UKUI you can download a Flavour for your preferred desktop environment and use that to install Ubuntu, pre-configured for the desktop environment of your choice.

Ubuntu

Debian is an operating system and a distribution of Free Software. It is maintained and updated through the work of many users who volunteer their time and effort.

Debian 11

Linux Mint is a modern, elegant, and comfortable open source operating system(based on Debian and Ubuntu), which is both powerful and easy to use for both new and advanced users. The flagsip version of Linux Mint uses the Cinnamon desktop environment similiar to Windows 7.

Linux Mint

Linux Mint Debian Edition (LMDE) uses Debian Bullseye as the base for a very stable and rock solid user experience with the Cinnamon desktop.

Linux Mint Debian Edition (LMDE)

Pop!_OS created by System76.

Pop!_OS

Fedora Linux is a polished, easy to use operating system for laptop & desktop computers, with a complete set of tools for developers and makers of all kinds. The OS serves as the foundation for which you can scale existing apps and roll out emerging technologies across bare-metal, virtual, container, and all types of cloud environments.

Fedora Spins is for those that prefer an alternative desktop environment such as KDE Plasma Desktop, MATE; or Xfce, you can download a spin for your preferred desktop environment and use that to install Fedora, pre-configured for the desktop environment of your choice.
Fedora Server is a powerful, flexible operating system that includes the best and latest datacenter technologies. It puts you in control of all your infrastructure and services.
Fedora ARM is an initiative to bring versions of Fedora tailored for running on ARM-based systems.
Fedora Silverblue is a variant of the Fedora Workstation that uses rpm-ostree to provide an immutable OS image with reliable updates and easy rollbacks.
Fedora Kinoite is an immutable desktop operating system. It aims to be extremely stable and reliable. It also aims to be an excellent platform for developers and for those using container-focused workflows. Kinoite is a variant of the Fedora KDE Spin.
Fedora CoreOS is an automatically-updating, minimal operating system for running containerized workloads securely and at scale.

Fedora Linux

CentOS Stream is a continuously delivered distro(uses the Fedora OS base) that tracks just ahead of Red Hat Enterprise Linux (RHEL) development, positioned as a midstream between Fedora Linux and RHEL.

CentOS Stream

Red Hat® Enterprise Linux® (RHEL) the world's leading enterprise Linux platform. The OS serves as the foundation for which you can scale existing apps and roll out emerging technologies across bare-metal, virtual, container, and all types of cloud environments.

Red Hat Enterprise Linux Desktop

AlmaLinux is an open source enterprise-ready Linux distribution forked from Red Hat Enterprise Linux(RHEL). It's a very stable/solid operating system especially for production envrionments such as servers, though, you can also install a GUI on AlmaLinux and use it as a desktop OS. It was founded by the team behind the CloudLinux OS.

AlmaLinux Desktop Setup and Install

Rocky Linux is a community enterprise operating system designed to be 100% bug-for-bug compatible and forked from Red Hat Enterprise Linux(RHEL) now that its downstream partner(Red Hat) has shifted direction.

Rocky Linux Desktop

SUSE is a leading Linux OS most adaptable Linux operating system and the only open Kubernetes management platform thanks to their acquistion of Rancher. They also developer of SUSE Linux Enterprise and the primary sponsor of the community-supported openSUSE Project, which develops the openSUSE Linux distribution.

openSUSE Leap is a brand new way of building openSUSE and is new type of a hybrid Linux distribution. Leap uses source from SUSE Linux Enterprise (SLE), which gives Leap a level of stability unmatched by other Linux distributions, and combines that with community developments to give users, developers and sysadmins the best stable Linux experience available.
openSUSE Tumbleweed is a pure rolling release version of openSUSE containing the latest "stable" versions of all software instead of relying on rigid periodic release cycles. The project does this for users that want the newest stable software.
openSUSE Kubic is a multi-purpose Standalone & Kubernetes Container Operating System based on openSUSE MicroOS. Kubic uses kubeadm to provide an easy way of configuring a Kubernetes cluster across multiple machines, while our MicroOS base keeps your operating system updated automatically, with fully atomic rollbacks if required.
openSUSE MicroOS is a M icro Service OS providing Transactional (Atomic) Updates upon a read-only btrfs root filesystem. It's designed to host container workloads with automated administration & patching.

SUSE Linux Enterprise 12

openSUSE

NixOS is a Linux distribution built on top of the Nix package manager. It has tools dedicated to DevOps and deployment tasks. NixOS Guide

Nix Tour is an interactive tour that uses the actual package manager to learn you the language by example, in the browser.
Nix is a package manager and build system that parses reproducible build instructions specified in the Nix Expression Language, is a pure functional language with lazy evaluation. Nix expressions are pure functions taking dependencies as arguments and producing derivation specifying a reproducible build environment for the package. Nix stores the results of the build in unique addresses specified by a hash of the complete dependency tree, creating an immutable package store that allows for atomic upgrades, rollbacks and concurrent installation of different versions of a package, essentially eliminating dependency hell.
Nix Expression Language is a pure, lazy, functional language. Purity means that operations in the language don't have side-effects (for instance, there is no variable assignment). The language is not a full-featured, general purpose language. Its main job is to describe packages, compositions of packages, and the variability within packages.
Nixpkgs is the largest repository of Nix packages(over 80,000 packages) and NixOS modules. The repository is hosted on GitHub and maintained by the community, with official backing from the NixOS Foundation. Additionally, checkout Language-specific package helpers and Alternative Package Sets.
NixOS Packages Search is a tool for searching through NixOS packages.

NixOS Packages Search

NixOS Desktop with the new Calamares Installer

NixOS with the Plasma Desktop

BSD

Back to the Top

FreeBSD is an Unix-like operating system used to power modern servers, desktops, and embedded platforms. A large community has continually developed it for more than thirty years. Its advanced networking, security, and storage features have made FreeBSD the platform of choice for many of the busiest web sites and most pervasive embedded networking and storage devices.

FreeBSD Software ports. Source: FreeBSD Software

OpenBSD is a security-focused, free and open-source, Unix-like operating system based on the Berkeley Software Distribution. It comes with a secure minimal firewall, webserver, mailserver, and an optional graphical desktop.

NetBSD is a free, fast, secure, and highly portable Unix-like Open Source operating system. It is available for a wide range of platforms, from large-scale servers and powerful desktop systems to handheld and embedded devices.

NetBSD Documentation

DragonFly BSD is a free and open-source Unix-like operating system forked from FreeBSD 4.8.

DragonFly Documentation

The BSD Desktop for the average user

GhostBSD is a simple desktop-oriented operating system based on FreeBSD with MATE, OpenRC and OS packages for simplicity. GhostBSD has a selection of commonly used software preinstalled and required to start using it to its full potential.

GhostBSD Desktop. Source: GhostBSD

Storage

Back to the Top

-Storage Reference Guide by Storage Review

Useful Tools for Storage Management

Scrutiny is a WebUI for smartd Hard Drive S.M.A.R.T Monitoring, Historical Trends & Real World Failure Thresholds.

smartd is SMART Disk Monitoring Daemon for Linux. It controls and monitors storage systems using the Self-Monitoring, Analysis and Reporting Technology System (SMART) built into most modern ATA/SATA, SCSI/SAS and NVMe disks. In many cases, these utilities will provide advanced warning of disk degradation and failure.

DUA (Disk Usage Analyzer) is a tool to conveniently learn about the usage of disk space of a given directory. It's parallel by default and will max out your SSD, providing relevant information as fast as possible. Optionally delete superfluous data, and do so more quickly than rm.

Perkeep is a set of open source formats, protocols, and software for modeling, storing, searching, sharing and synchronizing data. It can be easily accessed via a phone, browser or FUSE filesystem.

duf is a Disk Usage/Free Utility for Linux, BSD, macOS & Windows.

Dirstat-rs is a fast, cross-platform disk usage CLI, similar to Windirstat.

Dutree is a tool to analyze file system usage written in Rust.

Shufflecake is a tool for Linux that allows to create multiple hidden volumes on a storage device in such a way that it is very difficult, even under forensic inspection, to prove the existence of such volumes.

btdu is a sampling disk usage profiler for btrfs.

Btrfs maintenance toolbox is a set of scripts supplementing the btrfs filesystem and aims to automate a few maintenance tasks. This means the scrub, balance, trim or defragmentation.

File systems

Back to the Top

FSArchiver is a system tool that allows you to save the contents of a file system to a compressed archive file. The file system can be restored on a partition which has a different size and it can be restored on a different file system.

WekaFS is the world's fastest shared parallel file system and delivers unmatched performance at ANY scale while offering the same enterprise features and benefits of traditional storage. It meets all storage challenges, delivering 10x the performance of legacy network attached storage (NAS) systems and 3x the performance of local server storage.

GlusterFS is a free and open source scalable network filesystem. Gluster is a scalable network filesystem. Using common off-the-shelf hardware, you can create large, distributed storage solutions for media streaming, data analysis, and other data- and bandwidth-intensive tasks.

Ceph is a software-defined storage solution designed to address the object, block, and file storage needs of data centers adopting open source as the new norm for high-growth block storage, object stores and data lakes. Ceph provides enterprise scalable storage while keeping CAPEX and OPEX costs in line with underlying bulk commodity disk prices.

ZFS is an enterprise-ready open source file system and volume manager with unprecedented flexibility and an uncompromising commitment to data integrity.

ZFSBootMenu is a Linux bootloader that attempts to provide an experience similar to the FreeBSD bootloader. It takes advantage of ZFS features, it allows a user to have multiple “boot environments” (with different distros, for example), manipulate snapshots before booting, and even bootstrap a system installation via zfs recv.

OpenZFS is an open-source storage platform. It includes the functionality of both traditional file systems and volume manager. It has many advanced features including:

Protection against data corruption.
Integrity checking for both data and metadata.
Continuous integrity verification and automatic "self-healing" repair.

Btrfs is a modern copy on write (CoW) filesystem for Linux aimed at implementing advanced features while also focusing on fault tolerance, repair and easy administration. Its main features and benefits are:

Snapshots which do not make the full copy of files
RAID - support for software-based RAID 0, RAID 1, RAID 10
Self-healing - checksums for data and metadata, automatic detection of silent data corruptions

Composefs is a native Linux file system designed to help sharing filesystem contents, as well as ensuring said content is not modified. The initial target usecase are container images and ostree commits.

MergerFS is a union filesystem geared towards simplifying storage and management of files across numerous commodity storage devices. It is similar to mhddfs, unionfs, and aufs.

MergerFS Features

Configurable behaviors / file placement
Ability to add or remove filesystems at will
Resistance to individual filesystem failure
Support for extended attributes (xattrs)
Support for file attributes (chattr)
Runtime configurable (via xattrs)
Works with heterogeneous filesystem types
Moving of file when filesystem runs out of space while writing
Ignore read-only filesystems when creating files
Turn read-only files into symlinks to underlying file
Hard link copy-on-write / CoW
Support for POSIX ACLs

Proxmox Cluster File System (PMXCFS) is a File System used to transparently distribute the cluster configuration to all cluster nodes.

UnionFS is a filesystem service for Linux, FreeBSD and NetBSD which implements a union mount for other file systems. It allows files and directories of separate file systems, known as branches, to be transparently overlaid, forming a single coherent file system.

OverlayFS is a modern union filesystem that is similar to AUFS, but faster and with a simpler implementation. It's typically used on systems running on embed devices, like OpenWRT, where is useful to preserve a basic set of configurations and at the same time allowing the user to perform modifications.

Bcachefs is an advanced new filesystem for Linux, with an emphasis on reliability and robustness and the complete set of features one would expect from a modern filesystem. Scalability has been tested to 50+ TB, will eventually scale far higher.

Squashfs is a compressed read-only filesystem for Linux. It uses zlib, lz4, lzo, or xz compression to compress files, inodes and directories. Inodes in the system are very small and all blocks are packed to minimize data overhead.

SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.

CubeFS is a cloud native distributed storage platform. It's commonly used as the storage infrastructure for online applications, database or data processing services and machine learning jobs orchestrated by Kubernetes.

Apple File System (APFS) is the default file system for Mac computers using macOS 10.13 or later, features strong encryption, space sharing, snapshots, fast directory sizing, and improved file system fundamentals.

NTFS(New Technology File System) is the primary file system for recent versions of Windows and Windows Server—provides a full set of features including security descriptors, encryption, disk quotas, and rich metadata, and can be used with Cluster Shared Volumes (CSV) to provide continuously available volumes that can be accessed simultaneously from multiple nodes of a failover cluster.

exFAT(Extended File Allocation Table ) is the file system that was the successor to FAT32 in the FAT family of file systems. It was optimized for flash memory such as USB flash drives and SD cards.

Books

Back to the Top

Self-hosted Cookbook is a cookbook, for docker-compose based recipes, for self-hosted applications and services.
Geek's Cookbook is a collection of guides for establishing your own highly-available "private cloud" and using it to run self-hosted services such as GitLab, Plex, NextCloud, etc.
Database Books(PDFs)

Podcasts

Back to the Top

Self-Hosted Podcast is a chat show between Chris and Alex two long-time "self-hosters" who share their lessons and take you along for the journey as they learn new ones.
Self-Hosted SRE(Site Reality Engineer) Podcast is a feed to say thank you to our Self-Hosted Site Reality Engineers!
Home Assistant Podcast is a biweekly podcast with the latest news and interesting guests.

WireGuard

Back to the Top

What is WireGuard?

Back to the Top

WireGuard® is a straight-forward, fast and modern VPN that utilizes state-of-the-art cryptography. It aims to be faster, simpler, leaner, and more useful than IPsec while avoiding the massive headache. WireGuard is designed as a general-purpose VPN for running on embedded interfaces and super computers alike, fit for many circumstances. Initially released for the Linux kernel, it is now cross-platform (Windows, macOS, BSD, iOS, Android) and widely deployable.

What is Tailscale?

Back to the Top

How NAT Traversal works on a Home router. Credit: Tailscale.

Headscale is an open source, self-hosted implementation of the Tailscale coordination server.

What is Netmaker?

Back to the Top

Netmaker is a tool that enables you to create relays, gateways, full VPN meshes, and even zero trust networks. It's fully configurable to let you maximize the power of Wireguard.

NetMaker Architecture. Credit: Netmaker.

WireGuard Tools

Back to the Top

Wiretrustee is a WireGuard®-based mesh network that connects your devices into a single private network.

Wireguard Manager is a tool that enables you to build your own vpn under a minute.

Headscale is an open source, self-hosted implementation of the Tailscale coordination server.

Firezone is a self-hosted WireGuard®-based VPN server and Linux firewall.

NetBird is an open-source VPN management platform built on top of WireGuard® making it easy to create secure private networks for your organization or home.

Mistborn is a secure platform for easily standing up and managing your own cloud services: including firewall, ad-blocking, and multi-factor WireGuard VPN access.

Mistborn CLI is a Command-line interface for Mistborn.

BoringTun is an implementation of the WireGuard® protocol designed for portability and speed. It's successfully deployed on millions of iOS and Android consumer devices as well as thousands of Cloudflare Linux servers.

PiVPN is the simplest VPN installer, designed for Raspberry Pi.

Algo VPN is a set of Ansible scripts that simplify the setup of a personal WireGuard and IPsec VPN. It uses the most secure defaults available and works with common cloud providers.

Pro Custodibus is a tool for managing WireGuard with a variety of business VPN (Virtual Private Network) use cases, such as site-to-site connectivity, secure remote access from anywhere, secure access to the cloud (Amazon Web Services, Google Cloud Platform, Microsoft Azure, etc), and more.

Drago is a flexible configuration manager for WireGuard designed to make it simple to configure secure network overlays spanning heterogeneous nodes distributed across different clouds and physical locations. Drago is in active development, and we welcome contributions from the open-source community.

Netmaker is a tool that helps connect any computers together over a secure, fast, private network, and manage multiple networks from a central server.

Kilo is a multi-cloud network overlay built on WireGuard and designed for Kubernetes. Kilo connects nodes in a cluster by providing an encrypted layer 3 network that can span across data centers and public clouds. The Pod network created by Kilo is always fully connected, even when the nodes are in different networks or behind NAT. By allowing pools of nodes in different locations to communicate securely, Kilo enables the operation of multi-cloud clusters. Kilo's design allows clients to VPN to a cluster in order to securely access services running on the cluster.

Subspace is a simple WireGuard VPN server GUI.

WG UI is a basic, self-contained management service for WireGuard with a self-serve web UI.

WireHole is a combination of WireGuard, PiHole, and Unbound in a docker-compose project with the intent of enabling users to quickly and easily create and deploy a personally managed full or split-tunnel WireGuard VPN with ad blocking capabilities (via Pihole), and DNS caching with additional privacy options (via Unbound).

Gluetun is a lightwieght VPN client in a thin Docker container for multiple VPN providers, written in Go, and uses OpenVPN or Wireguard, DNS over TLS, with a few proxy servers built-in.

Ethr is a cross platform network performance measurement tool written in golang. The goal of this project is to provide a native tool for comprehensive network performance measurements of bandwidth, connections/s, packets/s, latency, loss & jitter, across multiple protocols such as TCP, UDP, HTTP, HTTPS, and across multiple platforms such as Windows, Linux and other Unix systems.

Setting up WireGuard with PiVPN

Back to the Top

Installing PiVPN:

sudo apt install curl -y

curl -L https://install.pivpn.io | bash

Setting up WireGuard on Unraid

Back to the Top

Select Apps, then search for WireGuard and install Wireguard-Easy.

VPN manager

Almost all of the settings can stay as default, however, there are a few that we will modify.

Set the WG_HOST variable to be the IP address of your Unraid server.
If you’d like to modify the WireGuard port (51820), you can do that here.
Change the default Web GUI password.

Setting up WireGuard on pfSense

Back to the Top

When looking at how to set up WireGuard on pfSense, the first thing that we need to do is install the package. Follow the instructions below to install the WireGuard package on pfSense.

Open the Package Manager and search for WireGuard, then Install the latest version of the package.

After the package has installed, select VPN then WireGuard and under the Tunnels section, select Add Tunnel.
In the Tunnel Configuration, set the Description as WireGuard, the Listen Port as 51820, then Generate private and public keys.
Copy the Public Key. We will need this for our client configuration.
Create the tunnel, then select Settings, and ensure that Enable WireGuard is selected. Then Save and Apply.

Setting up WireGuard on OpenWRT

Back to the Top

Quick Links:

In your router’s webUI, navigate to System - Software, click Update lists:

In the Filter field, type WireGuard, locate and install the wireguard, wireguard-tools, kmod-wireguard, and luci-app-wireguard packages. Note: The wireguard package is included in version 22.02.

Generate WireGuard keypair

SSH into your router as ‘root’ (OpenWrt Wiki):

ssh root@192.168.1.1

Generate WireGuard keys:

wg genkey | tee privatekey | wg pubkey > publickey

chmod 600 privatekey

Note your Private & Public keys, you will need them later:

cat privatekey

cat publickey

Creating an Interface

Navigate to Network - Interface,

Click the Add new interface... button and enter the following configuration:

Name - give it any name
Protocol - WireGuard VPN

Create interface

In the General Settings tab:

Bring up on boot - Checked
Private Key - copy and paste the generated previously Private key
IP Address - enter the WireGuard IP Address obtained in the Client Area ending with /32, e.g. 172.27.124.169/32

Add a Firewall zone

Navigate to Network - Firewall

Click the Add button and enter the following configuration:

Name - Give it any name
Input - Reject
Output - Accept
Forward - Reject
Masquerading - Checked
MSS clamping - Checked
Covered networks - select the previously created VPN tunnel interface
Allow forward to destination zones - Unspecified
Allow forward from source zones - lan

DNS

Navigate to Network - Interfaces

Click on the Edit button next to the WAN interface

In the Advanced Settings tab, uncheck the Use DNS servers advertised by peer and specify one of the following DNS servers in the Use custom DNS servers field:

172.16.0.1 = regular DNS with no blocking
10.0.254.2 = standard AntiTracker to block advertising and malware domains
10.0.254.3 = Hardcore Mode AntiTracker to also block Google and Facebook domains

Click the Save button.

Last Steps

A device reboot is not required, though it may be useful to confirm that everything behaves as expected.
Run a leak test at https://www.dnsleaktest.com via one of the internal network clients attached to your OpenWRT router.

Setting up WireGuard on Home Assistant

Back to the Top

Install Wireguard Add-on in Home Assistant

Next, open up Home Assistant. Go to Supervisor > Add-on store, and search for WireGuard.
Click the WireGuard addon, and the click Install.

Configure Wireguard Settings

After installing WireGuard, do not start it yet. We need to configure a few options first.

Click the Configuration tab at the very top.
There are two blocks of code here: server and peers. The server section is the WireGuard server info, and the peers section is where you’d add new devices that will connect to your VPN.

Server Configuration

Host: add the subdomain you just created. (vpn.mydomain.com)
Addresses: If your internal network is using the 192.168.x.x or 10.x.x.x range, you can leave the default IP addresses WireGuard has provided. (see note above)
DNS: Set to your router’s internal IP address (Open CMD > ipconfig /all > Under DNS servers) If you have Adguard or PiHole installed, you can use the IP address of those instead. This will allow you to block ads even when connected to the WireGuard VPN.

Peers Configuration

This is where you’ll create WireGuard configuration files for each of the devices you want to connect to WireGuard with. For this example, I’m using my phone and leaving allowed_ips and client_allowed_ips as is. If you adding multiple devices, then you’ll need to copy the entire block of code starting at name, give it a different name, and add the next available IP address (For example: 172.27.66.4)

Click Save once finished.

Then, go back to the Info tab and click Start.

Port Forward

The next step is to forward port 51820 from your Home Assistant server through your router. Unfortunately, there are so many different types of routers, each with different steps to port forward. The important thing to note is that you’ll be port forwarding 51820(wireguard port) from the internal IP of your Home Assistant instance (for example: 192.168.68.24) and choosing the UDP protocol only.

Download Wireguard app on mobile device

Download the WireGuard app from the Apple App Store or Google Play Store. You will need it for the next step.

If all goes well, you can click into the new tunnel connection from within the app. If you see data flowing under the Transfer section, that means you are good to go.

Improving Security

Once you have everything setup and working correctly, you should read through the WireGuard Addon docs to setup up allowed_ips and client_allowed_ips to further secure your VPN instance. There’s also some other helpful options you can configure such as log level, but these are all optional.

Nextcloud

Back to the Top

Nexcloud login screen

Nexcloud Hub

Nextcloud AIO (All In One) is a tool that provides easy deployment and maintenance with most features included in this one Nextcloud instance.

Features it includes:

Nextcloud
Nextcloud Office
High performance backend for Nextcloud Files
High performance backend for Nextcloud Talk
Backup solution (based on BorgBackup)
Imaginary
ClamAV
Fulltextsearch

Nextcloud Desktop Client is a tool to synchronize files from Nextcloud Server with your computer.

Nextcloud Deck is a kanban style organization tool aimed at personal planning and project organization for teams integrated with Nextcloud.

Nextcloud Home is a tool that allows you store your documents, calendar, contacts and photos on your server at home, at one of at one Nextcloud's providers or in a data center you trust.

Nextcloud Enterprise is a service that gives professional organizations software optimized and tested for mission critical environments.

Nextcloud Outlook Integration is a tool that automatically upload files to replace large attachments or integrate Calendars and Contacts in Microsoft Outlook.

LibreSign is a Libre digital signature app for Nextcloud.

Raspberry Pi

Back to the Top

Models of Raspberry Pi boards

Back to the Top

Raspberry Pi 4 Model B

Check out the Raspberry Pi 4

Raspberry Pi 4 Model B Hardware Specifications

Broadcom BCM2711, Quad core Cortex-A72 (ARM v8) 64-bit SoC @ 1.5GHz
2GB, 4GB or 8GB LPDDR4-3200 SDRAM (depending on model)
2.4 GHz and 5.0 GHz IEEE 802.11ac wireless
Bluetooth 5.0, BLE
Gigabit Ethernet
2 USB 3.0 ports; 2 USB 2.0 ports.
Raspberry Pi standard 40 pin GPIO header (fully backwards compatible with previous Pi boards)
2 × micro-HDMI ports (up to 4kp60 supported)
OpenGL ES 3.0 graphics

Raspberry Pi 400 Personal Computer Kit

Check out the Raspberry Pi 400 Personal Computer Kit

Raspberry Pi 400 Hardware Specifications

Broadcom BCM2711, Quad core Cortex-A72 (ARM v8) 64-bit SoC @ 1.8GHz
4GB LPDDR4-3200 SDRAM
2.4 GHz and 5.0 GHz IEEE 802.11ac wireless
Bluetooth 5.0, BLE
Gigabit Ethernet
2 USB 3.0 ports; 2 USB 2.0 ports.
Raspberry Pi standard 40 pin GPIO header
2 × micro-HDMI ports (up to 4kp60 supported)
OpenGL ES 3.0 graphics

Raspberry Pi Pico Microcontroller

Check out the Raspberry Pi Pico

Raspberry Pi Pico Hardware Specifications

RP2040 microcontroller chip designed by Raspberry Pi in the UK
Dual-core Arm Cortex-M0+ processor, flexible clock running up to 133 MHz
264KB on-chip SRAM
2MB on-board QSPI Flash
26 multifunction GPIO pins, including 3 analogue inputs
2 × UART, 2 × SPI controllers, 2 × I2C controllers, 16 × PWM channels
1 × USB 1.1 controller and PHY, with host and device support
8 × Programmable I/O (PIO) state machines for custom peripheral support
Castellated module allows soldering direct to carrier boards
Drag-and-drop programming using mass storage over USB
Low-power sleep and dormant modes
Accurate on-chip clock
Temperature sensor
Accelerated integer and floating-point libraries on-chip

Raspberry Pi OS. The default Operating System for every Raspberry Pi device

Check out Raspberry Pi OS

Raspberry Pi Learning Resources

Back to the Top

Raspberry Pi is an ARM powered single board computer(SBC) that is the size of a credit card and costs around $35.

Raspberry Pi Foundation is a UK-based charity that works to put the power of computing and digital making into the hands of people all over the world.

Microsecond accurate NTP with a Raspberry Pi and PPS GPS

Getting Started with Raspberry Pi Projects

Online learning for the Raspberry Pi

Raspberry Pi Training Program

Raspberry Pi Online Courses on Udemy

Raspberry Pi Online Courses on Coursera

The Raspberry Pi Platform and Python Programming course on Coursera

Learning Raspberry Pi with Online Courses on edX

Raspberry Pi Online Training Courses on LinkedIn Learning

Getting Started with Raspberry Pi course on FutureLearn

Home Assistant on Raspberry Pi

PiSwitch: Build your own Nintendo Switch-style console

Raspberry Pi Operating Systems

Back to the Top

Raspberry Pi OS

Hass.io(Home Assistant OS)

OmniROM(Android 11) based on ASOP

Manjaro Linux ARM

Arch Linux ARM

Ubuntu MATE for Raspberry Pi

Ubuntu Desktop for Raspberry Pi

Ubuntu Core on a Raspberry Pi

Ubuntu Server for ARM

Fedora ARM

Kali Linux for the Raspberry Pi

Raspberry Pi Tools

Back to the Top

Raspberry Pi Imager is the quick and easy way to install Raspberry Pi OS and other operating systems to a microSD card, ready to use with your Raspberry Pi.

Raspberry Pi Locator is a website to track Raspberry Pi 4 model B, Compute Module 4, Pi Zero 2 W, and Pico availability across multiple retailers in different countries.

Raspberry Pi Network Install (Beta) is a feature can be used to start the Raspberry Pi Imager application directly on a Raspberry Pi 4, or a Raspberry Pi 400, by downloading it from the internet using an Ethernet cable. The Raspberry Pi Imager application, which will run in memory on your Raspberry Pi, can then be used to flash the operating system onto a blank SD Card or USB disk, just like normal.

Raspberry Pi Bootloader is a feature, which is now available in beta, that utilize an EEPROM(Electrically Erasable Programmable Read-Only Memory) to store the system’s bootloader. This EEPROM is persistent storage that is located on the Pi’s mainboard. The advantage of using the EEPROM instead is that the Raspberry Pi 4 can perform tasks without needing any storage to be attached.

Etcher is an open source, cross-platform software that makes it easy to flash operating system images to a microSD card or USB device.

Gladys Assistant is a privacy-first, open-source home assistant and runs great on Raspberry Pi.

Kodi for Raspberry Pi is a free and open source media player application developed by the XBMC/Kodi Foundation.

PiKVM is a very simple and fully functional Raspberry Pi-based KVM over IP.

PiShrink is a bash script that automatically shrink a pi image that will then resize to the max size of the SD card on boot.

RPiPlay is an open-source implementation of an AirPlay mirroring server for the Raspberry Pi that supports iOS 9 and later.

Gpiozero is a simple interface to GPIO(General-Purpose Input/Output) devices with the Raspberry Pi.

Balena Sound is a single or multi-room streamer for an existing audio device using a Raspberry Pi! It supports Bluetooth, Airplay and Spotify Connect.

OpenBalena is a platform to deploy and manage connected devices.

Home Assistant

Back to the Top

Home Assistant is a container-based system for managing your Home Assistant Core installation and related applications. The system is controlled via Home Assistant which communicates with the Supervisor. The Supervisor provides an API to manage the installation. This includes changing network settings or installing and updating software.

Quick Links

Home Assistant Frontend is a frontend for Home Assistant.

Tools to write the HA image to your boot media(microSD card or USB device)

Raspberry Pi Imager is the quick and easy way to install Raspberry Pi OS and other operating systems to a microSD card, ready to use with your Raspberry Pi.

Etcher is an open source, cross-platform software that makes it easy to flash operating system images to a microSD card or USB device.

Home Assistant integrations

Home Assistant integrations. Credit: Home Assistant

ESPHome is a system to control your ESP8266/ESP32 by simple yet powerful configuration files and control them remotely through Home Automation systems.

Amazon Alexa is a smart virtual assistant software to manage Alexa-enabled devices, control music playback, view shopping lists on the go, keep track of upcoming reminders, check on active timers and much more.

Google Assistant is a smart virtual assistant software on mobile and home automation devices.

Apple HomeKit is a software framework that enables your app to coordinate and control home automation accessories from multiple vendors to present a coherent, user-focused interface. Using HomeKit, your app can: Discover HomeKit-compatible automation accessories and add them to a persistent, cross-device home configuration database.

Samsung SmartThings is a sofwtare frmaeowrk that you can connect, monitor and control multiple smart home devices quicker and easier. Connect your Samsung smart TVs, smart appliances, smart speakers and brands like Ring, Nest and Philips Hue all from one app.

Ecobee is a home automation company in Canada that makes thermostats for residential and commercial use.

Philips Hue is a smart lighting system. The smart lights, Hue Bridge, and smart controls will forever change the way you experience light.

Sonos is the wireless home sound system that fills as many rooms as you want with great-sounding music, movies, and TV.

openHAB is a cross-platform software with the aim to integrate all kinds of Smart Home technologies, devices, etc.

Z-Wave is the leading wireless communications protocol behind many of the secure, trusted brands that are working to make everyone's home smarter and safer.

Zwavejs2Mqtt is a fully configurable Zwave to MQTT Gateway and Control Panel Web UI.

Z-Wave JS Server is a small server wrapper around Z-Wave JS to access it via a WebSocket.

Z-Wave JS Config DB Browser is the official device configuration reference to find out if your device is supported. Currently supports 387 brands, spanning at least 2075 device configurations.

Homebridge

Back to the Top

Homebridge is a software frameowrk that allows you to integrate with smart home devices that do not natively support HomeKit. There are over 2,000 Homebridge plugins supporting thousands of different smart accessories.

Tools to write the Homebridge image to your boot media(microSD card or USB device)

Raspberry Pi Imager is the quick and easy way to install Raspberry Pi OS and other operating systems to a microSD card, ready to use with your Raspberry Pi.

Etcher is an open source, cross-platform software that makes it easy to flash operating system images to a microSD card or USB device.

Homebridge UI is a tool that provides an easy to use interface to manage your Homebridge plugins, configuration and accessories.

Install and configure Homebridge plugins.
Monitor your Homebridge server via a fully customisable widget-based dashboard.
View and control Homebridge accessories.
Backup and Restore your Homebridge instance.

Homebridge UI

ESPHome

Back to the Top

ESPHome is a system to control your ESP8266/ESP32 by simple yet powerful configuration files and control them remotely through Home Automation systems.

Quick Links

Install ESPHome using Home Assistant

In Home Assistant go to:

Configuration > Add-ons, Backups & Supervisor > Add-on Store (button in the lower right corner) or click on the My Home Assistant Link below:

Open your Home Assistant instance and show the Supervisor add-on store.

Next, search for ESPHome, click on the result and then click on the Install button.

When the installation is finished, the Install button will be replaced with Start button – click on it to start the ESPHome add-on.

Wait a few seconds for the ESPHome to start and then click on the Open Web UI button.

Install ESPHome using Docker

First thing is to pull the ESPHome Docker image from Docker Hub (Online).

docker pull esphome/esphome
Then, start the ESPHome wizard. This wizard will ask you about your device type, your device name, your WiFi credentials and finally will generate a yaml file containing all of the configurations for you.

docker run --rm -v "${PWD}":/config -it esphome/esphome wizard stl.yaml

Now, connect your ESP device to the device where Docker is running (either using an USB cable or Serial-To-USB adapter) and if you are on Linux type the following command :

dmesg | grep ttyUSB

Put your device in programming mode (if needed) and execute the next command to install the ESPHome on the device connected to the /dev/ttyUSB1 using the configuration stored in stl.yaml file

docker run --rm -v "${PWD}":/config --device=/dev/ttyUSB1 -it esphome/esphome run stl.yaml

Install ESPHome using Python

If you are on macOS or Linux check if Python 3.8 or later is installed by executing the command.

python3 --version

If you are on macOS, you need to install wheel and esphome packages by using the following command.

pip3 install wheel esphome

If you are on Linux, you have to install esphome package by using the following command.

pip3 install --user esphome

If you are on macOS or Linux you can start the ESPHome wizard using the following command.

esphome wizard stl-python.yaml

Finally, connect your ESP device to your Computer (using USB cable or Serial-To-usb adapter) and put it in programming mode (if needed). Then, Install ESPHome using the configuration in the stl-python.yaml file.

esphome run stl-python.yaml

Turning Raspberry Pi into a Router

Back to the Top

Software

OpenWrt Project is a Linux operating system targeting embedded devices. Instead of trying to create a single, static firmware, OpenWrt provides a fully writable filesystem with package management. It's primarily used on embedded devices to route network traffic.

OpenWrt Wiki - Raspberry Pi setup

Download the appropriate OpenWrt image for your Raspberry PI by going to the link above.

Tools to write the Operating System (OS) image to your boot media(microSD card)

Raspberry Pi Imager is the quick and easy way to install Raspberry Pi OS and other operating systems to a microSD card, ready to use with your Raspberry Pi.

Hardware

Raspberry Pi Router Board for CM4 module (Cost: $55 USD) is an expansion board based on the Raspberry Pi Compute Module 4. It brings Raspberry Pi CM4 two full-speed gigabit network ports and offers better performance, lower CPU usage, and higher stability for a long time work compared with a USB network card. It's compatible with Raspberry Pi OS, Ubuntu Server and other Raspberry Pi systems.

Raspberry Pi Router Board for CM4 module

Technical Specs:

Compatible Module: Raspberry Pi Compute Module 4 series.
BCM2711 4 core @ 1.5GHz Cortex-A72.
Support standard Raspberry Pi HAT interface.
Support POE HAT to supply power to the board.
Support POE HAT for external power supply.
Full-speed dual gigabit network interface.
Master-slave dual USB2.0 interface.
Micro SD card slot, used to support non-eMMC version of CM4.
Standard HDMI video output interface.
0.91 inch IIC OLED display.
5V DC fan interface(Support controlling via PWM signal).
Ethernet: high-performance Gigabit ethernet controller RTL8111E chip, JXD 2111x G2406s chip as isolation transformer.
- Port0: Compute Module 4 Built-In.
- Port1: PCI Express 1000BASE-T NIC.
GPIO: 40-Pin GPIO compatible with Raspberry Pi.

Setting Watchdog Timer (WDT) on Raspberry Pi

Back to the Top

Watchdog Timer (WDT) is a timer that monitors microcontroller (MCU) programs to see if they are out of control or have stopped operating.

Installing and enabling WDT service

To enable watchdog you have to change the boot parameters by adding dtparam=watchdog=on in /boot/config.txt using a text editor such as nano, vim, gedit, etc.. Also, install watchdog package and enable it to start at startup. Also, make sure to restart your Raspberry Pi for these settings to take effect.

pi@raspberrypi:~ $ sudo apt install watchdog

pi@raspberrypi:~ $sudo systemctl enable watchdog

Configure WDT service

Configuration file for watchdog can be found in /etc/watchdog.conf.

max-load-1 = 24
watchdog-device = /dev/watchdog
realtime = yes
priority = 1

To start the WTD service:

pi@raspberrypi:~ $ sudo systemctl start watchdog

Check watchdog status:

pi@raspberrypi:~ $ sudo systemctl status watchdog

To stop the service:

pi@raspberrypi:~ $ sudo systemctl stop watchdog

Raspberry Pi Upgrades

Back to the Top

Raspberry Pi Cases from Pi-Shop US

Raspberry Pi Cases from The Pi Hut

X825 expansion board provides a complete storage solution for newest Raspberry Pi 4 Model B, it supports up to 4TB 2.5-inch SATA hard disk drives (HDD) / solid-state drive (SSD).

Sabrent M.2 SSD [NGFF] to USB 3.0 / SATA III 2.5-Inch Aluminum Enclosure Adapter

Samsung 970 EVO 250GB - NVMe PCIe M.2 2280 SSD

Western Digital 1TB WD Blue SN550 NVMe Internal SSD

SAMSUNG T5 Portable SSD

Samsung SSD 860 EVO 250GB mSATA Internal SSD

Samsung 850 EVO 120GB SSD mSATA

Grafana

Back to the Top

Grafana Learning Resources

Grafana is an analytics platform that enables you to query and visualize data, then create and share dashboards based on your visualizations. Easily visualize metrics, logs, and traces from multiple sources such as Prometheus, Loki, Elasticsearch, InfluxDB, Postgres, Fluentd, Fluentbit, Logstash and many more.

Getting Started with Grafana

Grafana Community

Grafana Professional Services Training | Grafana Labs

Grafana Pro Training AWS | Grafana Labs

Grafana Tutorials

Top Grafana Courses on Udemy

Grafana Online Training Courses | LinkedIn Learning

Grafana Training Courses - NobleProg

Setting Up Grafana to Visualize Our Metrics Course on Coursera

Grafana Tools

Grafana Cloud is a composable observability platform, integrating metrics, traces and logs with Grafana. Leverage the best open source observability software – including Prometheus, Loki, and Tempo – without the overhead of installing, maintaining, and scaling your observability stack.

Grafana Cloud Integrations. Source: Grafana

Grafana Enterprise is a service that includes features that provide better scalability, collaboration, operations, and governance in a self-managed environment.

Grafana Enterprise Stack. Source: Grafana

Grafana Tempo is an open source high-scale distributed tarcing backend. Tempo is cost-efficient, requiring only object storage to operate, and is deeply integrated with Grafana, Loki, and Prometheus.

Grafana MetricTank is a multi-tenant timeseries platform for Graphite developed by Grafana Labs. MetricTank provides high-availability(HA) and efficient long-term storage, retrieval, and processing for large-scale environments.

Grafana Tanka is a robust configuration utility for your Kubernetes cluster, powered by the Jsonnet language.

Grafana Loki is a horizontally-scalable, highly-available(HA), multi-tenant log aggregation system inspired by Prometheus.

Cortex is a project that lets users query metrics from many Prometheusservers in a single place, without any gaps in the grpahs due to server failture. Also, Cortex lets you store Prometheus metrics for long term capacity planning and performance analysis.

Graphite is an open source monitoring system.

Networking

Back to the Top

Networking Tools & Concepts

cURL is a computer software project providing a library and command-line tool for transferring data using various network protocols(HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, DICT, TELNET, LDAP LDAPS, MQTT, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP or SMTPS). cURL is also used in cars, television sets, routers, printers, audio equipment, mobile phones, tablets, settop boxes, media players and is the Internet transfer engine for thousands of software applications in over ten billion installations.

cURL Fuzzer is a quality assurance testing for the curl project.

DoH is a stand-alone application for DoH (DNS-over-HTTPS) name resolves and lookups.

Authelia is an open-source highly-available authentication server providing single sign-on capability and two-factor authentication to applications running behind NGINX.

nginx(engine x) is an HTTP and reverse proxy server, a mail proxy server, and a generic TCP/UDP proxy server, originally written by Igor Sysoev.

Wireshark is a very popular network protocol analyzer that is commonly used for network troubleshooting, analysis, and communications protocol development. Learn more about the other useful Wireshark Tools available.

HTTPie is a command-line HTTP client. Its goal is to make CLI interaction with web services as human-friendly as possible. HTTPie is designed for testing, debugging, and generally interacting with APIs & HTTP servers.

HTTPStat is a tool that visualizes curl statistics in a simple layout.

Wuzz is an interactive cli tool for HTTP inspection. It can be used to inspect/modify requests copied from the browser's network inspector with the "copy as cURL" feature.

Websocat is a ommand-line client for WebSockets, like netcat (or curl) for ws:// with advanced socat-like functions.

• Connection: In networking, a connection refers to pieces of related information that are transferred through a network. This generally infers that a connection is built before the data transfer (by following the procedures laid out in a protocol) and then is deconstructed at the at the end of the data transfer.

• Packet: A packet is, generally speaking, the most basic unit that is transferred over a network. When communicating over a network, packets are the envelopes that carry your data (in pieces) from one end point to the other.

Packets have a header portion that contains information about the packet including the source and destination, timestamps, network hops. The main portion of a packet contains the actual data being transferred. It is sometimes called the body or the payload.

• Network Interface: A network interface can refer to any kind of software interface to networking hardware. For instance, if you have two network cards in your computer, you can control and configure each network interface associated with them individually.

A network interface may be associated with a physical device, or it may be a representation of a virtual interface. The "loop-back" device, which is a virtual interface to the local machine, is an example of this.

• LAN: LAN stands for "local area network". It refers to a network or a portion of a network that is not publicly accessible to the greater internet. A home or office network is an example of a LAN.

• WAN: WAN stands for "wide area network". It means a network that is much more extensive than a LAN. While WAN is the relevant term to use to describe large, dispersed networks in general, it is usually meant to mean the internet, as a whole.

If an interface is connected to the WAN, it is generally assumed that it is reachable through the internet.

• Protocol: A protocol is a set of rules and standards that basically define a language that devices can use to communicate. There are a great number of protocols in use extensively in networking, and they are often implemented in different layers.

Some low level protocols are TCP, UDP, IP, and ICMP. Some familiar examples of application layer protocols, built on these lower protocols, are HTTP (for accessing web content), SSH, TLS/SSL, and FTP.

• Port: A port is an address on a single machine that can be tied to a specific piece of software. It is not a physical interface or location, but it allows your server to be able to communicate using more than one application.

• Firewall: A firewall is a program that decides whether traffic coming into a server or going out should be allowed. A firewall usually works by creating rules for which type of traffic is acceptable on which ports. Generally, firewalls block ports that are not used by a specific application on a server.

• NAT: Network address translation is a way to translate requests that are incoming into a routing server to the relevant devices or servers that it knows about in the LAN. This is usually implemented in physical LANs as a way to route requests through one IP address to the necessary backend servers.

• VPN: Virtual private network is a means of connecting separate LANs through the internet, while maintaining privacy. This is used as a means of connecting remote systems as if they were on a local network, often for security reasons.

Network Layers

While networking is often discussed in terms of topology in a horizontal way, between hosts, its implementation is layered in a vertical fashion throughout a computer or network. This means is that there are multiple technologies and protocols that are built on top of each other in order for communication to function more easily. Each successive, higher layer abstracts the raw data a little bit more, and makes it simpler to use for applications and users. It also allows you to leverage lower layers in new ways without having to invest the time and energy to develop the protocols and applications that handle those types of traffic.

As data is sent out of one machine, it begins at the top of the stack and filters downwards. At the lowest level, actual transmission to another machine takes place. At this point, the data travels back up through the layers of the other computer. Each layer has the ability to add its own "wrapper" around the data that it receives from the adjacent layer, which will help the layers that come after decide what to do with the data when it is passed off.

One method of talking about the different layers of network communication is the OSI model. OSI stands for Open Systems Interconnect.This model defines seven separate layers. The layers in this model are:

• Application: The application layer is the layer that the users and user-applications most often interact with. Network communication is discussed in terms of availability of resources, partners to communicate with, and data synchronization.

• Presentation: The presentation layer is responsible for mapping resources and creating context. It is used to translate lower level networking data into data that applications expect to see.

• Session: The session layer is a connection handler. It creates, maintains, and destroys connections between nodes in a persistent way.

• Transport: The transport layer is responsible for handing the layers above it a reliable connection. In this context, reliable refers to the ability to verify that a piece of data was received intact at the other end of the connection. This layer can resend information that has been dropped or corrupted and can acknowledge the receipt of data to remote computers.

• Network: The network layer is used to route data between different nodes on the network. It uses addresses to be able to tell which computer to send information to. This layer can also break apart larger messages into smaller chunks to be reassembled on the opposite end.

• Data Link: This layer is implemented as a method of establishing and maintaining reliable links between different nodes or devices on a network using existing physical connections.

• Physical: The physical layer is responsible for handling the actual physical devices that are used to make a connection. This layer involves the bare software that manages physical connections as well as the hardware itself (like Ethernet).

The TCP/IP model, more commonly known as the Internet protocol suite, is another layering model that is simpler and has been widely adopted.It defines the four separate layers, some of which overlap with the OSI model:

• Application: In this model, the application layer is responsible for creating and transmitting user data between applications. The applications can be on remote systems, and should appear to operate as if locally to the end user.

The communication takes place between peers network.

• Transport: The transport layer is responsible for communication between processes. This level of networking utilizes ports to address different services. It can build up unreliable or reliable connections depending on the type of protocol used.

• Internet: The internet layer is used to transport data from node to node in a network. This layer is aware of the endpoints of the connections, but does not worry about the actual connection needed to get from one place to another. IP addresses are defined in this layer as a way of reaching remote systems in an addressable manner.

• Link: The link layer implements the actual topology of the local network that allows the internet layer to present an addressable interface. It establishes connections between neighboring nodes to send data.

Interfaces

Interfaces are networking communication points for your computer. Each interface is associated with a physical or virtual networking device. Typically, your server will have one configurable network interface for each Ethernet or wireless internet card you have. In addition, it will define a virtual network interface called the "loopback" or localhost interface. This is used as an interface to connect applications and processes on a single computer to other applications and processes. You can see this referenced as the "lo" interface in many tools.

Network Protocols

Networking works by piggybacks on a number of different protocols on top of each other. In this way, one piece of data can be transmitted using multiple protocols encapsulated within one another.

Media Access Control(MAC) is a communications protocol that is used to distinguish specific devices. Each device is supposed to get a unique MAC address during the manufacturing process that differentiates it from every other device on the internet. Addressing hardware by the MAC address allows you to reference a device by a unique value even when the software on top may change the name for that specific device during operation. Media access control is one of the only protocols from the link layer that you are likely to interact with on a regular basis.

The IP protocol is one of the fundamental protocols that allow the internet to work. IP addresses are unique on each network and they allow machines to address each other across a network. It is implemented on the internet layer in the IP/TCP model. Networks can be linked together, but traffic must be routed when crossing network boundaries. This protocol assumes an unreliable network and multiple paths to the same destination that it can dynamically change between. There are a number of different implementations of the protocol. The most common implementation today is IPv4, although IPv6 is growing in popularity as an alternative due to the scarcity of IPv4 addresses available and improvements in the protocols capabilities.

ICMP: internet control message protocol is used to send messages between devices to indicate the availability or error conditions. These packets are used in a variety of network diagnostic tools, such as ping and traceroute. Usually ICMP packets are transmitted when a packet of a different kind meets some kind of a problem. Basically, they are used as a feedback mechanism for network communications.

TCP: Transmission control protocol is implemented in the transport layer of the IP/TCP model and is used to establish reliable connections. TCP is one of the protocols that encapsulates data into packets. It then transfers these to the remote end of the connection using the methods available on the lower layers. On the other end, it can check for errors, request certain pieces to be resent, and reassemble the information into one logical piece to send to the application layer. The protocol builds up a connection prior to data transfer using a system called a three-way handshake. This is a way for the two ends of the communication to acknowledge the request and agree upon a method of ensuring data reliability. After the data has been sent, the connection is torn down using a similar four-way handshake. TCP is the protocol of choice for many of the most popular uses for the internet, including WWW, FTP, SSH, and email. It is safe to say that the internet we know today would not be here without TCP.

UDP: User datagram protocol is a popular companion protocol to TCP and is also implemented in the transport layer. The fundamental difference between UDP and TCP is that UDP offers unreliable data transfer. It does not verify that data has been received on the other end of the connection. This might sound like a bad thing, and for many purposes, it is. However, it is also extremely important for some functions. It’s not required to wait for confirmation that the data was received and forced to resend data, UDP is much faster than TCP. It does not establish a connection with the remote host, it simply fires off the data to that host and doesn't care if it is accepted or not. Since UDP is a simple transaction, it is useful for simple communications like querying for network resources. It also doesn't maintain a state, which makes it great for transmitting data from one machine to many real-time clients. This makes it ideal for VOIP, games, and other applications that cannot afford delays.

HTTP: Hypertext transfer protocol is a protocol defined in the application layer that forms the basis for communication on the web. HTTP defines a number of functions that tell the remote system what you are requesting. For instance, GET, POST, and DELETE all interact with the requested data in a different way.

FTP: File transfer protocol is in the application layer and provides a way of transferring complete files from one host to another. It is inherently insecure, so it is not recommended for any externally facing network unless it is implemented as a public, download-only resource.

DNS: Domain name system is an application layer protocol used to provide a human-friendly naming mechanism for internet resources. It is what ties a domain name to an IP address and allows you to access sites by name in your browser.

SSH: Secure shell is an encrypted protocol implemented in the application layer that can be used to communicate with a remote server in a secure way. Many additional technologies are built around this protocol because of its end-to-end encryption and ubiquity. There are many other protocols that we haven't covered that are equally important. However, this should give you a good overview of some of the fundamental technologies that make the internet and networking possible.

REST(REpresentational State Transfer) is an architectural style for providing standards between computer systems on the web, making it easier for systems to communicate with each other.

JSON Web Token (JWT) is a compact URL-safe means of representing claims to be transferred between two parties. The claims in a JWT are encoded as a JSON object that is digitally signed using JSON Web Signature (JWS).

OAuth 2.0 is an open source authorization framework that enables applications to obtain limited access to user accounts on an HTTP service, such as Amazon, Google, Facebook, Microsoft, Twitter GitHub, and DigitalOcean. It works by delegating user authentication to the service that hosts the user account, and authorizing third-party applications to access the user account.

Docker

Back to the Top

Container Architecture. Source: Containerd.io

Docker Learning Resources

Docker Training Program

Docker Certified Associate (DCA) certification

Docker Documentation | Docker Documentation

The Docker Workshop

Docker Courses on Udemy

Docker Courses on Coursera

Docker Courses on edX

Docker Courses on Linkedin Learning

Docker Tools

Docker is an open platform for developing, shipping, and running applications. Docker enables you to separate your applications from your infrastructure so you can deliver software quickly working in collaboration with cloud, Linux, and Windows vendors, including Microsoft.

Docker Enterprise is a subscription including software, supported and certified container platform for CentOS, Red Hat Enterprise Linux (RHEL), Ubuntu, SUSE Linux Enterprise Server (SLES), Oracle Linux, and Windows Server 2016, as well as for cloud providers AWS and Azure. In November 2019 Docker's Enterprise Platform business was acquired by Mirantis.

Docker Desktop is an application for MacOS and Windows machines for the building and sharing of containerized applications and microservices. Docker Desktop delivers the speed, choice and security you need for designing and delivering containerized applications on your desktop. Docker Desktop includes Docker App, developer tools, Kubernetes and version synchronization to production Docker Engines.

Docker Hub is the world's largest library and community for container images Browse over 100,000 container images from software vendors, open-source projects, and the community.

Docker Compose is a tool that was developed to help define and share multi-container applications. With Docker Compose, you can create a YAML file to define the services and with a single command, can spin everything up or tear it all down.

Docker Swarm is a Docker-native clustering system swarm is a simple tool which controls a cluster of Docker hosts and exposes it as a single "virtual" host.

Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image. Using docker build users can create an automated build that executes several command-line instructions in succession.

Docker Containers is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another.

Docker Engine is a container runtime that runs on various Linux (CentOS, Debian, Fedora, Oracle Linux, RHEL, SUSE, and Ubuntu) and Windows Server operating systems. Docker creates simple tooling and a universal packaging approach that bundles up all application dependencies inside a container which is then run on Docker Engine.

Docker Images is a lightweight, standalone, executable package of software that includes everything needed to run an application: code, runtime, system tools, system libraries and settings. Images have intermediate layers that increase reusability, decrease disk usage, and speed up docker build by allowing each step to be cached. These intermediate layers are not shown by default. The SIZE is the cumulative space taken up by the image and all its parent images.

Docker Network is a that displays detailed information on one or more networks.

Docker Daemon is a service started by a system utility, not manually by a user. This makes it easier to automatically start Docker when the machine reboots. The command to start Docker depends on your operating system. Currently, it only runs on Linux because it depends on a number of Linux kernel features, but there are a few ways to run Docker on MacOS and Windows as well by configuring the operating system utilities.

Docker Storage is a driver controls how images and containers are stored and managed on your Docker host.

Kitematic is a simple application for managing Docker containers on Mac, Linux and Windows letting you control your app containers from a graphical user interface (GUI).

Open Container Initiative is an open governance structure for the express purpose of creating open industry standards around container formats and runtimes.

Buildah is a command line tool to build Open Container Initiative (OCI) images. It can be used with Docker, Podman, Kubernetes.

Kubernetes

Back to the Top

Kubernetes Learning Resources

Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications.

Getting Kubernetes Certifications

Getting started with Kubernetes on AWS

Kubernetes on Microsoft Azure

Intro to Azure Kubernetes Service

Azure Red Hat OpenShift

Getting started with Google Cloud

Getting started with Kubernetes on Red Hat

Getting started with Kubernetes on IBM

Red Hat OpenShift on IBM Cloud

Enable OpenShift Virtualization on Red Hat OpenShift

YAML basics in Kubernetes

Elastic Cloud on Kubernetes

Docker and Kubernetes

Running Apache Spark on Kubernetes

Kubernetes Across VMware vRealize Automation

VMware Tanzu Kubernetes Grid

All the Ways VMware Tanzu Works with AWS

VMware Tanzu Education

Using Ansible in a Cloud-Native Kubernetes Environment

Managing Kubernetes (K8s) objects with Ansible

Setting up a Kubernetes cluster using Vagrant and Ansible

Running MongoDB with Kubernetes

Kubernetes Fluentd

Understanding the new GitLab Kubernetes Agent

Intro Local Process with Kubernetes for Visual Studio 2019

Kubernetes Contributors

KubeAcademy from VMware

Kubernetes Tutorials from Pulumi

Kubernetes Playground by Katacoda

Scalable Microservices with Kubernetes course from Udacity

Kubernetes Tools, Frameworks, and Projects

Open Container Initiative is an open governance structure for the express purpose of creating open industry standards around container formats and runtimes.

Buildah is a command line tool to build Open Container Initiative (OCI) images. It can be used with Docker, Podman, Kubernetes.

Google Kubernetes Engine (GKE) is a managed, production-ready environment for running containerized applications.

Azure Kubernetes Service (AKS) is serverless Kubernetes, with a integrated continuous integration and continuous delivery (CI/CD) experience, and enterprise-grade security and governance. Unite your development and operations teams on a single platform to rapidly build, deliver, and scale applications with confidence.

Amazon EKS is a tool that runs Kubernetes control plane instances across multiple Availability Zones to ensure high availability.

AWS Controllers for Kubernetes (ACK) is a new tool that lets you directly manage AWS services from Kubernetes. ACK makes it simple to build scalable and highly-available Kubernetes applications that utilize AWS services.

Container Engine for Kubernetes (OKE) is an Oracle-managed container orchestration service that can reduce the time and cost to build modern cloud native applications. Unlike most other vendors, Oracle Cloud Infrastructure provides Container Engine for Kubernetes as a free service that runs on higher-performance, lower-cost compute.

Anthos is a modern application management platform that provides a consistent development and operations experience for cloud and on-premises environments.

Red Hat Openshift is a fully managed Kubernetes platform that provides a foundation for on-premises, hybrid, and multicloud deployments.

OKD is a community distribution of Kubernetes optimized for continuous application development and multi-tenant deployment. OKD adds developer and operations-centric tools on top of Kubernetes to enable rapid application development, easy deployment and scaling, and long-term lifecycle maintenance for small and large teams.

Odo is a fast, iterative, and straightforward CLI tool for developers who write, build, and deploy applications on Kubernetes and OpenShift.

Kata Operator is an operator to perform lifecycle management (install/upgrade/uninstall) of Kata Runtime on Openshift as well as Kubernetes cluster.

Thanos is a set of components that can be composed into a highly available metric system with unlimited storage capacity, which can be added seamlessly on top of existing Prometheus deployments.

OpenShift Hive is an operator which runs as a service on top of Kubernetes/OpenShift. The Hive service can be used to provision and perform initial configuration of OpenShift 4 clusters.

Rook is a tool that turns distributed storage systems into self-managing, self-scaling, self-healing storage services. It automates the tasks of a storage administrator: deployment, bootstrapping, configuration, provisioning, scaling, upgrading, migration, disaster recovery, monitoring, and resource management.

VMware Tanzu is a centralized management platform for consistently operating and securing your Kubernetes infrastructure and modern applications across multiple teams and private/public clouds.

Kubespray is a tool that combines Kubernetes and Ansible to easily install Kubernetes clusters that can be deployed on AWS, GCE, Azure, OpenStack, vSphere, Packet (bare metal), Oracle Cloud Infrastructure (Experimental), or Baremetal.

KubeInit provides Ansible playbooks and roles for the deployment and configuration of multiple Kubernetes distributions.

Rancher is a complete software stack for teams adopting containers. It addresses the operational and security challenges of managing multiple Kubernetes clusters, while providing DevOps teams with integrated tools for running containerized workloads.

K3s is a highly available, certified Kubernetes distribution designed for production workloads in unattended, resource-constrained, remote locations or inside IoT appliances.

Helm is a Kubernetes Package Manager tool that makes it easier to install and manage Kubernetes applications.

Knative is a Kubernetes-based platform to build, deploy, and manage modern serverless workloads. Knative takes care of the operational overhead details of networking, autoscaling (even to zero), and revision tracking.

KubeFlow is a tool dedicated to making deployments of machine learning (ML) workflows on Kubernetes simple, portable and scalable.

Etcd is a distributed key-value store that provides a reliable way to store data that needs to be accessed by a distributed system or cluster of machines. Etcd is used as the backend for service discovery and stores cluster state and configuration for Kubernetes.

OpenEBS is a Kubernetes-based tool to create stateful applications using Container Attached Storage.

Container Storage Interface (CSI) is an API that lets container orchestration platforms like Kubernetes seamlessly communicate with stored data via a plug-in.

MicroK8s is a tool that delivers the full Kubernetes experience. In a Fully containerized deployment with compressed over-the-air updates for ultra-reliable operations. It is supported on Linux, Windows, and MacOS.

Charmed Kubernetes is a well integrated, turn-key, conformant Kubernetes platform, optimized for your multi-cloud environments developed by Canonical.

Grafana Kubernetes App is a toll that allows you to monitor your Kubernetes cluster's performance. It includes 4 dashboards, Cluster, Node, Pod/Container and Deployment. It allows for the automatic deployment of the required Prometheus exporters and a default scrape config to use with your in cluster Prometheus deployment.

KubeEdge is an open source system for extending native containerized application orchestration capabilities to hosts at Edge.It is built upon kubernetes and provides fundamental infrastructure support for network, app. deployment and metadata synchronization between cloud and edge.

Lens is the most powerful IDE for people who need to deal with Kubernetes clusters on a daily basis. It has support for MacOS, Windows and Linux operating systems.

Flux CD is a tool that automatically ensures that the state of your Kubernetes cluster matches the configuration you've supplied in Git. It uses an operator in the cluster to trigger deployments inside Kubernetes, which means that you don't need a separate continuous delivery tool.

Platform9 Managed Kubernetes (PMK) is a Kubernetes as a service that ensures fully automated Day-2 operations with 99.9% SLA on any environment, whether in data-centers, public clouds, or at the edge.

Ansible

Back to the Top

Mac Development Ansible Playbook by Jeff Geerling

Ansible Learning Resources

Ansible is a simple IT automation engine that automates cloud provisioning, configuration management, application deployment, intra-service orchestration, and many other IT needs. It uses a very simple language (YAML, in the form of Ansible Playbooks) that allows you to describe your automation jobs in a way that approaches plain English. Anisble works on Linux (Red Hat EnterPrise Linux(RHEL) and Ubuntu) and Microsoft Windows.

Red Hat Training for Ansible

Top Ansible Courses Online from Udemy

Introduction to Ansible: The Fundamentals on Coursera

Learning Ansible Fundamentals on Pluralsight

Introducing Red Hat Ansible Automation Platform 2.1

Ansible Documentation

Ansible Galaxy User Guide

Ansible Use Cases

Ansible Integrations

Ansible Collections Overview

Working with playbooks

Ansible for DevOps Examples by Jeff Geerling

Getting Started: Writing Your First Playbook - Ansible

Working With Modules in Ansible

Ansible Best Practices: Roles & Modules

Working with command line tools for Ansible

Encrypting content with Ansible Vault

Using vault in playbooks with Ansible

Using Ansible With Azure

Configuring Ansible on an Azure VM

How to Use Ansible: An Ansible Cheat Sheet Guide from DigitalOcean

Intro to Ansible on Linode | Spatial Labs

Ansible DevOps Tools Integration

Ansible Automation Hub is the official location to discover and download supported collections, included as part of an Ansible Automation Platform subscription. These content collections contain modules, plugins, roles, and playbooks in a downloadable package.

Collections are a distribution format for Ansible content that can include playbooks, roles, modules, and plugins. As modules move from the core Ansible repository into collections, the module documentation will move to the collections pages.

Ansible Lint is a command-line tool for linting playbooks, roles and collections aimed towards any Ansible users. Its main goal is to promote proven practices, patterns and behaviors while avoiding common pitfalls that can easily lead to bugs or make code harder to maintain.

Ansible cmdb is a tool that takes the output of Ansible’s fact gathering and converts it into a static HTML overview page containing system configuration information.

Ansible Inventory Grapher visually displays inventory inheritance hierarchies and at what level a variable is defined in inventory.

Ansible Playbook Grapher is a command line tool to create a graph representing your Ansible playbook tasks and roles.

Ansible Shell is an interactive shell for Ansible with built-in tab completion for all the modules.

Ansible Silo is a self-contained Ansible environment by Docker.

Ansigenome is a command line tool designed to help you manage your Ansible roles.

ARA is a records Ansible playbook runs and makes the recorded data available and intuitive for users and systems by integrating with Ansible as a callback plugin.

Capistrano is a remote server automation tool. It supports the scripting and execution of arbitrary tasks, and includes a set of sane-default deployment workflows.

Fabric is a high level Python (2.7, 3.4+) library designed to execute shell commands remotely over SSH, yielding useful Python objects in return. It builds on top of Invoke (subprocess command execution and command-line features) and Paramiko (SSH protocol implementation), extending their APIs to complement one another and provide additional functionality.

ansible-role-wireguard is an Ansible role for installing WireGuard VPN. Supports Ubuntu, Debian, Archlinx, Fedora and CentOS Stream.

wireguard_cloud_gateway is an Ansible role for setting up Wireguard as a gateway VPN server for cloud networks.

Red Hat OpenShift is focused on security at every level of the container stack and throughout the application lifecycle. It includes long-term, enterprise support from one of the leading Kubernetes contributors and open source software companies.

OpenShift Hive is an operator which runs as a service on top of Kubernetes/OpenShift. The Hive service can be used to provision and perform initial configuration of OpenShift 4 clusters.

Databases

Back to the Top

SQL/NoSQL Learning Resources

SQL is a standard language for storing, manipulating and retrieving data in relational databases.

NoSQL is a database that is interchangeably referred to as "nonrelational, or "non-SQL" to highlight that the database can handle huge volumes of rapidly changing, unstructured data in different ways than a relational (SQL-based) database with rows and tables.

Transact-SQL(T-SQL) is a Microsoft extension of SQL with all of the tools and applications communicating to a SQL database by sending T-SQL commands.

Introduction to Transact-SQL

SQL Tutorial by W3Schools

Learn SQL Skills Online from Coursera

SQL Courses Online from Udemy

SQL Online Training Courses from LinkedIn Learning

Learn SQL For Free from Codecademy

GitLab's SQL Style Guide

OracleDB SQL Style Guide Basics

Tableau CRM: BI Software and Tools

Databases on AWS

Best Practices and Recommendations for SQL Server Clustering in AWS EC2.

Connecting from Google Kubernetes Engine to a Cloud SQL instance.

Educational Microsoft Azure SQL resources

MySQL Certifications

SQL vs. NoSQL Databases: What's the Difference?

What is NoSQL?

SQL/NoSQL Tools and Databases

Azure Data Studio is an open source data management tool that enables working with SQL Server, Azure SQL DB and SQL DW from Windows, macOS and Linux.

MySQL is a fully managed database service to deploy cloud-native applications using the world's most popular open source database.

Amazon DynamoDB is a key-value and document database that delivers single-digit millisecond performance at any scale. It is a fully managed, multiregion, multimaster, durable database with built-in security, backup and restore, and in-memory caching for internet-scale applications.

Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. It can run Hadoop, Jenkins, Spark, Aurora, and other frameworks on a dynamically shared pool of nodes.

Apache Spark is a unified analytics engine for big data processing, with built-in modules for streaming, SQL, machine learning and graph processing.

Logstash is a tool for managing events and logs. When used generically, the term encompasses a larger system of log collection, processing, storage and searching activities.

Extract, transform, and load (ETL) is a data pipeline used to collect data from various sources, transform the data according to business rules, and load it into a destination data store.

MongoDB is a document database meaning it stores data in JSON-like documents.

OracleDB is a powerful fully managed database helps developers manage business-critical data with the highest availability, reliability, and security.

MariaDB is an enterprise open source database solution for modern, mission-critical applications.

Atlas is an in-memory dimensional time series database.

Cosmos DB Profiler is a real-time visual debugger allowing a development team to gain valuable insight and perspective into their usage of Cosmos DB database. It identifies over a dozen suspicious behaviors from your application’s interaction with Cosmos DB.

DBeaver is an open source database tool for developers and database administrators. It offers supports for JDBC compliant databases such as MySQL, Oracle, IBM DB2, SQL Server, Firebird, SQLite, Sybase, Teradata, Firebird, Apache Hive, Phoenix, and Presto.

DbVisualizer is a SQL management tool that allows users to manage a wide range of databases such as Oracle, Sybase, SQL Server, MySQL, H3, and SQLite.

Telco 5G

Back to the Top

VMware Cloud First Approach. Source: VMware.

VMware Telco Cloud Automation Components. Source: VMware.

Telco Learning Resources

HPE(Hewlett Packard Enterprise) Telco Blueprints overview

Network Functions Virtualization Infrastructure (NFVI) by Cisco

Introduction to vCloud NFV Telco Edge from VMware

VMware Telco Cloud Automation(TCA) Architecture Overview

5G Telco Cloud from VMware

Maturing OpenStack Together To Solve Telco Needs from Red Hat

Red Hat telco ecosystem program

OpenStack for Telcos by Canonical

Open source NFV platform for 5G from Ubuntu

Understanding 5G Technology from Verizon

Verizon and Unity partner to enable 5G & MEC gaming and enterprise applications

Understanding 5G Technology from Intel

Understanding 5G Technology from Qualcomm

Telco Acceleration with Xilinx

VIMs on OSM Public Wiki

Amazon EC2 Overview and Networking Introduction for Telecom Companies

Citrix Certified Associate – Networking(CCA-N)

Citrix Certified Professional – Virtualization(CCP-V)

CCNP Routing and Switching

Certified Information Security Manager(CISM)

Wireshark Certified Network Analyst (WCNA)

Juniper Networks Certification Program Enterprise (JNCP)

Cloud Native Computing Foundation Training and Certification Program

Tools

Open Stack is an open source cloud platform, deployed as infrastructure-as-a-service (IaaS) to orchestrate data center operations on bare metal, private cloud hardware, public cloud resources, or both (hybrid/multi-cloud architecture). OpenStack includes advance use of virtualization & SDN for network traffic optimization to handle the core cloud-computing services of compute, networking, storage, identity, and image services.

StarlingX is a complete cloud infrastructure software stack for the edge used by the most demanding applications in industrial IOT, telecom, video delivery and other ultra-low latency use cases.

Airship is a collection of open source tools for automating cloud provisioning and management. Airship provides a declarative framework for defining and managing the life cycle of open infrastructure tools and the underlying hardware.

Physical Network Function(PNF) is a physical network node which has not undergone virtualization. Both PNFs and VNFs (Virtualized Network Functions) can be used to form an overall Network Service.

Open Source Security

Back to the Top

Open Source Security Foundation (OpenSSF) is a cross-industry collaboration that brings together leaders to improve the security of open source software by building a broader community, targeted initiatives, and best practices. The OpenSSF brings together open source security initiatives under one foundation to accelerate work through cross-industry support. Along with the Core Infrastructure Initiative and the Open Source Security Coalition, and will include new working groups that address vulnerability disclosures, security tooling and more.

Security Standards, Frameworks and Benchmarks

STIGs Benchmarks - Security Technical Implementation Guides

CIS Benchmarks - CIS Center for Internet Security

NIST - Current FIPS

ISO Standards Catalogue

Common Criteria for Information Technology Security Evaluation (CC) is an international standard (ISO / IEC 15408) for computer security. It allows an objective evaluation to validate that a particular product satisfies a defined set of security requirements.

ISO 22301 is the international standard that provides a best-practice framework for implementing an optimised BCMS (business continuity management system).

ISO27001 is the international standard that describes the requirements for an ISMS (information security management system). The framework is designed to help organizations manage their security practices in one place, consistently and cost-effectively.

ISO 27701 specifies the requirements for a PIMS (privacy information management system) based on the requirements of ISO 27001. It is extended by a set of privacy-specific requirements, control objectives and controls. Companies that have implemented ISO 27001 will be able to use ISO 27701 to extend their security efforts to cover privacy management.

EU GDPR (General Data Protection Regulation) is a privacy and data protection law that supersedes existing national data protection laws across the EU, bringing uniformity by introducing just one main data protection law for companies/organizations to comply with.

CCPA (California Consumer Privacy Act) is a data privacy law that took effect on January 1, 2020 in the State of California. It applies to businesses that collect California residents’ personal information, and its privacy requirements are similar to those of the EU’s GDPR (General Data Protection Regulation).

Payment Card Industry (PCI) Data Security Standards (DSS) is a global information security standard designed to prevent fraud through increased control of credit card data.

SOC 2 is an auditing procedure that ensures your service providers securely manage your data to protect the interests of your comapny/organization and the privacy of their clients.

NIST CSF is a voluntary framework primarily intended for critical infrastructure organizations to manage and mitigate cybersecurity risk based on existing best practice.

Security Tools

SELinux is a security enhancement to Linux which allows users and administrators more control over access control. Access can be constrained on such variables as which users and applications can access which resources. These resources may take the form of files. Standard Linux access controls, such as file modes (-rwxr-xr-x) are modifiable by the user and the applications which the user runs. Conversely, SELinux access controls are determined by a policy loaded on the system which may not be changed by careless users or misbehaving applications.

AppArmor is an effective and easy-to-use Linux application security system. AppArmor proactively protects the operating system and applications from external or internal threats, even zero-day attacks, by enforcing good behavior and preventing both known and unknown application flaws from being exploited. AppArmor supplements the traditional Unix discretionary access control (DAC) model by providing mandatory access control (MAC). It has been included in the mainline Linux kernel since version 2.6.36 and its development has been supported by Canonical since 2009.

Control Groups(Cgroups) is a Linux kernel feature that allows you to allocate resources such as CPU time, system memory, network bandwidth, or any combination of these resources for user-defined groups of tasks (processes) running on a system.

EarlyOOM is a daemon for Linux that enables users to more quickly recover and regain control over their system in low-memory situations with heavy swap usage.

Libgcrypt is a general purpose cryptographic library originally based on code from GnuPG.

Kali Linux is an open source project that is maintained and funded by Offensive Security, a provider of world-class information security training and penetration testing services.

Aircrack-ng is a network software suite consisting of a detector, packet sniffer, WEP and WPA/WPA2-PSK cracker and analysis tool for 802.11 wireless LANs. It works with any wireless network interface controller whose driver supports raw monitoring mode and can sniff 802.11a, 802.11b and 802.11g traffic.

Burp Suite is a leading range of cybersecurity tools.

KernelCI is a community-based open source distributed test automation system focused on upstream kernel development. The primary goal of KernelCI is to use an open testing philosophy to ensure the quality, stability and long-term maintenance of the Linux kernel.

Continuous Kernel Integration project helps find bugs in kernel patches before they are commited to an upstram kernel tree. We are team of kernel developers, kernel testers, and automation engineers.

eBPF is a revolutionary technology that can run sandboxed programs in the Linux kernel without changing kernel source code or loading kernel modules. By making the Linux kernel programmable, infrastructure software can leverage existing layers, making them more intelligent and feature-rich without continuing to add additional layers of complexity to the system.

Cilium uses eBPF to accelerate getting data in and out of L7 proxies such as Envoy, enabling efficient visibility into API protocols like HTTP, gRPC, and Kafka.

Hubble is a Network, Service & Security Observability for Kubernetes using eBPF.

Istio is an open platform to connect, manage, and secure microservices. Istio's control plane provides an abstraction layer over the underlying cluster management platform, such as Kubernetes and Mesos.

Certgen is a convenience tool to generate and store certificates for Hubble Relay mTLS.

Scapy is a python-based interactive packet manipulation program & library.

syzkaller is an unsupervised, coverage-guided kernel fuzzer.

SchedViz is a tool for gathering and visualizing kernel scheduling traces on Linux machines.

oss-fuzz aims to make common open source software more secure and stable by combining modern fuzzing techniques with scalable, distributed execution.

OSSEC is a free, open-source host-based intrusion detection system. It performs log analysis, integrity checking, Windows registry monitoring, rootkit detection, time-based alerting, and active response.

Metasploit Project is a computer security project that provides information about security vulnerabilities and aids in penetration testing and IDS signature development.

Wfuzz was created to facilitate the task in web applications assessments and it is based on a simple concept: it replaces any reference to the FUZZ keyword by the value of a given payload.

Nmap is a security scanner used to discover hosts and services on a computer network, thus building a "map" of the network.

Patchwork is a web-based patch tracking system designed to facilitate the contribution and management of contributions to an open-source project.

pfSense is a free and open source firewall and router that also features unified threat management, load balancing, multi WAN, and more.

Snowpatch is a continuous integration tool for projects using a patch-based, mailing-list-centric git workflow. This workflow is used by a number of well-known open source projects such as the Linux kernel.

Snort is an open-source, free and lightweight network intrusion detection system (NIDS) software for Linux and Windows to detect emerging threats.

Wireshark is a free and open-source packet analyzer. It is used for network troubleshooting, analysis, software and communications protocol development, and education.

OpenSCAP is U.S. standard maintained by National Institute of Standards and Technology (NIST). It provides multiple tools to assist administrators and auditors with assessment, measurement, and enforcement of security baselines. OpenSCAP maintains great flexibility and interoperability by reducing the costs of performing security audits. Whether you want to evaluate DISA STIGs, NIST‘s USGCB, or Red Hat’s Security Response Team’s content, all are supported by OpenSCAP.

Tink is a multi-language, cross-platform, open source library that provides cryptographic APIs that are secure, easy to use correctly, and harder to misuse.

OWASP is an online community, produces freely-available articles, methodologies, documentation, tools, and technologies in the field of web application security.

Open Vulnerability and Assessment Language is a community effort to standardize how to assess and report upon the machine state of computer systems. OVAL includes a language to encode system details, and community repositories of content. Tools and services that use OVAL provide enterprises with accurate, consistent, and actionable information to improve their security.

ClamAV is an open source antivirus engine for detecting trojans, viruses, malware & other malicious threats.

Open Source Security Learning Resources

Microsoft Open Source Software Security

Cloudflare Open Source Security

The Seven Properties of Highly Secure Devices

How Layer 7 of the Internet Works

The 7 Kinds of Security

The Libgcrypt Reference Manual

The Open Web Application Security Project(OWASP) Foundation Top 10

Best Practices for Using Open Source Code from The Linux Foundation

AWS Certified Security - Specialty Certification

Microsoft Certified: Azure Security Engineer Associate

Google Cloud Certified Professional Cloud Security Engineer

Cisco Security Certifications

The Red Hat Certified Specialist in Security: Linux

Linux Professional Institute LPIC-3 Enterprise Security Certification

Cybersecurity Training and Courses from IBM Skills

Cybersecurity Courses and Certifications by Offensive Security

RSA Certification Program

Check Point Certified Security Expert(CCSE) Certification

Check Point Certified Security Administrator(CCSA) Certification

Check Point Certified Security Master (CCSM) Certification

Certified Cloud Security Professional(CCSP) Certification

Certified Information Systems Security Professional (CISSP) Certification

CCNP Routing and Switching

Certified Information Security Manager(CISM)

Wireshark Certified Network Analyst (WCNA)

Juniper Networks Certification Program Enterprise (JNCP)

Security Training Certifications and Courses from Udemy

Security Training Certifications and Courses from Coursera

Security Certifications Training from Pluarlsight

Differential Privacy

Back to the Top

Above is a simple diagram of how Differential Privacy-Preserving Data Sharing and Data Mining protects a User's Data

Differential Privacy Learning Resources

Differential Privacy is a system that simultaneously enables researchers and analysts to extract useful insights from datasets containing personal information and offers stronger privacy protections. This is achieved by introducing "statistical noise".

Statistical Noise is a process that small aletrations to masked datasets. The statistical noise hides identifiable characteristics of individuals, ensuring that the privacy of personal information is protected, but it's small enough to not materially impact the accuracy of the answers extracted by analysts and researchers.

Laplacian Noise is a mechanism that adds Laplacian-distributed noise to a function.

Differential Privacy Blog Series by the National Institute of Standards and Technology(NIST)

Apple's Differential Privacy Overview

Learning with Privacy at Scale with Apple Machine Learning

Microsoft Research Differential Privacy Overview

Responsible Machine Learning with Microsoft Azure

Responsible AI Resources with Microsoft AI

Preserve data privacy by using differential privacy and the SmartNoise package

Open Differential Privacy(OpenDP) Initiative by Microsoft and Harvard

Google's Differential Privacy Library

Computing Private Statistics with Privacy on Beam from Google Codelabs

Introducing TensorFlow Privacy: Learning with Differential Privacy for Training Data

TensorFlow Federated: Machine Learning on Decentralized Data

Federated Analytics: Collaborative Data Science without Data Collection

Differentially-Private Stochastic Gradient Descent(DP-SGD)

Learning Differential Privacy from Harvard University Privacy Tools Project

Harvard University Privacy Tools Project Courses & Educational Materials

The Weaknesses of Differential Privacy course on Coursera

The Differential Privacy of Bayesian Inference

Simultaneous private learning of multiple concepts

The Complexity of Computing the Optimal Composition of Differential Privacy

Order revealing encryption and the hardness of private learning

SAP HANA data anonymization using SAP Software Solutions

SAP HANA Security using their In-Memory Database

DEFCON Differential Privacy Training Launch

Secure and Private AI course on Udacity

Differential Privacy - Security and Privacy for Big Data - Part 1 course on Coursera

Differential Privacy - Security and Privacy for Big Data - Part 2 course on Coursera

Certified Ethical Emerging Technologist Professional Certificate course on Coursera

Differential Privacy Tools

PySyft is a Python library for secure and private Deep Learning. PySyft decouples private data from model training, using Federated Learning, Differential Privacy, and Encrypted Computation (like Multi-Party Computation (MPC) and Homomorphic Encryption (HE) within the main Deep Learning frameworks like PyTorch and TensorFlow.

TensorFlow Privacy is a Python library that includes implementations of TensorFlow optimizers for training machine learning models with differential privacy. The library comes with tutorials and analysis tools for computing the privacy guarantees provided.

TensorFlow Federated (TFF) is an open-source framework for machine learning and other computations on decentralized data. TFF has been developed to facilitate open research and experimentation with Federated Learning (FL), an approach to machine learning where a shared global model is trained across many participating clients that keep their training data locally.

Privacy on Beam is an end-to-end differential privacy solution built on Apache Beam. It is intended to be usable by all developers, regardless of their differential privacy expertise.

PyDP is a Python wrapper for Google's Differential Privacy project.

PennyLane is a cross-platform Python library for differentiable programming of quantum computers. By training a quantum computer the same way as a neural network.

BoTorch is a library for Bayesian Optimization built on PyTorch.

PyTorch Geometric (PyG) is a geometric deep learning extension library for PyTorch.

Skorch is a scikit-learn compatible neural network library that wraps PyTorch.

Diffprivlib is the IBM Differential Privacy Library for experimenting with, investigating and developing applications in, differential privacy.

Opacus is a library that enables training PyTorch models with differential privacy. It supports training with minimal code changes required on the client, has little impact on training performance and allows the client to online track the privacy budget expended at any given moment.

Smart Noise is a toolkit that uses state-of-the-art differential privacy (DP) techniques to inject noise into data, to prevent disclosure of sensitive information and manage exposure risk.

Machine Learning

Back to the Top

ML frameworks & applications

TensorFlow is an end-to-end open source platform for machine learning. It has a comprehensive, flexible ecosystem of tools, libraries and community resources that lets researchers push the state-of-the-art in ML and developers easily build and deploy ML powered applications.

Tensorman is a utility for easy management of Tensorflow containers by developed by System76.Tensorman allows Tensorflow to operate in an isolated environment that is contained from the rest of the system. This virtual environment can operate independent of the base system, allowing you to use any version of Tensorflow on any version of a Linux distribution that supports the Docker runtime.

Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano.It was developed with a focus on enabling fast experimentation. It is capable of running on top of TensorFlow, Microsoft Cognitive Toolkit, R, Theano, or PlaidML.

PyTorch is a library for deep learning on irregular input data such as graphs, point clouds, and manifolds. Primarily developed by Facebook's AI Research lab.

Amazon SageMaker is a fully managed service that provides every developer and data scientist with the ability to build, train, and deploy machine learning (ML) models quickly. SageMaker removes the heavy lifting from each step of the machine learning process to make it easier to develop high quality models.

Azure Databricks is a fast and collaborative Apache Spark-based big data analytics service designed for data science and data engineering. Azure Databricks, sets up your Apache Spark environment in minutes, autoscale, and collaborate on shared projects in an interactive workspace. Azure Databricks supports Python, Scala, R, Java, and SQL, as well as data science frameworks and libraries including TensorFlow, PyTorch, and scikit-learn.

Microsoft Cognitive Toolkit (CNTK) is an open-source toolkit for commercial-grade distributed deep learning. It describes neural networks as a series of computational steps via a directed graph. CNTK allows the user to easily realize and combine popular model types such as feed-forward DNNs, convolutional neural networks (CNNs) and recurrent neural networks (RNNs/LSTMs). CNTK implements stochastic gradient descent (SGD, error backpropagation) learning with automatic differentiation and parallelization across multiple GPUs and servers.

Apache Airflow is an open-source workflow management platform created by the community to programmatically author, schedule and monitor workflows. Install. Principles. Scalable. Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. Airflow is ready to scale to infinity.

Open Neural Network Exchange(ONNX) is an open ecosystem that empowers AI developers to choose the right tools as their project evolves. ONNX provides an open source format for AI models, both deep learning and traditional ML. It defines an extensible computation graph model, as well as definitions of built-in operators and standard data types.

Apache MXNet is a deep learning framework designed for both efficiency and flexibility. It allows you to mix symbolic and imperative programming to maximize efficiency and productivity. At its core, MXNet contains a dynamic dependency scheduler that automatically parallelizes both symbolic and imperative operations on the fly. A graph optimization layer on top of that makes symbolic execution fast and memory efficient. MXNet is portable and lightweight, scaling effectively to multiple GPUs and multiple machines. Support for Python, R, Julia, Scala, Go, Javascript and more.

AutoGluon is toolkit for Deep learning that automates machine learning tasks enabling you to easily achieve strong predictive performance in your applications. With just a few lines of code, you can train and deploy high-accuracy deep learning models on tabular, image, and text data.

Anaconda is a very popular Data Science platform for machine learning and deep learning that enables users to develop models, train them, and deploy them.

PlaidML is an advanced and portable tensor compiler for enabling deep learning on laptops, embedded devices, or other devices where the available computing hardware is not well supported or the available software stack contains unpalatable license restrictions.

OpenCV is a highly optimized library with focus on real-time computer vision applications. The C++, Python, and Java interfaces support Linux, MacOS, Windows, iOS, and Android.

Scikit-Learn is a Python module for machine learning built on top of SciPy, NumPy, and matplotlib, making it easier to apply robust and simple implementations of many popular machine learning algorithms.

Weka is an open source machine learning software that can be accessed through a graphical user interface, standard terminal applications, or a Java API. It is widely used for teaching, research, and industrial applications, contains a plethora of built-in tools for standard machine learning tasks, and additionally gives transparent access to well-known toolboxes such as scikit-learn, R, and Deeplearning4j.

Caffe is a deep learning framework made with expression, speed, and modularity in mind. It is developed by Berkeley AI Research (BAIR)/The Berkeley Vision and Learning Center (BVLC) and community contributors.

Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently including tight integration with NumPy.

nGraph is an open source C++ library, compiler and runtime for Deep Learning. The nGraph Compiler aims to accelerate developing AI workloads using any deep learning framework and deploying to a variety of hardware targets.It provides the freedom, performance, and ease-of-use to AI developers.

NVIDIA cuDNN is a GPU-accelerated library of primitives for deep neural networks. cuDNN provides highly tuned implementations for standard routines such as forward and backward convolution, pooling, normalization, and activation layers. cuDNN accelerates widely used deep learning frameworks, including Caffe2, Chainer, Keras, MATLAB, MxNet, PyTorch, and TensorFlow.

Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Jupyter is used widely in industries that do data cleaning and transformation, numerical simulation, statistical modeling, data visualization, data science, and machine learning.

Apache Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Scala, Java, Python, and R, and an optimized engine that supports general computation graphs for data analysis. It also supports a rich set of higher-level tools including Spark SQL for SQL and DataFrames, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for stream processing.

Apache Spark Connector for SQL Server and Azure SQL is a high-performance connector that enables you to use transactional data in big data analytics and persists results for ad-hoc queries or reporting. The connector allows you to use any SQL database, on-premises or in the cloud, as an input data source or output data sink for Spark jobs.

Apache PredictionIO is an open source machine learning framework for developers, data scientists, and end users. It supports event collection, deployment of algorithms, evaluation, querying predictive results via REST APIs. It is based on scalable open source services like Hadoop, HBase (and other DBs), Elasticsearch, Spark and implements what is called a Lambda Architecture.

Cluster Manager for Apache Kafka(CMAK) is a tool for managing Apache Kafka clusters.

BigDL is a distributed deep learning library for Apache Spark. With BigDL, users can write their deep learning applications as standard Spark programs, which can directly run on top of existing Spark or Hadoop clusters.

Koalas is project makes data scientists more productive when interacting with big data, by implementing the pandas DataFrame API on top of Apache Spark.

Apache Spark™ MLflow is an open source platform to manage the ML lifecycle, including experimentation, reproducibility, deployment, and a central model registry. MLflow currently offers four components:

MLflow Tracking: Record and query experiments: code, data, config, and results.

MLflow Projects: Package data science code in a format to reproduce runs on any platform.

MLflow Models: Deploy machine learning models in diverse serving environments.

Model Registry: Store, annotate, discover, and manage models in a central repository.

Eclipse Deeplearning4J (DL4J) is a set of projects intended to support all the needs of a JVM-based(Scala, Kotlin, Clojure, and Groovy) deep learning application. This means starting with the raw data, loading and preprocessing it from wherever and whatever format it is in to building and tuning a wide variety of simple and complex deep learning networks.

Numba is an open source, NumPy-aware optimizing compiler for Python sponsored by Anaconda, Inc. It uses the LLVM compiler project to generate machine code from Python syntax. Numba can compile a large subset of numerically-focused Python, including many NumPy functions. Additionally, Numba has support for automatic parallelization of loops, generation of GPU-accelerated code, and creation of ufuncs and C callbacks.

Chainer is a Python-based deep learning framework aiming at flexibility. It provides automatic differentiation APIs based on the define-by-run approach (dynamic computational graphs) as well as object-oriented high-level APIs to build and train neural networks. It also supports CUDA/cuDNN using CuPy for high performance training and inference.

cuML is a suite of libraries that implement machine learning algorithms and mathematical primitives functions that share compatible APIs with other RAPIDS projects. cuML enables data scientists, researchers, and software engineers to run traditional tabular ML tasks on GPUs without going into the details of CUDA programming. In most cases, cuML's Python API matches the API from scikit-learn.

Online ML Learning Resources

Machine Learning by Stanford University from Coursera

Machine Learning Courses Online from Coursera

Machine Learning Courses Online from Udemy

Learn Machine Learning with Online Courses and Classes from edX

IoT Protocols

Back to the Top

DBus is an open source software bus developed Red Hat for inter-process communication, and remote procedure call mechanism that allows communication between multiple processes running concurrently on the same machine.

SOAP is a messaging protocol specification for exchanging structured information in the implementation of web services in computer networks. SOAP can extend HTTP for XML messaging. SOAP provides data transport for Web services. SOAP can exchange complete documents or call a remote procedure. SOAP can be used for broadcasting a message.

gRPC is a modern, open source remote procedure call (RPC) framework developed by Google that can run anywhere. It enables client and server applications to communicate transparently, and makes it easier to build connected systems.It uses HTTP/2 for transport, Protocol Buffers as the interface description language, and provides features such as authentication, bidirectional streaming and flow control, blocking or nonblocking bindings, and cancellation and timeouts.

LWM2M is a protocol from the Open Mobile Alliance for M2M or IoT device management. Lightweight M2M enabler defines the application layer communication protocol between a LWM2M Server and a LWM2M Client, which is located in a LWM2M Device.

Advanced Message Queuing Protocol (AMQP) is an open standard for passing business messages between applications or organizations. It connects systems, feeds business processes with the information they need and reliably transmits onward the instructions that achieve their goals. The defining features of AMQP are message orientation, queuing, routing, reliability and security.

Constrained Application Protocol (CoAP) is a specialized web transfer protocol for use with constrained nodes and constrained networks in the Internet of Things. The protocol is designed for machine-to-machine (M2M) applications such as smart energy and building automation."

Extensible Messaging and Presence Protocol (XMPP) is a communication protocol for message-oriented middleware based on XML (Extensible Markup Language). It enables the near real-time exchange of structured yet extensible data between any two or more network entities.

OASIS Message Queuing Telemetry Transport (MQTT) is an open OASIS and ISO standard (ISO/IEC 20922) lightweight, publish-subscribe network protocol that transports messages between devices. The protocol usually runs over TCP/IP; however, any network protocol that provides ordered, lossless, bi-directional connections can support MQTT.

Very Simple Control Protocol (VSCP) is a free automation protocol suitable for all sorts of automation task where building- or home-automation is in the main focus. Its main advantage is that each VSCP-node can work completely autonomous, being part of distributed network of other nodes.

Operating systems

Back to the Top

Raspberry Pi OS

Hass.io(Home Assistant OS)

Manjaro Linux ARM

Arch Linux ARM

Ubuntu MATE for Raspberry Pi

Ubuntu Desktop for Raspberry Pi

Ubuntu Core on a Raspberry Pi

Ubuntu Server for ARM

Kali Linux for the Raspberry Pi

HeliOS is an embedded operating system that is free for anyone to use. While called an operating system for simplicity, HeliOS is better described as a multitasking kernel for embedded systems.

Simba is a small OS for an Embedded Programming Platform like Arduino. It aims to make embedded programming easy and portable.

Trampoline is a static RTOS for small embedded systems.

DuinOS is Framework (a wrapper) for use the FreeRTOSwith Arduino.

VxWorks is an industry-leading real-time operating systems (RTOS) for building embedded devices and systems for more than 30 years.

LynxOS is a native POSIX, hard real-time partitioning operating system developed by Lynx Software Technologies.

Zephyr OS is a popular security-oriented RTOS with a small-footprint kernel designed for use on resource-constrained and embedded systems. Zephyr has a small-foorprint Kernel focusing on embedded devices compatible with x86, ARM, RISC-V, Xtensa and others.

FreeRTOS is an open source, real-time operating system for microcontrollers that makes small, low-power edge devices easy to program, deploy, secure, connect, and manage.

Arm Mbed TLS provides a comprehensive SSL/TLS solution and makes it easy for developers to include cryptographic and SSL/TLS capabilities in their software and embedded products. As an SSL library, it provides an intuitive API, readable source code and a minimal and highly configurable code footprint.

Contiki-os is an operating system for networked, memory-constrained systems with a focus on low-power wireless Internet of Things devices.

Middleware

Back to the Top

IoTSyS is an integration middleware for the Internet of Things. It provides a communication stack for embedded devices based on IPv6, Web services, and OBIX to establish interoperable interfaces for smart objects.

OpenIoT is an open source middleware infrastructure will support flexible configuration and deployment of algorithms for collection, and filtering information streams stemming from the internet-connected objects, while at the same time generating and processing important business/applications events.

OpenRemote is an open source middleware project, which integrates many different protocols and solutions available for smart building, and smart city automation, and offers visualization tools.

Kaa is a Enterprise IoT Platform has been designed with heavy-duty, enterprise-grade IoT solutions in mind. It banishes a monolithic approach to architecture in favour of highly portable microservices, which allow for flexible rearrangement and customization even in the middle of the solution's lifecycle.

Node flow editors

Back to the Top

Node-RED is a programming tool for wiring together hardware devices, APIs and online services in new and interesting ways. It provides a browser-based editor that makes it easy to wire together flows using the wide range of nodes in the palette that can be deployed to its runtime in a single-click.

Toolkits

Back to the Top

KinomaJS is a visual code editor designed to help developers build starter projects for Kinoma Create and Kinoma Element. The project is built on Angular 2(RC7) and runs in a web browser. The live version is hosted using Google App Engine, but you can modify and build it yourself by following the instructions in this document.

IoT Toolkit is a collection of libraries that enables communication with modern IoT based environments and devices. It is a high-performance collection of libraries optimized for minimum memory consumption in RAM, ROM, high speed, and versatility working on any device.

Data Visualization

Back to the Top

Freeboard is an open source real-time dashboard builder for IOT and other web mashups. A free open-source alternative to Geckoboard.

ThingSpeak is an IoT analytics platform service that allows you to aggregate, visualize, and analyze live data streams in the cloud. You can send data to ThingSpeak from your devices, create instant visualization of live data, and send alerts.

Search

Back to the Top

Thingful is a Search Engine for the Internet of Things Find & use open IoT data from around the world.

Hardware

Back to the Top

Arduino Ethernet Shield 2 allows an Arduino board to connect to the internet using the Ethernet library and to read and write an SD card using the SD library.This shield is fully compatible with the former version, but relies on the newer W5500 chip.

Raspberry Pi is a series of small single-board computers developed in the United Kingdom by the Raspberry Pi Foundation to promote teaching of basic computer science in schools and in developing countries. Price range from $10-45 depending on model.

BeagleBone is a low-power open-source single-board computer produced by Texas Instruments. It runs Android, Ubuntu and other Linux flavors.

openPicus FlyportPro is a system on a module dedicated to IoT and M2M application, especially for professional use. Following some details on the solution: Why FlyportPRO SoM? A system-on-module is the best solution for those customers looking for flexibility and for development time and risk reduction.

Pinoccio is a solution to add mesh networking capability and WiFi-Internet access to all yout IoT devices, and it is Arduino compatible. Each board can assume the role of Scout in a Troop and one of the Scouts is the Lead to connect internet: Field Scouts talk to each other using a mesh network (called a Troop), using an extremely low-power radio.

In-memory data grids

Back to the Top

Ehcache is an open source, standards-based cache that boosts performance, offloads your database, and simplifies scalability. It's the most widely-used Java-based cache because it's robust, proven, full-featured, and integrates with other popular libraries and frameworks.

Hazelcast is an open source in-memory data grid based on Java.

Home automation

Back to the Top

Home Assistant is open source home automation that puts local control and privacy first. Powered by a worldwide community of tinkerers and DIY enthusiasts. Perfect to run on a Raspberry Pi or a local server.

openHAB is a cross-platform software with the aim to integrate all kinds of Smart Home technologies, devices, etc.

Eclipse SmartHome is a framework, not a ready-to-use solution. It offers a large set of features to choose from and leaves enough possibilities to design a Smart Home solution specific to your expectations. Its modular design brings millions of combinations and proves to be easily extensible by custom parts.

The Thing System is a set of software components and network protocols that aims to fix the Internet of Things. Our steward software is written in node.js making it both portable and easily extensible. It can run on your laptop, or fit onto a small single board computer like the Raspberry Pi.

Robotics

Back to the Top

Tools for Robotics

Open Source Robotics Foundation works with industry, academia, and government to create and support open software and hardware for use in robotics, from research and education to product development.

ROS is robotics middleware. Although ROS is not an operating system, it provides services designed for a heterogeneous computer cluster such as hardware abstraction, low-level device control, implementation of commonly used functionality, message-passing between processes, and package management.

ROS2 is a set of software libraries and tools that help you build robot applications. From drivers to state-of-the-art algorithms, and with powerful developer tools, ROS has what you need for your next robotics project. And it’s all open source.

Robot Framework is a generic open source automation framework. It can be used for test automation and robotic process automation. It has easy syntax, utilizing human-readable keywords. Its capabilities can be extended by libraries implemented with Python or Java.

The Robotics Library (RL) is a self-contained C++ library for robot kinematics, motion planning and control. It covers mathematics, kinematics and dynamics, hardware abstraction, motion planning, collision detection, and visualization.RL runs on many different systems, including Linux, macOS, and Windows. It uses CMake as a build system and can be compiled with Clang, GCC, and Visual Studio.

MoveIt is the most widely used software for manipulation and has been used on over 100 robots. It provides an easy-to-use robotics platform for developing advanced applications, evaluating new designs and building integrated products for industrial, commercial, R&D, and other domains.

AutoGluon is tool

autowarefoundation/autoware

Autoware - the world's leading open-source software project for autonomous driving

Autoware is an open-source software stack for self-driving vehicles, built on the Robot Operating System (ROS). It includes all of the necessary functions to drive an autonomous vehicles from localization and object detection to route planning and control, and was created with the aim of enabling as many individuals and organizations as possible to contribute to open innovations in autonomous driving technology.

Documentation

To learn more about using or developing Autoware, refer to the Autoware documentation site. You can find the source for the documentation in autowarefoundation/autoware-documentation.

Repository overview

autowarefoundation/autoware
- Meta-repository containing .repos files to construct an Autoware workspace.
- It is anticipated that this repository will be frequently forked by users, and so it contains minimal information to avoid unnecessary differences.
autowarefoundation/autoware_core
- Main repository for high-quality, stable ROS packages for Autonomous Driving.
- Based on Autoware.Auto and Autoware.Universe.
autowarefoundation/autoware_universe
- Repository for experimental, cutting-edge ROS packages for Autonomous Driving.
- Autoware Universe was created to make it easier for researchers and developers to extend the functionality of Autoware Core
autowarefoundation/autoware_launch
- Launch configuration repository containing node configurations and their parameters.
autowarefoundation/autoware-github-actions
- Contains reusable GitHub Actions workflows used by multiple repositories for CI.
- Utilizes the DRY concept.
autowarefoundation/autoware-documentation
- Documentation repository for Autoware users and developers.
- Since Autoware Core/Universe has multiple repositories, a central documentation repository is important to make information accessible from a single place.

Using Autoware.AI

If you wish to use Autoware.AI, the previous version of Autoware based on ROS 1, switch to autoware-ai repository. However, be aware that Autoware.AI has reached the end-of-life as of 2022, and we strongly recommend transitioning to Autoware Core/Universe for future use.

Contributing

There is no formal process to become a contributor - you can comment on any existing issues or make a pull request on any Autoware repository!
- Make sure to follow the Contribution Guidelines.
- Take a look at Autoware's various working groups to gain an understanding of any work in progress and to see how projects are managed.
If you have any technical questions, you can start a discussion in the Q&A category to request help and confirm if a potential issue is a bug or not.

Useful resources

goldbergyoni/nodebestpractices

✅ The Node.js best practices list (July 2024)

Node.js Best Practices

Follow us on Twitter! @nodepractices

Read in a different language: CN, FR, BR, RU, PL, JA, EU (ES, HE, KR and TR in progress! )

🎊 2024 edition is here!

🛰 Modernized to 2024: Tons of text edits, new recommended libraries, and some new best practices
✨ Easily focus on new content: Already visited before? Search for #new or #updated tags for new content only
🔖 Curious to see examples? We have a starter: Visit Practica.js, our application example and boilerplate (beta) to see some practices in action

Welcome! 3 Things You Ought To Know First

1. You are reading dozens of the best Node.js articles - this repository is a summary and curation of the top-ranked content on Node.js best practices, as well as content written here by collaborators

2. It is the largest compilation, and it is growing every week - currently, more than 80 best practices, style guides, and architectural tips are presented. New issues and pull requests are created every day to keep this live book updated. We'd love to see you contributing here, whether that is fixing code mistakes, helping with translations, or suggesting brilliant new ideas. See our writing guidelines here

3. Best practices have additional info - most bullets include a 🔗Read More link that expands on the practice with code examples, quotes from selected blogs, and more information

By Yoni Goldberg

Learn with me: As a consultant, I engage with worldwide teams on various activities like workshops and code reviews. 🎉AND... Hold on, I've just launched my beyond-the-basics testing course, which is on a 🎁 limited-time sale until August 7th

1. Project Architecture Practices (6)

1.1 Structure your solution by components #strategic #updated
1.2 Layer your components, keep the web layer within its boundaries #strategic #updated
1.3 Wrap common utilities as packages, consider publishing
1.4 Use environment aware, secure and hierarchical config #updated
1.5 Consider all the consequences when choosing the main framework #new
1.6 Use TypeScript sparingly and thoughtfully #new

2. Error Handling Practices (12)

2.1 Use Async-Await or promises for async error handling
2.2 Extend the built-in Error object #strategic #updated
2.3 Distinguish operational vs programmer errors #strategic #updated
2.4 Handle errors centrally, not within a middleware #strategic
2.5 Document API errors using OpenAPI or GraphQL
2.6 Exit the process gracefully when a stranger comes to town #strategic
2.7 Use a mature logger to increase errors visibility #updated
2.8 Test error flows using your favorite test framework #updated
2.9 Discover errors and downtime using APM products
2.10 Catch unhandled promise rejections #updated
2.11 Fail fast, validate arguments using a dedicated library
2.12 Always await promises before returning to avoid a partial stacktrace #new
2.13 Subscribe to event emitters 'error' event #new

3. Code Style Practices (12)

3.1 Use ESLint #strategic
3.2 Use Node.js eslint extension plugins #updated
3.3 Start a Codeblock's Curly Braces on the Same Line
3.4 Separate your statements properly
3.5 Name your functions
3.6 Use naming conventions for variables, constants, functions and classes
3.7 Prefer const over let. Ditch the var
3.8 Require modules first, not inside functions
3.9 Set an explicit entry point to a module/folder #updated
3.10 Use the === operator
3.11 Use Async Await, avoid callbacks #strategic
3.12 Use arrow function expressions (=>)
3.13 Avoid effects outside of functions #new

4. Testing And Overall Quality Practices (13)

4.1 At the very least, write API (component) testing #strategic
4.2 Include 3 parts in each test name #new
4.3 Structure tests by the AAA pattern #strategic
4.4 Ensure Node version is unified #new
4.5 Avoid global test fixtures and seeds, add data per-test #strategic
4.6 Tag your tests #advanced
4.7 Check your test coverage, it helps to identify wrong test patterns
4.8 Use production-like environment for e2e testing
4.9 Refactor regularly using static analysis tools
4.10 Mock responses of external HTTP services #advanced #new #advanced
4.11 Test your middlewares in isolation
4.12 Specify a port in production, randomize in testing #new
4.13 Test the five possible outcomes #strategic #new

5. Going To Production Practices (19)

5.1. Monitoring #strategic
5.2. Increase the observability using smart logging #strategic
5.3. Delegate anything possible (e.g. gzip, SSL) to a reverse proxy #strategic
5.4. Lock dependencies
5.5. Guard process uptime using the right tool
5.6. Utilize all CPU cores
5.7. Create a ‘maintenance endpoint’
5.8. Discover the unknowns using APM products #advanced #updated
5.9. Make your code production-ready
5.10. Measure and guard the memory usage #advanced
5.11. Get your frontend assets out of Node
5.12. Strive to be stateless #strategic
5.13. Use tools that automatically detect vulnerabilities
5.14. Assign a transaction id to each log statement #advanced
5.15. Set NODE_ENV=production
5.16. Design automated, atomic and zero-downtime deployments #advanced
5.17. Use an LTS release of Node.js
5.18. Log to stdout, avoid specifying log destination within the app #updated
5.19. Install your packages with npm ci #new

6. Security Practices (25)

6.1. Embrace linter security rules
6.2. Limit concurrent requests using a middleware
6.3 Extract secrets from config files or use packages to encrypt them #strategic
6.4. Prevent query injection vulnerabilities with ORM/ODM libraries #strategic
6.5. Collection of generic security best practices
6.6. Adjust the HTTP response headers for enhanced security
6.7. Constantly and automatically inspect for vulnerable dependencies #strategic
6.8. Protect Users' Passwords/Secrets using bcrypt or scrypt #strategic
6.9. Escape HTML, JS and CSS output
6.10. Validate incoming JSON schemas #strategic
6.11. Support blocklisting JWTs
6.12. Prevent brute-force attacks against authorization #advanced
6.13. Run Node.js as non-root user
6.14. Limit payload size using a reverse-proxy or a middleware
6.15. Avoid JavaScript eval statements
6.16. Prevent evil RegEx from overloading your single thread execution
6.17. Avoid module loading using a variable
6.18. Run unsafe code in a sandbox
6.19. Take extra care when working with child processes #advanced
6.20. Hide error details from clients
6.21. Configure 2FA for npm or Yarn #strategic
6.22. Modify session middleware settings
6.23. Avoid DOS attacks by explicitly setting when a process should crash #advanced
6.24. Prevent unsafe redirects
6.25. Avoid publishing secrets to the npm registry
6.26. 6.26 Inspect for outdated packages
6.27. Import built-in modules using the 'node:' protocol #new

7. Performance Practices (2) (Work In Progress️ ✍️)

7.1. Don't block the event loop
7.2. Prefer native JS methods over user-land utils like Lodash

8. Docker Practices (15)

8.1 Use multi-stage builds for leaner and more secure Docker images #strategic
8.2. Bootstrap using node command, avoid npm start
8.3. Let the Docker runtime handle replication and uptime #strategic
8.4. Use .dockerignore to prevent leaking secrets
8.5. Clean-up dependencies before production
8.6. Shutdown smartly and gracefully #advanced
8.7. Set memory limits using both Docker and v8 #advanced #strategic
8.8. Plan for efficient caching
8.9. Use explicit image reference, avoid latest tag
8.10. Prefer smaller Docker base images
8.11. Clean-out build-time secrets, avoid secrets in args #strategic #new
8.12. Scan images for multi layers of vulnerabilities #advanced
8.13 Clean NODE_MODULE cache
8.14. Generic Docker practices
8.15. Lint your Dockerfile #new

`1. Project Architecture Practices`

1.1 Structure your solution by business components

`📝 #updated`

TL;DR: The root of a system should contain folders or repositories that represent reasonably sized business modules. Each component represents a product domain (i.e., bounded context), like 'user-component', 'order-component', etc. Each component has its own API, logic, and logical database. What is the significant merit? With an autonomous component, every change is performed over a granular and smaller scope - the mental overload, development friction, and deployment fear are much smaller and better. As a result, developers can move much faster. This does not necessarily demand physical separation and can be achieved using a Monorepo or with a multi-repo

my-system
├─ apps (components)
│  ├─ orders
│  ├─ users
│  ├─ payments
├─ libraries (generic cross-component functionality)
│  ├─ logger
│  ├─ authenticator

Otherwise: when artifacts from various modules/topics are mixed together, there are great chances of a tightly-coupled 'spaghetti' system. For example, in an architecture where 'module-a controller' might call 'module-b service', there are no clear modularity borders - every code change might affect anything else. With this approach, developers who code new features struggle to realize the scope and impact of their change. Consequently, they fear breaking other modules, and each deployment becomes slower and riskier

1.2 Layer your components with 3-tiers, keep the web layer within its boundaries

`📝 #updated`

TL;DR: Each component should contain 'layers' - a dedicated folder for common concerns: 'entry-point' where controller lives, 'domain' where the logic lives, and 'data-access'. The primary principle of the most popular architectures is to separate the technical concerns (e.g., HTTP, DB, etc) from the pure logic of the app so a developer can code more features without worrying about infrastructural concerns. Putting each concern in a dedicated folder, also known as the 3-Tier pattern, is the simplest way to meet this goal

my-system
├─ apps (components)
│  ├─ component-a
   │  ├─ entry-points
   │  │  ├─ api # controller comes here
   │  │  ├─ message-queue # message consumer comes here
   │  ├─ domain # features and flows: DTO, services, logic
   │  ├─ data-access # DB calls w/o ORM

Otherwise: It's often seen that developer pass web objects like request/response to functions in the domain/logic layer - this violates the separation principle and makes it harder to access later the logic code by other clients like testing code, scheduled jobs, message queues, etc

🔗 Read More: layer your app

1.3 Wrap common utilities as packages, consider publishing

TL;DR: Place all reusable modules in a dedicated folder, e.g., "libraries", and underneath each module in its own folder, e.g., "/libraries/logger". Make the module an independent package with its own package.json file to increase the module encapsulation, and allows future publishing to a repository. In a Monorepo setup, modules can be consumed by 'npm linking' to their physical paths, using ts-paths or by publishing and installing from a package manager repository like the npm registry

my-system
├─ apps (components)
  │  ├─ component-a
├─ libraries (generic cross-component functionality)
│  ├─ logger
│  │  ├─ package.json
│  │  ├─ src
│  │  │ ├─ index.js

Otherwise: Clients of a module might import and get coupled to internal functionality of a module. With a package.json at the root, one can set a package.json.main or package.json.exports to explicitly tell which files and functions are part of the public interface

🔗 Read More: Structure by feature

1.4 Use environment aware, secure and hierarchical config

`📝 #updated`

TL;DR: A flawless configuration setup should ensure (a) keys can be read from file AND from environment variable (b) secrets are kept outside committed code (c) config is hierarchical for easier findability (d) typing support (e) validation for failing fast (f) Specify default for each key. There are a few packages that can help tick most of those boxes like convict, env-var, zod, and others

Otherwise: Consider a mandatory environment variable that wasn't provided. The app starts successfully and serve requests, some information is already persisted to DB. Then, it's realized that without this mandatory key the request can't complete, leaving the app in a dirty state

1.5 Consider all the consequences when choosing the main framework

`🌟 #new`

TL;DR: When building apps and APIs, using a framework is mandatory. It's easy to overlook alternative frameworks or important considerations and then finally land on a sub optimal option. As of 2023/2024, we believe that these four frameworks are worth considering: Nest.js, Fastify, express, and Koa. Click read more below for a detailed pros/cons of each framework. Simplistically, we believe that Nest.js is the best match for teams who wish to go OOP and/or build large-scale apps that can't get partitioned into smaller autonomous components. Fastify is our recommendation for apps with reasonably-sized components (e.g., Microservices) that are built around simple Node.js mechanics. Read our full considerations guide here

Otherwise: Due to the overwhelming amount of considerations, it's easy to make decisions based on partial information and compare apples with oranges. For example, it's believed that Fastify is a minimal web-server that should get compared with express only. In reality, it's a rich framework with many official plugins that cover many concerns

1.6 Use TypeScript sparingly and thoughtfully

`🌟 #new`

TL;DR: Coding without type safety is no longer an option, TypeScript is the most popular option for this mission. Use it to define variables and functions return types. With that, it is also a double edge sword that can greatly encourage complexity with its additional ~ 50 keywords and sophisticated features. Consider using it sparingly, mostly with simple types, and utilize advanced features only when a real need arises

Otherwise: Researches show that using TypeScript can help in detecting ~20% bugs earlier. Without it, also the developer experience in the IDE is intolerable. On the flip side, 80% of other bugs were not discovered using types. Consequently, typed syntax is valuable but limited. Only efficient tests can discover the whole spectrum of bugs, including type-related bugs. It might also defeat its purpose: sophisticated code features are likely to increase the code complexity, which by itself increases both the amount of bugs and the average bug fix time

⬆ Return to top

`2. Error Handling Practices`

2.1 Use Async-Await or promises for async error handling

TL;DR: Handling async errors in callback style is probably the fastest way to hell (a.k.a the pyramid of doom). The best gift you can give to your code is using Promises with async-await which enables a much more compact and familiar code syntax like try-catch

Otherwise: Node.js callback style, function(err, response), is a promising way to un-maintainable code due to the mix of error handling with casual code, excessive nesting, and awkward coding patterns

🔗 Read More: avoiding callbacks

2.2 Extend the built-in Error object

`📝 #updated`

TL;DR: Some libraries throw errors as a string or as some custom type – this complicates the error handling logic and the interoperability between modules. Instead, create app error object/class that extends the built-in Error object and use it whenever rejecting, throwing or emitting an error. The app error should add useful imperative properties like the error name/code and isCatastrophic. By doing so, all errors have a unified structure and support better error handling. There is no-throw-literal ESLint rule that strictly checks that (although it has some limitations which can be solved when using TypeScript and setting the @typescript-eslint/no-throw-literal rule)

Otherwise: When invoking some component, being uncertain which type of errors come in return – it makes proper error handling much harder. Even worse, using custom types to describe errors might lead to loss of critical error information like the stack trace!

2.3 Distinguish catastrophic errors from operational errors

`📝 #updated`

TL;DR: Operational errors (e.g. API received an invalid input) refer to known cases where the error impact is fully understood and can be handled thoughtfully. On the other hand, catastrophic error (also known as programmer errors) refers to unusual code failures that dictate to gracefully restart the application

Otherwise: You may always restart the application when an error appears, but why let ~5000 online users down because of a minor, predicted, operational error? The opposite is also not ideal – keeping the application up when an unknown catastrophic issue (programmer error) occurred might lead to an unpredicted behavior. Differentiating the two allows acting tactfully and applying a balanced approach based on the given context

2.4 Handle errors centrally, not within a middleware

TL;DR: Error handling logic such as logging, deciding whether to crash and monitoring metrics should be encapsulated in a dedicated and centralized object that all entry-points (e.g. APIs, cron jobs, scheduled jobs) call when an error comes in

Otherwise: Not handling errors within a single place will lead to code duplication and probably to improperly handled errors

2.5 Document API errors using OpenAPI or GraphQL

TL;DR: Let your API callers know which errors might come in return so they can handle these thoughtfully without crashing. For RESTful APIs, this is usually done with documentation frameworks like OpenAPI. If you're using GraphQL, you can utilize your schema and comments as well

Otherwise: An API client might decide to crash and restart only because it received back an error it couldn’t understand. Note: the caller of your API might be you (very typical in a microservice environment)

2.6 Exit the process gracefully when a stranger comes to town

TL;DR: When an unknown error occurs (catastrophic error, see best practice 2.3) - there is uncertainty about the application healthiness. In this case, there is no escape from making the error observable, shutting off connections and exiting the process. Any reputable runtime framework like Dockerized services or cloud serverless solutions will take care to restart

Otherwise: When an unfamiliar exception occurs, some object might be in a faulty state (e.g. an event emitter which is used globally and not firing events anymore due to some internal failure) and all future requests might fail or behave crazily

🔗 Read More: shutting the process

2.7 Use a mature logger to increase errors visibility

`📝 #updated`

TL;DR: A robust logging tools like Pino or Winston increases the errors visibility using features like log-levels, pretty print coloring and more. Console.log lacks these imperative features and should be avoided. The best in class logger allows attaching custom useful properties to log entries with minimized serialization performance penalty. Developers should write logs to stdout and let the infrastructure pipe the stream to the appropriate log aggregator

Otherwise: Skimming through console.logs or manually through messy text file without querying tools or a decent log viewer might keep you busy at work until late

🔗 Read More: using a mature logger

2.8 Test error flows using your favorite test framework

`📝 #updated`

TL;DR: Whether professional automated QA or plain manual developer testing – Ensure that your code not only satisfies positive scenarios but also handles and returns the right errors. On top of this, simulate deeper error flows like uncaught exceptions and ensure that the error handler treat these properly (see code examples within the "read more" section)

Otherwise: Without testing, whether automatically or manually, you can’t rely on your code to return the right errors. Without meaningful errors – there’s no error handling

🔗 Read More: testing error flows

2.9 Discover errors and downtime using APM products

TL;DR: Monitoring and performance products (a.k.a APM) proactively gauge your codebase or API so they can automagically highlight errors, crashes, and slow parts that you were missing

Otherwise: You might spend great effort on measuring API performance and downtimes, probably you’ll never be aware which are your slowest code parts under real-world scenario and how these affect the UX

🔗 Read More: using APM products

2.10 Catch unhandled promise rejections

`📝 #updated`

TL;DR: Any exception thrown within a promise will get swallowed and discarded unless a developer didn’t forget to explicitly handle it. Even if your code is subscribed to process.uncaughtException! Overcome this by registering to the event process.unhandledRejection

Otherwise: Your errors will get swallowed and leave no trace. Nothing to worry about

2.11 Fail fast, validate arguments using a dedicated library

TL;DR: Assert API input to avoid nasty bugs that are much harder to track later. The validation code is usually tedious unless you are using a modern validation library like ajv, zod, or typebox

Otherwise: Consider this – your function expects a numeric argument “Discount” which the caller forgets to pass, later on, your code checks if Discount!=0 (amount of allowed discount is greater than zero), then it will allow the user to enjoy a discount. OMG, what a nasty bug. Can you see it?

🔗 Read More: failing fast

2.12 Always await promises before returning to avoid a partial stacktrace

`🌟 #new`

TL;DR: Always do return await when returning a promise to benefit full error stacktrace. If a function returns a promise, that function must be declared as async function and explicitly await the promise before returning it

Otherwise: The function that returns a promise without awaiting won't appear in the stacktrace. Such missing frames would probably complicate the understanding of the flow that leads to the error, especially if the cause of the abnormal behavior is inside of the missing function

🔗 Read More: returning promises

2.13 Subscribe to event emitters and streams 'error' event

`🌟 #new`

TL;DR: Unlike typical functions, a try-catch clause won't get errors that originate from Event Emitters and anything inherited from it (e.g., streams). Instead of try-catch, subscribe to an event emitter's 'error' event so your code can handle the error in context. When dealing with EventTargets (the web standard version of Event Emitters) there are no 'error' event and all errors end in the process.on('error) global event - in this case, at least ensure that the process crash or not based on the desired context. Also, mind that error originating from asynchronous event handlers are not get caught unless the event emitter is initialized with {captureRejections: true}

Otherwise: Event emitters are commonly used for global and key application functionality such as DB or message queue connection. When this kind of crucial objects throw an error, at best the process will crash due to unhandled exception. Even worst, it will stay alive as a zombie while a key functionality is turned off

⬆ Return to top

`3. Code Patterns And Style Practices`

3.1 Use ESLint

TL;DR: ESLint is the de-facto standard for checking possible code errors and fixing code style, not only to identify nitty-gritty spacing issues but also to detect serious code anti-patterns like developers throwing errors without classification. Though ESLint can automatically fix code styles, other tools like prettier are more powerful in formatting the fix and work in conjunction with ESLint

Otherwise: Developers will focus on tedious spacing and line-width concerns and time might be wasted overthinking the project's code style

3.2 Use Node.js eslint extension plugins

`📝 #updated`

TL;DR: On top of ESLint standard rules that cover vanilla JavaScript, add Node.js specific plugins like eslint-plugin-node, eslint-plugin-mocha and eslint-plugin-node-security, eslint-plugin-require, /eslint-plugin-jest and other useful rules

Otherwise: Many faulty Node.js code patterns might escape under the radar. For example, developers might require(variableAsPath) files with a variable given as a path which allows attackers to execute any JS script. Node.js linters can detect such patterns and complain early

3.3 Start a Codeblock's Curly Braces on the Same Line

TL;DR: The opening curly braces of a code block should be on the same line as the opening statement

Code Example

// Do
function someFunction() {
  // code block
}

// Avoid
function someFunction()
{
  // code block
}

Otherwise: Deferring from this best practice might lead to unexpected results, as seen in the StackOverflow thread below:

3.4 Separate your statements properly

No matter if you use semicolons or not to separate your statements, knowing the common pitfalls of improper linebreaks or automatic semicolon insertion, will help you to eliminate regular syntax errors.

TL;DR: Use ESLint to gain awareness about separation concerns. Prettier or Standardjs can automatically resolve these issues.

Otherwise: As seen in the previous section, JavaScript's interpreter automatically adds a semicolon at the end of a statement if there isn't one, or considers a statement as not ended where it should, which might lead to some undesired results. You can use assignments and avoid using immediately invoked function expressions to prevent most of the unexpected errors.

Code example

// Do
function doThing() {
    // ...
}

doThing()

// Do

const items = [1, 2, 3]
items.forEach(console.log)

// Avoid — throws exception
const m = new Map()
const a = [1,2,3]
[...m.values()].forEach(console.log)
> [...m.values()].forEach(console.log)
>  ^^^
> SyntaxError: Unexpected token ...

// Avoid — throws exception
const count = 2 // it tries to run 2(), but 2 is not a function
(function doSomething() {
  // do something amazing
}())
// put a semicolon before the immediate invoked function, after the const definition, save the return value of the anonymous function to a variable or avoid IIFEs altogether

3.5 Name your functions

TL;DR: Name all functions, including closures and callbacks. Avoid anonymous functions. This is especially useful when profiling a node app. Naming all functions will allow you to easily understand what you're looking at when checking a memory snapshot

Otherwise: Debugging production issues using a core dump (memory snapshot) might become challenging as you notice significant memory consumption from anonymous functions

3.6 Use naming conventions for variables, constants, functions and classes

TL;DR: Use lowerCamelCase when naming constants, variables and functions, UpperCamelCase (capital first letter as well) when naming classes and UPPER_SNAKE_CASE when naming global or static variables. This will help you to easily distinguish between plain variables, functions, classes that require instantiation and variables declared at global module scope. Use descriptive names, but try to keep them short

Otherwise: JavaScript is the only language in the world that allows invoking a constructor ("Class") directly without instantiating it first. Consequently, Classes and function-constructors are differentiated by starting with UpperCamelCase

3.6 Code Example

// for global variables names we use the const/let keyword and UPPER_SNAKE_CASE
let MUTABLE_GLOBAL = "mutable value";
const GLOBAL_CONSTANT = "immutable value";
const CONFIG = {
  key: "value",
};

// examples of UPPER_SNAKE_CASE convention in nodejs/javascript ecosystem
// in javascript Math.PI module
const PI = 3.141592653589793;

// https://github.com/nodejs/node/blob/b9f36062d7b5c5039498e98d2f2c180dca2a7065/lib/internal/http2/core.js#L303
// in nodejs http2 module
const HTTP_STATUS_OK = 200;
const HTTP_STATUS_CREATED = 201;

// for class name we use UpperCamelCase
class SomeClassExample {
  // for static class properties we use UPPER_SNAKE_CASE
  static STATIC_PROPERTY = "value";
}

// for functions names we use lowerCamelCase
function doSomething() {
  // for scoped variable names we use the const/let keyword and lowerCamelCase
  const someConstExample = "immutable value";
  let someMutableExample = "mutable value";
}

3.7 Prefer const over let. Ditch the var

TL;DR: Using const means that once a variable is assigned, it cannot be reassigned. Preferring const will help you to not be tempted to use the same variable for different uses, and make your code clearer. If a variable needs to be reassigned, in a for loop, for example, use let to declare it. Another important aspect of let is that a variable declared using it is only available in the block scope in which it was defined. var is function scoped, not block-scoped, and shouldn't be used in ES6 now that you have const and let at your disposal

Otherwise: Debugging becomes way more cumbersome when following a variable that frequently changes

3.8 Require modules first, not inside functions

TL;DR: Require modules at the beginning of each file, before and outside of any functions. This simple best practice will not only help you easily and quickly tell the dependencies of a file right at the top but also avoids a couple of potential problems

Otherwise: Requires are run synchronously by Node.js. If they are called from within a function, it may block other requests from being handled at a more critical time. Also, if a required module or any of its dependencies throw an error and crash the server, it is best to find out about it as soon as possible, which might not be the case if that module is required from within a function

3.9 Set an explicit entry point to a module/folder

`📝 #updated`

TL;DR: When developing a module/library, set an explicit root file that exports the public and interesting code. Discourage the client code from importing deep files and becoming familiar with the internal structure. With commonjs (require), this can be done with an index.js file at the folder's root or the package.json.main field. With ESM (import), if a package.json exists on the root, the field "exports" allow specifying the module's root file. If no package.json exists, you may put an index.js file on the root which re-exports all the public functionality

Otherwise: Having an explicit root file acts like a public 'interface' that encapsulates the internal, directs the caller to the public code and facilitates future changes without breaking the contract

3.9 Code example - avoid coupling the client to the module structure

// Avoid: client has deep familiarity with the internals

// Client code
const SMSWithMedia = require("./SMSProvider/providers/media/media-provider.js");

// Better: explicitly export the public functions

//index.js, module code
module.exports.SMSWithMedia = require("./SMSProvider/providers/media/media-provider.js");

// Client code
const { SMSWithMedia } = require("./SMSProvider");

3.10 Use the `===` operator

TL;DR: Prefer the strict equality operator === over the weaker abstract equality operator ==. == will compare two variables after converting them to a common type. There is no type conversion in ===, and both variables must be of the same type to be equal

Otherwise: Unequal variables might return true when compared with the == operator

3.10 Code example

"" == "0"; // false
0 == ""; // true
0 == "0"; // true

false == "false"; // false
false == "0"; // true

false == undefined; // false
false == null; // false
null == undefined; // true

" \t\r\n " == 0; // true

All statements above will return false if used with ===

3.11 Use Async Await, avoid callbacks

TL;DR: Async-await is the simplest way to express an asynchronous flow as it makes asynchronous code look synchronous. Async-await will also result in much more compact code and support for try-catch. This technique now supersedes callbacks and promises in most cases. Using it in your code is probably the best gift one can give to the code reader

Otherwise: Handling async errors in callback style are probably the fastest way to hell - this style forces to check errors all over, deal with awkward code nesting, and makes it difficult to reason about the code flow

3.12 Use arrow function expressions (=>)

TL;DR: Though it's recommended to use async-await and avoid function parameters when dealing with older APIs that accept promises or callbacks - arrow functions make the code structure more compact and keep the lexical context of the root function (i.e. this)

Otherwise: Longer code (in ES5 functions) is more prone to bugs and cumbersome to read

3.13 Avoid effects outside of functions

`🌟 #new`

TL;DR: Avoid putting code with effects like network or DB calls outside of functions. Such a code will be executed immediately when another file requires the file. This 'floating' code might get executed when the underlying system is not ready yet. It also comes with a performance penalty even when this module's functions will finally not be used in runtime. Last, mocking these DB/network calls for testing is harder outside of functions. Instead, put this code inside functions that should get called explicitly. If some DB/network code must get executed right when the module loads, consider using the factory or revealing module patterns

Otherwise: A typical web framework sets error handler, environment variables and monitoring. When DB/network calls are made before the web framework is initialized, they won't be monitored or fail due to a lack of configuration data

⬆ Return to top

`4. Testing And Overall Quality Practices`

_We have dedicated guides for testing, see below. The best practices list here is a brief summary of these guides

a. JavaScript testing best practices b. Node.js testing - beyond the basics _

4.1 At the very least, write API (component) testing

TL;DR: Most projects just don't have any automated testing due to short timetables or often the 'testing project' ran out of control and was abandoned. For that reason, prioritize and start with API testing which is the easiest way to write and provides more coverage than unit testing (you may even craft API tests without code using tools like Postman). Afterwards, should you have more resources and time, continue with advanced test types like unit testing, DB testing, performance testing, etc

Otherwise: You may spend long days on writing unit tests to find out that you got only 20% system coverage

4.2 Include 3 parts in each test name

`🌟 #new`

TL;DR: Make the test speak at the requirements level so it's self-explanatory also to QA engineers and developers who are not familiar with the code internals. State in the test name what is being tested (unit under test), under what circumstances, and what is the expected result

Otherwise: A deployment just failed, a test named “Add product” failed. Does this tell you what exactly is malfunctioning?

4.3 Structure tests by the AAA pattern

`🌟 #new`

TL;DR: Structure your tests with 3 well-separated sections: Arrange, Act & Assert (AAA). The first part includes the test setup, then the execution of the unit under test, and finally the assertion phase. Following this structure guarantees that the reader spends no brain CPU on understanding the test plan

Otherwise: Not only you spend long daily hours on understanding the main code, but now also what should have been the simple part of the day (testing) stretches your brain

4.4 Ensure Node version is unified

`🌟 #new`

TL;DR: Use tools that encourage or enforce the same Node.js version across different environments and developers. Tools like nvm, and Volta allow specifying the project's version in a file so each team member can run a single command to conform with the project's version. Optionally, this definition can be replicated to CI and the production runtime (e.g., copy the specified value to .Dockerfile build and to the CI declaration file)

Otherwise: A developer might face or miss an error because she uses a different Node.js version than her teammates. Even worse - the production runtime might be different than the environment where tests were executed

4.5 Avoid global test fixtures and seeds, add data per-test

TL;DR: To prevent test coupling and easily reason about the test flow, each test should add and act on its own set of DB rows. Whenever a test needs to pull or assume the existence of some DB data - it must explicitly add that data and avoid mutating any other records

Otherwise: Consider a scenario where deployment is aborted due to failing tests, team is now going to spend precious investigation time that ends in a sad conclusion: the system works well, the tests however interfere with each other and break the build

4.6 Tag your tests

TL;DR: Different tests must run on different scenarios: quick smoke, IO-less, tests should run when a developer saves or commits a file, full end-to-end tests usually run when a new pull request is submitted, etc. This can be achieved by tagging tests with keywords like #cold #api #sanity so you can grep with your testing harness and invoke the desired subset. For example, this is how you would invoke only the sanity test group with Mocha: mocha --grep 'sanity'

Otherwise: Running all the tests, including tests that perform dozens of DB queries, any time a developer makes a small change can be extremely slow and keeps developers away from running tests

4.7 Check your test coverage, it helps to identify wrong test patterns

TL;DR: Code coverage tools like Istanbul/NYC are great for 3 reasons: it comes for free (no effort is required to benefit this reports), it helps to identify a decrease in testing coverage, and last but not least it highlights testing mismatches: by looking at colored code coverage reports you may notice, for example, code areas that are never tested like catch clauses (meaning that tests only invoke the happy paths and not how the app behaves on errors). Set it to fail builds if the coverage falls under a certain threshold

Otherwise: There won't be any automated metric telling you when a large portion of your code is not covered by testing

4.8 Use production-like environment for e2e testing

TL;DR: End to end (e2e) testing which includes live data used to be the weakest link of the CI process as it depends on multiple heavy services like DB. Use an environment which is as close to your real production environment as possible like a-continue (Missed -continue here, needs content. Judging by the Otherwise clause, this should mention docker-compose)

Otherwise: Without docker-compose, teams must maintain a testing DB for each testing environment including developers' machines, keep all those DBs in sync so test results won't vary across environments

4.9 Refactor regularly using static analysis tools

TL;DR: Using static analysis tools helps by giving objective ways to improve code quality and keeps your code maintainable. You can add static analysis tools to your CI build to fail when it finds code smells. Its main selling points over plain linting are the ability to inspect quality in the context of multiple files (e.g. detect duplications), perform advanced analysis (e.g. code complexity), and follow the history and progress of code issues. Two examples of tools you can use are Sonarqube (2,600+ stars) and Code Climate (1,500+ stars).

Otherwise: With poor code quality, bugs and performance will always be an issue that no shiny new library or state of the art features can fix

🔗 Read More: Refactoring!

4.10 Mock responses of external HTTP services

`🌟 #new`

TL;DR: Use network mocking tools to simulate responses of external collaborators' services that are approached over the network (e.g., REST, Graph). This is imperative not only to isolate the component under test but mostly to simulate non-happy path flows. Tools like nock (in-process) or Mock-Server allow defining a specific response of external service in a single line of code. Remember to simulate also errors, delays, timeouts, and any other event that is likely to happen in production

Otherwise: Allowing your component to reach real external services instances will likely result in naive tests that mostly cover happy paths. The tests might also be flaky and slow

🔗 Read More: Mock external services

4.11 Test your middlewares in isolation

TL;DR: When a middleware holds some immense logic that spans many requests, it is worth testing it in isolation without waking up the entire web framework. This can be easily achieved by stubbing and spying on the {req, res, next} objects

Otherwise: A bug in Express middleware === a bug in all or most requests

4.12 Specify a port in production, randomize in testing

`🌟 #new`

TL;DR: When testing against the API, it's common and desirable to initialize the web server inside the tests. Let the server randomize the web server port in testing to prevent collisions. If you're using Node.js http server (used by most frameworks), doing so demands nothing but passing a port number zero - this will randomize an available port

Otherwise: Specifying a fixed port will prevent two testing processes from running at the same time. Most of the modern test runners run with multiple processes by default

4.13 Test the five possible outcomes

`🌟 #new`

TL;DR: When testing a flow, ensure to cover five potential categories. Any time some action is triggered (e.g., API call), a reaction occurs, a meaningful outcome is produced and calls for testing. There are five possible outcome types for every flow: a response, a visible state change (e.g., DB), an outgoing API call, a new message in a queue, and an observability call (e.g., logging, metric). See a checklist here. Each type of outcome comes with unique challenges and techniques to mitigate those challenges - we have a dedicated guide about this topic: Node.js testing - beyond the basics

Otherwise: Consider a case when testing the addition of a new product to the system. It's common to see tests that assert on a valid response only. What if the product was failed to persist regardless of the positive response? what if when adding a new product demands calling some external service, or putting a message in the queue - shouldn't the test assert these outcomes as well? It's easy to overlook various paths, this is where a checklist comes handy

🔗 Read More: Test five outcomes

⬆ Return to top

`5. Going To Production Practices`

5.1. Monitoring

TL;DR: Monitoring is a game of finding out issues before customers do – obviously this should be assigned unprecedented importance. The market is overwhelmed with offers thus consider starting with defining the basic metrics you must follow (my suggestions inside), then go over additional fancy features and choose the solution that ticks all boxes. In any case, the 4 layers of observability must be covered: uptime, metrics with focus on user-facing symptoms and Node.js technical metrics like event loop lag, distributed flows measurement with Open Telemetry and logging. Click ‘Read More’ below for an overview of the solutions

Otherwise: Failure === disappointed customers. Simple

🔗 Read More: Monitoring!

5.2. Increase the observability using smart logging

TL;DR: Logs can be a dumb warehouse of debug statements or the enabler of a beautiful dashboard that tells the story of your app. Plan your logging platform from day 1: how logs are collected, stored and analyzed to ensure that the desired information (e.g. error rate, following an entire transaction through services and servers, etc) can really be extracted

Otherwise: You end up with a black box that is hard to reason about, then you start re-writing all logging statements to add additional information

5.3. Delegate anything possible (e.g. gzip, SSL) to a reverse proxy

TL;DR: Node is quite bad at doing CPU intensive tasks like gzipping, SSL termination, etc. You should use specialized infrastructure like nginx, HAproxy or cloud vendor services instead

Otherwise: Your poor single thread will stay busy doing infrastructural tasks instead of dealing with your application core and performance will degrade accordingly

5.4. Lock dependencies

TL;DR: Your code must be identical across all environments, but without a special lockfile npm lets dependencies drift across environments. Ensure to commit your package-lock.json so all the environments will be identical

Otherwise: QA will thoroughly test the code and approve a version that will behave differently in production. Even worse, different servers in the same production cluster might run different code

🔗 Read More: Lock dependencies

5.5. Guard process uptime using the right tool

TL;DR: The process must go on and get restarted upon failures. Modern runtime platforms like Docker-ized platforms (e.g. Kubernetes), and Serverless take care for this automatically. When the app is hosted on a bare metal server, one must take care for a process management tools like systemd. Avoid including a custom process management tool in a modern platform that monitors an app instance (e.g., Kubernetes) - doing so will hide failures from the infrastructure. When the underlying infrastructure is not aware of errors, it can't perform useful mitigation steps like re-placing the instance in a different location

Otherwise: Running dozens of instances without a clear strategy and too many tools together (cluster management, docker, PM2) might lead to DevOps chaos

5.6. Utilize all CPU cores

TL;DR: At its basic form, a Node app runs on a single CPU core while all others are left idling. It’s your duty to replicate the Node process and utilize all CPUs. Most of the modern run-times platform (e.g., Kubernetes) allow replicating instances of the app but they won't verify that all cores are utilized - this is your duty. If the app is hosted on a bare server, it's also your duty to use some process replication solution (e.g. systemd)

Otherwise: Your app will likely utilize only 25% of its available resources(!) or even less. Note that a typical server has 4 CPU cores or more, naive deployment of Node.js utilizes only 1 (even using PaaS services like AWS beanstalk!)

🔗 Read More: Utilize all CPU cores

5.7. Create a ‘maintenance endpoint’

TL;DR: Expose a set of system-related information, like memory usage and REPL, etc in a secured API. Although it’s highly recommended to rely on standard and battle-tested tools, some valuable information and operations are easier done using code

Otherwise: You’ll find that you’re performing many “diagnostic deploys” – shipping code to production only to extract some information for diagnostic purposes

5.8. Discover the unknowns using APM products

`📝 #updated`

TL;DR: Consider adding another safety layer to the production stack - APM. While the majority of symptoms and causes can be detected using traditional monitoring techniques, in a distributed system there is more than meets the eye. Application monitoring and performance products (a.k.a. APM) can auto-magically go beyond traditional monitoring and provide additional layer of discovery and developer-experience. For example, some APM products can highlight a transaction that loads too slow on the end-user's side while suggesting the root cause. APMs also provide more context for developers who try to troubleshoot a log error by showing what was the server busy with when the error occurred. To name a few example

Otherwise: You might spend great effort on measuring API performance and downtimes, probably you’ll never be aware which is your slowest code parts under real-world scenario and how these affect the UX

5.9. Make your code production-ready

TL;DR: Code with the end in mind, plan for production from day 1. This sounds a bit vague so I’ve compiled a few development tips that are closely related to production maintenance (click 'Read More')

Otherwise: A world champion IT/DevOps guy won’t save a system that is badly written

5.10. Measure and guard the memory usage

TL;DR: Node.js has controversial relationships with memory: the v8 engine has soft limits on memory usage (1.4GB) and there are known paths to leak memory in Node’s code – thus watching Node’s process memory is a must. In small apps, you may gauge memory periodically using shell commands but in medium-large apps consider baking your memory watch into a robust monitoring system

Otherwise: Your process memory might leak a hundred megabytes a day like how it happened at Walmart

5.11. Get your frontend assets out of Node

TL;DR: Serve frontend content using a specialized infrastructure (nginx, S3, CDN) because Node performance gets hurt when dealing with many static files due to its single-threaded model. One exception to this guideline is when doing server-side rendering

Otherwise: Your single Node thread will be busy streaming hundreds of html/images/angular/react files instead of allocating all its resources for the task it was born for – serving dynamic content

5.12. Strive to be stateless

TL;DR: Store any type of data (e.g. user sessions, cache, uploaded files) within external data stores. When the app holds data in-process this adds additional layer of maintenance complexity like routing users to the same instance and higher cost of restarting a process. To enforce and encourage a stateless approach, most modern runtime platforms allows 'reapp-ing' instances periodically

Otherwise: Failure at a given server will result in application downtime instead of just killing a faulty machine. Moreover, scaling-out elasticity will get more challenging due to the reliance on a specific server

5.13. Use tools that automatically detect vulnerabilities

TL;DR: Even the most reputable dependencies such as Express have known vulnerabilities (from time to time) that can put a system at risk. This can be easily tamed using community and commercial tools that constantly check for vulnerabilities and warn (locally or at GitHub), some can even patch them immediately

Otherwise: Keeping your code clean from vulnerabilities without dedicated tools will require you to constantly follow online publications about new threats. Quite tedious

5.14. Assign a transaction id to each log statement

TL;DR: Assign the same identifier, transaction-id: uuid(), to each log entry within a single request (also known as correlation-id/tracing-id/request-context). Then when inspecting errors in logs, easily conclude what happened before and after. Node has a built-in mechanism, AsyncLocalStorage, for keeping the same context across asynchronous calls. see code examples inside

Otherwise: Looking at a production error log without the context – what happened before – makes it much harder and slower to reason about the issue

5.15. Set `NODE_ENV=production`

TL;DR: Set the environment variable NODE_ENV to ‘production’ or ‘development’ to flag whether production optimizations should get activated – some npm packages determine the current environment and optimize their code for production

Otherwise: Omitting this simple property might greatly degrade performance when dealing with some specific libraries like Express server-side rendering

5.16. Design automated, atomic and zero-downtime deployments

TL;DR: Research shows that teams who perform many deployments lower the probability of severe production issues. Fast and automated deployments that don’t require risky manual steps and service downtime significantly improve the deployment process. You should probably achieve this using Docker combined with CI tools as they became the industry standard for streamlined deployment

Otherwise: Long deployments -> production downtime & human-related error -> team unconfident in making deployment -> fewer deployments and features

5.17. Use an LTS release of Node.js

TL;DR: Ensure you are using an LTS version of Node.js to receive critical bug fixes, security updates and performance improvements

Otherwise: Newly discovered bugs or vulnerabilities could be used to exploit an application running in production, and your application may become unsupported by various modules and harder to maintain

5.18. Log to stdout, avoid specifying log destination within the app

`📝 #updated`

TL;DR: Log destinations should not be hard-coded by developers within the application code, but instead should be defined by the execution environment the application runs in. Developers should write logs to stdout using a logger utility and then let the execution environment (container, server, etc.) pipe the stdout stream to the appropriate destination (i.e. Splunk, Graylog, ElasticSearch, etc.).

Otherwise: If developers set the log routing, less flexibility is left for the ops professional who wishes to customize it. Beyond this, if the app tries to log directly to a remote location (e.g., Elastic Search), in case of panic or crash - further logs that might explain the problem won't arrive

🔗 Read More: Log Routing

5.19. Install your packages with `npm ci`

TL;DR: Run npm ci to strictly do a clean install of your dependencies matching package.json and package-lock.json. Obviously production code must use the exact version of the packages that were used for testing. While package-lock.json file sets strict version for dependencies, in case of mismatch with the file package.json, the command 'npm install' will treat package.json as the source of truth. On the other hand, the command 'npm ci' will exit with error in case of mismatch between these files

Otherwise: QA will thoroughly test the code and approve a version that will behave differently in production. Even worse, different servers in the same production cluster might run different code.

🔗 Read More: Use npm ci

⬆ Return to top

`6. Security Best Practices`

6.1. Embrace linter security rules

TL;DR: Make use of security-related linter plugins such as eslint-plugin-security to catch security vulnerabilities and issues as early as possible, preferably while they're being coded. This can help catching security weaknesses like using eval, invoking a child process or importing a module with a string literal (e.g. user input). Click 'Read more' below to see code examples that will get caught by a security linter

Otherwise: What could have been a straightforward security weakness during development becomes a major issue in production. Also, the project may not follow consistent code security practices, leading to vulnerabilities being introduced, or sensitive secrets committed into remote repositories

🔗 Read More: Lint rules

6.2. Limit concurrent requests using a middleware

TL;DR: DOS attacks are very popular and relatively easy to conduct. Implement rate limiting using an external service such as cloud load balancers, cloud firewalls, nginx, rate-limiter-flexible package, or (for smaller and less critical apps) a rate-limiting middleware (e.g. express-rate-limit)

Otherwise: An application could be subject to an attack resulting in a denial of service where real users receive a degraded or unavailable service.

6.3 Extract secrets from config files or use packages to encrypt them

TL;DR: Never store plain-text secrets in configuration files or source code. Instead, make use of secret-management systems like Vault products, Kubernetes/Docker Secrets, or using environment variables. As a last resort, secrets stored in source control must be encrypted and managed (rolling keys, expiring, auditing, etc). Make use of pre-commit/push hooks to prevent committing secrets accidentally

Otherwise: Source control, even for private repositories, can mistakenly be made public, at which point all secrets are exposed. Access to source control for an external party will inadvertently provide access to related systems (databases, apis, services, etc).

🔗 Read More: Secret management

6.4. Prevent query injection vulnerabilities with ORM/ODM libraries

TL;DR: To prevent SQL/NoSQL injection and other malicious attacks, always make use of an ORM/ODM or a database library that escapes data or supports named or indexed parameterized queries, and takes care of validating user input for expected types. Never just use JavaScript template strings or string concatenation to inject values into queries as this opens your application to a wide spectrum of vulnerabilities. All the reputable Node.js data access libraries (e.g. Sequelize, Knex, mongoose) have built-in protection against injection attacks.

Otherwise: Unvalidated or unsanitized user input could lead to operator injection when working with MongoDB for NoSQL, and not using a proper sanitization system or ORM will easily allow SQL injection attacks, creating a giant vulnerability.

6.5. Collection of generic security best practices

TL;DR: This is a collection of security advice that is not related directly to Node.js - the Node implementation is not much different than any other language. Click read more to skim through.

6.6. Adjust the HTTP response headers for enhanced security

TL;DR: Your application should be using secure headers to prevent attackers from using common attacks like cross-site scripting (XSS), clickjacking and other malicious attacks. These can be configured easily using modules like helmet.

Otherwise: Attackers could perform direct attacks on your application's users, leading to huge security vulnerabilities

6.7. Constantly and automatically inspect for vulnerable dependencies

TL;DR: With the npm ecosystem it is common to have many dependencies for a project. Dependencies should always be kept in check as new vulnerabilities are found. Use tools like npm audit or snyk to track, monitor and patch vulnerable dependencies. Integrate these tools with your CI setup so you catch a vulnerable dependency before it makes it to production.

Otherwise: An attacker could detect your web framework and attack all its known vulnerabilities.

🔗 Read More: Dependency security

6.8. Protect Users' Passwords/Secrets using bcrypt or scrypt

TL;DR: Passwords or secrets (e.g. API keys) should be stored using a secure hash + salt function like bcrypt,scrypt, or worst case pbkdf2.

Otherwise: Passwords and secrets that are stored without using a secure function are vulnerable to brute forcing and dictionary attacks that will lead to their disclosure eventually.

🔗 Read More: User Passwords

6.9. Escape HTML, JS and CSS output

TL;DR: Untrusted data that is sent down to the browser might get executed instead of just being displayed, this is commonly referred as a cross-site-scripting (XSS) attack. Mitigate this by using dedicated libraries that explicitly mark the data as pure content that should never get executed (i.e. encoding, escaping)

Otherwise: An attacker might store malicious JavaScript code in your DB which will then be sent as-is to the poor clients

🔗 Read More: Escape output

6.10. Validate incoming JSON schemas

TL;DR: Validate the incoming requests' body payload and ensure it meets expectations, fail fast if it doesn't. To avoid tedious validation coding within each route you may use lightweight JSON-based validation schemas such as jsonschema or joi

Otherwise: Your generosity and permissive approach greatly increases the attack surface and encourages the attacker to try out many inputs until they find some combination to crash the application

6.11. Support blocklisting JWTs

TL;DR: When using JSON Web Tokens (for example, with Passport.js), by default there's no mechanism to revoke access from issued tokens. Once you discover some malicious user activity, there's no way to stop them from accessing the system as long as they hold a valid token. Mitigate this by implementing a blocklist of untrusted tokens that are validated on each request.

Otherwise: Expired, or misplaced tokens could be used maliciously by a third party to access an application and impersonate the owner of the token.

6.12. Prevent brute-force attacks against authorization

TL;DR: A simple and powerful technique is to limit authorization attempts using two metrics:

The first is number of consecutive failed attempts by the same user unique ID/name and IP address.
The second is number of failed attempts from an IP address over some long period of time. For example, block an IP address if it makes 100 failed attempts in one day.

Otherwise: An attacker can issue unlimited automated password attempts to gain access to privileged accounts on an application

🔗 Read More: Login rate limiting

6.13. Run Node.js as non-root user

TL;DR: There is a common scenario where Node.js runs as a root user with unlimited permissions. For example, this is the default behaviour in Docker containers. It's recommended to create a non-root user and either bake it into the Docker image (examples given below) or run the process on this user's behalf by invoking the container with the flag "-u username"

Otherwise: An attacker who manages to run a script on the server gets unlimited power over the local machine (e.g. change iptable and re-route traffic to their server)

6.14. Limit payload size using a reverse-proxy or a middleware

TL;DR: The bigger the body payload is, the harder your single thread works in processing it. This is an opportunity for attackers to bring servers to their knees without tremendous amount of requests (DOS/DDOS attacks). Mitigate this limiting the body size of incoming requests on the edge (e.g. firewall, ELB) or by configuring express body parser to accept only small-size payloads

Otherwise: Your application will have to deal with large requests, unable to process the other important work it has to accomplish, leading to performance implications and vulnerability towards DOS attacks

🔗 Read More: Limit payload size

6.15. Avoid JavaScript eval statements

TL;DR: eval is evil as it allows executing custom JavaScript code during run time. This is not just a performance concern but also an important security concern due to malicious JavaScript code that may be sourced from user input. Another language feature that should be avoided is new Function constructor. setTimeout and setInterval should never be passed dynamic JavaScript code either.

Otherwise: Malicious JavaScript code finds a way into text passed into eval or other real-time evaluating JavaScript language functions, and will gain complete access to JavaScript permissions on the page. This vulnerability is often manifested as an XSS attack.

6.16. Prevent evil RegEx from overloading your single thread execution

TL;DR: Regular Expressions, while being handy, pose a real threat to JavaScript applications at large, and the Node.js platform in particular. A user input for text to match might require an outstanding amount of CPU cycles to process. RegEx processing might be inefficient to an extent that a single request that validates 10 words can block the entire event loop for 6 seconds and set the CPU on 🔥. For that reason, prefer third-party validation packages like validator.js instead of writing your own Regex patterns, or make use of safe-regex to detect vulnerable regex patterns

Otherwise: Poorly written regexes could be susceptible to Regular Expression DoS attacks that will block the event loop completely. For example, the popular moment package was found vulnerable with malicious RegEx usage in November of 2017

6.17. Avoid module loading using a variable

TL;DR: Avoid requiring/importing another file with a path that was given as parameter due to the concern that it could have originated from user input. This rule can be extended for accessing files in general (i.e. fs.readFile()) or other sensitive resource access with dynamic variables originating from user input. Eslint-plugin-security linter can catch such patterns and warn early enough

Otherwise: Malicious user input could find its way to a parameter that is used to require tampered files, for example, a previously uploaded file on the file system, or access already existing system files.

🔗 Read More: Safe module loading

6.18. Run unsafe code in a sandbox

TL;DR: When tasked to run external code that is given at run-time (e.g. plugin), use any sort of 'sandbox' execution environment that isolates and guards the main code against the plugin. This can be achieved using a dedicated process (e.g. cluster.fork()), serverless environment or dedicated npm packages that act as a sandbox

Otherwise: A plugin can attack through an endless variety of options like infinite loops, memory overloading, and access to sensitive process environment variables

6.19. Take extra care when working with child processes

TL;DR: Avoid using child processes when possible and validate and sanitize input to mitigate shell injection attacks if you still have to. Prefer using child_process.execFile which by definition will only execute a single command with a set of attributes and will not allow shell parameter expansion.

Otherwise: Naive use of child processes could result in remote command execution or shell injection attacks due to malicious user input passed to an unsanitized system command.

6.20. Hide error details from clients

TL;DR: An integrated express error handler hides the error details by default. However, great are the chances that you implement your own error handling logic with custom Error objects (considered by many as a best practice). If you do so, ensure not to return the entire Error object to the client, which might contain some sensitive application details

Otherwise: Sensitive application details such as server file paths, third party modules in use, and other internal workflows of the application which could be exploited by an attacker, could be leaked from information found in a stack trace

6.21. Configure 2FA for npm or Yarn

TL;DR: Any step in the development chain should be protected with MFA (multi-factor authentication), npm/Yarn are a sweet opportunity for attackers who can get their hands on some developer's password. Using developer credentials, attackers can inject malicious code into libraries that are widely installed across projects and services. Maybe even across the web if published in public. Enabling 2-factor-authentication in npm leaves almost zero chances for attackers to alter your package code.

Otherwise: Have you heard about the eslint developer whose password was hijacked?

6.22. Modify session middleware settings

TL;DR: Each web framework and technology has its known weaknesses - telling an attacker which web framework we use is a great help for them. Using the default settings for session middlewares can expose your app to module- and framework-specific hijacking attacks in a similar way to the X-Powered-By header. Try hiding anything that identifies and reveals your tech stack (E.g. Node.js, express)

Otherwise: Cookies could be sent over insecure connections, and an attacker might use session identification to identify the underlying framework of the web application, as well as module-specific vulnerabilities

6.23. Avoid DOS attacks by explicitly setting when a process should crash

TL;DR: The Node process will crash when errors are not handled. Many best practices even recommend to exit even though an error was caught and got handled. Express, for example, will crash on any asynchronous error - unless you wrap routes with a catch clause. This opens a very sweet attack spot for attackers who recognize what input makes the process crash and repeatedly send the same request. There's no instant remedy for this but a few techniques can mitigate the pain: Alert with critical severity anytime a process crashes due to an unhandled error, validate the input and avoid crashing the process due to invalid user input, wrap all routes with a catch and consider not to crash when an error originated within a request (as opposed to what happens globally)

Otherwise: This is just an educated guess: given many Node.js applications, if we try passing an empty JSON body to all POST requests - a handful of applications will crash. At that point, we can just repeat sending the same request to take down the applications with ease

6.24. Prevent unsafe redirects

TL;DR: Redirects that do not validate user input can enable attackers to launch phishing scams, steal user credentials, and perform other malicious actions.

Otherwise: If an attacker discovers that you are not validating external, user-supplied input, they may exploit this vulnerability by posting specially-crafted links on forums, social media, and other public places to get users to click it.

6.25. Avoid publishing secrets to the npm registry

TL;DR: Precautions should be taken to avoid the risk of accidentally publishing secrets to public npm registries. An .npmignore file can be used to ignore specific files or folders, or the files array in package.json can act as an allow list.

Otherwise: Your project's API keys, passwords or other secrets are open to be abused by anyone who comes across them, which may result in financial loss, impersonation, and other risks.

6.26 Inspect for outdated packages

TL;DR: Use your preferred tool (e.g. npm outdated or npm-check-updates) to detect installed outdated packages, inject this check into your CI pipeline and even make a build fail in a severe scenario. For example, a severe scenario might be when an installed package is 5 patch commits behind (e.g. local version is 1.3.1 and repository version is 1.3.8) or it is tagged as deprecated by its author - kill the build and prevent deploying this version

Otherwise: Your production will run packages that have been explicitly tagged by their author as risky

6.27. Import built-in modules using the 'node:' protocol

`🌟 #new`

TL;DR: Import or require built-in Node.js modules using the 'node protocol' syntax:

import { functionName } from "node:module"; // note that 'node:' prefix

For example:

import { createServer } from "node:http";

This style ensures that there is no ambiguity with global npm packages and makes it clear for the reader that the code refers to a well-trusted official module. This style can be enforced with the eslint rule 'prefer-node-protocol'

Otherwise: Using the import syntax without 'node:' prefix opens the door for typosquatting attacks where one could mistakenly mistype a module name (e.g., 'event' instead of 'events) and get a malicious package that was built only to trick users into installing them

⬆ Return to top

`7. Draft: Performance Best Practices`

Our contributors are working on this section. Would you like to join?

7.1. Don't block the event loop

TL;DR: Avoid CPU intensive tasks as they will block the mostly single-threaded Event Loop and offload those to a dedicated thread, process or even a different technology based on the context.

Otherwise: As the Event Loop is blocked, Node.js will be unable to handle other request thus causing delays for concurrent users. 3000 users are waiting for a response, the content is ready to be served, but one single request blocks the server from dispatching the results back

7.2. Prefer native JS methods over user-land utils like Lodash

TL;DR: It's often more penalising to use utility libraries like lodash and underscore over native methods as it leads to unneeded dependencies and slower performance. Bear in mind that with the introduction of the new V8 engine alongside the new ES standards, native methods were improved in such a way that it's now about 50% more performant than utility libraries.

Otherwise: You'll have to maintain less performant projects where you could have simply used what was already available or dealt with a few more lines in exchange of a few more files.

⬆ Return to top

`8. Docker Best Practices`

🏅 Many thanks to Bret Fisher from whom we learned many of the following practices

8.1 Use multi-stage builds for leaner and more secure Docker images

TL;DR: Use multi-stage build to copy only necessary production artifacts. A lot of build-time dependencies and files are not needed for running your application. With multi-stage builds these resources can be used during build while the runtime environment contains only what's necessary. Multi-stage builds are an easy way to get rid of overweight and security threats.

Otherwise: Larger images will take longer to build and ship, build-only tools might contain vulnerabilities and secrets only meant for the build phase might be leaked.

Example Dockerfile for multi-stage builds

FROM node:14.4.0 AS build

COPY . .
RUN npm ci && npm run build


FROM node:slim-14.4.0

USER node
EXPOSE 8080

COPY --from=build /home/node/app/dist /home/node/app/package.json /home/node/app/package-lock.json ./
RUN npm ci --production

CMD [ "node", "dist/app.js" ]

🔗 Read More: Use multi-stage builds

8.2. Bootstrap using `node` command, avoid `npm start`

TL;DR: Use CMD ['node','server.js'] to start your app, avoid using npm scripts which don't pass OS signals to the code. This prevents problems with child-processes, signal handling, graceful shutdown and having zombie processes

Update: Starting from npm 7, npm claim to pass signals. We follow and will update accordingly

Otherwise: When no signals are passed, your code will never be notified about shutdowns. Without this, it will lose its chance to close properly possibly losing current requests and/or data

8.3. Let the Docker runtime handle replication and uptime

TL;DR: When using a Docker run time orchestrator (e.g., Kubernetes), invoke the Node.js process directly without intermediate process managers or custom code that replicate the process (e.g. PM2, Cluster module). The runtime platform has the highest amount of data and visibility for making placement decision - It knows best how many processes are needed, how to spread them and what to do in case of crashes

Otherwise: Container keeps crashing due to lack of resources will get restarted indefinitely by the process manager. Should Kubernetes be aware of that, it could relocate it to a different roomy instance

8.4. Use .dockerignore to prevent leaking secrets

TL;DR: Include a .dockerignore file that filters out common secret files and development artifacts. By doing so, you might prevent secrets from leaking into the image. As a bonus the build time will significantly decrease. Also, ensure not to copy all files recursively rather explicitly choose what should be copied to Docker

Otherwise: Common personal secret files like .env, .aws and .npmrc will be shared with anybody with access to the image (e.g. Docker repository)

🔗 Read More: Use .dockerignore

8.5. Clean-up dependencies before production

TL;DR: Although Dev-Dependencies are sometimes needed during the build and test life-cycle, eventually the image that is shipped to production should be minimal and clean from development dependencies. Doing so guarantees that only necessary code is shipped and the amount of potential attacks (i.e. attack surface) is minimized. When using multi-stage build (see dedicated bullet) this can be achieved by installing all dependencies first and finally running npm ci --production

Otherwise: Many of the infamous npm security breaches were found within development packages (e.g. eslint-scope)

🔗 Read More: Remove development dependencies

8.6. Shutdown smartly and gracefully

TL;DR: Handle the process SIGTERM event and clean-up all existing connection and resources. This should be done while responding to ongoing requests. In Dockerized runtimes, shutting down containers is not a rare event, rather a frequent occurrence that happen as part of routine work. Achieving this demands some thoughtful code to orchestrate several moving parts: The load balancer, keep-alive connections, the HTTP server and other resources

Otherwise: Dying immediately means not responding to thousands of disappointed users

🔗 Read More: Graceful shutdown

8.7. Set memory limits using both Docker and v8

TL;DR: Always configure a memory limit using both Docker and the JavaScript runtime flags. The Docker limit is needed to make thoughtful container placement decision, the --v8's flag max-old-space is needed to kick off the GC on time and prevent under utilization of memory. Practically, set the v8's old space memory to be a just bit less than the container limit

Otherwise: The docker definition is needed to perform thoughtful scaling decision and prevent starving other citizens. Without also defining the v8's limits, it will under utilize the container resources - Without explicit instructions it crashes when utilizing ~50-60% of its host resources

8.8. Plan for efficient caching

TL;DR: Rebuilding a whole docker image from cache can be nearly instantaneous if done correctly. The less updated instructions should be at the top of your Dockerfile and the ones constantly changing (like app code) should be at the bottom.

Otherwise: Docker build will be very long and consume lot of resources even when making tiny changes

8.9. Use explicit image reference, avoid `latest` tag

TL;DR: Specify an explicit image digest or versioned label, never refer to latest. Developers are often led to believe that specifying the latest tag will provide them with the most recent image in the repository however this is not the case. Using a digest guarantees that every instance of the service is running exactly the same code.

In addition, referring to an image tag means that the base image is subject to change, as image tags cannot be relied upon for a deterministic install. Instead, if a deterministic install is expected, a SHA256 digest can be used to reference an exact image.

Otherwise: A new version of a base image could be deployed into production with breaking changes, causing unintended application behaviour.

8.10. Prefer smaller Docker base images

TL;DR: Large images lead to higher exposure to vulnerabilities and increased resource consumption. Using leaner Docker images, such as Slim and Alpine Linux variants, mitigates this issue.

Otherwise: Building, pushing, and pulling images will take longer, unknown attack vectors can be used by malicious actors and more resources are consumed.

🔗 Read More: Prefer smaller images

8.11. Clean-out build-time secrets, avoid secrets in args

`🌟 #new`

TL;DR: Avoid secrets leaking from the Docker build environment. A Docker image is typically shared in multiple environment like CI and a registry that are not as sanitized as production. A typical example is an npm token which is usually passed to a dockerfile as argument. This token stays within the image long after it is needed and allows the attacker indefinite access to a private npm registry. This can be avoided by coping a secret file like .npmrc and then removing it using multi-stage build (beware, build history should be deleted as well) or by using Docker build-kit secret feature which leaves zero traces

Otherwise: Everyone with access to the CI and docker registry will also get access to some precious organization secrets as a bonus

8.12. Scan images for multi layers of vulnerabilities

TL;DR: Besides checking code dependencies vulnerabilities also scan the final image that is shipped to production. Docker image scanners check the code dependencies but also the OS binaries. This E2E security scan covers more ground and verifies that no bad guy injected bad things during the build. Consequently, it is recommended running this as the last step before deployment. There are a handful of free and commercial scanners that also provide CI/CD plugins

Otherwise: Your code might be entirely free from vulnerabilities. However it might still get hacked due to vulnerable version of OS-level binaries (e.g. OpenSSL, TarBall) that are commonly being used by applications

8.13 Clean NODE_MODULE cache

TL;DR: After installing dependencies in a container remove the local cache. It doesn't make any sense to duplicate the dependencies for faster future installs since there won't be any further installs - A Docker image is immutable. Using a single line of code tens of MB (typically 10-50% of the image size) are shaved off

Otherwise: The image that will get shipped to production will weigh 30% more due to files that will never get used

8.14. Generic Docker practices

TL;DR: This is a collection of Docker advice that is not related directly to Node.js - the Node implementation is not much different than any other language. Click read more to skim through.

8.15. Lint your Dockerfile

`🌟 #new`

TL;DR: Linting your Dockerfile is an important step to identify issues in your Dockerfile which differ from best practices. By checking for potential flaws using a specialised Docker linter, performance and security improvements can be easily identified, saving countless hours of wasted time or security issues in production code.

Otherwise: Mistakenly the Dockerfile creator left Root as the production user, and also used an image from unknown source repository. This could be avoided with with just a simple linter.

🔗 Read More: Lint your Dockerfile

⬆ Return to top

Milestones

To maintain this guide and keep it up to date, we are constantly updating and improving the guidelines and best practices with the help of the community. You can follow our milestones and join the working groups if you want to contribute to this project

Translations

All translations are contributed by the community. We will be happy to get any help with either completed, ongoing or new translations!

Completed translations

Brazilian Portuguese - Courtesy of Marcelo Melo
Chinese - Courtesy of Matt Jin
Russian - Courtesy of Alex Ivanov
Polish - Courtesy of Michal Biesiada
Japanese - Courtesy of Yuki Ota, Yuta Azumi
Basque - Courtesy of Ane Diaz de Tuesta & Joxefe Diaz de Tuesta

Translations in progress

French (Discussion)
Hebrew (Discussion)
Korean - Courtesy of Sangbeom Han (Discussion)
Spanish (Discussion)
Turkish (Discussion)

Steering Committee

Meet the steering committee members - the people who work together to provide guidance and future direction to the project. In addition, each member of the committee leads a project tracked under our GitHub projects.

Yoni Goldberg

Independent Node.js consultant who works with customers in the USA, Europe, and Israel on building large-scale Node.js applications. Many of the best practices above were first published at goldbergyoni.com. Reach Yoni at @goldbergyoni or me@goldbergyoni.com

Josh Hemphill

Full Stack Software Engineer / Developer specializing in Security, DevOps/DevSecOps, and ERP Integrations.

Raz Luvaton

Full Stack Developer who knows how to exit from Vim and loves Architecture, Virtualization and Security.

Contributing

If you've ever wanted to contribute to open source, now is your chance! See the contributing docs for more information.

Contributors ✨

Thanks goes to these wonderful people who have contributed to this repository!

_{Kevin Rambaud} 🖋	_{Michael Fine} 🖋	_{Shreya Dahal} 🖋	_{Matheus Cruz Rocha} 🖋	_{Yog Mehta} 🖋	_{Kudakwashe Paradzayi} 🖋	_t1st3 🖋
_{mulijordan1976} 🖋	_{Matan Kushner} 🖋	_{Fabio Hiroki} 🖋	_{James Sumners} 🖋	_{Dan Gamble} 🖋	_{PJ Trainor} 🖋	_{Remek Ambroziak} 🖋
_{Yoni Jah} 🖋	_{Misha Khokhlov} 🖋	_{Evgeny Orekhov} 🖋	_- 🖋	_{Isaac Halvorson} 🖋	_{Vedran Karačić} 🖋	_lallenlowe 🖋
_{Nathan Wells} 🖋	_{Paulo Reis} 🖋	_syzer 🖋	_{David Sancho} 🖋	_{Robert Manolea} 🖋	_{Xavier Ho} 🖋	_Aaron 🖋
_{Jan Charles Maghirang Adona} 🖋	_Allen 🖋	_{Leonardo Villela} 🖋	_{Michał Załęcki} 🖋	_{Chris Nicola} 🖋	_{Alejandro Corredor} 🖋	_cwar 🖋
_Yuwei 🖋	_{Utkarsh Bhatt} 🖋	_{Duarte Mendes} 🖋	_{Jason Kim} 🖋	_{Mitja O.} 🖋	_{Sandro Miguel Marques} 🖋	_Gabe 🖋
_{Ron Gross} 🖋	_{Valeri Karpov} 🖋	_{Sergio Bernal} 🖋	_{Nikola Telkedzhiev} 🖋	_{Vitor Godoy} 🖋	_{Manish Saraan} 🖋	_{Sangbeom Han} 🖋
_blackmatch 🖋	_{Joe Reeve} 🖋	_{Ryan Busby} 🖋	_{Iman Mohamadi} 🖋	_{Sergii Paryzhskyi} 🖋	_{Kapil Patel} 🖋	_迷渡 🖋
_Hozefa 🖋	_Ethan 🖋	_Sam 🖋	_Arlind 🖋	_{Teddy Toussaint} 🖋	_Lewis 🖋	_{Gabriel Lidenor} 🖋
_Roman 🖋	_Francozeira 🖋	_Invvard 🖋	_{Rômulo Garofalo} 🖋	_{Tho Q Luong} 🖋	_{Burak Shen} 🖋	_{Martin Muzatko} 🖋
_{Jared Collier} 🖋	_{Hilton Meyer} 🖋	_{ChangJoo Park(박창주)} 🖋	_{Masahiro Sakaguchi} 🖋	_{Keith Holliday} 🖋	_coreyc 🖋	_{Maximilian Berkmann} 🖋
_{Douglas Mariano Valero} 🖋	_{Marcelo Melo} 🖋	_{Mehmet Perk} 🖋	_{ryan ouyang} 🖋	_Shabeer 🖋	_{Eduard Kyvenko} 🖋	_{Deyvison Rocha} 🖋
_{George Mamer} 🖋	_{Konstantinos Leimonis} 🖋	_{Oliver Lluberes} 🌍	_{Tien Do} 🖋	_{Ranvir Singh} 🖋	_{Vadim Nicolaev} 🖋 🌍	_{German Gamboa Gonzalez} 🖋
_Hafez 🖋	_Chandiran 🖋	_{VinayaSathyanarayana} 🖋	_{Kim Kern} 🖋	_{Kenneth Freitas} 🖋	_songe 🖋	_{Kirill Shekhovtsov} 🖋
_Serge 🖋	_keyrwinz 🖋	_{Dmitry Nikitenko} 🖋	_bushuai 👀 🖋	_{Benjamin Gruenbaum} 🖋	_Ezequiel 🌍	_{Juan José Rodríguez} 🌍
_{Or Bin} 🖋	_{Andreo Vieira} 🖋	_{Michael Solomon} 🖋	_{Jimmy Callin} 🖋	_Siddharth 🖋	_{Ryan Smith} 🖋	_{Tom Boettger} 🖋
_{Joaquín Ormaechea} 🌍	_dfrzuz 🌍	_{Victor Homyakov} 🖋	_Josh 🖋 🛡️	_{Alec Francis} 🖋	_arjun6610 🖋	_{Jan Osch} 🖋
_{Thiago Rotondo Sampaio} 🌍	_Alexsey 🖋	_{Luis A. Acurero} 🌍	_{Lucas Romano} 🌍	_{Denise Case} 🖋	_{Nick Ribal} 🖋 👀	_0xflotus 🖋
_{Jonathan Chen} 🖋	_{Dilan Srilal} 🖋	_{vladthelittleone} 🌍	_{Nik Osvalds} 🖋	_{Daniel Kiss} 📖	_Forresst 🖋	_{Jonathan Svenheden} 🖋
_AustrisC 🖋	_{kyeongtae kim} 🌍	₀₀₇ 🖋	_{Ane Diaz de Tuesta} 🌍 🖋	_YukiOta 🌍	_{Frazer Smith} 🖋	_{Raz Luvaton} 🖋
_{Yuta Azumi} 🖋	_{andrewjbarbour} 🖋	_mr 🖋	_Aleksandar 🖋	_Owl 🖋	_{Yedidya Schwartz} 🖋 💡	_ari 🖋
_{Thomas König} 🖋	_{Kalle Lämsä} 🖋	_Wyatt 🖋	_{KHADIR Tayeb} 🖋	_{Shankar Regmi} 🖋	_Shubham 🖋	_{Lucas Alves} 🖋
_Benjamin 🖋	_{Yeoh Joer} 🖋	_Miigon 🖋	_{Rostislav Bogorad} 🖋	_Flouse 🖋	_{Tarantini Pereira} 🖋	_{Kazuki Matsuo} 🖋
_{Adam Smith} 🖋	_{Dohyeon Ko} 🖋	_{Vladislav Legkov} 🖋	_{Kerollos Magdy} 🖋	_{Erez Lieberman} 🖋	_{Breno Macedo} 🖋	_{Fernando Flores} 🌍
_{Rafael Brito} 🌍	_{Emiliano Peralta} 🌍	_{Shin, SJ} 🖋	_{Benjamin Forster} 🖋	_{Daniele Fedeli} 🖋	_djob195 🖋	_antspk 🖋
_정진영 🖋	_kkk-cashwalk 🖋	_{apainintheneck} 🖋	_{Fajar Budhi Iswanda} 🖋	_이주호 🖋	_Singh 🖋	_{Alex Dumitru} 🖋
_{Anton Lykhatskyi} 🖋	_sangwonlee 🖋	_{Eugenio Berretta} 🖋	_soranakk 🖋	_고준영 🖋 💻	_{Guilherme Portella} 🖋	_{André Esser} 🖋
_Scc 🌍	_{Mauro Accornero} 🖋	_no-yan 🖋	_hodbauer 🌍

Steering Committee Emeriti

Bruno Scheufler

💻 full-stack web engineer, Node.js & GraphQL enthusiast

Kyle Martin

Full Stack Developer & Site Reliability Engineer based in New Zealand, interested in web application security, and architecting and building Node.js applications to perform at global scale.

Kevyn Bruyere

Independent full-stack developer with a taste for Ops and automation.

Sagir Khan

Deep specialist in JavaScript and its ecosystem — React, Node.js, TypeScript, GraphQL, MongoDB, pretty much anything that involves JS/JSON in any layer of the system — building products using the web platform for the world’s most recognized brands. Individual Member of the Node.js Foundation.

LinkedInLearning/level-up-advanced-sql-4311094

This repo is for the Linkedin Learning course: Level Up: Advanced SQL

Level Up: Advanced SQL

This is the repository for the LinkedIn Learning course Level Up: Advanced SQL. The full course is available from LinkedIn Learning.

Want to test out your advanced SQL skills? This course was designed for you. These concise challenges let you stretch your brain and push your talents to the next level. Join Microsoft MVP and SQL expert Jess Pomfret to find out more about what it takes to become a master SQL user. Learn how to execute some of the more sophisticated SQLite queries, including complex join types, grouping, advanced select options, windowing functions, and more.

This course is integrated with GitHub Codespaces, an instant development environment that offers all the functionality of your favorite IDE without the need for any local machine setup. With GitHub Codespaces, you can get hands-on practice from any machine, at any time—all while using a tool that you’ll likely encounter in the workplace.

Each installment of the Level Up series offers at least 15 bite-sized opportunities to practice programming at various levels of difficulty, so you can challenge yourself and reinforce what you’ve learned. Check out the “Using GitHub Codespaces with this course” video to learn how to get a codespace up and running.

Instructor

Jess Pomfret

Database Platform Architect and Microsoft MVP

Check out my other courses on LinkedIn Learning.

laradock/laradock

Full PHP development environment for Docker.

Full PHP development environment based on Docker.

Use Docker First - Learn About It Later!

Awesome People

Laradock is an MIT-licensed open source project with its ongoing development made possible entirely by the support of you and all these awesome people. 💜

Project Maintainers

Mahmoud Zalt @mahmoudz	Bo-Yi Wu @appleboy	Philippe Trépanier @philtrep	Mike Erickson @mikeerickson	Dwi Fahni Denni @zeroc0d3
Thor Erik @thorerik	Winfried van Loon @winfried-van-loon	TJ Miller @sixlive	Yu-Lung Shao (Allen) @bestlong	Milan Urukalo @urukalo
Vince Chu @vwchu	Huadong Zuo @zuohuadong	Lan Phan @lanphan	Ahkui @ahkui	< Join Us > @laradock

Code Contributors

Financial Contributors (Backers)

Translations

中文文档 (Chinese)

License

vulhub/vulhub

Pre-Built Vulnerable Environments Based on Docker-Compose

Vulhub is an open-source collection of pre-built, ready-to-use vulnerable Docker environments. With just one command you can launch a vulnerable environment for security research, learning, or demonstration, no prior Docker experience required.

中文版本(Chinese version)

Quick Start

Install Docker (example for Ubuntu 24.04):

# Install the latest version docker
curl -s https://get.docker.com/ | sh

# Run docker service
systemctl start docker

For other operating systems, see the Docker documentation.

Although all Vulhub environments are running based on Docker Compose, you no longer need to install docker-compose separately. Instead, you can use the built-in docker compose command to start Vulhub environments.

Download and set up Vulhub:

git clone --depth 1 https://github.com/vulhub/vulhub

Launch a vulnerable environment:

cd vulhub/langflow/CVE-2025-3248  # Example: enter a vulnerability directory
docker compose up -d

Each environment directory contains a detailed README with reproduction steps and usage instructions.

Clean up after testing:

docker compose down -v

Note

Use a VPS or VM with at least 1GB RAM for best results
The your-ip in documentation refers to your host/VPS IP, not the Docker container IP
Ensure Docker has permission to access all files in the current directory to avoid permission errors
Some environments may not support ARM architectures, see Troubleshooting for details
All environments are for testing and educational purposes only. Do not use in production!

Troubleshooting

Docker image pull fails in mainland China

Docker Hub may be inaccessible from mainland China. Use a registry mirror or run Vulhub on an overseas VPS.

Environments fail to start on Apple Silicon (M-series) Macs

Most Vulhub environments run natively on Docker Desktop for Mac with M-series chips. If an environment fails, try setting the platform explicitly:

export DOCKER_DEFAULT_PLATFORM=linux/amd64
docker compose up -d

Environments fail on Kali Linux

Some environments may fail on Kali Linux due to a low ulimit nofile setting. See the FAQ for the fix.

If you encounter issues that you cannot resolve, feel free to seek help from the community:

Contributing

We welcome contributions! Please read our Contributing Guide to get started.

Thanks to all contributors:

Partners

Our partners and users:

Sponsor Vulhub on GitHub Sponsor, OpenCollective, or Patreon 🙏

More ways to donate.

License

Vulhub is licensed under the MIT License. See LICENSE for details.

JawadTechYT/KHAN-MD

A simple WhatsApp bot to manage groups

Built on Baileys • Designed for Speed • Enhanced for Stability • Powered by Jawad TechX

Deploy New Version 15.0.0

🛰️ Deployment Steps

🚀 First, star & fork the repo using the button below:

📦 Next deploy using your preferred method ( Heroku, Railway, etc.)

📁 Clone or Fork the repository
⚙️ Add your setting variables
🚀 Deploy using platform buttons or manual setup

🚀 Pairing Dashboard

Securely link your device using the official KHAN-MD pairing portals below: if connection issues may require vpn activation to fix this issue.

🌊 GET SESSION ID

📡 DEPLOYMENT SITES

Deploy your KHAN-MD Bot with one click on your favorite hosting platforms.

🔥 TRY JAWAD MD MINI

🚀 Deploy Free on VPS

Star this repository to support the project. Fork the repository to your own GitHub account Copy your GitHub > username and paste it on the VPS site. Get your Session ID from the Pair site and add it to the VPS setup. Click on Deploy and wait about 5 minutes for the bot to start automatically.

⚠️ Note: After deployment, use the .restart command to improve the bot's performance.

✨ Keep your bot updated regularly to enjoy the latest features and patches.

KHAN-MD FEATURES

⚒️ Feature	Status 🔰
🤖 Ultimate Work	Active ✅
🔁 Anti-Delete	Active ✅
🎵 24/7 Runtime	Active ✅
📥 Downloader	Active ✅
🧠 AI Chat	Active ✅
👮 Group Setting	Active ✅
📛 Auto Sticker	Active ✅
🎮 Games	Active ✅
🌐 Web Pairing	Active ✅
🎨 Sticker Maker	Active ✅

🪀 OFFICIAL SUPPORT CHANNEL

📢 Get updates, fixes, and sneak peeks directly from the official channel.

🧾 PROJECT STATS KHAN-MD

Credits ⚒️

Creator:
JawadTechX (Official Creator)

🧠 DEVELOPER NOTICE

🚫 This project is NOT FOR SALE and is provided for educational and personal use only.

📛 Do not use without giving proper credit to the original author.

🛑 Unauthorized selling, re-uploading, or branding as your own is strictly prohibited.

📜 DISCLAIMER

📌 Read Carefully

This bot is not affiliated with WhatsApp Inc.
Use at your own risk. We are not responsible for any misuse, abuse, or ban that results from improper use.
Inspired by the original work of Balleys, and enhanced by the community.

📝 LICENSE

This project is licensed under the Apache License 2.0
You are free to use, modify, and distribute it under the terms of this license.

🙏 Thanks for using!

If you like this project, consider giving it a ⭐ on GitHub.
Made with ❤️ by JawadTechX

iggredible/Learn-Vim

Learning Vim and Vimscript doesn't have to be hard. This is the guide that you're looking for 📖

Learn Vim (the Smart Way)

What's This?

Learn Vim (the Smart Way) is a guide to learn the good parts of Vim.

There are many places to learn Vim: the vimtutor is a great place to start and the help manual has all the references you will ever need. However, the average user needs something more than vimtutor and less than the help manual. This guide attempts to bridge that gap by highlighting only the key features to learn the most useful parts of Vim in the least amount of time possible.

This guide is written for both beginner and advanced Vimmers. It starts out with broad and simple concepts and ends with specific and advanced concepts. If you're an advanced user already, I encourage you to read this guide from start to finish anyway, because you may learn something new!

Read Locally (with Docker)

You can read this book locally with Docker. Just run:

docker compose -f book/docker-compose.yml up --build

Then open http://localhost:8080.

I Want More Vim Tips!

Follow @learnvim for updates, Vim tips, etc.

Support This Project

This guide is and will always be free.

If you want to financially support this project, you can either purchase this guide on Leanpub or buy me a coffee!

Prologue

Ch 0 - Read This First

Part 1: Learn Vim the Smart Way

Part 2: Customize Vim the Smart Way

Part 3: Learn Vimscript the Smart Way

Translations

Learn-Vim 中文翻译(zh-CN)
Learn-Vim Spanish(es)

License & Copyright

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International.

odoo/docker

About this Repo

This is the Git repo of the official Docker image for Odoo. See the Hub page for the full readme on how to use the Docker image and for information regarding contributing and issues.

The full readme is generated over in docker-library/docs, specifically in docker-library/docs/odoo.

corretto/corretto-docker

Dockerfiles for Amazon Corretto Official images.

Corretto Docker

Master repository where Dockerfiles for Amazon Corretto are hosted. These docker files are used to build images for Amazon Corretto Offical Images and ECR images.

Usage

The docker images are available on Amazon Corretto Official Images

To use Amazon Corretto Official Images from Docker hub run

docker pull amazoncorretto:17

The docker images are also available on Amazon ECR.

To get Corretto docker images from Amazon ECR please see Amazon Corretto's ECR Public Gallery as well as the Docker Official Images ECR Public Gallery

To use docker images from Corretto ECR Public Gallery run the following commands:

docker pull public.ecr.aws/amazoncorretto/amazoncorretto:17
docker run -it public.ecr.aws/amazoncorretto/amazoncorretto:17 /bin/bash

You can see the list of available images by going to: https://gallery.ecr.aws/amazoncorretto/amazoncorretto

Supported Tags

See https://hub.docker.com/_/amazoncorretto

Building

To build the docker images, you can use the following command.

docker build -t amazon-corretto-{major_version} -f ./{major_version}/{jdk|jre|slim}/{al2|alpine|debian}/Dockerfile .

Security

If you would like to report a potential security issue in this project, please do not create a GitHub issue. Instead, please follow the instructions here or email AWS security directly.

Why does security scanner show that a docker image has a CVE?

If a security scanner reports that an amazoncorretto image includes a CVE, the first recommended action is to pull an updated version of this image with docker pull amazoncorretto:<tag>.

If no updated image is available, run the appropriate command to update packages for the platform in your Dockerfiles or running systems to resolve the issue immediately.

AmazonLinux 2 and many other RPM based distrobutions: yum update -y --security
AmazonLinux 2023 dnf update -y --security --releasever=latest
AlpineLinux: apk -U upgrade

If no updated package is available, please treat this as a potential security issue and follow these instructions or email AWS security directly at aws-security@amazon.com.

It is the responsibility of the base docker image supplier to provide timely security updates to images and packages. The amazoncorretto images are automatically rebuilt when a new base image is made available, but we do not make changes to our Dockerfiles to pull in one-off package updates. If a new base image has not yet been made generally available by a base docker image maintainer, please contact that maintainer to request that the issue be addressed.

Note that there are multiple reasons why a CVE may appear to be present in a docker image, as explained in the docker library FAQs.

Security scanners may use heuristics or version checks of packages compared to a security advisory to determine if an image is potentially vulnerable. The generic Linux Corretto RPMs use a slightly different version than packages built specifically for Amazon Linux, images are available for both package types. When an Amazon Linux Security Advisory (ALAS) bulliten is published it will include the Corretto package name and version that contains the fix and that version will not correctly match the generic Linux package.

Types of images provided

amazoncorretto: The default image based on Amazon Linux 2, using the Corretto generic Linux RPM packages. The Corretto packages installed support a wide range of Linux versions, and not all GUI dependencies are installed. The Corretto generic linux packages use a slightly different version scheme than native packages, which may not match exact versions posted in ALAS bulletins. However, both generic linux and native Amazon Linux packages will contain the same code. Notice: in this image, the Corretto JDK comes with its own, bundled version of certificates under lib/security/cacerts.

amazoncorretto: -alpine Based on Alpine Linux that uses musl libc, with a focus on smaller image sizes. Images are available for each supported Alpine version. When new versions of Alpine come out, a pre-built image is typically provided on the next Corretto security release after the base image is available.

amazoncorretto: -al2-native Based on Amazon Linux 2 using the Corretto RPMs specifically built for the platform using the platform’s toolchain. These will include all dependencies, and the version of the Corretto packages will match ALAS bulletins. Notice: in this image, the Corretto JDK uses the default platform certificates and lib/security/cacerts is just a simlink to /etc/pki/java/cacerts.

amazoncorretto: -al2023 Based on Amazon Linux 2023 using the Corretto RPMs specifically built for the platform using the platform’s toolchain. These will include all dependencies and the version of the Corretto packages will match ALAS bulletins. Notice: in this image, the Corretto JDK uses the default platform certificates and lib/security/cacerts is just a simlink to /etc/pki/java/cacerts.

amazoncorretto: -debian The dockerfiles are provided as examples only. Corretto is supported on apt/deb based distributions, but does not provide pre-built images.

Image subtypes

Corretto 8

jre - Contains only the runtime components and not the compiler. Suitable for most services.

jdk - Full development environment with compiler and tools.

Corretto 11+

headless - Contains runtime components without GUI libraries and dependencies. This will be the smallest image and is suitable for most services.

headful - Runtime components with GUI libraries and dependencies.

jdk - Full development environment with compiler and tools.

Version Tags

Image tags contain either just the major version or a specific security update version. Corretto 8 version tags have a format of 8u<security_update_version>, for example 8u402 . Corretto 11 and later use <major_version>.0.<security_update_version>, for example 11.0.22 . Images for a major version always point to the latest security update. Once a new security update version is released, the old tag no longer gets base image updates but remains available.

docker-library/python

Docker Official Image packaging for Python

https://github.com/docker-library/python

Maintained by: the Docker Community

This is the Git repo of the Docker "Official Image" for python (not to be confused with any official python image provided by python upstream). See the Docker Hub page for the full readme on how to use this Docker image and for information regarding contributing and issues.

The full image description on Docker Hub is generated/maintained over in the docker-library/docs repository, specifically in the python directory.

See a change merged here that doesn't show up on Docker Hub yet?

For more information about the full official images change lifecycle, see the "An image's source changed in Git, now what?" FAQ entry.

For outstanding python image PRs, check PRs with the "library/python" label on the official-images repository. For the current "source of truth" for python, see the library/python file in the official-images repository.

microsoft/mssql-docker

Official Microsoft repository for SQL Server in Docker resources

SQL Server in Docker

This GitHub repository aims to provide a centralized location for community engagement. In here you will find documentation, Dockerfiles and additional developer resources.

SQL Server in Docker comes in two different flavors:

Linux-based containers: This Docker image uses SQL Server 2017 Developer Edition on Linux on top of an Ubuntu 16.04 base image. This is meant to be run on Docker Engine on its multiple platforms. There are also Dockerfiles here for building RHEL & CentOS based images.
Windows-based containers: These Docker images use SQL Server 2017 Express Edition and SQL Server 2017 Developer Edition. Both images are based on Windows Container technology and can only be run using Docker Engine for Windows Containers. You can currently sign-up for SQL Server 2019 on Windows Containers available in our Early Adopter Preview program.

Before choosing to run a SQL Server container for production use cases, please review our support policy for SQL Server Containers to ensure that you are running on a supported configuration.

SQL Server Command Line Tools(sqlcmd,bcp) are also available as a Docker Image. You can now deliver SQL Server management payload using this as a base image for your CI/CD scenarios. Check out the mssql-tools Docker Image to get started.

Visit the Microsoft Docker Hub page for more information and additional images.

Documentation

Take our survey

Let us know more about how you would like to use SQL containers by taking our survey.

Issues

For any issues with the repo, please file under this GitHub project on the Issues section.

If you require support with a production related issue, then please raise a support incident with Microsoft here.

There is also a Gitter channel for SQL Server in DevOps that you can join and discuss interesting topics with other container, SQL Server, and DevOps enthusiasts.

Troubleshooting & Frequently Asked Questions

"Unknown blob" error code: You are probably trying to run the Windows Containers-based Docker image on a Linux-based Docker Engine. If you want to continue running the Windows Container-based image, we recommend reading the following community article: Run Linux and Windows Containers on Windows 10.
When using the Windows Docker CLI you must use double quotes instead of single ticks for the environment variables, else the mssql-server-linux image won't find the ACCEPT_EULA or SA_PASSWORD variables which are required to start the container.
The 'sa' password has a minimum complexity requirement (8 characters, uppercase, lowercase, alphanumerical and/or non-alphanumerical)
Licensing for SQL Server in Docker: Regardless of where you run it - VM, Docker, physical, cloud, on prem - the licensing model is the same and it depends on which edition of SQL Server you are using. The Express and Developer Editions are free. Standard and Enterprise have a cost. More information here: https://www.microsoft.com/en-us/sql-server/sql-server-2016-editions

License

The Docker resource files for SQL Server are licensed under the MIT license. See the LICENSE file for more details.

GitHub Dockerfile Daily Trending

piyushsachdeva/CKA-2024

Certified Kubernetes Administrator(CKA) 20245

🔗 Join our community 👇

🏆 #40daysofkubernetes Challenge - COMPLETED!

🥇 Challenge Winners

🖊️ Blogging Champions

👥 Community Champions

Day0: Intro to the series

Day 1: Docker Fundamentals

Day 2 Video: Dockerize an application

Day3 Video: Docker Multi-Stage Builds

Day 4 Video: Why do We need Kubernetes?

Day 5 Video: Kubernetes Architecture

Day 6 Video: Install Kubernetes Cluster locally

Day7 Video: Pods in Kubernetes

Day8 Video: Replicasets and Deployments in Kubernetes:

Day9 Video: Services in Kubernetes:

Day 10 Video: Namespaces:

Day 11 Video: Multi-container Pods

Day 12 Video: Daemonset, Cronjob, and job

Day13 Video: Static Pods

Day14 Video: Taints and Tolerations

Day15 Video: Node Affinity

Day16 Video: Resource Requests and Limits

Day17 Video: Autoscaling in Kubernetes

Day18 Video: Probes in Kubernetes

Day19 Video: Config maps and Secrets

Day 20 Video: How SSL/TLS works

Day 21 Video: TLS in Kubernetes

Day 22 Video: Authorization in Kubernetes

Day 23 Video: Role-based access control (RBAC)

Day 24 Video: Cluster role and cluster role binding

Day 25 Video: Service Account

Day26 Video: Network Policies

Day27 Video: Use Kubeadm to install a Kubernetes cluster

Day28 Video: Docker storage fundamentals

Day29 Video: Storage in Kubernetes

Day30 Video: How does DNS work?

Day31 Video: DNS in kubernetes

Day32 Video: Kubernetes Networking

Day 33 Video: Ingress controller and Ingress resources

Day 34 Video: Perform a version upgrade on a Kubernetes cluster using Kubeadm

Day 35 Video: Implement etcd backup and restore

Day 36 Video: Monitoring, Logging and Alerting

Day 37 Video: Troubleshoot application failure

Day 38 Video: Troubleshoot cluster component failure

Day 39 Video: Network Troubleshooting

Day 40 Video: JSONPath, advance kubectl commands

Day 41 Mission CKA

Day 42 Video: Realtime project: Host your own container registry on Kubernetes

Day 43 Video: Helm

Day 44 Video - Kustomize

Day 45 Video - StatefulSets

Day 46 Video - Priority class and preemption

Frequently asked questions:

1) 𝐈 𝐜𝐚𝐧𝐧𝐨𝐭 𝐚𝐟𝐟𝐨𝐫𝐝 𝐭𝐡𝐞 𝐂𝐊𝐀 𝐞𝐱𝐚𝐦 𝐢𝐭'𝐬 𝐭𝐨𝐨 𝐞𝐱𝐩𝐞𝐧𝐬𝐢𝐯𝐞. 𝐒𝐡𝐨𝐮𝐥𝐝 𝐈 𝐬𝐭𝐢𝐥𝐥 𝐟𝐨𝐥𝐥𝐨𝐰 𝐚𝐥𝐨𝐧𝐠?

𝟐) 𝐖𝐡𝐚𝐭 𝐚𝐫𝐞 𝐭𝐡𝐞 𝐩𝐫𝐞𝐫𝐞𝐪𝐮𝐢𝐬𝐢𝐭𝐞𝐬 𝐟𝐨𝐫 𝐭𝐡𝐢𝐬 𝐜𝐨𝐮𝐫𝐬𝐞?

𝟑) 𝐖𝐡𝐢𝐜𝐡 𝐂𝐥𝐨𝐮𝐝 𝐩𝐫𝐨𝐯𝐢𝐝𝐞𝐫 𝐚𝐫𝐞 𝐰𝐞 𝐮𝐬𝐢𝐧𝐠 𝐟𝐨𝐫 𝐭𝐡𝐞 𝐝𝐞𝐦𝐨?

𝟒) 𝐀𝐫𝐞 𝐰𝐞 𝐝𝐨𝐢𝐧𝐠 𝐚𝐧𝐲 𝐫𝐞𝐚𝐥-𝐭𝐢𝐦𝐞 𝐩𝐫𝐨𝐣𝐞𝐜𝐭𝐬 𝐢𝐧 𝐭𝐡𝐢𝐬 𝐬𝐞𝐫𝐢𝐞𝐬?

5) 𝐇𝐨𝐰 𝐦𝐚𝐧𝐲 𝐯𝐢𝐝𝐞𝐨𝐬 𝐰𝐢𝐥𝐥 𝐛𝐞 𝐩𝐮𝐛𝐥𝐢𝐬𝐡𝐞𝐝 𝐞𝐯𝐞𝐫𝐲 𝐰𝐞𝐞𝐤?

6) 𝐖𝐡𝐚𝐭 𝐢𝐟 𝐰𝐞 𝐚𝐫𝐞 𝐬𝐭𝐮𝐜𝐤 𝐬𝐨𝐦𝐞𝐰𝐡𝐞𝐫𝐞 𝐚𝐧𝐝 𝐧𝐞𝐞𝐝 𝐚𝐬𝐬𝐢𝐬𝐭𝐚𝐧𝐜𝐞?

TechHutTV/homelab

My Homelab Setup

Table of Contents

Hardware

Compute

Networking

Security

Other Devices

Stars

mikeroyal/Self-Hosting-Guide

Self Hosting Guide

A guide for getting started with Self Hosting devices including software and hardware that will make you a better and more efficient Self Hosting.

Table of Contents

Getting Started with Self-Hosting

Tools for Self-Hosting

Containers

CI/CD

Development