Tools on Producthunt daily

Prompt-Engineering-Guide

Tue, 14 Oct 2025 15:30:06 +0800

dair-ai/Prompt-Engineering-Guide

Prompt Engineering Guide

Announcements / Updates

🎓 We now offer self-paced prompt engineering courses under our DAIR.AI Academy. Join Now!
🎓 New course on Prompt Engineering for LLMs announced! Enroll here!
💼 We now offer several services like corporate training, consulting, and talks.
🌐 We now support 13 languages! Welcoming more translations.
👩‍🎓 We crossed 3 million learners in January 2024!
🎉 We have launched a new web version of the guide here
🔥 We reached #1 on Hacker News on 21 Feb 2023
🎉 The First Prompt Engineering Lecture went live here

Join our Discord

Subscribe to our YouTube

Subscribe to our Newsletter

Guides

You can also find the most up-to-date guides on our new website https://www.promptingguide.ai/.

Lecture

We have published a 1 hour lecture that provides a comprehensive overview of prompting techniques, applications, and tools.

Running the guide locally

To run the guide locally, for example to check the correct implementation of a new translation, you will need to:

Install Node >=18.0.0
Install pnpm if not present in your system. Check here for detailed instructions.
Install the dependencies: pnpm i next react react-dom nextra nextra-theme-docs
Boot the guide with pnpm dev
Browse the guide at http://localhost:3000/

Appearances

Some places where we have been featured:

Wall Street Journal - ChatGPT Can Give Great Answers. But Only If You Know How to Ask the Right Question
Forbes - Mom, Dad, I Want To Be A Prompt Engineer
Markettechpost - Best Free Prompt Engineering Resources (2023)

If you are using the guide for your work or research, please cite us as follows:

@article{Saravia_Prompt_Engineering_Guide_2022,
author = {Saravia, Elvis},
journal = {https://github.com/dair-ai/Prompt-Engineering-Guide},
month = {12},
title = {{Prompt Engineering Guide}},
year = {2022}
}

License

MIT License

Feel free to open a PR if you think something is missing here. Always welcome feedback and suggestions. Just open an issue!

cai

Thu, 25 Sep 2025 15:29:24 +0800

aliasrobotics/cai

Cybersecurity AI (`CAI`)

Cybersecurity AI (CAI) is a lightweight, open-source framework that empowers security professionals to build and deploy AI-powered offensive and defensive automation. CAI is the de facto framework for AI Security, already used by thousands of individual users and hundreds of organizations. Whether you’re a security researcher, ethical hacker, IT professional, or organization looking to enhance your security posture, CAI provides the building blocks to create specialized AI agents that can assist with mitigation, vulnerability discovery, exploitation, and security assessment.

Key Features:

🤖 300+ AI Models: Support for OpenAI, Anthropic, DeepSeek, Ollama, and more
🔧 Built-in Security Tools: Ready-to-use tools for reconnaissance, exploitation, and privilege escalation
🏆 Battle-tested: Proven in HackTheBox CTFs, bug bounties, and real-world security case studies
🎯 Agent-based Architecture: Modular framework design to build specialized agents for different security tasks
🛡️ Guardrails Protection: Built-in defenses against prompt injection and dangerous command execution
📚 Research-oriented: Research foundation to democratize cybersecurity AI for the community

[!NOTE] Read the technical report: CAI: An Open, Bug Bounty-Ready Cybersecurity AI

For further readings, refer to our impact and CAI citation sections.

`OT` - CAI and alias0 on: Ecoforest Heat Pumps	`Robotics` - CAI and alias0 on: Mobile Industrial Robots (MiR)
CAI discovers critical vulnerability in Ecoforest heat pumps allowing unauthorized remote access and potential catastrophic failures. AI-powered security testing reveals exposed credentials and DES encryption weaknesses affecting all of their deployed units across Europe.	CAI-powered security testing of MiR (Mobile Industrial Robot) platform through automated ROS message injection attacks. This study demonstrates how AI-driven vulnerability discovery can expose unauthorized access to robot control systems and alarm triggers.

`IT` (Web) - CAI and alias0 on: Mercado Libre’s e-commerce	`OT` - CAI and alias0 on: MQTT broker
CAI-powered API vulnerability discovery at Mercado Libre through automated enumeration attacks. This study demonstrates how AI-driven security testing can expose user data exposure risks in e-commerce platforms at scale.	CAI-powered testing exposed critical flaws in an MQTT broker within a Dockerized OT network. Without authentication, CAI subscribed to temperature and humidity topics and injected false values, corrupting data shown in Grafana dashboards.

[!WARNING] :warning: CAI is in active development, so don’t expect it to work flawlessly. Instead, contribute by raising an issue or sending a PR.

Access to this library and the use of information, materials (or portions thereof), is not intended, and is prohibited, where such access or use violates applicable laws or regulations. By no means the authors encourage or promote the unauthorized tampering with running systems. This can cause serious human harm and material damages.

By no means the authors of CAI encourage or promote the unauthorized tampering with compute systems. Please don’t use the source code in here for cybercrime. Pentest for good instead. By downloading, using, or modifying this source code, you agree to the terms of the LICENSE and the limitations outlined in the DISCLAIMER file.

:bookmark: Table of Contents

Cybersecurity AI (CAI)

🎯 Impact

🏆 Competitions and challenges

📊 Research Impact

Pioneered LLM-powered AI Security with PentestGPT, establishing the foundation for the Cybersecurity AI research domain
Established the Cybersecurity AI research line with 6 papers and technical reports, with active research collaborations
Demonstrated 3,600× performance improvement over human penetration testers in standardized CTF benchmark evaluations
Identified CVSS 4.3-7.5 severity vulnerabilities in production systems through automated security assessment
Democratization of AI-empowered vulnerability research: CAI enables both non-security domain experts and experienced researchers to conduct more efficient vulnerability discovery, expanding the security research community while empowering small and medium enterprises to conduct autonomous security assessments
Systematic evaluation of large language models across both proprietary and open-weight architectures, revealing substantial gaps between vendor-reported capabilities and empirical cybersecurity performance metrics
Established the autonomy levels in cybersecurity and argued about autonomy vs automation in the field
Collaborative research initiatives with international academic institutions focused on developing cybersecurity education curricula and training methodologies
Contributed a comprehensive defense framework against prompt injection in AI security agents: developed and empirically validated a multi-layered defense system that addresses the identified prompt injection issues
Explord the Cybersecurity of Humanoid Robots with CAI and identified new attack vectors showing how it (a) operates simultaneously as a covert surveillance node and (b) can be purposed as an active cyber operations platform

📚 Research products: `Cybersecurity AI`

CAI, An Open, Bug Bounty-Ready Cybersecurity AI	The Dangerous Gap Between Automation and Autonomy	CAI Fluency, A Framework for Cybersecurity AI Fluency

Hacking the AI Hackers via Prompt Injection	Humanoid Robots as Attack Vectors	The Cybersecurity of a Humanoid Robot

PoCs

CAI with `alias0` on ROS message injection attacks in MiR-100 robot	CAI with `alias0` on API vulnerability discovery at Mercado Libre

CAI on JWT@PortSwigger CTF — Cybersecurity AI	CAI on HackableII Boot2Root CTF — Cybersecurity AI

More case studies and PoCs are available at https://aliasrobotics.com/case-studies-robot-cybersecurity.php.

Motivation

:bust_in_silhouette: Why CAI?

The cybersecurity landscape is undergoing a dramatic transformation as AI becomes increasingly integrated into security operations. We predict that by 2028, AI-powered security testing tools will outnumber human pentesters. This shift represents a fundamental change in how we approach cybersecurity challenges. AI is not just another tool - it’s becoming essential for addressing complex security vulnerabilities and staying ahead of sophisticated threats. As organizations face more advanced cyber attacks, AI-enhanced security testing will be crucial for maintaining robust defenses.

This work builds upon prior efforts¹ and similarly, we believe that democratizing access to advanced cybersecurity AI tools is vital for the entire security community. That’s why we’re releasing Cybersecurity AI (CAI) as an open source framework. Our goal is to empower security researchers, ethical hackers, and organizations to build and deploy powerful AI-driven security tools. By making these capabilities openly available, we aim to level the playing field and ensure that cutting-edge security AI technology isn’t limited to well-funded private companies or state actors.

Bug Bounty programs have become a cornerstone of modern cybersecurity, providing a crucial mechanism for organizations to identify and fix vulnerabilities in their systems before they can be exploited. These programs have proven highly effective at securing both public and private infrastructure, with researchers discovering critical vulnerabilities that might have otherwise gone unnoticed. CAI is specifically designed to enhance these efforts by providing a lightweight, ergonomic framework for building specialized AI agents that can assist in various aspects of Bug Bounty hunting - from initial reconnaissance to vulnerability validation and reporting. Our framework aims to augment human expertise with AI capabilities, helping researchers work more efficiently and thoroughly in their quest to make digital systems more secure.

Ethical principles behind CAI

You might be wondering if releasing CAI in-the-wild given its capabilities and security implications is ethical. Our decision to open-source this framework is guided by two core ethical principles:

Democratizing Cybersecurity AI: We believe that advanced cybersecurity AI tools should be accessible to the entire security community, not just well-funded private companies or state actors. By releasing CAI as an open source framework, we aim to empower security researchers, ethical hackers, and organizations to build and deploy powerful AI-driven security tools, leveling the playing field in cybersecurity.
Transparency in AI Security Capabilities: Based on our research results, understanding of the technology, and dissection of top technical reports, we argue that current LLM vendors are undermining their cybersecurity capabilities. This is extremely dangerous and misleading. By developing CAI openly, we provide a transparent benchmark of what AI systems can actually do in cybersecurity contexts, enabling more informed decisions about security postures.

CAI is built on the following core principles:

Cybersecurity oriented AI framework: CAI is specifically designed for cybersecurity use cases, aiming at semi- and fully-automating offensive and defensive security tasks.
Open source, free for research: CAI is open source and free for research purposes. We aim at democratizing access to AI and Cybersecurity. For professional or commercial use, including on-premise deployments, dedicated technical support and custom extensions reach out to obtain a license.
Lightweight: CAI is designed to be fast, and easy to use.
Modular and agent-centric design: CAI operates on the basis of agents and agentic patterns, which allows flexibility and scalability. You can easily add the most suitable agents and pattern for your cybersecuritytarget case.
Tool-integration: CAI integrates already built-in tools, and allows the user to integrate their own tools with their own logic easily.
Logging and tracing integrated: using phoenix, the open source tracing and logging tool for LLMs. This provides the user with a detailed traceability of the agents and their execution.
Multi-Model Support: more than 300 supported and empowered by LiteLLM. The most popular providers:
- Anthropic: Claude 3.7, Claude 3.5, Claude 3, Claude 3 Opus
- OpenAI: O1, O1 Mini, O3 Mini, GPT-4o, GPT-4.5 Preview
- DeepSeek: DeepSeek V3, DeepSeek R1
- Ollama: Qwen2.5 72B, Qwen2.5 14B, etc

Closed-source alternatives

Cybersecurity AI is a critical field, yet many groups are misguidedly pursuing it through closed-source methods for pure economic return, leveraging similar techniques and building upon existing closed-source (often third-party owned) models. This approach not only squanders valuable engineering resources but also represents an economic waste and results in redundant efforts, as they often end up reinventing the wheel. Here are some of the closed-source initiatives we keep track of and attempting to leverage genAI and agentic frameworks in cybersecurity AI:

Learn - `CAI` Fluency

[!NOTE]

CAI Fluency technical report (arXiv:2508.13588) establishes formal educational frameworks for cybersecurity AI literacy.

	Description	English	Spanish
Episode 0: What is CAI?	Cybersecurity AI (`CAI`) explained
Episode 1: The `CAI` Framework	Vision & Ethics - Explore the core motivation behind CAI and delve into the crucial ethical principles guiding its development. Understand the motivation behind CAI and how you can actively contribute to the future of cybersecurity and the CAI framework.
Episode 2: From Zero to Cyber Hero	Breaking into Cybersecurity with AI - A comprehensive guide for complete beginners to become cybersecurity practitioners using CAI and AI tools. Learn how to leverage artificial intelligence to accelerate your cybersecurity learning journey, from understanding basic security concepts to performing real-world security assessments, all without requiring prior cybersecurity experience.
Episode 3: Vibe-Hacking Tutorial	“My first Hack” - A Vibe-Hacking guide for newbies. We demonstrate a simple web security hack using a default agent and show how to leverage tools and interpret CIA output with the help of the CAI Python API. You’ll also learn to compare different LLM models to find the best fit for your hacking endeavors.
Episode 4: Intro ReAct	The Evolution of LLMs - Learn how LLMs evolved from basic language models to advanced multiagency AI systems. From basic LLMs to Chain-of-Thought and Reasoning LLMs towards ReAct and Multi-Agent Architectures. Get to know the basic terms
Episode 5: CAI on CTF challenges	Dive into Capture The Flag (CTF) competitions using CAI. Learn how to leverage AI agents to solve various cybersecurity challenges including web exploitation, cryptography, reverse engineering, and forensics. Discover how to configure CAI for competitive hacking scenarios and maximize your CTF performance with intelligent automation.

Annex 1: `CAI` 0.5.x release	Introduce version 0.5 of `CAI` including new multi-agent functionality, new commands such as `/history`, `/compact`, `/graph` or `/memory` and a case study showing how `CAI` found a critical security flaw in OT heap pumps spread around the world.
Annex 2: `CAI` 0.4.x release and `alias0`	Introducing version 0.4 of `CAI` with streaming and improved MCP support. We also introduce `alias0`, the Privacy-First Cybersecurity AI, a Model-of-Models Intelligence that implements a Privacy-by-Design architecture and obtains state-of-the-art results in cybersecurity benchmarks.
Annex 3: Cybersecurity AI Community Meeting #1	First Cybersecurity AI (`CAI`) community meeting, over 40 participants from academia, industry, and defense gathered to discuss the open-source scaffolding behind CAI — a project designed to build agentic AI systems for cybersecurity that are open, modular, and Bug Bounty-ready.

:nut_and_bolt: Install

`1`	`pip install cai-framework`

Always create a new virtual environment to ensure proper dependency installation when updating CAI.

The following subsections provide a more detailed walkthrough on selected popular Operating Systems. Refer to the Development section for developer-related install instructions.

OS X

brew update && \
    brew install git python@3.12

# Create virtual environment
python3.12 -m venv cai_env

# Install the package from the local directory
source cai_env/bin/activate && pip install cai-framework

# Generate a .env file and set up with defaults
echo -e 'OPENAI_API_KEY="sk-1234"\nANTHROPIC_API_KEY=""\nOLLAMA=""\nPROMPT_TOOLKIT_NO_CPR=1\nCAI_STREAM=false' > .env

# Launch CAI
cai  # first launch it can take up to 30 seconds

Ubuntu 24.04

sudo apt-get update && \
    sudo apt-get install -y git python3-pip python3.12-venv

# Create the virtual environment
python3.12 -m venv cai_env

# Install the package from the local directory
source cai_env/bin/activate && pip install cai-framework

# Generate a .env file and set up with defaults
echo -e 'OPENAI_API_KEY="sk-1234"\nANTHROPIC_API_KEY=""\nOLLAMA=""\nPROMPT_TOOLKIT_NO_CPR=1\nCAI_STREAM=false' > .env

# Launch CAI
cai  # first launch it can take up to 30 seconds

Ubuntu 20.04

sudo apt-get update && \
    sudo apt-get install -y software-properties-common

# Fetch Python 3.12
sudo add-apt-repository ppa:deadsnakes/ppa && sudo apt update
sudo apt install python3.12 python3.12-venv python3.12-dev -y

# Create the virtual environment
python3.12 -m venv cai_env

# Install the package from the local directory
source cai_env/bin/activate && pip install cai-framework

# Generate a .env file and set up with defaults
echo -e 'OPENAI_API_KEY="sk-1234"\nANTHROPIC_API_KEY=""\nOLLAMA=""\nPROMPT_TOOLKIT_NO_CPR=1\nCAI_STREAM=false' > .env

# Launch CAI
cai  # first launch it can take up to 30 seconds

Windows WSL

Go to the Microsoft page: https://learn.microsoft.com/en-us/windows/wsl/install. Here you will find all the instructions to install WSL

From Powershell write: wsl –install


sudo apt-get update && \
    sudo apt-get install -y git python3-pip python3-venv

# Create the virtual environment
python3 -m venv cai_env

# Install the package from the local directory
source cai_env/bin/activate && pip install cai-framework

# Generate a .env file and set up with defaults
echo -e 'OPENAI_API_KEY="sk-1234"\nANTHROPIC_API_KEY=""\nOLLAMA=""\nPROMPT_TOOLKIT_NO_CPR=1\nCAI_STREAM=false' > .env

# Launch CAI
cai  # first launch it can take up to 30 seconds

Android

We recommend having at least 8 GB of RAM:

First of all, install userland https://play.google.com/store/apps/details?id=tech.ula&hl=es
Install Kali minimal in basic options (for free). [Or any other kali option if preferred]
Update apt keys like in this example: https://superuser.com/questions/1644520/apt-get-update-issue-in-kali, inside UserLand’s Kali terminal execute

# Get new apt keys
wget http://http.kali.org/kali/pool/main/k/kali-archive-keyring/kali-archive-keyring_2024.1_all.deb

# Install new apt keys
sudo dpkg -i kali-archive-keyring_2024.1_all.deb && rm kali-archive-keyring_2024.1_all.deb

# Update APT repository
sudo apt-get update

# CAI requieres python 3.12, lets install it (CAI for kali in Android)
sudo apt-get update && sudo apt-get install -y git python3-pip build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev libsqlite3-dev wget libbz2-dev pkg-config
wget https://www.python.org/ftp/python/3.12.4/Python-3.12.4.tar.xz
tar xf Python-3.12.4.tar.xz
cd ./configure --enable-optimizations
sudo make altinstall # This command takes long to execute

# Clone CAI's source code
git clone https://github.com/aliasrobotics/cai && cd cai

# Create virtual environment
python3.12 -m venv cai_env

# Install the package from the local directory
source cai_env/bin/activate && pip3 install -e .

# Generate a .env file and set up
cp .env.example .env  # edit here your keys/models

# Launch CAI
cai

:nut_and_bolt: Setup `.env` file

CAI leverages the .env file to load configuration at launch. To facilitate the setup, the repo provides an exemplary .env.example file provides a template for configuring CAI’s setup and your LLM API keys to work with desired LLM models.

:warning: Important:

CAI does NOT provide API keys for any model by default. Don’t ask us to provide keys, use your own or host your own models.

:warning: Note:

The OPENAI_API_KEY must not be left blank. It should contain either “sk-123” (as a placeholder) or your actual API key. See https://github.com/aliasrobotics/cai/issues/27.

:warning: Note:

If you are using alias0 model, make sure that CAI is >0.4.0 version and here you have an .env example to be able to use it.

OPENAI_API_KEY="sk-1234"
OLLAMA=""
ALIAS_API_KEY="<sk-your-key>"  # note, add yours
CAI_STEAM=False

🔹 Custom OpenAI Base URL Support

CAI supports configuring a custom OpenAI API base URL via the OPENAI_BASE_URL environment variable. This allows users to redirect API calls to a custom endpoint, such as a proxy or self-hosted OpenAI-compatible service.

Example .env entry configuration:

`1`	`OLLAMA_API_BASE="https://custom-openai-proxy.com/v1"`

Or directly from the command line:

`1`	`OLLAMA_API_BASE="https://custom-openai-proxy.com/v1" cai`

:triangular_ruler: Architecture:

CAI focuses on making cybersecurity agent coordination and execution lightweight, highly controllable, and useful for humans. To do so it builds upon 8 pillars: Agents, Tools, Handoffs, Patterns, Turns, Tracing, Guardrails and HITL.

                  ┌───────────────┐           ┌───────────┐
                  │      HITL     │◀─────────▶│   Turns   │
                  └───────┬───────┘           └───────────┘
                          │
                          ▼
┌───────────┐       ┌───────────┐       ┌───────────┐      ┌───────────┐
│  Patterns │◀─────▶│  Handoffs │◀────▶ │   Agents  │◀────▶│    LLMs   │
└───────────┘       └─────┬─────┘       └─────┬─────┘      └───────────┘
                          │                   │
                          │                   ▼
┌────────────┐       ┌────┴──────┐       ┌───────────┐     ┌────────────┐
│ Extensions │◀─────▶│  Tracing  │       │   Tools   │◀───▶│ Guardrails │
└────────────┘       └───────────┘       └───────────┘     └────────────┘
                                              │
                          ┌─────────────┬─────┴────┬─────────────┐
                          ▼             ▼          ▼             ▼
                    ┌───────────┐┌───────────┐┌────────────┐┌───────────┐
                    │ LinuxCmd  ││ WebSearch ││    Code    ││ SSHTunnel │
                    └───────────┘└───────────┘└────────────┘└───────────┘

If you want to dive deeper into the code, check the following files as a start point for using CAI:

init.py
cli.py - entrypoint for command line interface
util.py - utility functions
agents - Agent implementations
internal - CAI internal functions (endpoints, metrics, logging, etc.)
prompts - Agent Prompt Database
repl - CLI aesthetics and commands
sdk - CAI command sdk
tools - agent tools

🔹 Agent

At its core, CAI abstracts its cybersecurity behavior via Agents and agentic Patterns. An Agent in an intelligent system that interacts with some environment. More technically, within CAI we embrace a robotics-centric definition wherein an agent is anything that can be viewed as a system perceiving its environment through sensors, reasoning about its goals and and acting accordingly upon that environment through actuators (adapted from Russel & Norvig, AI: A Modern Approach). In cybersecurity, an Agent interacts with systems and networks, using peripherals and network interfaces as sensors, reasons accordingly and then executes network actions as if actuators. Correspondingly, in CAI, Agents implement the ReACT (Reasoning and Action) agent model². For more information, see the example here for the full execution code, and refer to this jupyter notebook for a tutorial on how to use it.

from cai.sdk.agents import Agent, Runner, OpenAIChatCompletionsModel

import os
from openai import AsyncOpenAI
from dotenv import load_dotenv
load_dotenv()

agent = Agent(
      name="Custom Agent",
      instructions="""You are a Cybersecurity expert Leader""",
      model=OpenAIChatCompletionsModel(
          model=os.getenv('CAI_MODEL', "openai/gpt-4o"),
          openai_client=AsyncOpenAI(),
          )
      )

message = "Tell me about recursion in programming."
result = await Runner.run(agent, message)

🔹 Tools

Tools let cybersecurity agents take actions by providing interfaces to execute system commands, run security scans, analyze vulnerabilities, and interact with target systems and APIs - they are the core capabilities that enable CAI agents to perform security tasks effectively; in CAI, tools include built-in cybersecurity utilities (like LinuxCmd for command execution, WebSearch for OSINT gathering, Code for dynamic script execution, and SSHTunnel for secure remote access), function calling mechanisms that allow integration of any Python function as a security tool, and agent-as-tool functionality that enables specialized security agents (such as reconnaissance or exploit agents) to be used by other agents, creating powerful collaborative security workflows without requiring formal handoffs between agents. For more information, please refer to the example here for the complete configuration of custom functions.

from cai.sdk.agents import Agent, Runner, OpenAIChatCompletionsModel
from cai.tools.reconnaissance.exec_code import execute_code
from cai.tools.reconnaissance.generic_linux_command import generic_linux_command

import os
from openai import AsyncOpenAI
from dotenv import load_dotenv
load_dotenv()

agent = Agent(
      name="Custom Agent",
      instructions="""You are a Cybersecurity expert Leader""",
      tools= [
        generic_linux_command,
        execute_code
      ],
      model=OpenAIChatCompletionsModel(
          model=os.getenv('CAI_MODEL', "openai/gpt-4o"),
          openai_client=AsyncOpenAI(),
          )
      )

message = "Tell me about recursion in programming."
result = await Runner.run(agent, message)

You may find different tools. They are grouped in 6 major categories inspired by the security kill chain ³:

Reconnaissance and weaponization - reconnaissance (crypto, listing, etc)
Exploitation - exploitation
Privilege escalation - escalation
Lateral movement - lateral
Data exfiltration - exfiltration
Command and control - control

🔹 Handoffs

Handoffs allow an Agent to delegate tasks to another agent, which is crucial in cybersecurity operations where specialized expertise is needed for different phases of an engagement. In our framework, Handoffs are implemented as tools for the LLM, where a handoff/transfer function like transfer_to_flag_discriminator enables the ctf_agent to pass control to the flag_discriminator_agent once it believes it has found the flag. This creates a security validation chain where the first agent handles exploitation and flag discovery, while the second agent specializes in flag verification, ensuring proper segregation of duties and leveraging specialized capabilities of different models for distinct security tasks. For more information, please refer to the example here for the full execution code.

from cai.sdk.agents import function_tool
from cai.tools.common import run_command
from cai.sdk.agents import Agent, OpenAIChatCompletionsModel, HandoffInputData, Runner, function_tool, handoff, trace
from cai.sdk.agents.extensions import handoff_filters

import os
from openai import AsyncOpenAI
from dotenv import load_dotenv
load_dotenv()

@function_tool
def execute_cli_command(command: str) -> str:
    return run_command(command)

flag_discriminator = Agent(
    name="Flag discriminator",
    description="Agent focused on extracting the flag from the output",
    instructions="You are an agent tailored to extract the flag from a given output.",
    model=OpenAIChatCompletionsModel(
        model=os.getenv('CAI_MODEL', "qwen2.5:14b"),
        openai_client=AsyncOpenAI(),
    ) 
)

ctf_agent = Agent(
    name="CTF agent",
    description="Agent focused on conquering security challenges",
    instructions="You are a Cybersecurity expert Leader facing a CTF",
    tools=[
        execute_cli_command,
    ],
    model=OpenAIChatCompletionsModel(
        model= os.getenv('CAI_MODEL', "qwen2.5:14b"),
        openai_client=AsyncOpenAI(),
    ), 
    handoffs = [flag_discriminator]
)

🔹 Patterns

An agentic Pattern is a structured design paradigm in artificial intelligence systems where autonomous or semi-autonomous agents operate within a defined interaction framework (the pattern) to achieve a goal. These Patterns specify the organization, coordination, and communication methods among agents, guiding decision-making, task execution, and delegation.

An agentic pattern (AP) can be formally defined as a tuple:

\[ AP = (A, H, D, C, E) \]

wherein:

$A$ (Agents): A set of autonomous entities, $ A = \{a_1, a_2, …, a_n\} $, each with defined roles, capabilities, and internal states.
$H$ (Handoffs): A function $ H: A \times T \to A $ that governs how tasks $ T $ are transferred between agents based on predefined logic (e.g., rules, negotiation, bidding).
$D$ (Decision Mechanism): A decision function $ D: S \to A $ where $ S $ represents system states, and $ D $ determines which agent takes action at any given time.
$C$ (Communication Protocol): A messaging function $ C: A \times A \to M $, where $ M $ is a message space, defining how agents share information.
$E$ (Execution Model): A function $ E: A \times I \to O $ where $ I $ is the input space and $ O $ is the output space, defining how agents perform tasks.

When building Patterns, we generall y classify them among one of the following categories, though others exist:

Agentic `Pattern` categories	Description
`Swarm` (Decentralized)	Agents share tasks and self-assign responsibilities without a central orchestrator. Handoffs occur dynamically. An example of a peer-to-peer agentic pattern is the `CTF Agentic Pattern`, which involves a team of agents working together to solve a CTF challenge with dynamic handoffs.
`Hierarchical`	A top-level agent (e.g., “PlannerAgent”) assigns tasks via structured handoffs to specialized sub-agents. Alternatively, the structure of the agents is harcoded into the agentic pattern with pre-defined handoffs.
`Chain-of-Thought` (Sequential Workflow)	A structured pipeline where Agent A produces an output, hands it to Agent B for reuse or refinement, and so on. Handoffs follow a linear sequence. An example of a chain-of-thought agentic pattern is the `ReasonerAgent`, which involves a Reasoning-type LLM that provides context to the main agent to solve a CTF challenge with a linear sequence.⁴
`Auction-Based` (Competitive Allocation)	Agents “bid” on tasks based on priority, capability, or cost. A decision agent evaluates bids and hands off tasks to the best-fit agent.
`Recursive`	A single agent continuously refines its own output, treating itself as both executor and evaluator, with handoffs (internal or external) to itself. An example of a recursive agentic pattern is the `CodeAgent` (when used as a recursive agent), which continuously refines its own output by executing code and updating its own instructions.

For more information and examples of common agentic patterns, see the examples folder.

🔹 Turns and Interactions

During the agentic flow (conversation), we distinguish between interactions and turns.

Interactions are sequential exchanges between one or multiple agents. Each agent executing its logic corresponds with one interaction. Since an Agent in CAI generally implements the ReACT agent model², each interaction consists of 1) a reasoning step via an LLM inference and 2) act by calling zero-to-n Tools. This is defined inprocess_interaction() in core.py.
Turns: A turn represents a cycle of one ore more interactions which finishes when the Agent (or Pattern) executing returns None, judging there’re no further actions to undertake. This is defined in run(), see core.py.

[!NOTE] CAI Agents are not related to Assistants in the Assistants API. They are named similarly for convenience, but are otherwise completely unrelated. CAI is entirely powered by the Chat Completions API and is hence stateless between calls.

🔹 Tracing

CAI implements AI observability by adopting the OpenTelemetry standard and to do so, it leverages Phoenix which provides comprehensive tracing capabilities through OpenTelemetry-based instrumentation, allowing you to monitor and analyze your security operations in real-time. This integration enables detailed visibility into agent interactions, tool usage, and attack vectors throughout penetration testing workflows, making it easier to debug complex exploitation chains, track vulnerability discovery processes, and optimize agent performance for more effective security assessments.

🔹 Guardrails

Guardrails provide a critical security layer for CAI agents, protecting against prompt injection attacks and preventing execution of dangerous commands. These guardrails run in parallel to agents, validating both input and output to ensure safe operation. The framework includes:

Input Guardrails: Detect and block prompt injection attempts before they reach agents, using pattern matching, Unicode homograph detection, and AI-powered analysis
Output Guardrails: Validate agent outputs before execution, preventing dangerous commands like reverse shells, fork bombs, or data exfiltration
Multi-layered Defense: Protection at input, processing, and execution stages with tool-level validation
Base64/Base32 Aware: Automatically decodes and analyzes encoded payloads to detect hidden malicious commands
Configurable: Can be enabled/disabled via CAI_GUARDRAILS environment variable

For detailed implementation, see docs/guardrails.md and docs/cai_prompt_injection.md.

🔹 Human-In-The-Loop (HITL)

                      ┌─────────────────────────────────┐
                      │                                 │
                      │      Cybersecurity AI (CAI)     │
                      │                                 │
                      │       ┌─────────────────┐       │
                      │       │  Autonomous AI  │       │
                      │       └────────┬────────┘       │
                      │                │                │
                      │                │                │
                      │       ┌────────▼─────────┐      │
                      │       │ HITL Interaction │      │
                      │       └────────┬─────────┘      │
                      │                │                │
                      └────────────────┼────────────────┘
                                       │
                                       │ Ctrl+C (cli.py)
                                       │
                           ┌───────────▼───────────┐
                           │   Human Operator(s)   │
                           │  Expertise | Judgment │
                           │    Teleoperation      │
                           └───────────────────────┘

CAI delivers a framework for building Cybersecurity AIs with a strong emphasis on semi-autonomous operation, as the reality is that fully-autonomous cybersecurity systems remain premature and face significant challenges when tackling complex tasks. While CAI explores autonomous capabilities, we recognize that effective security operations still require human teleoperation providing expertise, judgment, and oversight in the security process.

Accordingly, the Human-In-The-Loop (HITL) module is a core design principle of CAI, acknowledging that human intervention and teleoperation are essential components of responsible security testing. Through the cli.py interface, users can seamlessly interact with agents at any point during execution by simply pressing Ctrl+C. This is implemented across core.py and also in the REPL abstractions REPL.

:rocket: Quickstart

To start CAI after installing it, just type cai in the CLI:

└─# cai

          CCCCCCCCCCCCC      ++++++++   ++++++++      IIIIIIIIII
       CCC::::::::::::C  ++++++++++       ++++++++++  I::::::::I
     CC:::::::::::::::C ++++++++++         ++++++++++ I::::::::I
    C:::::CCCCCCCC::::C +++++++++    ++     +++++++++ II::::::II
   C:::::C       CCCCCC +++++++     +++++     +++++++   I::::I
  C:::::C                +++++     +++++++     +++++    I::::I
  C:::::C                ++++                   ++++    I::::I
  C:::::C                 ++                     ++     I::::I
  C:::::C                  +   +++++++++++++++   +      I::::I
  C:::::C                    +++++++++++++++++++        I::::I
  C:::::C                     +++++++++++++++++         I::::I
   C:::::C       CCCCCC        +++++++++++++++          I::::I
    C:::::CCCCCCCC::::C         +++++++++++++         II::::::II
     CC:::::::::::::::C           +++++++++           I::::::::I
       CCC::::::::::::C             +++++             I::::::::I
          CCCCCCCCCCCCC               ++              IIIIIIIIII

                      Cybersecurity AI (CAI), vX.Y.Z
                          Bug bounty-ready AI

CAI>

That should initialize CAI and provide a prompt to execute any security task you want to perform. The navigation bar at the bottom displays important system information. This information helps you understand your environment while working with CAI.

Here’s a quick demo video to help you get started with CAI. We’ll walk through the basic steps — from launching the tool to running your first AI-powered task in the terminal. Whether you’re a beginner or just curious, this guide will show you how easy it is to begin using CAI.

From here on, type on CAI and start your security exercise. Best way to learn is by example:

Environment Variables

For using private models, you are given a .env.example file. Copy it and rename it as .env. Fill in your corresponding API keys, and you are ready to use CAI.

List of Environment Variables

Variable	Description
CTF_NAME	Name of the CTF challenge to run (e.g. “picoctf_static_flag”)
CTF_CHALLENGE	Specific challenge name within the CTF to test
CTF_SUBNET	Network subnet for the CTF container
CTF_IP	IP address for the CTF container
CTF_INSIDE	Whether to conquer the CTF from within container
CAI_MODEL	Model to use for agents
CAI_DEBUG	Set debug output level (0: Only tool outputs, 1: Verbose debug output, 2: CLI debug output)
CAI_BRIEF	Enable/disable brief output mode
CAI_MAX_TURNS	Maximum number of turns for agent interactions
CAI_TRACING	Enable/disable OpenTelemetry tracing
CAI_AGENT_TYPE	Specify the agents to use (boot2root, one_tool…)
CAI_STATE	Enable/disable stateful mode
CAI_MEMORY	Enable/disable memory mode (episodic, semantic, all)
CAI_MEMORY_ONLINE	Enable/disable online memory mode
CAI_MEMORY_OFFLINE	Enable/disable offline memory
CAI_ENV_CONTEXT	Add dirs and current env to llm context
CAI_MEMORY_ONLINE_INTERVAL	Number of turns between online memory updates
CAI_PRICE_LIMIT	Price limit for the conversation in dollars
CAI_REPORT	Enable/disable reporter mode (ctf, nis2, pentesting)
CAI_SUPPORT_MODEL	Model to use for the support agent
CAI_SUPPORT_INTERVAL	Number of turns between support agent executions
CAI_WORKSPACE	Defines the name of the workspace
CAI_WORKSPACE_DIR	Specifies the directory path where the workspace is located
CAI_GUARDRAILS	Enable/disable guardrails for prompt injection protection (default: true)

OpenRouter Integration

The Cybersecurity AI (CAI) platform offers seamless integration with OpenRouter, a unified interface for Large Language Models (LLMs). This integration is crucial for users who wish to leverage advanced AI capabilities in their cybersecurity tasks. OpenRouter acts as a bridge, allowing CAI to communicate with various LLMs, thereby enhancing the flexibility and power of the AI agents used within CAI.

To enable OpenRouter support in CAI, you need to configure your environment by adding specific entries to your .env file. This setup ensures that CAI can interact with the OpenRouter API, facilitating the use of sophisticated models like Meta-LLaMA. Here’s how you can configure it:

CAI_AGENT_TYPE=redteam_agent
CAI_MODEL=openrouter/meta-llama/llama-4-maverick
OPENROUTER_API_KEY=<sk-your-key>  # note, add yours
OPENROUTER_API_BASE=https://openrouter.ai/api/v1

Azure OpenAI

The Cybersecurity AI (CAI) platform integrates seamlessly with Azure OpenAI, enabling organizations to run CAI against enterprise-hosted models (e.g., gpt-4o). This pathway is ideal for teams that must operate within Azure governance while leveraging advanced model capabilities. To enable Azure OpenAI support in CAI, configure your environment by adding the following entries to your .env. This ensures CAI can reach your Azure deployment endpoint and authenticate correctly.

CAI_AGENT_TYPE=redteam_agent
CAI_MODEL=azure/<model-name-deployed>
# Required: keep non-empty even when using Azure
OPENAI_API_KEY=dummy
# Azure credentials and endpoint
AZURE_API_KEY=<your-azure-openai-key>
AZURE_API_BASE=https://<resource>.openai.azure.com/openai/deployments/<deployment-name>/chat/completions?api-version=2025-01-01-preview

MCP

CAI supports the Model Context Protocol (MCP) for integrating external tools and services with AI agents. MCP is supported via two transport mechanisms:

SSE (Server-Sent Events) - For web-based servers that push updates over HTTP connections:

`1`	`CAI>/mcp load http://localhost:9876/sse burp`

STDIO (Standard Input/Output) - For local inter-process communication:

`1`	`CAI>/mcp load stdio myserver python mcp_server.py`

Once connected, you can add the MCP tools to any agent:

CAI>/mcp add burp redteam_agent
Adding tools from MCP server 'burp' to agent 'Red Team Agent'...
                                 Adding tools to Red Team Agent
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Tool                              ┃ Status ┃ Details                                         ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ send_http_request                 │ Added  │ Available as: send_http_request                 │
│ create_repeater_tab               │ Added  │ Available as: create_repeater_tab               │
│ send_to_intruder                  │ Added  │ Available as: send_to_intruder                  │
│ url_encode                        │ Added  │ Available as: url_encode                        │
│ url_decode                        │ Added  │ Available as: url_decode                        │
│ base64encode                      │ Added  │ Available as: base64encode                      │
│ base64decode                      │ Added  │ Available as: base64decode                      │
│ generate_random_string            │ Added  │ Available as: generate_random_string            │
│ output_project_options            │ Added  │ Available as: output_project_options            │
│ output_user_options               │ Added  │ Available as: output_user_options               │
│ set_project_options               │ Added  │ Available as: set_project_options               │
│ set_user_options                  │ Added  │ Available as: set_user_options                  │
│ get_proxy_http_history            │ Added  │ Available as: get_proxy_http_history            │
│ get_proxy_http_history_regex      │ Added  │ Available as: get_proxy_http_history_regex      │
│ get_proxy_websocket_history       │ Added  │ Available as: get_proxy_websocket_history       │
│ get_proxy_websocket_history_regex │ Added  │ Available as: get_proxy_websocket_history_regex │
│ set_task_execution_engine_state   │ Added  │ Available as: set_task_execution_engine_state   │
│ set_proxy_intercept_state         │ Added  │ Available as: set_proxy_intercept_state         │
│ get_active_editor_contents        │ Added  │ Available as: get_active_editor_contents        │
│ set_active_editor_contents        │ Added  │ Available as: set_active_editor_contents        │
└───────────────────────────────────┴────────┴─────────────────────────────────────────────────┘
Added 20 tools from server 'burp' to agent 'Red Team Agent'.
CAI>/agent 13
CAI>Create a repeater tab

You can list all active MCP connections and their transport types:

`1`	`CAI>/mcp list`

https://github.com/user-attachments/assets/386a1fd3-3469-4f84-9396-2a5236febe1f

Development

Development is facilitated via VS Code dev. environments. To try out our development environment, clone the repository, open VS Code and enter de dev. container mode:

Contributions

If you want to contribute to this project, use Pre-commit before your MR

1
2
3

pip install pre-commit
pre-commit # files staged
pre-commit run --all-files # all files

Optional Requirements: caiextensions

Currently, the extensions are not publicly available as the engineering endeavour to maintain them is significant. Instead, we’re making selected custom caiextensions available for partner companies across collaborations.

:information_source: Usage Data Collection

CAI is provided free of charge for researchers. To improve CAI’s detection accuracy and publish open security research, instead of payment for research use cases, we ask you to contribute to the CAI community by allowing usage data collection. This data helps us identify areas for improvement, understand how the framework is being used, and prioritize new features. Legal basis of data collection is under Art. 6 (1)(f) GDPR — CAI’s legitimate interest in maintaining and improving security tooling, with Art. 89 safeguards for research. The collected data includes:

Basic system information (OS type, Python version)
Username and IP information
Tool usage patterns and performance metrics
Model interactions and token usage statistics

We take your privacy seriously and only collect what’s needed to make CAI better. For further info, reach out to research＠aliasrobotics.com. You can disable some of the data collection features via the CAI_TELEMETRY environment variable but we encourage you to keep it enabled and contribute back to research:

`1`	`CAI_TELEMETRY=False cai`

Reproduce CI-Setup locally

To simulate the CI/CD pipeline, you can run the following in the Gitlab runner machines:

docker run --rm -it \
  --privileged \
  --network=exploitflow_net \
  --add-host="host.docker.internal:host-gateway" \
  -v /cache:/cache \
  -v /var/run/docker.sock:/var/run/docker.sock:rw \
  registry.gitlab.com/aliasrobotics/alias_research/cai:latest bash

FAQ

OLLAMA is giving me 404 errors

Ollama’s API in OpenAI mode uses /v1/chat/completions whereas the openai library uses base_url + /chat/completions.

We adopt the latter for overall alignment with the gen AI community and empower the former by allowing users to add the v1 themselves via:

`1`	`OLLAMA_API_BASE=http://IP:PORT/v1`

See the following issues that treat this topic in more detail:

Where are all the caiextensions?

See all caiextensions

How do I install the report caiextension?

See here

How do I set up SSH access for Gitlab?

Generate a new SSH key

`1`	`ssh-keygen -t ed25519`

Add the key to the SSH agent

`1`	`ssh-add ~/.ssh/id_ed25519`

Add the public key to Gitlab Copy the key and add it to Gitlab under https://gitlab.com/-/user_settings/ssh_keys

`1`	`cat ~/.ssh/id_ed25519.pub`

To verify it:

1
2

ssh -T git@gitlab.com
Welcome to GitLab, @vmayoral!

How do I clear Python cache?

`1`	`find . -name "*.pyc" -delete && find . -name "__pycache__" -delete`

If host networking is not working with ollama check whether it has been disabled in Docker because you are not signed in

Docker in OS X behaves funny sometimes. Check if the following message has shown up:

Host networking has been disabled because you are not signed in. Please sign in to enable it.

Make sure this has been addressed and also that the Dev Container is not forwarding the 8000 port (click on x, if necessary in the ports section).

To verify connection, from within the VSCode devcontainer:

`1`	`curl -v http://host.docker.internal:8000/api/version`

Run CAI against any target

The starting user prompt in this case is: Target IP: 192.168.3.10, perform a full network scan.

The agent started performing a nmap scan. You could either interact with the agent and give it more instructions, or let it run to see what it explores next.

How do I interact with the agent? Type twice CTRL + C

If you want to use the HITL mode, you can do it by presssing twice Ctrl + C. This will allow you to interact (prompt) with the agent whenever you want. The agent will not lose the previous context, as it is stored in the history variable, which is passed to it and any agent that is called. This enables any agent to use the previous information and be more accurate and efficient.

Can I change the model while CAI is running? /model

Use /model to change the model.

How can I list all the agents available? /agent

Use /agent to list all the agents available.

Where can I list all the environment variables? /config

How to know more about the CLI? /help

How can I trace the whole execution?

The environment variable `CAI_TRACING` allows the user to set it to `CAI_TRACING=true` to enable tracing, or `CAI_TRACING=false` to disable it. When CAI is prompted by the first time, the user is provided with two paths, the execution log, and the tracing log.

Can I expand CAI capabilities using previous run logs?

Yes. Today CAI performs best by relying on In‑Context Learning (ICL). Rather than building long‑term stores, the recommended workflow is to load relevant prior logs directly into the current session so the model can reason with them in context.

Use the /load command to bring JSONL logs into CAI’s context (this replaces the legacy memory-loading tool):

CAI>/load logs/cai_20250408_111856.jsonl         # Load into current agent
CAI>/load <file> agent <name>                    # Load into a specific agent
CAI>/load <file> all                             # Distribute across all agents
CAI>/load <file> parallel                        # Match to configured parallel agents
# Tip: if you omit <file>, /load uses `logs/last`. Alias: /l

CAI prints the path to the current run’s JSONL log at startup (highlighted in orange), which you can pass to /load:

Legacy notes: earlier “memory extension” mechanisms (episodic/semantic stores and offline ingestion) are retained for reference only. See src/cai/agents/memory.py for background and legacy details. Our current direction prioritizes ICL over persistent memory.

Can I expand CAI capabilities using scripts or extra information?

Currently, CAI supports text based information. You can add any extra information on the target you are facing by copy-pasting it directly into the system or user prompt.

How? By adding it to the system (system_master_template.md) or the user prompt (user_master_template.md). You can always directly prompt the path to the model, and it will cat it.

How CAI licence works?

CAI’s current license does not restrict usage for research purposes. You are free to use CAI for security assessments (pentests), to develop additional features, and to integrate it into your research activities, as long as you comply with local laws.

If you or your organization start benefiting commercially from CAI (e.g., offering pentesting services powered by CAI), then a commercial license will be required to help sustain the project.

CAI itself is not a profit-seeking initiative. Our goal is to build a sustainable open-source project. We simply ask that those who profit from CAI contribute back and support our ongoing development.

I get a `Unable to locate package python3.12-venv` when installing the prerequisites on my debian based system!

The easiest way to get around this is to simply install python3.12 from source.

Citation

If you want to cite our work, please use the following:

@misc{mayoralvilches2025caiopenbugbountyready,
      title={CAI: An Open, Bug Bounty-Ready Cybersecurity AI},
      author={Víctor Mayoral-Vilches and Luis Javier Navarrete-Lozano and María Sanz-Gómez and Lidia Salas Espejo and Martiño Crespo-Álvarez and Francisco Oca-Gonzalez and Francesco Balassone and Alfonso Glera-Picón and Unai Ayucar-Carbajo and Jon Ander Ruiz-Alcalde and Stefan Rass and Martin Pinzger and Endika Gil-Uriarte},
      year={2025},
      eprint={2504.06017},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2504.06017},
}

@misc{mayoralvilches2025cybersecurityaidangerousgap,
      title={Cybersecurity AI: The Dangerous Gap Between Automation and Autonomy}, 
      author={Víctor Mayoral-Vilches},
      year={2025},
      eprint={2506.23592},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2506.23592}, 
}

@misc{mayoralvilches2025caifluencyframeworkcybersecurity,
      title={CAI Fluency: A Framework for Cybersecurity AI Fluency}, 
      author={Víctor Mayoral-Vilches and Jasmin Wachter and Cristóbal R. J. Veas Chavez and Cathrin Schachner and Luis Javier Navarrete-Lozano and María Sanz-Gómez},
      year={2025},
      eprint={2508.13588},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2508.13588}, 
}

@misc{mayoralvilches2025cybersecurityaihackingai,
      title={Cybersecurity AI: Hacking the AI Hackers via Prompt Injection}, 
      author={Víctor Mayoral-Vilches and Per Mannermaa Rynning},
      year={2025},
      eprint={2508.21669},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2508.21669}, 
}

Acknowledgements

CAI was initially developed by Alias Robotics and co-funded by the European EIC accelerator project RIS (GA 101161136) - HORIZON-EIC-2023-ACCELERATOR-01 call. The original agentic principles are inspired from OpenAI’s swarm library and translated into newer prototypes. This project also makes use of other relevant open source building blocks including LiteLLM, and phoenix

Academic Collaborations

CAI benefits from ongoing research collaborations with academic institutions. Researchers interested in collaborative projects, dataset access, or academic licenses should contact research@aliasrobotics.com. We provide special support for:

PhD research projects
Academic benchmarking studies
Security education initiatives
Open-source contributions from research labs

Deng, G., Liu, Y., Mayoral-Vilches, V., Liu, P., Li, Y., Xu, Y., … & Rass, S. (2024). {PentestGPT}: Evaluating and harnessing large language models for automated penetration testing. In 33rd USENIX Security Symposium (USENIX Security 24) (pp. 847-864). ↩︎
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2023, January). React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR). ↩︎ ↩︎
Kamhoua, C. A., Leslie, N. O., & Weisman, M. J. (2018). Game theoretic modeling of advanced persistent threat in internet of things. Journal of Cyber Security and Information Systems. ↩︎
Arguably, the Chain-of-Thought agentic pattern is a special case of the Hierarchical agentic pattern. ↩︎

koog

Wed, 03 Sep 2025 15:26:34 +0800

JetBrains/koog

Koog

Overview

Koog is a Kotlin-based framework designed to build and run AI agents entirely in idiomatic Kotlin. It lets you create agents that can interact with tools, handle complex workflows, and communicate with users.

Key features

Key features of Koog include:

Pure Kotlin implementation: Build AI agents entirely in natural and idiomatic Kotlin.
MCP integration: Connect to Model Context Protocol for enhanced model management.
Embedding capabilities: Use vector embeddings for semantic search and knowledge retrieval.
Custom tool creation: Extend your agents with tools that access external systems and APIs.
Ready-to-use components: Speed up development with pre-built solutions for common AI engineering challenges.
Intelligent history compression: Optimize token usage while maintaining conversation context using various pre-built strategies.
Powerful Streaming API: Process responses in real-time with streaming support and parallel tool calls.
Persistent agent memory: Enable knowledge retention across sessions and even different agents.
Comprehensive tracing: Debug and monitor agent execution with detailed and configurable tracing.
Flexible graph workflows: Design complex agent behaviors using intuitive graph-based workflows.
Modular feature system: Customize agent capabilities through a composable architecture.
Scalable architecture: Handle workloads from simple chatbots to enterprise applications.
Multiplatform: Run agents on JVM, JS, WasmJS, iOS targets with Kotlin Multiplatform.

Available LLM providers and platforms

The LLM providers and platforms whose LLMs you can use to power your agent capabilities:

Google
OpenAI
Anthropic
OpenRouter
Ollama

Quickstart example

To help you get started with AI agents, here is a quick example:

fun main() = runBlocking {
    // Before you run the example, assign a corresponding API key as an environment variable.
   val apiKey = System.getenv("OPENAI_API_KEY") // or Anthropic, Google, OpenRouter, etc.

   val agent = AIAgent(
      executor = simpleOpenAIExecutor(apiKey), // or Anthropic, Google, OpenRouter, etc.
      systemPrompt = "You are a helpful assistant. Answer user questions concisely.",
      llmModel = OpenAIModels.Chat.GPT4o
   )

   val result = agent.run("Hello! How can you help me?")
   println(result)
}

Using in your projects

Supported targets

Currently, the framework supports the JVM, JS, WasmJS and iOS targets.

On JVM, JDK 17 or higher is required to use the framework.

Please check the libs.versions.toml to know more about the Koog dependencies.

Gradle (Kotlin DSL)

Add dependencies to the build.gradle.kts file:

1
2
3

dependencies {
    implementation("ai.koog:koog-agents:0.4.1")
}

Make sure that you have mavenCentral() in the list of repositories.

Gradle (Groovy)

Add dependencies to the build.gradle file:

1
2
3

dependencies {
    implementation 'ai.koog:koog-agents:0.4.1'
}

Make sure that you have mavenCentral() in the list of repositories.

Maven

Add dependencies to the pom.xml file:

<dependency>
    <groupId>ai.koog</groupId>
    <artifactId>koog-agents-jvm</artifactId>
    <version>0.4.1</version>
</dependency>

Make sure that you have mavenCentral in the list of repositories.

Contributing

Read the Contributing Guidelines.

Code of Conduct

This project and the corresponding community are governed by the JetBrains Open Source and Community Code of Conduct. Please make sure you read it.

License

Koog is licensed under the Apache 2.0 License.

Support

Please feel free to ask any questions in our official Slack channel (link) and to use Koog official YouTrack project for filing feature requests and bug reports.

genai-toolbox

Thu, 28 Aug 2025 15:29:14 +0800

googleapis/genai-toolbox

MCP Toolbox for Databases

[!NOTE] MCP Toolbox for Databases is currently in beta, and may see breaking changes until the first stable release (v1.0).

MCP Toolbox for Databases is an open source MCP server for databases. It enables you to develop tools easier, faster, and more securely by handling the complexities such as connection pooling, authentication, and more.

This README provides a brief overview. For comprehensive details, see the full documentation.

[!NOTE] This solution was originally named “Gen AI Toolbox for Databases” as its initial development predated MCP, but was renamed to align with recently added MCP compatibility.

Why Toolbox?

Toolbox helps you build Gen AI tools that let your agents access data in your database. Toolbox provides:

Simplified development: Integrate tools to your agent in less than 10 lines of code, reuse tools between multiple agents or frameworks, and deploy new versions of tools more easily.
Better performance: Best practices such as connection pooling, authentication, and more.
Enhanced security: Integrated auth for more secure access to your data
End-to-end observability: Out of the box metrics and tracing with built-in support for OpenTelemetry.

⚡ Supercharge Your Workflow with an AI Database Assistant ⚡

Stop context-switching and let your AI assistant become a true co-developer. By connecting your IDE to your databases with MCP Toolbox, you can delegate complex and time-consuming database tasks, allowing you to build faster and focus on what matters. This isn’t just about code completion; it’s about giving your AI the context it needs to handle the entire development lifecycle.

Here’s how it will save you time:

Query in Plain English: Interact with your data using natural language right from your IDE. Ask complex questions like, “How many orders were delivered in 2024, and what items were in them?” without writing any SQL.
Automate Database Management: Simply describe your data needs, and let the AI assistant manage your database for you. It can handle generating queries, creating tables, adding indexes, and more.
Generate Context-Aware Code: Empower your AI assistant to generate application code and tests with a deep understanding of your real-time database schema. This accelerates the development cycle by ensuring the generated code is directly usable.
Slash Development Overhead: Radically reduce the time spent on manual setup and boilerplate. MCP Toolbox helps streamline lengthy database configurations, repetitive code, and error-prone schema migrations.

Learn how to connect your AI tools (IDEs) to Toolbox using MCP.

General Architecture

Toolbox sits between your application’s orchestration framework and your database, providing a control plane that is used to modify, distribute, or invoke tools. It simplifies the management of your tools by providing you with a centralized location to store and update tools, allowing you to share tools between agents and applications and update those tools without necessarily redeploying your application.

Getting Started

Installing the server

For the latest version, check the releases page and use the following instructions for your OS and CPU architecture.

Binary

To install Toolbox as a binary:

# see releases page for other versions
export VERSION=0.13.0
curl -O https://storage.googleapis.com/genai-toolbox/v$VERSION/linux/amd64/toolbox
chmod +x toolbox

Container image

You can also install Toolbox as a container:

1
2
3

# see releases page for other versions
export VERSION=0.13.0
docker pull us-central1-docker.pkg.dev/database-toolbox/toolbox/toolbox:$VERSION

Homebrew

To install Toolbox using Homebrew on macOS or Linux:

`1`	`brew install mcp-toolbox`

Compile from source

To install from source, ensure you have the latest version of Go installed, and then run the following command:

`1`	`go install github.com/googleapis/genai-toolbox@v0.13.0`

Running the server

Configure a tools.yaml to define your tools, and then execute toolbox to start the server:

`1`	`./toolbox --tools-file "tools.yaml"`

[!NOTE] Toolbox enables dynamic reloading by default. To disable, use the --disable-reload flag.

Homebrew Users

If you installed Toolbox using Homebrew, the toolbox binary is available in your system path. You can start the server with the same command:

`1`	`toolbox --tools-file "tools.yaml"`

You can use toolbox help for a full list of flags! To stop the server, send a terminate signal (ctrl+c on most platforms).

For more detailed documentation on deploying to different environments, check out the resources in the How-to section

Integrating your application

Once your server is up and running, you can load the tools into your application. See below the list of Client SDKs for using various frameworks:

Python (Github)

Core

Install Toolbox Core SDK:
1

pip install toolbox-core

Load tools:

from toolbox_core import ToolboxClient

# update the url to point to your server
async with ToolboxClient("http://127.0.0.1:5000") as client:

    # these tools can be passed to your application!
    tools = await client.load_toolset("toolset_name")

For more detailed instructions on using the Toolbox Core SDK, see the project’s README.

LangChain / LangGraph

Install Toolbox LangChain SDK:
1

pip install toolbox-langchain

Load tools:

from toolbox_langchain import ToolboxClient

# update the url to point to your server
async with ToolboxClient("http://127.0.0.1:5000") as client:

    # these tools can be passed to your application!
    tools = client.load_toolset()

For more detailed instructions on using the Toolbox LangChain SDK, see the project’s README.

LlamaIndex

Install Toolbox Llamaindex SDK:
1

pip install toolbox-llamaindex

Load tools:

from toolbox_llamaindex import ToolboxClient

# update the url to point to your server
async with ToolboxClient("http://127.0.0.1:5000") as client:

    # these tools can be passed to your application!
    tools = client.load_toolset()

For more detailed instructions on using the Toolbox Llamaindex SDK, see the project’s README.

Javascript/Typescript (Github)

Core

Install Toolbox Core SDK:
1

npm install @toolbox-sdk/core

Load tools:

import { ToolboxClient } from '@toolbox-sdk/core';

// update the url to point to your server
const URL = 'http://127.0.0.1:5000';
let client = new ToolboxClient(URL);

// these tools can be passed to your application!
const tools = await client.loadToolset('toolsetName');

For more detailed instructions on using the Toolbox Core SDK, see the project’s README.

LangChain / LangGraph

Install Toolbox Core SDK:
1

npm install @toolbox-sdk/core

Load tools:

import { ToolboxClient } from '@toolbox-sdk/core';

// update the url to point to your server
const URL = 'http://127.0.0.1:5000';
let client = new ToolboxClient(URL);

// these tools can be passed to your application!
const toolboxTools = await client.loadToolset('toolsetName');

// Define the basics of the tool: name, description, schema and core logic
const getTool = (toolboxTool) => tool(currTool, {
    name: toolboxTool.getName(),
    description: toolboxTool.getDescription(),
    schema: toolboxTool.getParamSchema()
});

// Use these tools in your Langchain/Langraph applications
const tools = toolboxTools.map(getTool);

Genkit

Install Toolbox Core SDK:
1

npm install @toolbox-sdk/core

Load tools:

import { ToolboxClient } from '@toolbox-sdk/core';
import { genkit } from 'genkit';

// Initialise genkit
const ai = genkit({
    plugins: [
        googleAI({
            apiKey: process.env.GEMINI_API_KEY || process.env.GOOGLE_API_KEY
        })
    ],
    model: googleAI.model('gemini-2.0-flash'),
});

// update the url to point to your server
const URL = 'http://127.0.0.1:5000';
let client = new ToolboxClient(URL);

// these tools can be passed to your application!
const toolboxTools = await client.loadToolset('toolsetName');

// Define the basics of the tool: name, description, schema and core logic
const getTool = (toolboxTool) => ai.defineTool({
    name: toolboxTool.getName(),
    description: toolboxTool.getDescription(),
    schema: toolboxTool.getParamSchema()
}, toolboxTool)

// Use these tools in your Genkit applications
const tools = toolboxTools.map(getTool);

Go (Github)

Core

Install Toolbox Go SDK:

`1`	`go get github.com/googleapis/mcp-toolbox-sdk-go`

Load tools:

package main

import (
  "github.com/googleapis/mcp-toolbox-sdk-go/core"
  "context"
)

func main() {
  // Make sure to add the error checks
  // update the url to point to your server
  URL := "http://127.0.0.1:5000";
  ctx := context.Background()

  client, err := core.NewToolboxClient(URL)

  // Framework agnostic tools
  tools, err := client.LoadToolset("toolsetName", ctx)
}

For more detailed instructions on using the Toolbox Go SDK, see the project’s README.

LangChain Go

Install Toolbox Go SDK:

`1`	`go get github.com/googleapis/mcp-toolbox-sdk-go`

Load tools:

package main

import (
  "context"
  "encoding/json"

  "github.com/googleapis/mcp-toolbox-sdk-go/core"
  "github.com/tmc/langchaingo/llms"
)

func main() {
  // Make sure to add the error checks
  // update the url to point to your server
  URL := "http://127.0.0.1:5000"
  ctx := context.Background()

  client, err := core.NewToolboxClient(URL)

  // Framework agnostic tool
  tool, err := client.LoadTool("toolName", ctx)

  // Fetch the tool's input schema
  inputschema, err := tool.InputSchema()

  var paramsSchema map[string]any
  _ = json.Unmarshal(inputschema, &paramsSchema)

  // Use this tool with LangChainGo
  langChainTool := llms.Tool{
    Type: "function",
    Function: &llms.FunctionDefinition{
      Name:        tool.Name(),
      Description: tool.Description(),
      Parameters:  paramsSchema,
    },
  }
}

Genkit

Install Toolbox Go SDK:

`1`	`go get github.com/googleapis/mcp-toolbox-sdk-go`

Load tools:

package main
import (
  "context"
  "encoding/json"

  "github.com/firebase/genkit/go/ai"
  "github.com/firebase/genkit/go/genkit"
  "github.com/googleapis/mcp-toolbox-sdk-go/core"
  "github.com/googleapis/mcp-toolbox-sdk-go/tbgenkit"
  "github.com/invopop/jsonschema"
)

func main() {
  // Make sure to add the error checks
  // Update the url to point to your server
  URL := "http://127.0.0.1:5000"
  ctx := context.Background()
  g, err := genkit.Init(ctx)

  client, err := core.NewToolboxClient(URL)

  // Framework agnostic tool
  tool, err := client.LoadTool("toolName", ctx)

  // Convert the tool using the tbgenkit package
  // Use this tool with Genkit Go
  genkitTool, err := tbgenkit.ToGenkitTool(tool, g)
  if err != nil {
    log.Fatalf("Failed to convert tool: %v\n", err)
  }
}

Go GenAI

Install Toolbox Go SDK:

`1`	`go get github.com/googleapis/mcp-toolbox-sdk-go`

Load tools:

package main

import (
  "context"
  "encoding/json"

  "github.com/googleapis/mcp-toolbox-sdk-go/core"
  "google.golang.org/genai"
)

func main() {
  // Make sure to add the error checks
  // Update the url to point to your server
  URL := "http://127.0.0.1:5000"
  ctx := context.Background()

  client, err := core.NewToolboxClient(URL)

  // Framework agnostic tool
  tool, err := client.LoadTool("toolName", ctx)

  // Fetch the tool's input schema
  inputschema, err := tool.InputSchema()

  var schema *genai.Schema
  _ = json.Unmarshal(inputschema, &schema)

  funcDeclaration := &genai.FunctionDeclaration{
    Name:        tool.Name(),
    Description: tool.Description(),
    Parameters:  schema,
  }

  // Use this tool with Go GenAI
  genAITool := &genai.Tool{
    FunctionDeclarations: []*genai.FunctionDeclaration{funcDeclaration},
  }
}

OpenAI Go

Install Toolbox Go SDK:

`1`	`go get github.com/googleapis/mcp-toolbox-sdk-go`

Load tools:

package main

import (
  "context"
  "encoding/json"

  "github.com/googleapis/mcp-toolbox-sdk-go/core"
  openai "github.com/openai/openai-go"
)

func main() {
  // Make sure to add the error checks
  // Update the url to point to your server
  URL := "http://127.0.0.1:5000"
  ctx := context.Background()

  client, err := core.NewToolboxClient(URL)

  // Framework agnostic tool
  tool, err := client.LoadTool("toolName", ctx)

  // Fetch the tool's input schema
  inputschema, err := tool.InputSchema()

  var paramsSchema openai.FunctionParameters
  _ = json.Unmarshal(inputschema, &paramsSchema)

  // Use this tool with OpenAI Go
  openAITool := openai.ChatCompletionToolParam{
    Function: openai.FunctionDefinitionParam{
      Name:        tool.Name(),
      Description: openai.String(tool.Description()),
      Parameters:  paramsSchema,
    },
  }

}

Configuration

The primary way to configure Toolbox is through the tools.yaml file. If you have multiple files, you can tell toolbox which to load with the --tools-file tools.yaml flag.

You can find more detailed reference documentation to all resource types in the Resources.

Sources

The sources section of your tools.yaml defines what data sources your Toolbox should have access to. Most tools will have at least one source to execute against.

sources:
  my-pg-source:
    kind: postgres
    host: 127.0.0.1
    port: 5432
    database: toolbox_db
    user: toolbox_user
    password: my-password

For more details on configuring different types of sources, see the Sources.

Tools

The tools section of a tools.yaml define the actions an agent can take: what kind of tool it is, which source(s) it affects, what parameters it uses, etc.

tools:
  search-hotels-by-name:
    kind: postgres-sql
    source: my-pg-source
    description: Search for hotels based on name.
    parameters:
      - name: name
        type: string
        description: The name of the hotel.
    statement: SELECT * FROM hotels WHERE name ILIKE '%' || $1 || '%';

For more details on configuring different types of tools, see the Tools.

Toolsets

The toolsets section of your tools.yaml allows you to define groups of tools that you want to be able to load together. This can be useful for defining different groups based on agent or application.

toolsets:
    my_first_toolset:
        - my_first_tool
        - my_second_tool
    my_second_toolset:
        - my_second_tool
        - my_third_tool

You can load toolsets by name:

# This will load all tools
all_tools = client.load_toolset()

# This will only load the tools listed in 'my_second_toolset'
my_second_toolset = client.load_toolset("my_second_toolset")

Versioning

This project uses semantic versioning (MAJOR.MINOR.PATCH). Since the project is in a pre-release stage (version 0.x.y), we follow the standard conventions for initial development:

Pre-1.0.0 Versioning

While the major version is 0, the public API should be considered unstable. The version will be incremented as follows:

0.MINOR.PATCH: The MINOR version is incremented when we add new functionality or make breaking, incompatible API changes.
0.MINOR.PATCH: The PATCH version is incremented for backward-compatible bug fixes.

Post-1.0.0 Versioning

Once the project reaches a stable 1.0.0 release, the versioning will follow the more common convention:

MAJOR.MINOR.PATCH: Incremented for incompatible API changes.
MAJOR.MINOR.PATCH: Incremented for new, backward-compatible functionality.
MAJOR.MINOR.PATCH: Incremented for backward-compatible bug fixes.

The public API that this applies to is the CLI associated with Toolbox, the interactions with official SDKs, and the definitions in the tools.yaml file.

Contributing

Contributions are welcome. Please, see the CONTRIBUTING to get started.

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms. See Contributor Code of Conduct for more information.

Community

Join our discord community to connect with our developers!

GitHubDaily

Wed, 27 Aug 2025 15:28:51 +0800

GitHubDaily/GitHubDaily

宗旨

多年以前，我曾看到 GitHub 开源项目作者、全栈工程师 TJ Holowaychunk 说过这么一句话：

“I don’t read books, never went to school, I just read other people’s code and always wonder how things work”。

从那时起，我便认为，通过阅读源码，站在前辈的角度上，去思考代码架构与程序逻辑，乃是提升编程技巧最好的方式。

因此，GitHub 也自然而然的，成为我最喜爱的开发者平台。

秉着挖掘开源价值的初衷，GitHubDaily 自 2015 年 10 月 10 日正式成立。

我们希望能通过这一举措，帮助开发者们发现当下最火的开源项目，掌控最新技术动态, 扩大技术视野, 并从开源项目的学习中获得编程能力的提升。

目前，GitHubDaily 已累积分享超过 8000 个开源项目，内容包括但不限于 GitHub 上的开源技术资料、开发者工具、编程网站以及成熟应用。

除了 GitHub 之外，我们也开始在下面多个社交媒体平台，帮助开发者传播与分享优质开源项目，挖掘其未来的技术应用前景。

如果你想接收最新的 GitHub 开源项目资讯，可以关注一下👇

公众号: GitHubDaily
微博: @GitHubDaily
知乎: @GitHubDaily
X (Twitter)：@GitHubDaily

有不错的开源项目，也欢迎到本仓库的 issues 推荐或自荐项目，我们期待你的分享。

2024 年复盘

下面是对 GitHubDaily 在 2024 年所推荐的项目进行分类整理，方便大家查找以往分享过的内容。

过往复盘 2023， 2022， 2021，2020，2019，2018

AI 技术

项目	简述	源
OmniParser	一款基于纯视觉的 GUI 智能体，能够准确识别界面上可交互图标以及理解截图中各元素语义，实现自动化界面交互场景，如自动化测试、自动化操作等。
ChatTTS	一款专门为对话场景设计的语音生成模型，主要用于 LLM 助手对话任务、对话语音以及视频介绍等，支持中英文混合文本合成语音，音色表现强，能达到真假难辨程度。
DLRover	一套 AI 大模型训练方案，可实现训练"自动驾驶"，提高训练效率，降低研发成本，具有容错性、快速恢复和自动扩展等特点。
SwiftInfer	一款基于 TensorRT 实现的 StreamingLLM 推理加速框架，可提升大模型推理性能 46 %，为多轮对话推理提供高效可靠的落地方案。
Audio2PhotoReal	一项由 Meta AI 发布的新技术，能够通过音频驱动生成全身逼真的虚拟人物对话，包括动作、表情和多人对话等，可广泛应用于虚拟现实、影视制作等领域。
OpenVoice	一项由 MyShell 团队开发的多功能即时声音克隆技术，只需提供简短音频样本，即可克隆原发言者的声音，生成多种语言语音，支持高精度音色克隆、灵活的声音风格调控以及无需样本的跨语言声音克隆。

↥ 返回目录

AI 工具

项目	简述	源
Shortest	一款利用 Claude 模型能力的端到端 AI 测试框架，通过自然语言编写可执行的单元测试，支持 GitHub 与 2FA 集成，提供安全认证方式。
STranslate	一款开箱即用的翻译和 OCR 工具，支持多种翻译方式、多家翻译服务、离线 OCR、回译、TTS 等功能，提升翻译效率。
x-kit	一款用于抓取和分析 Twitter 推友数据的工具，可自动抓取指定推友的基本信息和推文，支持定时更新时间线数据和本地存储。
MMAudio	一款视频自动配音工具，可根据视频内容或文本提示生成符合场景的音效，并与视频画面动作保持同步对齐。
Hoarder	一款自托管的书签管理 AI 工具，支持保存链接、笔记、图片和 PDF 文件，利用 AI 自动打标签分类存储，提供全文搜索能力。
Midscene.js	一款由 AI 驱动的 UI 自动化 SDK，通过自然语言描述对任意网页进行自动化操作、验证和数据提取，生成可视化测试报告，支持多种主流 AI 模型和自动化库集成。
Browser Use	一款开源工具，能让 AI 智能体自动化操作浏览器，像真人一样理解网页内容并完成复杂任务，如投递简历、查询航班等，具备视觉识别、HTML 解析、多标签页管理等功能。
Video Ocean	一款基于 Colossal - AI 优化的 Sora 视频模型的免费视频生成平台，提供极低成本、快速的视频生成服务，并将优化方案开源。
LogoCreator	一款开源的 AI Logo 生成器，使用 Flux Pro 1 . 1 模型快速生成 Logo 图标，支持编辑样式，代码完全开源并提供免费体验。
MarkltDown	一款由微软开源的工具，可将常见文件如 Word、Excel、PPT 等转换为 Markdown 格式，支持 OCR 和 AI 模型处理多媒体文件，提供命令行、Python API 和 Docker 三种使用方式。
PDFMathTranslate	一款开源的 PDF 文档翻译及双语对照工具，能够很好地保留公式、图表、目录和注释等排版，支持多种语言和翻译服务，可通过命令行、GUI 界面、Docker 等方式使用。
Gemini-Teacher	一款基于 Gemini 2 . 0 开发的英语口语练习 AI 助手，能实时识别发音并提供反馈和纠正建议，适合英语口语学习和教学。
XHS Note Generator	一款小红书笔记 AI 生成器，支持快速批量将视频或直播内容一键转换为符合小红书风格的爆款笔记，自动优化内容和配图。
ScreenPipe	一款强大的开源 AI 桌面应用，可 24 小时监控电脑，通过屏幕录制、OCR、音频输入和转录收集信息，保存到本地数据库，利用 LLM 对话、总结和回顾你所做的事情。
RMBG-2.0	一款高效的背景移除工具，支持处理各种图像，一键移除背景，效果出色且处理速度快，适用于电商、广告等场景。
Comfyui_Object_Migration	一套基于 ComfyUI 的服装迁移工作流，可将服装照片自然真实地迁移到模特身上，支持虚拟试穿和风格迁移（如动漫转现实风格），还原度较高。
NSFW Detector	一款开源免费的 NSFW 内容检测器，使用 Google 模型，准确度高，支持检测图片、PDF 文件、视频、压缩包等多种文件类型，支持纯 CPU 推理，提供 API 服务便于集成。
VideoCaptioner	一款功能强大的视频字幕翻译助手，提供可视化操作界面，支持字幕智能断句、校正、优化、翻译，一键生成双字幕视频。
MagicQuill	一款开源的 AI 互动式图像编辑工具，用户只需通过画笔涂抹和简单提示词，即可轻松实现插入元素、擦除物体、调整颜色等各种图像编辑操作。
Voice-Pro	一款集转录、翻译和文字转语音为一体的开源工具，提供简洁直观的可视化操作界面，支持实时转录和翻译，以及批量处理模式。
AdvancedLivePortrait-WebUI	一款基于 Gradio 开发的开源工具，能够对图像中的人物进行实时面部表情精确控制，如微笑、眨眼、摇头等，支持多种部署方式使用。
pdf-extract-api	一款基于 FastAPI 的 PDF 文档提取和解析工具，利用 OCR 技术和 Ollama 模型将 PDF 或图像转换为 Markdown 文本或 JSON 文档，支持表格、公式等格式解析，使用 Redis 缓存提高效率。
Cerebellum	一款基于智能体的浏览器自动化工具，通过使用 LLM 构建的智能体实现自动化操作键盘和鼠标，在网页上完成数据抓取、自动化测试等任务。
PromptFix	一款基于提示的 AI 图像修复工具，能够根据输入的提示对图像进行上色、移除物体、去除水印、高清化、调整光线等操作。
Maxun	一款开源的自动化网页数据爬取工具，无需编写代码，通过可视化界面构建自定义机器人实现网页数据捕获、提取和处理等功能。
AI Podcast Generator	一款开源的自动化 AI 播客生成工具，能够自动抓取网络新闻内容，并生成自然流畅的叙述性音频播客，提供简单易用的界面。
MegaParse	一款基于大型语言模型构建的功能强大的文档解析开源工具，可以轻松处理 PDF、PPT、Word、Excel 等常见格式，保证解析过程中不丢失任何信息，还能准确识别文档中的表格、目录、页眉页脚和图片等内容，解析速度快，效率高。
SoniTranslate	一款基于 Gradio 构建的简单易用的视频翻译工具，支持将视频一键翻译成多种语言，并提供视频同声翻译功能。
EveryoneNobel	一款开源的 AI 工具，可用于生成个性化诺贝尔奖项图片，利用 ComfyUI 进行图像生成，并结合 HTML 模板展示图片上的文本，只需提供个人肖像图即可生成诺贝尔奖项风格图像。
PodCastLM	一款开源免费的工具，可以将 PDF 内容转化为适合音频播客的自然对话，并输出为 MP3 文件。
Agent.exe	一款开源的 AI 自主操作计脑工具，利用 Claude 3 . 5 Sonnet 直接控制本地电脑，展示了 Claude 的 Computer Use 能力，可用于自动化智能体开发。
Zerox	一款简单易用的 OCR 文档工具，支持 PDF、Docx、图像等文件格式，可将文件转换为图像后利用 gpt - 4o - mini 模型识别图像信息并输出 Markdown 格式文件。
UVR5-UI	一款功能强大的人声伴奏分离 AI 工具，提供在线简洁易用的操作界面，支持选择不同的 AI 模型以及输出格式和质量，支持批量处理。
Virtual Try-On Application	一款基于 WhatsApp API 和 IDM - VTON 虚拟试穿模型的 AI 应用，用户只需在 WhatsApp 上传个人照片和服装照片，即可生成虚拟试穿效果图，提供便捷的虚拟试衣体验。
Claude Financial Data Analyst	一款由 Anthropic 开源的基于 Next . js 和 Claude AI 构建的金融数据分析助手，支持多种格式文件上传，通过聊天界面进行财务数据分析并生成可视化图表。
Tabled	一款基于 Surya 开发的 OCR 小工具，能够识别 PDF、图像、Word 和 PPT 等文件中的表格，并将表格内容提取转换为 Markdown、CSV 或 HTML 格式。
Surya	一款开源且强大的文档 OCR 工具，专注于文档图像的处理和分析，能够准确进行逐行文本检测和识别，支持 90 多种语言。
Animate-X	一款基于 AI 的动画生成工具，可以通过输入角色图片和参考动作视频，自动生成角色按照指定动作移动的动画效果，支持真人、游戏、卡通等多种类型的角色，保持角色一致性并允许大幅度动作。
AsrTools	一款开源的智能语音转字幕文本工具，集成了多家公司的语音识别接口，支持多种音频格式，提供批量处理和生成字幕文件功能，界面简单易用。
Podcastfy	一款开源工具，可将视频、PDF、论文、网站和文章等内容转换为对话式的播客音频。
gptme	一款功能强大的开源终端 AI 助手，支持在终端执行本地代码、读写文件、搜索浏览网页和进行视觉识别等操作，可连接 OpenAI、Anthropic、OpenRouter 等主流 LLM 提供商，也可使用 llama . cpp 在本地提供服务。
FinRL	一个开源的量化金融自动交易框架，包含市场环境、智能体和应用三层架构，提供先进算法核心支持连续交易决策。支持投资组合分配、加密货币交易、高频交易等金融任务和实时交易功能。
Youtube-Whisper	一款利用 OpenAI Whisper 模型实现音频转文本的轻量级开源免费工具，可输入 YouTube 视频链接提取音频并转换为文本，支持多种语言。
ebook2audiobookXTTS	一款开源的 AI 工具，利用 Calibre 和 Coqui TTS 将电子书转换为高质量的有声读物，支持多种语言和语音克隆功能。
Text Behind Image	一款简单有趣的开源免费工具，可以轻松将文本添加到图像中指定对象的背后，如动物、人物或物品等，并支持保存到本地。
Compiler Explorer	一款在线交互式编程语言编译器探索工具，支持超过 30 种语言如 C、C ++、Rust、Go 等，可在浏览器中实时编辑代码并查看编译后的汇编代码输出，提供多种编辑器、工具和可视化选项。
Local File Organizer	一款基于 AI 的本地文件整理器，可以自动扫描指定目录中的文件，理解文件内容并生成相关描述、文件夹名称和文件名，将文件整理到新的目录结构中。
OpenScanner	一款快速、易用且开源免费的文档扫描工具，具有自动文本识别、文档命名、地理位置标记、签名添加、Vision Pro 支持等功能，可保存、编辑、注释和分享扫描文档。
Diffusers Image Outpaint	一款在线 AI 扩图工具，可以免费使用，只需上传图片并选择扩展比例，即可一键完成图像扩展。
GOT-OCR2.0	一款端到端的开源 OCR 模型，被称为 OCR 2 . 0，支持识别场景文本、文档、乐谱、图表、数学公式等多种内容，在 BLEU 评测中取得 0 . 972 的高分，模型体积仅 1 . 43GB。
Python Tutor	除了 Python 外，还支持 Java、C、C ++和 JavaScript 等编程语言。
Fish Speech	一款开源的多语言 TTS 模型，使用 70 万小时数据训练，支持英语、中文、韩语、日语、法语、德语、阿拉伯语和西班牙语 8 种语言。具有快速语音合成、低延迟、声音丰富、混合语言和语音克隆等特点。
wcf.js	一款强大、快速、开源的微信机器人底层框架，提供消息监听、发送和群聊操作的一站式解决方案，内置强大易用的 Agent 库和开发者工具包，具有高效稳定的处理能力和自由控制发消息频率的功能。
markmap	一款开源免费的工具，可以将 Markdown 文档内容转换为直观可视化的思维导图，支持图像、链接、代码块、内联样式和数学公式等格式内容。
Video2x	一款开源免费的无损放大视频和图像工具，使用多种先进的超分辨率算法如 waifu2x、Anime4K、SRMD 和 RealSR 等实现视频/ GIF /图像的无损放大和提高帧速率。
PptxGenJS	一款基于 JavaScript 的开源库，可用于生成 PPT 演示文稿，支持添加图表、表格、图像、视频等多种元素，生成的文件与 PowerPoint 等应用兼容，并提供 HTML 转 PPT 功能。
DataEase	一款功能强大的开源数据可视化分析工具，可作为 Tableau 的替代品，提供丰富美观的图表展示、制作和数据引擎等功能，支持多种数据链接、拖拽式图表制作和与他人分享，并具备 AI 辅助分析和模板市场等创新功能。
MoneyPrinterPlus	一款开源免费的全自动化 AI 视频剪辑工具，借助 AI 技术一键生成和批量混剪各类短视频，支持自动发布到视频平台，助力变现。
Melty	一款专为 10x 工程师打造的开源 AI 代码编辑器，能够与开发者的整个开发流程集成，理解从终端到 GitHub 的操作，协助高效编写和重构代码，支持多文件大规模更改。
Easy Voice Toolkit	一套功能丰富的开源 AI 语音工具箱，包括音频处理、语音识别、语音转录、数据集制作和语音合成等，形成完整语音模型训练工作流。
ChartDB	一款基于 Web 的功能强大的数据库图表编辑器，支持多种数据库如 PostgreSQL、MySQL、SQL Server 等，提供基于 AI 的导出功能生成 DDL 脚本方便数据库迁移，可在线使用或本地部署。
FireCrawl	一款开源且强大的 Web 爬虫工具，可爬取任何网站内容并转换为 Markdown 或结构化数据，为大语言模型训练、检索增强生成（RAG）提供数据。
MaxKB	支持多种大语言模型对接，内置工作流引擎编排 AI 过程，可无缝嵌入第三方系统，在短时间内获得 9000 +星。
HivisionIDPhoto	一款轻量级的 AI 证件照制作工具，可识别多种用户拍照场景，实现抠图和生成标准尺寸的证件照。
VideoLingo	一款全自动视频翻译的 AI 工具，能够一键对视频进行字幕切割、翻译、精准对齐和个性化配音，生成高质量的字幕和配音。
Cursor	一款智能代码编辑神器，提供代码补全、错误修复等功能，支持多种编程语言。该项目收集了各语言的最佳配置提示词和使用教程。
NarratoAI	一款一站式自动化影视解说的 AI 视频剪辑工具，基于 LLM 实现文案撰写、自动化视频剪辑、配音和字幕生成，是影视解说剪辑神器，可高效创作内容。
voicechat2	一款响应快速且完全本地化的 AI 语音聊天工具，使用 WebSockets 实现低延迟语音交互并允许远程访问，可本地运行语音识别、文本转语音和大语言模型，在 4090 显卡上延迟低至 300 毫秒。
CyberScraper 2077	一款基于 OpenAI 大语言模型的强大网络爬虫工具，能够智能理解和解析网页内容，提供简单易用的可视化界面，无需编程知识即可操作使用。
awesome-digital-human-live2d	一个开源项目，旨在打造一个有温度的数字人，支持 Docker 快速部署、Dify 服务接入、ASR、LLM、TTS、Agent 模块化扩展、Live2d 人物模型扩展和控制、PC 端和移动端 Web 访问。
LLM-Aided OCR	一款基于大语言模型（LLM）的开源 OCR 扫描 PDF 工具，可将原始 OCR 扫描 PDF 文本转换成高准确度、格式正确且易于阅读的 Markdown 文档。
Product Hunt Daily Hot	一款基于 GitHub Action 的自动化工具，每天定时爬取并生成 Product Hunt 热门产品榜单的 Markdown 文件，使用 GPT - 4 模型翻译产品描述，帮助快速查看每日热门榜单。
moffee	一款开源免费的 PPT 制作工具，能够将 Markdown 文档一键转换为干净、专业的幻灯片，提供 web 界面实时预览效果。
Linly-Dubbing	一款开源且强大的视频多语言 AI 配音/翻译工具，能自动下载视频、翻译字幕、人声分离、克隆音色配音并合成视频。
Transformer Explainer	一个可视化交互式学习工具，以 GPT - 2 为例，通过输入文本实时观察 Transformer 各组件（嵌入层、自注意力机制、MLP 等）的工作过程和预测下一个 Token，适合教学和理解 Transformer 内部原理。
AI Scientist	一款全球首个自动化科学研究 AI 系统，能够从构思、编写代码、进行实验、总结结果到撰写完整论文和进行同行评审的全过程，支持多种主流大语言模型。
Clapper	一款集成多种 AI 功能的开源视频剪辑工具，可一键生成图像、视频、语音、音乐等素材，并进行二次剪辑，让用户无需专业技能即可通过互动迭代的方式制作视频。
ai-renamer	一款基于 Node . js 的命令行工具，可根据文件内容自动批量重命名本地文件、图像或视频，默认使用 Ollama 本地模型如 Gemma、Llama 等进行智能识别。
metahuman-stream	一个开源项目，实现实时交互流式数字人，支持音视频同步对话，具有多种功能如数字人模型选择、声音克隆、对话被打断处理、全身视频拼接、推流、视频编排和大语言模型对话等，基本可达到商用效果。
PPTX2MD	一个开源免费的工具，能够将 PPT 幻灯片转换为 Markdown 文件，支持保留标题、列表、粗体、斜体、颜色、超链接、图片、表格以及合并单元格等 PPT 内容格式。
Fake Screenshot Generator	一款可直接在线免费使用的字幕截图生成器工具，可生成逼真的带字幕的截图。
MinerU	一款一站式、开源、高质量的数据提取工具，能够将 PDF、网页以及多格式电子书转换为 Markdown 格式，提取图像、表格并转换公式为 LaTex，支持多种语言识别。
Llama Tutor	一款开源免费的 AI 个人导师工具，基于 Next . js 和 Tailwind CSS 构建，集成 Llama 3 . 1 和 Serper 提供强大的推理和搜索能力，可根据用户选择的教育水平生成个性化导师解答各种问题。
bilingual_book_maker	一款利用大语言模型（如 ChatGPT、Claude 等）对文件和图书进行多语言翻译的开源工具，支持 epub、txt 和 srt 等格式。
Claude Engineer	一款基于 Claude 3 . 5 模型的交互式命令行工具，整合了文件系统操作、网络搜索等功能，可协助完成各种软件开发任务，如智能代码分析、修改建议、项目管理等。
AudioNotes	一款开源免费的音视频转结构化笔记工具，基于 FunASR 和 Qwen2 构建，可快速提取音视频内容，并利用大模型能力整理成结构化 Markdown 笔记，方便快速阅读。
Swapy	一个简单的 JavaScript 开源工具，只需几行代码就可以将网页上任何布局转换为可拖动交换布局，适用于 React、Vue 等主流前端框架。
Kspider	一款开源免费的可视化爬虫平台，通过流程图配置即可完成数据爬取工作，无需编写代码，对小白极其友好。
AI Shell	一款开源的终端 AI 助手，可在命令行界面通过自然语言转换为对应的 Shell 命令执行，支持多种语言输入，提高终端操作效率。
Crawlee	一款开源免费且非常强大的 Python 网页抓取和浏览器自动化库，旨在构建可靠的爬虫为 AI、大语言模型、检索增强生成或 GPTs 提取数据。
Enchanted	一款开源的本地大模型 Mac 客户端，类似于 ChatGPT 客户端，需搭配 Ollama 使用，可轻松连接本地部署的私有模型，如 Llama2、Mistral 以及 Vicuna 等，提供在 iOS 生态系统中无过滤、安全、私密和多模态的使用体验。
Whisper Timestamped	一款基于 Transformers . js 的浏览器内语音识别工具，可本地运行 whisper - base 模型进行语音转文字，支持 100 种语言并生成单词级时间戳。
Vanna	一款 SQL 生成 AI 框架，允许在自有数据上训练 RAG 模型，通过自然语言准确生成 SQL 查询语句，支持多种主流大模型，使用简单，开源可自行部署。
Great Tables	一款用于制作高质量表格的 Python 库，内置丰富的表格组件和格式化选项，可以组合创建多种类型的精美表格。
Comic Translate	一款利用 GPT - 4 视觉能力的开源漫画自动翻译神器，支持多种格式如图像、PDF、Epub 等，可实现多种语言之间互译，包括日语、英语、中文、韩语、德语、荷兰语等，并提供可视化操作界面。
CodeGeeX	一款功能全面的国产开源代码模型，集成代码补全、生成、问答、解释、工具调用、联网搜索等多种能力，覆盖编程开发各种场景，在百亿参数以下性能最强。同时也提供 CodeGeeX 智能编程助手插件。
遇见李白	一个开源项目，旨在通过构建李白知识图谱的 AI 智能体，以生成式对话应用的形式推广和普及李白古诗词文化，最终目标是开发一款生成式对话应用，实现实时互动并提供个性化的李白诗歌鉴赏体验。
WebDesignAgent	一款基于人工智能的网站构建工具，支持多页面管理、用户自定义添加/删除、迭代优化和视觉优化等功能，可通过文本、图片、视觉线索或混合方式生成设计精美的网站。
AI-YinMei	一款功能齐全的 AI 虚拟主播（Vtuber）工具，集成了 FastGPT 知识库聊天、语音合成、Stable Diffusion 绘画、AI 唱歌等技术，可实现聊天、唱歌、绘画、跳舞、表情切换、换装、搜图、场景切换等多种功能。
OmniParse	一款开源工具，能够将各种非结构化数据如文档、表格、图像、视频、音频和网页转化为结构化的可操作数据，方便用于检索增强生成（RAG）和微调。
gptpdf	一款基于 GPT - 4o 视觉大语言模型的开源工具，仅用 293 行代码就能将 PDF 文件解析为 Markdown 格式，几乎完美地解析排版、数学公式、表格、图片和图表等内容。
Streamer-Sales	一款能够根据商品特点进行解说的卖货主播大模型，通过生成细腻、独到的解说词，激发用户购买欲望，提供多种功能如文案生成、语音转换、视频生成等，旨在成为销售助手，提升销量和用户体验。
Wiseflow	一款敏捷的信息挖掘开源免费工具，可从网站、公众号、社交平台等渠道爬取相关信息，擅长从公众号文章中提取信息。支持对爬取信息进行筛选、提炼、贴标签等处理，并可整合到任意 Agent 项目中作为动态知识库。可完全本地部署，无需 GPU，适合任何硬件环境。
GPT Computer Assistant	一款开源的 GPT 客户端应用程序，旨在为 Windows 和 Linux 系统提供类似 macOS 上 GPT 应用的功能，包括屏幕读取、麦克风长音频输入、编写和运行程序以及知识库管理等。
Diffutoon	一款开源的视频动漫化项目，提供 Colab 运行笔记，无需部署即可一键动漫化视频，转换后的视频画面稳定流畅，还可编辑和添加视频效果。
RTranslator	一款开源免费、可离线的实时翻译应用，能够实时将对方说的语言翻译成你能听懂的语言，支持对话模式、对讲模式和文本翻译模式，使用 Meta 的 NLLB 和 OpenAI 的 Whisper 进行翻译和语音识别。
AI Math Notes	一款基于 Python 开发的交互式绘图应用程序，允许用户在画布上手写数学方程，利用多模态大语言模型（LLM）计算并显示结果，实现了类似苹果"备忘录数学计算器"的功能。
Whisper Web	一款基于 OpenAI 的 Whisper 语音识别模型的开源 Web 应用程序，允许直接在浏览器中运行使用 Whisper 进行语音转文本，支持 WebGPU 加速，无需后端服务器，可导出 TXT 和 JSON 格式文件。
GPT Academic	一款功能强大的 GPT 学术优化开源项目，提供论文翻译、代码解析、知识获取、语法校对、论文润色、摘要生成等多种功能，支持接入多种 LLM 模型，提升学术研究效率。
Translation Agent	一套由吴恩达老师开源的 AI 翻译工作流程，基于大语言模型进行翻译、反思和优化，可控制语气、地区和术语翻译一致性，翻译质量媲美商业工具。
ScrapeGraphAI	一款基于大语言模型和直接图逻辑的 AI 网页爬虫工具，可以根据用户提示自动为网站和本地文档创建爬取管道，支持单页、多页和语音爬取，提高数据采集效率。
MiGPT	一个开源项目，旨在将小米小爱音箱打造成专属的 AI 语音助手，将小爱音箱和米家智能设备与 ChatGPT 等大模型的理解能力完美融合，提供 LLM 回答、角色扮演、流式响应、长短期记忆、自定义 TTS 和智能家居 Agent 等功能。
pyVideoTrans	一款开源的视频翻译配音工具，可将一种语言的视频自动翻译为指定语言的视频，生成字幕和配音，支持多种翻译服务和配音引擎，可本地离线使用。
cloudflare-ai-web	一个开源项目，利用 Cloudflare Workers AI 免费提供的大模型，可快速搭建多模态 AI 平台，支持一键部署、无需服务器、个性化定制等，集成了 ChatGPT、Stable Diffusion 等多种 AI 模型。
ChatTTS-ui	一款开箱即用的 ChatTTS 安装包，提供 Web 界面和 API 接口，支持 Windows、Linux、macOS 部署，Windows 用户可直接下载安装包一键安装使用。
Perplexica	一款由 AI 驱动的开源搜索工具，能够深入互联网提供精准答案，理解问题并优化搜索结果，提供带引用来源的明确答案。具有隐私保护、本地大语言模型支持、双模式搜索和专注模式等特征。
RAGapp	一款无需编码的可视化界面工具，用于配置基于 LlamaIndex 构建的 RAG 聊天机器人，类似于 OpenAI 的 GPT 模型，可轻松部署在自有云基础设施中。
MusicGPT	一款开源免费的 AI 音乐生成器，支持在 Windows、macOS 和 Linux 系统上本地高效运行最新音乐生成 AI 模型，如 Meta 的 MusicGen 模型，无需安装依赖，易于使用。
Khoj	一款融合本地文档和在线搜索的 AI 第二大脑工具，可连接个人知识库、提供强大搜索引擎、支持在线离线使用，并提供定制 AI 智能代理协助完成任务，完全开源免费。
Talk With Gemini	一款可一键免费部署的私人 Gemini 应用程序，支持最新的 Gemini 模型，如 Gemini 1 . 5 Pro、Gemini 1 . 5 Flash 等，具有多模态支持、语音模式、视觉识别、助理市场、Markdown 支持、上下文压缩、隐私安全、精心设计的 UI 等特性。
ugly-avatar	一个开源免费的潦草头像生成器，可以生成极具特色的头像，受到许多网友的喜爱。
DashPlayer	一款专为英语学习打造的视频播放器，提供双语字幕、按字幕跳转、查词查询、可调整界面尺寸、记录播放位置、蓝牙遥控操作、夜间模式、AI 字幕生成、长视频切分和视频下载等功能，助力通过观看视频提升英语水平。
OpenGlass	一款基于开源硬件的 AI 智能眼镜系统，只需 20 美元的成本即可将普通眼镜升级为智能眼镜，实现记录生活、识别物体、计算卡路里、实时翻译等多项 AI 功能。
PicProse	一款开源免费的封面图片制作工具，能帮助用户轻松为 Medium、Wordpress、微信等平台的文章制作精美封面。
Reader	一款开源免费的工具，专门用于将网页内容转换为 Markdown 格式，方便将网页内容整合到知识库中。
SuperMemory	一款开源免费的个人知识管理工具，可以帮助用户轻松收集和管理网上的有价值内容，通过 AI 快速查找和回顾已保存的内容，打造属于自己的"第二大脑"。
STORM	一款由斯坦福大学开发的创新型 AI 写作工具，可根据主题自动收集信息、创建大纲，模拟专家对话并撰写完整文章，帮助编写出具有深度和广度的高质量内容。
LLocalSearch	一款完全本地化的 AI 搜索集成工具，无需依赖任何外部 API，通过 LLM Agents 寻找答案，支持低配置硬件运行，提供详细进度日志和后续问题交互，界面美观支持浅色深色主题，支持 Docker Compose 部署。
FreeAskInternet	一款开源免费的基于搜索的问答 AI 工具，可本地运行无需 GPU 硬件支持，通过多引擎搜索并利用 GPT - 3 . 5 处理结果生成答案，确保隐私安全。
poster-design	一款开源的在线创意图片编辑器，适用于海报图片生成、电商分享图、文章长图等场景，提供丰富的编辑功能，如元素拖拽、图片编辑、SVG 编辑、画布自定义等，无需客户端即可轻松完成图文排版。
Open WebUI	一款专为大型语言模型（LLM）设计的可扩展且功能丰富的 Web 界面，支持运行各种 LLM 模型，包括 Ollama、OpenAI 兼容 API、Gemini、Groq、Claude 等，提供直观界面、响应式设计、代码语法高亮、Markdown 和 LaTeX 支持、本地检索增强生成（RAG）集成、多模型支持与多模态互动、安全和多用户管理等丰富功能。
Douyin_TikTok_Download_API	一款开源免费的高性能异步数据爬取工具，支持 API 调用和在线批量解析及下载无水印视频或图集，覆盖抖音、快手、TikTok、B 站等多个视频平台。
AI Comic Factory	一款基于 LLM + SDXL 技术的开源免费在线 AI 漫画生成工具，可自动生成精美漫画作品，支持自定义 LLM 和图像生成模型。
Suno-API	一个非官方的 Suno AI API 项目，支持生成歌曲、歌词等功能，具有自动维护 token、全异步响应、代码简单易维护等特点。
Claude-Journalist	一款基于 Claude 3 的 AI 写作助手，可根据输入主题自动搜索相关信息、撰写和编辑高质量文章，提高写作效率。
MoneyPrinterTurbo	一款开源免费的 AI 工具，可以根据提供的主题或关键词自动生成视频文案、素材、字幕和背景音乐，并合成高清短视频，支持多种功能如自定义文案、视频尺寸、批量生成、语音合成、字幕设置等，提高短视频制作效率。
LiveStory	一款利用 AI 进行实时语音绘图的工具，将传统的文本输入换成语音输入，实时生成图像，支持语音控制绘图。
Buzz	一款开源免费、简洁、可离线的音频转录和翻译工具，基于 OpenAI Whisper 开发，支持拖放导入音视频文件进行转录和翻译，转录文本可导出多种格式，支持多种语音识别框架和命令行操作。
MediaCrawler	一款开源的自媒体爬虫工具，支持小红书、抖音、快手、B 站、微博等平台的视频、图片、评论、点赞、转发数据抓取，具备多种登录方式、指定数据爬取、IP 代理池和多种数据格式保存等功能。
小浣熊智能助手	一款由商汤科技研发的功能全面的国产 AI 工具，内置代码模式和办公模式，可大幅提升编程开发和协作办公效率。
Comflowy	一款针对 Mac 用户优化的 ComfyUI 整合包，提供一键部署、模型管理、工作流管理和扩展管理等功能，使用体验比官方 ComfyUI 更佳。
Inpaint-web	一款基于 WebGPU 技术开发的开源免费图像修复和放大工具，可直接在浏览器上运行，无需客户端，提供图像局部擦除修复和超分辨率放大功能。
AutoPrompt	一个自动优化提示词的开源框架，可根据用户意图生成高质量详细的提示词，并通过迭代数据集不断优化提示词，减少工作量，解决敏感性和歧义性问题。
青梧字幕	一款基于 Whisper 的开源字幕提取工具，支持智能提取、编辑和高质量翻译字幕，可生成单/双语字幕并选择多种格式下载。
Chat with MLX	一款开源的本地 AI 对话助手工具，可与本地数据进行交互，支持多种文件格式和语言，并可集成多个开源大语言模型。
StickerBaker	一款开源的 AI 贴纸制作工具，结合了 Stickers SDXL Lora 和 BRIA 背景移除工具，只需输入简单的提示词，即可快速生成高清精美的贴纸。
OOTDiffusion	一款开源的 AI 虚拟试衣工具，支持一键试穿上半身/下半身/连衣裙，服装与模特贴合自然，试穿效果逼真。
MoneyPrinter	一款自动化创建短视频的工具，可根据主题自动生成带配乐及字幕的视频，并上传到 YouTube 获取收益。支持使用 GPT、DALL - E 等大模型生成脚本和图像。
NotesGPT	一款开源免费的语音笔记工具，界面简洁，支持语音输入转文字和自动总结，内置 Mixtral LLM 和 Whisper 转录引擎，支持中英混合输入。
Reor	一款由 AI 驱动的开源免费桌面笔记工具，能够智能整理和归纳笔记内容，自动关联相关想法，内置 LLM 提供问答和语义搜索功能，可作为个人知识管理工具使用。
Noi	一款开源的高颜值 AI 客户端，集成了 ChatGPT、Claude、Bard、Poe 等主流 AI 模型，并内置 GitHub、HuggingFace 和 VS Code 等工具，支持自定义 Prompt 管理、多语言、多主题等功能，可在 MacOS、Windows、Linux 系统上安装使用。
ChatGPT Web + Midjourney Proxy	一款功能全面的镜像站，支持 AI 对话、AI 换脸、Midjourney、GPTs、TTS Whisper 等多种 AI 功能，提供 Vercel 一键部署。
ChatGemini	一款开源的简洁 Google Gemini 网页客户端，界面和交互类似 ChatGPT，支持在对话框中上传图片并自动调用 Gemini - Pro - Vision 模型进行图像识别。
WhisperFusion	一款基于 WhisperLive 和 WhisperSpeech 构建的 AI 对话系统，集成了 Mistral 大语言模型，实现了极低延迟的语音与 AI 对话体验，并通过 TensorRT 引擎优化和 torch . compile 技术提升了运行效率。
search_with_lepton	一款使用不到 500 行代码构建的对话式搜索引擎，实现了类似 Perplexity 的效果，集成了大语言模型（Llama2、Mixtral）和搜索引擎（支持 Bing、Google 搜索），具有可自定义的个性化 UI 界面，支持搜索结果的共享和缓存。
AI红包封面生成器	一款基于 DALL - E 3 的 AI 红包封面生成工具，支持自定义生成红包封面图像，采用 Next . js 全栈开发，集成了多种功能如谷歌登录、图片上传、支付等。
Zed	一款开源的高性能可多人协作代码编辑器，集成了 AI 代码生成和重构功能，提供语言感知、集成终端、多种编辑模式、团队协作和远程代码操作等强大功能。
Web LLM	一款可在浏览器中直接运行大型语言模型的工具，支持 Llama 2 7B / 13B、Mistral 7B 和 WizadMath 等模型，并通过 WebGPU 实现加速，无需服务器支持。
WhisperSpeech	一款基于 Whisper 语音识别模型的开源文本转语音工具，目前仅支持英语，作者计划在下一个版本增加多语言支持，可用于商业用途。
GPT-SoVITS	一款开源的中文语音克隆工具，仅需 5 秒语音样本即可实现 80 %~ 95 %相似度的声音克隆，提供 1 分钟语音可逼近真人效果并生成高质量 TTS 模型。
QAnything	一款开源的知识库问答引擎，支持本地部署和调用云端大模型服务，可导入多种格式文档，提供准确快速可靠的问答体验。
AI Gateway	一款统一的 API 网关工具，可轻松快速接入 100 多种大语言模型，如 OpenAI、Anthropic、Mistral、LLama2、Google Gemini 等，具有占用空间小、处理速度快、支持负载均衡、故障转移、自动重试等优势，已在超过 100B Tokens 上进行实战测试。
cutword	一个简单快速的中文分词和命名实体识别工具，分词速度是知名"结巴"中文分词的两倍，字典文件根据最新数据统计得到，词频更加合理。
DDColor	一款基于 AI 的图像着色工具，可为黑白老旧照片和动漫游戏场景提供逼真自然的着色效果，实现照片级真实感。
ReplaceAnything	一款由阿里巴巴发布的 AI 内容替换工具，能够准确保留指定的物体（如人脸、人物、服装、物品等），并通过输入提示词实现超高质量的内容替换，可用于人物替换、服装替换、背景替换等多种场景。
Jan	一款开源的本地 AI 工具，可无需编码运行主流大语言模型如 Mistral、Llama、Mixtral 等，支持 Windows、Mac 和 Linux 系统，拥有简洁美观的 UI 界面。
Mobile ALOHA	一款多功能家务机器人，可以帮助浇花、拖地、做饭、逗猫、扔垃圾、洗衣服、铺床单、整理衣物等家务活动，具有模仿学习和远程操作功能。
Auto Job Find Assistant	一款结合 GPT 和 RPA 技术的自动投简历助手，可根据个人简历和职位要求自动匹配合适的工作机会，生成自我介绍和求职信，并自动发送给 HR，实现一键寻找工作的全流程自动化。

↥ 返回目录

免费书籍

项目	简述	源
Large Language Model in Action	一本专注于大语言模型实践应用的开源书籍，通过介绍工具和案例实践，帮助读者快速上手大模型应用开发。
Select Star SQL	一本免费的互动式学习 SQL 书籍，共五章内容，每章约 30 分钟，涵盖行操作、分组聚合、表连接等主题，结合实际案例学习。
The Little Book of ML Metrics	一本面向数据科学家的开源免费书籍，涵盖广泛的机器学习指标，包括回归、分类、聚类、排行、计算机视觉和自然语言处理等领域，旨在成为数据科学家的快速参考手册。
Linux From Scratch 中文翻译版	一本免费的在线书籍，教授从源代码开始构建整个 Linux 系统的过程，深入了解 Linux 工作原理，支持个性化定制和优化，提高系统安全性，通过实践获得 Linux 系统的深刻理解。
Hands-On Large Language Models	一本由吴恩达老师推荐的关于大语言模型的实践指南书籍，涵盖语言模型基础知识、文本分类、提示工程、语义搜索、多模态语言模型等内容，配有案例代码、插图和引用论文，帮助读者深入理解和使用大语言模型。
计算机体系结构基础（第三版）	一本由"龙芯之父"胡伟武老师等编写的开源免费在线书籍，涵盖指令系统结构、计算机硬件结构、CPU 微结构、并行处理结构、计算机性能分析等内容，适合本科生、研究生和技术人员学习参考。
Decision Making Under Uncertainty	一本由麻省理工学院出品的免费电子书，涵盖概率推理、序列决策问题、模型/状态不确定性以及多 Agent 系统等内容，使用大量示例和练习帮助读者理解不同算法的直觉和应用场景。
Grammar Club	一本开源且免费的在线英语语法学习书籍，采用循序渐进的方式，从初级到高级分为三大篇幅，涵盖简单句、复句、合句和简化从句等内容，旨在培养英语能力、建立信心并增强阅读兴趣。
《理解深度学习》中文版	一本由麻省理工开源的深度学习教材中文版，全面涵盖深度学习基础概念、监督学习、强化学习等内容，并提供 PPT、笔记和 Python 练习 Demo。
Understanding Deep Learning	一本来自麻省理工的开源书籍，全面涵盖了深度学习的基础概念、监督学习、强化学习、线性回归、神经网络、扩散模型等内容。该书提供了 PPT、笔记和 68 个 Python 练习 Demo 供学习和实践。
Python Guide for JavaScript Engineers	一本面向 JavaScript 工程师的 Python 学习指南，由一名前端工程师编写，系统介绍了 Python 环境搭建、项目开发等内容，并对比了 JavaScript 和 Python 的异同，帮助 JavaScript 工程师快速掌握 Python 语言。
深入超高可用架构原理与实践	一本开源的架构设计书籍，涵盖云计算、网络、分布式系统、容器技术、可观测性、服务网格、DevOps 等主题，帮助程序员深入理解相关技术的原理与实践。

↥ 返回目录

学习教程

项目	简述	源
Reasoning with o1	一门由 OpenAI AI 解决方案主管 Colin Jarvis 主讲的免费 AI 短课程，教授如何有效使用 OpenAI 最新推理模型 o1，包括多步骤任务规划、代码开发与编辑、图像理解等实践项目，帮助学习者掌握 o1 模型的高级推理能力。
A Programmer’s Guide to English	一份专为程序员设计的英语学习指南，通过程序员易于理解的方式剖析语言学习本质，提供适合不同水平的训练方法和学习资料推荐，并从构建英语识别程序的角度助力学习。
ML Retreat	一份从中级到高级的 AI 学习路径，包含作者在学习高级机器学习时的个人笔记和资源，涵盖从基础到更高级主题的深入理解，如从头构建大语言模型、LLM 幻觉深入研究以及 LLM 超越注意力机制等。
Machine Learning Course Notes	一份来自卡内基梅隆大学的机器学习入门课程笔记，涵盖了机器学习各主题的要点，包括非参数模型、线性回归、逻辑回归、神经网络、支持向量机等，针对初学者提供全面的概念解释、数学证明和算法步骤。
AI Python for Beginners	一门由吴恩达老师主讲的免费 AI 短课程，面向初学者，教授 Python 编程基础知识以及集成 AI 工具进行数据操作、分析和可视化，通过真实项目实践强化 Python 技能。
软件工程师学习指南	一份从零开始成为软件工程师的详细学习指南，涵盖计算机基础知识、软件开发工具和流程、数据库、面向对象编程、DevOps 工具等内容，从 Python 语言入门，最后提供编码面试准备、工作选择和持续学习建议。
Machine-Learning	一份从零开始学习机器学习算法的教程，包含神经网络和 Transformer 模型的详细解释和 Python 实现代码，如 Neuron、Layer 和 NeuralNetwork 类，还提供机器学习视频和 Jupyter Notebook 文件。
Spring-Reading	一份深入了解 Spring 源码的系列学习资料，涵盖了 Spring 框架的核心概念和关键功能，如资源加载、表达式语言、Bean 定义、Aware 接口、核心注解和 AOP 等，帮助更透彻地理解 Spring 内部工作机制。
Machine Learning Specialization	一套由吴恩达老师主讲的机器学习专业课程，包括机器学习回归与分类、高级学习算法、无监督学习等多个领域，涵盖监督学习、无监督学习、神经网络等内容，适合初学者和希望在人工智能领域发展的专业人士学习。
LLM Twin Course	一门全面的大型语言模型（LLM）免费学习课程，教授如何构建生产级别的 LLM 和基于 LLM 的检索增强生成（RAG）系统，涵盖系统设计、数据工程、特征管道、训练管道和推理管道等方面。
Stable Diffusion From Scratch	一套系统性的教程，讲解 Stable Diffusion 和扩散模型的工作原理及其背后的数学知识，并提供一系列步骤指导训练完成一个扩散模型。
From Tensor to Stable Diffusion	一份开源的深度学习教程，提供为期 9 周的课程大纲，从张量基础到稳定扩散模型实现，包括视频讲解、代码实现和论文学习。
大模型理论基础	一份关于大规模预训练语言模型的教程资料集合，涵盖数据准备、模型构建、训练策略、模型评估与改进等内容，旨在为读者提供深入的理论知识和实践方法。
AI Text-to-Video Model from Scratch	一份手把手教程，教你如何使用 Python 从零开始构建一个文本到视频生成模型，涵盖理论概念、架构设计和编码实现等全过程。
nginx-tutorial	一份开源的 Nginx 入门学习笔记，涵盖基础安装、参数说明、配置讲解、第三方模块安装、性能优化及常见使用场景等内容。
PyTorch Deep Learning Tutorial	一份开源的《从零到精通深度学习 PyTorch》教程，涵盖 PyTorch 基础、深度学习工作流程、计算机视觉、自定义数据集处理、模块化代码编写及模型部署等内容，提供视频和代码示例，适合有 Python 编程和基础机器学习知识的初学者。
Generative AI Handbook	一份系统化的生成式人工智能学习路线手册，包含 9 大章节，涵盖生成式 AI 的基础知识、实践应用和最新研究进展，并提供相关学习资源。
LingoHut	一个免费的在线语言学习平台，提供 45 种语言的学习资源，包括 125 节免费课程，涵盖日常用语和词汇，让你用母语轻松学习新语言。
WTF-zk	一份零知识证明入门教程，旨在让仅有高中数学基础的人也能入门零知识证明（zk）技术，教程使用 Python 复现相关算法，需要一些 Python 语言基础。
100-exercises-to-learn-rust	一套包含 100 个练习的 Rust 编程语言学习资源，旨在帮助初学者从零基础逐步掌握 Rust 编程，最终能够独立编写 Rust 程序。
《大语言模型》电子书籍	一份全面系统地介绍大语言模型的电子书籍，涵盖基础理论、预训练、微调与对齐、使用部署以及评测与应用等内容，为大模型入门读者提供详细指南。
《动手学大模型》	一份由上海交通大学《人工智能安全技术》课程讲义拓展而来的编程实战教程系列，通过实践帮助快速入门大模型相关技术，涵盖模型微调与部署、提示学习与思维链、知识编辑、模型水印以及大模型智能体与安全等内容。
machine-learning-list	一份较为全面的机器学习教程，涵盖从基础到高级再到实践与应用的内容，包括基本简介、训练和微调、推理和执行策略、数据集处理以及实际应用等。
50projects50days	一份包含 50 个使用 HTML、CSS 和 JavaScript 实现的小 Demo 的项目集合，适合前端初学者练手，涵盖动画导航、背景滑块、密码生成器、图像轮播等常见网站功能。
self-llm	一份超全面的开源大模型使用指南，提供针对国内初学者的开源大模型教程，通过 AutoDL 平台简化模型部署、使用和应用流程。
SQL之母	一个免费的闯关式 SQL 自学网站，通过有趣的交互式关卡，让初学者从 0 到 1 掌握常用 SQL 语法，包括 30 多个关卡、在线提交代码、自由选择关卡、自定义关卡等功能。
Hugging Face AI Courses	一套由 Hugging Face 出品的 4 门 AI 学习课程，涵盖自然语言处理、深度强化学习、音频处理和开源 AI 开发等主题，适合初学者免费学习。
earthworm	一个通过连词构句的方式帮助用户更好地学习英语的开源项目，从简单到复杂的句子逐步训练，让用户轻松掌握写出长句子，顺便还可以练习键盘英文打字。
90DaysOfCyberSecurity	一份 90 天网络安全学习计划，涵盖网络基础、安全基础、Linux、Python、流量分析、Git、ELK、AWS、Azure 和黑客攻击等主题，提供完整的学习资源和材料。
Build a Large Language Model (From Scratch)	一套从零开始构建类似 ChatGPT 的大型语言模型（LLM）的教程，涵盖了文本数据处理、注意力机制实现、模型构建、预训练、微调等内容，帮助读者深入了解 LLM 的工作原理和实现方式。
LLM Course	一份开源免费的大语言模型课程，包含详细的学习路线和实践笔记，分为三个部分：LLM 基础教程、LLM 深入探索和 LLM 应用与部署。
CS50	一门由哈佛大学免费开放的计算机科学和编程入门课程，教授编程基本原理和计算思维方法，涵盖 C、Python、SQL、HTML、CSS 和 JavaScript 等多种语言，最后以终极项目作为收官。
English-level-up-tips	一份系统化的英语学习指南，包含认知、单词、听力、阅读、口语、写作和扯淡等七大章节，旨在帮助用户高效提升英语水平。
Docker 从入门到实践	一份全面的 Docker 学习指南，涵盖了 Docker 的基础知识、安装、镜像使用、容器操作、数据管理、网络配置等内容，旨在教会新手有效使用 Docker。

↥ 返回目录

实用工具

项目	简述	源
VutronMusic	一款高颜值的第三方网易云音乐播放器，支持本地歌曲、离线歌单播放，读取歌曲封面、内嵌歌词等功能，同时支持线上信息匹配、云盘、对歌曲评论，界面纯净无广告。
Squish	一款开源免费的图像压缩工具，支持多种图像格式，具有高性能压缩优化功能，能保持图像质量不变，并提供批量处理、格式转换、实时预览等功能。
PairDrop	一款开箱即用的跨平台文件传输工具，无需注册和安装，只需浏览器即可在同一局域网内任意设备之间自由传输文件，还支持创建临时公共房间实现公网传输，采用点对点传输保证数据安全隐私。
Checkmate	一款高颜值的可视化实时监控工具，支持网站、页面加载速度、Docker 容器、Ping 响应等多种监控类型，并提供服务器 CPU、内存、磁盘和温度等状态信息。
MyIP	一款开源的 IP 工具箱，可以检查 IP 地址、地理位置、DNS 泄露、网速测试、Ping 测试和网站可用性等，提供 258 项安全检查清单及详细解释。
Presenterm	一款开源工具，能够在终端上演示由 Markdown 创建的精美 PPT，支持代码高亮、图片展示、LaTeX 公式渲染等功能，内置丰富主题风格并可自定义，还可导出 PDF 文件。
File Converter	一款实用的文件格式转换工具，支持视频、音频、图片和文档等多种文件格式转换，可以快速完成文件格式转换和压缩，支持批量处理。
CodeSnap	一款基于 Rust 开发的代码截图工具，可离线使用，支持 Sublime Text 代码语法高亮主题，可自定义字体、行号、水印等，支持 PNG、SVG 和 HTML 输出格式。
IP Helper	一款简洁的 IP 信息查询工具，支持 Web、命令行、Telnet 和 FTP 等多种查询方式，提供 IP 地理位置查询、Token 认证和自定义域名等功能。
FastSend	一款基于 WebRTC 技术的开源免费点对点文件传输工具，支持快速安全的文件和目录同步传输，具有加密传输、局域网优化、简洁界面等特性。
academic-project-page-template-vue	一个基于 Vue 开发的学术论文项目页面模板，支持轮播图、嵌入视频、数据可视化等丰富功能，还能一键复制引用格式，内置评论系统，非常适合研究人员快速直观地呈现论文基本信息。
SiteOne Crawler	一款简单易用且功能强大的网站分析工具，可一键完成网站分析、性能检测、SEO 优化建议，并导出完整的离线 HTML 分析结果，用于网站分析优化。
wechatDataBackup	一款微信聊天记录导出工具，提供类似微信的简易操作界面，支持一键导出聊天记录、图片、视频、链接、语音、文件和表情等信息。还支持按类型、日期、群成员进行检索，以及多开账号选择导出和数据切换。
Neko	一款开源的虚拟浏览器工具，支持多人同时访问和控制同一浏览器界面，实现画面和声音同步，内置聊天系统、文件传输等功能，可用于远程协作、在线观影等场景。
Koodo Reader	一款开源免费的跨平台电子书阅读器，支持多种格式导入、多种阅读模式、听书、翻译、词典等功能，并提供个性化排版、笔记标注等功能。
WindTerm	一款高颜值且功能强大的跨平台终端工具，支持多种协议连接、文件传输、多语言界面、命令补全、文件管理、分屏显示等功能，具有高性能、低延迟和低内存占用的特点。
MarkText	一款高颜值的开源 Markdown 编辑器，界面简洁干净，提供实时预览、多种主题和编辑模式等功能，支持数学公式、表情符号、快捷键等，可导出 HTML 和 PDF 文件，适用于 Windows、macOS 和 Linux 系统。
RightMenu Master	一款 Finder 右键菜单增强工具，内置多个实用预设操作，支持移动复制文件、创建新文档、代码高亮预览、快捷打开常用文件夹和终端等，旨在提升 Mac 用户的工作效率。
File Centipede	一款功能强大的文件上传/下载管理器，支持多种协议如 HTTPS、BT /磁力链接、FTP、WebDAV、m3u8 等，内置多种实用辅助功能，还提供浏览器插件抓取网站音视频。
Paperless-ngx	一款功能强大的文档管理系统，能够将纸质文档转换为在线可搜索的文档，支持 OCR 技术、自动分类、多种文件格式、多语言识别、全文搜索等功能，界面美观，提供权限管理。
Your Next Store	一套基于 Next . js 开发的 Shopify 独立站搭建开源解决方案，使用 Stripe 管理产品及支付，界面简洁美观，提供完整的购物流程。
ToastFish	一款利用 Windows 通知栏背单词的开源工具，可在工作或上课时隐蔽地背诵英语单词，支持自定义词库、设置背诵数量和单词测试等实用功能。
Puter	一款开源免费的个人云电脑工具，提供丰富功能、极致体验和高度个性化定制，可作为网盘、开发环境、远程桌面等使用，支持自托管保护隐私。
MD3-Windows	一款基于 Material You 设计风格的 Windows 桌面美化工具，能根据壁纸自动调整配色，提供丰富的桌面小组件和自定义功能，打造整洁实用的桌面体验。
fzf	一款功能强大的命令行模糊查找器，通过模糊匹配算法快速筛选列表中的命令、文件、历史记录等，支持多种显示模式和自定义选项，并与 bash、zsh、fish、Vim 和 Neovim 等集成。
NewsNow	一款优雅的实时热门新闻阅读工具，包括国内、国际、科技、财经等多个类别，整合了微博、知乎、抖音、澎湃、GitHub、华尔街、Hacker News、IT 之家等多个平台的热门资讯。
Joplin	一款开源的跨平台笔记应用，支持 Markdown 语法、多种内容插入和管理，可作为印象笔记、OneNote 的替代品，提供多种同步、导入导出、加密等功能，适用于学习整理、会议记录、文档管理等场景。
TV	一款 IPTV 电视直播源更新工具，可自定义频道菜单，根据模板频道自动获取并更新最新直播源接口，生成可用接口文件，支持多种运行方式。
QtScrcpy	一款开源的手机投屏电脑工具，支持通过 USB 或 WIFI 无需 root 权限连接 Android 设备到电脑，提供实时屏幕显示、键鼠控制、录制截图、文件传输等丰富功能。
LosslessCut	一款高效的无损音视频剪辑工具，能够在几秒内完成对音视频的无损压缩，特别适用于高清视频文件。还支持快速提取视频精华、添加配乐字幕轨道等功能。
1Remote	一款开源免费的远程会话管理工具，支持多种协议如 RDP、VNC、SSH 等，具有快速启动、多显示器支持、细节配置、界面定制等特性，提升远程管理效率。
wechat-article-exporter	一款开源的微信公众号文章批量下载工具，可在线导出公众号文章，支持音视频导出和私有部署，能还原文章样式，过滤已删除文章，根据规则自动下载并缓存数据。
LocalSend	一款无需联网的跨平台本地文件传输工具，类似 AirDrop，可在本地网络上安全共享文件和消息，简单易用且传输速度超快。
思源笔记	一款隐私优先的个人知识管理系统，支持细粒度块级引用和 Markdown 所见即所得编辑，拥有实时渲染、数学公式、图表、导出 HTML、Markdown 文件、AI 写作等特性，支持跨平台使用。
1Panel	一款开源的 Linux 可视化管理工具，提供实时监控、文件管理、快速建站、一键安装应用、计划任务管理等功能，让用户更方便地管理 Linux 服务器。
Docker-OSX	一款开源免费的黑苹果部署工具，可在 Windows 或 Linux 的 Docker 容器中一键部署安装 macOS 系统，支持共享文件夹、USB 设备热插拔、音频驱动配置等功能，提供接近原生的体验。
eSearch	一款开源的多功能工具，具有截屏、OCR、搜索、翻译、贴图、以图搜图以及录屏等功能，支持 Windows、Linux 和 macOS 系统。
GoGoGo	一款开源免费的虚拟定位 APP，无需 ROOT 权限即可轻松修改位置信息和模拟移动，提供历史记录、位置搜索和直接输入坐标等功能。
Notify Me	一款开源免费的 Android 应用，可以捕获手机来电和短信信息，并将数据转发到 Bark 服务器或电子邮箱，实现跨设备及时接收重要通知。
immich	一款开源免费的高性能照片和视频自托管解决方案，支持内容自动备份、快速上传下载、共享相册、足迹地图、内容时间线索引和元数据搜索等功能，提供 Docker 部署方式和安卓、iOS 客户端。
cloudflare_temp_email	一个基于 CloudFlare 免费服务搭建的功能全面的临时邮箱开源项目，支持收发邮件、访问密码、自动回复、查看附件等多种功能。
ProxyCat	一款能将临时 IP 变成固定 IP 的代理池中间件，支持多协议、动态获取、自动验证，配合高并发异步处理，适用于需要频繁更换 IP 的网络操作场景。
Nginx UI	一款功能强大且界面友好的 Nginx 可视化管理工具，提供服务器监控、证书管理、配置编辑、日志分析等多种功能，支持多语言和深色模式。
imFile	一款基于 Motrix 开发的开源免费下载工具，界面简洁易用，支持 HTTP、FTP、BT、磁力链等多种资源下载，具有多线程下载、限速、模拟 UA、系统通知等功能。
AB Download Manager	一款开源免费的下载管理器工具，能够提升高达 500 %的下载速度，批量管理下载队列，自定义下载速度，并提供浏览器扩展自动获取下载链接。
Files	一款基于 Windows 设计的功能强大的文件管理器工具，支持标签页管理多个文件夹、双面板对比文件夹内容、云端存储集成、压缩文件处理以及 Git 项目管理等功能。
123pan	一款能突破 123 云盘下载速度限制的 Python 脚本工具，提供列出文件、下载文件、上传文件、分享文件等功能，为 Windows 用户提供开箱即用安装包。
BBackupp	一款开源免费的 iOS 设备备份工具，支持备份到外置硬盘、监控备份进度、加密备份，可以无缝迁移到新设备。
md	一款开源的微信 Markdown 编辑器，支持自定义 CSS 样式、Markdown 语法、LaTeX 公式，提供多种实用功能如格式化、取色、图片上传等，界面简洁好用。
FSNotes	一款简洁且功能强大的笔记管理工具，支持 Markdown 和纯文本文件、多文件夹存储、代码语法高亮、内联图片、跨笔记链接、暗黑主题等丰富功能，适用于 macOS 和 iOS 系统。
Real-Address-Generator	一个基于 Cloudflare Workers 的真实地址生成器，可以生成不同国家的真实随机地址，包括姓名、性别、电话号码和详细地址等信息。
GeekDesk	一款高颜值、功能强大的桌面快速启动管理工具，集成了强大的文件搜索、自定义快捷键、壁纸、界面效果调整、系统图标选择和定时提醒等功能，旨在为用户打造高效且个性化的极客桌面体验。
Spacedrive	一款基于 Rust 开发的跨平台文件管理器，提供安全且直观的文件管理体验，能够结合云端存储和本地硬盘，打造个人分布式云。功能丰富，包括文件预览、识别、搜索、多视图布局、AI 标签、云存储支持、个性化主题和多语言支持等。
Pake	一款基于 Rust 的开源工具，能将任何网页轻松打包成多端桌面应用，相比 Electron 更加小巧、快速、高性能、低内存占用。
Certimate	一款开源的 SSL 证书管理工具，可自动申请、部署 SSL 证书并在证书即将过期时自动续期，提供安全便捷的 SSL 证书管理解决方案。
SimpleMindMap	一款基于 JavaScript 实现的开源 Web 思维导图工具，具备思维导图核心功能，支持插件扩展更多实用功能，提供在线体验，上手简单。
Plate	一款基于 React 的功能强大且高颜值的 Markdown 富文本编辑器框架，提供 50 多个插件开箱即用，支持无缝系统集成和组件定制开发，UI 响应式设计支持深浅色主题。
fscan	一款开源免费的内网综合扫描工具，支持主机存活探测、服务爆破、redis 批量写公钥、web 指纹识别和漏洞扫描等功能，可一键自动化全方位进行漏扫扫描。
Simple Live	一款开源免费的聚合直播工具，集成了各大主流直播平台，支持多账号登录、观看记录、链接解析、弹幕设置等功能，提供 Android、iOS、Windows、macOS 和 Android TV 客户端。
xiaomusic	一款开源免费的工具，可让小爱音箱播放本地音乐，支持多种音频格式，语音控制播放，自动搜索下载匹配歌曲，支持网络歌单、电台、Docker 部署等功能。
DeeplxFile	一款基于 Deeplx / playwright 实现的简单易用、快速且免费的文件翻译工具，不限制文件大小，支持超长文本翻译，使用 DeepL 作为翻译源，能够翻译超长且复杂的文档，如超大的 Excel 文件，甚至还能正确显示复杂的引用公式。
IPTVnator	一款开源的跨平台 IPTV 播放器，界面简洁且功能强大，支持导入播放列表、自动更新、频道搜索、电视存档、分组频道列表、多种主题等，支持外部播放器和 HTML 视频播放器，提供多语言支持。
EcoPaste	一款开源免费的剪贴板管理工具，具有轻量小巧、快捷唤醒、本地存储、智能分组、收藏功能、内置搜索、图片 OCR 等特性，功能全面且操作便捷。
RenderCV	一款基于 LaTeX 的简历框架，内置多种主题，可从 YAML 输入文件生成 PDF、LaTeX、Markdown、HTML 和 PNG 格式的高质量简历文档。
cloudflare-saas-stack	一套基于 Cloudflare 的全栈 SaaS 开发模板，可快速在 Cloudflare 上构建并部署全栈应用，包括数据库、认证、样式、存储等功能。
Pearcleaner	一款开源免费且简洁好用的 Mac 软件卸载工具，提供高颜值简洁的操作界面，支持多选一键卸载软件并清理残留文件。
Fideo	一款开源免费且易于使用的直播录制工具，能够简单快捷地录制 YouTube、TikTok、抖音、快手、B 站、微博、淘宝等平台的直播内容，并保存为 MP4 格式的视频文件，提供开箱即用的安装包，支持 Windows 和 macOS 系统。
vCard	一款使用 HTML、CSS 和 JavaScript 构建的纯静态网站模板，界面布局适配所有设备，可用于搭建在线个人介绍或个人简历。
Pintree	一款将浏览器书签转换成美观导航网站的开源工具，可直观展示和管理大量书签，提高使用效率。
ray.so	一套来自 Raycast 团队的开源 Web 工具箱，提供代码图片生成器、图标制作器、高颜值 Raycast 主题等多种实用工具。
PDF Guru	一款开源的功能强大的 PDF 工具箱，支持水印去除、批量添加书签、OCR、格式转换、合并、拆分、插入、替换等多种 PDF 实用功能，还提供卡片制作、视频笔记、字段批量修改等额外功能，可大幅提高办公和学习效率。
Revezone	一款以图形为中心的思维管理工具，集成了无限画布白板、所见即所得编辑器、边栏文件管理和灵活布局等功能。
MediaGo	一款开源免费的视频在线提取工具，支持流媒体下载、视频下载、m3u8 文件下载以及 B 站视频下载，具有无需抓包、移动播放、批量下载等特性，提供 Windows 和 macOS 桌面客户端。
Flow	一款开源免费的在线 ePub 阅读器，支持书内搜索、多主题切换、图片预览、链接分享、下载书籍、自定义排版、高亮笔记、数据导出以及云端存储等功能，兼容跨平台使用，可离线运行，允许同时阅读多本书。
Auto-SSL	一个自动化管理域名 SSL 证书的开源项目，利用 GitHub Actions 和 acme . sh 实现自动申请、续期 SSL 证书，并将证书保存到 Git 仓库。
PicX	一款基于 GitHub API 的图床工具，提供图片上传托管、生成图片链接和常用图片工具服务，具有开箱即用、操作简单、数据安全等特点。
academic-homepage	一个开源的 GitHub Pages 模板，可快速搭建个人学术主页，包含个人简介、教育经历、发布学术列表等内容，页面布局兼容移动端。
spotify_player	一款快速、易用且高度可配置的终端 Spotify 播放器，具有极简界面、个性化配置、远程控制、歌词显示、跨平台支持、图像渲染、桌面通知等功能，提供丰富的 CLI 命令。
ShotEasy	一款开源免费且无限制的在线图片编辑工具，能够进行裁剪旋转、添加水印、添加圆角、背景美化、背景移除替换以及快速压缩等操作，所有编辑都在浏览器中进行，不会上传到服务器，保护数据隐私安全。
Image ToolBox	一款开源免费且功能强大的图片编辑器，提供批量裁剪、滤镜、图片提取文本、图片拼接与叠加、背景移除、水印添加、格式转换等多种功能。
Docmost	一款开源免费的协作文档和 Wiki 工具，提供富文本编辑器、表格、数学公式等功能，支持多人实时协作编辑，内置权限管理系统，可作为 Confluence 和 Notion 的替代品。
Image Watermark Tool	一款开源免费的图片水印添加工具，可在本地设备上为图片添加水印，支持自定义水印颜色、大小、透明度和倾斜角度，保护隐私。
ProxyPin	一款开源免费的跨平台抓包工具，基于 Flutter 开发，界面简洁易用，可用于拦截、检查和重写 http（s）流量。
Quick Waitlist	一款开源项目，帮助快速搭建精心设计的 Waitlist 页面，集成 Resend 进行邮件管理和自定义模板，提供 SEO 优化解决方案，并支持通过 Vercel 一键部署，适合发布任何产品或服务时收集潜在客户邮箱地址。
Telegram Media Downloader	一个开源免费的 Telegram 媒体下载脚本，可在 Telegram 网页版上自由下载图片、GIF 和视频等内容，即使是私人频道限制下载功能也能解锁。
social-auto-upload	一款开源免费的自动化短视频上传和分发工具，支持主流视频平台如抖音、B 站、视频号、TikTok 等，可以多账号矩阵化执行任务，并支持自定义复杂发布时间。
cobalt	一款开源免费的视频下载工具，支持 B 站、YouTube、Instagram、TikTok 和 Twitter 视频下载，无广告干扰。
Hidden Bar	一款开源免费的 Mac 菜单栏管理工具，可以一键折叠隐藏菜单栏中的图标，自由调整图标的显隐状态，使菜单栏变得干净整洁。
Caesium Image Compressor	一款开源免费的图像压缩工具，可以在保持图像质量的前提下减小文件大小，支持 JPG、PNG 和 WebP 格式，提供浏览器版本和跨平台客户端。
IT-Tools	一套专为开发人员打造的在线实用工具库，包括 Token 生成器、大小写转换器、进制转换器、二维码生成器、Git 备忘录等，可通过 Docker 一键部署到自己的服务器上。
res-downloader	一款开源免费的网络资源嗅探下载器，可获取视频、音频、图片以及 m3u8 等不同类型资源，支持微信视频号、抖音、快手、小红书、QQ 音乐等主流平台。
Stream-rec	一款开源免费的直播录制工具，支持虎牙、抖音、斗鱼和 Twitch 等多个平台，可自动录制直播视频和弹幕，支持自定义录制质量、格式等设置，并提供 Web 界面配置和 Docker 部署。
Depth of Field Simulator	一个在线景深模拟器工具，可以通过调整光圈、焦距和拍摄对象距离等参数，直观展示这些因素对景深的影响，非常适合摄影爱好者学习和掌握拍摄技巧。
Douyin-Vue	一个模仿抖音的移动端短视频项目，使用最新的 Vue 技术栈开发，实现媲美原生 App 丝滑流畅的使用体验，可作为学习 Vue 最佳实践的项目。
QuickRecorder	一款多功能、轻量化、高性能的 macOS 屏幕工具，支持窗口录制、App 录制、窗口声音内录、鼠标高亮、隐藏桌面文件等功能，体积仅 4MB，操作简便。
Hanzi Writer	一个开源免费的 JavaScript 库，可根据汉字书写时的笔画顺序播放正确的描边动画和练习测试，包含 9000 多个常用汉字，库大小仅 30kb，可嵌入 Web、iOS 或 Android 等平台应用。
Pingvin Share	一款可自建的文件分享工具，支持自定义链接、上传限制、有效期、访问次数、密码保护等，并提供反病毒检查，界面简洁易用，可通过 Docker 一键部署到私有服务器。
Memos	一款开源免费的轻量级笔记服务，像极简版微博社区，可记录想法、知识和笔记，支持公开分享。
DrawDB	一款开源免费的在线数据库设计工具及 SQL 生成器，支持主流数据库，可轻松创建数据库图表、导出 SQL 脚本和自定义编辑器。
My TV	一款开源免费的安卓电视直播软件，内置直播源，安装即可使用，具有稳定、快速、免费和无广告等特点。
Seal	一款开源免费的 Android 视频/音频下载工具，底层使用 yt - dlp 下载框架，支持数千个视频平台，界面简洁友好，遵循 MD3 设计规范。
sensitive-word	一款基于 DFA 算法实现的高性能 Java 敏感词过滤工具框架，内置 6W +敏感词库，支持敏感词判断、返回、脱敏等操作，具有性能优异、功能丰富、使用简洁等特点。
Motrix	一款开源免费的全能下载工具，界面简洁易用，支持 HTTP、FTP、BT、磁力链等资源下载，具有多线程下载、BT 选择性下载、自动更新 Tracker 等功能，可替代迅雷。
Soybean Admin	一款清新优雅、高颜值且功能强大的开源后台管理模板，采用前沿技术栈，提供丰富的主题配置、组件和一站式后台管理解决方案，无需额外配置即可使用。
Reminders MenuBar	一款专为 macOS 设计的提醒菜单栏小工具，可以直接在菜单栏上管理提醒事项，支持同步、创建、编辑和筛选等操作，提高工作效率。
Zealot	一款开源的自部署托管移动应用和 macOS 应用的分发平台，提供 iOS、Android SDK、fastlane 等丰富组件库，支持企业打包分发流程、上传应用全套流程方案，可接入任意 CI 系统实现自动化处理。
Gmeek	一款超轻量级的开源个人博客框架，无需本地部署，基于 GitHub Pages、GitHub Issues 和 GitHub Actions，仅需 3 步 18 秒即可搭建好博客并开始写作。
DevHub	一款专为开发者打造的 macOS 应用工具箱，集成超过 30 种常用小工具，如 API 请求、UUID 生成器、编解码工具等，无需联网即可使用，旨在提升开发效率。
pinyin-pro	一款专业的 JavaScript 汉字拼音转换库，功能丰富、准确率高、性能优异，支持多种拼音转换模式，包括拼音/声母/韵母/首字母/音调等，并提供拼音输入转换、文本匹配等高级功能。
Vmail.DEV	一款开源免费的临时邮箱创建工具，使用 Cloudflare email worker 实现，支持接收、显示和存储电子邮件，可自托管部署。
Lux	一款跨平台的视频下载命令行工具，使用 Go 语言编写，支持几乎所有视频平台如抖音、B 站、YouTube 等，可控制下载视频的格式、清晰度和字幕。
Applite	一款开源免费的 Homebrew 可视化工具，提供简洁易用的界面，让非技术用户轻松安装和管理第三方应用。
pear-rec	一款简单高效且功能齐全的日常工具，支持截图、录屏、录像、录制 GIF、查看多媒体文件和修改图片等，同时支持跨平台，目前提供 Windows 安装包。
Huntly	一款开源免费的 RSS 阅读器，支持 RSS 订阅、全文搜索、自动保存等功能，可自托管或本地运行。
LapisCV	一份基于 Markdown 格式的简洁正式简历模板，适用于 Obsidian / Typora，开箱即用，风格简洁正式，适合求职求学场景。
Hitomi Downloader	一款开源免费的视频下载工具，提供简单易用的操作界面，支持多个视频平台如 B 站、YouTube、抖音等，可下载视频、图像、音频和文本文件。
WeWe RSS	一款优雅的微信公众号订阅工具，支持微信公众号订阅以及 RSS 生成，后台自动定时更新内容，支持全文内容输出，让阅读无障碍。
GitButler	一款由 GitHub 联合创始人开源的高颜值且好用的可视化 Git 客户端，支持自动创建分支、管理分支合并、使用 AI 编写提交信息、快速解决问题等功能，提升 Git 使用效率。
Web-Check	一款功能强大的开源免费网站分析工具，可以分析任何网站的 IP 信息、SSL 链、DNS 记录、Cookie、域名信息、服务器位置、网站性能等信息。
Stats	一款开源免费的 Mac 系统监控工具，功能全面，可监控 CPU、GPU、磁盘、内存、网络、电池、传感器、风扇、蓝牙设备和多时区时钟等，是 iStat Menus 的良好替代品。
NewPipe	一款开源免费的轻量级 YouTube 第三方客户端，无需 Google 服务框架和 YouTube 账号登录，支持 4K 视频播放、画中画模式、搜索视频/音频/频道/播放列表，可下载视频/音频/字幕等功能丰富。
Czkawka	一款开源免费的简单、快速且强大的不必要文件清理工具，支持扫描重复、空文件夹、大文件、相似多媒体文件、无效文件等内容，扫描速度极快，支持多语言和多平台。
campus-imaotai	一款自动化工具，可以在 i 茅台 App 上自动完成注册、预约、旅行等操作，支持多账号、门店选择、耐力值查询、消息推送等功能，并提供 Docker 一键部署。
Loop	一款开源免费的 MacOS 屏幕窗口管理工具，可轻松拖拽、调整和排列窗口，提高工作效率，支持自定义颜色和设置。
ChatGPT Source Watch	一个监控 ChatGPT 前端代码变化的工具，可以提前发现即将推出的新功能，为 ChatGPT 用户和竞争对手提供有价值的信息。
Google Indexing Script	一个开源脚本，利用 Google API 可以在 48 小时内让你的网站在 Google 上建立索引，无需复杂技术或黑客手段。
Reactive Resume	一款独特的免费开源简历生成器，支持中文和多种配置，可轻松创建、更新、共享和打印简历，支持个性化主题和布局，并集成 GPT 辅助编写修改。
cmd-wrapped	一款命令行工具，可以统计和分析过去一年中用户在终端上使用的命令情况，包括每日最活跃时段、常用命令统计等，并生成类似 GitHub 年度贡献图的命令分布图。
Look Scanned	一款纯前端网站工具，能够让 PDF 文件看起来像是扫描件一样，无需打印机和扫描仪，操作简单快捷，具有隐私保护、个性化调整和跨平台支持等优点。
China Holiday Calendar	一份开源维护的中国节假日补班日历，提供 2022 - 2024 年中国节假日、调休、补班信息，支持 ICS 格式和 API 获取，可订阅到 iPhone、Google Calendar、Outlook 等客户端。
movie-web	一个极简且易用的开源电影网站，支持搜索和在线观看电影，具有自动保存播放进度、收藏电影、界面简洁、搜索响应快等特点。
Spotube	一款开源的跨平台 Spotify 客户端，无需付费订阅，支持离线播放、歌词同步等功能，注重隐私且体验流畅。

↥ 返回目录

实用插件

项目	简述	源
FastURL	一款实用的开源 Chrome 插件，可以在任何输入框中通过快捷键粘贴网页正文的 Markdown 格式内容，支持清理广告和导航栏等多余内容，同时保留标题、列表、链接等 HTML 元素，操作在本地浏览器完成，速度快且数据安全。
WebChat	一款去中心化、无服务器的匿名聊天浏览器插件，利用 WebRTC 端到端加密技术，所有聊天数据保存在本地，确保安全隐私。无需注册，可在任何网站一键开启与他人聊天，适合即时交流需求场景。
Screenity	一款功能强大的 Chrome 录屏插件，支持屏幕任意位置录制、标注、录音，可导出多种视频格式并进行简单编辑。
Cloud Document Converter	一款开源的 Chrome 插件，支持下载和复制飞书文档为 Markdown 格式，兼容多种元素如标题、代码块、列表、表格、图片等。
uBlockOrigin-HUGE-AI-Blocklist	一份综合性的 AI 内容过滤列表，包含约 950 个网站，可用于 uBlock Origin 和 uBlacklist，主要目的是清理 Google Search、DuckDuckGo 和 Bing 等图像搜索引擎中的 AI 生成内容。
Luxirty Search	一款基于 Google 的开源搜索引擎插件，屏蔽内容农场、无广告、无跟踪、界面简洁，提升 GitHub、StackOverflow 等技术社区的搜索权重。
materialYouNewTab	一款采用 Google Material You 设计风格的浏览器主页插件，提供自定义搜索引擎、快捷网站图标、主题切换等功能，无需权限即可使用。
visual-try-on	一款基于快手 Kolors 图像模型开发的 Chrome 浏览器插件，可在任何电商网站上上传自己的照片，一键试穿喜欢的衣服。
code-box	一款开源免费的浏览器插件，适用于主流技术社区网站，提供一键复制代码、阅读全文和去除登录弹窗等实用功能，为技术开发者提供便利。
ComfyUI_Bxb	一款开源插件，能够将 ComfyUI 工作流转换成小程序，支持主流平台和支付变现功能，方便将现有工作流推广到更广阔的市场。
vscode-drawio	一款将 draw . io 集成到 VSCode 的插件，让开发者能够在 VSCode 中快速绘制流程图、思维导图或 UML 类图。
Aide	一款实用的 AI 编程助手 VSCode 扩展，为现有编程助手（如 Copilot、CodegeeX 或 Codeium 等）提供功能补充，让开发者能更高效地编程。
Reborn AI	一款开源的 VSCode 插件，能够帮助编写、重构和改进代码，支持多种语言模型和语种，包括中文和英文等。
gh-dash	一款高颜值的 GitHub CLI 扩展，以仪表板形式展示 GitHub 项目的拉取请求和问题信息，支持搜索、过滤、操作等丰富功能。
cat-catch (猫抓)	一款开源免费的网页资源嗅探插件，能够筛选列出当前页面的资源，提供嗅探、缓存捕捉、视频录制等工具帮助抓取资源，支持自定义抓取资源类型。
Responsive Viewer	一款开源的 Chrome 浏览器插件，可以在一个界面中同时展示多个屏幕尺寸，方便前端开发者测试响应式网站或应用在不同设备尺寸下的显示效果，提供多种设备尺寸选择、横竖屏显示和一键截图分享等功能。
kiss-translator	一款简约且好用的双语对照翻译插件和油猴脚本，支持多种浏览器和翻译服务，提供网页双语对照翻译、输入框翻译、YouTube 字幕翻译等功能，支持自定义翻译规则、译文样式和快捷键。
Stellar	一款内置文档系统的简约商务风 Hexo 主题，支持丰富的标签和动态数据组件，可帮助简单应对各种表达需求。

↥ 返回目录

资料集合

项目	简述	源
Awesome LLM Apps	一份精心整理的大语言模型在不同领域实际应用的集合，涵盖客服、法律、健康、旅游、数据分析等场景，以及结合 RGA 和具备记忆功能的应用，还收集了 LLM 微调、高级工具和框架使用教程等，提供详细介绍。
web3.0	一份颇为全面的学习 Web3 资料集合，涵盖 Web3 基础知识、应用及项目、相关分享博主、视频教程、相关书籍、开发资源以及工具等内容，还整理了一些与 Web3 相关的招聘工作机会。
How-To-Secure-A-Linux-Server	一份关于如何保护 Linux 服务器安全的操作指南，涵盖 SSH 配置、基本安全措施、网络防火墙、系统审计和其他安全增强措施，并提供详细步骤和建议。
awesome-falsehood	一份整理了程序员在开发过程中常犯的错误观点集合，包括时间日期处理、地理位置、国际化、电子邮件格式化、邮政地址等场景，旨在帮助开发者认识并避免这些误解。
m3u直播源收集列表	一份包含电视、直播、MyTV 以及体育等订阅列表的 m3u 直播源收集列表，持续更新。
awesome-LLM-resourses	一份颇为全面的大语言模型（LLM）资源集合，涵盖数据、微调、推理、知识库、智能体、书籍、相关课程、学习教程以及论文等内容，旨在成为全球最佳 LLM 资源汇总。
awesome-low-level-design	一份关于学习低级设计（LLD）的资源集合，涵盖面向对象编程概念、设计模式、UML 图、面试题目和相关书籍推荐等内容。
Prompt_Engineering	一份全面的提示词学习指南，涵盖从基础到高级的提示工程技术，包括基本概念、思维链、角色提示、结构提示以及可直接使用的提示模板等内容，旨在帮助学习和使用大语言模型。
Awesome_Math_Books	一份收集了各种数学领域经典书籍的集合，包括概率论、代数、微积分等，提供下载链接或在线阅读，还有高中数学问题和物理学基础知识书籍。
awesome-remote-job	一份整理了关于远程工作职位和资源的列表，涵盖文章、视频、书籍、招聘网站、面试资料、问答社区、远程工作社区、会议工具以及法律和财务建议等内容，为寻找远程工作提供帮助和有用信息。
Awesome Indie Hacker Tools	一份收录独立开发/出海开发相关技术栈及工具的 GitHub 仓库，涵盖原型设计、UI 框架/组件库、图标/字体库、前端/后端开发框架、支付、国际化、产品发布推广等实用内容。
DevOps-Guide	一份整理了丰富的 DevOps 学习资源列表，涵盖 CI / CD、数据库、开发运维实践、面试准备、操作系统、网络、终端命令等多个方面，并提供了入门 DevOps 的学习指南。
出海网站经验总结	一份关于出海网站收款全流程的总结，包括公司注册、申请 EIN、申请银行卡、激活 Stripe 等内容，对于将业务拓展到海外的产品非常有帮助。
Awesome-Dify-Workflow	一份整理了一系列好用的 Dify 工作流集合，包括中英文翻译、知识库聊天、搜索大师、爆款标题、文章仿写、运营一条龙、聊天机器人、代码生成等，所有工作流基本可免费使用。
Awesome_GPT_Super_Prompting	一份在 GitHub 上整理的 ChatGPT 超级提示词集合，收集了 ChatGPT 越狱提示词、GPT 智能体提示词、提示词注入及保护、各种 GPTs 提示词以及 Prompt Engineering 学习资料等相关资源，内容全面且持续更新。
Free Public APIs	一份收集了大量免费 API 的网站，每天定时测试 API 可用性，确保收录的 248 个 API 涵盖金融、娱乐、音乐、旅行、交通等类别均可供学生和开发者使用。
system-design	一份精心收集整理的系统设计面试资源列表，提供知名科技公司实战案例和系统设计基础知识，帮助通过系统设计面试。
awesomeMLSys	一份精选的机器学习系统入门资料集合，包括论文、视频和代码库，涵盖注意力机制、性能优化、量化、长上下文长度和稀疏性等主题。
RAG_Techniques	一份全面的检索增强生成（RAG）技术学习集合，涵盖 22 种 RAG 技术，提供详细文档、实现指南和示例代码。
ComfyUI-Yolain-Workflows	一份较全面的 ComfyUI 工作流合集，涵盖从基础到进阶再到落地应用，包括图生图、细节修复、背景去除、重绘/扩图、人脸迁移、重绘光影、电商产品主图等工作流。
developer2gwy	一份开源的公务员考试最佳实践指南，由从程序员转行成为公务员的朋友们共同编写，涵盖公考基础知识、备考实践、常见问题、面试手册和问答等内容。
llama3-Chinese-chat	一份关于针对 Llama 3 进行中文微调的教程和工具集合，包括微调、魔改版本权重、训练、推理、评测以及部署 Llama 3 和 Llama 3 . 1 的视频和文档等。
中文网字计划	一套可商用免费的中文 Web 字体库，收录众多中文字体，提供在线加载和查看字体信息的功能，并有全字符集中文渲染方案。
OpenAlternative	一份收集主流软件开源替代品的列表，对每个替代品进行了详细介绍，并列举了更多类似的开源选择，涵盖多个领域，方便查找所需开源软件。
Free Font	一份收录了 630 多种免费可商用的字体列表，包括汉字字体和英文字体，按黑体、宋体、楷体、艺术体和手绘体等分类，可在线查看并下载字体包。
Modern-CV	一套受 Awesome - CV 启发的现代简历模板集合，使用 Typst 编写，提供多种现代简历模板样式。
IPTV Channels Collection	一份收集了世界各地公开的 IPTV 频道集合，涵盖动画、商业、教育、历史、电影、音乐、科学等多种类别，并按语言、国家、地区进行分组。
awesome-social-media-downloader	一份精心收集的开源免费视频下载工具列表，包括能够免费下载油管、B 站、抖音、视频号等平台视频的工具，以及一些短视频去水印和视频压缩等辅助工具。
OI Wiki	一份免费开放且持续更新的编程竞赛知识整合站点，包含基础知识、常见题型、解题思路以及常用工具等内容，旨在帮助学习编程竞赛。
awesome-landing-page	一份精心收集的落地页模板集合，包含多种高颜值且实用的开源落地页模板，使用不同前端技术框架构建，还收集了相关制作教程和设计资源。
best-practice	一份 2024 年构建 SaaS 初创公司的最佳实践集合，涵盖技术栈、云服务、架构、数据库、AI、后端及站点生成器等资源。
海棠诗社	一个开源的精美古诗词网站，内含丰富的古诗词资源，支持多种检索和分类方式，每首诗词均有注解、翻译和评价，助力理解诗词。
Adobe-Alternatives	一份精选的 GitHub 列表，收录了可替代 Adobe 全家桶软件（如 Photoshop、Illustrator、Lightroom、Premiere 和 After Effects 等）的开源、免费或付费替代品，并使用图标标注了软件的付费模式。
GPTS-Prompt-Collection	一份收集各种精彩 GPT 提示词的集合，涵盖写作、开发、生产力、商业、教育、金融、健康、营销、销售、游戏、设计等多个领域，可以搜索关键词查找所需提示词，学习优秀提示词的写法并进行改进。
awesome-artificial-intelligence	一份精心收集的人工智能领域资源集合，涵盖机器学习、深度学习、自然语言处理等基础知识，以及应用工具、教程、书籍、视频和论文等内容。
categorized-raycast-extensions	一份收集了超过 1498 个 Raycast 第三方插件的集合，涵盖商业、设计、音视频、社交媒体、教育、娱乐、AI、自动化、实用工具等多个类别。
The Algorithms	一套收录了 Python、Java、C ++、JavaScript、Go 等多种编程语言的算法实现代码的开源算法库，包含二分查找、快速排序、斐波那契数列等热门算法。
Awesome Chinese LLM	一份精心收集和梳理的中文大语言模型（LLM）相关资源集合，包括多个垂直领域的开源微调中文模型、底层模型、数据集、微调框架、推理部署框架、评测方法和相关教程。
Awesome Cloudflare	一份精心收集的基于 Cloudflare 的各种资源列表，包括开源工具、学习指南、博客等，旨在帮助独立开发者降本增效。
cs-video-courses	一份精选的计算机科学视频课程列表，涵盖计算机科学导论、数据结构和算法、系统编程、软件工程、人工智能、机器学习等领域，包含知名大学公开课程及专业讲师课程。
data-engineer-handbook	一份面向数据工程师的学习手册，涵盖书籍、课程、面试资料、优秀博客、社区和值得关注的博主等全面内容。
awesome-generative-ai-guide	一份全面的生成式 AI 学习指南，涵盖最新研究动态、面试资料、免费课程、学习笔记等丰富内容，帮助学习者系统掌握生成式 AI 相关知识。
ComfyUI Workflows	一份包含 36 个 ComfyUI 工作流的开源合集，涵盖 14 大类别，如 Stable Cascade、3D、LLM + SD、Portrait Master、SVD 等，为 ComfyUI 学习者提供参考。
Open Source Mac Apps	一份精心整理的最受欢迎的 Mac 系统开源免费软件列表，涵盖自动化、开发工具、日常生活、音视频、生产力、社交媒体、文本文件等多个类别。
rectg	一份精心整理的 Telegram 群组、频道和机器人推荐列表，包含 5000 多个优质资源，帮助用户快速找到符合需求的优质信息渠道。
Awesome-GPTs	一份精心收集的有趣和创新 GPTs 列表，涵盖学术、写作、教育、生产力、编程、生活方式、娱乐等领域。
Awesome-GPT-Agents	一份精心收集的专注于网络安全领域的 GPT 代理列表，涵盖了网络安全助手、网络安全导师、黑客新闻、数据分析等多个方面，旨在帮助网络安全学习者和从业者提高效率。
Free Certifications	一份精心收集的免费在线课程和认证列表，涵盖常见技术栈、网络安全、数据库、项目管理、市场营销等领域，为学习者提供免费的课程资源和技能认证证书。
indie-hacker-tools	一份收录独立开发者出海技术栈和工具的合集，涵盖 Web 开发模板、Chrome 插件开发模板、前端、后端、数据库、支付等内容，旨在提升开发效率和降低成本。
Awesome-Love-Code	一份收集了 50 多份表白代码的代码库，为程序员提供浪漫的表白方式。
oh-my-foss-android	一份精心收集的实用、良心开源的安卓软件清单，涵盖应用商店、浏览器、视频播放器、相机、学习办公和日常工具等类别，经过作者严格筛选。
Awesome Database Learning	一份精心整理的数据库学习合集，涵盖课程、书籍、讲座、论文等数据库相关学习资料，帮助了解数据库内部工作原理。
The-Art-of-Linear-Algebra	一份对《给每个人的线性代数》书籍的图解笔记，通过可视化图释帮助理解矩阵分解、向量、矩阵计算和算法，提供中英日三种语言笔记。
Gemini API Cookbook	一份由 Google 在 GitHub 上分享的 Gemini API 学习手册，旨在帮助开发者更好地理解和使用 Gemini API，包括使用指南、快速入门教程和示例项目。

↥ 返回目录

其他

项目	简述	源
WordsFunny	一个全栈英语单词学习网站，涵盖多个版本的词汇资源，提供在线播读、翻译、同义词和例句等功能。
academic-project-astro-template	一个基于 Astro 和 Tailwind CSS 构建的精美学术项目页面模版，具有快速加载、响应式设计、易于访问和 SEO 友好等特性，支持使用 Markdown 编辑内容并显示标题、图片、公式、两列布局、代码块和视频等。
onur.dev	一个极简风格的三栏式个人主页网站，使用了 Next . js、Tailwind CSS、shadcn / ui、Contentful、Raindrop、Supabase 构建并部署在 Vercel 上，兼容移动端。

↥ 返回目录

声明

本作品采用知识共享署名-非商业性使用-禁止演绎 4.0 通用许可协议进行许可。

the-book-of-secret-knowledge

Sun, 03 Aug 2025 15:30:20 +0800

trimstray/the-book-of-secret-knowledge

"Knowledge is powerful, be careful how you use it!"

A collection of inspiring lists, manuals, cheatsheets, blogs, hacks, one-liners, cli/web tools, and more.

:notebook_with_decorative_cover: What is it?

This repository is a collection of various materials and tools that I use every day in my work. It contains a lot of useful information gathered in one piece. It is an invaluable source of knowledge for me that I often look back on.

:restroom: For whom?

For everyone, really. Here everyone can find their favourite tastes. But to be perfectly honest, it is aimed towards System and Network administrators, DevOps, Pentesters, and Security Researchers.

:information_source: Contributing

If you find something which doesn’t make sense, or something doesn’t seem right, please make a pull request and please add valid and well-reasoned explanations about your changes or comments.

A few simple rules for this project:

inviting and clear
not tiring
useful

These below rules may be better:

easy to contribute to (Markdown + HTML …)
easy to find (simple TOC, maybe it’s worth extending them?)

Url marked * is temporary unavailable. Please don’t delete it without confirming that it has permanently expired.

Before adding a pull request, please see the contributing guidelines. You should also remember about this:

`1`	`+ This repository is not meant to contain everything but only good quality stuff.`

All suggestions/PR are welcome!

Code Contributors

This project exists thanks to all the people who contribute.

Financial Contributors

:newspaper: RSS Feed & Updates

GitHub exposes an RSS/Atom feed of the commits, which may also be useful if you want to be kept informed about all changes.

:ballot_box_with_check: ToDo

Add new stuff…
Add useful shell functions
Add one-liners for collection tools (eg. CLI Tools)
Sort order in lists

New items are also added on a regular basis.

:anger: Table of Contents

Only main chapters:

:trident: The Book of Secret Knowledge (Chapters)

CLI Tools ^[TOC]

:black_small_square: Shells

   GNU Bash - is an sh-compatible shell that incorporates useful features from the Korn shell and C shell.
   Zsh - is a shell designed for interactive use, although it is also a powerful scripting language.
   tclsh - is a very powerful cross-platform shell, suitable for a huge range of uses.
   bash-it - is a framework for using, developing and maintaining shell scripts and custom commands.
   Oh My ZSH! - is the best framework for managing your Zsh configuration.
   Oh My Fish - the Fishshell framework.
   Starship - the cross-shell prompt written in Rust.
   powerlevel10k - is a fast reimplementation of Powerlevel9k ZSH theme.

:black_small_square: Shell plugins

   z - tracks the folder you use the most and allow you to jump, without having to type the whole path.
   fzf - is a general-purpose command-line fuzzy finder.
   zsh-autosuggestions - Fish-like autosuggestions for Zsh.
   zsh-syntax-highlighting - Fish shell like syntax highlighting for Zsh.
   Awesome ZSH Plugins - A list of frameworks, plugins, themes and tutorials for ZSH.

:black_small_square: Managers

   Midnight Commander - is a visual file manager, licensed under GNU General Public License.
   ranger - is a VIM-inspired filemanager for the console.
   nnn - is a tiny, lightning fast, feature-packed file manager.
   screen - is a full-screen window manager that multiplexes a physical terminal.
   tmux - is a terminal multiplexer, lets you switch easily between several programs in one terminal.
   tmux-cssh - is a tool to set comfortable and easy to use functionality tmux-sessions.

:black_small_square: Text editors

   vi - is one of the most common text editors on Unix.
   vim - is a highly configurable text editor.
   emacs - is an extensible, customizable, free/libre text editor, and more.
   micro - is a modern and intuitive terminal-based text editor.
   neovim - is a free open source, powerful, extensible and usable code editor.
   spacemacs - a community-driven Emacs distribution.
   spacevim - a community-driven vim distribution.

:black_small_square: Files and directories

fd - is a simple, fast and user-friendly alternative to find.
ncdu - is an easy to use, fast disk usage analyzer.

:black_small_square: Network

   PuTTY - is an SSH and telnet client, developed originally by Simon Tatham.
   Mosh - is a SSH wrapper designed to keep a SSH session alive over a volatile connection.
   Eternal Terminal - enables mouse-scrolling and tmux commands inside the SSH session.
   nmap - is a free and open source (license) utility for network discovery and security auditing.
   zmap - is a fast single packet network scanner designed for Internet-wide network surveys.
   Rust Scan - to find all open ports faster than Nmap.
   masscan - is the fastest Internet port scanner, spews SYN packets asynchronously.
   pbscan - is a faster and more efficient stateless SYN scanner and banner grabber.
   hping - is a command-line oriented TCP/IP packet assembler/analyzer.
   mtr - is a tool that combines the functionality of the 'traceroute' and 'ping' programs in a single tool.
   mylg - utility which combines the functions of the different network probes in one diagnostic tool.
   netcat - utility which reads and writes data across network connections, using the TCP/IP protocol.
   socat - utility which transfers data between two objects.
   tcpdump - is a powerful command-line packet analyzer.
   tshark - is a tool that allows us to dump and analyze network traffic (wireshark cli).
   Termshark - is a simple terminal user-interface for tshark.
   ngrep - is like GNU grep applied to the network layer.
   netsniff-ng - is a Swiss army knife for your daily Linux network plumbing if you will.
   sockdump - dump unix domain socket traffic.
   stenographer - is a packet capture solution which aims to quickly spool all packets to disk.
   tcpterm - visualize packets in TUI.
   bmon - is a monitoring and debugging tool to capture networking related statistics and prepare them visually.
   iptraf-ng - is a console-based network monitoring program for Linux that displays information about IP traffic.
   vnstat - is a network traffic monitor for Linux and BSD.
   iPerf3 - is a tool for active measurements of the maximum achievable bandwidth on IP networks.
   ethr - is a Network Performance Measurement Tool for TCP, UDP & HTTP.
   Etherate - is a Linux CLI based Ethernet and MPLS traffic testing tool.
   echoip - is a IP address lookup service.
   Nemesis - packet manipulation CLI tool; craft and inject packets of several protocols.
   packetfu - a mid-level packet manipulation library for Ruby.
   Scapy - packet manipulation library; forge, send, decode, capture packets of a wide number of protocols.
   impacket - is a collection of Python classes for working with network protocols.
   ssh-audit - is a tool for SSH server auditing.
   aria2 - is a lightweight multi-protocol & multi-source command-line download utility.
   iptables-tracer - observe the path of packets through the iptables chains.
   inception - a highly configurable tool to check for whatever you like against any number of hosts.
   mRemoteNG - a fork of mRemote, multi-tabbed PuTTy on steroids!

:black_small_square: Network (DNS)

   dnsdiag - is a DNS diagnostics and performance measurement tools.
   fierce - is a DNS reconnaissance tool for locating non-contiguous IP space.
   subfinder - is a subdomain discovery tool that discovers valid subdomains for websites.
   sublist3r - is a fast subdomains enumeration tool for penetration testers.
   amass - is tool that obtains subdomain names by scraping data sources, crawling web archives, and more.
   namebench - provides personalized DNS server recommendations based on your browsing history.
   massdns - is a high-performance DNS stub resolver for bulk lookups and reconnaissance.
   knock - is a tool to enumerate subdomains on a target domain through a wordlist.
   dnsperf - DNS performance testing tools.
   dnscrypt-proxy 2 - a flexible DNS proxy, with support for encrypted DNS protocols.
   dnsdbq - API client providing access to passive DNS database systems.
   grimd - fast dns proxy, built to black-hole internet advertisements and malware servers.
   dnstwist - detect typosquatters, phishing attacks, fraud, and brand impersonation.

:black_small_square: Network (HTTP)

   curl - is a command line tool and library for transferring data with URLs.
   kurly - is an alternative to the widely popular curl program, written in Golang.
   HTTPie - is an user-friendly HTTP client.
   wuzz - is an interactive cli tool for HTTP inspection.
   h2spec - is a conformance testing tool for HTTP/2 implementation.
   h2t - is a simple tool to help sysadmins to hardening their websites.
   htrace.sh - is a simple Swiss Army knife for http/https troubleshooting and profiling.
   httpstat - is a tool that visualizes curl statistics in a way of beauty and clarity.
   httplab - is an interactive web server.
   Lynx - is a text browser for the World Wide Web.
   Browsh - is a fully interactive, real-time, and modern text-based browser.
   HeadlessBrowsers - a list of (almost) all headless web browsers in existence.
   ab - is a single-threaded command line tool for measuring the performance of HTTP web servers.
   siege - is an http load testing and benchmarking utility.
   wrk - is a modern HTTP benchmarking tool capable of generating significant load.
   wrk2 - is a constant throughput, correct latency recording variant of wrk.
   vegeta - is a constant throughput, correct latency recording variant of wrk.
   bombardier - is a fast cross-platform HTTP benchmarking tool written in Go.
   gobench - http/https load testing and benchmarking tool.
   hey - HTTP load generator, ApacheBench (ab) replacement, formerly known as rakyll/boom.
   boom - is a script you can use to quickly smoke-test your web app deployment.
   SlowHTTPTest - is a tool that simulates some Application Layer Denial of Service attacks by prolonging HTTP.
   gobuster - is a free and open source directory/file & DNS busting tool written in Go.
   ssllabs-scan - command-line reference-implementation client for SSL Labs APIs.
   http-observatory - Mozilla HTTP Observatory cli version.
   Hurl - is a command line tool to run and test HTTP requests with plain text.

:black_small_square: SSL

   openssl - is a robust, commercial-grade, and full-featured toolkit for the TLS and SSL protocols.
   gnutls-cli - client program to set up a TLS connection to some other computer.
   sslyze - fast and powerful SSL/TLS server scanning library.
   sslscan - tests SSL/TLS enabled services to discover supported cipher suites.
   testssl.sh - testing TLS/SSL encryption anywhere on any port.
   cipherscan - a very simple way to find out which SSL ciphersuites are supported by a target.
   spiped - is a utility for creating symmetrically encrypted and authenticated pipes between socket addresses.
   Certbot - is EFF's tool to obtain certs from Let's Encrypt and (optionally) auto-enable HTTPS on your server.
   mkcert - simple zero-config tool to make locally trusted development certificates with any names you'd like.
   certstrap - tools to bootstrap CAs, certificate requests, and signed certificates.
   Sublert - is a security and reconnaissance tool to automatically monitor new subdomains.
   mkchain - open source tool to help you build a valid SSL certificate chain.
   ssl-cert-check - SSL Certification Expiration Checker.

:black_small_square: Security

   SELinux - provides a flexible Mandatory Access Control (MAC) system built into the Linux kernel.
   AppArmor - proactively protects the operating system and applications from external or internal threats.
   grapheneX - Automated System Hardening Framework.
   DevSec Hardening Framework - Security + DevOps: Automatic Server Hardening.

:black_small_square: Auditing Tools

   ossec - actively monitoring all aspects of system activity with file integrity monitoring.
   auditd - provides a way to track security-relevant information on your system.
   Tiger - is a security tool that can be use both as a security audit and intrusion detection system.
   Lynis - battle-tested security tool for systems running Linux, macOS, or Unix-based operating system.
   LinEnum - scripted Local Linux Enumeration & Privilege Escalation Checks.
   Rkhunter - scanner tool for Linux systems that scans backdoors, rootkits and local exploits on your systems.
   PE-sieve - is a light-weight tool that helps to detect malware running on the system.
   PEASS - privilege escalation tools for Windows and Linux/Unix and MacOS.

:black_small_square: System Diagnostics/Debuggers

   strace - diagnostic, debugging and instructional userspace utility for Linux.
   DTrace - is a performance analysis and troubleshooting tool.
   ltrace - is a library call tracer, used to trace calls made by programs to library functions.
   ptrace-burrito - is a friendly wrapper around ptrace.
   perf-tools - performance analysis tools based on Linux perf_events (aka perf) and ftrace.
   bpftrace - high-level tracing language for Linux eBPF.
   sysdig - system exploration and troubleshooting tool with first class support for containers.
   Valgrind - is an instrumentation framework for building dynamic analysis tools.
   gperftools - high-performance multi-threaded malloc() implementation, plus some performance analysis tools.
   glances - cross-platform system monitoring tool written in Python.
   htop - interactive text-mode process viewer for Unix systems. It aims to be a better 'top'.
   bashtop - Linux resource monitor written in pure Bash.
   nmon - a single executable for performance monitoring and data analysis.
   atop - ASCII performance monitor. Includes statistics for CPU, memory, disk, swap, network, and processes.
   lsof - displays in its output information about files that are opened by processes.
   FlameGraph - stack trace visualizer.
   lsofgraph - convert Unix lsof output to a graph showing FIFO and UNIX interprocess communication.
   rr - is a lightweight tool for recording, replaying and debugging execution of applications.
   Performance Co-Pilot - a system performance analysis toolkit.
   hexyl - a command-line hex viewer.
   Austin - Python frame stack sampler for CPython.

:black_small_square: Log Analyzers

   angle-grinder - slice and dice log files on the command line.
   lnav - log file navigator with search and automatic refresh.
   GoAccess - real-time web log analyzer and interactive viewer that runs in a terminal.
   ngxtop - real-time metrics for nginx server.

:black_small_square: Databases

   usql - universal command-line interface for SQL databases.
   pgcli - postgres CLI with autocompletion and syntax highlighting.
   mycli - terminal client for MySQL with autocompletion and syntax highlighting.
   litecli - SQLite CLI with autocompletion and syntax highlighting.
   mssql-cli - SQL Server CLI with autocompletion and syntax highlighting.
   OSQuery - is a SQL powered operating system instrumentation, monitoring, and analytics framework.
   pgsync - sync data from one Postgres database to another.
   iredis - a terminal client for redis with autocompletion and syntax highlighting.
   SchemaCrawler - generates an E-R diagram of your database.

:black_small_square: TOR

Nipe - script to make Tor Network your default gateway.
multitor - a tool that lets you create multiple TOR instances with a load-balancing.

:black_small_square: Messengers/IRC Clients

Irssi - is a free open source terminal based IRC client.
WeeChat - is an extremely extensible and lightweight IRC client.

:black_small_square: Productivity

taskwarrior - task management system, todo list

:black_small_square: Other

   sysadmin-util - tools for Linux/Unix sysadmins.
   incron - is an inode-based filesystem notification technology.
   lsyncd - synchronizes local directories with remote targets (Live Syncing Daemon).
   GRV - is a terminal based interface for viewing Git repositories.
   Tig - text-mode interface for Git.
   tldr - simplified and community-driven man pages.
   archiver - easily create and extract .zip, .tar, .tar.gz, .tar.bz2, .tar.xz, .tar.lz4, .tar.sz, and .rar.
   commander.js - minimal CLI creator in JavaScript.
   gron - make JSON greppable!
   bed - binary editor written in Go.

GUI Tools ^[TOC]

:black_small_square: Terminal emulators

   Guake - is a dropdown terminal made for the GNOME desktop environment.
   Terminator - is based on GNOME Terminal, useful features for sysadmins and other users.
   Kitty - is a GPU based terminal emulator that supports smooth scrolling and images.
   Alacritty - is a fast, cross-platform, OpenGL terminal emulator.

:black_small_square: Network

   Wireshark - is the world’s foremost and widely-used network protocol analyzer.
   Ettercap - is a comprehensive network monitor tool.
   EtherApe - is a graphical network monitoring solution.
   Packet Sender - is a networking utility for packet generation and built-in UDP/TCP/SSL client and servers.
   Ostinato - is a packet crafter and traffic generator.
   JMeter™ - open source software to load test functional behavior and measure performance.
   locust - scalable user load testing tool written in Python.

:black_small_square: Browsers

TOR Browser - protect your privacy and defend yourself against network surveillance and traffic analysis.

:black_small_square: Password Managers

   KeePassXC - store your passwords safely and auto-type them into your everyday websites and apps.
   Bitwarden - open source password manager with built-in sync.
   Vaultwarden - unofficial Bitwarden compatible server written in Rust.

:black_small_square: Messengers/IRC Clients

HexChat - is an IRC client based on XChat.
Pidgin - is an easy to use and free chat client used by millions.

:black_small_square: Messengers (end-to-end encryption)

   Signal - is an encrypted communications app.
   Wire - secure messaging, file sharing, voice calls and video conferences.
   TorChat - decentralized anonymous instant messenger on top of Tor Hidden Services.
   Matrix - an open network for secure, decentralized, real-time communication.

:black_small_square: Text editors

   Sublime Text - is a lightweight, cross-platform code editor known for its speed, ease of use.
   Visual Studio Code - an open-source and free source code editor developed by Microsoft.
   Atom - a hackable text editor for the 21st Century.

Web Tools ^[TOC]

:black_small_square: Browsers

   SSL/TLS Capabilities of Your Browser - test your browser's SSL implementation.
   Can I use - provides up-to-date browser support tables for support of front-end web technologies.
   Panopticlick 3.0 - is your browser safe against tracking?
   Privacy Analyzer - see what data is exposed from your browser.
   Web Browser Security - it's all about Web Browser fingerprinting.
   How's My SSL? - help a web server developer learn what real world TLS clients were capable of.
   sslClientInfo - client test (incl TLSv1.3 information).

:black_small_square: SSL/Security

   SSLLabs Server Test - performs a deep analysis of the configuration of any SSL web server.
   SSLLabs Server Test (DEV) - performs a deep analysis of the configuration of any SSL web server.
   ImmuniWeb® SSLScan - test SSL/TLS (PCI DSS, HIPAA and NIST).
   SSL Check - scan your website for non-secure content.
   SSL Scanner - analyze website security.
   CryptCheck - test your TLS server configuration (e.g. ciphers).
   urlscan.io - service to scan and analyse websites.
   Report URI - monitoring security policies like CSP and HPKP.
   CSP Evaluator - allows developers and security experts to check if a Content Security Policy.
   Useless CSP - public list about CSP in some big players (might make them care a bit more).
   Why No HTTPS? - top 100 websites by Alexa rank not automatically redirecting insecure requests.
   TLS Cipher Suite Search- cipher suite search engine.
   cipherli.st - strong ciphers for Apache, Nginx, Lighttpd, and more.*
   dhtool - public Diffie-Hellman parameter service/tool.
   badssl.com - memorable site for testing clients against bad SSL configs.
   tlsfun.de - registered for various tests regarding the TLS/SSL protocol.
   CAA Record Helper - generate a CAA policy.
   Common CA Database - repository of information about CAs, and their root and intermediate certificates.
   CERTSTREAM - real-time certificate transparency log update stream.
   crt.sh - discovers certificates by continually monitoring all of the publicly known CT.
   Hardenize - deploy the security standards.
   Cipher suite compatibility - test TLS cipher suite compatibility.
   urlvoid - this service helps you detect potentially malicious websites.
   security.txt - a proposed standard (generator) which allows websites to define security policies.
   ssl-config-generator - help you follow the Mozilla Server Side TLS configuration guidelines.
   TLScan - pure python, SSL/TLS protocol and cipher scanner/enumerator.

:black_small_square: HTTP Headers & Web Linters

   Security Headers - analyse the HTTP response headers (with rating system to the results).
   Observatory by Mozilla - set of tools to analyze your website.
   webhint - is a linting tool that will help you with your site's accessibility, speed, security, and more.

:black_small_square: DNS

   ViewDNS - one source for free DNS related tools and information.
   DNSLookup - is an advanced DNS lookup tool.
   DNSlytics - online DNS investigation tool.
   DNS Spy - monitor, validate and verify your DNS configurations.
   Zonemaster - helps you to control how your DNS works.
   Leaf DNS - comprehensive DNS tester.
   Find subdomains online - find subdomains for security assessment penetration test.
   DNSdumpster - dns recon & research, find & lookup dns records.
   DNS Table online - search for DNS records by domain, IP, CIDR, ISP.
   intoDNS - DNS and mail server health checker.
   DNS Bajaj - check the delegation of your domain.
   BuddyDNS Delegation LAB - check, trace and visualize delegation of your domain.
   dnssec-debugger - DS or DNSKEY records validator.
   PTRarchive.com - this site is responsible for the safekeeping of historical reverse DNS records.
   xip.io - wildcard DNS for everyone.
   nip.io - dead simple wildcard DNS for any IP Address.
   dnslookup (ceipam) - one of the best DNS propagation checker (and not only).
   What's My DNS - DNS propagation checking tool.
   DNSGrep - quickly searching large DNS datasets.

:black_small_square: Mail

   smtp-tls-checker - check an email domain for SMTP TLS support.
   MX Toolbox - all of your MX record, DNS, blacklist and SMTP diagnostics in one integrated tool.
   Secure Email - complete email test tools for email technicians.
   blacklistalert - checks to see if your domain is on a Real Time Spam Blacklist.
   MultiRBL - complete IP check for sending Mailservers.
   DKIM SPF & Spam Assassin Validator - checks mail authentication and scores messages with Spam Assassin.

:black_small_square: Encoders/Decoders and Regex testing

   URL Encode/Decode - tool from above to either encode or decode a string of text.
   Uncoder - the online translator for search queries on log data.
   Regex101 - online regex tester and debugger: PHP, PCRE, Python, Golang and JavaScript.
   RegExr - online tool to learn, build, & test Regular Expressions (RegEx / RegExp).
   RegEx Testing - online regex testing tool.
   RegEx Pal - online regex testing tool + other tools.
   The Cyber Swiss Army Knife - a web app for encryption, encoding, compression and data analysis.

:black_small_square: Net-tools

   Netcraft - detailed report about the site, helping you to make informed choices about their integrity.*
   RIPE NCC Atlas - a global, open, distributed Internet measurement platform.
   Robtex - uses various sources to gather public information about IP numbers, domain names, host names, etc.
   Security Trails - APIs for Security Companies, Researchers and Teams.
   Online Curl - curl test, analyze HTTP Response Headers.
   Online Tools for Developers - HTTP API tools, testers, encoders, converters, formatters, and other tools.
   Ping.eu - online Ping, Traceroute, DNS lookup, WHOIS and others.
   Network-Tools - network tools for webmasters, IT technicians & geeks.
   BGPview - search for any ASN, IP, Prefix or Resource name.
   Is BGP safe yet? - check BGP (RPKI) security of ISPs and other major Internet players.
   Riseup - provides online communication tools for people and groups working on liberatory social change.
   VirusTotal - analyze suspicious files and URLs to detect types of malware.

:black_small_square: Privacy

privacyguides.org - provides knowledge and tools to protect your privacy against global mass surveillance.
DNS Privacy Test Servers - DNS privacy recursive servers list (with a 'no logging' policy).

:black_small_square: Code parsers/playgrounds

   ShellCheck - finds bugs in your shell scripts.
   explainshell - get interactive help texts for shell commands.
   jsbin - live pastebin for HTML, CSS & JavaScript, and more.
   CodeSandbox - online code editor for web application development.
   PHP Sandbox - test your PHP code with this code tester.
   Repl.it - an instant IDE to learn, build, collaborate, and host all in one place.
   vclFiddle - is an online tool for experimenting with the Varnish Cache VCL.
   Haskell Dockerfile Linter - a smarter Dockerfile linter that helps you build best practice Docker images.

:black_small_square: Performance

   GTmetrix - analyze your site’s speed and make it faster.
   Sucuri loadtimetester - test here the performance of any of your sites from across the globe.
   Pingdom Tools - analyze your site’s speed around the world.
   PingMe.io - run website latency tests across multiple geographic regions.
   PageSpeed Insights - analyze your site’s speed and make it faster.
   web.dev - helps developers like you learn and apply the web's modern capabilities to your own sites and apps.
   Lighthouse - automated auditing, performance metrics, and best practices for the web.

:black_small_square: Mass scanners (search engines)

   Censys - platform that helps information security practitioners discover, monitor, and analyze devices.
   Shodan - the world's first search engine for Internet-connected devices.
   Shodan 2000 - this tool looks for randomly generated data from Shodan.
   GreyNoise - mass scanner such as Shodan and Censys.
   ZoomEye - search engine for cyberspace that lets the user find specific network components.
   netograph - tools to monitor and understand deep structure of the web.
   FOFA - is a cyberspace search engine.
   onyphe - is a search engine for open-source and cyber threat intelligence data collected.
   IntelligenceX - is a search engine and data archive.
   binaryedge - it scan the entire internet space and create real-time threat intelligence streams and reports.
   Spyse - Internet assets registry: networks, threats, web objects, etc.
   wigle - is a submission-based catalog of wireless networks. All the networks. Found by Everyone.
   PublicWWW - find any alphanumeric snippet, signature or keyword in the web pages HTML, JS and CSS code.
   IntelTechniques - this repository contains hundreds of online search utilities.
   hunter - lets you find email addresses in seconds and connect with the people that matter for your business.
   GhostProject? - search by full email address or username.
   databreaches - was my email affected by data breach?
   We Leak Info - world's fastest and largest data breach search engine.
   Pulsedive - scans of malicious URLs, IPs, and domains, including port scans and web requests.
   Buckets by Grayhatwarfar - database with public search for Open Amazon S3 Buckets and their contents.
   Vigilante.pw - the breached database directory.
   builtwith - find out what websites are built with.
   NerdyData - search the web's source code for technologies, across millions of sites.
   zorexeye - search for sites, images, apps, softwares & more.
   Mamont's open FTP Index - if a target has an open FTP site with accessible content it will be listed here.
   OSINT Framework - focused on gathering information from free tools or resources.
   maltiverse - is a service oriented to cybersecurity analysts.
   Leaked Source - is a collaboration of data found online in the form of a lookup.
   We Leak Info - to help everyday individuals secure their online life, avoiding getting hacked.
   pipl - is the place to find the person behind the email address, social username or phone number.
   abuse.ch - is operated by a random swiss guy fighting malware for non-profit.
   malc0de - malware search engine.
   Cybercrime Tracker - monitors and tracks various malware families that are used to perpetrate cyber crimes.
   shhgit - find GitHub secrets in real time.
   searchcode - helping you find real world examples of functions, API's and libraries.
   Insecam - the world biggest directory of online surveillance security cameras.
   index-of - contains great stuff like: security, hacking, reverse engineering, cryptography, programming etc.
   Rapid7 Labs Open Data - is a great resources of datasets from Project Sonar.
   Common Response Headers - the largest database of HTTP response headers.
   InQuest Labs - InQuest Labs is an open, interactive, and API driven data portal for security researchers.

:black_small_square: Generators

   thispersondoesnotexist - generate fake faces in one click - endless possibilities.
   AI Generated Photos - 100.000 AI generated faces.
   fakenamegenerator - your randomly generated identity.
   Intigriti Redirector - open redirect/SSRF payload generator.

:black_small_square: Passwords

   have i been pwned? - check if you have an account that has been compromised in a data breach.
   dehashed - is a hacked database search engine.
   Leaked Source - is a collaboration of data found online in the form of a lookup.

:black_small_square: CVE/Exploits databases

   CVE Mitre - list of publicly known cybersecurity vulnerabilities.
   CVE Details - CVE security vulnerability advanced database.
   Exploit DB - CVE compliant archive of public exploits and corresponding vulnerable software.
   0day.today - exploits market provides you the possibility to buy/sell zero-day exploits.
   sploitus - the exploit and tools database.
   cxsecurity - free vulnerability database.
   Vulncode-DB - is a database for vulnerabilities and their corresponding source code if available.
   cveapi - free API for CVE data.

:black_small_square: Mobile apps scanners

   ImmuniWeb® Mobile App Scanner - test security and privacy of mobile apps (iOS & Android).
   Quixxi - free Mobile App Vulnerability Scanner for Android & iOS.
   Ostorlab - analyzes mobile application to identify vulnerabilities and potential weaknesses.

:black_small_square: Private Search Engines

   Startpage - the world's most private search engine.
   searX - a privacy-respecting, hackable metasearch engine.
   darksearch - the 1st real Dark Web search engine.
   Qwant - the search engine that respects your privacy.
   DuckDuckGo - the search engine that doesn't track you.
   Swisscows - privacy safe web search
   Disconnect - the search engine that anonymizes your searches.
   MetaGer - the search engine that uses anonymous proxy and hidden Tor branches.

:black_small_square: Secure Webmail Providers

   CounterMail - online email service, designed to provide maximum security and privacy.
   Mail2Tor - is a Tor Hidden Service that allows anyone to send and receive emails anonymously.
   Tutanota - is the world's most secure email service and amazingly easy to use.
   Protonmail - is the world's largest secure email service, developed by CERN and MIT scientists.
   Startmail - private & encrypted email made easy.

:black_small_square: Crypto

Keybase - it's open source and powered by public-key cryptography.

:black_small_square: PGP Keyservers

SKS OpenPGP Key server - services for the SKS keyservers used by OpenPGP.

Systems/Services ^[TOC]

:black_small_square: Operating Systems

   Slackware - the most "Unix-like" Linux distribution.
   OpenBSD - multi-platform 4.4BSD-based UNIX-like operating system.
   HardenedBSD - HardenedBSD aims to implement innovative exploit mitigation and security solutions.
   Kali Linux - Linux distribution used for Penetration Testing, Ethical Hacking and network security assessments.
   Parrot Security OS - cyber security GNU/Linux environment.
   Backbox Linux - penetration test and security assessment oriented Ubuntu-based Linux distribution.
   BlackArch - is an Arch Linux-based penetration testing distribution for penetration testers.
   Pentoo - is a security-focused livecd based on Gentoo.
   Security Onion - Linux distro for intrusion detection, enterprise security monitoring, and log management.
   Tails - is a live system that aims to preserve your privacy and anonymity.
   vedetta - OpenBSD router boilerplate.
   Qubes OS - is a security-oriented OS that uses Xen-based virtualization.

:black_small_square: HTTP(s) Services

   Varnish Cache - HTTP accelerator designed for content-heavy dynamic web sites.
   Nginx - open source web and reverse proxy server that is similar to Apache, but very light weight.
   OpenResty - is a dynamic web platform based on NGINX and LuaJIT.
   Tengine - a distribution of Nginx with some advanced features.
   Caddy Server - is an open source, HTTP/2-enabled web server with HTTPS by default.
   HAProxy - the reliable, high performance TCP/HTTP load balancer.

:black_small_square: DNS Services

   Unbound - validating, recursive, and caching DNS resolver (with TLS).
   Knot Resolver - caching full resolver implementation, including both a resolver library and a daemon.
   PowerDNS - is an open source authoritative DNS server, written in C++ and licensed under the GPL.

:black_small_square: Other Services

3proxy - tiny free proxy server.

:black_small_square: Security/hardening

   Emerald Onion - is a 501(c)(3) nonprofit organization and transit internet service provider (ISP).
   pi-hole - the Pi-hole® is a DNS sinkhole that protects your devices from unwanted content.
   maltrail - malicious traffic detection system.
   security_monkey - monitors AWS, GCP, OpenStack, and GitHub orgs for assets and their changes over time.
   firecracker - secure and fast microVMs for serverless computing.
   streisand - sets up a new server running your choice of WireGuard, OpenSSH, OpenVPN, and more.

Networks ^[TOC]

:black_small_square: Tools

CapAnalysis - web visual tool to analyze large amounts of captured network traffic (PCAP analyzer).
netbox - IP address management (IPAM) and data center infrastructure management (DCIM) tool.

:black_small_square: Labs

NRE Labs - learn automation by doing it. Right now, right here, in your browser.

:black_small_square: Other

LBNL's Network Research Group - home page of the Network Research Group (NRG).

Containers/Orchestration ^[TOC]

:black_small_square: CLI Tools

gvisor - container runtime sandbox.
ctop - top-like interface for container metrics.

:black_small_square: Web Tools

   Moby - a collaborative project for the container ecosystem to assemble container-based system.
   Traefik - open source reverse proxy/load balancer provides easier integration with Docker and Let's encrypt.
   kong - The Cloud-Native API Gateway.
   rancher - complete container management platform.
   portainer - making Docker management easy.
   nginx-proxy - automated nginx proxy for Docker containers using docker-gen.
   bunkerized-nginx - nginx docker image "secure by default".

:black_small_square: Security

   docker-bench-security - checks for dozens of common best-practices around deploying Docker.
   trivy - vulnerability scanner for containers, suitable for CI.
   Harbor - cloud native registry project that stores, signs, and scans content.
   Houdini - hundreds of offensive and useful docker images for network intrusion.

:black_small_square: Manuals/Tutorials/Best Practices

   docker-cheat-sheet - a quick reference cheat sheet on Docker.
   awesome-docker - a curated list of Docker resources and projects.
   docker_practice - learn and understand Docker technologies, with real DevOps practice!
   labs - is a collection of tutorials for learning how to use Docker with various tools.
   dockerfiles - various Dockerfiles I use on the desktop and on servers.
   kubernetes-the-hard-way - bootstrap Kubernetes the hard way on Google Cloud Platform. No scripts.
   kubernetes-the-easy-way - bootstrap Kubernetes the easy way on Google Cloud Platform. No scripts.
   cheatsheet-kubernetes-A4 - Kubernetes CheatSheets in A4.
   k8s-security - kubernetes security notes and best practices.
   kubernetes-production-best-practices - checklists with best-practices for production-ready Kubernetes.
   kubernetes-production-best-practices - kubernetes security - best practice guide.
   kubernetes-failure-stories - is a compilation of public failure/horror stories related to Kubernetes.

Manuals/Howtos/Tutorials ^[TOC]

:black_small_square: Shell/Command line

   pure-bash-bible - is a collection of pure bash alternatives to external processes.
   pure-sh-bible - is a collection of pure POSIX sh alternatives to external processes.
   bash-guide - is a guide to learn bash.
   bash-handbook - for those who wanna learn Bash.
   The Bash Hackers Wiki - hold documentation of any kind about GNU Bash.
   Shell & Utilities - describes the commands offered to application programs by POSIX-conformant systems.
   the-art-of-command-line - master the command line, in one page.
   Shell Style Guide - a shell style guide for Google-originated open-source projects.

:black_small_square: Text Editors

Vim Cheat Sheet - great multi language vim guide.

:black_small_square: Python

   Awesome Python - a curated list of awesome Python frameworks, libraries, software and resources.
   python-cheatsheet - comprehensive Python cheatsheet.
   pythoncheatsheet.org - basic reference for beginner and advanced developers.

:black_small_square: Sed & Awk & Other

F’Awk Yeah! - advanced sed and awk usage (Parsing for Pentesters 3).

:black_small_square: *nix & Network

   nixCraft - linux and unix tutorials for new and seasoned sysadmin.
   TecMint - the ideal Linux blog for Sysadmins & Geeks.
   Omnisecu - free Networking, System Administration and Security tutorials.
   linux-cheat - Linux tutorials and cheatsheets. Minimal examples. Mostly user-land CLI utilities.
   linuxupskillchallenge - learn the skills required to sysadmin.
   Unix Toolbox - Unix/Linux/BSD commands and tasks which are useful for IT work or for advanced users.
   Linux Kernel Teaching - is a collection of lectures and labs Linux kernel topics.
   htop explained - explanation of everything you can see in htop/top on Linux.
   Linux Guide and Hints - tutorials on system administration in Fedora and CentOS.
   strace-little-book - a little book which introduces strace.
   linux-tracing-workshop - examples and hands-on labs for Linux tracing tools workshops.
   http2-explained - a detailed document explaining and documenting HTTP/2.
   http3-explained - a document describing the HTTP/3 and QUIC protocols.
   HTTP/2 in Action - an excellent introduction to the new HTTP/2 standard.
   Let's code a TCP/IP stack - great stuff to learn network and system programming at a deeper level.
   Nginx Admin's Handbook - how to improve NGINX performance, security and other important things.
   nginxconfig.io - NGINX config generator on steroids.
   openssh guideline - is to help operational teams with the configuration of OpenSSH server and client.
   SSH Handshake Explained - is a relatively brief description of the SSH handshake.
   ISC's Knowledgebase - you'll find some general information about BIND 9, ISC DHCP, and Kea DHCP.
   PacketLife.net - a place to record notes while studying for Cisco's CCNP certification.

:black_small_square: Microsoft

AD-Attack-Defense - attack and defend active directory using modern post exploitation activity.

:black_small_square: Large-scale systems

   The System Design Primer - learn how to design large-scale systems.
   Awesome Scalability - best practices in building High Scalability, High Availability, High Stability, and more.
   Web Architecture 101 - the basic architecture concepts.

:black_small_square: System hardening

   CIS Benchmarks - secure configuration settings for over 100 technologies, available as a free PDF.
   Security Harden CentOS 7 - this walks you through the steps required to security harden CentOS.
   CentOS 7 Server Hardening Guide - great guide for hardening CentOS; familiar with OpenSCAP.
   awesome-security-hardening - is a collection of security hardening guides, tools and other resources.
   The Practical Linux Hardening Guide - provides a high-level overview of hardening GNU/Linux systems.
   Linux Hardening Guide - how to harden Linux as much as possible for security and privacy.

:black_small_square: Security & Privacy

   Hacking Articles - LRaj Chandel's Security & Hacking Blog.
   AWS security tools - make your AWS cloud environment more secure.
   Rawsec's CyberSecurity Inventory - an inventory of tools and resources about CyberSecurity.
   The Illustrated TLS Connection - every byte of a TLS connection explained and reproduced.
   SSL Research - SSL and TLS Deployment Best Practices by SSL Labs.
   SELinux Game - learn SELinux by doing. Solve Puzzles, show skillz.
   Certificates and PKI - everything you should know about certificates and PKI but are too afraid to ask.
   The Art of Subdomain Enumeration - a reference for subdomain enumeration techniques.
   Quitting Google - the comprehensive guide to quitting Google.

:black_small_square: Web Apps

   OWASP - worldwide not-for-profit charitable organization focused on improving the security of software.
   OWASP ASVS 3.0.1 - OWASP Application Security Verification Standard Project.
   OWASP ASVS 3.0.1 Web App - simple web app that helps developers understand the ASVS requirements.
   OWASP ASVS 4.0 - is a list of application security requirements or tests.
   OWASP Testing Guide v4 - includes a "best practice" penetration testing framework.
   OWASP Dev Guide - this is the development version of the OWASP Developer Guide.
   OWASP WSTG - is a comprehensive open source guide to testing the security of web apps.
   OWASP API Security Project - focuses specifically on the top ten vulnerabilities in API security.
   Mozilla Web Security - help operational teams with creating secure web applications.
   security-bulletins - security bulletins that relate to Netflix Open Source.
   API-Security-Checklist - security countermeasures when designing, testing, and releasing your API.
   Enable CORS - enable cross-origin resource sharing.
   Application Security Wiki - is an initiative to provide all application security related resources at one place.
   Weird Proxies - reverse proxy related attacks; it is a result of analysis of various proxies.
   Webshells - great series about malicious payloads.
   Practical Web Cache Poisoning - show you how to compromise websites by using esoteric web features.
   Hidden directories and files - as a source of sensitive information about web application.
   Explosive blog - great blog about cybersec and pentests.
   Security Cookies - this paper will take a close look at cookie security.
   APISecurityBestPractices - help you keep secrets (API keys, db credentials, certificates) out of source code.

:black_small_square: All-in-one

   LZone Cheat Sheets - all cheat sheets.
   Dan’s Cheat Sheets’s - massive cheat sheets documentation.
   Rico's cheatsheets - this is a modest collection of cheatsheets.
   DevDocs API - combines multiple API documentations in a fast, organized, and searchable interface.
   cheat.sh - the only cheat sheet you need.
   gnulinux.guru - collection of cheat sheets about bash, vim and networking.
   Web Skills - visual overview of useful skills to learn as a web developer.

:black_small_square: Ebooks

free-programming-books - list of free learning resources in many languages.

:black_small_square: Other

   CTF Series : Vulnerable Machines - the steps below could be followed to find vulnerabilities and exploits.
   50M_CTF_Writeup - $50 million CTF from Hackerone - writeup.
   ctf-tasks - an archive of low-level CTF challenges developed over the years.
   How to start RE/malware analysis? - collection of some hints and useful links for the beginners.
   The C10K problem - it's time for web servers to handle ten thousand clients simultaneously, don't you think?
   How 1500 bytes became the MTU of the internet - great story about the Maximum Transmission Unit.
   poor man's profiler - like dtrace's don't really provide methods to see what programs are blocking on.
   HTTPS on Stack Overflow - this is the story of a long journey regarding the implementation of SSL.
   Julia's Drawings - some drawings about programming and unix world, zines about systems & debugging tools.
   Hash collisions - this great repository is focused on hash collisions exploitation.
   sha256-animation - animation of the SHA-256 hash function in your terminal.
   sha256algorithm - sha256 algorithm explained online step by step visually.
   BGP Meets Cat - after 3072 hours of manipulating BGP, Job Snijders has succeeded in drawing a Nyancat.
   bgp-battleships - playing battleships over BGP.
   What happens when... - you type google.com into your browser and press enter?
   how-web-works - based on the 'What happens when...' repository.
   HTTPS in the real world - great tutorial explain how HTTPS works in the real world.
   Gitlab and NFS bug - how we spent two weeks hunting an NFS bug in the Linux kernel.
   Gitlab melts down - postmortem on the database outage of January 31 2017 with the lessons we learned.
   How To Become A Hacker - if you want to be a hacker, keep reading.
   Operation Costs in CPU - should help to estimate costs of certain operations in CPU clocks.
   Let's Build a Simple Database - writing a sqlite clone from scratch in C.
   simple-computer - great resource to understand how computers work under the hood.
   The story of "Have I been pwned?" - working with 154 million records on Azure Table Storage.
   TOP500 Supercomputers - shows the 500 most powerful commercially available computer systems.
   How to build a 8 GPU password cracker - hours of frustration like desktop components do.
   CERN Data Centre - 3D visualizations of the CERN computing environments (and more).
   How fucked is my database - evaluate how fucked your database is with this handy website.
   Linux Troubleshooting 101 , 2016 Edition - everything is a DNS Problem...
   Five Whys - you know what the problem is, but you cannot solve it?
   Maersk, me & notPetya - how did ransomware successfully hijack hundreds of domain controllers?
   howhttps.works - how HTTPS works ...in a comic!
   howdns.works - a fun and colorful explanation of how DNS works.
   POSTGRESQLCO.NF - your postgresql.conf documentation and recommendations.

Inspiring Lists ^[TOC]

:black_small_square: SysOps/DevOps

   Awesome Sysadmin - amazingly awesome open source sysadmin resources.
   Awesome Shell - awesome command-line frameworks, toolkits, guides and gizmos.
   Command-line-text-processing - finding text to search and replace, sorting to beautifying, and more.
   Awesome Pcaptools - collection of tools developed by other researchers to process network traces.
   awesome-ebpf - a curated list of awesome projects related to eBPF.
   Linux Network Performance - where some of the network sysctl variables fit into the Linux/Kernel network flow.
   Awesome Postgres - list of awesome PostgreSQL software, libraries, tools and resources.
   quick-SQL-cheatsheet - a quick reminder of all SQL queries and examples on how to use them.
   Awesome-Selfhosted - list of Free Software network services and web applications which can be hosted locally.
   List of applications - huge list of apps sorted by category, as a reference for those looking for packages.
   CS-Interview-Knowledge-Map - build the best interview map.
   DevOps-Guide - DevOps Guide from basic to advanced with Interview Questions and Notes.
   FreeBSD Journal - it is a great list of periodical magazines about FreeBSD and other important things.
   devops-interview-questions - contains interview questions on various DevOps and SRE related topics.

:black_small_square: Developers

   Web Developer Roadmap - roadmaps, articles and resources to help you choose your path, learn and improve.
   Front-End-Checklist - the perfect Front-End Checklist for modern websites and meticulous developers.
   Front-End-Performance-Checklist - Front-End Performance Checklist that runs faster than the others.
   Python's Magic Methods - what are magic methods? They're everything in object-oriented Python.
   wtfpython - a collection of surprising Python snippets and lesser-known features.
   js-dev-reads - a list of books and articles for the discerning web developer to read.
   Commit messages guide - a guide to understand the importance of commit messages.

:black_small_square: Security/Pentesting

   Awesome Web Security - a curated list of Web Security materials and resources.
   awesome-cyber-skills - a curated list of hacking environments where you can train your cyber skills.
   awesome-devsecops - an authoritative list of awesome devsecops tools.
   awesome-osint - is a curated list of amazingly awesome OSINT.
   HolyTips - tips and tutorials on Bug Bounty Hunting and Web App Security.
   awesome-threat-intelligence - a curated list of Awesome Threat Intelligence resources.
   Red-Teaming-Toolkit - a collection of open source and commercial tools that aid in red team operations.
   awesome-burp-extensions - a curated list of amazingly awesome Burp Extensions.
   Free Security eBooks - list of a Free Security and Hacking eBooks.
   Hacking-Security-Ebooks - top 100 Hacking & Security E-Books.
   privacy-respecting - curated list of privacy respecting services and software.
   reverse-engineering - list of awesome reverse engineering resources.
   linux-re-101 - a collection of resources for linux reverse engineering.
   reverseengineering-reading-list - a list of Reverse Engineering articles, books, and papers.
   Awesome-WAF - a curated list of awesome web-app firewall (WAF) stuff.
   awesome-shodan-queries - interesting, funny, and depressing search queries to plug into shodan.io.
   RobotsDisallowed - a curated list of the most common and most interesting robots.txt disallowed directories.
   HackingNeuralNetworks - is a small course on exploiting and defending neural networks.
   wildcard-certificates - why you probably shouldn't use a wildcard certificate.
   Don't use VPN services - which is what every third-party "VPN provider" does.
   awesome-yara - a curated list of awesome YARA rules, tools, and people.
   macOS-Security-and-Privacy-Guide - guide to securing and improving privacy on macOS.
   macos_security - macOS Security Compliance Project.
   awesome-sec-talks - is a collected list of awesome security talks.
   Movies for Hackers - list of movies every hacker & cyberpunk must watch.
   Cryptography_1 - materials used whilst taking Prof. Dan Boneh Stanford Crypto course.
   Crypton - library to learn and practice Offensive and Defensive Cryptography.

:black_small_square: Other

   Cheatography - over 3,000 free cheat sheets, revision aids and quick references.
   awesome-static-analysis - static analysis tools for all programming languages.
   computer-science - path to a free self-taught education in Computer Science.
   post-mortems - is a collection of postmortems (config errors, hardware failures, and more).
   build-your-own-x - build your own (insert technology here).
   Project-Based-Tutorials-in-C - is a curated list of project-based tutorials in C.
   The-Documentation-Compendium - various README templates & tips on writing high-quality documentation.
   awesome-python-applications - free software that works great, and also happens to be open-source Python.
   awesome-public-datasets - a topic-centric list of HQ open datasets.
   machine-learning-algorithms - a curated list of all machine learning algorithms and concepts.

Blogs/Podcasts/Videos ^[TOC]

:black_small_square: SysOps/DevOps

Varnish for PHP developers - very interesting presentation of Varnish by Mattias Geniar.
A Netflix Guide to Microservices - talks about the chaotic and vibrant world of microservices at Netflix.

:black_small_square: Developers

Comparing C to machine lang - compare a simple C app with the compiled machine code of that program.

:black_small_square: Geeky Persons

   Brendan Gregg's Blog - is an industry expert in computing performance and cloud computing.
   Gynvael "GynDream" Coldwind - is a IT security engineer at Google.
   Michał "lcamtuf" Zalewski - white hat hacker, computer security expert.
   Mattias Geniar - developer, sysadmin, blogger, podcaster and public speaker.
   Nick Craver - software developer and systems administrator for Stack Exchange.
   Scott Helme - security researcher, speaker and founder of securityheaders.com and report-uri.com.
   Brian Krebs - The Washington Post and now an Independent investigative journalist.
   Bruce Schneier - is an internationally renowned security technologist, called a "security guru".
   Chrissy Morgan - advocate of practical learning, Chrissy also takes part in bug bounty programs.
   Andy Gill - is a hacker at heart who works as a senior penetration tester.
   Daniel Miessler - cybersecurity expert and writer.
   Samy Kamkar - is an American privacy and security researcher, computer hacker.
   Javvad Malik - is a security advocate at AlienVault, a blogger event speaker and industry commentator.
   Graham Cluley - public speaker and independent computer security analyst.
   Kacper Szurek - detection engineer at ESET.
   Troy Hunt - web security expert known for public education and outreach on security topics.
   raymii.org - sysadmin specializing in building high availability cloud environments.
   Robert Penz - IT security expert.

:black_small_square: Geeky Blogs

   Linux Audit - the Linux security blog about auditing, hardening and compliance by Michael Boelen.
   Linux Security Expert - trainings, howtos, checklists, security tools, and more.
   The Grymoire - collection of useful incantations for wizards, be you computer wizards, magicians, or whatever.
   Secjuice - is the only non-profit, independent and volunteer led publication in the information security space.
   Decipher - security news that informs and inspires.

:black_small_square: Geeky Vendor Blogs

   Tenable Podcast - conversations and interviews related to Cyber Exposure, and more.
   Sophos - threat news room, giving you news, opinion, advice and research on computer security issues.
   Tripwire State of Security - blog featuring the latest news, trends and insights on current security issues.
   Malwarebytes Labs Blog - security blog aims to provide insider news about cybersecurity.
   TrustedSec - latest news, and trends about cybersecurity.
   PortSwigger Web Security Blog - about web app security vulns and top tips from our team of web security.
   AT&T Cybersecurity blog - news on emerging threats and practical advice to simplify threat detection.
   Thycotic - where CISOs and IT Admins come to learn about industry trends, IT security, and more.

:black_small_square: Geeky Cybersecurity Podcasts

   Risky Business - is a weekly information security podcast featuring news and in-depth interviews.
   Cyber, by Motherboard - stories, and focus on the ideas about cybersecurity.
   Tenable Podcast - conversations and interviews related to Cyber Exposure, and more.
   Cybercrime Investigations - podcast by Geoff White about cybercrimes.
   The many hats club - featuring stories from a wide range of Infosec people (Whitehat, Greyhat and Blackhat).
   Darknet Diaries - true stories from the dark side of the Internet.
   OSINTCurious Webcasts - is the investigative curiosity that helps people be successful in OSINT.
   Security Weekly - the latest information security and hacking news.

:black_small_square: Geeky Cybersecurity Video Blogs

   rev3rse security - offensive, binary exploitation, web app security, hardening, red team, blue team.
   LiveOverflow - a lot more advanced topics than what is typically offered in paid online courses - but for free.
   J4vv4D - the important information regarding our internet security.
   CyberTalks - talks, interviews, and article about cybersecurity.

:black_small_square: Best Personal Twitter Accounts

   @blackroomsec - a white-hat hacker/pentester. Intergalactic Minesweeper Champion 1990.
   @MarcoCiappelli - Co-Founder @ITSPmagazine, at the intersection of IT security and society.
   @binitamshah - Linux Evangelist. Malwares. Kernel Dev. Security Enthusiast.
   @joe_carson - an InfoSec Professional and Tech Geek.
   @mikko - CRO at F-Secure, Reverse Engineer, TED Speaker, Supervillain.
   @esrtweet - often referred to as ESR, is an American software developer, and open-source software advocate.
   @gynvael - security researcher/programmer, @DragonSectorCTF founder/player, technical streamer.
   @x0rz - Security Researcher & Cyber Observer.
   @hasherezade - programmer, malware analyst. Author of PEbear, PEsieve, libPeConv.
   @TinkerSec - tinkerer, cypherpunk, hacker.
   @alisaesage - independent hacker and researcher.
   @SwiftOnSecurity - systems security, industrial safety, sysadmin, author of decentsecurity.com.
   @dakami - is one of just seven people with the authority to restore the DNS root keys.
   @samykamkar - is a famous "grey hat" hacker, security researcher, creator of the MySpace "Samy" worm.
   @securityweekly - founder & CTO of Security Weekly podcast network.
   @jack_daniel - @SecurityBSides co-founder.
   @thegrugq - Security Researcher.
   @matthew_d_green - a cryptographer and professor at Johns Hopkins University.

:black_small_square: Best Commercial Twitter Accounts

   @haveibeenpwned - check if you have an account that has been compromised in a data breach.
   @bugcrowd - trusted by more of the Fortune 500 than any other crowdsourced security platform.
   @Malwarebytes - most trusted security company. Unmatched threat visibility.
   @sansforensics - the world's leading Digital Forensics and Incident Response provider.
   @attcyber - AT&T Cybersecurity’s Edge-to-Edge technologies provide threat intelligence, and more.
   @TheManyHatsClub - an information security focused podcast and group of individuals from all walks of life.
   @hedgehogsec - Hedgehog Cyber. Gibraltar and Manchester's top boutique information security firm.
   @NCSC - the National Cyber Security Centre. Helping to make the UK the safest place to live and work online.
   @Synacktiv - IT security experts.

:black_small_square: A piece of history

How to Do Things at ARL - how to configure modems, scan images, record CD-ROMs, and other.*

:black_small_square: Other

Diffie-Hellman Key Exchange (short version) - how Diffie-Hellman Key Exchange worked.

Hacking/Penetration Testing ^[TOC]

:black_small_square: Pentesters arsenal tools

   Sandcat Browser - a penetration-oriented browser with plenty of advanced functionality already built in.
   Metasploit - tool and framework for pentesting system, web and many more.
   Burp Suite - tool for testing web app security, intercepting proxy to replay, inject, scan and fuzz.
   OWASP Zed Attack Proxy - intercepting proxy to replay, inject, scan and fuzz HTTP requests.
   w3af - is a Web Application Attack and Audit Framework.
   mitmproxy - an interactive TLS-capable intercepting HTTP proxy for penetration testers.
   Nikto2 - web server scanner which performs comprehensive tests against web servers for multiple items.
   sqlmap - tool that automates the process of detecting and exploiting SQL injection flaws.
   Recon-ng - is a full-featured Web Reconnaissance framework written in Python.
   AutoRecon - is a network reconnaissance tool which performs automated enumeration of services.
   Faraday - an Integrated Multiuser Pentest Environment.
   Photon - incredibly fast crawler designed for OSINT.
   XSStrike - most advanced XSS detection suite.
   Sn1per - automated pentest framework for offensive security experts.
   vuls - is an agent-less vulnerability scanner for Linux, FreeBSD, and other.
   tsunami - is a general purpose network security scanner with an extensible plugin system.
   aquatone - a tool for domain flyovers.
   BillCipher - information gathering tool for a website or IP address.
   WhatWaf - detect and bypass web application firewalls and protection systems.
   Corsy - CORS misconfiguration scanner.
   Raccoon - is a high performance offensive security tool for reconnaissance and vulnerability scanning.
   dirhunt - find web directories without bruteforce.
   John The Ripper - is a fast password cracker, currently available for many flavors of Unix, Windows, and other.
   hashcat - world's fastest and most advanced password recovery utility.
   p0f - is a tool to identify the players behind any incidental TCP/IP communications.
   ssh_scan - a prototype SSH configuration and policy scanner.
   LeakLooker - find open databases - powered by Binaryedge.io
   exploitdb - searchable archive from The Exploit Database.
   getsploit - is a command line utility for searching and downloading exploits.
   ctf-tools - some setup scripts for security research tools.
   pwntools - CTF framework and exploit development library.
   security-tools - collection of small security tools created mostly in Python. CTFs, pentests and so on.
   pentestpackage - is a package of Pentest scripts.
   python-pentest-tools - python tools for penetration testers.
   fuzzdb - dictionary of attack patterns and primitives for black-box application fault injection.
   AFL - is a free software fuzzer maintained by Google.
   AFL++ - is AFL with community patches.
   syzkaller - is an unsupervised, coverage-guided kernel fuzzer.
   pwndbg - exploit development and reverse engineering with GDB made easy.
   GDB PEDA - Python Exploit Development Assistance for GDB.
   IDA - multi-processor disassembler and debugger useful for reverse engineering malware.
   radare2 - framework for reverse-engineering and analyzing binaries.
   routersploit - exploitation framework for embedded devices.
   Ghidra - is a software reverse engineering (SRE) framework.
   Cutter - is an SRE platform integrating Ghidra's decompiler.
   Vulnreport - open-source pentesting management and automation platform by Salesforce Product Security.
   Mentalist - is a graphical tool for custom wordlist generation.
   archerysec - vulnerability assessment and management helps to perform scans and manage vulnerabilities.
   Osmedeus - fully automated offensive security tool for reconnaissance and vulnerability scanning.
   beef - the browser exploitation framework project.
   AutoSploit - automated mass exploiter.
   SUDO_KILLER - is a tool to identify and exploit sudo rules' misconfigurations and vulnerabilities.
   yara - the pattern matching swiss knife.
   mimikatz - a little tool to play with Windows security.
   sherlock - hunt down social media accounts by username across social networks.
   OWASP Threat Dragon - is a tool used to create threat model diagrams and to record possible threats.

:black_small_square: Pentests bookmarks collection

   PTES - the penetration testing execution standard.
   Pentests MindMap - amazing mind map with vulnerable apps and systems.
   WebApps Security Tests MindMap - incredible mind map for WebApps security tests.
   Brute XSS - master the art of Cross Site Scripting.
   XSS cheat sheet - contains many vectors that can help you bypass WAFs and filters.
   Offensive Security Bookmarks - security bookmarks collection, all things that author need to pass OSCP.
   Awesome Pentest Cheat Sheets - collection of the cheat sheets useful for pentesting.
   Awesome Hacking by HackWithGithub - awesome lists for hackers, pentesters and security researchers.
   Awesome Hacking by carpedm20 - a curated list of awesome hacking tutorials, tools and resources.
   Awesome Hacking Resources - collection of hacking/penetration testing resources to make you better.
   Awesome Pentest - collection of awesome penetration testing resources, tools and other shiny things.
   Awesome-Hacking-Tools - is a curated list of awesome Hacking Tools.
   Hacking Cheat Sheet - author hacking and pentesting notes.
   blackhat-arsenal-tools - official Black Hat arsenal security tools repository.
   Penetration Testing and WebApp Cheat Sheets - the complete list of Infosec related cheat sheets.
   Cyber Security Resources - includes thousands of cybersecurity-related references and resources.
   Pentest Bookmarks - there are a LOT of pentesting blogs.
   Cheatsheet-God - Penetration Testing Reference Bank - OSCP/PTP & PTX Cheatsheet.
   ThreatHunter-Playbook - to aid the development of techniques and hypothesis for hunting campaigns.
   Beginner-Network-Pentesting - notes for beginner network pentesting course.
   OSCPRepo - is a list of resources that author have been gathering in preparation for the OSCP.
   PayloadsAllTheThings - a list of useful payloads and bypass for Web Application Security and Pentest/CTF.
   payloads - git all the Payloads! A collection of web attack payloads.
   command-injection-payload-list - command injection payload list.
   Awesome Shodan Search Queries - great search queries to plug into Shodan.
   AwesomeXSS - is a collection of Awesome XSS resources.
   php-webshells - common php webshells.
   Pentesting Tools Cheat Sheet - a quick reference high level overview for typical penetration testing.
   OWASP Cheat Sheet Series - is a collection of high value information on specific application security topics.
   OWASP dependency-check - is an open source solution the OWASP Top 10 2013 entry.
   OWASP ProActive Controls - OWASP Top 10 Proactive Controls 2018.
   PENTESTING-BIBLE - hacking & penetration testing & red team & cyber security resources.
   pentest-wiki - is a free online security knowledge library for pentesters/researchers.
   DEF CON Media Server - great stuff from DEFCON.
   Awesome Malware Analysis - a curated list of awesome malware analysis tools and resources.
   SQL Injection Cheat Sheet - detailed technical stuff about the many different variants of the SQL Injection.
   Entersoft Knowledge Base - great and detailed reference about vulnerabilities.
   HTML5 Security Cheatsheet - a collection of HTML5 related XSS attack vectors.
   XSS String Encoder - for generating XSS code to check your input validation filters against XSS.
   GTFOBins - list of Unix binaries that can be exploited by an attacker to bypass local security restrictions.
   Guifre Ruiz Notes - collection of security, system, network and pentest cheatsheets.
   SSRF Tips - a collection of SSRF Tips.
   shell-storm repo CTF - great archive of CTFs.
   ctf - CTF (Capture The Flag) writeups, code snippets, notes, scripts.
   My-CTF-Web-Challenges - collection of CTF Web challenges.
   MSTG - The Mobile Security Testing Guide (MSTG) is a comprehensive manual for mobile app security testing.
   Internal-Pentest-Playbook - notes on the most common things for an Internal Network Penetration Test.
   KeyHacks - shows quick ways in which API keys leaked by a bug bounty program can be checked.
   securitum/research - various Proof of Concepts of security research performed by Securitum.
   public-pentesting-reports - is a list of public pentest reports released by several consulting security groups.
   awesome-bug-bounty - is a comprehensive curated list of available Bug Bounty.
   bug-bounty-reference - is a list of bug bounty write-ups.
   Awesome-Bugbounty-Writeups - is a curated list of bugbounty writeups.
   Bug bounty writeups - list of bug bounty writeups (2012-2020).
   hackso.me - a great journey into security.

:black_small_square: Backdoors/exploits

PHP-backdoors - a collection of PHP backdoors. For educational or testing purposes only.

:black_small_square: Wordlists and Weak passwords

   Weakpass - for any kind of bruteforce find wordlists or unleash the power of them all at once!
   Hashes.org - is a free online hash resolving service incorporating many unparalleled techniques.
   SecLists - collection of multiple types of lists used during security assessments, collected in one place.
   Probable-Wordlists - sorted by probability originally created for password generation and testing.
   skullsecurity passwords - password dictionaries and leaked passwords repository.
   Polish PREMIUM Dictionary - official dictionary created by the team on the forum bezpieka.org.* ¹
   statistically-likely-usernames - wordlists for creating statistically likely username lists.

:black_small_square: Bounty platforms

   YesWeHack - bug bounty platform with infosec jobs.
   Openbugbounty - allows any security researcher reporting a vulnerability on any website.
   hackerone - global hacker community to surface the most relevant security issues.
   bugcrowd - crowdsourced cybersecurity for the enterprise.
   Crowdshield - crowdsourced security & bug bounty management.
   Synack - crowdsourced security & bug bounty programs, crowd security intelligence platform, and more.
   Hacktrophy - bug bounty platform.

:black_small_square: Web Training Apps (local installation)

   OWASP-VWAD - comprehensive and well maintained registry of all known vulnerable web applications.
   DVWA - PHP/MySQL web application that is damn vulnerable.
   metasploitable2 - vulnerable web application amongst security researchers.
   metasploitable3 - is a VM that is built from the ground up with a large amount of security vulnerabilities.
   DSVW - is a deliberately vulnerable web application written in under 100 lines of code.
   OWASP Mutillidae II - free, open source, deliberately vulnerable web-application.
   OWASP Juice Shop Project - the most bug-free vulnerable application in existence.
   OWASP Node js Goat Project - OWASP Top 10 security risks apply to web apps developed using Node.js.
   juicy-ctf - run Capture the Flags and Security Trainings with OWASP Juice Shop.
   SecurityShepherd - web and mobile application security training platform.
   Security Ninjas - open source application security training program.
   hackazon - a modern vulnerable web app.
   dvna - damn vulnerable NodeJS application.
   django-DefectDojo - is an open-source application vulnerability correlation and security orchestration tool.
   Google Gruyere - web application exploits and defenses.
   Bodhi - is a playground focused on learning the exploitation of client-side web vulnerabilities.
   Websploit - single vm lab with the purpose of combining several vulnerable appliations in one environment.
   vulhub - pre-built Vulnerable Environments based on docker-compose.
   CloudGoat 2 - the new & improved "Vulnerable by Design" AWS deployment tool.
   secDevLabs - is a laboratory for learning secure web development in a practical manner.
   CORS-vulnerable-Lab - sample vulnerable code and its exploit code.
   RootTheBox - a Game of Hackers (CTF Scoreboard & Game Manager).
   KONTRA - application security training (OWASP Top Web & Api).

:black_small_square: Labs (ethical hacking platforms/trainings/CTFs)

   Offensive Security - true performance-based penetration testing training for over a decade.
   Hack The Box - online platform allowing you to test your penetration testing skills.
   Hacking-Lab - online ethical hacking, computer network and security challenge platform.
   pwnable.kr - non-commercial wargame site which provides various pwn challenges.
   Pwnable.tw - is a wargame site for hackers to test and expand their binary exploiting skills.
   picoCTF - is a free computer security game targeted at middle and high school students.
   CTFlearn - is an online platform built to help ethical hackers learn and practice their cybersecurity knowledge.
   ctftime - CTF archive and a place, where you can get some another CTF-related info.
   Silesia Security Lab - high quality security testing services.
   Practical Pentest Labs - pentest lab, take your Hacking skills to the next level.
   Root Me - the fast, easy, and affordable way to train your hacking skills.
   rozwal.to - a great platform to train your pentesting skills.
   TryHackMe - learning Cyber Security made easy.
   hackxor - is a realistic web application hacking game, designed to help players of all abilities develop their skills.
   Hack Yourself First - it's full of nasty app sec holes.
   OverTheWire - can help you to learn and practice security concepts in the form of fun-filled games.
   Wizard Labs - is an online Penetration Testing Lab.
   PentesterLab - provides vulnerable systems that can be used to test and understand vulnerabilities.
   RingZer0 - tons of challenges designed to test and improve your hacking skills.
   try2hack - several security-oriented challenges for your entertainment.
   Ubeeri - preconfigured lab environments.
   Pentestit - emulate IT infrastructures of real companies for legal pen testing and improving pentest skills.
   Microcorruption - reversal challenges done in the web interface.
   Crackmes - download crackmes to help improve your reverse engineering skills.
   DomGoat - DOM XSS security learning and practicing platform.
   Stereotyped Challenges - upgrade your web hacking techniques today!
   Vulnhub - allows anyone to gain practical 'hands-on' experience in digital security.
   W3Challs - is a penetration testing training platform, which offers various computer challenges.
   RingZer0 CTF - offers you tons of challenges designed to test and improve your hacking skills.
   Hack.me - a platform where you can build, host and share vulnerable web apps for educational purposes.
   HackThis! - discover how hacks, dumps and defacements are performed and secure your website.
   Enigma Group WebApp Training - these challenges cover the exploits listed in the OWASP Top 10 Project.
   Reverse Engineering Challenges - challenges, exercises, problems and tasks - by level, by type, and more.
   0x00sec - the home of the Hacker - Malware, Reverse Engineering, and Computer Science.
   We Chall - there are exist a lots of different challenge types.
   Hacker Gateway - is the go-to place for hackers who want to test their skills.
   Hacker101 - is a free class for web security.
   contained.af - a stupid game for learning about containers, capabilities, and syscalls.
   flAWS challenge! - a series of levels you'll learn about common mistakes and gotchas when using AWS.
   CyberSec WTF - provides web hacking challenges derived from bounty write-ups.
   CTF Challenge - CTF Web App challenges.
   gCTF - most of the challenges used in the Google CTF 2017.
   Hack This Site - is a free, safe and legal training ground for hackers.
   Attack & Defense - is a browser-based cloud labs.
   Cryptohack - a fun platform for learning modern cryptography.
   Cryptopals - the cryptopals crypto challenges.

:black_small_square: CTF platforms

fbctf - platform to host Capture the Flag competitions.
ctfscoreboard - scoreboard for Capture The Flag competitions.

:black_small_square: Other resources

   Bugcrowd University - open source education content for the researcher community.
   OSCPRepo - a list of resources and scripts that I have been gathering in preparation for the OSCP.
   OWASP Top 10: Real-World Examples - test your web apps with real-world examples (two-part series).
   phrack.org - an awesome collection of articles from several respected hackers and other thinkers.
   Practical-Ethical-Hacking-Resources - compilation of resources from TCM's Udemy Course.

Your daily knowledge and news ^[TOC]

:black_small_square: RSS Readers

Feedly - organize, read and share what matters to you.
Inoreader - similar to feedly with a support for filtering what you fetch from rss.

:black_small_square: IRC Channels

#hackerspaces - hackerspace IRC channels.

:black_small_square: Security

   The Hacker News - leading news source dedicated to promoting awareness for security experts and hackers.
   Latest Hacking News - provides the latest hacking news, exploits and vulnerabilities for ethical hackers.
   Security Newsletter - security news as a weekly digest (email notifications).
   Google Online Security Blog - the latest news and insights from Google on security and safety on the Internet.
   Qualys Blog - expert network security guidance and news.
   DARKReading - connecting the Information Security Community.
   Darknet - latest hacking tools, hacker news, cybersecurity best practices, ethical hacking & pen-testing.
   publiclyDisclosed - public disclosure watcher who keeps you up to date about the recently disclosed bugs.
   Reddit - Hacking - a subreddit dedicated to hacking and hackers.
   Packet Storm - information security services, news, files, tools, exploits, advisories and whitepapers.
   Sekurak - about security, penetration tests, vulnerabilities and many others (PL/EN).
   nf.sec - basic aspects and mechanisms of Linux operating system security (PL).

:black_small_square: Other/All-in-one

Changelog - is a community of hackers; news & podcasts for developers and hackers.

Other Cheat Sheets ^[TOC]

Build your own DNS Servers

   Unbound DNS Tutorial - a validating, recursive, and caching DNS server.
   Knot Resolver on Fedora - how to get faster and more secure DNS resolution with Knot Resolver on Fedora.
   DNS-over-HTTPS - tutorial to setup your own DNS-over-HTTPS (DoH) server.
   dns-over-https - a cartoon intro to DNS over HTTPS.
   DNS-over-TLS - following to your DoH server, setup your DNS-over-TLS (DoT) server.
   DNS Servers - how (and why) i run my own DNS Servers.

Build your own Certificate Authority

OpenSSL Certificate Authority - build your own certificate authority (CA) using the OpenSSL tools.
step-ca Certificate Authority - build your own certificate authority (CA) using open source step-ca.

Build your own System/Virtual Machine

   os-tutorial - how to create an OS from scratch.
   Write your Own Virtual Machine - how to write your own virtual machine (VM).
   x86 Bare Metal Examples - dozens of minimal operating systems to learn x86 system programming.
   simple-computer - the scott CPU from "But How Do It Know?" by J. Clark Scott.
   littleosbook - the little book about OS development.

DNS Servers list (privacy)

IP	URL
`84.200.69.80`	dns.watch
`94.247.43.254`	opennic.org
`64.6.64.6`	verisign.com
`89.233.43.71`	censurfridns.dk
`1.1.1.1`	cloudflare.com
`94.130.110.185`	dnsprivacy.at

TOP Browser extensions

Extension name	Description
`IPvFoo`	Display the server IP address and HTTPS information across all page elements.
`FoxyProxy`	Simplifies configuring browsers to access proxy-servers.
`HTTPS Everywhere`	Automatically use HTTPS security on many sites.
`uMatrix`	Point & click to forbid/allow any class of requests made by your browser.
`uBlock Origin`	An efficient blocker: easy on memory and CPU footprint.
`Session Buddy`	Manage browser tabs and bookmarks with ease.
`SuperSorter`	Sort bookmarks recursively, delete duplicates, merge folders, and more.
`Clear Cache`	Clear your cache and browsing data.
`d3coder`	Encoding/Decoding plugin for various types of encoding.
`Web Developer`	Adds a toolbar button with various web developer tools.
`ThreatPinch Lookup`	Add threat intelligence hover tool tips.

TOP Burp extensions

Extension name	Description
`Active Scan++`	Extends Burp’s active and passive scanning capabilities.
`Autorize`	Automatically detects authorization enforcement.
`AuthMatrix`	A simple matrix grid to define the desired levels of access privilege.
`Logger++`	Logs requests and responses for all Burp tools in a sortable table.
`Bypass WAF`	Adds headers useful for bypassing some WAF devices.
`JSON Beautifier`	Beautifies JSON content in the HTTP message viewer.
`JSON Web Tokens`	Enables Burp to decode and manipulate JSON web tokens.
`CSP Auditor`	Displays CSP headers for responses, and passively reports CSP weaknesses.
`CSP-Bypass`	Passively scans for CSP headers that contain known bypasses.
`Hackvertor`	Converts data using a tag-based configuration to apply various encoding.
`HTML5 Auditor`	Scans for usage of risky HTML5 features.
`Software Vulnerability Scanner`	Vulnerability scanner based on vulners.com audit API.
`Turbo Intruder`	Is a powerful bruteforcing tool.
`Upload Scanner`	Upload a number of different file types, laced with different forms of payload.

Hack Mozilla Firefox address bar

In Firefox’s address bar, you can limit results by typing special characters before or after your term:

^ - for matches in your browsing history
* - for matches in your bookmarks.
% - for matches in your currently open tabs.
# - for matches in page titles.
@ - for matches in web addresses.

Chrome hidden commands

chrome://chrome-urls - list of all commands
chrome://flags - enable experiments and development features
chrome://interstitials - errors and warnings
chrome://net-internals - network internals (events, dns, cache)
chrome://network-errors - network errors
chrome://net-export - start logging future network activity to a file
chrome://safe-browsing - safe browsing options
chrome://user-actions - record all user actions
chrome://restart - restart chrome
chrome://dino - ERR_INTERNET_DISCONNECTED…
cache:<website-address> - view the cached version of the web page

Bypass WAFs by Shortening IP Address (by 0xInfection)

IP addresses can be shortened by dropping the zeroes:

http://1.0.0.1 → http://1.1
http://127.0.0.1 → http://127.1
http://192.168.0.1 → http://192.168.1

http://0xC0A80001 or http://3232235521 → 192.168.0.1
http://192.168.257 → 192.168.1.1
http://192.168.516 → 192.168.2.4

This bypasses WAF filters for SSRF, open-redirect, etc where any IP as input gets blacklisted.

For more information please see How to Obscure Any URL and Magic IP Address Shortcuts.

Hashing, encryption and encoding (by Michal Špaček)

Hashing

plaintext :arrow_right: hash
hash :no_entry: plaintext

Symmetric encryption

plaintext :arrow_right: :key: :arrow_right: ciphertext
plaintext :arrow_left: :key: :arrow_left: ciphertext
(:key: shared key)

Asymmetric encryption

plaintext :arrow_right: :key: :arrow_right: ciphertext
plaintext :arrow_left: :part_alternation_mark: :arrow_left: ciphertext
(:key: public key, :part_alternation_mark: private key)

Encoding

text :arrow_right: encoded
text :arrow_left: encoded

Shell One-liners ^[TOC]

Tool: terminal

Reload shell without exit

`1`	`exec $SHELL -l`

Close shell keeping all subprocess running

`1`	`disown -a && exit`

Exit without saving shell history

1
2

kill -9 $$
unset HISTFILE && exit

Perform a branching conditional

1
2

true && echo success
false || echo failed

Pipe stdout and stderr to separate commands

`1`	`some_command > >(/bin/cmd_for_stdout) 2> >(/bin/cmd_for_stderr)`

Redirect stdout and stderr each to separate files and print both to the screen

`1`	`(some_command 2>&1 1>&3 \| tee errorlog ) 3>&1 1>&2 \| tee stdoutlog`

List of commands you use most often

history | \
awk '{CMD[$2]++;count++;}END { for (a in CMD)print CMD[a] " " CMD[a]/count*100 "% " a;}' | \
grep -v "./" | \
column -c3 -s " " -t | \
sort -nr | nl |  head -n 20

Sterilize bash history

function sterile() {

  history | awk '$2 != "history" { $1=""; print $0 }' | egrep -vi "\
curl\b+.*(-E|--cert)\b+.*\b*|\
curl\b+.*--pass\b+.*\b*|\
curl\b+.*(-U|--proxy-user).*:.*\b*|\
curl\b+.*(-u|--user).*:.*\b*
.*(-H|--header).*(token|auth.*)\b+.*|\
wget\b+.*--.*password\b+.*\b*|\
http.?://.+:.+@.*\
" > $HOME/histbuff; history -r $HOME/histbuff;

}

export PROMPT_COMMAND="sterile"

Look also: A naive utility to censor credentials in command history.

Quickly backup a file

`1`	`cp filename{,.orig}`

Empty a file (truncate to 0 size)

`1`	`>filename`

Delete all files in a folder that don’t match a certain file extension

`1`	`rm !(.foo\|.bar\|*.baz)`

Pass multi-line string to a file

# cat  >filename ... - overwrite the file
# cat >>filename ... - append to a file
cat > filename << __EOF__
data data data
__EOF__

Edit a file on a remote host using vim

`1`	`vim scp://user@host//etc/fstab`

Create a directory and change into it at the same time

`1`	`mkd() { mkdir -p "$@" && cd "$@"; }`

Convert uppercase files to lowercase files

`1`	`rename 'y/A-Z/a-z/' *`

Print a row of characters across the terminal

`1`	printf "%`tput cols`s" \| tr ' ' '#'

Show shell history without line numbers

1
2

history | cut -c 8-
fc -l -n 1 | sed 's/^\s*//'

Run command(s) after exit session

cat > /etc/profile << __EOF__
_after_logout() {

  username=$(whoami)

  for _pid in $(ps afx | grep sshd | grep "$username" | awk '{print $1}') ; do

    kill -9 $_pid

  done

}
trap _after_logout EXIT
__EOF__

Generate a sequence of numbers

for ((i=1; i<=10; i+=2)) ; do echo $i ; done
# alternative: seq 1 2 10

for ((i=5; i<=10; ++i)) ; do printf '%02d\n' $i ; done
# alternative: seq -w 5 10

for i in {1..10} ; do echo $i ; done

Simple Bash filewatching

`1`	`unset MAIL; export MAILCHECK=1; export MAILPATH='$FILE_TO_WATCH?$MESSAGE'`

Tool: busybox

Static HTTP web server

`1`	`busybox httpd -p $PORT -h $HOME [-c httpd.conf]`

Tool: mount

Mount a temporary ram partition

`1`	`mount -t tmpfs tmpfs /mnt -o size=64M`

-t - filesystem type
-o - mount options

Remount a filesystem as read/write

`1`	`mount -o remount,rw /`

Tool: fuser

Show which processes use the files/directories

1
2

fuser /var/log/daemon.log
fuser -v /home/supervisor

Kills a process that is locking a file

`1`	`fuser -ki filename`

-i - interactive option

Kills a process that is locking a file with specific signal

`1`	`fuser -k -HUP filename`

--list-signals - list available signal names

Show what PID is listening on specific port

`1`	`fuser -v 53/udp`

Show all processes using the named filesystems or block device

`1`	`fuser -mv /var/www`

Tool: lsof

Show process that use internet connection at the moment

`1`	`lsof -P -i -n`

Show process that use specific port number

`1`	`lsof -i tcp:443`

Lists all listening ports together with the PID of the associated process

`1`	`lsof -Pan -i tcp -i udp`

List all open ports and their owning executables

`1`	`lsof -i -P \| grep -i "listen"`

Show all open ports

`1`	`lsof -Pnl -i`

Show open ports (LISTEN)

`1`	`lsof -Pni4 \| grep LISTEN \| column -t`

List all files opened by a particular command

`1`	`lsof -c "process"`

View user activity per directory

`1`	`lsof -u username -a +D /etc`

Show 10 largest open files

1
2
3

lsof / | \
awk '{ if($7 > 1048576) print $7/1048576 "MB" " " $9 " " $1 }' | \
sort -n -u | tail | column -t

Show current working directory of a process

`1`	`lsof -p <PID> \| grep cwd`

Tool: ps

Show a 4-way scrollable process tree with full details

`1`	`ps awwfux \| less -S`

Processes per user counter

`1`	`ps hax -o user \| sort \| uniq -c \| sort -r`

Show all processes by name with main header

`1`	`ps -lfC nginx`

Tool: find

Find files that have been modified on your system in the past 60 minutes

`1`	`find / -mmin 60 -type f`

Find all files larger than 20M

`1`	`find / -type f -size +20M`

Find duplicate files (based on MD5 hash)

`1`	`find -type f -exec md5sum '{}' ';' \| sort \| uniq --all-repeated=separate -w 33`

Change permission only for files

1
2

cd /var/www/site && find . -type f -exec chmod 766 {} \;
cd /var/www/site && find . -type f -exec chmod 664 {} +

Change permission only for directories

1
2

cd /var/www/site && find . -type d -exec chmod g+x {} \;
cd /var/www/site && find . -type d -exec chmod g+rwx {} +

Find files and directories for specific user/group

# User:
find . -user <username> -print
find /etc -type f -user <username> -name "*.conf"

# Group:
find /opt -group <group>
find /etc -type f -group <group> -iname "*.conf"

Find files and directories for all without specific user/group

# User:
find . \! -user <username> -print

# Group:
find . \! -group <group>

Looking for files/directories that only have certain permission

# User
find . -user <username> -perm -u+rw # -rw-r--r--
find /home -user $(whoami) -perm 777 # -rwxrwxrwx

# Group:
find /home -type d -group <group> -perm 755 # -rwxr-xr-x

Delete older files than 60 days

`1`	`find . -type f -mtime +60 -delete`

Recursively remove all empty sub-directories from a directory

`1`	`find . -depth -type d -empty -exec rmdir {} \;`

How to find all hard links to a file

`1`	`find </path/to/dir> -xdev -samefile filename`

Recursively find the latest modified files

`1`	`find . -type f -exec stat --format '%Y :%y %n' "{}" \; \| sort -nr \| cut -d: -f2- \| head`

Recursively find/replace of a string with sed

`1`	`find . -not -path '/\.git' -type f -print0 \| xargs -0 sed -i 's/foo/bar/g'`

Recursively find/replace of a string in directories and file names

`1`	`find . -depth -name 'test' -execdir bash -c 'mv -v "$1" "${1//foo/bar}"' _ {} \;`

Recursively find suid executables

`1`	`find / $ -perm -4000 -o -perm -2000 $ -type f -exec ls -la {} \;`

Tool: top

Use top to monitor only all processes with the specific string

`1`	`top -p $(pgrep -d , <str>)`

<str> - process containing string (eg. nginx, worker)

Tool: vmstat

Show current system utilization (fields in kilobytes)

`1`	`vmstat 2 20 -t -w`

2 - number of times with a defined time interval (delay)
20 - each execution of the command (count)
-t - show timestamp
-w - wide output
-S M - output of the fields in megabytes instead of kilobytes

Show current system utilization will get refreshed every 5 seconds

`1`	`vmstat 5 -w`

Display report a summary of disk operations

`1`	`vmstat -D`

Display report of event counters and memory stats

`1`	`vmstat -s`

Display report about kernel objects stored in slab layer cache

`1`	`vmstat -m`

Tool: iostat

Show information about the CPU usage, and I/O statistics about all the partitions

`1`	`iostat 2 10 -t -m`

2 - number of times with a defined time interval (delay)
10 - each execution of the command (count)
-t - show timestamp
-m - fields in megabytes (-k - in kilobytes, default)

Show information only about the CPU utilization

`1`	`iostat 2 10 -t -m -c`

Show information only about the disk utilization

`1`	`iostat 2 10 -t -m -d`

Show information only about the LVM utilization

`1`	`iostat -N`

Tool: strace

Track with child processes

# 1)
strace -f -p $(pidof glusterfsd)

# 2)
strace -f $(pidof php-fpm | sed 's/\([0-9]*\)/\-p \1/g')

Track process with 30 seconds limit

`1`	`timeout 30 strace $(< /var/run/zabbix/zabbix_agentd.pid)`

Track processes and redirect output to a file

1
2

ps auxw | grep '[a]pache' | awk '{print " -p " $2}' | \
xargs strace -o /tmp/strace-apache-proc.out

Track with print time spent in each syscall and limit length of print strings

1
2

ps auxw | grep '[i]init_policy' | awk '{print " -p " $2}' | \
xargs strace -f -e trace=network -T -s 10000

Track the open request of a network port

`1`	`strace -f -e trace=bind nc -l 80`

Track the open request of a network port (show TCP/UDP)

`1`	`strace -f -e trace=network nc -lu 80`

Tool: kill

Kill a process running on port

`1`	`kill -9 $(lsof -i :<port> \| awk '{l=$2} END {print l}')`

Tool: diff

Compare two directory trees

`1`	`diff <(cd directory1 && find \| sort) <(cd directory2 && find \| sort)`

Compare output of two commands

`1`	`diff <(cat /etc/passwd) <(cut -f2 /etc/passwd)`

Tool: vimdiff

Highlight the exact differences, based on characters and words

`1`	`vimdiff file1 file2`

Compare two JSON files

`1`	`vimdiff <(jq -S . A.json) <(jq -S . B.json)`

Compare Hex dump

`1`	`d(){ vimdiff <(f $1) <(f $2);};f(){ hexdump -C $1 \| cut -d' ' -f3- \| tr -s ' ';}; d ~/bin1 ~/bin2`

diffchar

Save diffchar @ ~/.vim/plugins

Click F7 to switch between diff modes

Usefull vimdiff commands:

qa to exit all windows
:vertical resize 70 to resize window
set window width Ctrl+W [N columns]+(Shift+)<\>

Tool: tail

Annotate tail -f with timestamps

`1`	`tail -f file \| while read ; do echo "$(date +%T.%N) $REPLY" ; done`

Analyse an Apache access log for the most common IP addresses

`1`	`tail -10000 access_log \| awk '{print $1}' \| sort \| uniq -c \| sort -n \| tail`

Analyse web server log and show only 5xx http codes

`1`	`tail -n 100 -f /path/to/logfile \| grep "HTTP/[1-2].[0-1]\" [5]"`

Tool: tar

System backup with exclude specific directories

1
2
3

cd /
tar -czvpf /mnt/system$(date +%d%m%Y%s).tgz --directory=/ \
--exclude=proc/* --exclude=sys/* --exclude=dev/* --exclude=mnt/* .

System backup with exclude specific directories (pigz)

cd /
tar cvpf /backup/snapshot-$(date +%d%m%Y%s).tgz --directory=/ \
--exclude=proc/* --exclude=sys/* --exclude=dev/* \
--exclude=mnt/* --exclude=tmp/* --use-compress-program=pigz .

Tool: dump

System backup to file

`1`	`dump -y -u -f /backup/system$(date +%d%m%Y%s).lzo /`

Restore system from lzo file

1
2

cd /
restore -rf /backup/system$(date +%d%m%Y%s).lzo

Tool: cpulimit

Limit the cpu usage of a process

`1`	`cpulimit -p pid -l 50`

Tool: pwdx

Show current working directory of a process

`1`	`pwdx <pid>`

Tool: taskset

Start a command on only one CPU core

`1`	`taskset -c 0 <command>`

Tool: tr

Show directories in the PATH, one per line

`1`	`tr : '\n' <<<$PATH`

Tool: chmod

Remove executable bit from all files in the current directory

`1`	`chmod -R -x+X *`

Restore permission for /bin/chmod

# 1:
cp /bin/ls chmod.01
cp /bin/chmod chmod.01
./chmod.01 700 file

# 2:
/bin/busybox chmod 0700 /bin/chmod

# 3:
setfacl --set u::rwx,g::---,o::--- /bin/chmod

Tool: who

Find last reboot time

who -b

Detect a user sudo-su’d into the current shell

`1`	`[[ $(who -m \| awk '{ print $1 }') == $(whoami) ]] \|\| echo "You are su-ed to $(whoami)"`

Tool: last

Was the last reboot a panic?

1
2

(last -x -f $(ls -1t /var/log/wtmp* | head -2 | tail -1); last -x -f /var/log/wtmp) | \
grep -A1 reboot | head -2 | grep -q shutdown && echo "Expected reboot" || echo "Panic reboot"

Tool: screen

Start screen in detached mode

`1`	`screen -d -m <command>`

Attach to an existing screen session

`1`	`screen -r -d <pid>`

Tool: script

Record and replay terminal session

### Record session
# 1)
script -t 2>~/session.time -a ~/session.log

# 2)
script --timing=session.time session.log

### Replay session
scriptreplay --timing=session.time session.log

Tool: du

Show 20 biggest directories with ‘K M G’

du | \
sort -r -n | \
awk '{split("K M G",v); s=1; while($1>1024){$1/=1024; s++} print int($1)" "v[s]"\t"$2}' | \
head -n 20

Tool: inotifywait

Init tool everytime a file in a directory is modified

`1`	`while true ; do inotifywait -r -e MODIFY dir/ && ls dir/ ; done;`

Tool: openssl

Testing connection to the remote host

`1`	`echo \| openssl s_client -connect google.com:443 -showcerts`

Testing connection to the remote host (debug mode)

`1`	`echo \| openssl s_client -connect google.com:443 -showcerts -tlsextdebug -status`

Testing connection to the remote host (with SNI support)

`1`	`echo \| openssl s_client -showcerts -servername google.com -connect google.com:443`

Testing connection to the remote host with specific ssl version

`1`	`openssl s_client -tls1_2 -connect google.com:443`

Testing connection to the remote host with specific ssl cipher

`1`	`openssl s_client -cipher 'AES128-SHA' -connect google.com:443`

Verify 0-RTT

_host="example.com"

cat > req.in << __EOF__
HEAD / HTTP/1.1
Host: $_host
Connection: close
__EOF__

openssl s_client -connect ${_host}:443 -tls1_3 -sess_out session.pem -ign_eof < req.in
openssl s_client -connect ${_host}:443 -tls1_3 -sess_in session.pem -early_data req.in

Generate private key without passphrase

1
2
3

# _len: 2048, 4096
( _fd="private.key" ; _len="2048" ; \
openssl genrsa -out ${_fd} ${_len} )

Generate private key with passphrase

# _ciph: aes128, aes256
# _len: 2048, 4096
( _ciph="aes128" ; _fd="private.key" ; _len="2048" ; \
openssl genrsa -${_ciph} -out ${_fd} ${_len} )

Remove passphrase from private key

1
2

( _fd="private.key" ; _fd_unp="private_unp.key" ; \
openssl rsa -in ${_fd} -out ${_fd_unp} )

Encrypt existing private key with a passphrase

1
2
3

# _ciph: aes128, aes256
( _ciph="aes128" ; _fd="private.key" ; _fd_pass="private_pass.key" ; \
openssl rsa -${_ciph} -in ${_fd} -out ${_fd_pass}

Check private key

1
2

( _fd="private.key" ; \
openssl rsa -check -in ${_fd} )

Get public key from private key

1
2

( _fd="private.key" ; _fd_pub="public.key" ; \
openssl rsa -pubout -in ${_fd} -out ${_fd_pub} )

Generate private key and CSR

1
2

( _fd="private.key" ; _fd_csr="request.csr" ; _len="2048" ; \
openssl req -out ${_fd_csr} -new -newkey rsa:${_len} -nodes -keyout ${_fd} )

Generate CSR

1
2

( _fd="private.key" ; _fd_csr="request.csr" ; \
openssl req -out ${_fd_csr} -new -key ${_fd} )

Generate CSR (metadata from existing certificate)

Where private.key is the existing private key. As you can see you do not generate this CSR from your certificate (public key). Also you do not generate the “same” CSR, just a new one to request a new certificate.

1
2

( _fd="private.key" ; _fd_csr="request.csr" ; _fd_crt="cert.crt" ; \
openssl x509 -x509toreq -in ${_fd_crt} -out ${_fd_csr} -signkey ${_fd} )

Generate CSR with -config param

( _fd="private.key" ; _fd_csr="request.csr" ; \
openssl req -new -sha256 -key ${_fd} -out ${_fd_csr} \
-config <(
cat << __EOF__
[req]
default_bits        = 2048
default_md          = sha256
prompt              = no
distinguished_name  = dn
req_extensions      = req_ext

[ dn ]
C   = "<two-letter ISO abbreviation for your country>"
ST  = "<state or province where your organisation is legally located>"
L   = "<city where your organisation is legally located>"
O   = "<legal name of your organisation>"
OU  = "<section of the organisation>"
CN  = "<fully qualified domain name>"

[ req_ext ]
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = <fully qualified domain name>
DNS.2 = <next domain>
DNS.3 = <next domain>
__EOF__
))

Other values in [ dn ]:

countryName            = "DE"                     # C=
stateOrProvinceName    = "Hessen"                 # ST=
localityName           = "Keller"                 # L=
postalCode             = "424242"                 # L/postalcode=
postalAddress          = "Keller"                 # L/postaladdress=
streetAddress          = "Crater 1621"            # L/street=
organizationName       = "apfelboymschule"        # O=
organizationalUnitName = "IT Department"          # OU=
commonName             = "example.com"            # CN=
emailAddress           = "webmaster@example.com"  # CN/emailAddress=

Example of oids (you’ll probably also have to make OpenSSL know about the new fields required for EV by adding the following under [new_oids]):

[req]
...
oid_section         = new_oids

[ new_oids ]
postalCode = 2.5.4.17
streetAddress = 2.5.4.9

Full example:

( _fd="private.key" ; _fd_csr="request.csr" ; \
openssl req -new -sha256 -key ${_fd} -out ${_fd_csr} \
-config <(
cat << __EOF__
[req]
default_bits        = 2048
default_md          = sha256
prompt              = no
distinguished_name  = dn
req_extensions      = req_ext
oid_section         = new_oids

[ new_oids ]
serialNumber = 2.5.4.5
streetAddress = 2.5.4.9
postalCode = 2.5.4.17
businessCategory = 2.5.4.15

[ dn ]
serialNumber=00001111
businessCategory=Private Organization
jurisdictionC=DE
C=DE
ST=Hessen
L=Keller
postalCode=424242
streetAddress=Crater 1621
O=AV Company
OU=IT
CN=example.com

[ req_ext ]
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = example.com
__EOF__
))

For more information please look at these great explanations:

List available EC curves

`1`	`openssl ecparam -list_curves`

Print ECDSA private and public keys

( _fd="private.key" ; \
openssl ec -in ${_fd} -noout -text )

# For x25519 only extracting public key
( _fd="private.key" ; _fd_pub="public.key" ; \
openssl pkey -in ${_fd} -pubout -out ${_fd_pub} )

Generate ECDSA private key

# _curve: prime256v1, secp521r1, secp384r1
( _fd="private.key" ; _curve="prime256v1" ; \
openssl ecparam -out ${_fd} -name ${_curve} -genkey )

# _curve: X25519
( _fd="private.key" ; _curve="x25519" ; \
openssl genpkey -algorithm ${_curve} -out ${_fd} )

Generate private key and CSR (ECC)

# _curve: prime256v1, secp521r1, secp384r1
( _fd="domain.com.key" ; _fd_csr="domain.com.csr" ; _curve="prime256v1" ; \
openssl ecparam -out ${_fd} -name ${_curve} -genkey ; \
openssl req -new -key ${_fd} -out ${_fd_csr} -sha256 )

Generate self-signed certificate

# _len: 2048, 4096
( _fd="domain.key" ; _fd_out="domain.crt" ; _len="2048" ; _days="365" ; \
openssl req -newkey rsa:${_len} -nodes \
-keyout ${_fd} -x509 -days ${_days} -out ${_fd_out} )

Generate self-signed certificate from existing private key

# _len: 2048, 4096
( _fd="domain.key" ; _fd_out="domain.crt" ; _days="365" ; \
openssl req -key ${_fd} -nodes \
-x509 -days ${_days} -out ${_fd_out} )

Generate self-signed certificate from existing private key and csr

# _len: 2048, 4096
( _fd="domain.key" ; _fd_csr="domain.csr" ; _fd_out="domain.crt" ; _days="365" ; \
openssl x509 -signkey ${_fd} -nodes \
-in ${_fd_csr} -req -days ${_days} -out ${_fd_out} )

Generate DH public parameters

1
2

( _dh_size="2048" ; \
openssl dhparam -out /etc/nginx/ssl/dhparam_${_dh_size}.pem "$_dh_size" )

Display DH public parameters

`1`	`openssl pkeyparam -in dhparam.pem -text`

Extract private key from pfx

1
2

( _fd_pfx="cert.pfx" ; _fd_key="key.pem" ; \
openssl pkcs12 -in ${_fd_pfx} -nocerts -nodes -out ${_fd_key} )

Extract private key and certs from pfx

1
2

( _fd_pfx="cert.pfx" ; _fd_pem="key_certs.pem" ; \
openssl pkcs12 -in ${_fd_pfx} -nodes -out ${_fd_pem} )

Extract certs from p7b

# PKCS#7 file doesn't include private keys.
( _fd_p7b="cert.p7b" ; _fd_pem="cert.pem" ; \
openssl pkcs7 -inform DER -outform PEM -in ${_fd_p7b} -print_certs > ${_fd_pem})
# or:
openssl pkcs7 -print_certs -in -in ${_fd_p7b} -out ${_fd_pem})

Convert DER to PEM

1
2

( _fd_der="cert.crt" ; _fd_pem="cert.pem" ; \
openssl x509 -in ${_fd_der} -inform der -outform pem -out ${_fd_pem} )

Convert PEM to DER

1
2

( _fd_der="cert.crt" ; _fd_pem="cert.pem" ; \
openssl x509 -in ${_fd_pem} -outform der -out ${_fd_der} )

Verification of the private key

1
2

( _fd="private.key" ; \
openssl rsa -noout -text -in ${_fd} )

Verification of the public key

# 1)
( _fd="public.key" ; \
openssl pkey -noout -text -pubin -in ${_fd} )

# 2)
( _fd="private.key" ; \
openssl rsa -inform PEM -noout -in ${_fd} &> /dev/null ; \
if [ $? = 0 ] ; then echo -en "OK\n" ; fi )

Verification of the certificate

1
2

( _fd="certificate.crt" ; # format: pem, cer, crt \
openssl x509 -noout -text -in ${_fd} )

Verification of the CSR

1
2

( _fd_csr="request.csr" ; \
openssl req -text -noout -in ${_fd_csr} )

Check the private key and the certificate are match

1
2

(openssl rsa -noout -modulus -in private.key | openssl md5 ; \
openssl x509 -noout -modulus -in certificate.crt | openssl md5) | uniq

Check the private key and the CSR are match

1
2

(openssl rsa -noout -modulus -in private.key | openssl md5 ; \
openssl req -noout -modulus -in request.csr | openssl md5) | uniq

Tool: secure-delete

Secure delete with shred

1
2

shred -vfuz -n 10 file
shred --verbose --random-source=/dev/urandom -n 1 /dev/sda

Secure delete with scrub

1
2

scrub -p dod /dev/sda
scrub -p dod -r file

Secure delete with badblocks

1
2

badblocks -s -w -t random -v /dev/sda
badblocks -c 10240 -s -w -t random -v /dev/sda

Secure delete with secure-delete

srm -vz /tmp/file
sfill -vz /local
sdmem -v
swapoff /dev/sda5 && sswap -vz /dev/sda5

Tool: dd

Show dd status every so often

1
2

dd <dd_params> status=progress
watch --interval 5 killall -USR1 dd

Redirect output to a file with dd

`1`	`echo "string" \| dd of=filename`

Tool: gpg

Export public key

`1`	`gpg --export --armor "<username>" > username.pkey`

--export - export all keys from all keyrings or specific key
-a|--armor - create ASCII armored output

Encrypt file

`1`	`gpg -e -r "<username>" dump.sql`

-e|--encrypt - encrypt data
-r|--recipient - encrypt for specific

Decrypt file

`1`	`gpg -o dump.sql -d dump.sql.gpg`

-o|--output - use as output file
-d|--decrypt - decrypt data (default)

Search recipient

`1`	`gpg --keyserver hkp://keyserver.ubuntu.com --search-keys "<username>"`

--keyserver - set specific key server
--search-keys - search for keys on a key server

List all of the packets in an encrypted file

1
2

gpg --batch --list-packets archive.gpg
gpg2 --batch --list-packets archive.gpg

Tool: system-other

Reboot system from init

`1`	`exec /sbin/init 6`

Init system from single user mode

`1`	`exec /sbin/init`

Show current working directory of a process

`1`	`readlink -f /proc/<PID>/cwd`

Show actual pathname of the executed command

`1`	`readlink -f /proc/<PID>/exe`

Tool: curl

`1`	`curl -Iks https://www.google.com`

-I - show response headers only
-k - insecure connection when using ssl
-s - silent mode (not display body)

`1`	`curl -Iks --location -X GET -A "x-agent" https://www.google.com`

--location - follow redirects
-X - set method
-A - set user-agent

`1`	`curl -Iks --location -X GET -A "x-agent" --proxy http://127.0.0.1:16379 https://www.google.com`

--proxy [socks5://|http://] - set proxy server

`1`	`curl -o file.pdf -C - https://example.com/Aiju2goo0Ja2.pdf`

-o - write output to file
-C - resume the transfer

Find your external IP address (external services)

curl ipinfo.io
curl ipinfo.io/ip
curl icanhazip.com
curl ifconfig.me/ip ; echo

Repeat URL request

# URL sequence substitution with a dummy query string:
curl -ks https://example.com/?[1-20]

# With shell 'for' loop:
for i in {1..20} ; do curl -ks https://example.com/ ; done

Check DNS and HTTP trace with headers for specific domains

### Set domains and external dns servers.
_domain_list=(google.com) ; _dns_list=("8.8.8.8" "1.1.1.1")

for _domain in "${_domain_list[@]}" ; do

  printf '=%.0s' {1..48}

  echo

  printf "[\\e[1;32m+\\e[m] resolve: %s\\n" "$_domain"

  for _dns in "${_dns_list[@]}" ; do

    # Resolve domain.
    host "${_domain}" "${_dns}"

    echo

  done

  for _proto in http https ; do

    printf "[\\e[1;32m+\\e[m] trace + headers: %s://%s\\n" "$_proto" "$_domain"

    # Get trace and http headers.
    curl -Iks -A "x-agent" --location "${_proto}://${_domain}"

    echo

  done

done

unset _domain_list _dns_list

Tool: httpie

`1`	`http -p Hh https://www.google.com`

-p - print request and response headers
- H - request headers
- B - request body
- h - response headers
- b - response body

`1`	`http -p Hh https://www.google.com --follow --verify no`

-F, --follow - follow redirects
--verify no - skip SSL verification

1
2

http -p Hh https://www.google.com --follow --verify no \
--proxy http:http://127.0.0.1:16379

--proxy [http:] - set proxy server

Tool: ssh

Escape Sequence

# Supported escape sequences:
~.  - terminate connection (and any multiplexed sessions)
~B  - send a BREAK to the remote system
~C  - open a command line
~R  - Request rekey (SSH protocol 2 only)
~^Z - suspend ssh
~#  - list forwarded connections
~&  - background ssh (when waiting for connections to terminate)
~?  - this message
~~  - send the escape character by typing it twice

Compare a remote file with a local file

`1`	`ssh user@host cat /path/to/remotefile \| diff /path/to/localfile -`

SSH connection through host in the middle

`1`	`ssh -t reachable_host ssh unreachable_host`

Run command over SSH on remote host

cat > cmd.txt << __EOF__
cat /etc/hosts
__EOF__

ssh host -l user $(<cmd.txt)

Get public key from private key

`1`	`ssh-keygen -y -f ~/.ssh/id_rsa`

Get all fingerprints

`1`	`ssh-keygen -l -f .ssh/known_hosts`

SSH authentication with user password

`1`	`ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no user@remote_host`

SSH authentication with publickey

`1`	`ssh -o PreferredAuthentications=publickey -o PubkeyAuthentication=yes -i id_rsa user@remote_host`

Simple recording SSH session

function _ssh_sesslog() {

  _sesdir="<path/to/session/logs>"

  mkdir -p "${_sesdir}" && \
  ssh $@ 2>&1 | tee -a "${_sesdir}/$(date +%Y%m%d).log"

}

# Alias:
alias ssh='_ssh_sesslog'

Using Keychain for SSH logins

### Delete all of ssh-agent's keys.
function _scl() {

  /usr/bin/keychain --clear

}

### Add key to keychain.
function _scg() {

  /usr/bin/keychain /path/to/private-key
  source "$HOME/.keychain/$HOSTNAME-sh"

}

`1`	`ssh -tt user@host bash`

SSH local port forwarding

Example 1:

# Forwarding our local 2250 port to nmap.org:443 from localhost through localhost
host1> ssh -L 2250:nmap.org:443 localhost

# Connect to the service:
host1> curl -Iks --location -X GET https://localhost:2250

Example 2:

# Forwarding our local 9051 port to db.d.x:5432 from localhost through node.d.y
host1> ssh -nNT -L 9051:db.d.x:5432 node.d.y

# Connect to the service:
host1> psql -U db_user -d db_dev -p 9051 -h localhost

-n - redirects stdin from /dev/null
-N - do not execute a remote command
-T - disable pseudo-terminal allocation

SSH remote port forwarding

# Forwarding our local 9051 port to db.d.x:5432 from host2 through node.d.y
host1> ssh -nNT -R 9051:db.d.x:5432 node.d.y

# Connect to the service:
host2> psql -U postgres -d postgres -p 8000 -h localhost

Tool: linux-dev

Testing remote connection to port

`1`	`timeout 1 bash -c "</dev/<proto>/<host>/<port>" >/dev/null 2>&1 ; echo $?`

<proto - set protocol (tcp/udp)
<host> - set remote host
<port> - set destination port

Read and write to TCP or UDP sockets with common bash tools

`1`	`exec 5<>/dev/tcp/<host>/<port>; cat <&5 & cat >&5; exec 5>&-`

Tool: tcpdump

Filter incoming (on interface) traffic (specific ip:port)

`1`	`tcpdump -ne -i eth0 -Q in host 192.168.252.1 and port 443`

-n - don’t convert addresses (-nn will not resolve hostnames or ports)
-e - print the link-level headers
-i [iface|any] - set interface
-Q|-D [in|out|inout] - choose send/receive direction (-D - for old tcpdump versions)
host [ip|hostname] - set host, also [host not]
[and|or] - set logic
port [1-65535] - set port number, also [port not]

Filter incoming (on interface) traffic (specific ip:port) and write to a file

`1`	`tcpdump -ne -i eth0 -Q in host 192.168.252.1 and port 443 -c 5 -w tcpdump.pcap`

-c [num] - capture only num number of packets
-w [filename] - write packets to file, -r [filename] - reading from file

Capture all ICMP packets

`1`	`tcpdump -nei eth0 icmp`

Check protocol used (TCP or UDP) for service

`1`	`tcpdump -nei eth0 tcp port 22 -vv -X \| egrep "TCP\|UDP"`

Display ASCII text (to parse the output using grep or other)

`1`	`tcpdump -i eth0 -A -s0 port 443`

Grab everything between two keywords

`1`	`tcpdump -i eth0 port 80 -X \| sed -n -e '/username/,/=ldap/ p'`

Grab user and pass ever plain http

1
2
3

tcpdump -i eth0  port http -l -A | egrep -i \
'pass=|pwd=|log=|login=|user=|username=|pw=|passw=|passwd=|password=|pass:|user:|username:|password:|login:|pass |user ' \
--color=auto --line-buffered -B20

Extract HTTP User Agent from HTTP request header

`1`	`tcpdump -ei eth0 -nn -A -s1500 -l \| grep "User-Agent:"`

Capture only HTTP GET and POST packets

1
2

tcpdump -ei eth0 -s 0 -A -vv \
'tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420' or 'tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x504f5354'

or simply:

`1`	`tcpdump -ei eth0 -s 0 -v -n -l \| egrep -i "POST /\|GET /\|Host:"`

Rotate capture files

`1`	`tcpdump -ei eth0 -w /tmp/capture-%H.pcap -G 3600 -C 200`

-G <num> - pcap will be created every <num> seconds
-C <size> - close the current pcap and open a new one if is larger than <size>

Top hosts by packets

`1`	`tcpdump -ei enp0s25 -nnn -t -c 200 \| cut -f 1,2,3,4 -d '.' \| sort \| uniq -c \| sort -nr \| head -n 20`

Excludes any RFC 1918 private address

`1`	`tcpdump -nei eth0 'not (src net (10 or 172.16/12 or 192.168/16) and dst net (10 or 172.16/12 or 192.168/16))'`

Tool: tcpick

Analyse packets in real-time

`1`	`while true ; do tcpick -a -C -r dump.pcap ; sleep 2 ; clear ; done`

Tool: ngrep

`1`	`ngrep -d eth0 "www.domain.com" port 443`

-d [iface|any] - set interface
[domain] - set hostname
port [1-65535] - set port number

`1`	`ngrep -d eth0 "www.domain.com" src host 10.240.20.2 and port 443`

(host [ip|hostname]) - filter by ip or hostname
(port [1-65535]) - filter by port number

`1`	`ngrep -d eth0 -qt -O ngrep.pcap "www.domain.com" port 443`

-q - quiet mode (only payloads)
-t - added timestamps
-O [filename] - save output to file, -I [filename] - reading from file

`1`	`ngrep -d eth0 -qt 'HTTP' 'tcp'`

HTTP - show http headers
tcp|udp - set protocol
[src|dst] host [ip|hostname] - set direction for specific node

`1`	`ngrep -l -q -d eth0 -i "User-Agent: curl*"`

-l - stdout line buffered
-i - case-insensitive search

Tool: hping3

`1`	`hping3 -V -p 80 -s 5050 <scan_type> www.google.com`

-V|--verbose - verbose mode
-p|--destport - set destination port
-s|--baseport - set source port
<scan_type> - set scan type
- -F|--fin - set FIN flag, port open if no reply
- -S|--syn - set SYN flag
- -P|--push - set PUSH flag
- -A|--ack - set ACK flag (use when ping is blocked, RST response back if the port is open)
- -U|--urg - set URG flag
- -Y|--ymas - set Y unused flag (0x80 - nullscan), port open if no reply
- -M 0 -UPF - set TCP sequence number and scan type (URG+PUSH+FIN), port open if no reply

`1`	`hping3 -V -c 1 -1 -C 8 www.google.com`

-c [num] - packet count
-1 - set ICMP mode
-C|--icmptype [icmp-num] - set icmp type (default icmp-echo = 8)

`1`	`hping3 -V -c 1000000 -d 120 -S -w 64 -p 80 --flood --rand-source <remote_host>`

--flood - sent packets as fast as possible (don’t show replies)
--rand-source - random source address mode
-d --data - data size
-w|--win - winsize (default 64)

Tool: nmap

Ping scans the network

`1`	`nmap -sP 192.168.0.0/24`

Show only open ports

`1`	`nmap -F --open 192.168.0.0/24`

Full TCP port scan using with service version detection

`1`	`nmap -p 1-65535 -sV -sS -T4 192.168.0.0/24`

Nmap scan and pass output to Nikto

`1`	`nmap -p80,443 192.168.0.0/24 -oG - \| nikto.pl -h -`

Recon specific ip:service with Nmap NSE scripts stack

# Set variables:
_hosts="192.168.250.10"
_ports="80,443"

# Set Nmap NSE scripts stack:
_nmap_nse_scripts="+dns-brute,\
                   +http-auth-finder,\
                   +http-chrono,\
                   +http-cookie-flags,\
                   +http-cors,\
                   +http-cross-domain-policy,\
                   +http-csrf,\
                   +http-dombased-xss,\
                   +http-enum,\
                   +http-errors,\
                   +http-git,\
                   +http-grep,\
                   +http-internal-ip-disclosure,\
                   +http-jsonp-detection,\
                   +http-malware-host,\
                   +http-methods,\
                   +http-passwd,\
                   +http-phpself-xss,\
                   +http-php-version,\
                   +http-robots.txt,\
                   +http-sitemap-generator,\
                   +http-shellshock,\
                   +http-stored-xss,\
                   +http-title,\
                   +http-unsafe-output-escaping,\
                   +http-useragent-tester,\
                   +http-vhosts,\
                   +http-waf-detect,\
                   +http-waf-fingerprint,\
                   +http-xssed,\
                   +traceroute-geolocation.nse,\
                   +ssl-enum-ciphers,\
                   +whois-domain,\
                   +whois-ip"

# Set Nmap NSE script params:
_nmap_nse_scripts_args="dns-brute.domain=${_hosts},http-cross-domain-policy.domain-lookup=true,"
_nmap_nse_scripts_args+="http-waf-detect.aggro,http-waf-detect.detectBodyChanges,"
_nmap_nse_scripts_args+="http-waf-fingerprint.intensive=1"

# Perform scan:
nmap --script="$_nmap_nse_scripts" --script-args="$_nmap_nse_scripts_args" -p "$_ports" "$_hosts"

Tool: netcat

`1`	`nc -kl 5000`

-l - listen for an incoming connection
-k - listening after client has disconnected
>filename.out - save receive data to file (optional)

`1`	`nc 192.168.0.1 5051 < filename.in`

< filename.in - send data to remote host

`1`	`nc -vz 10.240.30.3 5000`

-v - verbose output
-z - scan for listening daemons

`1`	`nc -vzu 10.240.30.3 1-65535`

-u - scan only udp ports

Transfer data file (archive)

1
2

server> nc -l 5000 | tar xzvfp -
client> tar czvfp - /path/to/dir | nc 10.240.30.3 5000

Launch remote shell

# 1)
server> nc -l 5000 -e /bin/bash
client> nc 10.240.30.3 5000

# 2)
server> rm -f /tmp/f; mkfifo /tmp/f
server> cat /tmp/f | /bin/bash -i 2>&1 | nc -l 127.0.0.1 5000 > /tmp/f
client> nc 10.240.30.3 5000

Simple file server

`1`	`while true ; do nc -l 5000 \| tar -xvf - ; done`

Simple minimal HTTP Server

`1`	`while true ; do nc -l -p 1500 -c 'echo -e "HTTP/1.1 200 OK\n\n $(date)"' ; done`

Simple HTTP Server

Restarts web server after each request - remove while condition for only single connection.

cat > index.html << __EOF__
<!doctype html>
    <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
        <title></title>
        <meta name="description" content="">
        <meta name="viewport" content="width=device-width, initial-scale=1">
    </head>
    <body>

    <p>

      Hello! It's a site.

    </p>

    </body>
</html>
__EOF__

server> while : ; do \
(echo -ne "HTTP/1.1 200 OK\r\nContent-Length: $(wc -c <index.html)\r\n\r\n" ; cat index.html;) | \
nc -l -p 5000 \
; done

-p - port number

Simple HTTP Proxy (single connection)

#!/usr/bin/env bash

if [[ $# != 2 ]] ; then
  printf "%s\\n" \
         "usage: ./nc-proxy listen-port bk_host:bk_port"
fi

_listen_port="$1"
_bk_host=$(echo "$2" | cut -d ":" -f1)
_bk_port=$(echo "$2" | cut -d ":" -f2)

printf "  lport: %s\\nbk_host: %s\\nbk_port: %s\\n\\n" \
       "$_listen_port" "$_bk_host" "$_bk_port"

_tmp=$(mktemp -d)
_back="$_tmp/pipe.back"
_sent="$_tmp/pipe.sent"
_recv="$_tmp/pipe.recv"

trap 'rm -rf "$_tmp"' EXIT

mkfifo -m 0600 "$_back" "$_sent" "$_recv"

sed "s/^/=> /" <"$_sent" &
sed "s/^/<=  /" <"$_recv" &

nc -l -p "$_listen_port" <"$_back" | \
tee "$_sent" | \
nc "$_bk_host" "$_bk_port" | \
tee "$_recv" >"$_back"

server> chmod +x nc-proxy && ./nc-proxy 8080 192.168.252.10:8000
  lport: 8080
bk_host: 192.168.252.10
bk_port: 8000

client> http -p h 10.240.30.3:8080
HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: max-age=31536000
Content-Length: 2748
Content-Type: text/html; charset=utf-8
Date: Sun, 01 Jul 2018 20:12:08 GMT
Last-Modified: Sun, 01 Apr 2018 21:53:37 GMT

Create a single-use TCP or UDP proxy

### TCP -> TCP
nc -l -p 2000 -c "nc [ip|hostname] 3000"

### TCP -> UDP
nc -l -p 2000 -c "nc -u [ip|hostname] 3000"

### UDP -> UDP
nc -l -u -p 2000 -c "nc -u [ip|hostname] 3000"

### UDP -> TCP
nc -l -u -p 2000 -c "nc [ip|hostname] 3000"

Tool: gnutls-cli

Testing connection to remote host (with SNI support)

`1`	`gnutls-cli -p 443 google.com`

Testing connection to remote host (without SNI support)

`1`	`gnutls-cli --disable-sni -p 443 google.com`

Tool: socat

Testing remote connection to port

`1`	`socat - TCP4:10.240.30.3:22`

- - standard input (STDIO)
TCP4:<params> - set tcp4 connection with specific params
- [hostname|ip] - set hostname/ip
- [1-65535] - set port number

Redirecting TCP-traffic to a UNIX domain socket under Linux

`1`	`socat TCP-LISTEN:1234,bind=127.0.0.1,reuseaddr,fork,su=nobody,range=127.0.0.0/8 UNIX-CLIENT:/tmp/foo`

TCP-LISTEN:<params> - set tcp listen with specific params
- [1-65535] - set port number
- bind=[hostname|ip] - set bind hostname/ip
- reuseaddr - allows other sockets to bind to an address
- fork - keeps the parent process attempting to produce more connections
- su=nobody - set user
- range=[ip-range] - ip range
UNIX-CLIENT:<params> - communicates with the specified peer socket
- filename - define socket

Tool: p0f

Set iface in promiscuous mode and dump traffic to the log file

`1`	`p0f -i enp0s25 -p -d -o /dump/enp0s25.log`

-i - listen on the specified interface
-p - set interface in promiscuous mode
-d - fork into background
-o - output file

Tool: netstat

Graph # of connections for each hosts

1
2

netstat -an | awk '/ESTABLISHED/ { split($5,ip,":"); if (ip[1] !~ /^$/) print ip[1] }' | \
sort | uniq -c | awk '{ printf("%s\t%s\t",$2,$1) ; for (i = 0; i < $1; i++) {printf("*")}; print "" }'

Monitor open connections for specific port including listen, count and sort it per IP

`1`	`watch "netstat -plan \| grep :443 \| awk {'print \$5'} \| cut -d: -f 1 \| sort \| uniq -c \| sort -nk 1"`

Grab banners from local IPv4 listening ports

`1`	`netstat -nlt \| grep 'tcp ' \| grep -Eo "[1-9][0-9]*" \| xargs -I {} sh -c "echo "" \| nc -v -n -w1 127.0.0.1 {}"`

Tool: rsync

Rsync remote data as root using sudo

`1`	`rsync --rsync-path 'sudo rsync' username@hostname:/path/to/dir/ /local/`

Tool: host

Resolves the domain name (using external dns server)

`1`	`host google.com 9.9.9.9`

Checks the domain administrator (SOA record)

`1`	`host -t soa google.com 9.9.9.9`

Tool: dig

Resolves the domain name (short output)

`1`	`dig google.com +short`

Lookup NS record for specific domain

`1`	`dig @9.9.9.9 google.com NS`

Query only answer section

`1`	`dig google.com +nocomments +noquestion +noauthority +noadditional +nostats`

Query ALL DNS Records

`1`	`dig google.com ANY +noall +answer`

DNS Reverse Look-up

`1`	`dig -x 172.217.16.14 +short`

Tool: certbot

Generate multidomain certificate

`1`	`certbot certonly -d example.com -d www.example.com`

Generate wildcard certificate

`1`	`certbot certonly --manual --preferred-challenges=dns -d example.com -d *.example.com`

Generate certificate with 4096 bit private key

`1`	`certbot certonly -d example.com -d www.example.com --rsa-key-size 4096`

Tool: network-other

Get all subnets for specific AS (Autonomous system)

AS="AS32934"
whois -h whois.radb.net -- "-i origin ${AS}" | \
grep "^route:" | \
cut -d ":" -f2 | \
sed -e 's/^[ \t]//' | \
sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4 | \
cut -d ":" -f2 | \
sed -e 's/^[ \t]/allow /' | \
sed 's/$/;/' | \
sed 's/allow  */subnet -> /g'

Resolves domain name from dns.google.com with curl and jq

`1`	`_dname="google.com" ; curl -s "https://dns.google.com/resolve?name=${_dname}&type=A" \| jq .`

Tool: git

Log alias for a decent view of your repo

# 1)
git log --oneline --decorate --graph --all

# 2)
git log --graph \
--pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' \
--abbrev-commit

Tool: python

Static HTTP web server

# Python 3.x
python3 -m http.server 8000 --bind 127.0.0.1

# Python 2.x
python -m SimpleHTTPServer 8000

Static HTTP web server with SSL support

# Python 3.x
from http.server import HTTPServer, BaseHTTPRequestHandler
import ssl

httpd = HTTPServer(('localhost', 4443), BaseHTTPRequestHandler)

httpd.socket = ssl.wrap_socket (httpd.socket,
        keyfile="path/to/key.pem",
        certfile='path/to/cert.pem', server_side=True)

httpd.serve_forever()

# Python 2.x
import BaseHTTPServer, SimpleHTTPServer
import ssl

httpd = BaseHTTPServer.HTTPServer(('localhost', 4443),
        SimpleHTTPServer.SimpleHTTPRequestHandler)

httpd.socket = ssl.wrap_socket (httpd.socket,
        keyfile="path/tp/key.pem",
        certfile='path/to/cert.pem', server_side=True)

httpd.serve_forever()

Encode base64

`1`	`python -m base64 -e <<< "sample string"`

Decode base64

`1`	`python -m base64 -d <<< "dGhpcyBpcyBlbmNvZGVkCg=="`

Tool: awk

Search for matching lines

1
2

# egrep foo
awk '/foo/' filename

Search non matching lines

1
2

# egrep -v foo
awk '!/foo/' filename

Print matching lines with numbers

1
2

# egrep -n foo
awk '/foo/{print FNR,$0}' filename

Print the last column

`1`	`awk '{print $NF}' filename`

Find all the lines longer than 80 characters

`1`	`awk 'length($0)>80{print FNR,$0}' filename`

Print only lines of less than 80 characters

`1`	`awk 'length < 80' filename`

Print double new lines a file

`1`	`awk '1; { print "" }' filename`

Print line numbers

1
2

awk '{ print FNR "\t" $0 }' filename
awk '{ printf("%5d : %s\n", NR, $0) }' filename   # in a fancy manner

Print line numbers for only non-blank lines

`1`	`awk 'NF { $0=++a " :" $0 }; { print }' filename`

Print the line and the next two (i=5) lines after the line matching regexp

`1`	`awk '/foo/{i=5+1;}{if(i){i--; print;}}' filename`

Print the lines starting at the line matching ‘server {’ until the line matching ‘}’

`1`	`awk '/server {/,/}/' filename`

Print multiple columns with separators

`1`	`awk -F' ' '{print "ip:\t" $2 "\n port:\t" $3' filename`

Remove empty lines

awk 'NF > 0' filename

# alternative:
awk NF filename

Delete trailing white space (spaces, tabs)

`1`	`awk '{sub(/[ \t]*$/, "");print}' filename`

Delete leading white space

`1`	`awk '{sub(/^[ \t]+/, ""); print}' filename`

Remove duplicate consecutive lines

1
2

# uniq
awk 'a !~ $0{print}; {a=$0}' filename

Remove duplicate entries in a file without sorting

`1`	`awk '!x[$0]++' filename`

Exclude multiple columns

`1`	`awk '{$1=$3=""}1' filename`

Substitute foo for bar on lines matching regexp

`1`	`awk '/regexp/{gsub(/foo/, "bar")};{print}' filename`

Add some characters at the beginning of matching lines

`1`	`awk '/regexp/{sub(/^/, "++++"); print;next;}{print}' filename`

Get the last hour of Apache logs

1
2

awk '/'$(date -d "1 hours ago" "+%d\\/%b\\/%Y:%H:%M")'/,/'$(date "+%d\\/%b\\/%Y:%H:%M")'/ { print $0 }' \
/var/log/httpd/access_log

Tool: sed

Print a specific line from a file

`1`	`sed -n 10p /path/to/file`

Remove a specific line from a file

1
2

sed -i 10d /path/to/file
# alternative (BSD): sed -i'' 10d /path/to/file

Remove a range of lines from a file

`1`	`sed -i <file> -re '<start>,<end>d'`

Replace newline(s) with a space

sed ':a;N;$!ba;s/\n/ /g' /path/to/file

# cross-platform compatible syntax:
sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/ /g' /path/to/file

:a create a label a
N append the next line to the pattern space
$! if not the last line, ba branch (go to) label a
s substitute, /\n/ regex for new line, / / by a space, /g global match (as many times as it can)

Alternatives:

# perl version (sed-like speed):
perl -p -e 's/\n/ /' /path/to/file

# bash version (slow):
while read line ; do printf "%s" "$line " ; done < file

Delete string +N next lines

`1`	`sed '/start/,+4d' /path/to/file`

Tool: grep

Search for a “pattern” inside all files in the current directory

1
2
3

grep -rn "pattern"
grep -RnisI "pattern" *
fgrep "pattern" * -R

Show only for multiple patterns

grep 'INFO*'\''WARN' filename
grep 'INFO\|WARN' filename
grep -e INFO -e WARN filename
grep -E '(INFO|WARN)' filename
egrep "INFO|WARN" filename

Except multiple patterns

`1`	`grep -vE '(error\|critical\|warning)' filename`

Show data from file without comments

`1`	`grep -v ^[[:space:]]*# filename`

Show data from file without comments and new lines

`1`	`egrep -v '#\|^$' filename`

Show strings with a dash/hyphen

1
2
3

grep -e -- filename
grep -- -- filename
grep "\-\-" filename

Remove blank lines from a file and save output to new file

`1`	`grep . filename > newfilename`

Tool: perl

Search and replace (in place)

`1`	`perl -i -pe's/SEARCH/REPLACE/' filename`

Edit of `*.conf` files changing all foo to bar (and backup original)

`1`	`perl -p -i.orig -e 's/\bfoo\b/bar/g' *.conf`

Prints the first 20 lines from `*.conf` files

`1`	`perl -pe 'exit if $. > 20' *.conf`

Search lines 10 to 20

`1`	`perl -ne 'print if 10 .. 20' filename`

Delete first 10 lines (and backup original)

`1`	`perl -i.orig -ne 'print unless 1 .. 10' filename`

Delete all but lines between foo and bar (and backup original)

`1`	`perl -i.orig -ne 'print unless /^foo$/ .. /^bar$/' filename`

Reduce multiple blank lines to a single line

`1`	`perl -p -i -00pe0 filename`

Convert tabs to spaces (1t = 2sp)

`1`	`perl -p -i -e 's/\t/ /g' filename`

Read input from a file and report number of lines and characters

`1`	`perl -lne '$i++; $in += length($_); END { print "$i lines, $in characters"; }' filename`

Shell Tricks ^[TOC]

When you get a shell, it is generally not very clean, but after following these steps, you will have a fairly clean and comfortable shell to work with.

script /dev/null -c bash
Ctrl-Z (to send it to background)
stty raw -echo; fg (returns the shell to foreground)
reset (to reset terminal)
xterm (when asked for terminal type)
export TERM=xterm; export SHELL=bash

Shell functions ^[TOC]

Domain resolve

# Dependencies:
#   - curl
#   - jq

function DomainResolve() {

  local _host="$1"

  local _curl_base="curl --request GET"
  local _timeout="15"

  _host_ip=$($_curl_base -ks -m "$_timeout" "https://dns.google.com/resolve?name=${_host}&type=A" | \
  jq '.Answer[0].data' | tr -d "\"" 2>/dev/null)

  if [[ -z "$_host_ip" ]] || [[ "$_host_ip" == "null" ]] ; then

    echo -en "Unsuccessful domain name resolution.\\n"

  else

    echo -en "$_host > $_host_ip\\n"

  fi

}

Example:

shell> DomainResolve nmap.org
nmap.org > 45.33.49.119

shell> DomainResolve nmap.org
Unsuccessful domain name resolution.

Get ASN

# Dependencies:
#   - curl

function GetASN() {

  local _ip="$1"

  local _curl_base="curl --request GET"
  local _timeout="15"

  _asn=$($_curl_base -ks -m "$_timeout" "http://ip-api.com/line/${_ip}?fields=as")

  _state=$(echo $?)

  if [[ -z "$_ip" ]] || [[ "$_ip" == "null" ]] || [[ "$_state" -ne 0 ]]; then

    echo -en "Unsuccessful ASN gathering.\\n"

  else

    echo -en "$_ip > $_asn\\n"

  fi

}

Example:

shell> GetASN 1.1.1.1
1.1.1.1 > AS13335 Cloudflare, Inc.

shell> GetASN 0.0.0.0
Unsuccessful ASN gathering.

Self-Hosting-Guide

Fri, 27 Jun 2025 15:31:40 +0800

mikeroyal/Self-Hosting-Guide

Self Hosting Guide

A guide for getting started with Self Hosting devices including software and hardware that will make you a better and more efficient Self Hosting.

Note: You can easily convert this markdown file to a PDF in VSCode using this handy extension Markdown PDF.

Note 2: This guide will constantly be updated with new info as becomes available and please feel to make an issue if you think something should be added.

Getting Started with Self-Hosting

Back to the Top

Self-Hosting is the practice of locally hosting(on premises & private web servers) and managing software applications by a person or organization instead of monthly subscriptions from Software as a service (SaaS) providers.

Most self-hosted software can be installed using Docker, a packaging system which allows software to bundle their configuration and dependencies and isolate them from your operating system. Software using docker can be installed using the command line or via graphical interfaces such as Portainer. Software is installed with Docker by downloading an image file containing the application, then creating a copy that sets up its own dependencies and configuration within what is called a container. Without containers you would often need to install different versions of the same programming languages or tools to satisfy the dependencies for the software you want to use which can get complicated.

Tools for Self-Hosting

Back to the Top

Containers

Back to the Top

Container is a standard unit of software that packages up code and all its dependencies(including CPU, memory, file storage, and network connections) so the application runs quickly and reliably from one computing environment to another.

Application Container Security Guide | NIST (PDF)

Container Image is a lightweight, standalone, executable package of software that includes everything needed to run an application such as the code, runtime, system tools, system libraries, and settings.

Best places to get Container Images:

Docker Compose is a tool that was developed to help define and share multi-container applications. With Compose, we can create a YAML file to define the services and with a single command, can spin everything up or tear it all down.

Docker Include is a Compose application can declare dependency on another Compose application. This is useful if you want to reuse other Compose files. Also, if you need to factor out parts of your application model into separate Compose files so they can be managed separately or shared with others.

Kompose is a conversion tool for Docker Compose to container orchestrators such as Kubernetes or OpenShift.

SwarmKit is a toolkit for orchestrating distributed systems at any scale. It includes primitives for node discovery, raft-based consensus, task scheduling and more.

Containerd is a daemon that manages the complete container lifecycle of its host system, from image transfer and storage to container execution and supervision to low-level storage to network attachments and beyond. It is available for Linux and Windows.

ContainersSSH is an SSH Server that Launches Containers in Kubernetes and Docker on demand.

Podman is a daemonless, open source, Linux native tool designed to make it easy to find, run, build, share and deploy applications using Open Containers Initiative (OCI) Containers and Container Images. Podman provides a command line interface (CLI) familiar to anyone who has used the Docker Container Engine.

Lima is a tool that launches Linux virtual machines with automatic file sharing and port forwarding (similar to WSL2), and containerd. It’s a great free and open-source alternative for Docker Desktop.

Colima is a container runtimes on macOS (and Linux) with minimal setup.

Portainer Community Edition is a lightweight service delivery platform for containerized applications that can be used to manage Docker, Swarm, Kubernetes and ACI environments. It is designed to be as simple to deploy as it is to use.

Yacht is a container management UI with a focus on templates and 1-click deployments.

Kitematic is a simple application for managing Docker containers on Mac, Linux and Windows letting you control your app containers from a graphical user interface (GUI).

HashiCorp Nomad is a simple and flexible scheduler and orchestrator to deploy and manage containers and non-containerized applications across on-premises and clouds at scale.

Open Container Initiative is an open governance structure for the express purpose of creating open industry standards around container formats and runtimes.

OpenNebula is an open source platform delivering a simple but feature-rich and flexible solution to build and manage enterprise clouds for virtualized services, containerized applications and serverless computing.

Buildah is a command line tool to build Open Container Initiative (OCI) images. It can be used with Docker, Podman, Kubernetes.

Red Hat Universal Base Images (UBI) is a tool that offers a way to build your container images on a foundation of Red Hat Enterprise Linux software. They are OCI-compliant, container-based, operating system images with complementary runtime languages and packages that are freely redistributable. Easily find UBI images in the Red Hat container catalog, and they are buildable and deployable anywhere.

Red Hat Quay is a project that Builds, Stores, and Distributes your Applications and Containers.

ctop is a tool that provides a concise and condensed overview of real-time metrics for multiple containers as well as a single container view for inspecting a specific container. It comes with built-in support for Docker and runC; connectors for other container and cluster systems are planned for future releases.

runc is a CLI tool for spawning and running containers on Linux according to the OCI specification.

container-images is a collection of container images used in CI across various opencontainers projects.

Clair is an open source project for the static analysis of vulnerabilities in application containers (currently including OCI and Docker).

Shipwright is a WebUI to generate templates for Yacht, Portainer, Docker-Compose, and Unraid.

Alnoda Workspaces is an open-source portable containerized browser-based development environments in Docker containers. You can create your own custom workspace or customize any of the workspaces with your preferred stack of applications without knowing much of the Docker.

Autoheal monitors and restarts unhealthy docker containers.

Dozzle is a small lightweight application with a web based interface to monitor Docker logs. It doesn’t store any log files. It is for live monitoring of your container logs only.

Diun is a tool that receive notifications when a Docker image is updated on a Docker registry.

WatchTower is a process for automating Docker container base image updates.

Kasm Workspaces is a a highly configurable container streaming platform that enables you to stream and deliver containerized applications over the web. It offers tools that you can use to create desktop workspaces and provide access to virtual desktops to end users. It also ensures data loss prevention as well as secure and private web browsing.

Nginx Proxy is an automation tool that sets up a container running nginx and docker-gen. Docker-gen generates reverse proxy configs for nginx and reloads nginx when containers are started and stopped.

Visual Studio Code Dev Containers is an extension that lets you use a Docker container as a full-featured development environment. It allows you to open any folder inside (or mounted into) a container and take advantage of Visual Studio Code’s full feature set. A devcontainer.json file in your project tells VS Code how to access (or create) a development container with a well-defined tool and runtime stack.

CI/CD

Back to the Top

CI/CD: Continuous Integration and Continuous Delivery

Drone is a Continuous Delivery system built on container technology. Drone uses a simple YAML configuration file, a superset of docker-compose, to define and execute Pipelines inside Docker containers.

Woodpecker is a CI service, a community fork of Drone.

Travis CI is a hosted continuous integration service used to build and test software projects hosted at GitHub.

Circle CI is a continuous integration and continuous delivery platform that helps software teams work smarter, faster.

Buddy is a fully-featured DevOps platform with no learning curve that packs everything you need from a CI/CD tool.

Buildbot is a continuous integration tool which automates the compile or test cycle required to validate changes to the project code base. It queues jobs, executes the jobs when the required resources are available, and reports the results.

Development

Back to the Top

Proxmox VE(Virtual Environment) is an open-source platform for enterprise virtualization. It has a built-in web interface that you can use to easily manage VMs and containers, software-defined storage and networking, high-availability clustering, and multiple out-of-the-box tools on a single solution.

Terraform provider plugin for Proxmox is a Terraform provider for the Proxmox virtualization platform and exposes Terraform resources to provision QEMU VMs and LXC Containers.

OTF is an open source alternative to Terraform Enterprise. Includes SSO, team management, agents, and as many applies as you can throw hardware at.

Semaphore UI is a modern UI for Ansible. It lets you easily run Ansible playbooks, get notifications about fails, control access to deployment system.

APITable is an API-oriented low-code platform for building collaborative apps and better than all other Airtable open-source alternatives.

Chisel Kubernetes Operator is a Kubernetes operator for Chisel. It allows you to use Chisel as a LoadBalancer provider for your Kubernetes cluster, similar to inlets-operator.

Docker-pgautoupgrade is a PostgreSQL Docker container that automatically upgrades your database. It’s whole purpose in life is to automatically detect the version of PostgreSQL used in the existing PostgreSQL data directory, and automatically upgrade it (if needed) to the required version of PostgreSQL.

IT-Tools is a collection of handy online tools for developers, with great UX.

Lazygit is a simple terminal UI for git commands, written in Go with the gocui library.

LazyDocker is a simple terminal UI for both docker and docker-compose, written in Go with the gocui library.

Code-Server is Visual Studio Code running on a remote server, accessible through the browser.

Turbopilot is an open source large-language-model based code completion engine that runs locally on your CPU.

Self-Hosted Sentry nightly is an official bootstrap for running your own Sentry with Docker. Sentry, feature-complete and packaged up for low-volume deployments and proofs-of-concept.

Visual Studio Live Share is a service/extension that enables you to collaboratively edit and debug with others in real time, regardless of the programming languages you’re using or app types you’re building. You can instantly and securely share your current project, start a joint debugging session, share terminal instances, forward localhost web apps, have voice calls, and more.

GistPad is a Visual Studio Code extension that allows you to edit GitHub Gists and repositories from the comfort of your favorite editor. You can open, create, delete, fork and star gists and repositories, and then seamlessly begin editing files as if they were local, without ever cloning, pushing or pulling anything.

Live Server is an extension for Visual Studio Code that launches a development local Server with live reload feature for static & dynamic pages.

Gitea is a community managed painless self-hosted Git service.

Act is a a tool to run your GitHub Actions locally.

Act runner is a runner for Gitea based on act.

GitLab is an open source end-to-end software development platform with built-in version control, issue tracking, code review, CI/CD, and more. Self-host GitLab on your own servers, in a container, or on a cloud provider.

Bonobo Git Server - Set up your own self hosted git server on IIS for Windows. Manage users and have full control over your repositories with a nice user friendly graphical interface.

Fossil - Distributed version control system featuring wiki and bug tracker.

Gerrit - A code review and project management tool for Git based projects.

Gitblit - Pure Java stack for managing, viewing, and serving Git repositories.

gitbucket - Easily installable GitHub clone powered by Scala.

Gitea - Community managed fork of Gogs, lightweight code hosting solution.

Gitlist - Web-based git repository browser - GitList allows you to browse repositories using your favorite browser, viewing files under different revisions, commit history and diffs.

Gitolite - Gitolite allows you to setup git hosting on a central server, with fine-grained access control and many more powerful features.

GitPrep - Portable Github clone.

Gogs - Painless self-hosted Git Service written in Go.

Kallithea - Source code management system that supports two leading version control systems, Mercurial and Git, with a web interface.

Klaus - Simple, easy-to-set-up Git web viewer that Just Works.

Lavagna - Lavagna is an open-source issue/project management tool designed for small teams. Lightweight, pure Java, easy to install, easy to use.

Leantime - Leantime is a lean project management system for small teams and startups helping to manage projects from ideation through delivery.

Taiga is an open-source project management software for cross-functional teams that work agile across both scrum and kanban frameworks.

Planka is a realtime kanban board for workgroups built with React and Redux.

Microgit - Git hosting service made in Crystal and Lucky. MIT Crystal

OneDev - All-In-One DevOps Platform. With Git Management, Issue Tracking, and CI/CD. Simple yet Powerful.

OpenProject - OpenProject is a web-based project management system.

Pagure - A lightweight, powerful, and flexible git-centric forge with features laying the foundation for federated and decentralized development.

Phorge - Phorge is an open source, community driven platform for collaborating, managing, organizing and reviewing software development projects.

Redmine - Redmine is a flexible project management web application. (Demo,

RhodeCode - RhodeCode is an open source platform for software development teams. It unifies and simplifies repository management for Git, Subversion, and Mercurial.

SCM Manager - The easiest way to share and manage your Git, Mercurial and Subversion repositories over http.

Taiga - Agile Project Management Tool based on the Kanban and Scrum methods.

Titra - Time-tracking solution for freelancers and small teams.

Traq - Project management and issue tracking system written in PHP.

Tuleap - Tuleap is a libre suite to plan, track, code and collaborate on software projects.

UVDesk - UVDesk community is a service oriented, event driven extensible opensource helpdesk system that can be used by your organization to provide efficient support to your clients effortlessly whichever way you imagine.

ZenTao - An agile(scrum) project management system/tool.

k3s-ansible is the easiest way to bootstrap a self-hosted High Availability Kubernetes cluster. A fully automated HA k3s etcd install with kube-vip, MetalLB, and more.

Soft Serve is a tasty, self-hostable Git server for the command line.

Coolify is an open-source & self-hostable Heroku/Netlify alternative.

Corosync Cluster Engine is a Group Communication System with additional features for implementing high availability within applications.

Glow is a terminal based markdown reader designed from the ground up to bring out the beauty—and power—of the CLI. It’s used to discover markdown files, read documentation directly on the command line and stash markdown files to your own private collection, so you can read them anywhere.

Deep Lake is a data lake for deep learning applications. Our open-source dataset format is optimized for rapid streaming and querying of data while training models at scale, and it includes a simple API for creating, storing, and collaborating on AI datasets of any size. It can be deployed locally or in the cloud, and it enables you to store all of your data in one place, ranging from simple annotations to large videos.

Node-Red is a low-code programming for event-driven applications.

krunvm is a CLI-based utility for creating microVMs from OCI images, using libkrun and buildah.

Zeal is an offline documentation browser for software developers inspired by Dash.

Web servers

Back to The Top

Web servers

Apache - Most popular web server.

OpenResty Manager - The easiest using, powerful and beautiful OpenResty Manager(Nginx Enhanced Version), open source alternative to OpenResty Edge.

Beakon - A self-host location sharing webserver. Beakon aims to leak as little data as possible and uses mostly self-contained libraries and local database files. Where possible, it will reference local files and not reach out over any network.

Caddy - The HTTP/2 Web Server with Fully Managed TLS.

Cherokee - Lightweight, high-performance web server/reverse proxy.

Lighttpd - Web server more optimized for speed-critical environments.

Nginx - Reverse proxy, load balancer, HTTP cache, and web server.

uWSGI - The uWSGI project aims at developing a full stack for building hosting services.

Web Performance

HAProxy - Software based load Balancing, SSL offloading and performance optimization, compression, and general web routing.

Squid - Caching proxy for the web supporting HTTP, HTTPS, FTP, and more.

Traefik - Taefik is a modern HTTP reverse proxy and load balancer made to deploy microservices with ease.

Varnish - HTTP based web application accelerator focusing on optimizing caching and compression.

LLMs

Back to the Top

Large Language Models (LLMs) is a language model that uses artificial neural networks to generate text (AI chatbots/search engines). Some notable ones are GPT-3, GPT-4, BLOOM, and LLaMA.

llama.cpp is a Port of Facebook’s LLaMA model in C/C++.

ollama is a tool to get up and running with Llama 2 and other large language models locally.

LocalAI is a self-hosted, community-driven, local OpenAI-compatible API. Drop-in replacement for OpenAI running LLMs on consumer-grade hardware with no GPU required. It’s an API to run ggml compatible models: llama, gpt4all, rwkv, whisper, vicuna, koala, gpt4all-j, cerebras, falcon, dolly, starcoder, and many others.

Serge is a web interface for chatting with Alpaca through llama.cpp. Fully self-hosted & dockerized, with an easy to use API.

OpenLLM is an open platform for operating large language models (LLMs) in production. Fine-tune, serve, deploy, and monitor any LLMs with ease.

Llama-gpt is a self-hosted, offline, ChatGPT-like chatbot. Powered by Llama 2. 100% private, with no data leaving your device.

Llama2 webui is a tool to run any Llama 2 locally with gradio UI on GPU or CPU from anywhere (Linux/Windows/Mac). Use llama2-wrapper as your local llama2 backend for Generative Agents/Apps.

Llama2.c is a tool to Train the Llama 2 LLM architecture in PyTorch then inference it with one simple 700-line C file (run.c).

Alpaca.cpp is a fast ChatGPT-like model locally on your device. It combines the LLaMA foundation model with an open reproduction of Stanford Alpaca a fine-tuning of the base model to obey instructions (akin to the RLHF used to train ChatGPT) and a set of modifications to llama.cpp to add a chat interface.

GPT4All is an ecosystem of open-source chatbots trained on a massive collections of clean assistant data including code, stories and dialogue based on LLaMa.

MiniGPT-4 is an enhancing Vision-language Understanding with Advanced Large Language Models

LoLLMS WebUI is a the hub for LLM (Large Language Model) models. It aims to provide a user-friendly interface to access and utilize various LLM models for a wide range of tasks. Whether you need help with writing, coding, organizing data, generating images, or seeking answers to your questions.

LM Studio is a tool to Discover, download, and run local LLMs.

Ava PLS small, all-in-one desktop app to run LLMs locally.

Gradio Web UI is a tool for Large Language Models. Supports transformers, GPTQ, llama.cpp (ggml/gguf), Llama models.

OpenPlayground is a playfround for running ChatGPT-like models locally on your device.

Vicuna is an open source chatbot trained by fine tuning LLaMA. It apparently achieves more than 90% quality of chatgpt and costs $300 to train.

Yeagar ai is a Langchain Agent creator designed to help you build, prototype, and deploy AI-powered agents with ease.

KoboldCpp is an easy-to-use AI text-generation software for GGML models. It’s a single self contained distributable from Concedo, that builds off llama.cpp, and adds a versatile Kobold API endpoint, additional format support, backward compatibility, as well as a fancy UI with persistent stories, editing tools, save formats, memory, world info, author’s note, characters, and scenarios.

Minima is a configurable conversational RAG system that runs LLM locally and on-premises using containers.

ChatGPT

Back to the Top

Chatbot UI for ChatGPT

Chatbot UI by mckaywrigley is an advanced chatbot kit for OpenAI’s chat models built on top of Chatbot UI Lite using Next.js, TypeScript, and Tailwind CSS. This version of ChatBot UI supports both GPT-3.5 and GPT-4 models. Conversations are stored locally within your browser. You can export and import conversations to safeguard against data loss. See a demo.

Chatbot UI Lite by mckaywrigley is a simple chatbot starter kit for OpenAI’s chat model using Next.js, TypeScript, and Tailwind CSS. See a demo.

Build locally with Docker:

1
2

docker build -t chatgpt-ui .
docker run -e OPENAI_API_KEY=xxxxxxxx -p 3000:3000 chatgpt-ui

Running Locally on Windows, MacOS, and Linux:

1. Clone Project Repo

`1`	`git clone https://github.com/mckaywrigley/chatbot-ui.git`

2. Install Dependencies

npm i

3. Provide OpenAI API Key

Create a .env.local file in the root of the repo with your OpenAI API Key:

`1`	`OPENAI_API_KEY=YOUR_KEY`

You can set OPENAI_API_HOST where access to the official OpenAI host is restricted or unavailable, allowing users to configure an alternative host for their specific needs.
Additionally, if you have multiple OpenAI Organizations, you can set OPENAI_ORGANIZATION to specify one.

4. Run App

`1`	`npm run dev`

You done you should be able to start chatting with ChatGPT!

Chatbot UI

MiniGPT-4 is an enhancing Vision-language Understanding with Advanced Large Language Models

Launching Demo Locally

Try out the demo demo.py on your local machine by running

python demo.py --cfg-path eval_configs/minigpt4_eval.yaml --gpu-id 0

Here, the demo loads Vicuna as 8 bit by default to save some GPU memory usage. Besides, the default beam search width is 1. Under this setting, the demo cost about 23G GPU memory. If you have a more powerful GPU with larger GPU memory, you can run the model in 16 bit by setting low_resource to False in the config file minigpt4_eval.yaml and use a larger beam search width.

MiniGPT-4 Demo

GPT4All is an ecosystem of open-source chatbots trained on a massive collections of clean assistant data including code, stories and dialogue based on LLaMa.

GPT4All UI is a Flask web application that provides a chat UI for interacting with the GPT4All chatbot.

llama.cpp is a Port of Facebook’s LLaMA model in C/C++.

Serge is a web interface for chatting with Alpaca through llama.cpp. Fully self-hosted & dockerized, with an easy to use API.

OpenPlayground is a playfround for running ChatGPT-like models locally on your device.

Vicuna is an open source chatbot trained by fine tuning LLaMA. It apparently achieves more than 90% quality of chatgpt and costs $300 to train.

Yeagar ai is a Langchain Agent creator designed to help you build, prototype, and deploy AI-powered agents with ease.

DoctorGPT is a lightweight self-contained binary that monitors your application logs for problems and diagnoses them.

HttpGPT is an Unreal Engine 5 plugin that facilitates integration with OpenAI’s GPT based services (ChatGPT and DALL-E) through asynchronous REST requests, making it easy for developers to communicate with these services. It also includes Editor Tools to integrate Chat GPT and DALL-E image generation directly in the Engine.

Automation

Back to the Top

Accelerated Text - Automatically generate multiple natural language descriptions of your data varying in wording and structure.

Activepieces - No-code business automation tool like Zapier or Tray. For example, you can send a Slack notification for each new Trello card.

ActiveWorkflow - An intelligent process and workflow automation platform based on software agents.

Alltube - Web GUI for youtube-dl, a program to download videos and audio from more than 100 websites.

AmIUnique - Learn how identifiable you are on the Internet (browser fingerprinting tool).

Automatisch - Business automation tool that lets you connect different services like Twitter, Slack, and more to automate your business processes (Open source Zapier alternative).

Baserow - Open source online database tool and Airtable alternative. Create your own database without technical experience.

betanin - Music organization man-in-the-middle of your torrent client and music player. Based on beets.io, similar to Sonarr and Radarr.

ChiefOnboarding - Employee onboarding platform that allows you to provision user accounts and create sequences with todo items, resources, text/email/Slack messages, and more! Available as a web portal and Slack bot.

Datasette - An open source multi-tool for exploring and publishing data, easy import and export and database management.

Eonza - Eonza is used to create scripts and automate tasks on servers or VPS hosting. Manage your servers from any browser on any device.

Exadel CompreFace - Face recognition system that provides REST API for face recognition, face detection, and other face services, and is easily deployed with docker. There are SDKs for Python and JavaScript languages. Can be used without prior machine learning skills.

feed2toot - Feed2toot parses a RSS feed, extracts the last entries and sends them to Mastodon.

feedmixer - FeedMixer is a WSGI (Python3) micro web service which takes a list of feed URLs and returns a new feed consisting of the most recent n entries from each given feed(Returns Atom, RSS, or JSON).

Headphones - Automated music downloader for NZB and Torrent, written in Python. It supports SABnzbd, NZBget, Transmission, µTorrent, Deluge and Blackhole.

Healthchecks - Django app which listens for pings and sends alerts when pings are late.

HRConvert2 - Drag-and-drop file conversion server with session based authentication, automatic temporary file maintenance, and logging capability.

Huginn - Allows you to build agents that monitor and act on your behalf.

Kibitzr - Lightweight personal web assistant with powerful integrations.

Krayin - Free and Opensource Laravel CRM Application.

Leon - Open-source personal assistant who can live on your server.

Lidarr - Lidarr is a music collection manager for Usenet and BitTorrent users.

Matchering - A containerized web app for automated music mastering. An open-source alternative to LANDR, eMastered, and MajorDecibel.

Medusa - Automatic Video Library Manager for TV Shows. It watches for new episodes of your favorite shows, and when they are posted it does its magic. (Source Code) GPL-3.0 Python

MeTube - Web GUI for youtube-dl, with playlist support. Allows downloading videos from dozens of websites. AGPL-3.0 Python/Nodejs/Docker

Nautobot is a Network Source of Truth and Network Automation Platform built as a web application atop the Django Python framework with a PostgreSQL or MySQL database.

nefarious - Web application that automates downloading Movies and TV Shows.

NocoDB - No-code platform that turns any database into a smart spreadsheet. It can be considered as an Airtable or Smartsheet alternative.

OliveTin - OliveTin is a web interface for running Linux shell commands.

Patrowl - Open Source, Smart and Scalable Security Operations Orchestration Platform.

Podgrab - Lightweight podcast manager and automatic podcast episode downloader. It will monitor podcasts for your and download them automatically whenever a new episode goes live.

pyLoad - Lightweight, customizable and remotely manageable downloader for 1-click-hosting sites like rapidshare.com or uploaded.to.

Radarr - Radarr is an independent fork of Sonarr reworked for automatically downloading movies via Usenet and BitTorrent, à la Couchpotato.

SickRage - SickRage is an automatic Video Library Manager for TV Shows. Automatic torrent/nzb searching, downloading, and processing at the qualities you want.

SiteInspector - Web-based tool for catching spelling errors, grammatical errors, broken links, and other errors on websites.

Sonarr - Automatic TV Shows downloader and manager for Usenet and BitTorrent. It can grab, sort and rename new episodes and automatically upgrade the quality of files already downloaded when a better quality format becomes available.

StackStorm - StackStorm (aka IFTTT for Ops) is event-driven automation for auto-remediation, security responses, troubleshooting, deployments, and more. Includes rules engine, workflow, 160 integration packs with 6000+ actions and ChatOps.

µTask - Automation engine that models and executes business processes declared in yaml.

Configuration Management

Back to The Top

Ansible - is a tool is a powerful, agentless tool that works everywhere and with everything. When you add in proven enterprise engineering and support from Red Hat that’s written in Python.

Ansible.Ai is an AI for Ansible Content Development tool to automate in your IT infrastructure and it will generate syntactically correct playbook to help you get there.

CFEngine - is a Lightweight agent system where the configuration state is specified via a declarative language.

mgmt - is a next generation config management written in Go.

Pallet - is a Infrastructure definition, configuration and management via a Clojure DSL.

Puppet - is an automated administrative engine for your Linux, Unix, and Windows systems, performs administrative tasks (such as adding users, installing packages, and updating server configurations) based on a centralized specification.

Chef - is a powerful automation platform that transforms infrastructure into code automating how infrastructure is configured, deployed and managed across any environment.

(R)?ex - is a friendly automation framework to any combinations of local and remote execution, push and pull style of management, or imperative and declarative approach.

Salt - is an event-driven automation tool and framework to deploy, configure, and manage complex IT systems. It automates common infrastructure administration tasks and ensure that all the components of your infrastructure are operating in a consistent desired state.

Fleek is an all-in-one management system for everything you need to be productive on your computer.

Cloud Storage

Back to The Top

Swift - A highly available, distributed, eventually consistent object/blob store.

Syncthing - Open Source system for private, encrypted and authenticated distribution of data.

git-annex assistant - A synchronized folder on each of your MacOS and Linux computers, Android devices, removable drives, NAS appliances, and cloud services.

NextCloud - Provides access to your files via the web.

ownCloud - Provides universal access to your files via the web, your computer or your mobile devices.

Seafile - Another Open Source Cloud Storage solution.

SparkleShare - Provides cloud storage and file synchronization services. By default, it uses Git as a storage backend.

Cloud

Back to the Top

Linode

Linode is a cloud hosting company that provides virtual private servers and variety of other cloud services.

Linode Tools

Linode Cloud Manager is a user- and mobile-friendly interface to deploy and manage virtual machines, configure networking, and control user accounts.

Linode API is a tool that makes easy to configure, manage, and deploy user management, billing, support tickets, and more with programmatic access to Linode products and services.

Linode CLI is a tool to deploy and manage Linux servers from Linode without leaving the command line.

Linode Images is a service to capture, store, and deploy your custom images across Linodes or data centers. Easily create your own raw disk image and upload a compressed .gz image file (up to 5 GB) using the Cloud Manager or API to easily deploy to the Linode size and data center you need.

Linode Integrations is a collection of integrations lets you connect infrastructure and dev tools to the Linode platform. That let’s you manage your Linode resources using the tools you know and love.

StackScripts is a tool to automatically configure new Linode instances using simple scripts. Create your own StackScript or browse the community StackScript library.

Linode Bare Metal is the single-tenant solution for applications and organizations with security, compliance, and performance needs. Bare Metal combines direct hardware access and the flexibility of a virtual machine.

Nextcloud

Back to the Top

Nextcloud is an industry-leading, on-premises content collaboration platform for file sync & share and communication server. It is fully open source and you can host it yourself or pay a company to do it for you. Also checkout the following links below:

Nexcloud login screen

Nextcloud Hub is a tool that allows you to share and collaborate on documents, send and receive email, manage your calendar and have video chats without data leaks. As fully on-premises solution, Nextcloud Hub provides the benefits of online collaboration without the compliance and security risks.

Nexcloud Hub

Nextcloud AIO (All In One) is a tool that provides easy deployment and maintenance with most features included in this one Nextcloud instance.

Nextcloud Desktop Client is a tool to synchronize files from Nextcloud Server with your computer.

Nextcloud Deck is a kanban style organization tool aimed at personal planning and project organization for teams integrated with Nextcloud.

Nextcloud Files is a tool tool that allows your employees have easy access to their files, photos and documents to work and can share and collaborate with team members, customers and partners. So IT knows nobody besides those they shared with has access to those files.

Nextcloud Talk is a tool that protects your communication better than other team collaboration platforms like Microsoft Teams or Slack, making sure your data stays on your servers. It also goes further than other encrypted communication technologies by keeping even metadata from leaking.

Nextcloud Home is a tool that allows you store your documents, calendar, contacts and photos on your server at home, at one of at one Nextcloud’s providers or in a data center you trust.

Nextcloud Enterprise is a service that gives professional organizations software optimized and tested for mission critical environments.

Nextcloud Outlook Integration is a tool that automatically upload files to replace large attachments or integrate Calendars and Contacts in Microsoft Outlook.

Collabora Online in Nextcloud is a powerful LibreOffice-based online office suite with collaborative editing, which supports all major document, spreadsheet and presentation file formats and works in all modern browsers.

ONLYOFFICE integration in Nextcloud is a service that empowers your users to collaborate on office documents with team members in real time. It has compatibility with Microsoft Office formats means perfect documents, every time.

Nextcloud VM(virtual machine appliance) is a set of carefully crafted family of *nix scripts, which interactively guide you through a quality-controlled installation of a Nextcloud instance for Home/SME Server and scripts for Raspberry Pi 4. It is Community developed and maintained.

LibreSign is a Libre digital signature app for Nextcloud.

DigitalOcean

Back to the Top

DigitalOcean is a cloud infrastructure provider that provides developers cloud services that help to deploy and scale applications that run simultaneously on multiple computers with data centers worldwide.

DigitalOcean Tools

DigitalOcean API is a service that manages your DigitalOcean infrastructure with our RESTful API.

DigitalOcean Client libraries is a collection of libraries lets you use the DigitalOcean API in a variety of programming languages.

DigitalOcean CLI is a service that manages your DigitalOcean infrastructure through your terminal with our open source Command Line Interface (CLI).

Terraform provider is a service that allows the user treat their DigitalOcean infrastructure like code with Terraform.

DigitalOcean Custom images is a service that quickly builds your environment in the cloud by provisioning servers with your own custom image, or choose from various Linux distributions.

Container Registry is a service that easily stores, manages, and protects private container images.

Back4app Web Deployment

Back4app Web Deployment is a Container as a Service (CaaS) provider platform that allows the dev teams to build and deploy containerized applications with no downtime. You can simply connect it to a GitHub repository and publish the code within seconds.

MinIO Object Storage

Back to the Top

MinIO is a High Performance Object Storage released under GNU Affero General Public License v3.0. It is API compatible with Amazon S3 cloud storage service. Use MinIO to build high performance infrastructure for machine learning, analytics and application data workloads. It’s one of the fastest object storage platforms globally, with a read/write speed of 183GB/s-171GB/s if you use standard hardware. It can function as the main storage tier for many workloads like Spark, TensorFlow, Presto, Hadoop HDFS, and H2O.

MinIO UI

Run the following command to run the latest stable image of MinIO as a container using an ephemeral data volume:

Podman

1
2

podman run -p 9000:9000 -p 9001:9001 \
  quay.io/minio/minio server /data --console-address ":9001"

Docker

`1`	`#docker run -p 9000:9000 --name minio -d minio/minio server /export`

If you’re using an SSD mounted at /mnt/sdd, then we can run the following to use it instead:

`1`	`# docker run -v /mnt/ssd:/export -p 9000:9000 --name minio -d minio/minio server /export`

MacOS

1
2

brew install minio/stable/minio
minio server /data

Binary Download for MacOS

1
2
3

wget https://dl.min.io/server/minio/release/darwin-amd64/minio
chmod +x minio
./minio server /data

Linux

1
2
3

wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
./minio server /data

Architecture	URL
64-bit Intel/AMD	https://dl.min.io/server/minio/release/linux-amd64/minio
64-bit ARM	https://dl.min.io/server/minio/release/linux-arm64/minio
64-bit PowerPC LE (ppc64le)	https://dl.min.io/server/minio/release/linux-ppc64le/minio
IBM Z-Series (S390X)	https://dl.min.io/server/minio/release/linux-s390x/minio

Windows

To run MinIO on 64-bit Windows hosts, download the MinIO executable from the following URL:

https://dl.min.io/server/minio/release/windows-amd64/minio.exe

Use the following command to run a standalone MinIO server on the Windows host. Replace D:\ with the path to the drive or directory in which you want MinIO to store data. You must change the terminal or powershell directory to the location of the minio.exe executable, or add the path to that directory to the system $PATH:

minio.exe server D:\

Install from Source

Use the following commands to compile and run a standalone MinIO server from source. Source installation is only intended for developers and advanced users. If you do not have a working Golang environment, please follow How to install Golang. The minimum version required is go1.19.

go install github.com/minio/minio@latest

After you install MinIO:

The MinIO deployment starts using default root credentials minioadmin:minioadmin. You can test the deployment using the MinIO Console, an embedded web-based object browser built into MinIO Server. Point a web browser running on the host machine to http://127.0.0.1:9000 and log in with the root credentials. You can use the Browser to create buckets, upload objects, and browse the contents of the MinIO server.

When you run Minio you will be issued a key and a secret. These are used by the client or the web front-end to connect securely. I found my codes by typing in docker logs minio.

Created minio configuration file at /root/.minio

Endpoint:  http://172.17.0.2:9000  http://127.0.0.1:9000
AccessKey: accessCode
SecretKey: secretCode
Region:    us-west-1
SQS ARNs:  <none>

Browser Access:
   http://172.17.0.2:9000  http://127.0.0.1:9000

Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
   $ mc config host add myminio http://172.17.0.2:9000 accessCode secretCode

Object API (Amazon S3 compatible):
   Go:         https://docs.minio.io/docs/golang-client-quickstart-guide
   Java:       https://docs.minio.io/docs/java-client-quickstart-guide
   Python:     https://docs.minio.io/docs/python-client-quickstart-guide
   JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide

Drive Capacity: 50 GiB Free, 70 GiB Total

If you’d like to learn more then most of the Minio client commands support a help flag or give info on the command line:

NAME:
  mc - Minio Client for cloud storage and filesystems.

USAGE:
  mc [FLAGS] COMMAND [COMMAND FLAGS | -h] [ARGUMENTS...]

COMMANDS:
  ls       List files and folders.
  mb       Make a bucket or a folder.
  cat      Display file and object contents.
  pipe     Redirect STDIN to an object or file or STDOUT.
  share    Generate URL for sharing.
  cp       Copy files and objects.
  mirror   Mirror buckets and folders.
  diff     Show differences between two folders or buckets.
  rm       Remove files and objects.
  events   Manage object notifications.
  watch    Watch for files and objects events.
  policy   Manage anonymous access to objects.
  session  Manage saved sessions for cp and mirror commands.
  config   Manage mc configuration file.
  update   Check for new mc update.
  version  Print version info.
  help, h  Shows a list of commands or help for one command

Advanced options

You can have your client point to multiple Minio servers, which is really neat especially if you’re working on a distributed team.

Minio’s test-server called “play” is already configured in the default client, you can see all the servers you have configured with mc config host list.

To upload the photo to Minio’s “play” S3 server just type in:

# mc mb play/somebucketname

# mc cp ~/Downloads/IMG_2016120-25.jpg play/somebucketname

Recursive uploads:

If you want to test something larger out you could try uploading your entire Downloads photo, and then you should use the –recursive flag to make sure nothing’s missed:

# mc cp --recursive ~/Downloads/IMG_2016120-25.jpg myminio/photos

Databases

Back to the Top

SQL

SQL is a standard language for storing, manipulating and retrieving data in relational databases.

Coolify is an open-source & self-hostable Heroku/Netlify alternative.

MySQL is a fully managed database service to deploy cloud-native applications using the world’s most popular open source database.

PostgreSQL is a powerful, open source object-relational database system with over 30 years of active development that has earned it a strong reputation for reliability, feature robustness, and performance.

PostgREST is a tool that serves a fully RESTful API from any existing PostgreSQL database. It provides a cleaner, more standards-compliant, faster API than you are likely to write from scratch.

NocoDB is an open source #NoCode platform that turns any database into a smart spreadsheet. It turns any MySQL, PostgreSQL, SQL Server, SQLite & MariaDB into a smart-spreadsheet.

DBeaver is a free multi-platform database tool for developers, database administrators, analysts and all people who need to work with databases. Supports all popular databases: MySQL, PostgreSQL, SQLite, Oracle, DB2, SQL Server, Sybase, MS Access, Teradata, Firebird, Apache Hive, Phoenix, Presto, etc.

OmniDB is a web-based tool for database management.

Navicat is a series of graphical database management and development software produced by CyberTech Ltd. for MySQL, MariaDB, MongoDB, Oracle, SQLite, PostgreSQL and Microsoft SQL Server.

HeidiSQL is free software, and has the aim to be easy to learn. It lets you see and edit data and structures from computers running one of the database systems MariaDB, MySQL, Microsoft SQL, PostgreSQL and SQLite.

Beekeeper Studio is a cross-platform SQL editor and database manager(MySQL, Postgres, SQLite, SQL Server, and more.) available for Linux, Mac, and Windows.

UI Bakery is a web-based low-code internal tool builder. It can visualize the data pulled from PostgreSQL, MongoDB, MySQL, MicrosoftSQL, Redis.

IBM DB2 is a collection of hybrid data management products offering a complete suite of AI-empowered capabilities designed to help you manage both structured and unstructured data on premises as well as in private and public cloud environments. Db2 is built on an intelligent common SQL engine designed for scalability and flexibility.

OracleDB is a powerful fully managed database helps developers manage business-critical data with the highest availability, reliability, and security.

MariaDB is an enterprise open source database solution for modern, mission-critical applications.

EventQL is a distributed, analytical database. It allows you to store massive amounts of structured data and explore it using SQL and other programmatic query facilities.

CockroachDB is the SQL database for building global, scalable cloud services that survive disasters.

SQLite is a C-language library that implements a small, fast, self-contained, high-reliability, full-featured, SQL database engine.SQLite is the most used database engine in the world. SQLite is built into all mobile phones and most computers and comes bundled inside countless other applications that people use every day.

SQLite Database Browser is an open source SQL tool that allows users to create, design and edits SQLite database files. It lets users show a log of all the SQL commands that have been issued by them and by the application itself.

TimescaleDB is an open-source database designed to make SQL scalable for time-series data. It is engineered up from PostgreSQL and packaged as a PostgreSQL extension, providing automatic partitioning across time and space (partitioning key), as well as full SQL support.

InfluxDB is an open source time series platform. This includes APIs for storing and querying data, processing it in the background for ETL or monitoring and alerting purposes, user dashboards, Internet of Things sensor data, and visualizing and exploring the data and more. It also has support for processing data from Graphite.

Atlas is an in-memory dimensional time series database.

dbWatch is a complete database monitoring/management solution for SQL Server, Oracle, PostgreSQL, Sybase, MySQL and Azure. Designed for proactive management and automation of routine maintenance in large scale on-premise, hybrid/cloud database environments.

Adminer is an SQL management client tool for managing databases, tables, relations, indexes, users. Adminer has support for all the popular database management systems such as MySQL, MariaDB, PostgreSQL, SQLite, MS SQL, Oracle, Firebird, SimpleDB, Elasticsearch and MongoDB.

Knex is a query builder for PostgreSQL, MySQL, CockroachDB, SQL Server, SQLite3 and Oracle, designed to be flexible, portable, and fun to use.

rqlite is an easy-to-use, lightweight, distributed relational database, which uses SQLite as its storage engine.

osquery is a SQL powered operating system instrumentation, monitoring, and analytics framework.

SQLModel is a library for interacting with SQL databases from Python code, with Python objects. It is designed to be intuitive, easy to use, highly compatible, and robust.

Citus is a PostgreSQL extension that transforms Postgres into a distributed database—so you can achieve high performance at any scale.

DbVisualizer is a SQL management tool that allows users to manage a wide range of databases such as Oracle, Sybase, SQL Server, MySQL, H3, and SQLite.

AppDynamics Database is a management product for Microsoft SQL Server. With AppDynamics you can monitor and trend key performance metrics such as resource consumption, database objects, schema statistics and more, allowing you to proactively tune and fix issues in a High-Volume Production Environment.

Toad is a SQL Server DBMS toolset developed by Quest. It increases productivity by using extensive automation, intuitive workflows, and built-in expertise. This SQL management tool resolve issues, manage change and promote the highest levels of code quality for both relational and non-relational databases.

Lepide SQL Server is an open source storage manager utility to analyse the performance of SQL Servers. It provides a complete overview of all configuration and permission changes being made to your SQL Server environment through an easy-to-use, graphical user interface.

Sequel Pro is a fast MacOS database management tool for working with MySQL. This SQL management tool helpful for interacting with your database by easily to adding new databases, new tables, and new rows.

ElasticSearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch is developed in Java.

Logstash is a tool for managing events and logs. When used generically, the term encompasses a larger system of log collection, processing, storage and searching activities.

Kibana is an open source data visualization plugin for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. Users can create bar, line and scatter plots, or pie charts and maps on top of large volumes of data.

Trino is a Distributed SQL query engine for big data. It is able to tremendously speed up ETL processes, allow them all to use standard SQL statement, and work with numerous data sources and targets all in the same system.

Tableau is a Data Visualization software used in relational databases, cloud databases, and spreadsheets. Tableau was acquired by Salesforce in August 2019.

DataGrip is a professional DataBase IDE developed by Jet Brains that provides context-sensitive code completion, helping you to write SQL code faster. Completion is aware of the tables structure, foreign keys, and even database objects created in code you’re editing.

RStudio is an integrated development environment for R and Python, with a console, syntax-highlighting editor that supports direct code execution, and tools for plotting, history, debugging and workspace management.

NoSQL

Back to the Top

NoSQL is a database that is interchangeably referred to as “nonrelational, or “non-SQL” to highlight that the database can handle huge volumes of rapidly changing, unstructured data in different ways than a relational (SQL-based) database with rows and tables.

Scylla is the real-time big data database that is API-compatible with Apache Cassandra and Amazon DynamoDB.

Apache Cassandra™ is an open source NoSQL distributed database trusted by thousands of companies for scalability and high availability without compromising performance. Cassandra provides linear scalability and proven fault-tolerance on commodity hardware or cloud infrastructure make it the perfect platform for mission-critical data.

Apache HBase™ is an open-source, NoSQL, distributed big data store. It enables random, strictly consistent, real-time access to petabytes of data. HBase is very effective for handling large, sparse datasets. HBase serves as a direct input and output to the Apache MapReduce framework for Hadoop, and works with Apache Phoenix to enable SQL-like queries over HBase tables.

Hadoop Distributed File System (HDFS) is a distributed file system that handles large data sets running on commodity hardware. It is used to scale a single Apache Hadoop cluster to hundreds (and even thousands) of nodes. HDFS is one of the major components of Apache Hadoop, the others being MapReduce and YARN.

Redis(REmote DIctionary Server) is an open source (BSD licensed), in-memory data structure store, used as a database, cache, and message broker. It provides data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes, and streams.

FoundationDB is an open source distributed database designed to handle large volumes of structured data across clusters of commodity servers. It organizes data as an ordered key-value store and employs ACID transactions for all operations. It is especially well-suited for read/write workloads but also has excellent performance for write-intensive workloads. FoundationDB was acquired by Apple in 2015.

CouchbaseDB is an open source distributed multi-model NoSQL document-oriented database. It creates a key-value store with managed cache for sub-millisecond data operations, with purpose-built indexers for efficient queries and a powerful query engine for executing SQL queries.

MongoDB is a document database meaning it stores data in JSON-like documents.

NoSQLBooster is a cross-platform IDE for MongoDB v2.6-5.0, which provides a build-in MongoDB script debugger, SQL query, server monitoring tools, chaining fluent query, query code generator, task scheduling, ES2020 support, and advanced IntelliSense experience.

ClickHouse® is an open-source column-oriented database management system that allows generating analytical data reports in real-time.

Neo4j is a graph database management system that provides an array of tools, libraries, and frameworks to make development faster and easier.

Remote Access

Back to the Top

FreeRDP is a free remote desktop protocol library and clients.

Rustdesk is an open source virtual/remote desktop infrastructure for everyone. Display and control your PC (Windows, macOS, and Linux) and Android devices.

TinyPilot is a tool that enables KVM over IP letting you control any computer remotely.

X2Go is open source remote desktop software for Linux that uses a modified NX 3 protocol. It gives remote access to a Linux system’s GUI.

Apache Guacamole is a clientless remote desktop gateway. It supports standard protocols like VNC, RDP, and SSH.

Remmina is a Remote access screen and file sharing to your desktop. It has Remote Access Protocol Plugins for RDP, SSH, SPICE, VNC, X2Go, HTTP/HTTPS.

Remotely is a remote control and remote scripting solution, built with .NET 6, Blazor, SignalR Core, and WebRTC.

P2P Remote Desktop is a portable, no configuration or installation needed remote desktop tool.

Cloudflare Tunnel is a tunneling daemon that proxies traffic from the Cloudflare network to your origins. This daemon sits between Cloudflare network and your origin (a webserver). This attracts client requests and sends them to you via this daemon, without requiring you to poke holes on your firewall and your origin(webserver) can remain as closed as possible.

WireGuard® is a straight-forward, fast and modern VPN that utilizes state-of-the-art cryptography. It aims to be faster, simpler, leaner, and more useful than IPsec while avoiding the massive headache. WireGuard is designed as a general-purpose VPN for running on embedded interfaces and super computers alike, fit for many circumstances. It’s cross-platform (Windows, macOS, BSD, iOS, Android) and widely deployable.

NetBird is an open-source VPN management platform built on top of WireGuard® making it easy to create secure private networks for your organization or home.

Tailscale is a WireGuard-based app that makes secure, private networks easy for teams of any scale. It works like an overlay network between the computers of your networks using all kinds of NAT traversal sorcery.

Headscale is an open source, self-hosted implementation of the Tailscale coordination server.

MeshCentral is a full computer management web site. It can run your own web server to remotely manage and control computers on a local network or anywhere on the internet. Once you get the server started, create device group and download and install an agent on each computer you want to manage.

VNC Viewer is a free remote desktop application that use can use on your iPhone, iPad, Mac, Windows and Linux computers from anywhere in the world.

TightVNC is a free remote desktop application. It can see the desktop of a remote machine and control it with your local mouse and keyboard, just like you would do it sitting in the front of that computer.

KRDC is a client application that allows you to view or even control the desktop session on another machine that is running a compatible server. VNC and RDP is supported.

Krfb Desktop Sharing is a server application that allows you to share your current session with a user on another machine, who can use a VNC client to view or even control the desktop.

wayvnc is a VNC server for wlroots-based Wayland compositors (no_entry Gnome, KDE and Weston are not supported). It attaches to a running Wayland session, creates virtual input devices, and exposes a single display via the RFB protocol.

Waypipe is a proxy for Wayland clients. It forwards Wayland messages and serializes changes to shared memory buffers over a single socket.

Virtualization

Back to the Top

HVM (Hardware Virtual Machine) is a virtualization type that provides the ability to run an operating system directly on top of a virtual machine without any modification, as if it were run on the bare-metal hardware.

PV(ParaVirtualization) is an efficient and lightweight virtualization technique introduced by the Xen Project team, later adopted by other virtualization solutions. PV does not require virtualization extensions from the host CPU and thus enables virtualization on hardware architectures that do not support Hardware-assisted virtualization.

Network functions virtualization (NFV) is the replacement of network appliance hardware with virtual machines. The virtual machines use a hypervisor to run networking software and processes such as routing and load balancing. NFV allows for the separation of communication services from dedicated hardware, such as routers and firewalls. This separation means network operations can provide new services dynamically and without installing new hardware. Deploying network components with network functions virtualization only takes hours compared to months like with traditional networking solutions.

Software Defined Networking (SDN) is an approach to networking that uses software-based controllers or application programming interfaces (APIs) to communicate with underlying hardware infrastructure and direct traffic on a network. This model differs from that of traditional networks, which use dedicated hardware devices (routers and switches) to control network traffic.

Virtualized Infrastructure Manager (VIM) is a service delivery and reduce costs with high performance lifecycle management Manage the full lifecycle of the software and hardware comprising your NFV infrastructure (NFVI), and maintaining a live inventory and allocation plan of both physical and virtual resources.

Management and Orchestration(MANO) is an ETSI-hosted initiative to develop an Open Source NFV Management and Orchestration (MANO) software stack aligned with ETSI NFV. Two of the key components of the ETSI NFV architectural framework are the NFV Orchestrator and VNF Manager, known as NFV MANO.

Magma is an open source software platform that gives network operators an open, flexible and extendable mobile core network solution. Their mission is to connect the world to a faster network by enabling service providers to build cost-effective and extensible carrier-grade networks. Magma is 3GPP generation (2G, 3G, 4G or upcoming 5G networks) and access network agnostic (cellular or WiFi). It can flexibly support a radio access network with minimal development and deployment effort.

OpenRAN is an intelligent Radio Access Network(RAN) integrated on general purpose platforms with open interface between software defined functions. Open RANecosystem enables enormous flexibility and interoperability with a complete openess to multi-vendor deployments.

Open vSwitch(OVS)is an open source production quality, multilayer virtual switch licensed under the open source Apache 2.0 license. It is designed to enable massive network automation through programmatic extension, while still supporting standard management interfaces and protocols (NetFlow, sFlow, IPFIX, RSPAN, CLI, LACP, 802.1ag).

Edge is a distributed computing framework that brings enterprise applications closer to data sources such as IoT devices or local edge servers. This proximity to data at its source can deliver strong business benefits, including faster insights, improved response times and better bandwidth availability.

Multi-access edge computing (MEC) is an Industry Specification Group (ISG) within ETSI to create a standardized, open environment which will allow the efficient and seamless integration of applications from vendors, service providers, and third-parties across multi-vendor Multi-access Edge Computing platforms.

Virtualized network functions(VNFs) is a software application used in a Network Functions Virtualization (NFV) implementation that has well defined interfaces, and provides one or more component networking functions in a defined way. For example, a security VNF provides Network Address Translation (NAT) and firewall component functions.

Cloud-Native Network Functions(CNF) is a network function designed and implemented to run inside containers. CNFs inherit all the cloud native architectural and operational principles including Kubernetes(K8s) lifecycle management, agility, resilience, and observability.

Physical Network Function(PNF) is a physical network node which has not undergone virtualization. Both PNFs and VNFs (Virtualized Network Functions) can be used to form an overall Network Service.

Network functions virtualization infrastructure(NFVI) is the foundation of the overall NFV architecture. It provides the physical compute, storage, and networking hardware that hosts the VNFs. Each NFVI block can be thought of as an NFVI node and many nodes can be deployed and controlled geographically.

Virtualization-based Security (VBS) is a hardware virtualization feature to create and isolate a secure region of memory from the normal operating system.

Hypervisor-Enforced Code Integrity (HVCI) is a mechanism whereby a hypervisor, such as Hyper-V, uses hardware virtualization to protect kernel-mode processes against the injection and execution of malicious or unverified code. Code integrity validation is performed in a secure environment that is resistant to attack from malicious software, and page permissions for kernel mode are set and maintained by the hypervisor.

NVIDIA virtual GPU (vGPU) is a software enables powerful GPU performance for workloads ranging from graphics-rich virtual workstations to data science and AI, enabling IT to leverage the management and security benefits of virtualization as well as the performance of NVIDIA GPUs required for modern workloads.

AMD MxGPU is a hardware-based virtualized GPU solution, is built on industry standard SR-IOV (Single-Root I/O Virtualization) technology and allows multiple virtualized users per physical GPU to work remotely.

Proxmox Virtual Environment(VE) is a complete open-source platform for enterprise virtualization. It inlcudes a built-in web interface that you can easily manage VMs and containers, software-defined storage and networking, high-availability clustering, and multiple out-of-the-box tools on a single solution.

KVM (for Kernel-based Virtual Machine) is a full virtualization solution for Linux on x86 hardware containing virtualization extensions (Intel VT or AMD-V). It consists of a loadable kernel module, kvm.ko, that provides the core virtualization infrastructure and a processor specific module, kvm-intel.ko or kvm-amd.ko.

QEMU is a fast processor emulator using a portable dynamic translator. QEMU emulates a full system, including a processor and various peripherals. It can be used to launch a different Operating System without rebooting the PC or to debug system code.

Quickemu is a program that quickly create and run optimised Windows, macOS and Linux desktop virtual machines.

Hyper-V enables running virtualized computer systems on top of a physical host. These virtualized systems can be used and managed just as if they were physical computer systems, however they exist in virtualized and isolated environment. Special software called a hypervisor manages access between the virtual systems and the physical hardware resources. Virtualization enables quick deployment of computer systems, a way to quickly restore systems to a previously known good state, and the ability to migrate systems between physical hosts.

Cloud Hypervisor is an open source Virtual Machine Monitor (VMM) that runs on top of KVM. The project focuses on exclusively running modern, cloud workloads, on top of a limited set of hardware architectures and platforms. Cloud workloads refers to those that are usually run by customers inside a cloud provider. Cloud Hypervisor is implemented in Rust and is based on the rust-vmm crates.

VirtManager is a graphical tool for managing virtual machines via libvirt. Most usage is with QEMU/KVM virtual machines, but Xen and libvirt LXC containers are well supported. Common operations for any libvirt driver should work.

oVirt is an open-source distributed virtualization solution, designed to manage your entire enterprise infrastructure. oVirt uses the trusted KVM hypervisor and is built upon several other community projects, including libvirt, Gluster, PatternFly, and Ansible. Founded by Red Hat as a community project on which Red Hat Enterprise Virtualization is based allowing for centralized management of virtual machines, compute, storage and networking resources, from an easy-to-use web-based front-end with platform independent access.

Firecracker is an open source virtualization technology that is purpose-built for creating and managing secure, multi-tenant container and function-based services that provide serverless operational models. It runs workloads in lightweight virtual machines, called microVMs, which combine the security and isolation properties provided by hardware virtualization technology with the speed and flexibility of containers.

Foreman is a free open source project that gives you the power to easily automate repetitive tasks, quickly deploy applications, and proactively manage your servers life cycle, on-premises or in the cloud.

Harvester is an open source hyper-converged infrastructure (HCI) software built on Kubernetes.

Anthos is a modern application management platform that provides a consistent development and operations experience for cloud and on-premises environments.

HyperKit is a toolkit for embedding hypervisor capabilities in your application. It includes a complete hypervisor, based on xhyve/bhyve, which is optimized for lightweight virtual machines and container deployment. It is designed to be interfaced with higher-level components such as the VPNKit and DataKit. HyperKit currently only supports macOS using the Hypervisor.framework making it a core component of Docker Desktop for Mac.

Intel® Graphics Virtualization Technology (Intel® GVT) is a full GPU virtualization solution with mediated pass-through, starting from 4th generation Intel Core (TM) processors with Intel processor graphics(Broadwell and newer). It can be used to virtualize the GPU for multiple guest virtual machines, effectively providing near-native graphics performance in the virtual machine and still letting your host use the virtualized GPU normally.

Apple Hypervisor is a frameowrk that builds virtualization solutions on top of a lightweight hypervisor, without third-party kernel extensions. Hypervisor provides C APIs so you can interact with virtualization technologies in user space, without writing kernel extensions (KEXTs). As a result, the apps you create using this framework are suitable for distribution on the Mac App Store.

Apple Virtualization Framework is a framework that provides high-level APIs for creating and managing virtual machines on Apple silicon and Intel-based Mac computers. This framework is used to boot and run a Linux-based operating system in a custom environment that you define. It also supports the Virtio specification, which defines standard interfaces for many device types, including network, socket, serial port, storage, entropy, and memory-balloon devices.

Apple Paravirtualized Graphics Framework is a framework that implements hardware-accelerated graphics for macOS running in a virtual machine, hereafter known as the guest. The operating system provides a graphics driver that runs inside the guest, communicating with the framework in the host operating system to take advantage of Metal-accelerated graphics.

Cilicon is a macOS App that leverages Apple’s Virtualization Framework to create, provision and run ephemeral virtual machines with minimal setup or maintenance effort. You should be able to get up and running with your self-hosted CI in less than an hour.

Xen is focused on advancing virtualization in a number of different commercial and open source applications, including server virtualization, Infrastructure as a Services (IaaS), desktop virtualization, security applications, embedded and hardware appliances, and automotive/aviation.

Ganeti is a virtual machine cluster management tool built on top of existing virtualization technologies such as Xen or KVM and other open source software. Once installed, the tool assumes management of the virtual instances (Xen DomU).

Packer is an open source tool for creating identical machine images for multiple platforms from a single source configuration. Packer is lightweight, runs on every major operating system, and is highly performant, creating machine images for multiple platforms in parallel. Packer does not replace configuration management like Chef or Puppet. In fact, when building images, Packer is able to use tools like Chef or Puppet to install software onto the image.

Vagrant is a tool for building and managing virtual machine environments in a single workflow. With an easy-to-use workflow and focus on automation, Vagrant lowers development environment setup time, increases production parity, and makes the “works on my machine” excuse a relic of the past. It provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.

Password Management

Back to the Top

Bitwarden is a free and open-source password management service that stores sensitive information such as website credentials in an encrypted vault.

Bitwarden Server is a project contains the APIs, database, and other core infrastructure items needed for the “backend” of all bitwarden client applications. Checkout Bitwarden’s self-hosted release repository.

Vaultwarden is an unofficial Bitwarden compatible server written in Rust, formerly known as bitwarden_rs.

Passbolt is an open-source/self-hosted password manager for teams. It allows you to securely share and store credentials. For instance, the wifi password of your office, the administrator password of a router or your organization’s social media account passwords, all of them can be secured using passbolt.

KeePassXC is a modern, secure, and open-source password manager that stores and manages your most sensitive information. You can run KeePassXC on Windows, macOS, and Linux systems. It saves many different types of information, such as usernames, passwords, URLs, attachments, and notes in an offline, encrypted file that can be stored in any location, including private and public cloud solutions.

AuthPass.app is an Open-Source Password Manager for mobile and desktop that is Keepass 2.x (kdbx 3.x) compatible.

pass is an open-source unix-based password utilitiy with various gui clients

SSH

Back to The Top

Secure Shell Protocol (SSH) is a cryptographic network protocol for operating network services securely over an unsecured network.

Advanced SSH config is a tool that enhances ssh_config file capabilities, completely transparent.

AutoSSH is a tool that automatically respawn ssh session after network interruption.

ContainersSSH is an SSH Server that Launches Containers in Kubernetes and Docker on demand.

Cluster SSH is a tool that controls a number of xterm windows via a single graphical console.

DSH is a Dancer’s shell / distributed shell - Wrapper for executing multiple remote shell commands from one command line.

Flightplan is a node.js library for streamlining application deployment or systems administration tasks (local and remote hosts).

Mosh is a command-line program, like SSH. You can use it inside xterm, gnome-terminal, urxvt, Terminal.app, iTerm, emacs, screen, or tmux.

Parallel SSH is an asynchronous parallel SSH library designed for large scale automation. It differentiates ifself from alternatives, other libraries and higher level frameworks like Ansible or Chef.

SSH Audit is a tool for SSH server & client configuration auditing (banner, key exchange, encryption, mac, compression, compatibility, security, etc).

Sshwifty is a SSH and Telnet connector made for the Web. It can be deployed on your computer or server to provide SSH and Telnet access interface for any compatible (standard) web browser.

SSHrc is a tool that sources ~/.sshrc on your local computer after logging in remotely.

StormSSH is a command line tool to manage SSH connections.

Tailscale SSH is a service that allows Tailscale to manage the authentication and authorization of SSH connections on your tailnet.

VPN

Back to The Top

VPN (Virtual Private Network) is a service that encrypts your internet traffic on unsecured networks to protect your online identity, hide your IP address, and shield your online data from third parties.

Wireguard - New minimal VPN Solution that is very fast.

OpenVPN - Uses a custom security protocol that utilizes SSL/TLS for key exchange.

Pritunl - OpenVPN based solution that’s easy to set up.

SoftEther - Multi-protocol software VPN with advanced features.

sshuttle - Poor man’s VPN.

strongSwan - Complete IPsec implementation for Linux.

tinc - Distributed p2p VPN.

LDAP

Back to The Top

LDAP Servers

389 Directory Server - Developed by Red Hat.

Apache Directory Server - Apache Software Foundation project written in Java.

Fusion Directory - Improve the Management of the services and the company directory based on OpenLDAP. OpenDJ - Fork of OpenDS.

OpenDS - Another directory server written in Java.

OpenLDAP - Developed by the OpenLDAP Project.

LDAP management

Apache Directory Studio - The Eclipse-based LDAP browser and directory client

Log Management

Back to The Top

Echofish - A web based real-time event log aggregation, analysis, monitoring and management system.

Fluentd - Log Collector and Shipper.

Flume - Distributed log collection and aggregation system.

Graylog2 - Pluggable Log and Event Analysis Server with Alerting options.

Heka - Stream processing system which may be used for log aggregation.

Elasticsearch - A Lucene Based Document store mainly used for log indexing, storage and analysis.

Kibana - Visualize logs and time-stamped data.

Logstash - Tool for managing events and logs.

Octopussy - Log Management Solution (Visualize/Alert/Report).

DNS

Back to The Top

Duckdns - A free service which will point a DNS (sub domains of duckdns.org) to an IP of your choice. The service is completely free, and doesn’t require reactivation or forum posts to maintain its existence.

dnsmasq - A lightweight service providing DNS, DHCP and TFTP services to small-scale networks.

MagicDNS is a tool that automatically registers DNS names for devices in your network.

Bind - The most widely used name server software.

djbdns - A collection of DNS applications, including tinydns.

Designate - DNS REST API that support several DNS servers as its backend.

Knot - High performance authoritative-only DNS server.

Lexicon is a tool that provides a way to manipulate DNS records on multiple DNS providers in a standardized way.

NSD - Authoritative only, high performance, simple name server.

PowerDNS - DNS server with a variety of data storage back-ends and load balancing features.

CoreDNS is a DNS server/forwarder, written in Go, that chains plugins. Each plugin performs a (DNS) function.

Unbound - Validating, recursive, and caching DNS resolver.

Yadifa - Lightweight authoritative Name Server with DNSSEC capabilities powering the .eu top-level domain.

Network Tools

Back to the Top

MQTT is an OASIS standard messaging protocol for the Internet of Things (IoT). It is designed as an extremely lightweight publish/subscribe messaging transport that is ideal for connecting remote devices with a small code footprint and minimal network bandwidth.

Mongoose is a networking library for C/C++. It implements event-driven non-blocking APIs for TCP, UDP, HTTP, WebSocket, MQTT. It is designed for connecting devices and bringing them online.

Nautobot is a Network Source of Truth and Network Automation Platform built as a web application atop the Django Python framework with a PostgreSQL or MySQL database.

Eclipse Mosquitto is an open source implementation of a server for version 5.0, 3.1.1, and 3.1 of the MQTT protocol.

Ejabberd is an open-source, robust, scalable and extensible realtime platform built using Erlang/OTP, that includes XMPP Server, MQTT Broker and SIP Service.

Nebula is a scalable overlay networking tool with a focus on performance, simplicity and security. It lets you seamlessly connect computers anywhere in the world. Nebula is portable, and runs on Linux, OSX, Windows, iOS, and Android. It can be used to connect a small number of computers, but is also able to connect tens of thousands of computers.

LibreSpeed is a network speed test tool that can be run on your LAN or hosted in the cloud.

SmokePing is a deluxe latency measurement tool. It can measure, store and display latency, latency distribution and packet loss. It uses RRDtool to maintain a longterm data-store and to draw pretty graphs, giving up to the minute information on the state of each network connection.

Tailnet is your private network. When you log in for the first time to Tailscale on your phone, laptop, desktop, or cloud VM, a tailnet is created. For personal users, you are a tailnet of many devices and one person. Each device gets a private Tailscale IP address in the CGNAT range and every device can talk directly to every other device, wherever they are on the internet.

Tailscale SSH is a service that allows Tailscale to manage the authentication and authorization of SSH connections on your tailnet.

Tailscale Funnel is a feature that allows you to route traffic from the wider internet to one or more of your Tailscale nodes. You can think of this as publicly sharing a node for anyone to access, even if they don’t have Tailscale themselves.

Cockpit is a web-based graphical interface for servers, intended for everyone. It uses your system’s normal user logins and privileges. Network-wide logins are also supported through single-sign-on and other authentication techniques.

NetBox is a leading solution for modeling and documenting modern networks. It combines the traditional disciplines of IP address management (IPAM) and datacenter infrastructure management (DCIM) with powerful APIs and extensions, NetBox provides the ideal “source of truth” to power network automation.

Network UPS Tools (NUT) is a project that provides support for Power Devices, such as Uninterruptible Power Supplies, Power Distribution Units, Automatic Transfer Switches, Power Supply Units and Solar Controllers. NUT provides a common protocol and set of tools to monitor and manage such devices, and to consistently name equivalent features and data points, across a vast range of vendor-specific protocols and connection media types.

Dnsmasq is a tool that provides network infrastructure for small networks: DNS, DHCP, router advertisement and network boot. It is designed to be lightweight and have a small footprint, suitable for resource constrained routers and firewalls. It has also been widely used for tethering on smartphones and portable hotspots, and to support virtual networking in virtualisation frameworks. Supported platforms include Linux (with glibc and uclibc), Android, BSD, and MacOS.

Nginx proxy manager (NPM) is a reverse proxy management system running on Docker. It’s easy to set up and does not require users to know how to work with Nginx servers or SSL certificates.

Netdata is high-fidelity infrastructure monitoring and troubleshooting, real-time monitoring Agent collects thousands of metrics from systems, hardware, containers, and applications with zero configuration. It runs permanently on all your physical/virtual servers, containers, cloud deployments, and edge/IoT devices, and is perfectly safe to install on your systems mid-incident without any preparation.

Pi-hole is a DNS sinkhole that protects your devices from unwanted content, without installing any client-side software, intended for use on a private network. It is designed for use on embedded devices with network capability, such as the Raspberry Pi, but it can be used on other machines running Linux and cloud implementations.

OWASP Amass is a tool that performs network mapping of attack surfaces and external asset discovery using open source information gathering and active reconnaissance techniques.

Smap is a port scanner built with shodan.io’s free API. It takes same command line arguments as Nmap and produces the same output which makes it a drop-in replacament for Nmap.

ORY Oathkeeper is an Identity & Access Proxy (IAP) and Access Control Decision API that authorizes HTTP requests based on sets of Access Rules.

Ory Kratos is a developer-friendly, security-hardened and battle-test Identity, User Management and Authentication system for the Cloud. The Kratos identity server (similiar to Auth0, Okta, Firebase) with Ory-hardened authentication, MFA, FIDO2, TOTP, WebAuthn, profile management, identity schemas, social sign in, registration, account recovery, passwordless.

Ory Hydra is a hardened, OpenID Certified OAuth 2.0 Server and OpenID Connect Provider optimized for low-latency, high throughput, and low resource consumption. Ory Hydra is not an identity provider (user sign up, user login, password reset flow), but connects to your existing identity provider through a login and consent app.

Ory Keto is an Open Source (Go) implementation of Zanzibar: Google’s Consistent, Global Authorization System. It ships gRPC, REST APIs, newSQL, and an easy and granular permission language. Supports ACL, RBAC, and other access models.

AdGuard Home is a DNS relay station with ad/tracker/other blocking, IP address redirections, and DNS-over-HTTPS.

NetBird is an open-source VPN management platform built on top of WireGuard® making it easy to create secure private networks for your organization or home.

Supabase is an open source Firebase alternative. It is building the features of Firebase using enterprise-grade open source tools.

Plik is a scalable & friendly temporary file upload system (Wetransfer like) in golang.

Restify is a framework, utilizing connect style middleware for building REST APIs.

Traefik is an open source Edge Router that makes publishing your services a fun and easy experience. It receives requests on behalf of your system and finds out which components are responsible for handling them. What sets Traefik apart, besides its many features, is that it automatically discovers the right configuration for your services.

Traefik Mesh is a simple, yet full-featured service mesh. It is container-native and fits as your de-facto service mesh in your Kubernetes cluster. It supports the latest Service Mesh Interface specification SMI that facilitates integration with pre-existing solution.

DuckDNS is a free service that allows you to bind your own favorite subdomain under duckdns.org to the public IP address in use from your router, even though such address is dynamically allocated by your internet service provider and therefore changes over time.

Trust-DNS is a Rust based DNS client, server, and Resolver, built to be safe and secure from the ground up.

Hugo is a static HTML and CSS website generator written in Go. It is optimized for speed, ease of use, and configurability. Hugo takes a directory with content and templates and renders them into a full HTML website.

sshuttle is a transparent proxy server that works as a poor man’s VPN that forwards connection over ssh. It works with Linux and MacOS and supports DNS tunneling.

NetHopper is a Multi-Cloud Application Network as a Service. The easiest way to visualize, connect, secure, protect, and monitor microservices across any cluster, site, cloud, or network.

Cypress is a tool that makes it fast, easy and reliable testing for anything that runs in a browser.

Kimchi is an HTML5 based management tool for KVM. It is designed to make it as easy as possible to get started with KVM and create your first guest.

ION is a distributed real-time communication system, the goal is to chat anydevice, anytime, anywhere.

FreeRDP is a free remote desktop protocol library and clients.

Pimox is a port of Proxmox to the Raspberry Pi allowing you to build a Proxmox cluster of Rapberry Pi’s or even a hybrid cluster of Pis and x86 hardware.

PiKVM is a very simple and fully functional Raspberry Pi-based KVM over IP.

Firezone is a self-hosted WireGuard®-based VPN server and Linux firewall.

Monoid is an open-source suite of tools for automating data privacy.

Pinecone is an experimental overlay routing protocol suite which is the foundation of the current P2P Matrix demos. It is designed to provide end-to-end encrypted connectivity between devices at a global scale over any compatible medium (currently TCP, WebSockets, Bluetooth Low Energy etc), allowing multi-hop peer-to-peer connectivity between devices even in places where there is no Internet connectivity.

Service Discovery

Back to The Top

Consul is a tool for service discovery, monitoring and configuration. Install Consul on Self-Hosted Kubernetes Clusters.

Linkerd is an ultralight, security-first service mesh for Kubernetes. Linkerd adds critical security, observability, and reliability features to your Kubernetes stack with no code change required.

Doozerd is a highly-available, completely consistent store for small amounts of extremely important data.

Admiral is a tool for for service discovery that provides automatic configuration and service discovery for multicluster Istio service mesh.

ScaleCube is a library that simplifies the development of reactive and distributed applications by providing an embeddable microservices library. It connects distributed microservices in a way that resembles a fabric when viewed collectively. It greatly simplifies and streamlines asynchronous programming and provides a tool-set for managing microservices architecture.

DPS(dns-proxy-server) is a lightweight end user (Developers, Server Administrators) DNS server tool for service discovery, which make it easy to develop in systems where one hostname can solve to different IPs based on the configured environment, so you can:

Solve hostnames from local configuration database.
Solve hostnames from docker containers using docker hostname option or HOSTNAMES env.
Solve hostnames from a list of configured remote DNS servers(as a proxy) if no answer of two above .
Graphic interface to Create/List/Update/Delete A/CNAME records.
Solve host machine IP using host.docker hostname.
Access container by its container name / service name.
Specify from which network solve container IP.

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.

Security

Back to The Top

Blackbox - Safely store secrets in Git/Mercurial. Provides tooling to automatically encrypt secrets like passwords.

CrowdSec - Locally scans log files and optionnaly requests, detecting and blocking malicious behaviors. AppSec capabilities to enable virtual-patching and turn your install into a WAF. Share attacks signals and benefit from real time blocklist of the most agressive IPs attacking CrowdSec’s network.

Denyhosts - Thwart SSH dictionary based attacks and brute force attacks.

Fail2Ban - Scans log files and takes action on IPs that show malicious behavior.

fwknop - Protects ports via Single Packet Authorization in your firewall.

Glastopf - A low-interaction web application honeypot to emulate vulnerabilities and gather attack data.

Kippo - A medium-interaction SSH honeypot, mostly used as a standalone SSH daemon with a configurable Filesystem sandbox. OSSEC - OSSEC is a HIDS that performs log analysis, FIM, rootkit detection, and much more.

OSQuery - Query your servers status and info using a SQL like interface.

OPNsense is an open source firewall and routing software developed by Deciso. It offers an integrated Netflow analyser without the need for additional plugins or tools, similar to what you may find in high-end commercial products.

pfSense - Firewall and Router FreeBSD distribution.

Snort - Snort is a free and open source network intrusion prevention system (NIPS) and network intrusion detection system (NIDS) created by Martin Roesch in 1998.

SpamAssassin - A powerful and popular email spam filter employing a variety of detection technique.

BounCA - is a personal SSL / Certificate Authority Key management tool. Create self-signed SSL certificates via your browser.

Troubleshooting

Back to The Top

NETworkManager - A powerful tool for managing networks and troubleshoot network problems. It contains features like a WiFi analyzer, IP scanner, port scanner, ping monitor, traceroute, DNS lookup or a LLDP/CDP capture.

Wireshark - The world’s foremost network protocol analyzer.

Selfspy is a daemon for Unix/X11, MacOS (thanks to @ljos) and Windows (thanks to @Foxboron), that continuously monitors and stores what you are doing on your computer. This way, you can get all sorts of nifty statistics and reminders on what you have been up to.

Cilium - A networking, observability, and security solution with an eBPF-based dataplane. It provides a simple flat Layer 3 network with the ability to span multiple clusters in either a native routing or overlay mode.

Netshoot - A Docker + Kubernetes network trouble-shooting swiss-army container.

Kubevious - A suite of app-centric assurance, validation, and introspection products for Kubernetes. It helps running modern Kubernetes applications without disasters and costly outages by continuously validating application manifests, cluster state, and configuration.

HOMER - A robust, carrier-grade, scalable Packet and Event capture system and VoiP/RTC Monitoring Application based on the HEP/EEP protocol and ready to process & store insane amounts of signaling, rtc events, logs and statistics with instant search, end-to-end analysis and drill-down capabilities.

mitmproxy - A Python tool used for intercepting, viewing and modifying network traffic. Invaluable in troubleshooting certain problems.

Sysdig - Capture system state and activity from a running Linux instance, then save, filter and analyze.

Sysdig Inspect - A powerful opensource interface for container troubleshooting and security investigation.

Monitoring

Back to the Top

Proxmox Mail Gateway is an open-source email security solution protecting your mail server against all email threats from the moment they emerge.

M2MLabs MainSpring is an application framework for building machine-to-machine applications like vehicle tracking or machine remote monitoring. In such applications typically a remote device equipped with sensors (e.g. gps, temperature, pressure) and actors communicates with a server application that is running the device communication protocol, device configuration, storage of data sent by the devices as well as the application business logic and the presentation layer.

VictoriaMetrics is a fast and scalable open source time series database and monitoring solution which exists in a Single and in a cluster version. It is compatible with Prometheus pull model and supports a wide variety of ingestion protocols: Influx, Graphite, Prometheus remote_write, Prometheus exposion format, OpenTSDB put message, JSON line format, Arbitrary CSV data, native binary formant, DataDog agent or DogStatsD; as way as many ways to query data via PromQL or MetricsQL from Grafana or own VMUI.

Kestra is an infinitely scalable orchestration and scheduling platform, creating, running, scheduling, and monitoring millions of complex pipelines.

InfluxDB is an open source time series database, purpose-built by InfluxData for monitoring metrics and events, provides real-time visibility into stacks, sensors, and systems. Use InfluxDB to capture, analyze, and store millions of points per second, meet demanding SLA’s, and chart a path to automation.

Grafana is a tool that allows you to query, visualize, alert on and understand your metrics no matter where they are stored.

Prometheus is a free software application used for event monitoring and alerting. It records real-time metrics in a time series database (allowing for high dimensionality) built using a HTTP pull model, with flexible queries and real-time alerting.

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream.

Thanos is a set of components that can be composed into a highly available metric system with unlimited storage capacity, which can be added seamlessly on top of existing Prometheus deployments.

Wyze is a great security and monitoring application to live stream HD video from the security cameras from anywhere in the world.

Uptime Kuma is a fancy self-hosted monitoring tool.

Gatus is a developer-oriented health dashboard that gives you the ability to monitor your services using HTTP, ICMP, TCP, and even DNS queries as well as evaluate the result of said queries by using a list of conditions on values like the status code, the response time, the certificate expiration, the body and many others.

Upptime is the open-source uptime monitor and status page, powered entirely by GitHub Actions, Issues, and Pages.

HertzBeat is an open-source, real-time monitoring system with custom-monitor and agentless. It supports web service, database, os, middleware and more.

Tautulli is a python based web application for monitoring, analytics and notifications for Plex Media Server.

Flower is a web based tool for monitoring and administrating Celery clusters.

Weave Scope is a tool for Troubleshooting & Monitoring for Docker & Kubernetes. It automatically generates a map of your application, enabling you to intuitively understand, monitor, and control your containerized, microservices-based application.

Statping (Status Page & Monitoring Server) is an easy to use Status Page for your websites and applications. Statping will automatically fetch the application and render a beautiful status page with tons of features for you to build an even better status page.

Vector is a high-performance, end-to-end (agent & aggregator) observability data pipeline that puts you in control of your observability data. Collect, transform, and route all your logs, metrics, and traces to any vendors you want today and any other vendors you may want tomorrow.

Open Service Mesh (OSM) is a lightweight, extensible, cloud native service mesh that allows users to uniformly manage, secure, and get out-of-the-box observability features for highly dynamic microservice environments.

Ciao is a tool that checks HTTP(S) URL endpoints for a HTTP status code (or errors on the lower TCP stack) and sends a notification on status change via E-Mail or Webhooks.

Server is a simple server for sending and receiving messages in real-time per WebSocket.

Ngxtop is a real-time metrics for nginx server (and others).

Blocky is a fast and lightweight DNS proxy as ad-blocker for local network with many features

Dashy is a self-hostable personal dashboard built for you. Includes status-checking, widgets, themes, icon packs, a UI editor and tons more.

Restic is a modern backup program that can back up your files: from Linux, BSD, Mac and Windows. To many different storage types, including self-hosted and online services. easily, being a single executable that you can run without a server or complex setup. effectively, only transferring the parts that actually changed in the files you back up.

Autorestic is a wrapper around the amazing restic. While being amazing the restic cli can be a bit overwhelming and difficult to manage if you have many different locations that you want to backup to multiple locations.

MinIO is a high performance object storage tool that provides the world’s fastest object storage server. With READ/WRITE speeds of 325 GiB/s and 165 GiB/s on standard hardware with default parity (EC:4), object storage can operate as the primary storage tier for a diverse set of workloads ranging from Spark, Presto, TensorFlow.

Greyhole is a tool that uses Samba to create a storage pool of all your available hard drives (whatever their size, however they are connected), and allows you to create redundant copies of the files you store, in order to prevent data loss when part of your hardware fails.

Falcon LogScale is purpose-built to help any organization achieve the benefits of large-scale logging and analysis. Falcon LogScale has virtually no latency, even at ingest volumes of 1PB(Petabyte) per day.

Googerteller is a tool that makes an audible sound any time your computer sends a packet to a Google tracker or a Google service, which excludes Google Cloud users.

TeslaMate is a powerful, self-hosted data logger for your Tesla.

OneUptime is an open-source complete SRE and DevOps platform. It monitors your website, dashboards, API’s, and more and alerts your team when downtime happens.

Parca is a tool for continuous profiling for analysis of CPU and memory usage, down to the line number and throughout time. Saving infrastructure cost, improving performance, and increasing reliability.

DeviceHive is a free, highly scalable open-source IoT platform for data collection, processing and analysis, visualization, and device management with the broad range of integration options.

Distributed Services Architecture (DSA) is an open source IoT platform that facilitates device inter-communication, logic and applications at every layer of the Internet of Things infrastructure. The objective is to unify the disparate devices, services and applications into a structured and adaptable real-time data model.

IoTivity is an open source software framework enabling seamless device-to-device connectivity to address the emerging needs of the Internet of Things.

Eclipse IoT Project provides open source technology that will be used to build IoT solutions for industry and consumers.

Dashboards

Back to The Top

Adagios is a Web based Nagios configuration interface.

Dash is a low-overhead monitoring web dashboard for a GNU/Linux machine.

Thruk is a Multibackend monitoring web interface with support for Naemon, Nagios, Icinga and Shinken.

Uchiwa is a simple dashboard for the Sensu monitoring framework.

Grafana is a tool that allows you to query, visualize, alert on and understand your metrics no matter where they are stored.

Analytics

Back to the Top

Plausible Analytics - Simple, open-source, lightweight (< 1 KB) and privacy-friendly web analytics.

PostHog - Product analytics, session recording, feature flagging and a/b testing that you can self-host.

Ackee - Self-hosted analytics tool for those who care about privacy.

AWStats - Generate statistics from web, streaming, ftp or mail server logfiles.

Chartbrew - Web application that can connect directly to databases and APIs and use the data to create beautiful charts.

Countly Community Edition - Real time mobile and web analytics, crash reporting and push notifications platform.

Druid - Distributed, column-oriented, real-time analytics data store.

EDA - Web application for data analysis and visualization.

GoAccess - Real-time web log analyzer and interactive viewer that runs in a terminal.

GoatCounter - Easy web statistics without tracking of personal data.

Metabase - Easy, open-source way for everyone in your company to ask questions and learn from data.

Offen - Fair, lightweight and open web analytics tool. Gain insights while your users have full access to their data.

Open Web Analytics - Web analytics framework that lets you stay in control of how you instrument and analyze the use of your websites and applications.

Redash - Connect and query your data sources, build dashboards to visualize data and share them with your company.

RudderStack - Collect, unify, transform, and store your customer data, and route it to a wide range of common, popular marketing, sales, and product tools. alternative to Segment.

Shynet - Modern, privacy-friendly, and detailed web analytics that works without cookies or JS.

Superset - Modern data exploration and visualization platform.

Umami - Simple, fast, privacy-focused alternative to Google Analytics.

Search

Back to the Top

Meilisearch is a lightning-fast search engine that fits effortlessly into your apps, websites, and workflow.

Shodan is the world’s first search engine for Internet-connected (IoT) devices.

Whoogle Search is a self-hosted, ad-free, privacy-respecting metasearch engine.

SearX is a Privacy-respecting, hackable metasearch engine.

SearXNG is a free internet metasearch engine which aggregates results from various search services and databases.

Sonic is a fast, lightweight & schema-less search backend. An alternative to Elasticsearch that runs on a few MBs of RAM.

Zinc is a search engine that does full text indexing. It is a lightweight alternative to Elasticsearch and runs using a fraction of the resources.

Cylect.io is the ultimate searching tool that is here to assist anyone looking for specific information through vast amounts of websites, search engines, and data collectors.

Lyra is a fast, in-memory, typo-tolerant, full-text search engine written in TypeScript.

Hugo Lyra is a typescript module for creating LyraSearch indexes for static Hugo sites, it comes with server and client libraries.

Typesense is a fast, typo-tolerant search engine for building delightful search experiences.

Tantivy is a full-text search engine library inspired by Apache Lucene and written in Rust.

Toshi is meant to be a full-text search engine similar to Elasticsearch. Toshi strives to be to Elasticsearch what Tantivy is to Lucene.

FlexSearch is a Next-Generation full text search library for Browser and Node.js.

fd is a program to find entries in your filesystem. It is a simple, fast and user-friendly alternative to find.

k8s at home search is a tool that indexs Flux HelmReleases from Github repositories with the k8s-at-home topic on GitHub.

OpenFind is an app to find text in real life. Easily search your entire photo library in split seconds. This runs 100% offline. No servers, nothing weird going on.

Notifications

Back to the Top

Apprise is a tool that allows you to send a notification to almost all of the most popular notification services available to us today such as: Telegram, Discord, Slack, Amazon SNS, Gotify, etc.

ntfy is a simple HTTP-based pub-sub notification service. It allows you to send notifications to your phone or desktop via scripts from any computer, entirely without signup, cost or setup. It’s also open source if you want to run your own.

Countly is a product analytics solution and innovation enabler that helps teams track product performance and customer journey and behavior across mobile, web, and desktop applications. Ensuring privacy by design, Countly allows you to innovate and enhance your products to provide personalized and customized customer experiences, and meet key business and revenue goals.

notifiers is a general wrapper for a variety of 3rd party providers and built in ones (like SMTP) aimed solely at sending notifications.

Pushover is a tool that makes it easy to get real-time notifications on your Android, Android Wear, iPhone, iPad, Apple Watch and Desktop.

Simplepush is a tool to send end-to-end encrypted push notifications to your Android and iPhone.

UnifiedPush is a set of specifications and tools that lets the user choose how push notifications are delivered. All in a free and open source way.

RSS

Back to the Top

RSS Guard is a simple RSS/ATOM feed reader for Windows, Linux, BSD, OS/2 or macOS which can work with RSS/ATOM/JSON feeds as well as many online feed services:

Feedly
Gmail
Google Reader API (Bazqux, FreshRSS, Inoreader, Miniflux, Reedah, The Old Reader and more)
Nextcloud News
Tiny Tiny RSS

Feedly is an RSS tool where you can privately organize and research the topics and trends that matter to you. It offers useful integrations with Facebook, Twitter, Evernote, Buffer, OneNote, Pinterest, LinkedIn, IFTTT, and Zapier so that you can easily share stories with your networks and teammates.

FreshRSS is a self-hosted RSS and Atom feed aggregator. It is lightweight, easy to work with, powerful, and customizable.

ArchiveBox is a powerful, self-hosted internet archiving solution to collect, save, and view sites you want to preserve offline. It takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more.

RSSHub is an open source, easy to use, and extensible RSS feed generator. It’s capable of generating RSS feeds from pretty much everything.

Miniflux V2 is a minimalist and opinionated feed reader.

Websites/Blogs

Back to the Top

Lyra is a fast, in-memory, typo-tolerant, full-text search engine written in TypeScript.

Hugo Lyra is a typescript module for creating LyraSearch indexes for static Hugo sites, it comes with server and client libraries.

Kopage is a self-hosted Website Builder. It’s compatible with cPanel and other popular hosting control panels. Compatible with cPanel and other popular hosting control panels.

Ghost is a fully-managed PaaS & self-hosted open source software, and can be installed and maintained relatively easily on just about any VPS hosting provider.

Cloudron is a self-hosted immutable infrastructure design allows easy migration of apps across servers. In fact, you can move your entire server along with all its apps to another cloud provider in no time.

Directus is a real-time API and App dashboard for managing SQL database content.

Haven is a Self-hosted private blog instead of using Facebook.

Antville is an open source project aimed at the development of a simple site hosting system with many advanced features.

October is a Self-hosted Content Management System (CMS) and web platform whose sole purpose is to make your development workflow simple again.

Grav is a Fast, Simple, and Flexible, file-based Web-platform. There is Zero installation required. Just extract the ZIP archive, and you are already up and running. It comes with a powerful Package Management System to allow for simple installation and upgrading of plugins and themes, as well as simple updating of Grav itself.

Orchard is a free, open source, community-focused Content Management System built on the ASP.NET MVC platform.

Netlify CMS is a CMS for static site generators. Give users a simple way to edit and add content to any site built with a static site generator.

Zola is a fast static site generator in a single binary with everything built-in.

FlatPress is a lightweight, easy-to-set-up blogging engine.

Chyrp Lite is an ultra-lightweight blogging engine. It provides four beautiful blog themes and a friendly administration console, all fully navigable on a broad range of devices, thanks to the power of responsive HTML5.

WriteFreely is an open source platform for building a writing space on the web.

Sandstorm is an open source project built by a community of volunteers with the goal of making it really easy to run open source web applications.

YunoHost is a Debian-based distribution which strives to make it easy to quickly set up a server and host web applications.

Back to the Top

Mattermost is a secure, open source platform for communication, collaboration, and workflow orchestration across tools and teams.

Mastadon is a a decentralized social media platform that supports audio, video and picture posts, accessibility descriptions, polls, content warnings, animated avatars, custom emojis, thumbnail crop control, and more, to help you express yourself online.

Telegram is a cross-platform, cloud-based instant messaging service. It has an open API and source code free for everyone. Telegram also provides end-to-end encrypted video calling, VoIP, file sharing and several other features.

ActivityPub is a decentralized social networking protocol based on the ActivityStreams 2.0 data format. It provides a client to server API for creating, updating and deleting content, as well as a federated server to server API for delivering notifications and subscribing to content.

Lemmy is similar to sites like Reddit, Lobste.rs, or Hacker News. Where you subscribe to forums you’re interested in, post links and discussions, then vote, and comment on them. Behind the scenes, it is very different; anyone can easily run a server, and all these servers are federated, and connected to the same universe, called the Fediverse.

Lemmy-UI is the official web app for Lemmy, written in inferno.

Mlem is a Lemmy client for iOS.

Jerboa is an Android client for Lemmy, a federated reddit alternative.

GoToSocial is an ActivityPub social network server, written in Golang.

Berty is a secure peer-to-peer messaging app that works with or without internet access, cellular data or trust in the network.

Pleroma is a free and open communication for everyone. Pleroma is social networking software compatible with other Fediverse software such as Misskey, Pixelfed, Mastodon and many others.

Matrix is a tool that gives you simple HTTP APIs and SDKs (iOS, Android, Web) to create chatrooms, direct chats and chat bots, complete with end-to-end encryption, file transfer, synchronised conversation history, formatted messages, read receipts and more.

Element is a Matrix web client built using the Matrix React SDK.

Nostr(Notes and Other Stuff Transmitted by Relays) is a truly censorship-resistant alternative to Twitter that has a chance of working.

Fritter is an open source frontend for Twitter on mobile devices, focusing on giving you the best experience and keeping your data private, local and in your hands.

Nitter is a free and open source alternative Twitter front-end focused on privacy and performance. All requests go through the backend, meaning the client never talks to Twitter and prevents Twitter from tracking your IP or JavaScript fingerprint.

Diaspora is a privacy-aware, distributed, open source social network.

Hubzilla is a general purpose communication server integrated with a web publishing system and a decentralised permission system.

Expanse is a fully selfhosted multi-user web app for externally storing Reddit items (saved, created, upvoted, downvoted, hidden) to bypass Reddit’s 1000-item listing limits.

Apollo is a beautiful Reddit app built for fast navigation with an incredibly powerful set of features.

Infinity is a Reddit client on Android written in Java. It does not have any ads and it features a clean UI and smooth browsing experience.

RedReader is an unofficial open source Reddit client for Android.

Nostr

Back to the Top

Nostr (Notes and Other Stuff Transmitted by Relays) is a protocol, designed for simplicity, that aims to create a censorship-resistant global social network. The protocol is based on very simple & flexible event objects (which are passed around as plain JSON) and uses standard elliptic-curve cryptography for keys and signing.

nostr - overview and FAQ.
NIPs - the “Nostr Implementation Possibilities” describe the protocol in technical detail.
nostr, a basic tour - an intro to nostr.
UseNostr - A small guide for anyone who wants to learn more about how nostr works and what it can do.
nostr.how - Quick-start to onboard desktop users with Alby & Astral.
nostr.guide - A guide to all things nostr.
nostr address book - A directory of twitter users accounts and their NOSTR addresses.
NNostr - a C# relay.
nostr-rs-relay - a minimalistic relay written in Rust that saves data on SQLite.
Relayer Basic - a simple relay based on relayer backed by Postgres.
nodestr - a Node.js implementation.
sovereign-stack - a tool that helps you deploy nostr relays and create self-hosted (bitcoin-only) Value4Value websites.
expensive relay - a relay that requires payment for registration
me.untethr.nostr-relay - a relay written in Clojure
Minds Nostr Relay - a relay for Minds, an open-source social network
- Minds Engine - Nostr - relevant Minds API code for reading/writing Minds posts using Nostr
NostrPostr Relay - a Kotlin Relay supporting both SQLite and Postgresql
nostrpy - relay, client, and other tooling in python
nostream - a nostr relay written in Typescript backed by PostgreSQL (renamed from nostr-ts-relay)
nostr_relay – a nostr relay written in python, backed by SQLite
søstr – a private nostr relay written in rust, saves all notes from one pubkey and publish them to anyone that requests them
knostr – a nostr relay implemented in Kotlin with support for Postgres and metrics(micrometer).
PyRelay – a python implementation of a nostr relay, using asyncio.
strfry – C++ implementation backed by LMDB with efficient syncing of events using merkle trees
Astro – Elixir based implementation built to be performant and highly distributed.
Nex - A powerful and scalable Nostr relay written in Elixir with Postgres DB.
gnost-relay - nostr relay written in go backed by postgresql database.
nostring - A Nostr relay written in Deno.
Denostr - Deno based, cloud native nostr implemention support by ByteTrade and Revo.
nostr-relay-nestjs- A Nostr relay implemented using the NestJS framework
Ephemerelay - An in-memory Nostr relay that doesn’t store data.
Servus - A self-contained, single executable, CMS / blogging engine reminiscent of Jekyll which also acts as a personal Nostr relay for your blog posts. Written in Rust.
nostr relay registry - real-time checking of status of some known relays.
nostr.info - real-time checking of status of some known relays.
nostr.watch - real-time checking of status of some known relays.
Astral - a branle fork with global feed and UI makeover
damus - a twitter-like nostr client for iOS and MacOS.
more-speech - desktop client for nostr written in Clojure.
futr - nostr client desktop app written in Haskell.
Minds - open source social network. Supports reading and creating posts using the Nostr protocol.
Jester - Chess over nostr.
- Jester instance
Sendstr - shared clipboard between devices over nostr.
nosbin - pastebin over nostr.
noscl - a basic command-line client written in Go.
loquaz - a desktop app written in Rust for direct encrypted chat.
nostr console - a nostr command line client written in Dart. Binaries available for Windows, Linux, and MacOS.
ArcadeCity - Public group chats and P2P services (WIP) over nostr.
second exchange - an experiment to work out something of like medium, something of creator economy where users are rewarded for engaging in quality discussion and most importantly engaging in governance-related discussion.
scalastr - A barebones nostr client written in scala.
Nostros - A nostr mobile client for Android.
Nostrify.me - Nostr client built in SvelteKit.
NostrEmitter - Simple E2E encrypted client and EventEmitter object
Lightning.Pub - A nostr daemon for Lightning nodes.
shockwallet - A Lightning wallet that uses nostr and lnurl to connect to nodes.
coracle - A nostr web client.
nostrweb - another nostr web client in vanilla JS.
- nostr.ch - live instance
Bija - A desktop client written in python. Currently Linux only.
Nosky - A native Android client for Nostr. Still in development.
Stackerstan - A decentralised organisation built on Bitcoin and Nostr, implemented as a replicated state machine in Golang.
nostr-java - A nostr client API written in java, for generating, signing and publishing events to relays.
bolt.fun - A bitcoin lightning makers community that supports reading and creating comments using Nostr.
iris - A nostr web client.
- iris.to - live instance
- Android app
gossip - A desktop client in rust presented with egui.
Attached - Open-Source ReactNative Expo app for Nostr (iOS, Android). Currently under app stores review.
Member - Progressive Web App Client. Works on desktop and mobile.
- member.cash - live instance
dispute - A cross-platform (Linux, Android, iOS, MacOs, Windows and Web) client for NOSTR
Snort - Nostr UI written in react
Hamstr - A twitter-style web client built with Vue.js
Nozzle - A Twitter-like native Android client written with Jetpack Compose
electron-nostr - A bare-bones desktop nostr client using electron-react-boilerplate. Goal is to be an easy template for people to experiment with different ideas on decentralized ratings, reputation, and web of trust.
Nostrid - Multi-platform client currently offering binaries for Android, Windows, MacOS and Linux.
- Nostrid.Web - Web version running completely on the browser. It can be installed locally as PWA.
nostr-chat-widget-react - A React component that provides a live-chat widget over nostr that can be embedded into any website.
Blockcore Notes - Progressive Web App that can be installed on mobile and desktop, organize following in circles and have both public and private following lists. Dynamic interface for different uses, such as optimized for photograph viewing.
Noteon- Yet another nostr client focused on private chat and group chat with a simple, elegant and secure interface.
emon - Encrypted DMs over nostr with lightning payments integrated (WIP).
notebin.org - Nostr UI created with NextJS, support for markdown and code highlighting.
Daisy - Mobile client for Android and iOS.
Flycat - A 2000s old-school style web client which support blogging on Nostr.
Amethyst - An Android client for nostr written in Kotlin.
MeShell - Web, iOS and Android blog type client destined to publish articles and researches for independent journalists.
Disgus - A comment widget like Disqus, but for Nostr.
Tamga - An offline first nostr contact & profile manager for iOS!
nostromat- A Twitter-style Nostr web client, written in Clojurescript/React.
nostrom.at - live instance.
blogstack.io - Blogging site for nostr, supports markdown.
Votestr - Poll web app with nostr authentication and blind signature unlinkability.
gnost-deflate-client- A CLI nostr client written in go implementing permessage-deflate websocket compression.
algia - A cli application for nostr.
algia-web - A small resource consumption oriented Nostr web client.
Blowater - A desktop Web client focusing on chat with delightful UX.
Written Self hosted blog using nostr long-form content (NIP-23) and it shows only posts by selected authors.
Nostr Nests - Nostr Nests is an audio space for chatting, brainstorming, debating, jamming, micro-conferences and more.
nblog - a self-host nostr ghost blog
Nostribe.com - Nostr client web app built with Next.js 13 and TypeScript.
- Nostribe.com - Live instance.
Nostrtium - Post to Nostr directly from within WordPress
uBlog - A minimalist blog on nostr that allows anyone to easily create their own personal micro-blog.
Listr - A Nostr Web Client for Making Lists, built with Next.js 13 and TypeScript.
nostr-ruby - a Ruby implementation of the nostr protocol.
nostr - a Ruby Nostr gem for use by clients.
NNostr.Client - a C# Nostr library for use by clients.
nostr-tools - a JavaScript client that abstracts the relay management code for use by clients.
nostr-relaypool-ts - a TypeScript relay pool library on top of nostr-tools that simplifies handling subscriptions to multiple servers.
nostr-react - React Hooks for Nostr.
go-nostr - a Go library that implements relay management, plus event encoding and signing utils.
nostr_rust - Functional Rust implementation of the nostr protocol.
nostr-js - a javascript implementation of the nostr protocol.
nostr-rs - a Rust implementation of the nostr protocol
nostr
- nostr: Rust implementation of Nostr protocol.
- nostr-sdk: High level client library.
- bindings: UniFFI (Kotlin, Swift, Python, Ruby) bindings.
relayer - a server framework for writing custom relays.
NostrPostr - a Kotlin Nostr library for clients or relays.
python-nostr - a python library for making clients.
nostr-bot - a Rust library for writing bots.
NostrKit - a Swift library for interacting with relays.
nostr-relay-inspector - A library that returns useful information about relays based on nostr-js.
schorr_snap - A snap plugin for Metamask Flask, supports nostr.
nostr-deno - a client library for Deno javascript runtime.
nostr-types - a rust library defining types useful for the nostr protocol.
dart-nostr - a Dart library for Flutter.
nostr-connect - Nostr Connect SDK for TypeScript is a library that allows you to easily integrate Nostr Connect into your web application.
pynostr - a python library for nostr.
nostr-php - a PHP library for nostr.
smtp nostr gateway - a bridge that forwards emails to pubkeys as encrypted direct messages.
matrix-nostr-bridge - a simple Matrix-to-Nostr or Nostr-to-Matrix bridge.
Mostr - a bridge between Nostr and the Fediverse (Mastodon, ActivityPub, etc.).
nostrich.fun - A feature-rich directory of nostr projects. A fork of LightningNetworkStores.com
git-nostr-tools - A cli tool for sending code patches over nostr.
nostr-cln-events - A CLN plugin to push clightning node events to nostr.
nostr registry - a database of known relays with their uptime and NIP support tables
nostr-fzf - Nostr Directory; a tool for searching usernames and channels.
nostr-notify - desktop nostr notifications using libnotify.
nostr-launch - A tool for launching a bunch of relays and clients locally for development and testing.
nostr GitHub Action - send events from GitHub Actions.
nostrefresh - A simple refresh function for nostr web pages.
anonroom - anonymous chat room inside nostr.
nostril - A C cli tool for creating nostr events.
nostr-rs-relay-compose - A Docker compose deployment for nostr-rs-relay with SSL support based on Traefik.
nostr.guru - a nostr web gateway for viewing events by their ID.
nostrandom.netlify.app - generate publish-able Nostr event with random keys.
nashboard - A Nostr network dashboard with network statistics, reachable here.
ndxstr - nostr’s layer 2 indexing nodes, with more advanced querying capability than currently supported by relays.
nostrillery - A tool for running performance tests against Nostr relays.
nostr-terminal - A SSH-like access to your machine via web terminal, powered by Nostr.
nostreq - A Nostr relay event request generator.
nostr.io - A network statistics with last published notes, top 50 publishers, and top 50 followed users.
nostr-commander - A simple but convenient CLI-based Nostr app for following users, sending DMs, etc.
nostr.directory - A searchable database of nostr users and their other social media links.
nostr-tool - A Rust CLI tool to generate and publish events.
frostr - Create joint nostr identities and require t-of-n signatures to post.
nostr.rest - Mine proof of work public keys with user specified prefixes.
lnpass - A key manager for Lightning and nostr.
sb.nostr.band - Search bots that you can create and follow to receive new posts matching a keyword right into your feed.
rss.nostr.band - Create custom RSS feeds with posts matching your keywords and consume using your favorite RSS reader.
nostrview - A nostr search engine. Search by content, tags, events or pub keys.
nostr-bulk-dms - A tool that allows you to send DMs over nostr to many recipients in bulk.
nostrify - A Core Lightning plugin that sends events (forwards, connect, disconnect, etc.) to nostr.
nip06-web - a website to generate or restore NIP-06 seed phrases
nip06-cli - a Node.js CLI to generate or restore NIP-06 seed phrases.
nostr-broadcast - A tool lets you take your events from some relays and broadcast them to another relay. Could be helpful for backing up your notes to a private relay.
nostr-follow-bundler - A tool lets you create lists of profiles that other users can then see and follow themselves.
nostr-proxy - Push and get events to your Proxy, get results from multiple Nostr relays.
nostrends - Trending on Nostr, like Twitter trends. Live at nostrends.vercel.app.
homebrew-nostr - Homebrew tap for Nostr software.
heyxynip5 - A CLI helper for converting nostr npub/nsec to their hex format for NIP-05 verification.
http-nostr-publisher - A Cloudflare worker to publish Nostr events to relays through a non-blocking HTTP interface .
blastr - A nostr cloudflare workers proxy relay that publishes to all known relays.
keystr-rs - An application for managing Nostr keys. Written in Rust, with simple UI (Iced).
nostr_simple_publish - Drupal module to publish content to Nostr.
nostr-spam-detection - An experiment in building a machine learning model to label Nostr spam content for filtering and relay rejection.
blogsync - Self-host blog articles from long-form notes e.g. via Caddy server.
NostrFlu - A tool to collect and resend following lists from relays. You can also check badges.
nkcli - A CLI tool for nostr key manage and serve NIP-46.
git-nostr- A tool to enhance git cli with nostr communications.
nostr-wtf - A set of nostr tools available and deployed on a web app including a pubkey converter and relay query tool.
strfry policies- A collection of moderation & antispam policies for the strfry relay developed in TypeScript/Deno.
nostrum - Nostrum it’s a mobile app that allows you to sign transactions and messages with your Nostr keys. Nostrum is the reference implementation for a remote signer app (ie. Wallet) of the Nostr Connect protocol.
nostr-signing-device - Signing device for Nostr built on ESP32.

iMessage

Back to the Top

iMessage-exporter is a binary exports iMessage data to txt or html formats. It can also run diagnostics to find problems with the iMessage database.

pypush is a POC demo of my recent iMessage reverse-engineering. It can currently register as a new device on an Apple ID, set up encryption keys, and send and receive iMessages!

Self-Host Beeper is a self-hosted universal chat app that can chat with your friends on iMessage from your Android device using your phone number. You can also join iMessage Group Chats with your phone number and blue bubbles, and share full-resolution images, videos, and audio.

Beeper Mini is an Andorid app that can chat with your friends on iMessage from your Android device using your phone number. You can also join iMessage Group Chats with your phone number and blue bubbles, and share full-resolution images, videos, and audio.

Beeper Bridge Manager is a tool for running self-hosted bridges with the Beeper Matrix server.

Matrix Ansible and Docker Deploy is a Matrix (An open network for secure, decentralized communication) server setup using Ansible and Docker.

Communications

Back to the Top

Postmoogle is an actual SMTP server that allows you to send and receive emails on your matrix server. It can’t be used with arbitrary email providers, because it acts as an actual email provider itself, so you can use it to send emails from your apps and scripts as well.

SimpleX is a privacy redefined messenger without user IDs. Other apps have user IDs: Signal, Matrix, Session, Briar, Jami, Cwtch, etc. SimpleX does not, not even random numbers.

Element is a Matrix web client built using the Matrix React SDK.

Mattermost is a secure, open source platform for communication, collaboration, and workflow orchestration across tools and teams.

Berty is a secure peer-to-peer messaging app that works with or without internet access, cellular data or trust in the network.

Pleroma is a free and open communication for everyone. Pleroma is social networking software compatible with other Fediverse software such as Misskey, Pixelfed, Mastodon and many others.

ffsend is a easily and securely share files from the command line. A fully featured Firefox Send client.

Nostr(Notes and Other Stuff Transmitted by Relays) is a truly censorship-resistant alternative to Twitter that has a chance of working.

Diaspora is a privacy-aware, distributed, open source social network.

Hubzilla is a general purpose communication server integrated with a web publishing system and a decentralised permission system.

Expanse is a fully selfhosted multi-user web app for externally storing Reddit items (saved, created, upvoted, downvoted, hidden) to bypass Reddit’s 1000-item listing limits.

giscus is a comments system powered by GitHub Discussions. Let visitors leave comments and reactions on your website via GitHub.

Mailroute is a great tool that provides the best email filtering & security( CMMC, NIST 800-171, DFARS, DISA, HIPPA). It protects your inbox, stop spam, viruses, ransomware, security threats & more with email filtering services. With an easy setup on Office 365, Google & more.

Docker Mailserver is a production-ready fullstack but simple mail server (SMTP, IMAP, LDAP, Antispam, Antivirus, etc.) running inside a container. Only configuration files, no SQL database.

Diun is a CLI application written in Go and delivered as a single executable (and a Docker image) to receive notifications when a Docker image is updated on a Docker registry.

iRedMail is a self-hosted email server.

iRedMail Easy is a web-based deployment platform, it offers an easy to use web interface to help you deploy iRedMail server, keep your server up to date, also get fast and professional technical support from iRedMail team.

Spider Email Archiver is an On-Premises Email Archiving Software.

MailCow is a self-hosted email server.

Nextcloud Talk is a on-premises, private audio/video conferencing and text chat through browser and mobile interfaces with integrated screen sharing and SIP integration.

Poste.io Email Server is self-hosted SMTP + IMAP + POP3 + Antispam + Antivirus Web administration + Web email. It is easy setup with a DNS guide for protect from spam.

Business Management

Back to the Top

Nextcloud is a suite of enterprise client-server software for creating and using file hosting services. It offers an on-premise Universal File Access and sync platform with powerful collaboration capabilities and desktop, mobile and web interfaces.

Odoo is a suite of open source business apps that cover all your company needs: CRM, eCommerce, accounting, inventory, point of sale, project management, etc.

Kanboard is project management software that focuses on the Kanban methodology.

Eden Workplace is a complete workplace management platform that lets you achieve more. Desk Booking Software to make desk reservations easier for your team, including assigning permanent and hybrid desks, providing wayfinding solutions for employees.

Matomo is an ethical alternative where you won’t make privacy sacrifices or compromise your site. Matomo is the Google Analytics alternative that protects your data and your customer’s privacy.

Plausible Analytics is a simple, lightweight (< 1 KB), open-source and privacy-friendly alternative to Google Analytics. It doesn’t use cookies and is fully compliant with GDPR, CCPA and PECR. You can self-host Plausible or have us run it for you in the Cloud.

InvoicePlane is a self-hosted open source application for managing your quotes, invoices, clients and payments.

Collaboration & Synchronization

Back to the Top

Syncthing is a continuous file synchronization program. It synchronizes files between two or more computers in real time.

Synology is a tool that allows you to easily access and manage files in your Synology Drive on the go. Apart from common file types, such as documents, images, videos and music, you can also open Synology Office document, spreadsheets and slides in the user-friendly viewer provided by Drive.

Nextcloud is a suite of client-server software for creating and using file hosting services. It offers an on-premise Universal File Access and sync platform with powerful collaboration capabilities and desktop, mobile and web interfaces.

Lsyncd (Live Syncing Mirror Daemon) is a tool used in Linux systems to keep directories synchronized. These directories can be found locally, within the same machine, or remotely, on different machines. For remote synchronization, this article focuses on using SSH to accomplish it.

FileRun is a self-hosted Google Drive alternative. It is a full featured web based file manager with an easy to use user interface.

FileBrowser provides a file managing interface within a specified directory and it can be used to upload, delete, preview, rename and edit your files. It allows the creation of multiple users and each user can have its own directory.

Rsync is a utility in the command line which enables users to transfer and synchronize files efficiently between a computer and an external hard drive in the entire connected network.

Warpinator is a free, open-source tool for sending and receiving files between computers that are on the same network.

LocalSend is a free and open-source tool that allows you to send files and messages over the local LAN network to nearby devices. Everything is sent securely over HTTPS. The TLS/SSL certificate is generated on the fly on each device. It’s avilable on Windows, macOS, Linux, iOS, and Android.

FileZilla Client is a fast and reliable cross-platform FTP, FTPS and SFTP client with lots of useful features and an intuitive graphical user interface.

Dragit is an application for intuitive file sharing between devices. It’s useful for when you want to send file from one computer to another with minimal effort. Dragit automatically detects devices in the local network with help of mDNS protocol and allows you to send file immediately.

WinFsp is a set of software components for Windows computers that allows the creation of user mode file systems. In this sense it is similar to FUSE (Filesystem in Userspace), which provides the same functionality on UNIX-like computers.

SSHFS-Win is a minimal port of SSHFS to Windows. Looking under the hood it uses Cygwin for the POSIX environment and WinFsp for the FUSE (Filesystem in Userspace) functionality.

RiftShare is a cross platform (Windows, MacOS, Linux) file sharing tool that supports fully encrypted transfers both on the local network and off network using a simple passphrase. RiftShare uses magic-wormhole under the hood and is compatible with other magic-wormhole clients. It is also fully open source and licensed under the GPLv3.

Usermode FTP Server is a tool that let’s you start an FTP server as user and transfer files with any FTP client. Allowing you to access your files directly with many file browsers’ builtin FTP support: Windows File Explorer, Thunar, Gnome Files, Dolphin and many more.

TagSpaces is a free, no vendor lock-in, open source application for organizing, annotating and managing local files with the help of tags. It features advanced note taking functionalities and some capabilities of to-do apps. It’s available for Windows, Linux, Mac OS and Android.

Listmonk is a standalone, self-hosted, newsletter and mailing list manager. It is fast, feature-rich, and packed into a single binary.

Encryption

Back to the Top

VeraCrypt is free open-source disk encryption software for Windows, Mac OS X and Linux. The file encryption, data encryption performed by VeraCrypt is real-time (on-the-fly), automatic, transparent, needs very little memory, and does not involve temporary unencrypted files.
AxCrypt is an inexpensive and effective encryption tool for Windows, macOS, iOS, and Android.

AESCrypt is an advanced file encryption utility that integrates with the Windows shell or runs from the Linux command prompt to provide a simple, yet powerful, tool for encrypting files using the Advanced Encryption Standard (AES). It is available for Windows, MacOS, and Linux.

Linux Unified Key Setup (LUKS) is a disk encryption specification created by Clemens Fruhwirth in 2004 and was originally intended for Linux. It uses device mapper crypt ( dm-crypt) as a kernel module to handle encryption on the block device level.

GNU Privacy Guard (GnuPG) is a complete and free implementation of the OpenPGP standard as defined by RFC4880 (also known as PGP ). It allows you to encrypt and sign your data and communications; it features a versatile key management system, along with access modules for all kinds of public key directories.

Pretty Good Privacy (PGP) is an encryption program that provides cryptographic privacy and authentication for data communication. It’s used for signing, encrypting, and decrypting texts, e-mails, files, directories, and whole disk partitions and to increase the security of e-mail communications.

Deadbolt is a Dead-simple file encryption for any OS.

Infisical is an open-source, end-to-end encrypted platform to sync secrets and configs across your team and infrastructure.

Hemmelig.app is a tool that keeps your sensitive information out of chat logs, emails, and more with encrypted secrets.

How Encryption Keys work

Symmetric is a data encryption method whereby the same private key is used to encode and decode information.
Asymmetric is a data encryption method that allows users to encrypt information using shared keys. For example, if you need to send a message across the internet, but you don’t want anyone but the intended recipient to see what you’ve written.

Types of Encryption

Triple DES (Triple Data Encryption Algorithm) is a symmetric-key block cipher, which applies the DES cipher algorithm three times to each data block(contains 64 bits of data).
AES (Advanced Encryption Standard) is an algorithm that encrypts and decrypts data in blocks of 128 bits. It can do this using 128-bit, 192-bit, or 256-bit keys.
RSA (Rivest–Shamir–Adleman) is a type of public-key cryptography used for secure data transmission of e-mail and other digital transactions over the Internet.
Twofish is a symmetric key block cipher with a block size of 128 bits and key sizes up to 256 bits. It is an advanced version of Blowfish encryption.
Format Preserving Encryption (FPE) is a valid encryption algorithm to be used for compliance with NIST standards. It is mostly used in on-premise encryption and tokenization solutions.

Application Level Encryption

Hashes is a function that converts an input of letters and numbers into an encrypted output of a fixed length. For example, algorithms such as MD5 (Message Digest 5) or SHA (Secure Hash Algorithm).
Digital Certificates is a file that verifies the identity of a device or user and enables encrypted connections. A digital signature is a hashing approach that uses a numeric string to provide authenticity and validate identity. Digital certificates are typically issued by a certificate authority (CA), which is a trusted third-party entity that issues digital certificates for use by other parties.

Backups

Back to the Top

Proxmox Backup Server is an enterprise backup solution for backing up and restoring VMs, containers, and physical hosts. The open-source solution supports incremental backups, deduplication, Zstandard compression, and authenticated encryption.

BackupPC is a high-performance, enterprise-grade system for backing up Linux, Windows and macOS PCs and laptops to a server’s disk. BackupPC is highly configurable and easy to install and maintain.

BorgWarehouse is a fast and modern WebUI for a BorgBackup’s central repository server.

Emborg is a simple command line utility to orchestrate backups. It is built as a front-end to Borg, a powerful and fast de-duplicating backup program.

Borgmatic is a simple, configuration-driven backup software for servers and workstations. It protects your files with client-side encryption. Backup your databases too. Monitor it all with integrated third-party services.

Vorta is a backup client for macOS and Linux desktops. It integrates the mighty Borg Backup with your favorite desktop environment to protect your data from disk failure, ransomware and theft.

UrBackup is an easy to setup Open Source client/server backup system, that through a combination of image and file backups accomplishes both data safety and a fast restoration time. File and image backups are made while the system is running without interrupting current processes. Available for Windows, macOS, and Linux.

Kopia is a user-friendly desktop app for Windows, macOS, and Linux which allows you to create snapshots, define policies, and restore files quickly with Fast and Encrypted Backups.

Clonezilla is a partition and disk imaging/cloning program. It helps you to do system deployment, bare metal backup and recovery. Three types of Clonezilla are available, Clonezilla live, Clonezilla lite server, and Clonezilla SE (server edition).

rsnapshot is a filesystem snapshot utility based on rsync. This makes it easy to make periodic snapshots of local machines, and remote machines over ssh.

Duplicity is a tool that backs directories by producing encrypted tar-format volumes and uploading them to a remote or local file server. Because duplicity uses librsync, the incremental archives are space efficient and only record the parts of files that have changed since the last backup.

ZnapZend is a high performance open source ZFS backup with mbuffer and ssh support. It uses the built-in snapshot functionality of ZFS for fully consistent backups. For each fileset, a pre- and post-snapshot command can be configured to quiet down any software writing to the fileset prior to snapshotting.

SnapRAID is a folder-based backup tool that behaves like a software or hardware RAID5/6 disk raid, but is not a disk raid itself. There is no realtime recovery, free space between disks cannot be combined and manual excution of backup is needed.

rsync.net is a Cloud Storage for Offsite Backup that give you an empty UNIX filesystem to access with any SSH tool. Built on ZFS for data security and fault tolerance with support for rsync/sftp/scp/borg/rclone/restic/git-annex.

Snapshots Management/System Recovery

Back to the Top

rsnapshot is a filesystem snapshot utility based on rsync. This makes it easy to make periodic snapshots of local machines, and remote machines over ssh.

Sanoid is a policy-driven snapshot management tool for ZFS filesystems.

ZFSBootMenu is a Linux bootloader that attempts to provide an experience similar to FreeBSD’s. This allows a user to have multiple “boot environments” (with different distributions, for example), manipulate snapshots before booting, and, for the adventurous user, even bootstrap a system installation via zfs recv.

Btrfs maintenance toolbox is a set of scripts supplementing the btrfs filesystem and aims to automate a few maintenance tasks. This means the scrub, balance, snapshots, trim or defragmentation.

Btrbk is a backup tool for btrfs subvolumes, taking advantage of btrfs specific capabilities to create atomic snapshots and transfer them incrementally to your backup locations.

ksync is a toool that sync files between your local system and a kubernetes cluster. It transparently updates containers running on the cluster from your local checkout.

Verify is a snapshot tool that simplifies the assertion of complex data models and documents.

Timeshift is a Linux application for providing functionality to restore your system just like Windows System Restore tool. Timeshift makes snapshots of your system in regular intervals which are further used at the time of restoration or undo all changes in the system.

CRIU (Checkpoint and Restore in Userspace) is a utility to checkpoint/restore Linux tasks. Using this tool, you can freeze a running application (or part of it) and checkpoint it to a hard drive as a collection of files. You can then use the files to restore and run the application from the point it was frozen at.

Rsync time backup is a Time Machine style backup with rsync. It creates incremental backups of files and directories to the destination of your choice. The backups are structured in a way that makes it easy to recover any file at any point in time. It works on Linux, macOS and Windows (via WSL).

rdiff-backup is a simple backup tool which can be used locally and remotely, on Linux and Windows, and even cross-platform between both. Users have reported using it successfully on FreeBSD and MacOS.

Mainframer is a tool that executes a command on a remote machine while syncing files back and forth. The process is known as remote execution (in general) and remote build (in particular cases).

Archiving

Back to the Top

Access to Memory (AtoM) - Web-based, open source application for standards-based archival description and access in a multilingual, multi-repository environment.

ArchiveBox - Self-hosted wayback machine that creates HTML & screenshot archives of sites from your bookmarks, browsing history, RSS feeds, or other sources.

Archivematica - Mature digital preservation system designed to maintain standards-based, long-term access to collections of digital objects.

ArchivesSpace - Archives information management application for managing and providing Web access to archives, manuscripts and digital objects.

CKAN - CKAN is a tool for making open data websites.

Collective Access - Providence - Highly configurable Web-based framework for management, description, and discovery of digital and physical collections supporting a variety of metadata standards, data types, and media formats.

Omeka S - Omeka S is a web publication system for universities, galleries, libraries, archives, and museums. It consists of a local network of independently curated exhibits sharing a collaboratively built pool of items, media, and their metadata.

Wayback - A self-hosted toolkit for archiving webpages to the Internet Archive, archive.today, IPFS, and local file systems.

Home Server

Back to the Top

Home Assistant is an open source home automation that puts local control and privacy first. Home Assistant is powered by a worldwide community of tinkerers and DIY enthusiasts that runs great on Raspberry Pi.

Homebridge is a software framework that allows you to integrate with smart home devices that do not natively support HomeKit. There are over 2,000 Homebridge plugins supporting thousands of different smart accessories.

Homebridge UI is a tool that provides an easy to use interface to manage your Homebridge plugins, configuration and accessories.

Install and configure Homebridge plugins.
Monitor your Homebridge server via a fully customisable widget-based dashboard.
View and control Homebridge accessories.
Backup and Restore your Homebridge instance.

ESPHome is a system to control your ESP8266/ESP32 by simple yet powerful configuration files and control them remotely through Home Automation systems.

Shelly Cloud is a Smart home control tool that has been perfected and provides precise monitoring of your Shelly devices no matter where you are. Shelly devices are compatible with Alexa, Google Home, Android, and iOS.

Zigbee is the full-stack, secure, reliable, and market-proven solution used by a majority of large smart home ecosystem providers, such as Amazon’s Echo Plus, Samsung SmartThings, Signify (Philips Hue), and more.

openHAB is a cross-platform software with the aim to integrate all kinds of Smart Home technologies, devices, etc.

Z-Wave is the leading wireless communications protocol behind many of the secure, trusted brands that are working to make everyone’s home smarter and safer.

Homey is an applciation to control, automate and monitor your entire smart home from your phone, tablet or desktop.

Caddy is the only web server to use HTTPS automatically and by default. Caddy obtains and renews TLS certificates for your sites automatically.

Bazarr is a companion application to Sonarr and Radarr. It can manage and download subtitles based on your requirements. You define your preferences by TV show or movie and Bazarr takes care of everything for you.

Sonarr is a PVR for Usenet and BitTorrent users. It can monitor multiple RSS feeds for new episodes of your favorite shows and will grab, sort and rename them.

Homarr is a customizable browser’s home page to interact with your homeserver’s Docker containers (e.g. Sonarr/Radarr)

Midarr is a free and open source (and always will be), Midarr aims to provide a tailored experience for you and your users:

Beautifully crafted user interface.
Real-time online statuses.
Simple and easy invite system.
Integrates with your existing services, Radarr and Sonarr.

Rustdesk is an open source virtual/remote desktop infrastructure for everyone. Display and control your PC (Windows, macOS, and Linux) and Android devices.

TinyPilot is a tool that enables KVM over IP letting you control any computer remotely.

PM2 is a production process manager for Node.js applications with a built-in load balancer. It allows you to keep applications alive forever, to reload them without downtime and to facilitate common system admin tasks.

authentik is an open-source Identity Provider focused on flexibility and versatility. You can use authentik in an existing environment to add support for new protocols. authentik is also a great solution for implementing signup/recovery/etc in your application, so you don’t have to deal with it.

ESPHome Remote IS a WI-FI smart home remote with display that runs on ESPHome. It uses Lilygo T-Display or M5Stack Fire.

Tdarr is a distributed transcode automation application using FFmpeg/HandBrake + Audio/Video library analytics + video health checking (Windows, macOS, Linux & Docker). A common use for Tdarr is to simply convert video files from h264 to h265 (hevc), saving 40%-50% in size.

AppFlowy is an open-source alternative to Notion where you’re in charge of your data and customizations.

deemix is a barebone deezer downloader library built from the ashes of Deezloader Remix.

Neko is a self hosted virtual browser that runs in docker and uses WebRTC.

QNAP Switch System (QSS) is a configuration interface for QNAP’s managed switch series. Enable management functions such as link aggregation, VLAN, and RSTP, to take care of your network topology with ease.

ASUSTOR is a subsidiary of ASUS and a leading provider of network attached storage (NAS). It specializes in the development and integration of storage, backup, multimedia, video surveillance and mobile applications for home and enterprise users.

Seafile is an open-source, cross-platform file-hosting software system. Seafile organize files into libraries stored on a central server. Each library can be synced into any desktop computer(Windows, Mac and Linux) and mobile devices through apps.

FreeNAS is an Open Source Storage Platform and supports sharing across Windows, Apple, and UNIX-like systems. This includes ZFS (high storage capacities and integrates file systems and volume management into a single piece of software). It supports UPS, CIFS/SMB, FTP, NFS, RSYNC, SSH, AFP, Unison, UPnP, Webserver, iSCSI protocols, local and MS AD authentication, and disk enctyption.

Gladys Assistant is a privacy-first, open-source home assistant and runs great on Raspberry Pi.

Audiobookshelf is a self-hosted audiobook and podcast server.

Mistborn is a secure platform for easily standing up and managing your own cloud services: including firewall, ad-blocking, and multi-factor WireGuard VPN access.

Media Server

Back to the Top

Overseerr is a free and open source software application for managing requests for your media library. It integrates with your existing services, such as Sonarr, Radarr, and Plex.

Jellyfin is a Free Software Media System that puts you in control of managing and streaming your media. It is an alternative to the proprietary Emby and Plex, to provide media from a dedicated server to end-user devices via multiple apps.

Swiftfin is a modern video client for the Jellyfin media server. Redesigned in Swift to maximize direct play with the power of VLC and look native on all classes of Apple devices.

Intro Skipper is a tool that analyzes the audio of television episodes to detect and skip over intro sequences in Jellyfin.

Jellyseerr is a free and open source software application for managing requests for your media library. It is a a fork of Overseerr built to bring support for Jellyfin & Emby media servers.

Midarr is a free and open source (and always will be), Midarr aims to provide a tailored experience for you and your users:

Beautifully crafted user interface.
Real-time online statuses.
Simple and easy invite system.
Integrates with your existing services, Radarr and Sonarr.

Kirino Media Server is a lightweight, modular alternative to Plex and Jellyfin.

Emby is a home media server built on top of other popular open source technologies such as Service Stack, jQuery, jQuery mobile, and Mono. It features a REST-based API with built-in documention to facilitate client development.

OpenMediaVault is a next generation network attached storage (NAS) solution based on Debian Linux. It contains services like SSH, (S)FTP, SMB/CIFS, AFS, UPnP media server, DAAP media server, RSync, BitTorrent client and many more.

MediaElch is a MediaManager for Kodi. Information about Movies, TV Shows, Concerts and Music are stored as NFO files.

tinyMediaManager is a media management tool written in Java/Swing. It is written to provide metadata for the Kodi Media Center (formerly known as XBMC), MediaPortal and Plex media server.

FileBot is the ultimate tool for renaming and organizing your movies, TV shows and Anime. Match and rename media files against online databases, download artwork and cover images, fetch subtitles, write metadata, and more, all at once in matter of seconds.

Plex media server is a application that gives you the power to add, access and share all the entertainment that matters to you, on almost any device. With 50,000+ on demand titles and hundreds of channels of live TV, plus your own personal media collection, using one powerful app.

Tautulli is a 3rd party application that you can run alongside your Plex Media Server to monitor activity and track various statistics.

Plex DupeFinder is a python script that finds duplicate versions of media (TV episodes and movies) in your Plex Library and tells Plex to remove the lowest rated files/versions (based on user-specified scoring) to leave behind a single file/version.

Prometheus Exporter for Plex is an expose library playback, storage, and host metrics in a Prometheus format.

Infuse is a Video Player for iOS, Apple TV, and Mac. It plays every video file ever created to avoid wasting hours converting and transcoding files.

InfuseSync is a plugin for Emby and Jellyfin media servers that tracks all media changes to decrease sync times with Infuse clients.

InvidTUI is an invidious client, which fetches data from invidious instances and displays a user interface in the terminal, and allows for selecting and playing Youtube audio and video.

Polaris is a music streaming application, designed to let you enjoy your music collection from any computer or mobile device. Polaris works by streaming music directly from your computer (or cloud server), without uploading it to a third-party.

AirSonic is a free, web-based media streamer, providing ubiquitous access to your music.

TubeSync is a PVR (personal video recorder) for YouTube. Or, like Sonarr but for YouTube (with a built-in download client). It is designed to synchronize channels and playlists from YouTube to local directories and update your media server once media is downloaded.

yt-fts is a simple python script that uses yt-dlp to scrape all of a youtube channels subtitles and load them into an sqlite database that is searchable from the command line. It allows you to query a channel for specific key word or phrase and will generate time stamped youtube urls to the video containing the keyword.

Tube Archivist is a self hosted YouTube media server.

PeerTube is an ActivityPub-federated video streaming platform using P2P directly in your web browser.

Ant Media Server is a streaming engine software that provides adaptive, ultra low latency streaming by using WebRTC technology with ~0.5 seconds latency.

Castopod is an open-source hosting platform made for podcasters who want engage and interact with their audience.

Festival is a music player for local album collections.

HD HomeRun Scribe 4K is a free local live TV with DVR with 4 tuners and 150 hours of recording storage all-in-one amazing box for watching and recording free TV all around your home.

RuneAudio is a free and open source software that turns embedded hardware into Hi-Fi music players.

Volumio (The Audiophile Music Player) is a very powerful and convenient music aggregator, now Volumio can also be used with great results in all sorts of different situations.

Snapcast is a multiroom client-server audio player, where all clients are time synchronized with the server to play perfectly synced audio. It’s not a standalone player, but an extension that turns your existing audio player into a Sonos-like multiroom solution.

SonoBus is an easy to use application for streaming high-quality, low-latency peer-to-peer audio between devices over the internet or a local network.

MythTV is a Free Open Source software digital video recorder (DVR) project distributed under the terms of the GNU GPL.

Smart Home Automation

Back to the Top

Smart home is a process that allows homeowners to control appliances, thermostats, lights, and other smart devices remotely using a smartphone or tablet through an internet connection.

Most smart devices have their own Virtual Local Area Network (VLAN) with little to no internet access with broadcasts forwarding to LAN Subnet aka Subnetwork for discovery. Using software such as Home Assistant, Homebridge, ESPHome, etc. help simplify the process of controlling and automating all your smart devices.

Matter is an open standard for smart home technology that lets your device work with any Matter-certified ecosystem using a single protocol. Matter comes from the Connectivity Standards Alliance, an organization of hundreds of companies(Amazon, Apple, Google, Comcast, Zigbee Alliance, and Connectivity Standards Alliance (CSA) creating products for the smart home.

Proprietary Smart Devices

Amazon Alexa is a smart virtual assistant software to manage Alexa-enabled devices, control music playback, view shopping lists on the go, keep track of upcoming reminders, check on active timers and much more.
Google Assistant is a smart virtual assistant software on mobile and home automation devices.
Apple HomeKit is a software framework that enables your app to coordinate and control home automation accessories from multiple vendors to present a coherent, user-focused interface. Using HomeKit, your app can: Discover HomeKit-compatible automation accessories and add them to a persistent, cross-device home configuration database.
Samsung SmartThings is a sofwtare framework that you can connect, monitor and control multiple smart home devices quicker and easier. Connect your Samsung smart TVs, smart appliances, smart speakers and brands like Ring, Nest and Philips Hue all from one app.
Philips Hue is a smart lighting system. The smart lights, Hue Bridge, and smart controls will forever change the way you experience light.
Sonos is the wireless home sound system that fills as many rooms as you want with great-sounding music, movies, and TV.

——————————————————————

Add-ons are additional applications and services, that can be run alongside Home Assistant. The Home Assistant OS and Supervised installations types, provide the Supervisor, which is capable of running and managing these add-ons.

Home Assistant Official Add-ons

Addons created and maintained by the Home Assistant team.

DuckDNS - This updates your Duck DNS IP address and generate SSL using Let’s Encrypt.
Almond - An Open, Privacy-Preserving Virtual Assistant.
HomeMatic - HomeMatic central based on OCCU.
Let’s Encrypt - Get a free SSL certificate from Let’s Encrypt; an open and automated certificate authority (CA).
MariaDB - An open source relational database (fork of MySQL).
File editor - Browser-based configuration file editor.
Mosquitto - Fast and reliable MQTT broker.
Terminal & SSH - Allows logging in remotely to using a web terminal or SSH client.
Samba - Access your configuration files using Windows network shares.
NGINX SSL proxy - Reverse proxy with SSL termination.
deCONZ - Control a ZigBee network using ConBee or RaspBee hardware by Dresden Elektronik.
TellStick - Run a TellStick and TellStick Duo service.
Ada - Ada is voice assistant powered by Almond which is open and privacy-preserving.
Fully Kiosk Browser is a powerful kiosk browser for Android devices. It provides a number of features for monitoring and controlling your Android device. This integration gives you access to control your device and view the status in Home Assistant.

Home Assistant Third Party Add-ons

Add-ons created by the community.

Dasshio - Easily use your Amazon Dash Buttons.
InfluxDB - Scalable datastore for metrics, events, and real-time analytics.
Grafana - Open platform for beautiful analytics and monitoring.
Tor - Protect your privacy and access your instance via Tor.
Spotify Connect - Spotify Connect client for playing music on your Home Assistant device.
SSH & Web Terminal - SSH and Web-based terminal with tons of pre-loaded useful tools.
UniFi Controller - The UniFi Controller allows you to manage your UniFi network using a web browser.
Node-RED - Flow-based programming for the Internet of Things.
Plex Media Server - Your recorded media beautifully organized and ready to stream.
IDE - Advanced web-based IDE, based on Cloud9 IDE.
zigbee2mqtt - Zigbee to MQTT bridge, get rid of your proprietary Zigbee bridges.
Matrix - A secure and decentralized communication platform.
AdGuard Home - A network-wide ad-and-tracker blocking DNS server with parental control.
Traccar - Traccar is modern GPS Tracking Platform.
Home Panel - A touch-compatible web frontend for controlling the home.
Hass.io Google Drive Backup - A complete and easy to configure solution for backing up your snapshots to Google Drive.
Grocy - A groceries & household management solution for your home.
EmonCMS - A powerful open-source web app for processing, logging, and visualizing energy, temperature, and other environmental data.
CrowdSec - A next-gen collaborative IPS/IDS to protect you from intrusion.
AppDaemon - Python Apps and HADashboard.
TasmoAdmin - Centrally manage all your Sonoff-Tasmota devices.
Aircast - AirPlay capabilities for your Chromecast players.
AirSonos - AirPlay capabilities for your Sonos players.
Log Viewer - Browser-based live log viewing utility.
Tautulli - Monitor and get statistics from your Plex server.
motionEye - Simple, elegant and feature-rich CCTV/NVR for your cameras.
JupyterLab - Create documents containing live code, equations, visualizations, and explanatory text.
Glances - A cross-platform system monitoring tool written in Python.

Home Assistant Custom Cards

The Home Assistant Dashboards allows people to build custom cards on top of it, which you can easily add to your instance.

Simple Thermostat - A simpler and more flexible thermostat card.
Card Modder - Style your Lovelace cards.
Bar Card - Customizable animated bar card.
forked-daapd Card - Control a forked daapd instance.
Dual Gauge Card - Shows two gauges in one.
Atomic Calendar Revive - Calendar card with advanced settings.
Simple Weather Card - A minimalistic weather card, inspired by Google Material Design.
Auto-Entities Card - Dynamically adds entities.
Canvas Gauge Card - Use awesome gauges from canvas-gauges.com.
Big Number Card - Display big numbers for sensors, including severity level as background.

Animated Weather Card - Nice looking card showing the weather, with subtle animations.
Thermostat Card - Thermostat control card that looks like a Nest Thermostat.

Raspberry Pi Status Card - Show status of your Raspberry Pis.
Mini Media Player - A minimalistic media player card.
Mini Graph Card - A minimalistic sensor graph card.
Button card - Button card for your entities.
Slider Entity Row - Add a slider to adjust, e.g., the brightness of lights in lovelace entity cards.
Power Wheel Card - An intuitive way to represent the power that your home is consuming or producing.
Home Card - A quick glance of the state of your home.
Banner Card - A fluffy linkable banner with interactive glances to spice up your home dashboards.
Spotify Card - List and select from current available devices and users top playlists on Spotify.
Battery Entity - Displaying battery levels for battery entities.
Multiple Entity Row - Show multiple entity states or attributes on entity rows.
Home Feed Card - Display a combination of persistent notifications, calendar events, and entities in the style of a feed.
Config Template Card - Allow using templates in Lovelace.
RGB Light Card - Colorful buttons to control your RGB Lights.
Restriction Card - A card to provide restrictions on Lovelace cards defined within.
Vacuum Map Card - This card provides a user-friendly way to fully control Xiaomi (Roborock/Viomi/Dreame/Roidmi) and Neato (+ possibly other) vacuums.
Vacuum Card - A card to card for controlling a vacuum cleaner robot.
Purifier Card - A card for controlling air purifiers.

Home Assistant Custom Integrations

Additional integrations for Home Assistant created by the community.

Lutron Caseta Pro - Integrates Lutron Caseta Smart Bridge PRO / RA2 Select.
SmartIR - Integrates devices using Broadlink IR.
Alexa Media Player - Allow control of Amazon Alexa devices.
Circadian Lighting - Circadian Lighting slowly synchronizes your color changing lights with the regular naturally occuring color temperature of the sky throughout the day.
Volkswagen Carnet - Integrates Volkswagen Carnet (requires valid Carnet subscription).
Untappd - Connects with your Untappd account.
Elasticsearch - Publishes events to Elasticsearch.
HASS Aarlo - Asynchronous Arlo integration. Similar to the Arlo web site; monitors events and states for all base stations, cameras and doorbells.
Xiaomi Cloud Map Extractor - Presents a live view of a map for Xiaomi (Roborock/Viomi/Roidmi/Dreame) vacuums without a need for rooting.
Xiaomi Hygrothermo - Sensor platform for Xiaomi Mijia BT Hygrothermo temperature and humidity sensor.
WebRTC Camera - View RTSP streams from IP Cameras in real-time through WebRTC or MSE with Pan/Zoom controls.
Sonoff LAN - Control Sonoff devices with eWeLink (original) firmware over LAN and/or Cloud.
Spotcast - Start Spotify playback on an idle Chromecast device as well as control Spotify connect devices.
The Watchman - Keep track of missing entities and services in your config files.

Home Assistant integrations. Credit: Home Assistant

Homebridge UI is a tool that provides an easy to use interface to manage your Homebridge plugins, configuration and accessories.

Install and configure Homebridge plugins.
Monitor your Homebridge server via a fully customisable widget-based dashboard.
View and control Homebridge accessories.
Backup and Restore your Homebridge instance.

Homebridge Raspberry Pi Image is a free Raspbian based Raspberry Pi image with Homebridge and Homebridge Config UI X pre-installed.

Homebridge Config UI X is a web based management tool for Homebridge that allows you to manage all aspects of your Homebridge setup.

Homebridge webOS TV is a plugin for homebridge which allows you to control your LG webOS TV from your Home app! It should work with all TVs that support webOS2 and newer.

Homebridge Unifi Protect is a Homebridge plugin that provides HomeKit support to the UniFi Protect device ecosystem. UniFi Protect is Ubiquiti’s next-generation video security platform, with rich camera, doorbell, and NVR controller hardware options for you to choose from, as well as an app which you can use to view, configure and manage your video camera and doorbells.

Homebridge Camera FFmpeg is a Homebridge Plugin Providing FFmpeg-based Camera Support.

Homebridge Mi Aqara is a homebridge plugin for XiaoMi Aqara plugin.

Homebridge Camera UI is a tool that allows you to expose cameras from camera.ui to HomeKit via Homebridge.

HOOBS is a tool that makes smart accessories(over 2000 options) compatible with your favorite ecosystem. Whether you prefer Apple Homekit, Google Home, or Amazon Alexa, you’re unlikely to find compatible accessories and services that all work together nicely under one roof.

ESPHome is a system to control your ESP8266/ESP32 by simple yet powerful configuration files and control them remotely through Home Automation systems.

Homey is an applciation to control, automate and monitor your entire smart home from your phone, tablet or desktop.

Ecobee is a home automation company in Canada that makes thermostats for residential and commercial use.

Lutron Caséta is a smart lighting control system that is a great solution for giving any client smart lighting control. It was purposely built to work in homes of all ages and it works with older wiring as well as new.

Insteon switches is a Hub for controlling and configuring your home’s devices is quick, easy and fun. The setup takes a couple of minutes and a few moments per light switch, sensor, etc. It bridges your devices to 3rd party services like Amazon Alexa and Google Assistant, while being controlled from any smartphone or tablet.

Jeedom is an open source software; taht gives you complete access to the software that manages your home automation. It’s compatible with various protocols such as, ZigBee, Z-Wave, EnOcean, KNX, LoRaWAN, BACnet, Modbus, etc..

Beestat is a tool that connects with your thermostat and provides you with useful charts and analytics so that you can make informed decisions and see how the changes you make lower your energy footprint.

openHAB is a cross-platform software with the aim to integrate all kinds of Smart Home technologies, devices, etc.

Z-Wave is the leading wireless communications protocol behind many of the secure, trusted brands that are working to make everyone’s home smarter and safer.

pfSense is a firewall/router computer software distribution based on FreeBSD.

AdGuard Home is a DNS relay station with ad/tracker/other blocking, IP address redirections, and DNS-over-HTTPS.

OpenWRT is an open-source project for embedded operating systems based on Linux, primarily used on embedded devices to route network traffic.

ZoneMinder is a full-featured, open source, state-of-the-art video surveillance software system. Monitor your home, office, or wherever you want.

Voice Assistants

Back to the Top

$13 voice assistant remote for Home Assistant

Wyoming is a peer-to-peer protocol for voice assistants (basically JSONL + PCM audio). It’s used in Rhasspy and the Home Assistant for communication with voice services.

Wyoming Faster Whisper is a Wyoming protocol server for the faster-whisper speech to text system.

Wyoming Porcupine1 is a Wyoming protocol server for the porcupine1 wake word detection system.

Wyoming Snowboy is a Wyoming protocol server for the snowboy wake word detection system.

faster-whisper is a reimplementation of OpenAI’s Whisper model using CTranslate2, which is a fast inference engine for Transformer models.

Porcupine is a highly-accurate and lightweight wake word engine. It enables building always-listening voice-enabled applications. It uses deep neural networks trained in real-world environments.

Rhasspy is an open source voice assistant toolkit for many human languages.

openWakeWord is an open-source wakeword library that can be used to create voice-enabled applications and interfaces. It includes pre-trained models for common words & phrases that work well in real-world environments.

Conversation is an integration allows you to converse with Home Assistant. You can either converse by pressing the microphone in the frontend (supported browsers only (no iOS)) or by calling the conversation/process service with the transcribed text.

Piper is a fast, local neural text to speech system that sounds great and is optimized for the Raspberry Pi 4.

Mycroft is an open source voice assistant that is private by default and completely customizable.

DeepSpeech is an open source embedded (offline, on-device) speech-to-text engine which can run in real time on devices ranging from a Raspberry Pi 4 to high power GPU servers.

Leon is your open-source personal assistant.

Olivia is an open-source chatbot built in Golang using Machine Learning technologies. Its goal is to provide a free and open-source alternative to big services like DialogFlow.

Alan SDK is an voice assistant SDK to build a voice interface for websites and web apps (JavaScript, React, Angular, Vue, Ember, Electron).

OpenAssistant is a chat-based assistant that understands tasks, can interact with third-party systems, and retrieve information dynamically to do so.

Video Surveillance

Back to the Top

Frigate is an open source NVR built around real-time AI object detection. All processing is performed locally on your own hardware, and your camera feeds never leave your home.

hkcam is an open-source implementation of an HomeKit IP camera. It uses ffmpeg to access the camera stream and publishes the stream to HomeKit using hap. The camera stream can be viewed in a HomeKit app.

OpenDataCam is an open source tool to quantify the world. It quantifies and tracks moving objects with live video analysis. It is designed to be an accessible, affordable and open-source solution to better understand interactions in urban environments. It never records any photo or video data. The system only saves surveyed meta-data, in particular the path an object moved or number of counted objects at a certain point.

Viseron is a Self-hosted, local only NVR and AI Computer Vision software.

zmninja is a high performance, cross platform ionic app for Home/Commerical Security Surveillance using ZoneMinder.

Moonfire NVR is a security camera network video recorder.

Shinobi Pro is a Next Generation in Open-Source Video Management Software with support for over 6000 IP and USB Cameras.

WyzeHacks is a project contains a set of scripts trying to provide additional features not implemented by the official firmware. Currently, it provides the following functions:

Enable telnetd on your camera.
Customize the default root password for telnet login.
Redirect all the recordings to an NFS share.
Redirect console logs into an NFS share.
Automatically reboot the camera at certain time.
Automatically archive the recordings.

Text-To-Speech Synthesis (TTS)

Back to the Top

whisper.cpp is a high-performance inference of OpenAI’s Whisper automatic speech recognition (ASR) model.

WaaS is a Whisper as a Service (GUI and API for OpenAI Whisper).

Web Whisper is a OpenAI’s whisper on your web browser. Demo

Vosk is an offline open source speech recognition toolkit. It enables speech recognition for 20+ languages and dialects.

Coqui TTS is a deep learning toolkit for Text-to-Speech, battle-tested in research and production.

Mozilla TTS is a library for advanced Text-to-Speech generation. It’s built on the latest research, was designed to achieve the best trade-off among ease-of-training, speed and quality.

NVIDIA NeMo is a conversational AI toolkit built for researchers working on automatic speech recognition (ASR), text-to-speech synthesis (TTS), large language models (LLMs), and natural language processing (NLP).

Video and Audio Processing

Back to the Top

Intel® Quick Sync Video is a tools that uses the dedicated media processing capabilities of Intel® Graphics Technology to decode and encode fast, enabling the processor to complete other tasks and improving system responsiveness.

Intel® QuickAssist Technology (Intel® QAT) is a scalable, flexible, and extendable way to accelerate data encryption/decryption and compression for applications from networking to enterprise, cloud to storage, and content delivery to database.

FFmpeg is a leading multimedia framework that can decode, encode, transcode, mux, demux, stream, filter and play pretty much anything that humans and machines have created. It supports the most obscure ancient formats up to the cutting edge ones on multiple platforms such as Windows, macOS, and Linux.

FFmpeg.guide is a simple GUI tool to create complex FFmpeg filtergraphs quickly and correctly, without having to mess with the cumbersome filter syntax.

HandBrake is a tool for transcoding video from almost any format with a selection of widely supported codecs. It is supported on Window, macOS, and Linux.

Tdarr is a cross-platform conditional based transcoding application for automating media library transcode/remux management in order to process your media files as required. It can set rules for the required codecs, containers, languages etc that your media should have which helps keeps things organized and can increase compatability with your devices. A common use for Tdarr is to simply convert video files from h264 to h265 (hevc), saving 40%-50% in size.

SRS is a simple, high efficiency and realtime video server, supports RTMP, WebRTC, HLS, HTTP-FLV, SRT and GB28181.

obsws-python is a Python SDK for OBS Studio WebSocket v5.0.

Video/Audio Standards

AAC(Advanced Audio Coding) is an audio coding standard for lossy digital audio compression. It’s endorsed by ISO and IEC as MPEG-2 and MPEG-4 standards for video streams.

H.264(AVC) is a video compression standard based on block-oriented and motion-compensated integer-DCT coding that defines multiple profiles (tools) and levels (max bitrates and resolutions) with support up to 8K.

H.265(HEVC) is a video compression standard that is the successor to H.264(AVC). It offers a 25% to 50% better data compression at the same level of video quality, or improved video quality at the same bit-rate.

HTTP Live Streaming (HLS) is a communications protocol developed by Apple that sends live and on‐demand audio and video to iPhone, iPad, Mac, Apple Watch, Apple TV, and PC.

Dynamic Adaptive Streaming over HTTP (DASH) is an adaptive streaming protocol that allows for a video stream to switch between bit rates on the basis of network performance, in order to keep a video playing.

OpenMAX™ is a cross-platform API that provides comprehensive streaming media codec and application portability by enabling accelerated multimedia components to be developed, integrated and programmed across multiple operating systems and silicon platforms.

GStreamer is a library for constructing graphs of media-handling components. The applications it supports range from simple Ogg/Vorbis playback, audio/video streaming to complex audio (mixing) and video (non-linear editing) processing. Applications can take advantage of advances in codec and filter technology transparently.

Media Source Extensions (MSE) is a W3C specification that allows JavaScript to send byte streams to media codecs within Web browsers that support HTML5 video and audio. Also, this allows the implementation of client-side prefetching and buffering code for streaming media entirely in JavaScript.

WebRTC is an open-source project that adds real-time communication capabilities to your application that works on top of an open standard. It supports video, voice, and generic data to be sent between peers, allowing developers to build powerful voice- and video-communication solutions.

Podcasting

Back to the Top

Castopod is an open-source hosting platform made for podcasters who want engage and interact with their audience.

Sovereign Feeds is a tool to Search for your podcasts and add them to your favorites.

IPFS Podcasting is a Decentralized Podcast Distribution over IPFS where you can crowd hosting podcast episodes with storage & bandwidth provided by volunteer nodes.

Audiobookshelf is a self-hosted audiobook and podcast server.

Vod2Pod-RSS is a tool that converts a YouTube or Twitch channel into a podcast with ease. It creates a podcast RSS that can be listened to directly inside any podcast client. VODs are transcoded to MP3 on the fly and no server storage is needed.

Podverse is creating an open source podcast app for iOS, Android, F-Droid, and Web.

Alby is a Bitcoin Lightning App for your Browser.

Alby wallet API is an implemented OAuth an open standard that apps use to provide client applications with secure delegated access. Thus Podverse users create an Alby wallet account or use their existing Alby account to grant specific access rights to their Alby wallet.

Blubrry is a podcast hosting service for publishing platform, live customer support, stress-free migration, and impactful statistics.

SATurn is a tool that lets you connect your getalby.com Account and see which content resonates most with your audience and recognize your top contributors.

AntennaPod is a podcast player that is completely open. The app is open-source and you can subscribe to any RSS feed. AntennaPod is built by volunteers without commercial interest, so it respects your privacy while giving you full control.

Podgrab is a self-hosted podcast manager/downloader/archiver tool to download podcast episodes as soon as they become live with an integrated player.

Podify is a self-hosted service that allows you to download videos and audio from any source supported by youtube-dl, sort the downloads into feeds, and subscribe to these feeds using your favorite podcast app.

dir2cast is designed to turn a directory of MP3s into a podcast - automatically. Perfect for, say, radio shows - upload the MP3s to a folder, and use dir2cast.php as your PodCast URL.

Snipd is an AI-powered podcast player. Search for moments in the transcript, get summaries, share clips to social media, and export to your second brain & note taking apps. Discover, save, and share highlights from podcasts.

Wave Share is a serverless, peer-to-peer, local file sharing through sound.

KBD Audio is a collection of command-line and GUI tools for capturing and analyzing audio data.

AudioBooks

Back to the Top

Audioserve is a simple personal server to serve audio files from directories. Intended primarily for audio books, but anything with decent directories structure will do. Focus here is on simplicity and minimalist design.

Audiobookshelf is a self-hosted audiobook and podcast server.

Jellyfin Bookshelf Plugin

Health

Back to the Top

Connect is the swiss army knife of healthcare integration.

Fasten is an open-source, self-hosted, personal/family electronic medical record aggregator, designed to integrate with 1000’s of insurances/hospitals/clinics

ERPNext is a Free and Open Source Enterprise Resource Planning (ERP) for managing businesses.

OpenEMR is a Free and Open Source electronic health records and medical practice management application. It features fully integrated electronic health records, practice management, scheduling, electronic billing, internationalization, free support, a vibrant community, and a whole lot more. It runs on Windows, Linux, MacOS, and many other platforms.

Ryot (Roll Your Own Tracker) is a self hosted platform for tracking various facets of your life - media, fitness etc.

Gardening

Back to the Top

OpenSprinkler is an open-source, web-based smart sprinkler controller for lawn and plant watering, drip irrigation, farm irrigation, hydroponics etc. The current version is OS 3.2, with built-in WiFi (based on ESP8266) and OLED display.

Droplet is an ALL-IN-ONE Irrigation and monitoring system for ESPHome and Home Assistant.

9 Valve Sprinkler Controller is a 9 valve sprinkler controller for use with customized firmware such as ESPHOME.

GardenBot is an open source garden monitoring system. Their website is a collection of tutorials for how to build things (like a soil moisture sensor), software for running GardenBot, resources, links, and more.

farmOS is a web-based application for farm management, planning, and record keeping. It is developed by a community of farmers, developers, researchers, and organizations with the aim of providing a standard platform for agricultural data collection and management.

OpenFarm is a free and open database and web application for farming and gardening knowledge. One might think of it as the Wikipedia for growing plants, though it functions more like a cooking recipes site.

Growstuff is an open source/open data project for food gardeners. They crowdsource information on what our members are growing and harvesting, aggregate it, and make it available as open data via their API.

Harvest Helper is a tool that provides growing, harvesting and recipe information for the 45 plants in the database as well as a json api so that people can hopefully use this data to build other apps.

HappyPlants is a mobile web application that’s all about collecting, organising, and adding all kinds of information of your plants. Basically, creating your own plant database in a visual way.

Automated irrigation system is an open source application to water plants automatically. Up to now there is almost no free professional software and instructions available to build a DYI irrigation that is scalable, accurate and most importantly, durable.

Pigrow is a garden automation suite designed to help gardeners monitor, log, graph and control their grow space using a raspberry Pi, various sensor and a few relay modules.

Tania is a farm management software for the hobbyist and smallholder farmer.

Maps

Back to the Top

Magic Earth is aTurn-by-turn navigation, OpenStreetMap, Crowd-Sourced Traffic, 3D maps, Satellite maps, Offline maps and Transit.

Organic Maps is a free Android & iOS offline maps app for travelers, tourists, hikers, and cyclists. It uses crowd-sourced OpenStreetMap data and is developed with love by MapsWithMe (MapsMe) founders and our community. No ads, no tracking, no data collection, no crapware.

MapTiler Server is a self-hosted Aerial and satellite imagery maps of the entire world from your own server or laptop.

GPSLogger is a GPS tool that uses the GPS capabilities of your Android phone to log coordinates to GPS format files at regular intervals. This can be particularly useful if you want to geotag your photos after a day out or share your travel route with someone.

KelperJs is a open source full-stack geosocial network platform.

OpenStreetMap(OSM) is a map of the world, created by people like you and free to use under an open license. Hosting is supported by UCL, Fastly, Bytemark Hosting, and other partners.

uMap is a tool that lets you create maps with OpenStreetMap layers in a minute and embed them in your site.

Martin is a tile server able to generate vector tiles from large PostGIS databases on the fly, or serve tiles from PMTile and MBTile files. Martin optimizes for speed and heavy traffic, and is written in Rust.

MapLibre GL JS is an open-source library for publishing maps on your websites or webview based apps. Fast displaying of maps is possible thanks to GPU-accelerated vector tile rendering.

MapLibre Native is an Interactive vector tile maps for iOS, Android and other platforms.

Maplibre-rs is an Experimental Maps for Web, Mobile and Desktop.

Bookmarks

Back to the Top

Linkding is a simple bookmark service that you can host yourself. It’s designed be to be minimal, fast, and easy to set up using Docker.

Linkwarden is a fully self-hostable, open-source collaborative bookmark manager to collect, organize and archive webpages. Linkwarden Docker Image

LinkAce is a self-hosted bookmark archive to collect links of your favorite websites.

Eagle is a tool to collect, search and organize your design files in a logical way and all in one place.

Shlink is a self-hosted URL shortener that keeps control over all your shortened URLs, by serving them under your own domains, using this simple yet powerful tool.

Pinry is a tiling image board system for people who want to save, tag, and share images, videos and webpages in an easy to skim through format.

Shaark is a self-hosted platform to keep and share your content: web links, posts, passwords and pictures.

Maglit is an encrypted and privacy respecting Link Shortener service that supports not only your regular website links but also Magnet Links which are extensively used to download and share torrents.

Photos

Back to the Top

PhotoPrism® is an AI-powered app for browsing, organizing & sharing your photo collection. It makes use of the latest technologies to tag and find pictures automatically without getting in your way. You can run it at home, on a private server, or in the cloud.

Immich is a high performance self-hosted photo and video backup solution directly from your mobile phone.

Piwigo is a full featured, self-hosted, and open source photo gallery application for the web. It comes with more than 200 templates, plugins, and configurations that let you personalize how your photos are presented. It allows users to upload photos from digiKam, Shotwell, Lightroom or mobile applications. Lastly, users can create galleries and give viewing permissions to their clients. They can download individual photos or whole albums, post comments, give ratings, mark photos as favorites, and perform searches.

Czkawka is a Multi-functional app to find duplicates, empty folders, similar images, etc.

Phockup is a Media sorting tool to organize photos and videos from your camera in folders by year, month and day.

PiGallery 2 is a fast directory-first photo gallery website, with rich UI, optimized for running on low resource servers (especially on Raspberry Pi).

Photoview is a simple self-hosted and user-friendly photo gallery that’s made for photographers and aims to provide an easy and fast way to navigate directories, with thousands of high-resolution photos.

digiKam is a free and open-source Professional Photo Management tool.

ShareX is a free and open source program that lets you capture or record any area of your screen and share it with a single press of a key. It also allows uploading images, text or other types of files to many supported destinations you can choose from.

PhotoSync is a service to wirelessly transfer, backup & share photos/videos to your computer, NAS, other phones and popular cloud/photo services. It’s available for Windows, MacOS, Linux, Android, and iOS.

Lychee is a great looking and easy-to-use photo-management-system you can run on your server, to manage and share photos.

Photoview is a simple and user-friendly photo gallery that’s made for photographers and aims to provide an easy and fast way to navigate directories, with thousands of high-resolution photos.

Gimme-iPhotos is a tool uses pyicloud to synchronize photos and videos from iCloud to your local machine.

PyiCloud is a module which allows pythonistas to interact with iCloud webservices. It’s powered by the fantastic requests HTTP library.

Pixelfed is a fresh take on photo sharing. It decentralized ActivityPub protocol so you can comment, follow, and interact with remote Pixelfed, Mastodon and Pleroma posts and profiles from your Pixelfed account as if you were both on the same website.

Chevereto is an image hosting software that allows you to create a beautiful and full-featured image hosting website on your own server.

Got Your Back (GYB) is a command line tool for backing up your Gmail messages to your computer using Gmail’s API over HTTPS.

Upscayl is a free and open source desktop application that lets you upscale your low resolution images using advanced AI Models. Upscayl is a Linux-First Application that prioritizes Linux builds but is also cross-platform.

Librephotos is a self-hosted open source photo management service. This is the repository of the backend.

Librephotos frontend is a self-hosted open source photo management service. This is the repository of the frontend.

Librephotos Mobile is an open-source Android and iOS Mobile Application for self-hosted Librephotos Server.

Librephotos Docker is the Dockerfiles for the automated build process of LibrePhotos.

OneFolder is a Desktop app to sort your images the same way you would in Google Photos, but locally, no need to run a server (and compatible with NAS).

Pastebins

Back to the Top

Bepasty is a pastebin for all kinds of files.

Bin is a paste bin that’s actually minimalist.

Dpaste is a Simple pastebin with multiple text and code option, with short url result easy to remember.

Drift is a Self-hosted Github Gist clone.

EdPaste is a Self-hosted pastebin written in Laravel (PHP Framework).

ExBin is a pastebin with public/private snippets and netcat server.

Fiche is a Command line pastebin, all you need is netcat.

Filite is a simple, light and standalone pastebin, URL shortener and file-sharing service.

FlashPaper is a one-time encrypted zero-knowledge password/secret sharing application focused on simplicity and security. No database or complicated set-up required.

Hasty Paste is a place to quickly paste some text and share it. Mostly used for sharing debug logs and such to help developers provide tech support. The project aims to be both fast and minimal.

Lenpaste is a Web service that allows you to share notes anonymously, an alternative to pastebin.

LogPaste is a Minimal pastebin web app that’s easy to self-host and persists data to any S3-compatible backend.

MicroBin is a super tiny, feature rich, configurable, self-contained and self-hosted paste bin web application. It is very easy to set up and use, and will only require a few megabytes of memory and disk storage. MicroBin Docker install setup.

Opengist is a Self-hosted pastebin powered by Git.

Paaster is a secure by default end-to-end encrypted pastebin built with the objective of simplicity.

Pastefy - Beautiful, simple and easy to deploy Pastebin with optional Client-Encryption, Multitab-Pastes, an API, a highlighted Editor and more.

Pastila is a Minimalistic paste service. Single page, zero click experience.

Pasty is a fast and lightweight code pasting server.

PrivateBin is a minimalist, opensource online pastebin/discussion board where the server has zero knowledge of hosted data.

Prologic pastebin is a Simple pastebin service with convenient api and CLI.

PurritoBin is a Ultra fast, minimalistic, encrypted command line paste-bin, where the server has no knowledge of the paste data.

Rustypaste is a minimal file upload/pastebin service.

Spacebin is a Reliable Pastebin server in Golang and Fiber.

Sup3rS3cretMes5age is a simple to deploy and use secret message service using Hashicorp Vault as a secrets storage.

Wastebin is a Lightweight, minimal and fast pastebin with an SQLite backend.

YABin is a pastebin that contains plentiful features while remaining simple. Supports optional E2E encryption, a client-side CLI app, syntax highlighting, minimalistic UI, APIs, keyboard shortcuts, and more. It can even be run in serverless environments.

Note-Taking

Back to the Top

Joplin is an open source note-taking app that you can securely access from any device.

HedgeDoc is an open-source, web-based, self-hosted, collaborative markdown editor.

Lapce is a Lightning-fast And Powerful Code Editor written in pure Rust with a UI in Druid (which is also written in Rust).

nb is a CLI and local web plain text note‑taking, bookmarking, and archiving with linking, tagging, filtering, search, Git versioning & syncing, Pandoc conversion, + more, in a single portable script.

Outline is the fastest knowledge base for growing teams. It provides a beautiful, realtime collaborative, feature packed, and markdown compatible.

Rustpad is an open-source collaborative text editor based on the operational transformation algorithm. Share a link to this pad with others, and they can edit from their browser while seeing your changes in real time.

Turtl is a secure, collaborative notebook for bookmarks or passwords, files or shopping lists.

The Everything App is an app where you can do everything: Protect your thoughts & data with end-to-end encryption. Local, on-device encryption. Only you have encryption keys. Offline account creation: control your keys, own your data. No server, no gatekeeper: peer-to-peer sync on local networks. Locally store your data, self-host your backups where you please.

TiddlyWiki is a single-file mode wiki application for todo lists, effective project management tool and of course writing drafts and notes. It has extensions for all the major browsers.

Laverna is a note taking application with Markdown editor and encryption support. Consider it like open source alternative to Evernote.

Notesnook is a fully open source & end-to-end encrypted note taking alternative to Evernote.

Zettlr is an open-source Markdown editor for the 21st century.

Carnet is a complete open source note taking app. It has extensions for all the major browsers.

Frog is a tool that quickly extract text from almost any source: youtube, screencasts, PDFs, webpages, photos, etc. Grab the image and get the text.

Zeal is an offline documentation browser for software developers inspired by Dash.

Time Monitoring

Back to the Top

ActivityWatch is an app that automatically tracks how you spend time on your devices.

Kimai is a free & open source timetracker. It tracks work time and prints out a summary of your activities on demand.

Solidtime is an open source time tracking software for individuals and teams, with a modern user interface and reporting.

TimeTagger is an open source time-tracker based on an interactive timeline and powerful reporting.

Traggo is a tag-based time tracking tool. In Traggo there are no tasks, only tagged time spans.

Wikis

Back to the Top

Archivy is a self-hostable knowledge repository that allows you to learn and retain information in your own personal and extensible wiki.

BookStack - BookStack is a simple, self-hosted, easy-to-use platform for organizing and storing information. It allows for documentation to be stored in a book like fashion.

Cowyo - Cowyo is a feature-rich wiki for minimalists.

django-wiki - Wiki system with complex functionality for simple integration and a superb interface. Store your knowledge with style: Use django models.

Documize - Modern Docs + Wiki software with built-in workflow, single binary executable, just bring MySQL/Percona.

Dokuwiki - Easy to use, lightweight, standards-compliant wiki engine with a simple syntax allowing reading the data outside the wiki. All data is stored in plain text files, therefore no database is required.

Gitit - Wiki program that stores pages and uploaded files in a git repository, which can then be modified using the VCS command line tools or the wiki’s web interface.

Gollum - Simple, Git-powered wiki with a sweet API and local frontend.

Instiki - Instiki is a wiki clone so pretty and easy to set up, you’ll wonder if it’s really a wiki. Runs on Rails and focuses on portability and stability.

Mediawiki - MediaWiki is a free and open-source wiki software package written in PHP. It serves as the platform for Wikipedia and the other Wikimedia projects, used by hundreds of millions of people each month.

Pepperminty Wiki - Complete markdown-powered wiki contained in a single PHP file.

PineDocs - Simple, fast, customizable and lightweight site for browsing files.

PmWiki - Wiki-based system for collaborative creation and maintenance of websites.

PukiWiki - PukiWiki is a free, simple, open-source wiki management system. Minimalistic and simple design with many tools for collaborative work. Created in Japanese, so translator needed.

Raneto - Raneto is an open source Knowledgebase platform that uses static Markdown files to power your Knowledgebase.

TiddlyWiki - Reusable non-linear personal web notebook.

Tiki - Wiki CMS Groupware with the most built-in features.

TWiki - TWiki is a Perl-based structured wiki application, typically used to run a collaboration platform, knowledge or document management system, a knowledge base, or team portal.

WackoWiki - WackoWiki is a light and easy to install multilingual Wiki-engine.

Gramax - Free, open-source application for creating, editing, and publishing Git-driven documentation sites using Markdown and a visual editor.

Gaming

Back to the Top

Cartridge is a self-hosted game library made with Laravel + Vue.js.

Cartridge Features

Scan for ROM files and match with IGDB game information
Serve ROM download links alongside game details
Manage access to library with user creation and permissions (WIP)
Allow users to request games (Planned)
Play select ROMs in-browser using JS emulators (Planned)
Track played and favorite games (even ones that aren’t available for download) (Planned)

Moonlight Game Streaming is a program that let you stream from your PC games over the Internet with no configuration required. Stream from almost any device, whether you’re in another room or miles away from your gaming rig. Sunshine is a Game stream host for Moonlight that is a self-hosted, low latency, cloud gaming solution with support for AMD, Intel, and NVIDIA GPUs. It is an open source implementation of NVIDIA’s GameStream.

Chiaki is a Free and Open Source Software Client for PlayStation 4 and PlayStation 5 Remote Play for Linux, FreeBSD, OpenBSD, Android, macOS, Windows, Nintendo Switch and potentially even more platforms.

Game Emulators

EmuDeck is a tool that takes care of everything for your retrogaming needs from RetroArch Configuration, Bezels, Gamepad Configuration for GameCube, Wii, Citra, SNES, etc. EmuDeck will even install EmulationStation Desktop Edition and carry over all their custom configurations and no need to configure ROM paths or anything.

EmulationStation Desktop Edition (ES-DE) is a frontend application for browsing and launching games from your multi-platform game collection. It’s available for Unix/Linux, macOS(M1 & Intel) and Windows.

RetroPie is a frontend for emulators that allows you to turn your Raspberry Pi, ODroid C1/C2, or PC into a retro-gaming machine. It builds upon Raspbian, EmulationStation, RetroArch and many other projects to enable you to play your favourite Arcade, home-console, and classic PC games with the minimum set-up.

RetroArch is a frontend for emulators, game engines and media players. It enables you to run classic games on a wide range of computers and consoles through its slick graphical interface. Settings are also unified so configuration is done once and for all. RetroArch Flatpak

Pterodactyl is a free, open-source game server management panel built with PHP, React, and Go. Designed with security in mind, Pterodactyl runs all game servers in isolated Docker containers while exposing a beautiful and intuitive UI to end users.

LinuxGSM (Linux Game Server Managers) is a command-line tool for quick, simple deployment and management of Linux dedicated game servers.

Cartridge is a self-hosted game library made with Laravel + Vue.js.

Cartridge Features

Scan for ROM files and match with IGDB game information
Serve ROM download links alongside game details
Manage access to library with user creation and permissions (WIP)
Allow users to request games (Planned)
Play select ROMs in-browser using JS emulators (Planned)
Track played and favorite games (even ones that aren’t available for download) (Planned)

Cartridge UI

Dolphin is an emulator for two recent Nintendo video game consoles: the GameCube and the Wii. It allows PC gamers to enjoy games for these two consoles in full HD (1080p) with several enhancements: compatibility with all PC controllers, turbo speed, networked multiplayer, and even more.

Citra is an open-source emulator for the Nintendo 3DS capable of playing many of your favorite games.

yuzu is an experimental open-source emulator for the Nintendo Switch from the creators of Citra.[

m64p is a Nintendo 64 Emulator. It uses mupen64plus-gui, a brand new mupen64plus frontend written in Qt5. It supports all of the things you’d expect from a frontend (savestate management, pausing, screenshots).

DeSmuME is a Nintendo DS emulator.

Snes9x is a portable, freeware Super Nintendo Entertainment System (SNES) emulator.

bsnes is a Super Nintendo (SNES) emulator focused on performance, features, and ease of use.

mGBA is a new emulator for running Game Boy Advance games. It aims to be faster and more accurate than many existing Game Boy Advance emulators, as well as adding features that other emulators lack.

DOSBox is an open-source DOS emulator which primarily focuses on running DOS Games.

DOSBox Staging is a full x86 CPU emulator (independent of host architecture), capable of running DOS programs that require real or protected mode.

Flycast is a multi-platform Sega Dreamcast, Naomi and Atomiswave emulator derived from reicast.

PCSX2 is a PlayStation 2 ’emulator’, a free program that tries to replicate the PlayStation 2 console to enable you to play PS2 games on your PC.

RPCS3 is an experimental open-source Sony PlayStation 3 emulator and debugger written in C++ for Windows and Linux. RPCS3 started development in May of 2011 by its founders DH and Hykem. The emulator is currently capable of running over 1800 commercial titles powered by Vulkan and OpenGL.

MAME is a Arcade Machine Emulator.

xemu is an original Xbox emulator.

Xenia is an Xbox 360 Emulator.

Also checkout these subreddits for more great Game Emulators recommendations

Foundations/Projects

Back to the Top

Open Source Hardware Association (OSHWA) is a non-profit organization that advocates for open-source hardware. It aims to act as a hub of open source hardware activity of all types while actively cooperating with other initiatives such as the TAPR Open Hardware License, open-source development groups at CERN, and the Open Source Initiative (OSI).

The Open Connectivity Foundation is dedicated to ensuring secure interoperability for consumers, businesses and industries by delivering a standard communications platform, a bridging specification, an open source implementation and a certification program allowing devices to communicate regardless of form factor, operating system, service provider, transport technology or ecosystem.

Raspberry Pi Foundation is a UK-based charity with the mission to enable young people to realise their full potential through the power of computing and digital technologies.

OpenSSF(Open Source Security Foundation) is a cross-industry forum for a collaborative effort to improve open source software security.

OpenJS Foundation is the premier home for critical open source JavaScript projects, including Appium, Dojo, jQuery, Node.js, and webpack, and 27 more.

EdgeX Foundry is a vendor-neutral project under the Linux Foundation. The initiative is aligned around a common goal: the simplification and standardization of the foundation for edge computing architectures in the Industrial IoT market, while still allowing the ecosystem to add significant value.

Eclipse Foundation provides our global community of individuals and organizations with a mature, scalable and commercially-friendly environment for open source software collaboration and innovation.

System Hardware

Back to the Top

CPUs

Intel Processors(x86)

Back to the Top

I recommend using Intel CPUs no older than the second generation of the Intel Core processors (Core i7, i5, i3) AKA Sandy Bridge(Jan. 2011) for those that want to utilize Intel® Quick Sync Video. Though, if you’re concerned about power efficiency(~5W idle) I would recommend 7th Generation or newer.

Also, I recommend using Intel® QuickAssist Technology (Intel® QAT) a scalable, flexible, and extendable way to accelerate data encryption/decryption and compression for applications from networking to enterprise, cloud to storage, and content delivery to database. Available in 3rd Gen Intel® Xeon® Scalable Processors and Intel Atom® Processor C Series/P Series.

AMD Processors(x86)

Back to the Top

Devices

Back to the Top

Note: Will be adding more device soon!

Raspberry Pi
Turing Pi 2
Home Assistant Yellow
ZimaBoard
ODROID-H3 and H3+
Intel® NUC Mini PCs
Beelink mini PC
M1 Mac Mini
Nexcom Industrial Computers
Aeotec MultiSensor 7, 6-in-1 Zwave Sensors
reTerminal Raspberry Pi (CM4 module) all-in-one board
KOOLCORE R1 - The smallest mini PC with 4 x 2.5G LANs
Khadas VIM1S
Asustor DriveStor 4 NAS
TRENDnet TEG-S350 (2.5 GbE) Switch
Storinator™ is a line of Ultra-Large, Direct-Wired storage Servers by 45Drives.
HL15 from 45HomeLab is an open-source, open-platform, 15-bay homelab server. The HL15 features enterprise architecture and strength brought to a scale that works for the homelab. The server’s direct-wired architecture can provide blazing fast transfer speed of up to 2GB per second.
LattePanda Sigma is a powerful and compact x86 Windows single board computer (SBC). It features the 13th Intel® Core™ i5-1340P Rapter Lake (12-Core, 16-Thread) processor and 16GB Dual-Channel LPDDR5-6400MHz memory.
Apex Storage X21 is a storage solution that gives you have the freedom to choose system hardware thatworks best for you with the following benefits.
- Host 21 x M.2 Gen 4 NVME SSD’s
- 168 TB + Storage Per Card
- 31 GBps Read/Write Speeds
- Industry Leading IOPS
- 100 PCIe 4.0 Lanes
- Full UEFI/Secure Boot Support
GL.iNet is a leading developer of OpenWrt Wi-Fi and IoT Network Solutions. They build Wi-Fi routers, IoT gateways and remote device management platforms for a wide range of scenarios. All their routers include powerful built-in firewall, Shadow also supports OpenVPN, WireGuard® and customized DNS server in order to level up your online security.
Protectli Vault is a series of small computers that have firewalls with advanced firmware protection to keep your network safe. All Vaults are Opertaing System (OS) agnostic and support a variety of popular open source software distributions. Run a firewall like OPNsense, pfSense, or others. Also, run any hypervisor on your system.
Espclicker is a small device that you can use to integrate not IoT devices into your smart home setup, It connects directly to the push buttons of the device, which you want to control, and then simulates pressing them. You can simulate button clicks, hold time, and multiple clicks.
Pockethernet is a smartphone connected Ethernet network analyzer & cable tester that fits into your pocket. It helps any IT admin dealing with Ethernet network installation and maintenance. It lets you check the ethernet link, find cable faults, PoE voltage, VLAN, DHCP results and much more with the press of a button.
$13 USD voice assistant remote for Home Assistant
ATOM ECHO is a Programmable Smart Speaker based on the M5ATOM design. Music can be played using the BT (Bluetooth Technology) capabilities of the ESP32 from a mobile phone or tablet. The device could be programmed to access AWS, Azure, and other cloud platforms, using the built-in microphone and speaker for voice interaction.

Operating Systems

Back to the Top

Creating a bootable media device(USB/MicroSD card)

Rufus is a utility that helps format and create bootable USB flash drives.

Rufus

Etcher is an open source, cross-platform software that makes it easy to flash operating system images to a microSD card or USB device.

Etcher UI

A List of Operating Systems that are great for either settig up a personal Home Server or a Enterprise Server for your Organization/Company.

Home Assistant OS is a container-based system for managing your Home Assistant Core installation and related applications. The system is controlled via Home Assistant which communicates with the Supervisor. The Supervisor provides an API to manage the installation. This includes changing network settings or installing and updating software.

Home Assistant OS

Umbrel is an OS for running a personal server in your home. It can Self-host open source apps like Nextcloud, Bitcoin node, and more.

Umbrel

CasaOS is a simple, easy-to-use, elegant open-source Home Cloud system.

CasaOS

TrueNAS® CORE is the world’s most popular storage OS because it gives you the power to build your own professional-grade storage system to use in a variety of data-intensive applications without any software costs. It’s based on FreeBSD and Linux, using the OpenZFS file system.

TrueNAS CORE

Alpine Linux is a security-oriented, lightweight Linux distribution based on musl libc and busybox.

Xfce4 Desktop

Enable the Community repository, then execute command:

apk add xfce4

Alpine Linux Xfce

Mate Desktop

Enable the Community repository, then execute command:

apk add mate-desktop-environment

Alpine Linux MATE

Ubuntu is a modern open source operating system on Linux for the enterprise Server, Desktop, Cloud, and IoT developed by Canonical.

Ubuntu Server
Ubuntu for ARM
Ubuntu for Raspberry Pi
Ubuntu Flavours is for those that prefer an alternative desktop environment such as KDE Plasma Desktop, MATE, Xfce, LXQt, Budgie, and UKUI you can download a Flavour for your preferred desktop environment and use that to install Ubuntu, pre-configured for the desktop environment of your choice.

Ubuntu

Debian is an operating system and a distribution of Free Software. It is maintained and updated through the work of many users who volunteer their time and effort.

Debian 11

Linux Mint is a modern, elegant, and comfortable open source operating system(based on Debian and Ubuntu), which is both powerful and easy to use for both new and advanced users. The flagsip version of Linux Mint uses the Cinnamon desktop environment similiar to Windows 7.

Linux Mint

Linux Mint Debian Edition (LMDE) uses Debian Bullseye as the base for a very stable and rock solid user experience with the Cinnamon desktop.

Linux Mint Debian Edition (LMDE)

Pop!_OS created by System76.

Pop!_OS

Fedora Linux is a polished, easy to use operating system for laptop & desktop computers, with a complete set of tools for developers and makers of all kinds. The OS serves as the foundation for which you can scale existing apps and roll out emerging technologies across bare-metal, virtual, container, and all types of cloud environments.

Fedora Spins is for those that prefer an alternative desktop environment such as KDE Plasma Desktop, MATE; or Xfce, you can download a spin for your preferred desktop environment and use that to install Fedora, pre-configured for the desktop environment of your choice.
Fedora Server is a powerful, flexible operating system that includes the best and latest datacenter technologies. It puts you in control of all your infrastructure and services.
Fedora ARM is an initiative to bring versions of Fedora tailored for running on ARM-based systems.
Fedora Silverblue is a variant of the Fedora Workstation that uses rpm-ostree to provide an immutable OS image with reliable updates and easy rollbacks.
Fedora Kinoite is an immutable desktop operating system. It aims to be extremely stable and reliable. It also aims to be an excellent platform for developers and for those using container-focused workflows. Kinoite is a variant of the Fedora KDE Spin.
Fedora CoreOS is an automatically-updating, minimal operating system for running containerized workloads securely and at scale.

Fedora Linux

CentOS Stream is a continuously delivered distro(uses the Fedora OS base) that tracks just ahead of Red Hat Enterprise Linux (RHEL) development, positioned as a midstream between Fedora Linux and RHEL.

CentOS Stream

Red Hat® Enterprise Linux® (RHEL) the world’s leading enterprise Linux platform. The OS serves as the foundation for which you can scale existing apps and roll out emerging technologies across bare-metal, virtual, container, and all types of cloud environments.

Red Hat Enterprise Linux Desktop

AlmaLinux is an open source enterprise-ready Linux distribution forked from Red Hat Enterprise Linux(RHEL). It’s a very stable/solid operating system especially for production envrionments such as servers, though, you can also install a GUI on AlmaLinux and use it as a desktop OS. It was founded by the team behind the CloudLinux OS.

AlmaLinux Desktop Setup and Install

Rocky Linux is a community enterprise operating system designed to be 100% bug-for-bug compatible and forked from Red Hat Enterprise Linux(RHEL) now that its downstream partner(Red Hat) has shifted direction.

Rocky Linux Desktop

SUSE is a leading Linux OS most adaptable Linux operating system and the only open Kubernetes management platform thanks to their acquistion of Rancher. They also developer of SUSE Linux Enterprise and the primary sponsor of the community-supported openSUSE Project, which develops the openSUSE Linux distribution.

openSUSE Leap is a brand new way of building openSUSE and is new type of a hybrid Linux distribution. Leap uses source from SUSE Linux Enterprise (SLE), which gives Leap a level of stability unmatched by other Linux distributions, and combines that with community developments to give users, developers and sysadmins the best stable Linux experience available.
openSUSE Tumbleweed is a pure rolling release version of openSUSE containing the latest “stable” versions of all software instead of relying on rigid periodic release cycles. The project does this for users that want the newest stable software.
openSUSE Kubic is a multi-purpose Standalone & Kubernetes Container Operating System based on openSUSE MicroOS. Kubic uses kubeadm to provide an easy way of configuring a Kubernetes cluster across multiple machines, while our MicroOS base keeps your operating system updated automatically, with fully atomic rollbacks if required.
openSUSE MicroOS is a M icro Service OS providing Transactional (Atomic) Updates upon a read-only btrfs root filesystem. It’s designed to host container workloads with automated administration & patching.

SUSE Linux Enterprise 12

openSUSE

NixOS is a Linux distribution built on top of the Nix package manager. It has tools dedicated to DevOps and deployment tasks. NixOS Guide

Nix Tour is an interactive tour that uses the actual package manager to learn you the language by example, in the browser.
Nix is a package manager and build system that parses reproducible build instructions specified in the Nix Expression Language, is a pure functional language with lazy evaluation. Nix expressions are pure functions taking dependencies as arguments and producing derivation specifying a reproducible build environment for the package. Nix stores the results of the build in unique addresses specified by a hash of the complete dependency tree, creating an immutable package store that allows for atomic upgrades, rollbacks and concurrent installation of different versions of a package, essentially eliminating dependency hell.
Nix Expression Language is a pure, lazy, functional language. Purity means that operations in the language don’t have side-effects (for instance, there is no variable assignment). The language is not a full-featured, general purpose language. Its main job is to describe packages, compositions of packages, and the variability within packages.
Nixpkgs is the largest repository of Nix packages(over 80,000 packages) and NixOS modules. The repository is hosted on GitHub and maintained by the community, with official backing from the NixOS Foundation. Additionally, checkout Language-specific package helpers and Alternative Package Sets.
NixOS Packages Search is a tool for searching through NixOS packages.

NixOS Packages Search

NixOS Desktop with the new Calamares Installer

NixOS with the Plasma Desktop

BSD

Back to the Top

FreeBSD is an Unix-like operating system used to power modern servers, desktops, and embedded platforms. A large community has continually developed it for more than thirty years. Its advanced networking, security, and storage features have made FreeBSD the platform of choice for many of the busiest web sites and most pervasive embedded networking and storage devices.

FreeBSD Software ports. Source: FreeBSD Software

OpenBSD is a security-focused, free and open-source, Unix-like operating system based on the Berkeley Software Distribution. It comes with a secure minimal firewall, webserver, mailserver, and an optional graphical desktop.

NetBSD is a free, fast, secure, and highly portable Unix-like Open Source operating system. It is available for a wide range of platforms, from large-scale servers and powerful desktop systems to handheld and embedded devices.

NetBSD Documentation

DragonFly BSD is a free and open-source Unix-like operating system forked from FreeBSD 4.8.

DragonFly Documentation

The BSD Desktop for the average user

GhostBSD is a simple desktop-oriented operating system based on FreeBSD with MATE, OpenRC and OS packages for simplicity. GhostBSD has a selection of commonly used software preinstalled and required to start using it to its full potential.

GhostBSD Desktop. Source: GhostBSD

Storage

Back to the Top

-Storage Reference Guide by Storage Review

Useful Tools for Storage Management

Scrutiny is a WebUI for smartd Hard Drive S.M.A.R.T Monitoring, Historical Trends & Real World Failure Thresholds.

smartd is SMART Disk Monitoring Daemon for Linux. It controls and monitors storage systems using the Self-Monitoring, Analysis and Reporting Technology System (SMART) built into most modern ATA/SATA, SCSI/SAS and NVMe disks. In many cases, these utilities will provide advanced warning of disk degradation and failure.

DUA (Disk Usage Analyzer) is a tool to conveniently learn about the usage of disk space of a given directory. It’s parallel by default and will max out your SSD, providing relevant information as fast as possible. Optionally delete superfluous data, and do so more quickly than rm.

Perkeep is a set of open source formats, protocols, and software for modeling, storing, searching, sharing and synchronizing data. It can be easily accessed via a phone, browser or FUSE filesystem.

duf is a Disk Usage/Free Utility for Linux, BSD, macOS & Windows.

Dirstat-rs is a fast, cross-platform disk usage CLI, similar to Windirstat.

Dutree is a tool to analyze file system usage written in Rust.

Shufflecake is a tool for Linux that allows to create multiple hidden volumes on a storage device in such a way that it is very difficult, even under forensic inspection, to prove the existence of such volumes.

btdu is a sampling disk usage profiler for btrfs.

Btrfs maintenance toolbox is a set of scripts supplementing the btrfs filesystem and aims to automate a few maintenance tasks. This means the scrub, balance, trim or defragmentation.

File systems

Back to the Top

FSArchiver is a system tool that allows you to save the contents of a file system to a compressed archive file. The file system can be restored on a partition which has a different size and it can be restored on a different file system.

WekaFS is the world’s fastest shared parallel file system and delivers unmatched performance at ANY scale while offering the same enterprise features and benefits of traditional storage. It meets all storage challenges, delivering 10x the performance of legacy network attached storage (NAS) systems and 3x the performance of local server storage.

GlusterFS is a free and open source scalable network filesystem. Gluster is a scalable network filesystem. Using common off-the-shelf hardware, you can create large, distributed storage solutions for media streaming, data analysis, and other data- and bandwidth-intensive tasks.

Ceph is a software-defined storage solution designed to address the object, block, and file storage needs of data centers adopting open source as the new norm for high-growth block storage, object stores and data lakes. Ceph provides enterprise scalable storage while keeping CAPEX and OPEX costs in line with underlying bulk commodity disk prices.

ZFS is an enterprise-ready open source file system and volume manager with unprecedented flexibility and an uncompromising commitment to data integrity.

ZFSBootMenu is a Linux bootloader that attempts to provide an experience similar to the FreeBSD bootloader. It takes advantage of ZFS features, it allows a user to have multiple “boot environments” (with different distros, for example), manipulate snapshots before booting, and even bootstrap a system installation via zfs recv.

OpenZFS is an open-source storage platform. It includes the functionality of both traditional file systems and volume manager. It has many advanced features including:

Protection against data corruption.
Integrity checking for both data and metadata.
Continuous integrity verification and automatic “self-healing” repair.

Btrfs is a modern copy on write (CoW) filesystem for Linux aimed at implementing advanced features while also focusing on fault tolerance, repair and easy administration. Its main features and benefits are:

Snapshots which do not make the full copy of files
RAID - support for software-based RAID 0, RAID 1, RAID 10
Self-healing - checksums for data and metadata, automatic detection of silent data corruptions

Composefs is a native Linux file system designed to help sharing filesystem contents, as well as ensuring said content is not modified. The initial target usecase are container images and ostree commits.

MergerFS is a union filesystem geared towards simplifying storage and management of files across numerous commodity storage devices. It is similar to mhddfs, unionfs, and aufs.

MergerFS Features

Configurable behaviors / file placement
Ability to add or remove filesystems at will
Resistance to individual filesystem failure
Support for extended attributes (xattrs)
Support for file attributes (chattr)
Runtime configurable (via xattrs)
Works with heterogeneous filesystem types
Moving of file when filesystem runs out of space while writing
Ignore read-only filesystems when creating files
Turn read-only files into symlinks to underlying file
Hard link copy-on-write / CoW
Support for POSIX ACLs

Proxmox Cluster File System (PMXCFS) is a File System used to transparently distribute the cluster configuration to all cluster nodes.

UnionFS is a filesystem service for Linux, FreeBSD and NetBSD which implements a union mount for other file systems. It allows files and directories of separate file systems, known as branches, to be transparently overlaid, forming a single coherent file system.

OverlayFS is a modern union filesystem that is similar to AUFS, but faster and with a simpler implementation. It’s typically used on systems running on embed devices, like OpenWRT, where is useful to preserve a basic set of configurations and at the same time allowing the user to perform modifications.

Bcachefs is an advanced new filesystem for Linux, with an emphasis on reliability and robustness and the complete set of features one would expect from a modern filesystem. Scalability has been tested to 50+ TB, will eventually scale far higher.

Squashfs is a compressed read-only filesystem for Linux. It uses zlib, lz4, lzo, or xz compression to compress files, inodes and directories. Inodes in the system are very small and all blocks are packed to minimize data overhead.

SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.

CubeFS is a cloud native distributed storage platform. It’s commonly used as the storage infrastructure for online applications, database or data processing services and machine learning jobs orchestrated by Kubernetes.

Apple File System (APFS) is the default file system for Mac computers using macOS 10.13 or later, features strong encryption, space sharing, snapshots, fast directory sizing, and improved file system fundamentals.

NTFS(New Technology File System) is the primary file system for recent versions of Windows and Windows Server—provides a full set of features including security descriptors, encryption, disk quotas, and rich metadata, and can be used with Cluster Shared Volumes (CSV) to provide continuously available volumes that can be accessed simultaneously from multiple nodes of a failover cluster.

exFAT(Extended File Allocation Table ) is the file system that was the successor to FAT32 in the FAT family of file systems. It was optimized for flash memory such as USB flash drives and SD cards.

Books

Back to the Top

Self-hosted Cookbook is a cookbook, for docker-compose based recipes, for self-hosted applications and services.
Geek’s Cookbook is a collection of guides for establishing your own highly-available “private cloud” and using it to run self-hosted services such as GitLab, Plex, NextCloud, etc.
Database Books(PDFs)

Podcasts

Back to the Top

Self-Hosted Podcast is a chat show between Chris and Alex two long-time “self-hosters” who share their lessons and take you along for the journey as they learn new ones.
Self-Hosted SRE(Site Reality Engineer) Podcast is a feed to say thank you to our Self-Hosted Site Reality Engineers!
Home Assistant Podcast is a biweekly podcast with the latest news and interesting guests.

WireGuard

Back to the Top

What is WireGuard?

Back to the Top

WireGuard® is a straight-forward, fast and modern VPN that utilizes state-of-the-art cryptography. It aims to be faster, simpler, leaner, and more useful than IPsec while avoiding the massive headache. WireGuard is designed as a general-purpose VPN for running on embedded interfaces and super computers alike, fit for many circumstances. Initially released for the Linux kernel, it is now cross-platform (Windows, macOS, BSD, iOS, Android) and widely deployable.

What is Tailscale?

Back to the Top

How NAT Traversal works on a Home router. Credit: Tailscale.

Headscale is an open source, self-hosted implementation of the Tailscale coordination server.

What is Netmaker?

Back to the Top

Netmaker is a tool that enables you to create relays, gateways, full VPN meshes, and even zero trust networks. It’s fully configurable to let you maximize the power of Wireguard.

NetMaker Architecture. Credit: Netmaker.

WireGuard Tools

Back to the Top

Wiretrustee is a WireGuard®-based mesh network that connects your devices into a single private network.

Wireguard Manager is a tool that enables you to build your own vpn under a minute.

Headscale is an open source, self-hosted implementation of the Tailscale coordination server.

Firezone is a self-hosted WireGuard®-based VPN server and Linux firewall.

NetBird is an open-source VPN management platform built on top of WireGuard® making it easy to create secure private networks for your organization or home.

Mistborn is a secure platform for easily standing up and managing your own cloud services: including firewall, ad-blocking, and multi-factor WireGuard VPN access.

Mistborn CLI is a Command-line interface for Mistborn.

BoringTun is an implementation of the WireGuard® protocol designed for portability and speed. It’s successfully deployed on millions of iOS and Android consumer devices as well as thousands of Cloudflare Linux servers.

PiVPN is the simplest VPN installer, designed for Raspberry Pi.

Algo VPN is a set of Ansible scripts that simplify the setup of a personal WireGuard and IPsec VPN. It uses the most secure defaults available and works with common cloud providers.

Pro Custodibus is a tool for managing WireGuard with a variety of business VPN (Virtual Private Network) use cases, such as site-to-site connectivity, secure remote access from anywhere, secure access to the cloud (Amazon Web Services, Google Cloud Platform, Microsoft Azure, etc), and more.

Drago is a flexible configuration manager for WireGuard designed to make it simple to configure secure network overlays spanning heterogeneous nodes distributed across different clouds and physical locations. Drago is in active development, and we welcome contributions from the open-source community.

Netmaker is a tool that helps connect any computers together over a secure, fast, private network, and manage multiple networks from a central server.

Kilo is a multi-cloud network overlay built on WireGuard and designed for Kubernetes. Kilo connects nodes in a cluster by providing an encrypted layer 3 network that can span across data centers and public clouds. The Pod network created by Kilo is always fully connected, even when the nodes are in different networks or behind NAT. By allowing pools of nodes in different locations to communicate securely, Kilo enables the operation of multi-cloud clusters. Kilo’s design allows clients to VPN to a cluster in order to securely access services running on the cluster.

Subspace is a simple WireGuard VPN server GUI.

WG UI is a basic, self-contained management service for WireGuard with a self-serve web UI.

WireHole is a combination of WireGuard, PiHole, and Unbound in a docker-compose project with the intent of enabling users to quickly and easily create and deploy a personally managed full or split-tunnel WireGuard VPN with ad blocking capabilities (via Pihole), and DNS caching with additional privacy options (via Unbound).

Gluetun is a lightwieght VPN client in a thin Docker container for multiple VPN providers, written in Go, and uses OpenVPN or Wireguard, DNS over TLS, with a few proxy servers built-in.

Ethr is a cross platform network performance measurement tool written in golang. The goal of this project is to provide a native tool for comprehensive network performance measurements of bandwidth, connections/s, packets/s, latency, loss & jitter, across multiple protocols such as TCP, UDP, HTTP, HTTPS, and across multiple platforms such as Windows, Linux and other Unix systems.

Setting up WireGuard with PiVPN

Back to the Top

Installing PiVPN:

sudo apt install curl -y

curl -L https://install.pivpn.io | bash

Setting up WireGuard on Unraid

Back to the Top

Select Apps, then search for WireGuard and install Wireguard-Easy.

VPN manager

Almost all of the settings can stay as default, however, there are a few that we will modify.

Set the WG_HOST variable to be the IP address of your Unraid server.
If you’d like to modify the WireGuard port (51820), you can do that here.
Change the default Web GUI password.

Setting up WireGuard on pfSense

Back to the Top

When looking at how to set up WireGuard on pfSense, the first thing that we need to do is install the package. Follow the instructions below to install the WireGuard package on pfSense.

Open the Package Manager and search for WireGuard, then Install the latest version of the package.

After the package has installed, select VPN then WireGuard and under the Tunnels section, select Add Tunnel.
In the Tunnel Configuration, set the Description as WireGuard, the Listen Port as 51820, then Generate private and public keys.
Copy the Public Key. We will need this for our client configuration.
Create the tunnel, then select Settings, and ensure that Enable WireGuard is selected. Then Save and Apply.

Setting up WireGuard on OpenWRT

Back to the Top

Quick Links:

In your router’s webUI, navigate to System - Software, click Update lists:

In the Filter field, type WireGuard, locate and install the wireguard, wireguard-tools, kmod-wireguard, and luci-app-wireguard packages. Note: The wireguard package is included in version 22.02.

Generate WireGuard keypair

SSH into your router as ‘root’ (OpenWrt Wiki):

ssh root@192.168.1.1

Generate WireGuard keys:

wg genkey | tee privatekey | wg pubkey > publickey

chmod 600 privatekey

Note your Private & Public keys, you will need them later:

cat privatekey

cat publickey

Creating an Interface

Navigate to Network - Interface,

Click the Add new interface… button and enter the following configuration:

Name - give it any name
Protocol - WireGuard VPN

Create interface

In the General Settings tab:

Bring up on boot - Checked
Private Key - copy and paste the generated previously Private key
IP Address - enter the WireGuard IP Address obtained in the Client Area ending with /32, e.g. 172.27.124.169/32

Add a Firewall zone

Navigate to Network - Firewall

Click the Add button and enter the following configuration:

Name - Give it any name
Input - Reject
Output - Accept
Forward - Reject
Masquerading - Checked
MSS clamping - Checked
Covered networks - select the previously created VPN tunnel interface
Allow forward to destination zones - Unspecified
Allow forward from source zones - lan

DNS

Navigate to Network - Interfaces

Click on the Edit button next to the WAN interface

In the Advanced Settings tab, uncheck the Use DNS servers advertised by peer and specify one of the following DNS servers in the Use custom DNS servers field:

172.16.0.1 = regular DNS with no blocking
10.0.254.2 = standard AntiTracker to block advertising and malware domains
10.0.254.3 = Hardcore Mode AntiTracker to also block Google and Facebook domains

Click the Save button.

Last Steps

A device reboot is not required, though it may be useful to confirm that everything behaves as expected.
Run a leak test at https://www.dnsleaktest.com via one of the internal network clients attached to your OpenWRT router.

Setting up WireGuard on Home Assistant

Back to the Top

Install Wireguard Add-on in Home Assistant

Next, open up Home Assistant. Go to Supervisor > Add-on store, and search for WireGuard.
Click the WireGuard addon, and the click Install.

Configure Wireguard Settings

After installing WireGuard, do not start it yet. We need to configure a few options first.

Click the Configuration tab at the very top.
There are two blocks of code here: server and peers. The server section is the WireGuard server info, and the peers section is where you’d add new devices that will connect to your VPN.

Server Configuration

Host: add the subdomain you just created. (vpn.mydomain.com)
Addresses: If your internal network is using the 192.168.x.x or 10.x.x.x range, you can leave the default IP addresses WireGuard has provided. (see note above)
DNS: Set to your router’s internal IP address (Open CMD > ipconfig /all > Under DNS servers) If you have Adguard or PiHole installed, you can use the IP address of those instead. This will allow you to block ads even when connected to the WireGuard VPN.

Peers Configuration

This is where you’ll create WireGuard configuration files for each of the devices you want to connect to WireGuard with. For this example, I’m using my phone and leaving allowed_ips and client_allowed_ips as is. If you adding multiple devices, then you’ll need to copy the entire block of code starting at name, give it a different name, and add the next available IP address (For example: 172.27.66.4)

Click Save once finished.

Then, go back to the Info tab and click Start.

Port Forward

The next step is to forward port 51820 from your Home Assistant server through your router. Unfortunately, there are so many different types of routers, each with different steps to port forward. The important thing to note is that you’ll be port forwarding 51820(wireguard port) from the internal IP of your Home Assistant instance (for example: 192.168.68.24) and choosing the UDP protocol only.

Download Wireguard app on mobile device

Download the WireGuard app from the Apple App Store or Google Play Store. You will need it for the next step.

If all goes well, you can click into the new tunnel connection from within the app. If you see data flowing under the Transfer section, that means you are good to go.

Improving Security

Once you have everything setup and working correctly, you should read through the WireGuard Addon docs to setup up allowed_ips and client_allowed_ips to further secure your VPN instance. There’s also some other helpful options you can configure such as log level, but these are all optional.

Nextcloud

Back to the Top

Nexcloud login screen

Nexcloud Hub

Nextcloud AIO (All In One) is a tool that provides easy deployment and maintenance with most features included in this one Nextcloud instance.

Features it includes:

Nextcloud
Nextcloud Office
High performance backend for Nextcloud Files
High performance backend for Nextcloud Talk
Backup solution (based on BorgBackup)
Imaginary
ClamAV
Fulltextsearch

Nextcloud Desktop Client is a tool to synchronize files from Nextcloud Server with your computer.

Nextcloud Deck is a kanban style organization tool aimed at personal planning and project organization for teams integrated with Nextcloud.

Nextcloud Home is a tool that allows you store your documents, calendar, contacts and photos on your server at home, at one of at one Nextcloud’s providers or in a data center you trust.

Nextcloud Enterprise is a service that gives professional organizations software optimized and tested for mission critical environments.

Nextcloud Outlook Integration is a tool that automatically upload files to replace large attachments or integrate Calendars and Contacts in Microsoft Outlook.

LibreSign is a Libre digital signature app for Nextcloud.

Raspberry Pi

Back to the Top

Models of Raspberry Pi boards

Back to the Top

Raspberry Pi 4 Model B

Check out the Raspberry Pi 4

Raspberry Pi 4 Model B Hardware Specifications

Broadcom BCM2711, Quad core Cortex-A72 (ARM v8) 64-bit SoC @ 1.5GHz
2GB, 4GB or 8GB LPDDR4-3200 SDRAM (depending on model)
2.4 GHz and 5.0 GHz IEEE 802.11ac wireless
Bluetooth 5.0, BLE
Gigabit Ethernet
2 USB 3.0 ports; 2 USB 2.0 ports.
Raspberry Pi standard 40 pin GPIO header (fully backwards compatible with previous Pi boards)
2 × micro-HDMI ports (up to 4kp60 supported)
OpenGL ES 3.0 graphics

Raspberry Pi 400 Personal Computer Kit

Check out the Raspberry Pi 400 Personal Computer Kit

Raspberry Pi 400 Hardware Specifications

Broadcom BCM2711, Quad core Cortex-A72 (ARM v8) 64-bit SoC @ 1.8GHz
4GB LPDDR4-3200 SDRAM
2.4 GHz and 5.0 GHz IEEE 802.11ac wireless
Bluetooth 5.0, BLE
Gigabit Ethernet
2 USB 3.0 ports; 2 USB 2.0 ports.
Raspberry Pi standard 40 pin GPIO header
2 × micro-HDMI ports (up to 4kp60 supported)
OpenGL ES 3.0 graphics

Raspberry Pi Pico Microcontroller

Check out the Raspberry Pi Pico

Raspberry Pi Pico Hardware Specifications

RP2040 microcontroller chip designed by Raspberry Pi in the UK
Dual-core Arm Cortex-M0+ processor, flexible clock running up to 133 MHz
264KB on-chip SRAM
2MB on-board QSPI Flash
26 multifunction GPIO pins, including 3 analogue inputs
2 × UART, 2 × SPI controllers, 2 × I2C controllers, 16 × PWM channels
1 × USB 1.1 controller and PHY, with host and device support
8 × Programmable I/O (PIO) state machines for custom peripheral support
Castellated module allows soldering direct to carrier boards
Drag-and-drop programming using mass storage over USB
Low-power sleep and dormant modes
Accurate on-chip clock
Temperature sensor
Accelerated integer and floating-point libraries on-chip

Raspberry Pi OS. The default Operating System for every Raspberry Pi device

Check out Raspberry Pi OS

Raspberry Pi Learning Resources

Back to the Top

Raspberry Pi is an ARM powered single board computer(SBC) that is the size of a credit card and costs around $35.

Raspberry Pi Foundation is a UK-based charity that works to put the power of computing and digital making into the hands of people all over the world.

Microsecond accurate NTP with a Raspberry Pi and PPS GPS

Getting Started with Raspberry Pi Projects

Online learning for the Raspberry Pi

Raspberry Pi Training Program

Raspberry Pi Online Courses on Udemy

Raspberry Pi Online Courses on Coursera

The Raspberry Pi Platform and Python Programming course on Coursera

Learning Raspberry Pi with Online Courses on edX

Raspberry Pi Online Training Courses on LinkedIn Learning

Getting Started with Raspberry Pi course on FutureLearn

Home Assistant on Raspberry Pi

PiSwitch: Build your own Nintendo Switch-style console

Raspberry Pi Operating Systems

Back to the Top

Raspberry Pi OS

Hass.io(Home Assistant OS)

OmniROM(Android 11) based on ASOP

Manjaro Linux ARM

Arch Linux ARM

Ubuntu MATE for Raspberry Pi

Ubuntu Desktop for Raspberry Pi

Ubuntu Core on a Raspberry Pi

Ubuntu Server for ARM

Fedora ARM

Kali Linux for the Raspberry Pi

Raspberry Pi Tools

Back to the Top

Raspberry Pi Imager is the quick and easy way to install Raspberry Pi OS and other operating systems to a microSD card, ready to use with your Raspberry Pi.

Raspberry Pi Locator is a website to track Raspberry Pi 4 model B, Compute Module 4, Pi Zero 2 W, and Pico availability across multiple retailers in different countries.

Raspberry Pi Network Install (Beta) is a feature can be used to start the Raspberry Pi Imager application directly on a Raspberry Pi 4, or a Raspberry Pi 400, by downloading it from the internet using an Ethernet cable. The Raspberry Pi Imager application, which will run in memory on your Raspberry Pi, can then be used to flash the operating system onto a blank SD Card or USB disk, just like normal.

Raspberry Pi Bootloader is a feature, which is now available in beta, that utilize an EEPROM(Electrically Erasable Programmable Read-Only Memory) to store the system’s bootloader. This EEPROM is persistent storage that is located on the Pi’s mainboard. The advantage of using the EEPROM instead is that the Raspberry Pi 4 can perform tasks without needing any storage to be attached.

Etcher is an open source, cross-platform software that makes it easy to flash operating system images to a microSD card or USB device.

Gladys Assistant is a privacy-first, open-source home assistant and runs great on Raspberry Pi.

Kodi for Raspberry Pi is a free and open source media player application developed by the XBMC/Kodi Foundation.

PiKVM is a very simple and fully functional Raspberry Pi-based KVM over IP.

PiShrink is a bash script that automatically shrink a pi image that will then resize to the max size of the SD card on boot.

RPiPlay is an open-source implementation of an AirPlay mirroring server for the Raspberry Pi that supports iOS 9 and later.

Gpiozero is a simple interface to GPIO(General-Purpose Input/Output) devices with the Raspberry Pi.

Balena Sound is a single or multi-room streamer for an existing audio device using a Raspberry Pi! It supports Bluetooth, Airplay and Spotify Connect.

OpenBalena is a platform to deploy and manage connected devices.

Home Assistant

Back to the Top

Home Assistant is a container-based system for managing your Home Assistant Core installation and related applications. The system is controlled via Home Assistant which communicates with the Supervisor. The Supervisor provides an API to manage the installation. This includes changing network settings or installing and updating software.

Quick Links

Home Assistant Frontend is a frontend for Home Assistant.

Tools to write the HA image to your boot media(microSD card or USB device)

Raspberry Pi Imager is the quick and easy way to install Raspberry Pi OS and other operating systems to a microSD card, ready to use with your Raspberry Pi.

Etcher is an open source, cross-platform software that makes it easy to flash operating system images to a microSD card or USB device.

Home Assistant integrations

Home Assistant integrations. Credit: Home Assistant

ESPHome is a system to control your ESP8266/ESP32 by simple yet powerful configuration files and control them remotely through Home Automation systems.

Amazon Alexa is a smart virtual assistant software to manage Alexa-enabled devices, control music playback, view shopping lists on the go, keep track of upcoming reminders, check on active timers and much more.

Google Assistant is a smart virtual assistant software on mobile and home automation devices.

Apple HomeKit is a software framework that enables your app to coordinate and control home automation accessories from multiple vendors to present a coherent, user-focused interface. Using HomeKit, your app can: Discover HomeKit-compatible automation accessories and add them to a persistent, cross-device home configuration database.

Samsung SmartThings is a sofwtare frmaeowrk that you can connect, monitor and control multiple smart home devices quicker and easier. Connect your Samsung smart TVs, smart appliances, smart speakers and brands like Ring, Nest and Philips Hue all from one app.

Ecobee is a home automation company in Canada that makes thermostats for residential and commercial use.

Philips Hue is a smart lighting system. The smart lights, Hue Bridge, and smart controls will forever change the way you experience light.

Sonos is the wireless home sound system that fills as many rooms as you want with great-sounding music, movies, and TV.

openHAB is a cross-platform software with the aim to integrate all kinds of Smart Home technologies, devices, etc.

Z-Wave is the leading wireless communications protocol behind many of the secure, trusted brands that are working to make everyone’s home smarter and safer.

Zwavejs2Mqtt is a fully configurable Zwave to MQTT Gateway and Control Panel Web UI.

Z-Wave JS Server is a small server wrapper around Z-Wave JS to access it via a WebSocket.

Z-Wave JS Config DB Browser is the official device configuration reference to find out if your device is supported. Currently supports 387 brands, spanning at least 2075 device configurations.

Homebridge

Back to the Top

Homebridge is a software frameowrk that allows you to integrate with smart home devices that do not natively support HomeKit. There are over 2,000 Homebridge plugins supporting thousands of different smart accessories.

Tools to write the Homebridge image to your boot media(microSD card or USB device)

Raspberry Pi Imager is the quick and easy way to install Raspberry Pi OS and other operating systems to a microSD card, ready to use with your Raspberry Pi.

Etcher is an open source, cross-platform software that makes it easy to flash operating system images to a microSD card or USB device.

Homebridge UI is a tool that provides an easy to use interface to manage your Homebridge plugins, configuration and accessories.

Install and configure Homebridge plugins.
Monitor your Homebridge server via a fully customisable widget-based dashboard.
View and control Homebridge accessories.
Backup and Restore your Homebridge instance.

Homebridge UI

ESPHome

Back to the Top

ESPHome is a system to control your ESP8266/ESP32 by simple yet powerful configuration files and control them remotely through Home Automation systems.

Quick Links

Install ESPHome using Home Assistant

In Home Assistant go to:

Configuration > Add-ons, Backups & Supervisor > Add-on Store (button in the lower right corner) or click on the My Home Assistant Link below:

Open your Home Assistant instance and show the Supervisor add-on store.

Next, search for ESPHome, click on the result and then click on the Install button.

When the installation is finished, the Install button will be replaced with Start button – click on it to start the ESPHome add-on.

Wait a few seconds for the ESPHome to start and then click on the Open Web UI button.

Install ESPHome using Docker

First thing is to pull the ESPHome Docker image from Docker Hub (Online).

docker pull esphome/esphome
Then, start the ESPHome wizard. This wizard will ask you about your device type, your device name, your WiFi credentials and finally will generate a yaml file containing all of the configurations for you.

docker run --rm -v "${PWD}":/config -it esphome/esphome wizard stl.yaml

Now, connect your ESP device to the device where Docker is running (either using an USB cable or Serial-To-USB adapter) and if you are on Linux type the following command :

dmesg | grep ttyUSB

Put your device in programming mode (if needed) and execute the next command to install the ESPHome on the device connected to the /dev/ttyUSB1 using the configuration stored in stl.yaml file

docker run --rm -v "${PWD}":/config --device=/dev/ttyUSB1 -it esphome/esphome run stl.yaml

Install ESPHome using Python

If you are on macOS or Linux check if Python 3.8 or later is installed by executing the command.

python3 --version

If you are on macOS, you need to install wheel and esphome packages by using the following command.

pip3 install wheel esphome

If you are on Linux, you have to install esphome package by using the following command.

pip3 install --user esphome

If you are on macOS or Linux you can start the ESPHome wizard using the following command.

esphome wizard stl-python.yaml

Finally, connect your ESP device to your Computer (using USB cable or Serial-To-usb adapter) and put it in programming mode (if needed). Then, Install ESPHome using the configuration in the stl-python.yaml file.

esphome run stl-python.yaml

Turning Raspberry Pi into a Router

Back to the Top

Software

OpenWrt Project is a Linux operating system targeting embedded devices. Instead of trying to create a single, static firmware, OpenWrt provides a fully writable filesystem with package management. It’s primarily used on embedded devices to route network traffic.

OpenWrt Wiki - Raspberry Pi setup

Download the appropriate OpenWrt image for your Raspberry PI by going to the link above.

Tools to write the Operating System (OS) image to your boot media(microSD card)

Raspberry Pi Imager is the quick and easy way to install Raspberry Pi OS and other operating systems to a microSD card, ready to use with your Raspberry Pi.

Hardware

Raspberry Pi Router Board for CM4 module (Cost: $55 USD) is an expansion board based on the Raspberry Pi Compute Module 4. It brings Raspberry Pi CM4 two full-speed gigabit network ports and offers better performance, lower CPU usage, and higher stability for a long time work compared with a USB network card. It’s compatible with Raspberry Pi OS, Ubuntu Server and other Raspberry Pi systems.

Raspberry Pi Router Board for CM4 module

Technical Specs:

Compatible Module: Raspberry Pi Compute Module 4 series.
BCM2711 4 core @ 1.5GHz Cortex-A72.
Support standard Raspberry Pi HAT interface.
Support POE HAT to supply power to the board.
Support POE HAT for external power supply.
Full-speed dual gigabit network interface.
Master-slave dual USB2.0 interface.
Micro SD card slot, used to support non-eMMC version of CM4.
Standard HDMI video output interface.
0.91 inch IIC OLED display.
5V DC fan interface(Support controlling via PWM signal).
Ethernet: high-performance Gigabit ethernet controller RTL8111E chip, JXD 2111x G2406s chip as isolation transformer.
- Port0: Compute Module 4 Built-In.
- Port1: PCI Express 1000BASE-T NIC.
GPIO: 40-Pin GPIO compatible with Raspberry Pi.

Setting Watchdog Timer (WDT) on Raspberry Pi

Back to the Top

Watchdog Timer (WDT) is a timer that monitors microcontroller (MCU) programs to see if they are out of control or have stopped operating.

Installing and enabling WDT service

To enable watchdog you have to change the boot parameters by adding dtparam=watchdog=on in /boot/config.txt using a text editor such as nano, vim, gedit, etc.. Also, install watchdog package and enable it to start at startup. Also, make sure to restart your Raspberry Pi for these settings to take effect.

pi@raspberrypi:~ $ sudo apt install watchdog

pi@raspberrypi:~ $sudo systemctl enable watchdog

Configure WDT service

Configuration file for watchdog can be found in /etc/watchdog.conf.

max-load-1 = 24
watchdog-device = /dev/watchdog
realtime = yes
priority = 1

To start the WTD service:

pi@raspberrypi:~ $ sudo systemctl start watchdog

Check watchdog status:

pi@raspberrypi:~ $ sudo systemctl status watchdog

To stop the service:

pi@raspberrypi:~ $ sudo systemctl stop watchdog

Raspberry Pi Upgrades

Back to the Top

Raspberry Pi Cases from Pi-Shop US

Raspberry Pi Cases from The Pi Hut

X825 expansion board provides a complete storage solution for newest Raspberry Pi 4 Model B, it supports up to 4TB 2.5-inch SATA hard disk drives (HDD) / solid-state drive (SSD).

Sabrent M.2 SSD [NGFF] to USB 3.0 / SATA III 2.5-Inch Aluminum Enclosure Adapter

Samsung 970 EVO 250GB - NVMe PCIe M.2 2280 SSD

Western Digital 1TB WD Blue SN550 NVMe Internal SSD

SAMSUNG T5 Portable SSD

Samsung SSD 860 EVO 250GB mSATA Internal SSD

Samsung 850 EVO 120GB SSD mSATA

Grafana

Back to the Top

Grafana Learning Resources

Grafana is an analytics platform that enables you to query and visualize data, then create and share dashboards based on your visualizations. Easily visualize metrics, logs, and traces from multiple sources such as Prometheus, Loki, Elasticsearch, InfluxDB, Postgres, Fluentd, Fluentbit, Logstash and many more.

Getting Started with Grafana

Grafana Community

Grafana Professional Services Training | Grafana Labs

Grafana Pro Training AWS | Grafana Labs

Grafana Tutorials

Top Grafana Courses on Udemy

Grafana Online Training Courses | LinkedIn Learning

Grafana Training Courses - NobleProg

Setting Up Grafana to Visualize Our Metrics Course on Coursera

Grafana Tools

Grafana Cloud is a composable observability platform, integrating metrics, traces and logs with Grafana. Leverage the best open source observability software – including Prometheus, Loki, and Tempo – without the overhead of installing, maintaining, and scaling your observability stack.

Grafana Cloud Integrations. Source: Grafana

Grafana Enterprise is a service that includes features that provide better scalability, collaboration, operations, and governance in a self-managed environment.

Grafana Enterprise Stack. Source: Grafana

Grafana Tempo is an open source high-scale distributed tarcing backend. Tempo is cost-efficient, requiring only object storage to operate, and is deeply integrated with Grafana, Loki, and Prometheus.

Grafana MetricTank is a multi-tenant timeseries platform for Graphite developed by Grafana Labs. MetricTank provides high-availability(HA) and efficient long-term storage, retrieval, and processing for large-scale environments.

Grafana Tanka is a robust configuration utility for your Kubernetes cluster, powered by the Jsonnet language.

Grafana Loki is a horizontally-scalable, highly-available(HA), multi-tenant log aggregation system inspired by Prometheus.

Cortex is a project that lets users query metrics from many Prometheusservers in a single place, without any gaps in the grpahs due to server failture. Also, Cortex lets you store Prometheus metrics for long term capacity planning and performance analysis.

Graphite is an open source monitoring system.

Networking

Back to the Top

Networking Tools & Concepts

cURL is a computer software project providing a library and command-line tool for transferring data using various network protocols(HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, DICT, TELNET, LDAP LDAPS, MQTT, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP or SMTPS). cURL is also used in cars, television sets, routers, printers, audio equipment, mobile phones, tablets, settop boxes, media players and is the Internet transfer engine for thousands of software applications in over ten billion installations.

cURL Fuzzer is a quality assurance testing for the curl project.

DoH is a stand-alone application for DoH (DNS-over-HTTPS) name resolves and lookups.

Authelia is an open-source highly-available authentication server providing single sign-on capability and two-factor authentication to applications running behind NGINX.

nginx(engine x) is an HTTP and reverse proxy server, a mail proxy server, and a generic TCP/UDP proxy server, originally written by Igor Sysoev.

Wireshark is a very popular network protocol analyzer that is commonly used for network troubleshooting, analysis, and communications protocol development. Learn more about the other useful Wireshark Tools available.

HTTPie is a command-line HTTP client. Its goal is to make CLI interaction with web services as human-friendly as possible. HTTPie is designed for testing, debugging, and generally interacting with APIs & HTTP servers.

HTTPStat is a tool that visualizes curl statistics in a simple layout.

Wuzz is an interactive cli tool for HTTP inspection. It can be used to inspect/modify requests copied from the browser’s network inspector with the “copy as cURL” feature.

Websocat is a ommand-line client for WebSockets, like netcat (or curl) for ws:// with advanced socat-like functions.

• Connection: In networking, a connection refers to pieces of related information that are transferred through a network. This generally infers that a connection is built before the data transfer (by following the procedures laid out in a protocol) and then is deconstructed at the at the end of the data transfer.

• Packet: A packet is, generally speaking, the most basic unit that is transferred over a network. When communicating over a network, packets are the envelopes that carry your data (in pieces) from one end point to the other.

Packets have a header portion that contains information about the packet including the source and destination, timestamps, network hops. The main portion of a packet contains the actual data being transferred. It is sometimes called the body or the payload.

• Network Interface: A network interface can refer to any kind of software interface to networking hardware. For instance, if you have two network cards in your computer, you can control and configure each network interface associated with them individually.

A network interface may be associated with a physical device, or it may be a representation of a virtual interface. The “loop-back” device, which is a virtual interface to the local machine, is an example of this.

• LAN: LAN stands for "local area network". It refers to a network or a portion of a network that is not publicly accessible to the greater internet. A home or office network is an example of a LAN.

• WAN: WAN stands for "wide area network". It means a network that is much more extensive than a LAN. While WAN is the relevant term to use to describe large, dispersed networks in general, it is usually meant to mean the internet, as a whole.

If an interface is connected to the WAN, it is generally assumed that it is reachable through the internet.

• Protocol: A protocol is a set of rules and standards that basically define a language that devices can use to communicate. There are a great number of protocols in use extensively in networking, and they are often implemented in different layers.

Some low level protocols are TCP, UDP, IP, and ICMP. Some familiar examples of application layer protocols, built on these lower protocols, are HTTP (for accessing web content), SSH, TLS/SSL, and FTP.

• Port: A port is an address on a single machine that can be tied to a specific piece of software. It is not a physical interface or location, but it allows your server to be able to communicate using more than one application.

• Firewall: A firewall is a program that decides whether traffic coming into a server or going out should be allowed. A firewall usually works by creating rules for which type of traffic is acceptable on which ports. Generally, firewalls block ports that are not used by a specific application on a server.

• NAT: Network address translation is a way to translate requests that are incoming into a routing server to the relevant devices or servers that it knows about in the LAN. This is usually implemented in physical LANs as a way to route requests through one IP address to the necessary backend servers.

• VPN: Virtual private network is a means of connecting separate LANs through the internet, while maintaining privacy. This is used as a means of connecting remote systems as if they were on a local network, often for security reasons.

Network Layers

While networking is often discussed in terms of topology in a horizontal way, between hosts, its implementation is layered in a vertical fashion throughout a computer or network. This means is that there are multiple technologies and protocols that are built on top of each other in order for communication to function more easily. Each successive, higher layer abstracts the raw data a little bit more, and makes it simpler to use for applications and users. It also allows you to leverage lower layers in new ways without having to invest the time and energy to develop the protocols and applications that handle those types of traffic.

As data is sent out of one machine, it begins at the top of the stack and filters downwards. At the lowest level, actual transmission to another machine takes place. At this point, the data travels back up through the layers of the other computer. Each layer has the ability to add its own "wrapper" around the data that it receives from the adjacent layer, which will help the layers that come after decide what to do with the data when it is passed off.

One method of talking about the different layers of network communication is the OSI model. OSI stands for Open Systems Interconnect.This model defines seven separate layers. The layers in this model are:

• Application: The application layer is the layer that the users and user-applications most often interact with. Network communication is discussed in terms of availability of resources, partners to communicate with, and data synchronization.

• Presentation: The presentation layer is responsible for mapping resources and creating context. It is used to translate lower level networking data into data that applications expect to see.

• Session: The session layer is a connection handler. It creates, maintains, and destroys connections between nodes in a persistent way.

• Transport: The transport layer is responsible for handing the layers above it a reliable connection. In this context, reliable refers to the ability to verify that a piece of data was received intact at the other end of the connection. This layer can resend information that has been dropped or corrupted and can acknowledge the receipt of data to remote computers.

• Network: The network layer is used to route data between different nodes on the network. It uses addresses to be able to tell which computer to send information to. This layer can also break apart larger messages into smaller chunks to be reassembled on the opposite end.

• Data Link: This layer is implemented as a method of establishing and maintaining reliable links between different nodes or devices on a network using existing physical connections.

• Physical: The physical layer is responsible for handling the actual physical devices that are used to make a connection. This layer involves the bare software that manages physical connections as well as the hardware itself (like Ethernet).

The TCP/IP model, more commonly known as the Internet protocol suite, is another layering model that is simpler and has been widely adopted.It defines the four separate layers, some of which overlap with the OSI model:

• Application: In this model, the application layer is responsible for creating and transmitting user data between applications. The applications can be on remote systems, and should appear to operate as if locally to the end user.

The communication takes place between peers network.

• Transport: The transport layer is responsible for communication between processes. This level of networking utilizes ports to address different services. It can build up unreliable or reliable connections depending on the type of protocol used.

• Internet: The internet layer is used to transport data from node to node in a network. This layer is aware of the endpoints of the connections, but does not worry about the actual connection needed to get from one place to another. IP addresses are defined in this layer as a way of reaching remote systems in an addressable manner.

• Link: The link layer implements the actual topology of the local network that allows the internet layer to present an addressable interface. It establishes connections between neighboring nodes to send data.

Interfaces

Interfaces are networking communication points for your computer. Each interface is associated with a physical or virtual networking device. Typically, your server will have one configurable network interface for each Ethernet or wireless internet card you have. In addition, it will define a virtual network interface called the “loopback” or localhost interface. This is used as an interface to connect applications and processes on a single computer to other applications and processes. You can see this referenced as the “lo” interface in many tools.

Network Protocols

Networking works by piggybacks on a number of different protocols on top of each other. In this way, one piece of data can be transmitted using multiple protocols encapsulated within one another.

Media Access Control(MAC) is a communications protocol that is used to distinguish specific devices. Each device is supposed to get a unique MAC address during the manufacturing process that differentiates it from every other device on the internet. Addressing hardware by the MAC address allows you to reference a device by a unique value even when the software on top may change the name for that specific device during operation. Media access control is one of the only protocols from the link layer that you are likely to interact with on a regular basis.

The IP protocol is one of the fundamental protocols that allow the internet to work. IP addresses are unique on each network and they allow machines to address each other across a network. It is implemented on the internet layer in the IP/TCP model. Networks can be linked together, but traffic must be routed when crossing network boundaries. This protocol assumes an unreliable network and multiple paths to the same destination that it can dynamically change between. There are a number of different implementations of the protocol. The most common implementation today is IPv4, although IPv6 is growing in popularity as an alternative due to the scarcity of IPv4 addresses available and improvements in the protocols capabilities.

ICMP: internet control message protocol is used to send messages between devices to indicate the availability or error conditions. These packets are used in a variety of network diagnostic tools, such as ping and traceroute. Usually ICMP packets are transmitted when a packet of a different kind meets some kind of a problem. Basically, they are used as a feedback mechanism for network communications.

TCP: Transmission control protocol is implemented in the transport layer of the IP/TCP model and is used to establish reliable connections. TCP is one of the protocols that encapsulates data into packets. It then transfers these to the remote end of the connection using the methods available on the lower layers. On the other end, it can check for errors, request certain pieces to be resent, and reassemble the information into one logical piece to send to the application layer. The protocol builds up a connection prior to data transfer using a system called a three-way handshake. This is a way for the two ends of the communication to acknowledge the request and agree upon a method of ensuring data reliability. After the data has been sent, the connection is torn down using a similar four-way handshake. TCP is the protocol of choice for many of the most popular uses for the internet, including WWW, FTP, SSH, and email. It is safe to say that the internet we know today would not be here without TCP.

UDP: User datagram protocol is a popular companion protocol to TCP and is also implemented in the transport layer. The fundamental difference between UDP and TCP is that UDP offers unreliable data transfer. It does not verify that data has been received on the other end of the connection. This might sound like a bad thing, and for many purposes, it is. However, it is also extremely important for some functions. It’s not required to wait for confirmation that the data was received and forced to resend data, UDP is much faster than TCP. It does not establish a connection with the remote host, it simply fires off the data to that host and doesn’t care if it is accepted or not. Since UDP is a simple transaction, it is useful for simple communications like querying for network resources. It also doesn’t maintain a state, which makes it great for transmitting data from one machine to many real-time clients. This makes it ideal for VOIP, games, and other applications that cannot afford delays.

HTTP: Hypertext transfer protocol is a protocol defined in the application layer that forms the basis for communication on the web. HTTP defines a number of functions that tell the remote system what you are requesting. For instance, GET, POST, and DELETE all interact with the requested data in a different way.

FTP: File transfer protocol is in the application layer and provides a way of transferring complete files from one host to another. It is inherently insecure, so it is not recommended for any externally facing network unless it is implemented as a public, download-only resource.

DNS: Domain name system is an application layer protocol used to provide a human-friendly naming mechanism for internet resources. It is what ties a domain name to an IP address and allows you to access sites by name in your browser.

SSH: Secure shell is an encrypted protocol implemented in the application layer that can be used to communicate with a remote server in a secure way. Many additional technologies are built around this protocol because of its end-to-end encryption and ubiquity. There are many other protocols that we haven’t covered that are equally important. However, this should give you a good overview of some of the fundamental technologies that make the internet and networking possible.

REST(REpresentational State Transfer) is an architectural style for providing standards between computer systems on the web, making it easier for systems to communicate with each other.

JSON Web Token (JWT) is a compact URL-safe means of representing claims to be transferred between two parties. The claims in a JWT are encoded as a JSON object that is digitally signed using JSON Web Signature (JWS).

OAuth 2.0 is an open source authorization framework that enables applications to obtain limited access to user accounts on an HTTP service, such as Amazon, Google, Facebook, Microsoft, Twitter GitHub, and DigitalOcean. It works by delegating user authentication to the service that hosts the user account, and authorizing third-party applications to access the user account.

Docker

Back to the Top

Container Architecture. Source: Containerd.io

Docker Learning Resources

Docker Training Program

Docker Certified Associate (DCA) certification

Docker Documentation | Docker Documentation

The Docker Workshop

Docker Courses on Udemy

Docker Courses on Coursera

Docker Courses on edX

Docker Courses on Linkedin Learning

Docker Tools

Docker is an open platform for developing, shipping, and running applications. Docker enables you to separate your applications from your infrastructure so you can deliver software quickly working in collaboration with cloud, Linux, and Windows vendors, including Microsoft.

Docker Enterprise is a subscription including software, supported and certified container platform for CentOS, Red Hat Enterprise Linux (RHEL), Ubuntu, SUSE Linux Enterprise Server (SLES), Oracle Linux, and Windows Server 2016, as well as for cloud providers AWS and Azure. In November 2019 Docker’s Enterprise Platform business was acquired by Mirantis.

Docker Desktop is an application for MacOS and Windows machines for the building and sharing of containerized applications and microservices. Docker Desktop delivers the speed, choice and security you need for designing and delivering containerized applications on your desktop. Docker Desktop includes Docker App, developer tools, Kubernetes and version synchronization to production Docker Engines.

Docker Hub is the world’s largest library and community for container images Browse over 100,000 container images from software vendors, open-source projects, and the community.

Docker Compose is a tool that was developed to help define and share multi-container applications. With Docker Compose, you can create a YAML file to define the services and with a single command, can spin everything up or tear it all down.

Docker Swarm is a Docker-native clustering system swarm is a simple tool which controls a cluster of Docker hosts and exposes it as a single “virtual” host.

Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image. Using docker build users can create an automated build that executes several command-line instructions in succession.

Docker Containers is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another.

Docker Engine is a container runtime that runs on various Linux (CentOS, Debian, Fedora, Oracle Linux, RHEL, SUSE, and Ubuntu) and Windows Server operating systems. Docker creates simple tooling and a universal packaging approach that bundles up all application dependencies inside a container which is then run on Docker Engine.

Docker Images is a lightweight, standalone, executable package of software that includes everything needed to run an application: code, runtime, system tools, system libraries and settings. Images have intermediate layers that increase reusability, decrease disk usage, and speed up docker build by allowing each step to be cached. These intermediate layers are not shown by default. The SIZE is the cumulative space taken up by the image and all its parent images.

Docker Network is a that displays detailed information on one or more networks.

Docker Daemon is a service started by a system utility, not manually by a user. This makes it easier to automatically start Docker when the machine reboots. The command to start Docker depends on your operating system. Currently, it only runs on Linux because it depends on a number of Linux kernel features, but there are a few ways to run Docker on MacOS and Windows as well by configuring the operating system utilities.

Docker Storage is a driver controls how images and containers are stored and managed on your Docker host.

Kitematic is a simple application for managing Docker containers on Mac, Linux and Windows letting you control your app containers from a graphical user interface (GUI).

Open Container Initiative is an open governance structure for the express purpose of creating open industry standards around container formats and runtimes.

Buildah is a command line tool to build Open Container Initiative (OCI) images. It can be used with Docker, Podman, Kubernetes.

Kubernetes

Back to the Top

Kubernetes Learning Resources

Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications.

Getting Kubernetes Certifications

Getting started with Kubernetes on AWS

Kubernetes on Microsoft Azure

Intro to Azure Kubernetes Service

Azure Red Hat OpenShift

Getting started with Google Cloud

Getting started with Kubernetes on Red Hat

Getting started with Kubernetes on IBM

Red Hat OpenShift on IBM Cloud

Enable OpenShift Virtualization on Red Hat OpenShift

YAML basics in Kubernetes

Elastic Cloud on Kubernetes

Docker and Kubernetes

Running Apache Spark on Kubernetes

Kubernetes Across VMware vRealize Automation

VMware Tanzu Kubernetes Grid

All the Ways VMware Tanzu Works with AWS

VMware Tanzu Education

Using Ansible in a Cloud-Native Kubernetes Environment

Managing Kubernetes (K8s) objects with Ansible

Setting up a Kubernetes cluster using Vagrant and Ansible

Running MongoDB with Kubernetes

Kubernetes Fluentd

Understanding the new GitLab Kubernetes Agent

Intro Local Process with Kubernetes for Visual Studio 2019

Kubernetes Contributors

KubeAcademy from VMware

Kubernetes Tutorials from Pulumi

Kubernetes Playground by Katacoda

Scalable Microservices with Kubernetes course from Udacity

Kubernetes Tools, Frameworks, and Projects

Open Container Initiative is an open governance structure for the express purpose of creating open industry standards around container formats and runtimes.

Buildah is a command line tool to build Open Container Initiative (OCI) images. It can be used with Docker, Podman, Kubernetes.

Google Kubernetes Engine (GKE) is a managed, production-ready environment for running containerized applications.

Azure Kubernetes Service (AKS) is serverless Kubernetes, with a integrated continuous integration and continuous delivery (CI/CD) experience, and enterprise-grade security and governance. Unite your development and operations teams on a single platform to rapidly build, deliver, and scale applications with confidence.

Amazon EKS is a tool that runs Kubernetes control plane instances across multiple Availability Zones to ensure high availability.

AWS Controllers for Kubernetes (ACK) is a new tool that lets you directly manage AWS services from Kubernetes. ACK makes it simple to build scalable and highly-available Kubernetes applications that utilize AWS services.

Container Engine for Kubernetes (OKE) is an Oracle-managed container orchestration service that can reduce the time and cost to build modern cloud native applications. Unlike most other vendors, Oracle Cloud Infrastructure provides Container Engine for Kubernetes as a free service that runs on higher-performance, lower-cost compute.

Anthos is a modern application management platform that provides a consistent development and operations experience for cloud and on-premises environments.

Red Hat Openshift is a fully managed Kubernetes platform that provides a foundation for on-premises, hybrid, and multicloud deployments.

OKD is a community distribution of Kubernetes optimized for continuous application development and multi-tenant deployment. OKD adds developer and operations-centric tools on top of Kubernetes to enable rapid application development, easy deployment and scaling, and long-term lifecycle maintenance for small and large teams.

Odo is a fast, iterative, and straightforward CLI tool for developers who write, build, and deploy applications on Kubernetes and OpenShift.

Kata Operator is an operator to perform lifecycle management (install/upgrade/uninstall) of Kata Runtime on Openshift as well as Kubernetes cluster.

Thanos is a set of components that can be composed into a highly available metric system with unlimited storage capacity, which can be added seamlessly on top of existing Prometheus deployments.

OpenShift Hive is an operator which runs as a service on top of Kubernetes/OpenShift. The Hive service can be used to provision and perform initial configuration of OpenShift 4 clusters.

Rook is a tool that turns distributed storage systems into self-managing, self-scaling, self-healing storage services. It automates the tasks of a storage administrator: deployment, bootstrapping, configuration, provisioning, scaling, upgrading, migration, disaster recovery, monitoring, and resource management.

VMware Tanzu is a centralized management platform for consistently operating and securing your Kubernetes infrastructure and modern applications across multiple teams and private/public clouds.

Kubespray is a tool that combines Kubernetes and Ansible to easily install Kubernetes clusters that can be deployed on AWS, GCE, Azure, OpenStack, vSphere, Packet (bare metal), Oracle Cloud Infrastructure (Experimental), or Baremetal.

KubeInit provides Ansible playbooks and roles for the deployment and configuration of multiple Kubernetes distributions.

Rancher is a complete software stack for teams adopting containers. It addresses the operational and security challenges of managing multiple Kubernetes clusters, while providing DevOps teams with integrated tools for running containerized workloads.

K3s is a highly available, certified Kubernetes distribution designed for production workloads in unattended, resource-constrained, remote locations or inside IoT appliances.

Helm is a Kubernetes Package Manager tool that makes it easier to install and manage Kubernetes applications.

Knative is a Kubernetes-based platform to build, deploy, and manage modern serverless workloads. Knative takes care of the operational overhead details of networking, autoscaling (even to zero), and revision tracking.

KubeFlow is a tool dedicated to making deployments of machine learning (ML) workflows on Kubernetes simple, portable and scalable.

Etcd is a distributed key-value store that provides a reliable way to store data that needs to be accessed by a distributed system or cluster of machines. Etcd is used as the backend for service discovery and stores cluster state and configuration for Kubernetes.

OpenEBS is a Kubernetes-based tool to create stateful applications using Container Attached Storage.

Container Storage Interface (CSI) is an API that lets container orchestration platforms like Kubernetes seamlessly communicate with stored data via a plug-in.

MicroK8s is a tool that delivers the full Kubernetes experience. In a Fully containerized deployment with compressed over-the-air updates for ultra-reliable operations. It is supported on Linux, Windows, and MacOS.

Charmed Kubernetes is a well integrated, turn-key, conformant Kubernetes platform, optimized for your multi-cloud environments developed by Canonical.

Grafana Kubernetes App is a toll that allows you to monitor your Kubernetes cluster’s performance. It includes 4 dashboards, Cluster, Node, Pod/Container and Deployment. It allows for the automatic deployment of the required Prometheus exporters and a default scrape config to use with your in cluster Prometheus deployment.

KubeEdge is an open source system for extending native containerized application orchestration capabilities to hosts at Edge.It is built upon kubernetes and provides fundamental infrastructure support for network, app. deployment and metadata synchronization between cloud and edge.

Lens is the most powerful IDE for people who need to deal with Kubernetes clusters on a daily basis. It has support for MacOS, Windows and Linux operating systems.

Flux CD is a tool that automatically ensures that the state of your Kubernetes cluster matches the configuration you’ve supplied in Git. It uses an operator in the cluster to trigger deployments inside Kubernetes, which means that you don’t need a separate continuous delivery tool.

Platform9 Managed Kubernetes (PMK) is a Kubernetes as a service that ensures fully automated Day-2 operations with 99.9% SLA on any environment, whether in data-centers, public clouds, or at the edge.

Ansible

Back to the Top

Mac Development Ansible Playbook by Jeff Geerling

Ansible Learning Resources

Ansible is a simple IT automation engine that automates cloud provisioning, configuration management, application deployment, intra-service orchestration, and many other IT needs. It uses a very simple language (YAML, in the form of Ansible Playbooks) that allows you to describe your automation jobs in a way that approaches plain English. Anisble works on Linux (Red Hat EnterPrise Linux(RHEL) and Ubuntu) and Microsoft Windows.

Red Hat Training for Ansible

Top Ansible Courses Online from Udemy

Introduction to Ansible: The Fundamentals on Coursera

Learning Ansible Fundamentals on Pluralsight

Introducing Red Hat Ansible Automation Platform 2.1

Ansible Documentation

Ansible Galaxy User Guide

Ansible Use Cases

Ansible Integrations

Ansible Collections Overview

Working with playbooks

Ansible for DevOps Examples by Jeff Geerling

Getting Started: Writing Your First Playbook - Ansible

Working With Modules in Ansible

Ansible Best Practices: Roles & Modules

Working with command line tools for Ansible

Encrypting content with Ansible Vault

Using vault in playbooks with Ansible

Using Ansible With Azure

Configuring Ansible on an Azure VM

How to Use Ansible: An Ansible Cheat Sheet Guide from DigitalOcean

Intro to Ansible on Linode | Spatial Labs

Ansible DevOps Tools Integration

Ansible Automation Hub is the official location to discover and download supported collections, included as part of an Ansible Automation Platform subscription. These content collections contain modules, plugins, roles, and playbooks in a downloadable package.

Collections are a distribution format for Ansible content that can include playbooks, roles, modules, and plugins. As modules move from the core Ansible repository into collections, the module documentation will move to the collections pages.

Ansible Lint is a command-line tool for linting playbooks, roles and collections aimed towards any Ansible users. Its main goal is to promote proven practices, patterns and behaviors while avoiding common pitfalls that can easily lead to bugs or make code harder to maintain.

Ansible cmdb is a tool that takes the output of Ansible’s fact gathering and converts it into a static HTML overview page containing system configuration information.

Ansible Inventory Grapher visually displays inventory inheritance hierarchies and at what level a variable is defined in inventory.

Ansible Playbook Grapher is a command line tool to create a graph representing your Ansible playbook tasks and roles.

Ansible Shell is an interactive shell for Ansible with built-in tab completion for all the modules.

Ansible Silo is a self-contained Ansible environment by Docker.

Ansigenome is a command line tool designed to help you manage your Ansible roles.

ARA is a records Ansible playbook runs and makes the recorded data available and intuitive for users and systems by integrating with Ansible as a callback plugin.

Capistrano is a remote server automation tool. It supports the scripting and execution of arbitrary tasks, and includes a set of sane-default deployment workflows.

Fabric is a high level Python (2.7, 3.4+) library designed to execute shell commands remotely over SSH, yielding useful Python objects in return. It builds on top of Invoke (subprocess command execution and command-line features) and Paramiko (SSH protocol implementation), extending their APIs to complement one another and provide additional functionality.

ansible-role-wireguard is an Ansible role for installing WireGuard VPN. Supports Ubuntu, Debian, Archlinx, Fedora and CentOS Stream.

wireguard_cloud_gateway is an Ansible role for setting up Wireguard as a gateway VPN server for cloud networks.

Red Hat OpenShift is focused on security at every level of the container stack and throughout the application lifecycle. It includes long-term, enterprise support from one of the leading Kubernetes contributors and open source software companies.

OpenShift Hive is an operator which runs as a service on top of Kubernetes/OpenShift. The Hive service can be used to provision and perform initial configuration of OpenShift 4 clusters.

Databases

Back to the Top

SQL/NoSQL Learning Resources

SQL is a standard language for storing, manipulating and retrieving data in relational databases.

NoSQL is a database that is interchangeably referred to as “nonrelational, or “non-SQL” to highlight that the database can handle huge volumes of rapidly changing, unstructured data in different ways than a relational (SQL-based) database with rows and tables.

Transact-SQL(T-SQL) is a Microsoft extension of SQL with all of the tools and applications communicating to a SQL database by sending T-SQL commands.

Introduction to Transact-SQL

SQL Tutorial by W3Schools

Learn SQL Skills Online from Coursera

SQL Courses Online from Udemy

SQL Online Training Courses from LinkedIn Learning

Learn SQL For Free from Codecademy

GitLab’s SQL Style Guide

OracleDB SQL Style Guide Basics

Tableau CRM: BI Software and Tools

Databases on AWS

Best Practices and Recommendations for SQL Server Clustering in AWS EC2.

Connecting from Google Kubernetes Engine to a Cloud SQL instance.

Educational Microsoft Azure SQL resources

MySQL Certifications

SQL vs. NoSQL Databases: What’s the Difference?

What is NoSQL?

SQL/NoSQL Tools and Databases

Azure Data Studio is an open source data management tool that enables working with SQL Server, Azure SQL DB and SQL DW from Windows, macOS and Linux.

MySQL is a fully managed database service to deploy cloud-native applications using the world’s most popular open source database.

Amazon DynamoDB is a key-value and document database that delivers single-digit millisecond performance at any scale. It is a fully managed, multiregion, multimaster, durable database with built-in security, backup and restore, and in-memory caching for internet-scale applications.

Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. It can run Hadoop, Jenkins, Spark, Aurora, and other frameworks on a dynamically shared pool of nodes.

Apache Spark is a unified analytics engine for big data processing, with built-in modules for streaming, SQL, machine learning and graph processing.

Logstash is a tool for managing events and logs. When used generically, the term encompasses a larger system of log collection, processing, storage and searching activities.

Extract, transform, and load (ETL) is a data pipeline used to collect data from various sources, transform the data according to business rules, and load it into a destination data store.

MongoDB is a document database meaning it stores data in JSON-like documents.

OracleDB is a powerful fully managed database helps developers manage business-critical data with the highest availability, reliability, and security.

MariaDB is an enterprise open source database solution for modern, mission-critical applications.

Atlas is an in-memory dimensional time series database.

Cosmos DB Profiler is a real-time visual debugger allowing a development team to gain valuable insight and perspective into their usage of Cosmos DB database. It identifies over a dozen suspicious behaviors from your application’s interaction with Cosmos DB.

DBeaver is an open source database tool for developers and database administrators. It offers supports for JDBC compliant databases such as MySQL, Oracle, IBM DB2, SQL Server, Firebird, SQLite, Sybase, Teradata, Firebird, Apache Hive, Phoenix, and Presto.

DbVisualizer is a SQL management tool that allows users to manage a wide range of databases such as Oracle, Sybase, SQL Server, MySQL, H3, and SQLite.

Telco 5G

Back to the Top

VMware Cloud First Approach. Source: VMware.

VMware Telco Cloud Automation Components. Source: VMware.

Telco Learning Resources

HPE(Hewlett Packard Enterprise) Telco Blueprints overview

Network Functions Virtualization Infrastructure (NFVI) by Cisco

Introduction to vCloud NFV Telco Edge from VMware

VMware Telco Cloud Automation(TCA) Architecture Overview

5G Telco Cloud from VMware

Maturing OpenStack Together To Solve Telco Needs from Red Hat

Red Hat telco ecosystem program

OpenStack for Telcos by Canonical

Open source NFV platform for 5G from Ubuntu

Understanding 5G Technology from Verizon

Verizon and Unity partner to enable 5G & MEC gaming and enterprise applications

Understanding 5G Technology from Intel

Understanding 5G Technology from Qualcomm

Telco Acceleration with Xilinx

VIMs on OSM Public Wiki

Amazon EC2 Overview and Networking Introduction for Telecom Companies

Citrix Certified Associate – Networking(CCA-N)

Citrix Certified Professional – Virtualization(CCP-V)

CCNP Routing and Switching

Certified Information Security Manager(CISM)

Wireshark Certified Network Analyst (WCNA)

Juniper Networks Certification Program Enterprise (JNCP)

Cloud Native Computing Foundation Training and Certification Program

Tools

Open Stack is an open source cloud platform, deployed as infrastructure-as-a-service (IaaS) to orchestrate data center operations on bare metal, private cloud hardware, public cloud resources, or both (hybrid/multi-cloud architecture). OpenStack includes advance use of virtualization & SDN for network traffic optimization to handle the core cloud-computing services of compute, networking, storage, identity, and image services.

StarlingX is a complete cloud infrastructure software stack for the edge used by the most demanding applications in industrial IOT, telecom, video delivery and other ultra-low latency use cases.

Airship is a collection of open source tools for automating cloud provisioning and management. Airship provides a declarative framework for defining and managing the life cycle of open infrastructure tools and the underlying hardware.

Physical Network Function(PNF) is a physical network node which has not undergone virtualization. Both PNFs and VNFs (Virtualized Network Functions) can be used to form an overall Network Service.

Open Source Security

Back to the Top

Open Source Security Foundation (OpenSSF) is a cross-industry collaboration that brings together leaders to improve the security of open source software by building a broader community, targeted initiatives, and best practices. The OpenSSF brings together open source security initiatives under one foundation to accelerate work through cross-industry support. Along with the Core Infrastructure Initiative and the Open Source Security Coalition, and will include new working groups that address vulnerability disclosures, security tooling and more.

Security Standards, Frameworks and Benchmarks

STIGs Benchmarks - Security Technical Implementation Guides

CIS Benchmarks - CIS Center for Internet Security

NIST - Current FIPS

ISO Standards Catalogue

Common Criteria for Information Technology Security Evaluation (CC) is an international standard (ISO / IEC 15408) for computer security. It allows an objective evaluation to validate that a particular product satisfies a defined set of security requirements.

ISO 22301 is the international standard that provides a best-practice framework for implementing an optimised BCMS (business continuity management system).

ISO27001 is the international standard that describes the requirements for an ISMS (information security management system). The framework is designed to help organizations manage their security practices in one place, consistently and cost-effectively.

ISO 27701 specifies the requirements for a PIMS (privacy information management system) based on the requirements of ISO 27001. It is extended by a set of privacy-specific requirements, control objectives and controls. Companies that have implemented ISO 27001 will be able to use ISO 27701 to extend their security efforts to cover privacy management.

EU GDPR (General Data Protection Regulation) is a privacy and data protection law that supersedes existing national data protection laws across the EU, bringing uniformity by introducing just one main data protection law for companies/organizations to comply with.

CCPA (California Consumer Privacy Act) is a data privacy law that took effect on January 1, 2020 in the State of California. It applies to businesses that collect California residents’ personal information, and its privacy requirements are similar to those of the EU’s GDPR (General Data Protection Regulation).

Payment Card Industry (PCI) Data Security Standards (DSS) is a global information security standard designed to prevent fraud through increased control of credit card data.

SOC 2 is an auditing procedure that ensures your service providers securely manage your data to protect the interests of your comapny/organization and the privacy of their clients.

NIST CSF is a voluntary framework primarily intended for critical infrastructure organizations to manage and mitigate cybersecurity risk based on existing best practice.

Security Tools

SELinux is a security enhancement to Linux which allows users and administrators more control over access control. Access can be constrained on such variables as which users and applications can access which resources. These resources may take the form of files. Standard Linux access controls, such as file modes (-rwxr-xr-x) are modifiable by the user and the applications which the user runs. Conversely, SELinux access controls are determined by a policy loaded on the system which may not be changed by careless users or misbehaving applications.

AppArmor is an effective and easy-to-use Linux application security system. AppArmor proactively protects the operating system and applications from external or internal threats, even zero-day attacks, by enforcing good behavior and preventing both known and unknown application flaws from being exploited. AppArmor supplements the traditional Unix discretionary access control (DAC) model by providing mandatory access control (MAC). It has been included in the mainline Linux kernel since version 2.6.36 and its development has been supported by Canonical since 2009.

Control Groups(Cgroups) is a Linux kernel feature that allows you to allocate resources such as CPU time, system memory, network bandwidth, or any combination of these resources for user-defined groups of tasks (processes) running on a system.

EarlyOOM is a daemon for Linux that enables users to more quickly recover and regain control over their system in low-memory situations with heavy swap usage.

Libgcrypt is a general purpose cryptographic library originally based on code from GnuPG.

Kali Linux is an open source project that is maintained and funded by Offensive Security, a provider of world-class information security training and penetration testing services.

Aircrack-ng is a network software suite consisting of a detector, packet sniffer, WEP and WPA/WPA2-PSK cracker and analysis tool for 802.11 wireless LANs. It works with any wireless network interface controller whose driver supports raw monitoring mode and can sniff 802.11a, 802.11b and 802.11g traffic.

Burp Suite is a leading range of cybersecurity tools.

KernelCI is a community-based open source distributed test automation system focused on upstream kernel development. The primary goal of KernelCI is to use an open testing philosophy to ensure the quality, stability and long-term maintenance of the Linux kernel.

Continuous Kernel Integration project helps find bugs in kernel patches before they are commited to an upstram kernel tree. We are team of kernel developers, kernel testers, and automation engineers.

eBPF is a revolutionary technology that can run sandboxed programs in the Linux kernel without changing kernel source code or loading kernel modules. By making the Linux kernel programmable, infrastructure software can leverage existing layers, making them more intelligent and feature-rich without continuing to add additional layers of complexity to the system.

Cilium uses eBPF to accelerate getting data in and out of L7 proxies such as Envoy, enabling efficient visibility into API protocols like HTTP, gRPC, and Kafka.

Hubble is a Network, Service & Security Observability for Kubernetes using eBPF.

Istio is an open platform to connect, manage, and secure microservices. Istio’s control plane provides an abstraction layer over the underlying cluster management platform, such as Kubernetes and Mesos.

Certgen is a convenience tool to generate and store certificates for Hubble Relay mTLS.

Scapy is a python-based interactive packet manipulation program & library.

syzkaller is an unsupervised, coverage-guided kernel fuzzer.

SchedViz is a tool for gathering and visualizing kernel scheduling traces on Linux machines.

oss-fuzz aims to make common open source software more secure and stable by combining modern fuzzing techniques with scalable, distributed execution.

OSSEC is a free, open-source host-based intrusion detection system. It performs log analysis, integrity checking, Windows registry monitoring, rootkit detection, time-based alerting, and active response.

Metasploit Project is a computer security project that provides information about security vulnerabilities and aids in penetration testing and IDS signature development.

Wfuzz was created to facilitate the task in web applications assessments and it is based on a simple concept: it replaces any reference to the FUZZ keyword by the value of a given payload.

Nmap is a security scanner used to discover hosts and services on a computer network, thus building a “map” of the network.

Patchwork is a web-based patch tracking system designed to facilitate the contribution and management of contributions to an open-source project.

pfSense is a free and open source firewall and router that also features unified threat management, load balancing, multi WAN, and more.

Snowpatch is a continuous integration tool for projects using a patch-based, mailing-list-centric git workflow. This workflow is used by a number of well-known open source projects such as the Linux kernel.

Snort is an open-source, free and lightweight network intrusion detection system (NIDS) software for Linux and Windows to detect emerging threats.

Wireshark is a free and open-source packet analyzer. It is used for network troubleshooting, analysis, software and communications protocol development, and education.

OpenSCAP is U.S. standard maintained by National Institute of Standards and Technology (NIST). It provides multiple tools to assist administrators and auditors with assessment, measurement, and enforcement of security baselines. OpenSCAP maintains great flexibility and interoperability by reducing the costs of performing security audits. Whether you want to evaluate DISA STIGs, NIST‘s USGCB, or Red Hat’s Security Response Team’s content, all are supported by OpenSCAP.

Tink is a multi-language, cross-platform, open source library that provides cryptographic APIs that are secure, easy to use correctly, and harder to misuse.

OWASP is an online community, produces freely-available articles, methodologies, documentation, tools, and technologies in the field of web application security.

Open Vulnerability and Assessment Language is a community effort to standardize how to assess and report upon the machine state of computer systems. OVAL includes a language to encode system details, and community repositories of content. Tools and services that use OVAL provide enterprises with accurate, consistent, and actionable information to improve their security.

ClamAV is an open source antivirus engine for detecting trojans, viruses, malware & other malicious threats.

Open Source Security Learning Resources

Microsoft Open Source Software Security

Cloudflare Open Source Security

The Seven Properties of Highly Secure Devices

How Layer 7 of the Internet Works

The 7 Kinds of Security

The Libgcrypt Reference Manual

The Open Web Application Security Project(OWASP) Foundation Top 10

Best Practices for Using Open Source Code from The Linux Foundation

AWS Certified Security - Specialty Certification

Microsoft Certified: Azure Security Engineer Associate

Google Cloud Certified Professional Cloud Security Engineer

Cisco Security Certifications

The Red Hat Certified Specialist in Security: Linux

Linux Professional Institute LPIC-3 Enterprise Security Certification

Cybersecurity Training and Courses from IBM Skills

Cybersecurity Courses and Certifications by Offensive Security

RSA Certification Program

Check Point Certified Security Expert(CCSE) Certification

Check Point Certified Security Administrator(CCSA) Certification

Check Point Certified Security Master (CCSM) Certification

Certified Cloud Security Professional(CCSP) Certification

Certified Information Systems Security Professional (CISSP) Certification

CCNP Routing and Switching

Certified Information Security Manager(CISM)

Wireshark Certified Network Analyst (WCNA)

Juniper Networks Certification Program Enterprise (JNCP)

Security Training Certifications and Courses from Udemy

Security Training Certifications and Courses from Coursera

Security Certifications Training from Pluarlsight

Differential Privacy

Back to the Top

Above is a simple diagram of how Differential Privacy-Preserving Data Sharing and Data Mining protects a User's Data

Differential Privacy Learning Resources

Differential Privacy is a system that simultaneously enables researchers and analysts to extract useful insights from datasets containing personal information and offers stronger privacy protections. This is achieved by introducing “statistical noise”.

Statistical Noise is a process that small aletrations to masked datasets. The statistical noise hides identifiable characteristics of individuals, ensuring that the privacy of personal information is protected, but it’s small enough to not materially impact the accuracy of the answers extracted by analysts and researchers.

Laplacian Noise is a mechanism that adds Laplacian-distributed noise to a function.

Differential Privacy Blog Series by the National Institute of Standards and Technology(NIST)

Apple’s Differential Privacy Overview

Learning with Privacy at Scale with Apple Machine Learning

Microsoft Research Differential Privacy Overview

Responsible Machine Learning with Microsoft Azure

Responsible AI Resources with Microsoft AI

Preserve data privacy by using differential privacy and the SmartNoise package

Open Differential Privacy(OpenDP) Initiative by Microsoft and Harvard

Google’s Differential Privacy Library

Computing Private Statistics with Privacy on Beam from Google Codelabs

Introducing TensorFlow Privacy: Learning with Differential Privacy for Training Data

TensorFlow Federated: Machine Learning on Decentralized Data

Federated Analytics: Collaborative Data Science without Data Collection

Differentially-Private Stochastic Gradient Descent(DP-SGD)

Learning Differential Privacy from Harvard University Privacy Tools Project

Harvard University Privacy Tools Project Courses & Educational Materials

The Weaknesses of Differential Privacy course on Coursera

The Differential Privacy of Bayesian Inference

Simultaneous private learning of multiple concepts

The Complexity of Computing the Optimal Composition of Differential Privacy

Order revealing encryption and the hardness of private learning

SAP HANA data anonymization using SAP Software Solutions

SAP HANA Security using their In-Memory Database

DEFCON Differential Privacy Training Launch

Secure and Private AI course on Udacity

Differential Privacy - Security and Privacy for Big Data - Part 1 course on Coursera

Differential Privacy - Security and Privacy for Big Data - Part 2 course on Coursera

Certified Ethical Emerging Technologist Professional Certificate course on Coursera

Differential Privacy Tools

PySyft is a Python library for secure and private Deep Learning. PySyft decouples private data from model training, using Federated Learning, Differential Privacy, and Encrypted Computation (like Multi-Party Computation (MPC) and Homomorphic Encryption (HE) within the main Deep Learning frameworks like PyTorch and TensorFlow.

TensorFlow Privacy is a Python library that includes implementations of TensorFlow optimizers for training machine learning models with differential privacy. The library comes with tutorials and analysis tools for computing the privacy guarantees provided.

TensorFlow Federated (TFF) is an open-source framework for machine learning and other computations on decentralized data. TFF has been developed to facilitate open research and experimentation with Federated Learning (FL), an approach to machine learning where a shared global model is trained across many participating clients that keep their training data locally.

Privacy on Beam is an end-to-end differential privacy solution built on Apache Beam. It is intended to be usable by all developers, regardless of their differential privacy expertise.

PyDP is a Python wrapper for Google’s Differential Privacy project.

PennyLane is a cross-platform Python library for differentiable programming of quantum computers. By training a quantum computer the same way as a neural network.

BoTorch is a library for Bayesian Optimization built on PyTorch.

PyTorch Geometric (PyG) is a geometric deep learning extension library for PyTorch.

Skorch is a scikit-learn compatible neural network library that wraps PyTorch.

Diffprivlib is the IBM Differential Privacy Library for experimenting with, investigating and developing applications in, differential privacy.

Opacus is a library that enables training PyTorch models with differential privacy. It supports training with minimal code changes required on the client, has little impact on training performance and allows the client to online track the privacy budget expended at any given moment.

Smart Noise is a toolkit that uses state-of-the-art differential privacy (DP) techniques to inject noise into data, to prevent disclosure of sensitive information and manage exposure risk.

Machine Learning

Back to the Top

ML frameworks & applications

TensorFlow is an end-to-end open source platform for machine learning. It has a comprehensive, flexible ecosystem of tools, libraries and community resources that lets researchers push the state-of-the-art in ML and developers easily build and deploy ML powered applications.

Tensorman is a utility for easy management of Tensorflow containers by developed by System76.Tensorman allows Tensorflow to operate in an isolated environment that is contained from the rest of the system. This virtual environment can operate independent of the base system, allowing you to use any version of Tensorflow on any version of a Linux distribution that supports the Docker runtime.

Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano.It was developed with a focus on enabling fast experimentation. It is capable of running on top of TensorFlow, Microsoft Cognitive Toolkit, R, Theano, or PlaidML.

PyTorch is a library for deep learning on irregular input data such as graphs, point clouds, and manifolds. Primarily developed by Facebook’s AI Research lab.

Amazon SageMaker is a fully managed service that provides every developer and data scientist with the ability to build, train, and deploy machine learning (ML) models quickly. SageMaker removes the heavy lifting from each step of the machine learning process to make it easier to develop high quality models.

Azure Databricks is a fast and collaborative Apache Spark-based big data analytics service designed for data science and data engineering. Azure Databricks, sets up your Apache Spark environment in minutes, autoscale, and collaborate on shared projects in an interactive workspace. Azure Databricks supports Python, Scala, R, Java, and SQL, as well as data science frameworks and libraries including TensorFlow, PyTorch, and scikit-learn.

Microsoft Cognitive Toolkit (CNTK) is an open-source toolkit for commercial-grade distributed deep learning. It describes neural networks as a series of computational steps via a directed graph. CNTK allows the user to easily realize and combine popular model types such as feed-forward DNNs, convolutional neural networks (CNNs) and recurrent neural networks (RNNs/LSTMs). CNTK implements stochastic gradient descent (SGD, error backpropagation) learning with automatic differentiation and parallelization across multiple GPUs and servers.

Apache Airflow is an open-source workflow management platform created by the community to programmatically author, schedule and monitor workflows. Install. Principles. Scalable. Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. Airflow is ready to scale to infinity.

Open Neural Network Exchange(ONNX) is an open ecosystem that empowers AI developers to choose the right tools as their project evolves. ONNX provides an open source format for AI models, both deep learning and traditional ML. It defines an extensible computation graph model, as well as definitions of built-in operators and standard data types.

Apache MXNet is a deep learning framework designed for both efficiency and flexibility. It allows you to mix symbolic and imperative programming to maximize efficiency and productivity. At its core, MXNet contains a dynamic dependency scheduler that automatically parallelizes both symbolic and imperative operations on the fly. A graph optimization layer on top of that makes symbolic execution fast and memory efficient. MXNet is portable and lightweight, scaling effectively to multiple GPUs and multiple machines. Support for Python, R, Julia, Scala, Go, Javascript and more.

AutoGluon is toolkit for Deep learning that automates machine learning tasks enabling you to easily achieve strong predictive performance in your applications. With just a few lines of code, you can train and deploy high-accuracy deep learning models on tabular, image, and text data.

Anaconda is a very popular Data Science platform for machine learning and deep learning that enables users to develop models, train them, and deploy them.

PlaidML is an advanced and portable tensor compiler for enabling deep learning on laptops, embedded devices, or other devices where the available computing hardware is not well supported or the available software stack contains unpalatable license restrictions.

OpenCV is a highly optimized library with focus on real-time computer vision applications. The C++, Python, and Java interfaces support Linux, MacOS, Windows, iOS, and Android.

Scikit-Learn is a Python module for machine learning built on top of SciPy, NumPy, and matplotlib, making it easier to apply robust and simple implementations of many popular machine learning algorithms.

Weka is an open source machine learning software that can be accessed through a graphical user interface, standard terminal applications, or a Java API. It is widely used for teaching, research, and industrial applications, contains a plethora of built-in tools for standard machine learning tasks, and additionally gives transparent access to well-known toolboxes such as scikit-learn, R, and Deeplearning4j.

Caffe is a deep learning framework made with expression, speed, and modularity in mind. It is developed by Berkeley AI Research (BAIR)/The Berkeley Vision and Learning Center (BVLC) and community contributors.

Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently including tight integration with NumPy.

nGraph is an open source C++ library, compiler and runtime for Deep Learning. The nGraph Compiler aims to accelerate developing AI workloads using any deep learning framework and deploying to a variety of hardware targets.It provides the freedom, performance, and ease-of-use to AI developers.

NVIDIA cuDNN is a GPU-accelerated library of primitives for deep neural networks. cuDNN provides highly tuned implementations for standard routines such as forward and backward convolution, pooling, normalization, and activation layers. cuDNN accelerates widely used deep learning frameworks, including Caffe2, Chainer, Keras, MATLAB, MxNet, PyTorch, and TensorFlow.

Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Jupyter is used widely in industries that do data cleaning and transformation, numerical simulation, statistical modeling, data visualization, data science, and machine learning.

Apache Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Scala, Java, Python, and R, and an optimized engine that supports general computation graphs for data analysis. It also supports a rich set of higher-level tools including Spark SQL for SQL and DataFrames, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for stream processing.

Apache Spark Connector for SQL Server and Azure SQL is a high-performance connector that enables you to use transactional data in big data analytics and persists results for ad-hoc queries or reporting. The connector allows you to use any SQL database, on-premises or in the cloud, as an input data source or output data sink for Spark jobs.

Apache PredictionIO is an open source machine learning framework for developers, data scientists, and end users. It supports event collection, deployment of algorithms, evaluation, querying predictive results via REST APIs. It is based on scalable open source services like Hadoop, HBase (and other DBs), Elasticsearch, Spark and implements what is called a Lambda Architecture.

Cluster Manager for Apache Kafka(CMAK) is a tool for managing Apache Kafka clusters.

BigDL is a distributed deep learning library for Apache Spark. With BigDL, users can write their deep learning applications as standard Spark programs, which can directly run on top of existing Spark or Hadoop clusters.

Koalas is project makes data scientists more productive when interacting with big data, by implementing the pandas DataFrame API on top of Apache Spark.

Apache Spark™ MLflow is an open source platform to manage the ML lifecycle, including experimentation, reproducibility, deployment, and a central model registry. MLflow currently offers four components:

MLflow Tracking: Record and query experiments: code, data, config, and results.

MLflow Projects: Package data science code in a format to reproduce runs on any platform.

MLflow Models: Deploy machine learning models in diverse serving environments.

Model Registry: Store, annotate, discover, and manage models in a central repository.

Eclipse Deeplearning4J (DL4J) is a set of projects intended to support all the needs of a JVM-based(Scala, Kotlin, Clojure, and Groovy) deep learning application. This means starting with the raw data, loading and preprocessing it from wherever and whatever format it is in to building and tuning a wide variety of simple and complex deep learning networks.

Numba is an open source, NumPy-aware optimizing compiler for Python sponsored by Anaconda, Inc. It uses the LLVM compiler project to generate machine code from Python syntax. Numba can compile a large subset of numerically-focused Python, including many NumPy functions. Additionally, Numba has support for automatic parallelization of loops, generation of GPU-accelerated code, and creation of ufuncs and C callbacks.

Chainer is a Python-based deep learning framework aiming at flexibility. It provides automatic differentiation APIs based on the define-by-run approach (dynamic computational graphs) as well as object-oriented high-level APIs to build and train neural networks. It also supports CUDA/cuDNN using CuPy for high performance training and inference.

cuML is a suite of libraries that implement machine learning algorithms and mathematical primitives functions that share compatible APIs with other RAPIDS projects. cuML enables data scientists, researchers, and software engineers to run traditional tabular ML tasks on GPUs without going into the details of CUDA programming. In most cases, cuML’s Python API matches the API from scikit-learn.

Online ML Learning Resources

Machine Learning by Stanford University from Coursera

Machine Learning Courses Online from Coursera

Machine Learning Courses Online from Udemy

Learn Machine Learning with Online Courses and Classes from edX

IoT Protocols

Back to the Top

DBus is an open source software bus developed Red Hat for inter-process communication, and remote procedure call mechanism that allows communication between multiple processes running concurrently on the same machine.

SOAP is a messaging protocol specification for exchanging structured information in the implementation of web services in computer networks. SOAP can extend HTTP for XML messaging. SOAP provides data transport for Web services. SOAP can exchange complete documents or call a remote procedure. SOAP can be used for broadcasting a message.

gRPC is a modern, open source remote procedure call (RPC) framework developed by Google that can run anywhere. It enables client and server applications to communicate transparently, and makes it easier to build connected systems.It uses HTTP/2 for transport, Protocol Buffers as the interface description language, and provides features such as authentication, bidirectional streaming and flow control, blocking or nonblocking bindings, and cancellation and timeouts.

LWM2M is a protocol from the Open Mobile Alliance for M2M or IoT device management. Lightweight M2M enabler defines the application layer communication protocol between a LWM2M Server and a LWM2M Client, which is located in a LWM2M Device.

Advanced Message Queuing Protocol (AMQP) is an open standard for passing business messages between applications or organizations. It connects systems, feeds business processes with the information they need and reliably transmits onward the instructions that achieve their goals. The defining features of AMQP are message orientation, queuing, routing, reliability and security.

Constrained Application Protocol (CoAP) is a specialized web transfer protocol for use with constrained nodes and constrained networks in the Internet of Things. The protocol is designed for machine-to-machine (M2M) applications such as smart energy and building automation.”

Extensible Messaging and Presence Protocol (XMPP) is a communication protocol for message-oriented middleware based on XML (Extensible Markup Language). It enables the near real-time exchange of structured yet extensible data between any two or more network entities.

OASIS Message Queuing Telemetry Transport (MQTT) is an open OASIS and ISO standard (ISO/IEC 20922) lightweight, publish-subscribe network protocol that transports messages between devices. The protocol usually runs over TCP/IP; however, any network protocol that provides ordered, lossless, bi-directional connections can support MQTT.

Very Simple Control Protocol (VSCP) is a free automation protocol suitable for all sorts of automation task where building- or home-automation is in the main focus. Its main advantage is that each VSCP-node can work completely autonomous, being part of distributed network of other nodes.

Operating systems

Back to the Top

Raspberry Pi OS

Hass.io(Home Assistant OS)

Manjaro Linux ARM

Arch Linux ARM

Ubuntu MATE for Raspberry Pi

Ubuntu Desktop for Raspberry Pi

Ubuntu Core on a Raspberry Pi

Ubuntu Server for ARM

Kali Linux for the Raspberry Pi

HeliOS is an embedded operating system that is free for anyone to use. While called an operating system for simplicity, HeliOS is better described as a multitasking kernel for embedded systems.

Simba is a small OS for an Embedded Programming Platform like Arduino. It aims to make embedded programming easy and portable.

Trampoline is a static RTOS for small embedded systems.

DuinOS is Framework (a wrapper) for use the FreeRTOSwith Arduino.

VxWorks is an industry-leading real-time operating systems (RTOS) for building embedded devices and systems for more than 30 years.

LynxOS is a native POSIX, hard real-time partitioning operating system developed by Lynx Software Technologies.

Zephyr OS is a popular security-oriented RTOS with a small-footprint kernel designed for use on resource-constrained and embedded systems. Zephyr has a small-foorprint Kernel focusing on embedded devices compatible with x86, ARM, RISC-V, Xtensa and others.

FreeRTOS is an open source, real-time operating system for microcontrollers that makes small, low-power edge devices easy to program, deploy, secure, connect, and manage.

Arm Mbed TLS provides a comprehensive SSL/TLS solution and makes it easy for developers to include cryptographic and SSL/TLS capabilities in their software and embedded products. As an SSL library, it provides an intuitive API, readable source code and a minimal and highly configurable code footprint.

Contiki-os is an operating system for networked, memory-constrained systems with a focus on low-power wireless Internet of Things devices.

Middleware

Back to the Top

IoTSyS is an integration middleware for the Internet of Things. It provides a communication stack for embedded devices based on IPv6, Web services, and OBIX to establish interoperable interfaces for smart objects.

OpenIoT is an open source middleware infrastructure will support flexible configuration and deployment of algorithms for collection, and filtering information streams stemming from the internet-connected objects, while at the same time generating and processing important business/applications events.

OpenRemote is an open source middleware project, which integrates many different protocols and solutions available for smart building, and smart city automation, and offers visualization tools.

Kaa is a Enterprise IoT Platform has been designed with heavy-duty, enterprise-grade IoT solutions in mind. It banishes a monolithic approach to architecture in favour of highly portable microservices, which allow for flexible rearrangement and customization even in the middle of the solution’s lifecycle.

Node flow editors

Back to the Top

Node-RED is a programming tool for wiring together hardware devices, APIs and online services in new and interesting ways. It provides a browser-based editor that makes it easy to wire together flows using the wide range of nodes in the palette that can be deployed to its runtime in a single-click.

Toolkits

Back to the Top

KinomaJS is a visual code editor designed to help developers build starter projects for Kinoma Create and Kinoma Element. The project is built on Angular 2(RC7) and runs in a web browser. The live version is hosted using Google App Engine, but you can modify and build it yourself by following the instructions in this document.

IoT Toolkit is a collection of libraries that enables communication with modern IoT based environments and devices. It is a high-performance collection of libraries optimized for minimum memory consumption in RAM, ROM, high speed, and versatility working on any device.

Data Visualization

Back to the Top

Freeboard is an open source real-time dashboard builder for IOT and other web mashups. A free open-source alternative to Geckoboard.

ThingSpeak is an IoT analytics platform service that allows you to aggregate, visualize, and analyze live data streams in the cloud. You can send data to ThingSpeak from your devices, create instant visualization of live data, and send alerts.

Search

Back to the Top

Thingful is a Search Engine for the Internet of Things Find & use open IoT data from around the world.

Hardware

Back to the Top

Arduino Ethernet Shield 2 allows an Arduino board to connect to the internet using the Ethernet library and to read and write an SD card using the SD library.This shield is fully compatible with the former version, but relies on the newer W5500 chip.

Raspberry Pi is a series of small single-board computers developed in the United Kingdom by the Raspberry Pi Foundation to promote teaching of basic computer science in schools and in developing countries. Price range from $10-45 depending on model.

BeagleBone is a low-power open-source single-board computer produced by Texas Instruments. It runs Android, Ubuntu and other Linux flavors.

openPicus FlyportPro is a system on a module dedicated to IoT and M2M application, especially for professional use. Following some details on the solution: Why FlyportPRO SoM? A system-on-module is the best solution for those customers looking for flexibility and for development time and risk reduction.

Pinoccio is a solution to add mesh networking capability and WiFi-Internet access to all yout IoT devices, and it is Arduino compatible. Each board can assume the role of Scout in a Troop and one of the Scouts is the Lead to connect internet: Field Scouts talk to each other using a mesh network (called a Troop), using an extremely low-power radio.

In-memory data grids

Back to the Top

Ehcache is an open source, standards-based cache that boosts performance, offloads your database, and simplifies scalability. It’s the most widely-used Java-based cache because it’s robust, proven, full-featured, and integrates with other popular libraries and frameworks.

Hazelcast is an open source in-memory data grid based on Java.

Home automation

Back to the Top

Home Assistant is open source home automation that puts local control and privacy first. Powered by a worldwide community of tinkerers and DIY enthusiasts. Perfect to run on a Raspberry Pi or a local server.

openHAB is a cross-platform software with the aim to integrate all kinds of Smart Home technologies, devices, etc.

Eclipse SmartHome is a framework, not a ready-to-use solution. It offers a large set of features to choose from and leaves enough possibilities to design a Smart Home solution specific to your expectations. Its modular design brings millions of combinations and proves to be easily extensible by custom parts.

The Thing System is a set of software components and network protocols that aims to fix the Internet of Things. Our steward software is written in node.js making it both portable and easily extensible. It can run on your laptop, or fit onto a small single board computer like the Raspberry Pi.

Robotics

Back to the Top

Tools for Robotics

Open Source Robotics Foundation works with industry, academia, and government to create and support open software and hardware for use in robotics, from research and education to product development.

ROS is robotics middleware. Although ROS is not an operating system, it provides services designed for a heterogeneous computer cluster such as hardware abstraction, low-level device control, implementation of commonly used functionality, message-passing between processes, and package management.

ROS2 is a set of software libraries and tools that help you build robot applications. From drivers to state-of-the-art algorithms, and with powerful developer tools, ROS has what you need for your next robotics project. And it’s all open source.

Robot Framework is a generic open source automation framework. It can be used for test automation and robotic process automation. It has easy syntax, utilizing human-readable keywords. Its capabilities can be extended by libraries implemented with Python or Java.

The Robotics Library (RL) is a self-contained C++ library for robot kinematics, motion planning and control. It covers mathematics, kinematics and dynamics, hardware abstraction, motion planning, collision detection, and visualization.RL runs on many different systems, including Linux, macOS, and Windows. It uses CMake as a build system and can be compiled with Clang, GCC, and Visual Studio.

MoveIt is the most widely used software for manipulation and has been used on over 100 robots. It provides an easy-to-use robotics platform for developing advanced applications, evaluating new designs and building integrated products for industrial, commercial, R&D, and other domains.

Gazebo accurately and efficiently simulates indoor and outdoor robots. You get a robust physics engine, high-quality graphics, and programmatic and graphical interfaces.

Robotics System Toolbox provides tools and algorithms for designing, simulating, and testing manipulators, mobile robots, and humanoid robots. For manipulators and humanoid robots, the toolbox includes algorithms for collision checking, trajectory generation, forward and inverse kinematics, and dynamics using a rigid body tree representation. For mobile robots, it includes algorithms for mapping, localization, path planning, path following, and motion control. The toolbox provides reference examples of common industrial robot applications. It also includes a library of commercially available industrial robot models that you can import, visualize, and simulate.

Intel Robot DevKit is the tool to generate Robotics Software Development Kit (RDK) designed for autonomous devices, including the ROS2 core and capacibilities packages like perception, planning, control driver etc. It provides flexible build/runtime configurations to meet different autonomous requirement on top of diversity hardware choices, for example use different hareware engine CPU/GPU/VPU to accelerate AI related features.

Arduino is an open-source platform used for building electronics projects. Arduino consists of both a physical programmable circuit board (often referred to as a microcontroller) and a piece of software, or IDE (Integrated Development Environment) that runs on your computer, used to write and upload computer code to the physical board.

ArduPilot enables the creation and use of trusted, autonomous, unmanned vehicle systems for the peaceful benefit of all. ArduPilot provides a comprehensive suite of tools suitable for almost any vehicle and application.

AirSim is a simulator for drones, cars and more, built on Unreal Engine (we now also have an experimental Unity release). It is open-source, cross platform, and supports hardware-in-loop with popular flight controllers such as PX4 for physically and visually realistic simulations.

F´ (F Prime) is a component-driven framework that enables rapid development and deployment of spaceflight and other embedded software applications. Originally developed at the Jet Propulsion Laboratory, F´ has been successfully deployed on several space applications.

The JPL Open Source Rover is an open source, build it yourself, scaled down version of the 6 wheel rover design that JPL uses to explore the surface of Mars. The Open Source Rover is designed almost entirely out of consumer off the shelf (COTS) parts. This project is intended to be a teaching and learning experience for those who want to get involved in mechanical engineering, software, electronics, or robotics.

Light Detection and Ranging(LiDAR) is a remote sensing method that uses light in the form of a pulsed laser at an object, and uses the time and wavelength of the reflected beam of light to estimate the distance and in some applications (Laser Imaging), to create a 3D representation of the object and its surface characteristics. This technology is commonly used in aircraft and self-driving vehicles.

Neurorobotics Platform (NRP) is an Internet-accessible simulation system that allows the simulation of robots controlled by spiking neural networks.

ViSP is an open-source visual servoing platform library, is able to compute control laws that can be applied to robotic systems.

ROS Behavior Trees is an open-source library to create robot’s behaviors in form of Behavior Trees running in ROS (Robot Operating System).

g2core is an open-source motion control software for CNC and Robotics, designed to run on Arduino Due class microcontrollers.

ur5controller is an open-source OpenRAVE controller for UR5 robot integrated with ROS.

RBDL is an open-source (zlib) C++ libray for both forward and inverse dynamics and kinematics. Also supports contacts and loops.

Unity Robotics Hub is a Central repository for open-source Unity packages, tutorials, and other resources demonstrating how to use Unity for robotics simulations. Includes new support for ROS integration.

AliceVision is a Photogrammetric Computer Vision Framework which provides a 3D Reconstruction and Camera Tracking algorithms. AliceVision aims to provide strong software basis with state-of-the-art computer vision algorithms that can be tested, analyzed and reused. The project is a result of collaboration between academia and industry to provide cutting-edge algorithms with the robustness and the quality required for production usage.

CARLA is an open-source simulator for autonomous driving research. CARLA has been developed from the ground up to support development, training, and validation of autonomous driving systems. In addition to open-source code and protocols, CARLA provides open digital assets (urban layouts, buildings, vehicles) that were created for this purpose and can be used freely. The simulation platform supports flexible specification of sensor suites and environmental conditions.

ROS bridge is a package to bridge ROS for CARLA Simulator.

ROS-Industrial is an open source project that extends the advanced capabilities of ROS software to manufacturing.

AWS RoboMaker is the most complete cloud solution for robotic developers to simulate, test and securely deploy robotic applications at scale. RoboMaker provides a fully-managed, scalable infrastructure for simulation that customers use for multi-robot simulation and CI/CD integration with regression testing in simulation.

Microsoft Robotics Developer Studio is a free .NET-based programming environment for building robotics applications.

Visual Studio Code Extension for ROS is an extension provides support for Robot Operating System (ROS) development.

Azure Kinect ROS Driver is a node which publishes sensor data from the Azure Kinect Developer Kit to the Robot Operating System (ROS). Developers working with ROS can use this node to connect an Azure Kinect Developer Kit to an existing ROS installation.

Azure IoT Hub for ROS is a ROS package works with the Microsoft Azure IoT Hub service to relay telemetry messages from the Robot to Azure IoT Hub or reflect properties from the Digital Twin to the robot using dynamic reconfigure.

ROS 2 with ONNX Runtime is a program that uses ROS 2 to run on different hardware platforms using their respective AI acceleration libraries for optimized execution of the ONNX model.

Azure Cognitive Services LUIS ROS Node is a ROS node that bridges between ROS and the Azure Language Understanding Service. it can be configured to process audio directly from a microphone, or can subscribe to a ROS audio topic, then processes speech and generates “intent” ROS messages which can be processed by another ROS node to generate ROS commands.

Robotics Learning Resources

Robotics courses from Coursera

Learn Robotics with Online Courses and Classes from edX

Top Robotics Courses Online from Udemy

Free Online AI & Robotics Courses

REC Foundation Robotics Industry Certification

Carnegie Mellon Robotics Academy

RIA Robotic Integrator Certification Program

AWS RoboMaker – Develop, Test, Deploy, and Manage Intelligent Robotics Apps

Microsoft AI School

Language Understanding (LUIS) for Azure Cognitive Services

Azure VM templates to bootstrap ROS and ROS 2 environments

Google Robotics Research

Mesh networks

Back to the Top

Open Garden develops the FireChat mobile application, which enables peer-to-peer mobile Internet connection sharing with faster and more efficient data transmissions by automatically and actively choosing and switching to the best available network without requiring users to manually sift through available networks to find the best one available.

OpenWSN is a project created at the University of California Berkeley and extended at the INRIA and at the Open University of Catalonia which aims to build an open standard-based and open source implementation of a complete constrained network protocol stack for wireless sensor networks and Internet of Things.

Blockchain Development

Back to the Top

Blockchain Learning Resources

Blockchain is a distributed database meaning that the storage devices used for the database are not all connected to a common processor. It maintains a growing decentralized digital list of ordered records, called blocks. Each block has a timestamp and a link to a previous block. By the system being decentralization it allows for complete transparency in all shared information.

IBM Blockchain Essentials

Getting started with Blockchain Development

Blockchain Developer Nanodegree program | Udacity

Blockchain Programming: Build a blockchain and cryptocurrency from scratch using Python | Udemy

Blockchain Courses by Coursera

Blockchain Courses by edX

Blockchain Certifications

Blockchain Tools, Libraries and Frameworks

Lightning Network (LN) is a decentralized network using smart contract functionality in the blockchain to enable instant payments across a network of participants. It was created in response to scalability issues with Bitcoin, namely the speed and cost of Bitcoin transactions. Bitcoin’s current theoretical maximum transactions per second (TPS) is 10, though in reality it is between 3 and 7.

Hyperledger is an open source community focused on developing a suite of stable frameworks, tools and libraries for enterprise-grade blockchain deployments. It serves as a neutral home for various distributed ledger frameworks including Hyperledger Fabric, Sawtooth, Indy, as well as tools like Hyperledger Caliper and libraries like Hyperledger Ursa.

Hyperledger Caliper is a blockchain benchmark tool, it allows users to measure the performance of a blockchain implementation with a set of predefined use cases. Hyperledger Caliper will produce reports containing a number of performance indicators to serve as a reference when using the following blockchain solutions: Hyperledger Besu, Hyperledger Burrow, Ethereum, Hyperledger Fabric, FISCO BCOS, Hyperledger Iroha and Hyperledger Sawtooth.

Hyperledger Ursa is a shared cryptographic library, it enables implementations to avoid duplicating other cryptographic work and hopefully increase security in the process. The library is an opt-in repository (for Hyperledger and non Hyperledger projects) to place and use crypto. Hyperledger Ursa consists of sub-projects, which are cohesive implementations of cryptographic code or interfaces to cryptographic code.

Hyperledger Fabric is an enterprise-grade permissioned distributed ledger framework for developing solutions and applications. Its modular and versatile design satisfies a broad range of industry use cases. It offers a unique approach to consensus that enables performance at scale while preserving privacy.

Geth is an implementation of an Ethereum node in the Go programming language.

Solidity is a purposefully slimmed down, loosely-typed language with a syntax very similar to ECMAScript (Javascript) that is used for the creation of smart contracts on the Ethereum blockchain.

GanacheCLI (NodeJS package) is a fast and customizable blockchain emulator. It simulates the Ethereum network on a single computer and allows you to make calls to the blockchain without any of the hassles of running a real Ethereum node.

Truffle is a development environment, testing framework and asset pipeline for Ethereum, aiming to make life as an Ethereum developer easier.

Ethers.js is a library which was made for and is used by ethers.io. It is designed to make it easier to write client-side JavaScript based wallets, keeping the private key on the owner’s machine at all times.

OpenEthereum is a fast and feature-rich multi-network Ethereum client.

Quorum is an Ethereum-based distributed ledger protocol with transaction/contract privacy and new consensus mechanisms.

Diesel is a safe, extensible ORM and Query Builder for Rust.

BlockShell is a minimal command line utility for learning Blockchain technical concepts like chaining & mining.

Tendermint Core is Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine written in any programming language and securely replicates it on many machines.

inDEX is a comprehensive list of decentralized exchanges (DEX) of cryptocurrencies, tokens, derivatives and futures, and their protocols.

Squeezer is a platform that empowers new-entry blockchain developers to build serverless dApps easily.

Node.js Development

Back to the Top

Node.js Learning Resources

Node.js is a JavaScript runtime built on Chrome’s V8 JavaScript engine that lets developers write command line tools and server-side scripts outside of a browser.

Node.js Build Working Group maintains and controls infrastructure used for continuous integration (CI), releases, benchmarks, web hosting (of nodejs.org and other Node.js web properties) and more.

The OpenJS Foundation is made up of 32 open source JavaScript projects including Appium, Dojo, Electron, jQuery, Node.js, and webpack. The foundation’s mission is to support the healthy growth of JavaScript and web technologies by providing a neutral organization to host and sustain projects, as well as collaboratively fund activities that benefit the ecosystem as a whole.

Set up NodeJS on WSL 2

Getting started with Node.js in Google Cloud

Getting Started with Node.js in AWS

Node.js App Hosting & Deployment in Microsoft Azure

The Node.js best practices list

Introduction to Node.js by W3Schools

The Node.js Community Committee

Node.js Mentorship Program Initiative

Node.js tutorial in Visual Studio Code

Server-side Development with NodeJS, Express and MongoDB on Coursera

Node.js Tools

NPM is the company behind Node package manager, the npm Registry, and npm CLI.

node-gyp is a cross-platform command-line tool written in Node.js for compiling native addon modules for Node.js. It contains a vendored copy of the gyp-next project that was previously used by the Chromium team, extended to support the development of Node.js native addons.

nvm is a version manager for node.js, designed to be installed per-user, and invoked per-shell. nvm works on any POSIX-compliant shell (sh, dash, ksh, zsh, bash), in particular on these platforms: unix, macOS, and windows WSL.

node-docker is the official Node.js docker image, made with love by the node community.

Mocha is a simple, flexible, fun JavaScript test framework for Node.js & The Browser.

AVA is a test runner for Node.js with a concise API, detailed error output, embrace of new language features and process isolation that lets you develop with confidence.

egg is a born to build better enterprise frameworks and apps with Node.js & Koa.

mysqljs is a pure node.js JavaScript Client implementing the MySQL protocol.

axios is a promise based HTTP client for the browser and node.js.

Fastify is a fast and low overhead web framework, for Node.js.

Express is a fast, unopinionated, minimalist web framework for node.

Meteor is an ultra-simple environment for building modern web applications with JavavScript.

NW.js is an app runtime based on Chromium and node.js. You can write native apps in HTML and JavaScript with NW.js. It also lets you call Node.js modules directly from the DOM and enables a new way of writing native applications with all Web technologies.

NestJS is a framework for building efficient, scalable Node.js web applications. It uses modern JavaScript, is built with TypeScript and combines elements of OOP (Object Oriented Progamming), FP (Functional Programming), and FRP (Functional Reactive Programming).

jenkins-nodejs is a Jenkins plugin for Node.js that provides the NodeJS auto-installer, allowing to create as many NodeJS installations “profiles” as you want.

Strapi is an open source Node.js Headless CMS to easily build customisable APIs.

Standard is a JavaScript Style Guide, with linter & automatic code fixer.

React Starter Kit is an isomorphic web app boilerplate for web development built on top of Node.js, Express, GraphQL and React, containing modern web development tools such as Webpack, Babel and Browsersync. Helping you to stay productive following the best practices.

Hexo is a A fast, simple & powerful blog framework, powered by Node.js.

C/C++ Development

Back to the Top

C/C++ Learning Resources

C++ is a cross-platform language that can be used to build high-performance applications developed by Bjarne Stroustrup, as an extension to the C language.

C is a general-purpose, high-level language that was originally developed by Dennis M. Ritchie to develop the UNIX operating system at Bell Labs. It supports structured programming, lexical variable scope, and recursion, with a static type system. C also provides constructs that map efficiently to typical machine instructions, which makes it one was of the most widely used programming languages today.

Embedded C is a set of language extensions for the C programming language by the C Standards Committee to address issues that exist between C extensions for different embedded systems. The extensions hep enhance microprocessor features such as fixed-point arithmetic, multiple distinct memory banks, and basic I/O operations. This makes Embedded C the most popular embedded software language in the world.

C & C++ Developer Tools from JetBrains

Open source C++ libraries on cppreference.com

C++ Graphics libraries

C++ Libraries in MATLAB

C++ Tools and Libraries Articles

Google C++ Style Guide

Introduction C++ Education course on Google Developers

C++ style guide for Fuchsia

C and C++ Coding Style Guide by OpenTitan

Chromium C++ Style Guide

C++ Core Guidelines

C++ Style Guide for ROS

Learn C++

Learn C : An Interactive C Tutorial

C++ Institute

C++ Online Training Courses on LinkedIn Learning

C++ Tutorials on W3Schools

Learn C Programming Online Courses on edX

Learn C++ with Online Courses on edX

Learn C++ on Codecademy

Coding for Everyone: C and C++ course on Coursera

C++ For C Programmers on Coursera

Top C Courses on Coursera

C++ Online Courses on Udemy

Top C Courses on Udemy

C++ For Programmers Course on Udacity

C++ Fundamentals Course on Pluralsight

Introduction to C++ on MIT Free Online Course Materials

Introduction to C++ for Programmers | Harvard

Online C Courses | Harvard University

C/C++ Tools

Visual Studio is an integrated development environment (IDE) from Microsoft; which is a feature-rich application that can be used for many aspects of software development. Visual Studio makes it easy to edit, debug, build, and publish your app. By using Microsoft software development platforms such as Windows API, Windows Forms, Windows Presentation Foundation, and Windows Store.

Visual Studio Code is a code editor redefined and optimized for building and debugging modern web and cloud applications.

Vcpkg is a C++ Library Manager for Windows, Linux, and MacOS.

ReSharper C++ is a Visual Studio Extension for C++ developers developed by JetBrains.

AppCode is constantly monitoring the quality of your code. It warns you of errors and smells and suggests quick-fixes to resolve them automatically. AppCode provides lots of code inspections for Objective-C, Swift, C/C++, and a number of code inspections for other supported languages. All code inspections are run on the fly.

CLion is a cross-platform IDE for C and C++ developers developed by JetBrains.

Code::Blocks is a free C/C++ and Fortran IDE built to meet the most demanding needs of its users. It is designed to be very extensible and fully configurable. Built around a plugin framework, Code::Blocks can be extended with plugins.

CppSharp is a tool and set of libraries which facilitates the usage of native C/C++ code with the .NET ecosystem. It consumes C/C++ header and library files and generates the necessary glue code to surface the native API as a managed API. Such an API can be used to consume an existing native library in your managed code or add managed scripting support to a native codebase.

Conan is an Open Source Package Manager for C++ development and dependency management into the 21st century and on par with the other development ecosystems.

High Performance Computing (HPC) SDK is a comprehensive toolbox for GPU accelerating HPC modeling and simulation applications. It includes the C, C++, and Fortran compilers, libraries, and analysis tools necessary for developing HPC applications on the NVIDIA platform.

Thrust is a C++ parallel programming library which resembles the C++ Standard Library. Thrust’s high-level interface greatly enhances programmer productivity while enabling performance portability between GPUs and multicore CPUs. Interoperability with established technologies such as CUDA, TBB, and OpenMP integrates with existing software.

Boost is an educational opportunity focused on cutting-edge C++. Boost has been a participant in the annual Google Summer of Code since 2007, in which students develop their skills by working on Boost Library development.

Automake is a tool for automatically generating Makefile.in files compliant with the GNU Coding Standards. Automake requires the use of GNU Autoconf.

Cmake is an open-source, cross-platform family of tools designed to build, test and package software. CMake is used to control the software compilation process using simple platform and compiler independent configuration files, and generate native makefiles and workspaces that can be used in the compiler environment of your choice.

GDB is a debugger, that allows you to see what is going on `inside’ another program while it executes or what another program was doing at the moment it crashed.

GCC is a compiler Collection that includes front ends for C, C++, Objective-C, Fortran, Ada, Go, and D, as well as libraries for these languages.

GSL is a numerical library for C and C++ programmers. It is free software under the GNU General Public License. The library provides a wide range of mathematical routines such as random number generators, special functions and least-squares fitting. There are over 1000 functions in total with an extensive test suite.

OpenGL Extension Wrangler Library (GLEW) is a cross-platform open-source C/C++ extension loading library. GLEW provides efficient run-time mechanisms for determining which OpenGL extensions are supported on the target platform.

Libtool is a generic library support script that hides the complexity of using shared libraries behind a consistent, portable interface. To use Libtool, add the new generic library building commands to your Makefile, Makefile.in, or Makefile.am.

Maven is a software project management and comprehension tool. Based on the concept of a project object model (POM), Maven can manage a project’s build, reporting and documentation from a central piece of information.

TAU (Tuning And Analysis Utilities) is capable of gathering performance information through instrumentation of functions, methods, basic blocks, and statements as well as event-based sampling. All C++ language features are supported including templates and namespaces.

Clang is a production quality C, Objective-C, C++ and Objective-C++ compiler when targeting X86-32, X86-64, and ARM (other targets may have caveats, but are usually easy to fix). Clang is used in production to build performance-critical software like Google Chrome or Firefox.

OpenCV is a highly optimized library with focus on real-time applications. Cross-Platform C++, Python and Java interfaces support Linux, MacOS, Windows, iOS, and Android.

Libcu++ is the NVIDIA C++ Standard Library for your entire system. It provides a heterogeneous implementation of the C++ Standard Library that can be used in and between CPU and GPU code.

ANTLR (ANother Tool for Language Recognition) is a powerful parser generator for reading, processing, executing, or translating structured text or binary files. It’s widely used to build languages, tools, and frameworks. From a grammar, ANTLR generates a parser that can build parse trees and also generates a listener interface that makes it easy to respond to the recognition of phrases of interest.

Oat++ is a light and powerful C++ web framework for highly scalable and resource-efficient web application. It’s zero-dependency and easy-portable.

JavaCPP is a program that provides efficient access to native C++ inside Java, not unlike the way some C/C++ compilers interact with assembly language.

Cython is a language that makes writing C extensions for Python as easy as Python itself. Cython is based on Pyrex, but supports more cutting edge functionality and optimizations such as calling C functions and declaring C types on variables and class attributes.

Spdlog is a very fast, header-only/compiled, C++ logging library.

Infer is a static analysis tool for Java, C++, Objective-C, and C. Infer is written in OCaml.

Java Development

Back to the Top

Java Learning Resources

Java is a popular programming language and development platform(JDK). It reduces costs, shortens development timeframes, drives innovation, and improves application services. With millions of developers running more than 51 billion Java Virtual Machines worldwide.

The Eclipse Foundation is home to a worldwide community of developers, the Eclipse IDE, Jakarta EE and over 375 open source projects, including runtimes, tools and frameworks for Java and other languages.

Getting Started with Java

Oracle Java certifications from Oracle University

Google Developers Training

Google Developers Certification

Java Tutorial by W3Schools

Building Your First Android App in Java

Getting Started with Java in Visual Studio Code

Google Java Style Guide

AOSP Java Code Style for Contributors

Chromium Java style guide

Get Started with OR-Tools for Java

Getting started with Java Tool Installer task for Azure Pipelines

Gradle User Manual

Java Tools & Frameworks

Java SE contains several tools to assist in program development and debugging, and in the monitoring and troubleshooting of production applications.

JDK Development Tools includes the Java Web Start Tools (javaws) Java Troubleshooting, Profiling, Monitoring and Management Tools (jcmd, jconsole, jmc, jvisualvm); and Java Web Services Tools (schemagen, wsgen, wsimport, xjc).

Android Studio is the official integrated development environment for Google’s Android operating system, built on JetBrains’ IntelliJ IDEA software and designed specifically for Android development. Availble on Windows, macOS, Linux, Chrome OS.

IntelliJ IDEA is an IDE for Java, but it also understands and provides intelligent coding assistance for a large variety of other languages such as Kotlin, SQL, JPQL, HTML, JavaScript, etc., even if the language expression is injected into a String literal in your Java code.

NetBeans is an IDE provides Java developers with all the tools needed to create professional desktop, mobile and enterprise applications. Creating, Editing, and Refactoring. The IDE provides wizards and templates to let you create Java EE, Java SE, and Java ME applications.

Java Design Patterns is a collection of the best formalized practices a programmer can use to solve common problems when designing an application or system.

Elasticsearch is a distributed RESTful search engine built for the cloud written in Java.

RxJava is a Java VM implementation of Reactive Extensions: a library for composing asynchronous and event-based programs by using observable sequences. It extends the observer pattern to support sequences of data/events and adds operators that allow you to compose sequences together declaratively while abstracting away concerns about things like low-level threading, synchronization, thread-safety and concurrent data structures.

Guava is a set of core Java libraries from Google that includes new collection types (such as multimap and multiset), immutable collections, a graph library, and utilities for concurrency, I/O, hashing, caching, primitives, strings, and more! It is widely used on most Java projects within Google, and widely used by many other companies as well.

okhttp is a HTTP client for Java and Kotlin developed by Square.

Retrofit is a type-safe HTTP client for Android and Java develped by Square.

LeakCanary is a memory leak detection library for Android develped by Square.

Apache Flink is an open source stream processing framework with powerful stream- and batch-processing capabilities with elegant and fluent APIs in Java and Scala.

Fastjson is a Java library that can be used to convert Java Objects into their JSON representation. It can also be used to convert a JSON string to an equivalent Java object.

libGDX is a cross-platform Java game development framework based on OpenGL (ES) that works on Windows, Linux, Mac OS X, Android, your WebGL enabled browser and iOS.

Jenkins is the leading open-source automation server. Built with Java, it provides over 1700 plugins to support automating virtually anything, so that humans can actually spend their time doing things machines cannot.

DBeaver is a free multi-platform database tool for developers, SQL programmers, database administrators and analysts. Supports any database which has JDBC driver (which basically means - ANY database). EE version also supports non-JDBC datasources (MongoDB, Cassandra, Redis, DynamoDB, etc).

Redisson is a Redis Java client with features of In-Memory Data Grid. Over 50 Redis based Java objects and services: Set, Multimap, SortedSet, Map, List, Queue, Deque, Semaphore, Lock, AtomicLong, Map Reduce, Publish / Subscribe, Bloom filter, Spring Cache, Tomcat, Scheduler, JCache API, Hibernate, MyBatis, RPC, and local cache.

GraalVM is a universal virtual machine for running applications written in JavaScript, Python, Ruby, R, JVM-based languages like Java, Scala, Clojure, Kotlin, and LLVM-based languages such as C and C++.

Gradle is a build automation tool for multi-language software development. From mobile apps to microservices, from small startups to big enterprises, Gradle helps teams build, automate and deliver better software, faster. Write in Java, C++, Python or your language of choice.

Apache Groovy is a powerful, optionally typed and dynamic language, with static-typing and static compilation capabilities, for the Java platform aimed at improving developer productivity thanks to a concise, familiar and easy to learn syntax. It integrates smoothly with any Java program, and immediately delivers to your application powerful features, including scripting capabilities, Domain-Specific Language authoring, runtime and compile-time meta-programming and functional programming.

JaCoCo is a free code coverage library for Java, which has been created by the EclEmma team based on the lessons learned from using and integration existing libraries for many years.

Apache JMeter is used to test performance both on static and dynamic resources, Web dynamic applications. It also used to simulate a heavy load on a server, group of servers, network or object to test its strength or to analyze overall performance under different load types.

Junit is a simple framework to write repeatable tests. It is an instance of the xUnit architecture for unit testing frameworks.

Mockito is the most popular Mocking framework for unit tests written in Java.

SpotBugs is a program which uses static analysis to look for bugs in Java code.

SpringBoot is a great tool that helps you to create Spring-powered, production-grade applications and services with absolute minimum fuss. It takes an opinionated view of the Spring platform so that new and existing users can quickly get to the bits they need.

YourKit is a technology leader, creator of the most innovative and intelligent tools for profiling Java & .NET applications.

Python Development

Back to the Top

Python Learning Resources

Python is an interpreted, high-level programming language. Python is used heavily in the fields of Data Science and Machine Learning.

Python Developer’s Guide is a comprehensive resource for contributing to Python – for both new and experienced contributors. It is maintained by the same community that maintains Python.

Azure Functions Python developer guide is an introduction to developing Azure Functions using Python. The content below assumes that you’ve already read the Azure Functions developers guide.

CheckiO is a programming learning platform and a gamified website that teaches Python through solving code challenges and competing for the most elegant and creative solutions.

Python Institute

PCEP – Certified Entry-Level Python Programmer certification

PCAP – Certified Associate in Python Programming certification

PCPP – Certified Professional in Python Programming 1 certification

PCPP – Certified Professional in Python Programming 2

MTA: Introduction to Programming Using Python Certification

Getting Started with Python in Visual Studio Code

Google’s Python Style Guide

Google’s Python Education Class

Real Python

The Python Open Source Computer Science Degree by Forrest Knight

Intro to Python for Data Science

Intro to Python by W3schools

Codecademy’s Python 3 course

Learn Python with Online Courses and Classes from edX

Python Courses Online from Coursera

Python Frameworks and Tools

Python Package Index (PyPI) is a repository of software for the Python programming language. PyPI helps you find and install software developed and shared by the Python community.

PyCharm is the best IDE I’ve ever used. With PyCharm, you can access the command line, connect to a database, create a virtual environment, and manage your version control system all in one place, saving time by avoiding constantly switching between windows.

Python Tools for Visual Studio(PTVS) is a free, open source plugin that turns Visual Studio into a Python IDE. It supports editing, browsing, IntelliSense, mixed Python/C++ debugging, remote Linux/MacOS debugging, profiling, IPython, and web development with Django and other frameworks.

Pylance is an extension that works alongside Python in Visual Studio Code to provide performant language support. Under the hood, Pylance is powered by Pyright, Microsoft’s static type checking tool.

Pyright is a fast type checker meant for large Python source bases. It can run in a “watch” mode and performs fast incremental updates when files are modified.

Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design.

Flask is a micro web framework written in Python. It is classified as a microframework because it does not require particular tools or libraries.

Web2py is an open-source web application framework written in Python allowing allows web developers to program dynamic web content. One web2py instance can run multiple web sites using different databases.

AWS Chalice is a framework for writing serverless apps in python. It allows you to quickly create and deploy applications that use AWS Lambda.

Tornado is a Python web framework and asynchronous networking library. Tornado uses a non-blocking network I/O, which can scale to tens of thousands of open connections.

HTTPie is a command line HTTP client that makes CLI interaction with web services as easy as possible. HTTPie is designed for testing, debugging, and generally interacting with APIs & HTTP servers.

Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

Sentry is a service that helps you monitor and fix crashes in realtime. The server is in Python, but it contains a full API for sending events from any language, in any application.

Pipenv is a tool that aims to bring the best of all packaging worlds (bundler, composer, npm, cargo, yarn, etc.) to the Python world.

Python Fire is a library for automatically generating command line interfaces (CLIs) from absolutely any Python object.

Bottle is a fast, simple and lightweight WSGI micro web-framework for Python. It is distributed as a single file module and has no dependencies other than the Python Standard Library.

CherryPy is a minimalist Python object-oriented HTTP web framework.

Sanic is a Python 3.6+ web server and web framework that’s written to go fast.

Pyramid is a small and fast open source Python web framework. It makes real-world web application development and deployment more fun and more productive.

TurboGears is a hybrid web framework able to act both as a Full Stack framework or as a Microframework.

Falcon is a reliable, high-performance Python web framework for building large-scale app backends and microservices with support for MongoDB, Pluggable Applications and autogenerated Admin.

Neural Network Intelligence(NNI) is an open source AutoML toolkit for automate machine learning lifecycle, including Feature Engineering, Neural Architecture Search, Model Compression and Hyperparameter Tuning.

Dash is a popular Python framework for building ML & data science web apps for Python, R, Julia, and Jupyter.

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built-in.

Locust is an easy to use, scriptable and scalable performance testing tool.

spaCy is a library for advanced Natural Language Processing in Python and Cython.

NumPy is the fundamental package needed for scientific computing with Python.

Pillow is a friendly PIL(Python Imaging Library) fork.

IPython is a command shell for interactive computing in multiple programming languages, originally developed for the Python programming language, that offers enhanced introspection, rich media, additional shell syntax, tab completion, and rich history.

GraphLab Create is a Python library, backed by a C++ engine, for quickly building large-scale, high-performance machine learning models.

Pandas is a fast, powerful, and easy to use open source data structrures, data analysis and manipulation tool, built on top of the Python programming language.

PuLP is an Linear Programming modeler written in python. PuLP can generate LP files and call on use highly optimized solvers, GLPK, COIN CLP/CBC, CPLEX, and GUROBI, to solve these linear problems.

Matplotlib is a 2D plotting library for creating static, animated, and interactive visualizations in Python. Matplotlib produces publication-quality figures in a variety of hardcopy formats and interactive environments across platforms.

Scikit-Learn is a simple and efficient tool for data mining and data analysis. It is built on NumPy,SciPy, and mathplotlib.

Rust Development

Back to the Top

Rust Learning Resources

Rust is a multi-paradigm programming language focused on performance and safety. Rust has a comparable amount of runtime to C and C++, and has set up its standard library to be amenable towards OS development. Specifically, the standard library is split into two parts: core and std. Core is the lowest-level aspects only, and doesn’t include things like allocation, threading, and other higher-level features.

The Rust Language Reference

The Rust Programming Language Book

Learning Rust

Why AWS loves Rust

Rust Programming courses on Udemy

Safety in Systems Programming with Rust at Standford by Ryan Eberhardt

WebAssembly meets Kubernetes with Krustlet using Rust

Microsoft’s Project Verona

Rust Tools

Cargo is a package manager that downloads your Rust project’s dependencies and compiles your project.

Crater is a tool to run experiments across parts of the Rust ecosystem. Its primary purpose is to detect regressions in the Rust compiler, and it does this by building a large number of crates, running their test suites and comparing the results between two versions of the Rust compiler. It can operate locally (with Docker as the only dependency) or distributed on the cloud.

VSCode-Rust is plugin that adds language support for Rust to Visual Studio Code. Rust support is powered by a separate language server - either by the official Rust Language Server (RLS) or rust-analyzer, depending on the user’s preference. If you don’t have it installed, the extension will install it for you (with permission). This extension is built and maintained by the Rust IDEs and editors team with the focus on providing a stable, high quality extension that makes the best use of the respective language server.

Apache Arrow is a development platform for in-memory analytics. It contains a set of technologies that enable big data systems to process and move data fast. Arrow’s libraries are available for C, C++, C#, Go, Java, JavaScript, MATLAB, Python, R, Ruby, and Rust.

Wasmer enables super lightweight containers based on WebAssembly that can run anywhere such as the Desktop to the Cloud and IoT devices, and also embedded in any programming language.

Firecracker is an open source virtualization technology that is purpose-built for creating and managing secure, multi-tenant container and function-based services that provide serverless operational models. Firecracker runs workloads in lightweight virtual machines, called microVMs, which combine the security and isolation properties provided by hardware virtualization technology with the speed and flexibility of containers. Firecracker has also been integrated in container runtimes, for example Kata Containers and Weaveworks Ignite.

Tokio is an event-driven, non-blocking I/O platform for writing asynchronous applications with the Rust programming language.

TiKV is an open-source distributed transactional key-value database that also provides classical key-vlue APIs, but also transactional APIs with ACID compliance.

Sonic is a fast, lightweight and schema-less search backend similar to Elasticsearch in some use-cases.

Hyper is a fast and correct HTTP library for Rust.

Rocket is an async web framework for Rust with a focus on usability, security, extensibility, and speed.

Clippy is a collection of lints to catch common mistakes and improve your Rust code.

Servo is a prototype web browser engine written in the Rust language.

Vector is a high-performance, end-to-end (agent & aggregator) observability data platform that puts the user in control of their observability data.

RustPython is a Python Interpreter written in Rust.

Miri is an interpreter for Rust’s mid-level intermediate representation. It can run binaries and test suites of cargo projects and detect certain classes of undefined behavior. Miri will alsowill also tell you about memory leaks: when there is memory still allocated at the end of the execution, and that memory is not reachable from a global static, Miri will raise an error.

Chalk is an implementation and definition of the Rust trait system using a PROLOG-like logic solver.

stdarch is Rust’s standard library vendor-specific APIs and run-time feature detection.

Simpleinfra is rep that contains the tools and automation written by the Rust infrastructure team to manage our services. Using some of the tools in this repo require privileges only infra team members have.

Rustlings is a small set of exercises to get you used to reading and writing Rust code.

Krustlet acts as a Kubernetes Kubelet(written in Rust) by listening on the event stream for new pods that the scheduler assigns to it based on specific Kubernetes tolerations. The project is currently experimental.

Operating System

Redox is a Unix-like Operating System written in Rust, aiming to bring the innovations of Rust to a modern microkernel and full set of applications. Acitvely being developed by Jeremy Soeller.

Bottlerocket OS is an open-source Linux-based operating system meant for hosting containers. Bottlerocket focuses on security and maintainability, providing a reliable, consistent, and safe platform for container-based workloads.

Tock is an embedded operating system designed for running multiple concurrent, mutually distrustful applications on Cortex-M and RISC-V based embedded platforms. Tock’s design centers around protection, both from potentially malicious applications and from device drivers. Tock uses two mechanisms to protect different components of the operating system. First, the kernel and device drivers are written in Rust, a systems programming language that provides compile-time memory safety, type safety and strict aliasing. Tock uses Rust to protect the kernel (the scheduler and hardware abstraction layer) from platform specific device drivers as well as isolate device drivers from each other. Second, Tock uses memory protection units to isolate applications from each other and the kernel.

Rust on Chrome OS is a document that provides information on creating Rust projects for installation within Chrome OS and Chrome OS SDK.

Writing an OS in Rust is a blog series creates a small operating system in the Rust programming language by Philipp Oppermann.

Swift Development

Back to the Top

Swift Learning Resources

Swift is Apple’s main programming language for iOS, macOS, watchOS, and tvOS app development. Though, many parts of Swift will be familiar to developers from their experience of developing in C and Objective-C.

Swift Evolution maintains proposals for changes and user-visible enhancements to the Swift Programming Language.

Xcode + Swift makes developing applications for MacOS and iOS fast and fun.

Swift 5.3 Basics

Start Developing iOS Apps with Swift

Apple Developer Documentation

Apple Foundation Framework

Apple Core Animation Framework

Apple Core Graphics Framework

Getting Started with LLDB

Mac Catalyst - iOS - Human Interface Guidelines

Amazon EC2 Mac Instances

Swift GitHub

Apple Developer Forums

Swift Forums

Google’s Swift Style Guide

Swift Courses Online from Coursera

Swift Courses Online from Udemy

Learning Swift course from Codecademy

Swift Tools

Xcode includes everything developers need to create great applications for Mac, iPhone, iPad, Apple TV, and Apple Watch. Xcode provides developers a unified workflow for user interface design, coding, testing, and debugging. Xcode 12 is built as an Universal app that runs 100% natively on Intel-based CPUs and Apple Silicon. It includes a unified macOS SDK that features all the frameworks, compilers, debuggers, and other tools you need to build apps that run natively on Apple Silicon and the Intel x86_64 CPU.

SwiftUI is a user interface toolkit that provides views, controls, and layout structures for declaring your app’s user interface. The SwiftUI framework provides event handlers for delivering taps, gestures, and other types of input to your application.

UIKit is a framework provides the required infrastructure for your iOS or tvOS apps. It provides the window and view architecture for implementing your interface, the event handling infrastructure for delivering Multi-Touch and other types of input to your app, and the main run loop needed to manage interactions among the user, the system, and your app.

AppKit is a graphical user interface toolkit that contains all the objects you need to implement the user interface for a macOS app such as windows, panels, buttons, menus, scrollers, and text fields, and it handles all the details for you as it efficiently draws on the screen, communicates with hardware devices and screen buffers, clears areas of the screen before drawing, and clips views.

ARKit is a set set of software development tools to enable developers to build augmented-reality apps for iOS developed by Apple. The latest version ARKit 3.5 takes advantage of the new LiDAR Scanner and depth sensing system on iPad Pro(2020) to support a new generation of AR apps that use Scene Geometry for enhanced scene understanding and object occlusion.

RealityKit is a framework to implement high-performance 3D simulation and rendering with information provided by the ARKit framework to seamlessly integrate virtual objects into the real world.

SceneKit is a high-level 3D graphics framework that helps you create 3D animated scenes and effects in your iOS apps.

Mac Catalyst is a set of Apple APIs that developers can use to rapidly port their iOS apps to Apple Silicon M1 Chip and take full advantage of the new capabilities on the new Apple hardware.

Instruments is a powerful and flexible performance-analysis and testing tool that’s part of the Xcode tool set. It’s designed to help you profile your iOS, watchOS, tvOS, and macOS apps, processes, and devices in order to better understand and optimize their behavior and performance.

Cocoapods is a dependency manager for Swift and Objective-C used in Xcode projects by specifying the dependencies for your project in a simple text file. CocoaPods then recursively resolves dependencies between libraries, fetches source code for all dependencies, and creates and maintains an Xcode workspace to build your project.

Vapor is a web framework for Swift. It provides a beautifully expressive and easy to use foundation for your next website, API, or cloud project.

Hero is a library for building iOS view controller transitions. It provides a declarative layer on top of the UIKit’s cumbersome transition APIs—making custom transitions an easy task for developers.

Kingfisher is a powerful, pure-Swift library for downloading and caching images from the web. It provides you a chance to use a pure-Swift way to work with remote images in your next app.

Realm is a mobile database that runs directly inside phones, tablets or wearables. This repository holds the source code for the iOS, macOS, tvOS & watchOS versions of Realm Swift & Realm Objective-C.

Perfect is a complete and powerful toolbox, framework, and application server for Linux, iOS, and macOS (OS X). It provides everything a Swift engineer needs for developing lightweight, maintainable, and scalable apps and other REST services entirely in the Swift programming language for both client-facing and server-side applications.

Alamofire is an HTTP networking library written in Swift.

Eureka is an elegant iOS form builder in Swift

Carthage is intended to be the simplest way to add frameworks to your Cocoa application. Carthage builds your dependencies and provides you with binary frameworks, but you retain full control over your project structure and setup. Carthage does not automatically modify your project files or your build settings.

ReactiveCocoa is reactive extensions to Cocoa frameworks, built on top of ReactiveSwift.

XML Development

Back to the Top

XML Learning Resources

XML(Extensible Markup Language) is a markup language that is similar to HTML but without predefined tags. Instead, the user define their own tags designed specifically for their needs.

XAML is a declarative XML-based markup language that is applied to the .NET Core programming model, which XAML simplifies creating a UI for a .NET Core app.

Using XML editor in Visual Studio

XMLmind XML Editor Tutorial

XMLReader for Android Developers

Adding app configuration policies for managed iOS/iPadOS devices

XML Processing Modules for Python 3

PHP: XML Parser Manual

Introduction to XML on W3Schools

Introduction to XML by IBM

XML Fundamentals Training Program from Oracle University

XML Online Training Courses on Udemy

XML Schema (XSD) Crash Course for Beginners on Udemy

XML Online Training Courses on Coursera

XML Online Training Courses on LinkedIn Learning

Introduction to Microsoft Office Open XML on Pluralsight

XML Fundamentals in C# on Pluralsight

Working with XML in Java Using JAXB

Querying JSON, XML, and Temporal Data with T-SQL

XML Training Courses on NobleProg

XML Tools and Frameworks

MSBuild is the build platform for .NET and Visual Studio. MSBuild, provides an XML schema for a project file that controls how the build platform processes and builds software. Visual Studio uses MSBuild to perform team builds through Azure DevOps Server, but MSBuild can run without Visual Studio.

WebStorm is a professional IDE for JavaScript(including support for both HTML and CSS) developed by JetBrains. WebStorm comes with intelligent code completion, on-the-fly error detection, powerful navigation and refactoring for JavaScript, TypeScript, stylesheet languages, and all the most popular frameworks(Angular, React, Vue.js, Ionic, Apache Cordova, React Native, Node.js, Meteor, and Electron).

Uno Platform is a Universal Windows Platform Bridge that allows UWP-based code (C# and XAML) to run on iOS, Android, macOS, WebAssembly, Linux and Windows. It provides the full definitions of the UWP Windows 10 2004 (19041), and the implementation of a growing number of parts of the UWP API, such as Windows.UI.Xaml, to enable UWP and WinUI applications to run on these platforms.

Oxygen XML Editor is a cross platform tool setting the standard in XML editing. The advanced XML editor provides the most intuitive tools for XML editing, authoring, and development. It is availble on Windows, macOS, and Linux.

XMLmind XML Editor is a strictly validating XML editor. It has Extensive DITA 1.3 support, and includes lightweight DITA, XDITA, HDITA, MDITA.

Refit is the automatic type-safe REST library for .NET Core,.NET and Xamarin.

Pugixml is a C++ XML processing library, which consists of a DOM-like interface with rich traversal/modification capabilities, an extremely fast XML parser which constructs the DOM tree from an XML file/buffer, and an XPath 1.0 implementation for complex data-driven tree queries.

Serializer is a component that makes it easier to handle serializing data structures, including object graphs, into array structures or other formats like XML and JSON. It can also handle deserializing XML and JSON back to object graphs.

Erlsom is an XML parser for Erlang.

REST-assured is a Java DSL for easy testing of REST services.

Contribute

If would you like to contribute to this guide simply make a Pull Request.

License

Back to the Top

Distributed under the Creative Commons Attribution 4.0 International (CC BY 4.0) Public License.

ruby-sdk

Fri, 06 Jun 2025 15:29:47 +0800

modelcontextprotocol/ruby-sdk

MCP Ruby SDK

The official Ruby SDK for Model Context Protocol servers and clients.

Installation

Add this line to your application’s Gemfile:

`1`	`gem 'mcp'`

And then execute:

`1`	`$ bundle install`

Or install it yourself as:

`1`	`$ gem install mcp`

MCP Server

The MCP::Server class is the core component that handles JSON-RPC requests and responses. It implements the Model Context Protocol specification, handling model context requests and responses.

Key Features

Implements JSON-RPC 2.0 message handling
Supports protocol initialization and capability negotiation
Manages tool registration and invocation
Supports prompt registration and execution
Supports resource registration and retrieval

Supported Methods

initialize - Initializes the protocol and returns server capabilities
ping - Simple health check
tools/list - Lists all registered tools and their schemas
tools/call - Invokes a specific tool with provided arguments
prompts/list - Lists all registered prompts and their schemas
prompts/get - Retrieves a specific prompt by name
resources/list - Lists all registered resources and their schemas
resources/read - Retrieves a specific resource by name
resources/templates/list - Lists all registered resource templates and their schemas

Unsupported Features ( to be implemented in future versions )

Notifications
Log Level
Resource subscriptions
Completions
Complete StreamableHTTP implementation with streaming responses

Usage

Rails Controller

When added to a Rails controller on a route that handles POST requests, your server will be compliant with non-streaming StreamableHTTP transport requests.

You can use the Server#handle_json method to handle requests.

class ApplicationController < ActionController::Base

  def index
    server = MCP::Server.new(
      name: "my_server",
      version: "1.0.0",
      tools: [SomeTool, AnotherTool],
      prompts: [MyPrompt],
      server_context: { user_id: current_user.id },
    )
    render(json: server.handle_json(request.body.read))
  end
end

Stdio Transport

If you want to build a local command-line application, you can use the stdio transport:

#!/usr/bin/env ruby
require "mcp"
require "mcp/server/transports/stdio_transport"

# Create a simple tool
class ExampleTool < MCP::Tool
  description "A simple example tool that echoes back its arguments"
  input_schema(
    properties: {
      message: { type: "string" },
    },
    required: ["message"]
  )

  class << self
    def call(message:, server_context:)
      MCP::Tool::Response.new([{
        type: "text",
        text: "Hello from example tool! Message: #{message}",
      }])
    end
  end
end

# Set up the server
server = MCP::Server.new(
  name: "example_server",
  tools: [ExampleTool],
)

# Create and start the transport
transport = MCP::Server::Transports::StdioTransport.new(server)
transport.open

You can run this script and then type in requests to the server at the command line.

1
2
3

$ ./examples/stdio_server.rb
{"jsonrpc":"2.0","id":"1","method":"ping"}
{"jsonrpc":"2.0","id":"2","method":"tools/list"}

Configuration

The gem can be configured using the MCP.configure block:

MCP.configure do |config|
  config.exception_reporter = ->(exception, server_context) {
    # Your exception reporting logic here
    # For example with Bugsnag:
    Bugsnag.notify(exception) do |report|
      report.add_metadata(:model_context_protocol, server_context)
    end
  }

  config.instrumentation_callback = ->(data) {
    puts "Got instrumentation data #{data.inspect}"
  }
end

or by creating an explicit configuration and passing it into the server. This is useful for systems where an application hosts more than one MCP server but they might require different instrumentation callbacks.

configuration = MCP::Configuration.new
configuration.exception_reporter = ->(exception, server_context) {
  # Your exception reporting logic here
  # For example with Bugsnag:
  Bugsnag.notify(exception) do |report|
    report.add_metadata(:model_context_protocol, server_context)
  end
}

configuration.instrumentation_callback = ->(data) {
  puts "Got instrumentation data #{data.inspect}"
}

server = MCP::Server.new(
  # ... all other options
  configuration:,
)

Server Context and Configuration Block Data

`server_context`

The server_context is a user-defined hash that is passed into the server instance and made available to tools, prompts, and exception/instrumentation callbacks. It can be used to provide contextual information such as authentication state, user IDs, or request-specific data.

Type:

`1`	`server_context: { [String, Symbol] => Any }`

Example:

server = MCP::Server.new(
  name: "my_server",
  server_context: { user_id: current_user.id, request_id: request.uuid }
)

This hash is then passed as the server_context argument to tool and prompt calls, and is included in exception and instrumentation callbacks.

Configuration Block Data

Exception Reporter

The exception reporter receives:

exception: The Ruby exception object that was raised
server_context: The context hash provided to the server

Signature:

`1`	`exception_reporter = ->(exception, server_context) { ... }`

Instrumentation Callback

The instrumentation callback receives a hash with the following possible keys:

method: (String) The protocol method called (e.g., “ping”, “tools/list”)
tool_name: (String, optional) The name of the tool called
prompt_name: (String, optional) The name of the prompt called
resource_uri: (String, optional) The URI of the resource called
error: (String, optional) Error code if a lookup failed
duration: (Float) Duration of the call in seconds

Type:

1
2

instrumentation_callback = ->(data) { ... }
# where data is a Hash with keys as described above

Example:

1
2
3

config.instrumentation_callback = ->(data) {
  puts "Instrumentation: #{data.inspect}"
}

Server Protocol Version

The server’s protocol version can be overridden using the protocol_version class method:

`1`	`MCP::Server.protocol_version = "2024-11-05"`

This will make all new server instances use the specified protocol version instead of the default version. The protocol version can be reset to the default by setting it to nil:

`1`	`MCP::Server.protocol_version = nil`

Be sure to check the MCP spec for the protocol version to understand the supported features for the version being set.

Exception Reporting

The exception reporter receives two arguments:

exception: The Ruby exception object that was raised
server_context: A hash containing contextual information about where the error occurred

The server_context hash includes:

For tool calls: { tool_name: "name", arguments: { ... } }
For general request handling: { request: { ... } }

When an exception occurs:

The exception is reported via the configured reporter
For tool calls, a generic error response is returned to the client: { error: "Internal error occurred", isError: true }
For other requests, the exception is re-raised after reporting

If no exception reporter is configured, a default no-op reporter is used that silently ignores exceptions.

Tools

MCP spec includes Tools which provide functionality to LLM apps.

This gem provides a MCP::Tool class that can be used to create tools in two ways:

As a class definition:

class MyTool < MCP::Tool
  description "This tool performs specific functionality..."
  input_schema(
    properties: {
      message: { type: "string" },
    },
    required: ["message"]
  )
  annotations(
    title: "My Tool",
    read_only_hint: true,
    destructive_hint: false,
    idempotent_hint: true,
    open_world_hint: false
  )

  def self.call(message:, server_context:)
    MCP::Tool::Response.new([{ type: "text", text: "OK" }])
  end
end

tool = MyTool

By using the MCP::Tool.define method with a block:

tool = MCP::Tool.define(
  name: "my_tool",
  description: "This tool performs specific functionality...",
  annotations: {
    title: "My Tool",
    read_only_hint: true
  }
) do |args, server_context|
  Tool::Response.new([{ type: "text", text: "OK" }])
end

The server_context parameter is the server_context passed into the server and can be used to pass per request information, e.g. around authentication state.

Tool Annotations

Tools can include annotations that provide additional metadata about their behavior. The following annotations are supported:

title: A human-readable title for the tool
read_only_hint: Indicates if the tool only reads data (doesn’t modify state)
destructive_hint: Indicates if the tool performs destructive operations
idempotent_hint: Indicates if the tool’s operations are idempotent
open_world_hint: Indicates if the tool operates in an open world context

Annotations can be set either through the class definition using the annotations class method or when defining a tool using the define method.

Prompts

MCP spec includes Prompts, which enable servers to define reusable prompt templates and workflows that clients can easily surface to users and LLMs.

The MCP::Prompt class provides two ways to create prompts:

As a class definition with metadata:

class MyPrompt < MCP::Prompt
  prompt_name "my_prompt"  # Optional - defaults to underscored class name
  description "This prompt performs specific functionality..."
  arguments [
    Prompt::Argument.new(
      name: "message",
      description: "Input message",
      required: true
    )
  ]

  class << self
    def template(args, server_context:)
      Prompt::Result.new(
        description: "Response description",
        messages: [
          Prompt::Message.new(
            role: "user",
            content: Content::Text.new("User message")
          ),
          Prompt::Message.new(
            role: "assistant",
            content: Content::Text.new(args["message"])
          )
        ]
      )
    end
  end
end

prompt = MyPrompt

Using the MCP::Prompt.define method:

prompt = MCP::Prompt.define(
  name: "my_prompt",
  description: "This prompt performs specific functionality...",
  arguments: [
    Prompt::Argument.new(
      name: "message",
      description: "Input message",
      required: true
    )
  ]
) do |args, server_context:|
  Prompt::Result.new(
    description: "Response description",
    messages: [
      Prompt::Message.new(
        role: "user",
        content: Content::Text.new("User message")
      ),
      Prompt::Message.new(
        role: "assistant",
        content: Content::Text.new(args["message"])
      )
    ]
  )
end

The server_context parameter is the server_context passed into the server and can be used to pass per request information, e.g. around authentication state or user preferences.

Key Components

Prompt::Argument - Defines input parameters for the prompt template
Prompt::Message - Represents a message in the conversation with a role and content
Prompt::Result - The output of a prompt template containing description and messages
Content::Text - Text content for messages

Usage

server = MCP::Server.new(
  name: "my_server",
  prompts: [MyPrompt],
  server_context: { user_id: current_user.id },
)

The server will handle prompt listing and execution through the MCP protocol methods:

prompts/list - Lists all registered prompts and their schemas
prompts/get - Retrieves and executes a specific prompt with arguments

Instrumentation

The server allows registering a callback to receive information about instrumentation. To register a handler pass a proc/lambda to as instrumentation_callback into the server constructor.

MCP.configure do |config|
  config.instrumentation_callback = ->(data) {
    puts "Got instrumentation data #{data.inspect}"
  }
end

The data contains the following keys: method: the metod called, e.g. ping, tools/list, tools/call etc tool_name: the name of the tool called prompt_name: the name of the prompt called resource_uri: the uri of the resource called error: if looking up tools/prompts etc failed, e.g. tool_not_found duration: the duration of the call in seconds

tool_name, prompt_name and resource_uri are only populated if a matching handler is registered. This is to avoid potential issues with metric cardinality

Resources

MCP spec includes Resources

The MCP::Resource class provides a way to register resources with the server.

resource = MCP::Resource.new(
  uri: "example.com/my_resource",
  mime_type: "text/plain",
  text: "Lorem ipsum dolor sit amet"
)

server = MCP::Server.new(
  name: "my_server",
  resources: [resource],
)

The server must register a handler for the resources/read method to retrieve a resource dynamically.

server.resources_read_handler do |params|
  [{
    uri: params[:uri],
    mimeType: "text/plain",
    text: "Hello, world!",
  }]
end

otherwise ‘resources/read’ requests will be a no-op.

Releases

This gem is published to RubyGems.org

Releases are triggered by PRs to the main branch updating the version number in lib/mcp/version.rb.

Update the version number in lib/mcp/version.rb, following semver
Create A PR and get approval from a maintainer
Merge your PR to the main branch - This will automatically trigger the release workflow via GitHub Actions

When changes are merged to the main branch, the GitHub Actions workflow (.github/workflows/release.yml) is triggered and the gem is published to RubyGems.

Tools on Producthunt daily

Prompt-Engineering-Guide

dair-ai/Prompt-Engineering-Guide

Prompt Engineering Guide

Sponsored by

Announcements / Updates

Guides

Lecture

Running the guide locally

Appearances

License

cai

aliasrobotics/cai

Cybersecurity AI (CAI)

:bookmark: Table of Contents

🎯 Impact

🏆 Competitions and challenges

📊 Research Impact

📚 Research products: Cybersecurity AI

PoCs

Motivation

:bust_in_silhouette: Why CAI?

Ethical principles behind CAI

Closed-source alternatives

Learn - CAI Fluency

:nut_and_bolt: Install

OS X

Ubuntu 24.04

Ubuntu 20.04

Windows WSL

Android

:nut_and_bolt: Setup .env file

🔹 Custom OpenAI Base URL Support

:triangular_ruler: Architecture:

🔹 Agent

🔹 Tools

🔹 Handoffs

🔹 Patterns

🔹 Turns and Interactions

🔹 Tracing

🔹 Guardrails

🔹 Human-In-The-Loop (HITL)

:rocket: Quickstart

Environment Variables

OpenRouter Integration

Azure OpenAI

MCP

Development

Contributions

Optional Requirements: caiextensions

:information_source: Usage Data Collection

Reproduce CI-Setup locally

FAQ

Citation

Acknowledgements

Academic Collaborations

koog

JetBrains/koog

Koog

Overview

Key features

Available LLM providers and platforms

Quickstart example

Using in your projects

Supported targets

Gradle (Kotlin DSL)

Gradle (Groovy)

Maven

Contributing

Code of Conduct

License

Support

genai-toolbox

googleapis/genai-toolbox

MCP Toolbox for Databases

Table of Contents

Why Toolbox?

General Architecture

Getting Started

Installing the server

Cybersecurity AI (`CAI`)

📚 Research products: `Cybersecurity AI`

Learn - `CAI` Fluency

:nut_and_bolt: Setup `.env` file

CLI Tools ^[TOC]

GUI Tools ^[TOC]

Web Tools ^[TOC]