OpenAI API on Producthunt daily

llama.cpp

Tue, 14 Oct 2025 15:29:21 +0800

ggml-org/llama.cpp

llama.cpp

Manifesto / ggml / ops

LLM inference in C/C++

Recent API changes

Hot topics

guide : running gpt-oss with llama.cpp
[FEEDBACK] Better packaging for llama.cpp to support downstream consumers 🤗
Support for the gpt-oss model with native MXFP4 format has been added | PR | Collaboration with NVIDIA | Comment
Hot PRs: All | Open
Multimodal support arrived in llama-server: #12898 | documentation
VS Code extension for FIM completions: https://github.com/ggml-org/llama.vscode
Vim/Neovim plugin for FIM completions: https://github.com/ggml-org/llama.vim
Introducing GGUF-my-LoRA https://github.com/ggml-org/llama.cpp/discussions/10123
Hugging Face Inference Endpoints now support GGUF out of the box! https://github.com/ggml-org/llama.cpp/discussions/9669
Hugging Face GGUF editor: discussion | tool

Quick start

Getting started with llama.cpp is straightforward. Here are several ways to install it on your machine:

Install llama.cpp using brew, nix or winget
Run with Docker - see our Docker documentation
Download pre-built binaries from the releases page
Build from source by cloning this repository - check out our build guide

Once installed, you’ll need a model to work with. Head to the Obtaining and quantizing models section to learn more.

Example command:

# Use a local model file
llama-cli -m my_model.gguf

# Or download and run a model directly from Hugging Face
llama-cli -hf ggml-org/gemma-3-1b-it-GGUF

# Launch OpenAI-compatible API server
llama-server -hf ggml-org/gemma-3-1b-it-GGUF

Description

The main goal of llama.cpp is to enable LLM inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

Plain C/C++ implementation without any dependencies
Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
AVX, AVX2, AVX512 and AMX support for x86 architectures
1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
Vulkan and SYCL backend support
CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is the main playground for developing new features for the ggml library.

Models

Typically finetunes of the base models below are supported as well.

Instructions for adding support for new models: HOWTO-add-model.md

Text-only

LLaMA 🦙
LLaMA 2 🦙🦙
LLaMA 3 🦙🦙🦙
Mistral 7B
Mixtral MoE
DBRX
Falcon
Chinese LLaMA / Alpaca and Chinese LLaMA-2 / Alpaca-2
Vigogne (French)
BERT
Koala
Baichuan 1 & 2 + derivations
Aquila 1 & 2
Starcoder models
Refact
MPT
Bloom
Yi models
StableLM models
Deepseek models
Qwen models
PLaMo-13B
Phi models
PhiMoE
GPT-2
Orion 14B
InternLM2
CodeShell
Gemma
Mamba
Grok-1
Xverse
Command-R models
SEA-LION
GritLM-7B + GritLM-8x7B
OLMo
OLMo 2
OLMoE
Granite models
GPT-NeoX + Pythia
Snowflake-Arctic MoE
Smaug
Poro 34B
Bitnet b1.58 models
Flan T5
Open Elm models
ChatGLM3-6b + ChatGLM4-9b + GLMEdge-1.5b + GLMEdge-4b
GLM-4-0414
SmolLM
EXAONE-3.0-7.8B-Instruct
FalconMamba Models
Jais
Bielik-11B-v2.3
RWKV-6
QRWKV-6
GigaChat-20B-A3B
Trillion-7B-preview
Ling models
LFM2 models
Hunyuan models

Multimodal

Bindings

Python: ddh0/easy-llama
Python: abetlen/llama-cpp-python
Go: go-skynet/go-llama.cpp
Node.js: withcatai/node-llama-cpp
JS/TS (llama.cpp server client): lgrammel/modelfusion
JS/TS (Programmable Prompt Engine CLI): offline-ai/cli
JavaScript/Wasm (works in browser): tangledgroup/llama-cpp-wasm
Typescript/Wasm (nicer API, available on npm): ngxson/wllama
Ruby: yoshoku/llama_cpp.rb
Rust (more features): edgenai/llama_cpp-rs
Rust (nicer API): mdrokz/rust-llama.cpp
Rust (more direct bindings): utilityai/llama-cpp-rs
Rust (automated build from crates.io): ShelbyJenkins/llm_client
C#/.NET: SciSharp/LLamaSharp
C#/VB.NET (more features - community license): LM-Kit.NET
Scala 3: donderom/llm4s
Clojure: phronmophobic/llama.clj
React Native: mybigday/llama.rn
Java: kherud/java-llama.cpp
Java: QuasarByte/llama-cpp-jna
Zig: deins/llama.cpp.zig
Flutter/Dart: netdur/llama_cpp_dart
Flutter: xuegao-tzx/Fllama
PHP (API bindings and features built on top of llama.cpp): distantmagic/resonance (more info)
Guile Scheme: guile_llama_cpp
Swift srgtuszy/llama-cpp-swift
Swift ShenghaiWang/SwiftLlama
Delphi Embarcadero/llama-cpp-delphi

UIs

(to have a project listed here, it should clearly state that it depends on llama.cpp)

AI Sublime Text plugin (MIT)
cztomsik/ava (MIT)
Dot (GPL)
eva (MIT)
iohub/collama (Apache-2.0)
janhq/jan (AGPL)
johnbean393/Sidekick (MIT)
KanTV (Apache-2.0)
KodiBot (GPL)
llama.vim (MIT)
LARS (AGPL)
Llama Assistant (GPL)
LLMFarm (MIT)
LLMUnity (MIT)
LMStudio (proprietary)
LocalAI (MIT)
LostRuins/koboldcpp (AGPL)
MindMac (proprietary)
MindWorkAI/AI-Studio (FSL-1.1-MIT)
Mobile-Artificial-Intelligence/maid (MIT)
Mozilla-Ocho/llamafile (Apache-2.0)
nat/openplayground (MIT)
nomic-ai/gpt4all (MIT)
ollama/ollama (MIT)
oobabooga/text-generation-webui (AGPL)
PocketPal AI (MIT)
psugihara/FreeChat (MIT)
ptsochantaris/emeltal (MIT)
pythops/tenere (AGPL)
ramalama (MIT)
semperai/amica (MIT)
withcatai/catai (MIT)
Autopen (GPL)

Tools

akx/ggify – download PyTorch models from HuggingFace Hub and convert them to GGML
akx/ollama-dl – download models from the Ollama library to be used directly with llama.cpp
crashr/gppm – launch llama.cpp instances utilizing NVIDIA Tesla P40 or P100 GPUs with reduced idle power consumption
gpustack/gguf-parser - review/check the GGUF file and estimate the memory usage
Styled Lines (proprietary licensed, async wrapper of inference part for game development in Unity3d with pre-built Mobile and Web platform wrappers and a model example)

Infrastructure

Paddler - Open-source LLMOps platform for hosting and scaling AI in your own infrastructure
GPUStack - Manage GPU clusters for running LLMs
llama_cpp_canister - llama.cpp as a smart contract on the Internet Computer, using WebAssembly
llama-swap - transparent proxy that adds automatic model switching with llama-server
Kalavai - Crowdsource end to end LLM deployment at any scale
llmaz - ☸️ Easy, advanced inference platform for large language models on Kubernetes.

Games

Lucy’s Labyrinth - A simple maze game where agents controlled by an AI model will try to trick you.

Supported backends

Backend	Target devices
Metal	Apple Silicon
BLAS	All
BLIS	All
SYCL	Intel and Nvidia GPU
MUSA	Moore Threads GPU
CUDA	Nvidia GPU
HIP	AMD GPU
Vulkan	GPU
CANN	Ascend NPU
OpenCL	Adreno GPU
IBM zDNN	IBM Z & LinuxONE
WebGPU [In Progress]	All
RPC	All

Obtaining and quantizing models

The Hugging Face platform hosts a number of LLMs compatible with llama.cpp:

You can either manually download the GGUF file or directly use any llama.cpp-compatible models from Hugging Face or other model hosting sites, such as ModelScope, by using this CLI argument: -hf <user>/<model>[:quant]. For example:

`1`	`llama-cli -hf ggml-org/gemma-3-1b-it-GGUF`

By default, the CLI would download from Hugging Face, you can switch to other options with the environment variable MODEL_ENDPOINT. For example, you may opt to downloading model checkpoints from ModelScope or other model sharing communities by setting the environment variable, e.g. MODEL_ENDPOINT=https://www.modelscope.cn/.

After downloading a model, use the CLI tools to run it locally - see below.

llama.cpp requires the model to be stored in the GGUF file format. Models in other data formats can be converted to GGUF using the convert_*.py Python scripts in this repo.

The Hugging Face platform provides a variety of online tools for converting, quantizing and hosting models with llama.cpp:

Use the GGUF-my-repo space to convert to GGUF format and quantize model weights to smaller sizes
Use the GGUF-my-LoRA space to convert LoRA adapters to GGUF format (more info: https://github.com/ggml-org/llama.cpp/discussions/10123)
Use the GGUF-editor space to edit GGUF meta data in the browser (more info: https://github.com/ggml-org/llama.cpp/discussions/9268)
Use the Inference Endpoints to directly host llama.cpp in the cloud (more info: https://github.com/ggml-org/llama.cpp/discussions/9669)

To learn more about model quantization, read this documentation

`llama-cli`

A CLI tool for accessing and experimenting with most of `llama.cpp`’s functionality.

Run in conversation mode

Models with a built-in chat template will automatically activate conversation mode. If this doesn’t occur, you can manually enable it by adding -cnv and specifying a suitable chat template with --chat-template NAME

llama-cli -m model.gguf

# > hi, who are you?
# Hi there! I'm your helpful assistant! I'm an AI-powered chatbot designed to assist and provide information to users like you. I'm here to help answer your questions, provide guidance, and offer support on a wide range of topics. I'm a friendly and knowledgeable AI, and I'm always happy to help with anything you need. What's on your mind, and how can I assist you today?
#
# > what is 1+1?
# Easy peasy! The answer to 1+1 is... 2!

Run in conversation mode with custom chat template

# use the "chatml" template (use -h to see the list of supported templates)
llama-cli -m model.gguf -cnv --chat-template chatml

# use a custom template
llama-cli -m model.gguf -cnv --in-prefix 'User: ' --reverse-prompt 'User:'

Run simple text completion

To disable conversation mode explicitly, use -no-cnv

1
2
3

llama-cli -m model.gguf -p "I believe the meaning of life is" -n 128 -no-cnv

# I believe the meaning of life is to find your own truth and to live in accordance with it. For me, this means being true to myself and following my passions, even if they don't align with societal expectations. I think that's what I love about yoga – it's not just a physical practice, but a spiritual one too. It's about connecting with yourself, listening to your inner voice, and honoring your own unique journey.

Constrain the output with a custom grammar
1 2 3

llama-cli -m model.gguf -n 256 --grammar-file grammars/json.gbnf -p 'Request: schedule a call at 8pm; Command:' # {"appointmentTime": "8pm", "appointmentDetails": "schedule a a call"}
The grammars/ folder contains a handful of sample grammars. To write your own, check out the GBNF Guide.

For authoring more complex JSON grammars, check out https://grammar.intrinsiclabs.ai/

`llama-server`

A lightweight, OpenAI API compatible, HTTP server for serving LLMs.

Start a local HTTP server with default configuration on port 8080

llama-server -m model.gguf --port 8080

# Basic web UI can be accessed via browser: http://localhost:8080
# Chat completion endpoint: http://localhost:8080/v1/chat/completions

Support multiple-users and parallel decoding

1
2

# up to 4 concurrent requests, each with 4096 max context
llama-server -m model.gguf -c 16384 -np 4

Enable speculative decoding

1
2

# the draft.gguf model should be a small variant of the target model.gguf
llama-server -m model.gguf -md draft.gguf

Serve an embedding model

1
2

# use the /embedding endpoint
llama-server -m model.gguf --embedding --pooling cls -ub 8192

Serve a reranking model

1
2

# use the /reranking endpoint
llama-server -m model.gguf --reranking

Constrain all outputs with a grammar

# custom grammar
llama-server -m model.gguf --grammar-file grammar.gbnf

# JSON
llama-server -m model.gguf --grammar-file grammars/json.gbnf

`llama-perplexity`

A tool for measuring the perplexity ¹ (and other quality metrics) of a model over a given text.

Measure the perplexity over a text file

llama-perplexity -m model.gguf -f file.txt

# [1]15.2701,[2]5.4007,[3]5.3073,[4]6.2965,[5]5.8940,[6]5.6096,[7]5.7942,[8]4.9297, ...
# Final estimate: PPL = 5.4007 +/- 0.67339

Measure KL divergence
1

# TODO

`llama-bench`

Benchmark the performance of the inference for various parameters.

Run default benchmark

llama-bench -m model.gguf

# Output:
# | model               |       size |     params | backend    | threads |          test |                  t/s |
# | ------------------- | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |
# | qwen2 1.5B Q4_0     | 885.97 MiB |     1.54 B | Metal,BLAS |      16 |         pp512 |      5765.41 ± 20.55 |
# | qwen2 1.5B Q4_0     | 885.97 MiB |     1.54 B | Metal,BLAS |      16 |         tg128 |        197.71 ± 0.81 |
#
# build: 3e0ba0e60 (4229)

`llama-run`

A comprehensive example for running `llama.cpp` models. Useful for inferencing. Used with RamaLama ².

Run a model with a specific prompt (by default it's pulled from Ollama registry)
1

llama-run granite-code

`llama-simple`

A minimal example for implementing apps with `llama.cpp`. Useful for developers.

Basic text completion

1
2
3

llama-simple -m model.gguf

# Hello my name is Kaitlyn and I am a 16 year old girl. I am a junior in high school and I am currently taking a class called "The Art of

Contributing

Contributors can open PRs
Collaborators will be invited based on contributions
Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
Any help with managing issues, PRs and projects is very appreciated!
See good first issues for tasks suitable for first contributions
Read the CONTRIBUTING.md for more information
Make sure to read this: Inference at the edge
A bit of backstory for those who are interested: Changelog podcast

XCFramework

The XCFramework is a precompiled version of the library for iOS, visionOS, tvOS, and macOS. It can be used in Swift projects without the need to compile the library from source. For example:

// swift-tools-version: 5.10
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
    name: "MyLlamaPackage",
    targets: [
        .executableTarget(
            name: "MyLlamaPackage",
            dependencies: [
                "LlamaFramework"
            ]),
        .binaryTarget(
            name: "LlamaFramework",
            url: "https://github.com/ggml-org/llama.cpp/releases/download/b5046/llama-b5046-xcframework.zip",
            checksum: "c19be78b5f00d8d29a25da41042cb7afa094cbf6280a225abe614b03b20029ab"
        )
    ]
)

The above example is using an intermediate build b5046 of the library. This can be modified to use a different version by changing the URL and checksum.

Completions

Command-line completion is available for some environments.

Bash Completion

1
2

$ build/bin/llama-cli --completion-bash > ~/.llama-completion.bash
$ source ~/.llama-completion.bash

Optionally this can be added to your .bashrc or .bash_profile to load it automatically. For example:

`1`	`$ echo "source ~/.llama-completion.bash" >> ~/.bashrc`

Dependencies

yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
stb-image - Single-header image format decoder, used by multimodal subsystem - Public domain
nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
minja - Minimal Jinja parser in C++, used by various tools/examples - MIT License
linenoise.cpp - C++ library that provides readline-like line editing capabilities, used by llama-run - BSD 2-Clause License
curl - Client-side URL transfer library, used by various tools/examples - CURL License
miniaudio.h - Single-header audio format decoder, used by multimodal subsystem - Public domain

generative-ai-for-beginners

Tue, 09 Sep 2025 15:28:49 +0800

microsoft/generative-ai-for-beginners

21 Lessons teaching everything you need to know to start building Generative AI applications

🌐 Multi-Language Support

Supported via GitHub Action (Automated & Always Up-to-Date)

Generative AI for Beginners (Version 3) - A Course

Learn the fundamentals of building Generative AI applications with our 21-lesson comprehensive course by Microsoft Cloud Advocates.

🌱 Getting Started

This course has 21 lessons. Each lesson covers its own topic so start wherever you like!

Lessons are labeled either “Learn” lessons explaining a Generative AI concept or “Build” lessons that explain a concept and code examples in both Python and TypeScript when possible.

For .NET Developers checkout Generative AI for Beginners (.NET Edition)!

Each lesson also includes a “Keep Learning” section with additional learning tools.

What You Need

To run the code of this course, you can use either:

Azure OpenAI Service - Lessons: “aoai-assignment”
GitHub Marketplace Model Catalog - Lessons: “githubmodels”
OpenAI API - Lessons: “oai-assignment”
Basic knowledge of Python or TypeScript is helpful - *For absolute beginners check out these Python and TypeScript courses
A GitHub account to fork this entire repo to your own GitHub account

We have created a Course Setup lesson to help you with setting up your development environment.

Don’t forget to star (🌟) this repo to find it easier later.

🧠 Ready to Deploy?

If you are looking for more advanced code samples, check out our collection of Generative AI Code Samples in both Python and TypeScript.

🗣️ Meet Other Learners, Get Support

Join our official Azure AI Foundry Discord server to meet and network with other learners taking this course and get support.

Ask questions or share product feedback in our Azure AI Foundry Developer Forum on Github.

🚀 Building a Startup?

Visit Microsoft for Startups to find out how to get started building with Azure credits today.

🙏 Want to help?

Do you have suggestions or found spelling or code errors? Raise an issue or Create a pull request

📂 Each lesson includes:

A short video introduction to the topic
A written lesson located in the README
Python and TypeScript code samples supporting Azure OpenAI and OpenAI API
Links to extra resources to continue your learning

🗃️ Lessons

#	Lesson Link	Description	Video	Extra Learning
00	Course Setup	Learn: How to Setup Your Development Environment	Video Coming Soon	Learn More
01	Introduction to Generative AI and LLMs	Learn: Understanding what Generative AI is and how Large Language Models (LLMs) work.	Video	Learn More
02	Exploring and comparing different LLMs	Learn: How to select the right model for your use case	Video	Learn More
03	Using Generative AI Responsibly	Learn: How to build Generative AI Applications responsibly	Video	Learn More
04	Understanding Prompt Engineering Fundamentals	Learn: Hands-on Prompt Engineering Best Practices	Video	Learn More
05	Creating Advanced Prompts	Learn: How to apply prompt engineering techniques that improve the outcome of your prompts.	Video	Learn More
06	Building Text Generation Applications	Build: A text generation app using Azure OpenAI / OpenAI API	Video	Learn More
07	Building Chat Applications	Build: Techniques for efficiently building and integrating chat applications.	Video	Learn More
08	Building Search Apps Vector Databases	Build: A search application that uses Embeddings to search for data.	Video	Learn More
09	Building Image Generation Applications	Build: An image generation application	Video	Learn More
10	Building Low Code AI Applications	Build: A Generative AI application using Low Code tools	Video	Learn More
11	Integrating External Applications with Function Calling	Build: What is function calling and its use cases for applications	Video	Learn More
12	Designing UX for AI Applications	Learn: How to apply UX design principles when developing Generative AI Applications	Video	Learn More
13	Securing Your Generative AI Applications	Learn: The threats and risks to AI systems and methods to secure these systems.	Video	Learn More
14	The Generative AI Application Lifecycle	Learn: The tools and metrics to manage the LLM Lifecycle and LLMOps	Video	Learn More
15	Retrieval Augmented Generation (RAG) and Vector Databases	Build: An application using a RAG Framework to retrieve embeddings from a Vector Databases	Video	Learn More
16	Open Source Models and Hugging Face	Build: An application using open source models available on Hugging Face	Video	Learn More
17	AI Agents	Build: An application using an AI Agent Framework	Video	Learn More
18	Fine-Tuning LLMs	Learn: The what, why and how of fine-tuning LLMs	Video	Learn More
19	Building with SLMs	Learn: The benefits of building with Small Language Models	Video Coming Soon	Learn More
20	Building with Mistral Models	Learn: The features and differences of the Mistral Family Models	Video Coming Soon	Learn More
21	Building with Meta Models	Learn: The features and differences of the Meta Family Models	Video Coming Soon	Learn More

🌟 Special thanks

Special thanks to John Aziz for creating all of the GitHub Actions and workflows

Bernhard Merkle for making key contributions to each lesson to improve the learner and code experience.

🎒 Other Courses

Our team produces other courses! Check out:

KrillinAI

Wed, 16 Apr 2025 15:29:17 +0800

krillinai/KrillinAI

AI Audio&Video Translation and Dubbing Tool

English｜简体中文｜日本語｜한국어｜Français｜Deutsch｜Español｜Português｜Русский｜اللغة العربية

📢 New Release for Win & Mac Desktop Version – Welcome to Test and Provide Feedback

Overview

Krillin AI is an all-in-one solution for effortless video localization and enhancement. This minimalist yet powerful tool handles everything from translation, dubbing to voice cloning，formatting—seamlessly converting videos between landscape and portrait modes for optimal display across all content platforms(YouTube, TikTok, Bilibili, Douyin, WeChat Channel, RedNote, Kuaishou). With its end-to-end workflow, Krillin AI transforms raw footage into polished, platform-ready content in just a few clicks.

Key Features:

🎯 One-Click Start - Launch your workflow instantly,New desktop version available—easier to use!

📥 Video download - yt-dlp and local file uploading supported

📜 Precise Subtitles - Whisper-powered high-accuracy recognition

🧠 Smart Segmentation - LLM-based subtitle chunking & alignment

🌍 Professional Translation - Paragraph-level translation for consistency

🔄 Term Replacement - One-click domain-specific vocabulary swap

🎙️ Dubbing and Voice Cloning - CosyVoice selected or cloning voices

🎬 Video Composition - Auto-formatting for horizontal/vertical layouts

Showcase

The following picture demonstrates the effect after the subtitle file, which was generated through a one-click operation after importing a 46-minute local video, was inserted into the track. There was no manual adjustment involved at all. There are no missing or overlapping subtitles, the sentence segmentation is natural, and the translation quality is also quite high.

🔍 Speech Recognition Support

All local models in the table below support automatic installation of executable files + model files. Just make your selection, and KrillinAI will handle everything else for you.

Service	Supported Platforms	Model Options	Local/Cloud	Notes
OpenAI Whisper	Cross-platform	-	Cloud	Fast with excellent results
FasterWhisper	Windows/Linux	`tiny`/`medium`/`large-v2` (recommend medium+)	Local	Faster speed, no cloud service overhead
WhisperKit	macOS (Apple Silicon only)	`large-v2`	Local	Native optimization for Apple chips
Alibaba Cloud ASR	Cross-platform	-	Cloud	Bypasses China mainland network issues

🚀 Large Language Model Support

✅ Compatible with all OpenAI API-compatible cloud/local LLM services including but not limited to:

OpenAI
DeepSeek
Qwen (Tongyi Qianwen)
Self-hosted open-source models
Other OpenAI-format compatible API services

🌍 Language Support

Input languages: Chinese, English, Japanese, German, Turkish supported (more languages being added)
Translation languages: 56 languages supported, including English, Chinese, Russian, Spanish, French, etc.

Interface Preview

🚀 Quick Start

Basic Steps

First, download the Release executable file that matches your device’s system. Follow the instructions below to choose between the desktop or non-desktop version, then place the software in an empty folder. Running the program will generate some directories, so keeping it in an empty folder makes management easier.

[For the desktop version (release files with “desktop” in the name), refer here]
The desktop version is newly released to address the difficulty beginners face in editing configuration files correctly. It still has some bugs and is being continuously updated.

Double-click the file to start using it.

[For the non-desktop version (release files without “desktop” in the name), refer here]
The non-desktop version is the original release, with more complex configuration but stable functionality. It is also suitable for server deployment, as it provides a web-based UI.

Create a config folder in the directory, then create a config.toml file inside it. Copy the contents of the config-example.toml file from the source code’s config directory into your config.toml and fill in your configuration details. (If you want to use OpenAI models but don’t know how to get a key, you can join the group for free trial access.)

Double-click the executable or run it in the terminal to start the service.

Open your browser and enter http://127.0.0.1:8888 to begin using it. (Replace 8888 with the port number you specified in the config file.)

To: macOS Users

[For the desktop version, i.e., release files with “desktop” in the name, refer here]
The current packaging method for the desktop version cannot support direct double-click execution or DMG installation due to signing issues. Manual trust configuration is required as follows:

Open the directory containing the executable file (assuming the filename is KrillinAI_1.0.0_desktop_macOS_arm64) in Terminal
Execute the following commands sequentially:

1
2
3

sudo xattr -cr ./KrillinAI_1.0.0_desktop_macOS_arm64  
sudo chmod +x ./KrillinAI_1.0.0_desktop_macOS_arm64  
./KrillinAI_1.0.0_desktop_macOS_arm64

[For the non-desktop version, i.e., release files without “desktop” in the name, refer here]
This software is not signed, so after completing the file configuration in the “Basic Steps,” you will need to manually trust the application on macOS. Follow these steps:

Open the terminal and navigate to the directory where the executable file (assuming the file name is KrillinAI_1.0.0_macOS_arm64) is located.
Execute the following commands in sequence:

1
2
3

sudo xattr -rd com.apple.quarantine ./KrillinAI_1.0.0_macOS_arm64
sudo chmod +x ./KrillinAI_1.0.0_macOS_arm64
./KrillinAI_1.0.0_macOS_arm64

This will start the service.

Docker Deployment

This project supports Docker deployment. Please refer to the Docker Deployment Instructions.

If you encounter video download failures, please refer to the Cookie Configuration Instructions to configure your cookie information.

Configuration Help

The quickest and most convenient configuration method:

Select openai for both transcription_provider and llm_provider. In this way, you only need to fill in openai.apikey in the following three major configuration item categories, namely openai, local_model, and aliyun, and then you can conduct subtitle translation. (Fill in app.proxy, model and openai.base_url as per your own situation.)

The configuration method for using the local speech recognition model (macOS is not supported for the time being) (a choice that takes into account cost, speed, and quality):

Fill in fasterwhisper for transcription_provider and openai for llm_provider. In this way, you only need to fill in openai.apikey and local_model.faster_whisper in the following three major configuration item categories, namely openai and local_model, and then you can conduct subtitle translation. The local model will be downloaded automatically. (The same applies to app.proxy and openai.base_url as mentioned above.)

The following usage situations require the configuration of Alibaba Cloud:

If llm_provider is filled with aliyun, it indicates that the large model service of Alibaba Cloud will be used. Consequently, the configuration of the aliyun.bailian item needs to be set up.
If transcription_provider is filled with aliyun, or if the “voice dubbing” function is enabled when starting a task, the voice service of Alibaba Cloud will be utilized. Therefore, the configuration of the aliyun.speech item needs to be filled in.
If the “voice dubbing” function is enabled and local audio files are uploaded for voice timbre cloning at the same time, the OSS cloud storage service of Alibaba Cloud will also be used. Hence, the configuration of the aliyun.oss item needs to be filled in. Configuration Guide: Alibaba Cloud Configuration Instructions

Frequently Asked Questions

Please refer to Frequently Asked Questions

Contribution Guidelines

Do not submit unnecessary files like .vscode, .idea, etc. Please make good use of .gitignore to filter them.
Do not submit config.toml; instead, submit config-example.toml.

OpenAI API on Producthunt daily

llama.cpp

llama.cpp

Recent API changes

Hot topics

Quick start

Description

Text-only

Multimodal

Supported backends

Obtaining and quantizing models

A CLI tool for accessing and experimenting with most of llama.cpp’s functionality.

A lightweight, OpenAI API compatible, HTTP server for serving LLMs.

A tool for measuring the perplexity 1 (and other quality metrics) of a model over a given text.

Benchmark the performance of the inference for various parameters.

A comprehensive example for running llama.cpp models. Useful for inferencing. Used with RamaLama 2.

A minimal example for implementing apps with llama.cpp. Useful for developers.

Contributing

Other documentation

Development documentation

Seminal papers and background on the models

XCFramework

Completions

Bash Completion

Dependencies

generative-ai-for-beginners

21 Lessons teaching everything you need to know to start building Generative AI applications

🌐 Multi-Language Support

Supported via GitHub Action (Automated & Always Up-to-Date)

Generative AI for Beginners (Version 3) - A Course

🌱 Getting Started

What You Need

To run the code of this course, you can use either:

🧠 Ready to Deploy?

🗣️ Meet Other Learners, Get Support

🚀 Building a Startup?

🙏 Want to help?

📂 Each lesson includes:

🗃️ Lessons

🌟 Special thanks

🎒 Other Courses

KrillinAI

AI Audio&Video Translation and Dubbing Tool

📢 New Release for Win & Mac Desktop Version – Welcome to Test and Provide Feedback

Overview

Key Features:

Showcase

Subtitle Translation

Dubbing

Portrait

🔍 Speech Recognition Support

🚀 Large Language Model Support

🌍 Language Support

Interface Preview

🚀 Quick Start

Basic Steps

To: macOS Users

Docker Deployment

Cookie Configuration Instructions

Configuration Help

Frequently Asked Questions

Contribution Guidelines

Star History

A CLI tool for accessing and experimenting with most of `llama.cpp`’s functionality.

A tool for measuring the perplexity ¹ (and other quality metrics) of a model over a given text.

A comprehensive example for running `llama.cpp` models. Useful for inferencing. Used with RamaLama ².

A minimal example for implementing apps with `llama.cpp`. Useful for developers.