Image on Producthunt daily

blind_watermark

Sun, 26 Oct 2025 15:25:42 +0800

guofei9987/blind_watermark

Blind watermark based on DWT-DCT-SVD.

Documentation: https://BlindWatermark.github.io/blind_watermark/#/en/
文档： https://BlindWatermark.github.io/blind_watermark/#/zh/
中文 readme README_cn.md
Source code: https://github.com/guofei9987/blind_watermark

install

`1`	`pip install blind-watermark`

For the current developer version:

1
2
3

git clone git@github.com:guofei9987/blind_watermark.git
cd blind_watermark
pip install .

How to use

Use in bash

# embed watermark into image:
blind_watermark --embed --pwd 1234 examples/pic/ori_img.jpeg "watermark text" examples/output/embedded.png
# extract watermark from image:
blind_watermark --extract --pwd 1234 --wm_shape 111 examples/output/embedded.png

Use in Python

Original Image + Watermark = Watermarked Image

+ ‘@guofei9987 开源万岁！’ =

See the codes

Embed watermark:

from blind_watermark import WaterMark

bwm1 = WaterMark(password_img=1, password_wm=1)
bwm1.read_img('pic/ori_img.jpg')
wm = '@guofei9987 开源万岁！'
bwm1.read_wm(wm, mode='str')
bwm1.embed('output/embedded.png')
len_wm = len(bwm1.wm_bit)
print('Put down the length of wm_bit {len_wm}'.format(len_wm=len_wm))

Extract watermark:

1
2
3

bwm1 = WaterMark(password_img=1, password_wm=1)
wm_extract = bwm1.extract('output/embedded.png', wm_shape=len_wm, mode='str')
print(wm_extract)

Output:

@guofei9987 开源万岁！

attacks on Watermarked Image

attack method	image after attack	extracted watermark
Rotate 45 Degrees		‘@guofei9987 开源万岁！’
Random crop		‘@guofei9987 开源万岁！’
Masks		‘@guofei9987 开源万岁！’
Vertical cut		‘@guofei9987 开源万岁！’
Horizontal cut		‘@guofei9987 开源万岁！’
Resize		‘@guofei9987 开源万岁！’
Pepper Noise		‘@guofei9987 开源万岁！’
Brightness 10% Down		‘@guofei9987 开源万岁！’

embed images

embed watermark:

from blind_watermark import WaterMark

bwm1 = WaterMark(password_wm=1, password_img=1)
# read original image
bwm1.read_img('pic/ori_img.jpg')
# read watermark
bwm1.read_wm('pic/watermark.png')
# embed
bwm1.embed('output/embedded.png')

Extract watermark:

1
2
3

bwm1 = WaterMark(password_wm=1, password_img=1)
# notice that wm_shape is necessary
bwm1.extract(filename='output/embedded.png', wm_shape=(128, 128), out_wm_name='output/extracted.png', )

attack method	image after attack	extracted watermark
Rotate 45 Degrees
Random crop
Mask

embed array of bits

See it here

As demo, we embed 6 bytes data:

`1`	`wm = [True, False, True, True, True, False]`

Embed:

from blind_watermark import WaterMark

bwm1 = WaterMark(password_img=1, password_wm=1)
bwm1.read_ori_img('pic/ori_img.jpg')
bwm1.read_wm([True, False, True, True, True, False], mode='bit')
bwm1.embed('output/embedded.png')

Extract:

1
2
3

bwm1 = WaterMark(password_img=1, password_wm=1, wm_shape=6)
wm_extract = bwm1.extract('output/打上水印的图.png', mode='bit')
print(wm_extract)

Notice that wm_shape (shape of watermark) is necessary

The output wm_extract is an array of float. set a threshold such as 0.5.

Concurrency

`1`	`WaterMark(..., processes=None)`

processes number of processes, can be integer. Default None, which means using all processes.

text_blind_watermark (Embed message into text): https://github.com/guofei9987/text_blind_watermark
HideInfo（hide as image, hide as sounds, hide as text）：https://github.com/guofei9987/HideInfo

marker

Sat, 16 Aug 2025 15:27:34 +0800

datalab-to/marker

Marker

Marker converts documents to markdown, JSON, chunks, and HTML quickly and accurately.

Converts PDF, image, PPTX, DOCX, XLSX, HTML, EPUB files in all languages
Formats tables, forms, equations, inline math, links, references, and code blocks
Extracts and saves images
Removes headers/footers/other artifacts
Extensible with your own formatting and logic
Does structured extraction, given a JSON schema (beta)
Optionally boost accuracy with LLMs (and your own prompt)
Works on GPU, CPU, or MPS

Performance

Marker benchmarks favorably compared to cloud services like Llamaparse and Mathpix, as well as other open source tools.

The above results are running single PDF pages serially. Marker is significantly faster when running in batch mode, with a projected throughput of 25 pages/second on an H100.

See below for detailed speed and accuracy benchmarks, and instructions on how to run your own benchmarks.

Hybrid Mode

For the highest accuracy, pass the --use_llm flag to use an LLM alongside marker. This will do things like merge tables across pages, handle inline math, format tables properly, and extract values from forms. It can use any gemini or ollama model. By default, it uses gemini-2.0-flash. See below for details.

Here is a table benchmark comparing marker, gemini flash alone, and marker with use_llm:

As you can see, the use_llm mode offers higher accuracy than marker or gemini alone.

Examples

PDF	File type	Markdown	JSON
Think Python	Textbook	View	View
Switch Transformers	arXiv paper	View	View
Multi-column CNN	arXiv paper	View	View

Commercial usage

I want marker to be as widely accessible as possible, while still funding my development/training costs. Research and personal usage is always okay, but there are some restrictions on commercial usage.

The weights for the models are licensed cc-by-nc-sa-4.0, but I will waive that for any organization under $2M USD in gross revenue in the most recent 12-month period AND under $2M in lifetime VC/angel funding raised. You also must not be competitive with the Datalab API. If you want to remove the GPL license requirements (dual-license) and/or use the weights commercially over the revenue limit, check out the options here.

Hosted API

There’s a hosted API for marker available here:

Supports PDF, image, PPT, PPTX, DOC, DOCX, XLS, XLSX, HTML, EPUB files
1/4th the price of leading cloud-based competitors
Fast - ~15s for a 250 page PDF
Supports LLM mode
High uptime (99.99%)

Community

Discord is where we discuss future development.

Installation

You’ll need python 3.10+ and PyTorch.

Install with:

`1`	`pip install marker-pdf`

If you want to use marker on documents other than PDFs, you will need to install additional dependencies with:

`1`	`pip install marker-pdf[full]`

Usage

First, some configuration:

Your torch device will be automatically detected, but you can override this. For example, TORCH_DEVICE=cuda.
Some PDFs, even digital ones, have bad text in them. Set --force_ocr to force OCR on all lines, or the strip_existing_ocr to keep all digital text, and strip out any existing OCR text.
If you care about inline math, set force_ocr to convert inline math to LaTeX.

Interactive App

I’ve included a streamlit app that lets you interactively try marker with some basic options. Run it with:

1
2

pip install streamlit streamlit-ace
marker_gui

Convert a single file

`1`	`marker_single /path/to/file.pdf`

You can pass in PDFs or images.

Options:

--page_range TEXT: Specify which pages to process. Accepts comma-separated page numbers and ranges. Example: --page_range "0,5-10,20" will process pages 0, 5 through 10, and page 20.
--output_format [markdown|json|html|chunks]: Specify the format for the output results.
--output_dir PATH: Directory where output files will be saved. Defaults to the value specified in settings.OUTPUT_DIR.
--paginate_output: Paginates the output, using \n\n{PAGE_NUMBER} followed by - * 48, then \n\n
--use_llm: Uses an LLM to improve accuracy. You will need to configure the LLM backend - see below.
--force_ocr: Force OCR processing on the entire document, even for pages that might contain extractable text. This will also format inline math properly.
--block_correction_prompt: if LLM mode is active, an optional prompt that will be used to correct the output of marker. This is useful for custom formatting or logic that you want to apply to the output.
--strip_existing_ocr: Remove all existing OCR text in the document and re-OCR with surya.
--redo_inline_math: If you want the absolute highest quality inline math conversion, use this along with --use_llm.
--disable_image_extraction: Don’t extract images from the PDF. If you also specify --use_llm, then images will be replaced with a description.
--debug: Enable debug mode for additional logging and diagnostic information.
--processors TEXT: Override the default processors by providing their full module paths, separated by commas. Example: --processors "module1.processor1,module2.processor2"
--config_json PATH: Path to a JSON configuration file containing additional settings.
config --help: List all available builders, processors, and converters, and their associated configuration. These values can be used to build a JSON configuration file for additional tweaking of marker defaults.
--converter_cls: One of marker.converters.pdf.PdfConverter (default) or marker.converters.table.TableConverter. The PdfConverter will convert the whole PDF, the TableConverter will only extract and convert tables.
--llm_service: Which llm service to use if --use_llm is passed. This defaults to marker.services.gemini.GoogleGeminiService.
--help: see all of the flags that can be passed into marker. (it supports many more options then are listed above)

The list of supported languages for surya OCR is here. If you don’t need OCR, marker can work with any language.

Convert multiple files

`1`	`marker /path/to/input/folder`

marker supports all the same options from marker_single above.
--workers is the number of conversion workers to run simultaneously. This is automatically set by default, but you can increase it to increase throughput, at the cost of more CPU/GPU usage. Marker will use 5GB of VRAM per worker at the peak, and 3.5GB average.

Convert multiple files on multiple GPUs

`1`	`NUM_DEVICES=4 NUM_WORKERS=15 marker_chunk_convert ../pdf_in ../md_out`

NUM_DEVICES is the number of GPUs to use. Should be 2 or greater.
NUM_WORKERS is the number of parallel processes to run on each GPU.

Use from python

See the PdfConverter class at marker/converters/pdf.py function for additional arguments that can be passed.

from marker.converters.pdf import PdfConverter
from marker.models import create_model_dict
from marker.output import text_from_rendered

converter = PdfConverter(
    artifact_dict=create_model_dict(),
)
rendered = converter("FILEPATH")
text, _, images = text_from_rendered(rendered)

rendered will be a pydantic basemodel with different properties depending on the output type requested. With markdown output (default), you’ll have the properties markdown, metadata, and images. For json output, you’ll have children, block_type, and metadata.

Custom configuration

You can pass configuration using the ConfigParser. To see all available options, do marker_single --help.

from marker.converters.pdf import PdfConverter
from marker.models import create_model_dict
from marker.config.parser import ConfigParser

config = {
    "output_format": "json",
    "ADDITIONAL_KEY": "VALUE"
}
config_parser = ConfigParser(config)

converter = PdfConverter(
    config=config_parser.generate_config_dict(),
    artifact_dict=create_model_dict(),
    processor_list=config_parser.get_processors(),
    renderer=config_parser.get_renderer(),
    llm_service=config_parser.get_llm_service()
)
rendered = converter("FILEPATH")

Extract blocks

Each document consists of one or more pages. Pages contain blocks, which can themselves contain other blocks. It’s possible to programmatically manipulate these blocks.

Here’s an example of extracting all forms from a document:

from marker.converters.pdf import PdfConverter
from marker.models import create_model_dict
from marker.schema import BlockTypes

converter = PdfConverter(
    artifact_dict=create_model_dict(),
)
document = converter.build_document("FILEPATH")
forms = document.contained_blocks((BlockTypes.Form,))

Look at the processors for more examples of extracting and manipulating blocks.

Other converters

You can also use other converters that define different conversion pipelines:

Extract tables

The TableConverter will only convert and extract tables:

from marker.converters.table import TableConverter
from marker.models import create_model_dict
from marker.output import text_from_rendered

converter = TableConverter(
    artifact_dict=create_model_dict(),
)
rendered = converter("FILEPATH")
text, _, images = text_from_rendered(rendered)

This takes all the same configuration as the PdfConverter. You can specify the configuration force_layout_block=Table to avoid layout detection and instead assume every page is a table. Set output_format=json to also get cell bounding boxes.

You can also run this via the CLI with

`1`	`marker_single FILENAME --use_llm --force_layout_block Table --converter_cls marker.converters.table.TableConverter --output_format json`

OCR Only

If you only want to run OCR, you can also do that through the OCRConverter. Set --keep_chars to keep individual characters and bounding boxes.

from marker.converters.ocr import OCRConverter
from marker.models import create_model_dict

converter = OCRConverter(
    artifact_dict=create_model_dict(),
)
rendered = converter("FILEPATH")

This takes all the same configuration as the PdfConverter.

You can also run this via the CLI with

`1`	`marker_single FILENAME --converter_cls marker.converters.ocr.OCRConverter`

Structured Extraction (beta)

You can run structured extraction via the ExtractionConverter. This requires an llm service to be setup first (see here for details). You’ll get a JSON output with the extracted values.

from marker.converters.extraction import ExtractionConverter
from marker.models import create_model_dict
from marker.config.parser import ConfigParser
from pydantic import BaseModel

class Links(BaseModel):
    links: list[str]
    
schema = Links.model_json_schema()
config_parser = ConfigParser({
    "page_schema": schema
})

converter = ExtractionConverter(
    artifact_dict=create_model_dict(),
    config=config_parser.generate_config_dict(),
    llm_service=config_parser.get_llm_service(),
)
rendered = converter("FILEPATH")

Rendered will have an original_markdown field. If you pass this back in next time you run the converter, as the existing_markdown config key, you can skip re-parsing the document.

Output Formats

Markdown

Markdown output will include:

image links (images will be saved in the same folder)
formatted tables
embedded LaTeX equations (fenced with $$)
Code is fenced with triple backticks
Superscripts for footnotes

HTML

HTML output is similar to markdown output:

Images are included via img tags
equations are fenced with <math> tags
code is in pre tags

JSON

JSON output will be organized in a tree-like structure, with the leaf nodes being blocks. Examples of leaf nodes are a single list item, a paragraph of text, or an image.

The output will be a list, with each list item representing a page. Each page is considered a block in the internal marker schema. There are different types of blocks to represent different elements.

Pages have the keys:

id - unique id for the block.
block_type - the type of block. The possible block types can be seen in marker/schema/__init__.py. As of this writing, they are [“Line”, “Span”, “FigureGroup”, “TableGroup”, “ListGroup”, “PictureGroup”, “Page”, “Caption”, “Code”, “Figure”, “Footnote”, “Form”, “Equation”, “Handwriting”, “TextInlineMath”, “ListItem”, “PageFooter”, “PageHeader”, “Picture”, “SectionHeader”, “Table”, “Text”, “TableOfContents”, “Document”]
html - the HTML for the page. Note that this will have recursive references to children. The content-ref tags must be replaced with the child content if you want the full html. You can see an example of this at marker/output.py:json_to_html. That function will take in a single block from the json output, and turn it into HTML.
polygon - the 4-corner polygon of the page, in (x1,y1), (x2,y2), (x3, y3), (x4, y4) format. (x1,y1) is the top left, and coordinates go clockwise.
children - the child blocks.

The child blocks have two additional keys:

section_hierarchy - indicates the sections that the block is part of. 1 indicates an h1 tag, 2 an h2, and so on.
images - base64 encoded images. The key will be the block id, and the data will be the encoded image.

Note that child blocks of pages can have their own children as well (a tree structure).

{
      "id": "/page/10/Page/366",
      "block_type": "Page",
      "html": "<content-ref src='/page/10/SectionHeader/0'></content-ref><content-ref src='/page/10/SectionHeader/1'></content-ref><content-ref src='/page/10/Text/2'></content-ref><content-ref src='/page/10/Text/3'></content-ref><content-ref src='/page/10/Figure/4'></content-ref><content-ref src='/page/10/SectionHeader/5'></content-ref><content-ref src='/page/10/SectionHeader/6'></content-ref><content-ref src='/page/10/TextInlineMath/7'></content-ref><content-ref src='/page/10/TextInlineMath/8'></content-ref><content-ref src='/page/10/Table/9'></content-ref><content-ref src='/page/10/SectionHeader/10'></content-ref><content-ref src='/page/10/Text/11'></content-ref>",
      "polygon": [[0.0, 0.0], [612.0, 0.0], [612.0, 792.0], [0.0, 792.0]],
      "children": [
        {
          "id": "/page/10/SectionHeader/0",
          "block_type": "SectionHeader",
          "html": "<h1>Supplementary Material for <i>Subspace Adversarial Training</i> </h1>",
          "polygon": [
            [217.845703125, 80.630859375], [374.73046875, 80.630859375],
            [374.73046875, 107.0],
            [217.845703125, 107.0]
          ],
          "children": null,
          "section_hierarchy": {
            "1": "/page/10/SectionHeader/1"
          },
          "images": {}
        },
        ...
        ]
    }

Chunks

Chunks format is similar to JSON, but flattens everything into a single list instead of a tree. Only the top level blocks from each page show up. It also has the full HTML of each block inside, so you don’t need to crawl the tree to reconstruct it. This enable flexible and easy chunking for RAG.

Metadata

All output formats will return a metadata dictionary, with the following fields:

{
    "table_of_contents": [
      {
        "title": "Introduction",
        "heading_level": 1,
        "page_id": 0,
        "polygon": [...]
      }
    ], // computed PDF table of contents
    "page_stats": [
      {
        "page_id":  0, 
        "text_extraction_method": "pdftext",
        "block_counts": [("Span", 200), ...]
      },
      ...
    ]
}

LLM Services

When running with the --use_llm flag, you have a choice of services you can use:

Gemini - this will use the Gemini developer API by default. You’ll need to pass --gemini_api_key to configuration.
Google Vertex - this will use vertex, which can be more reliable. You’ll need to pass --vertex_project_id. To use it, set --llm_service=marker.services.vertex.GoogleVertexService.
Ollama - this will use local models. You can configure --ollama_base_url and --ollama_model. To use it, set --llm_service=marker.services.ollama.OllamaService.
Claude - this will use the anthropic API. You can configure --claude_api_key, and --claude_model_name. To use it, set --llm_service=marker.services.claude.ClaudeService.
OpenAI - this supports any openai-like endpoint. You can configure --openai_api_key, --openai_model, and --openai_base_url. To use it, set --llm_service=marker.services.openai.OpenAIService.
Azure OpenAI - this uses the Azure OpenAI service. You can configure --azure_endpoint, --azure_api_key, and --deployment_name. To use it, set --llm_service=marker.services.azure_openai.AzureOpenAIService.

These services may have additional optional configuration as well - you can see it by viewing the classes.

Internals

Marker is easy to extend. The core units of marker are:

Providers, at marker/providers. These provide information from a source file, like a PDF.
Builders, at marker/builders. These generate the initial document blocks and fill in text, using info from the providers.
Processors, at marker/processors. These process specific blocks, for example the table formatter is a processor.
Renderers, at marker/renderers. These use the blocks to render output.
Schema, at marker/schema. The classes for all the block types.
Converters, at marker/converters. They run the whole end to end pipeline.

To customize processing behavior, override the processors. To add new output formats, write a new renderer. For additional input formats, write a new provider.

Processors and renderers can be directly passed into the base PDFConverter, so you can specify your own custom processing easily.

API server

There is a very simple API server you can run like this:

1
2

pip install -U uvicorn fastapi python-multipart
marker_server --port 8001

This will start a fastapi server that you can access at localhost:8001. You can go to localhost:8001/docs to see the endpoint options.

You can send requests like this:

import requests
import json

post_data = {
    'filepath': 'FILEPATH',
    # Add other params here
}

requests.post("http://localhost:8001/marker", data=json.dumps(post_data)).json()

Note that this is not a very robust API, and is only intended for small-scale use. If you want to use this server, but want a more robust conversion option, you can use the hosted Datalab API.

Troubleshooting

There are some settings that you may find useful if things aren’t working the way you expect:

If you have issues with accuracy, try setting --use_llm to use an LLM to improve quality. You must set GOOGLE_API_KEY to a Gemini API key for this to work.
Make sure to set force_ocr if you see garbled text - this will re-OCR the document.
TORCH_DEVICE - set this to force marker to use a given torch device for inference.
If you’re getting out of memory errors, decrease worker count. You can also try splitting up long PDFs into multiple files.

Debugging

Pass the debug option to activate debug mode. This will save images of each page with detected layout and text, as well as output a json file with additional bounding box information.

Benchmarks

Overall PDF Conversion

We created a benchmark set by extracting single PDF pages from common crawl. We scored based on a heuristic that aligns text with ground truth text segments, and an LLM as a judge scoring method.

Method	Avg Time	Heuristic Score	LLM Score
marker	2.83837	95.6709	4.23916
llamaparse	23.348	84.2442	3.97619
mathpix	6.36223	86.4281	4.15626
docling	3.69949	86.7073	3.70429

Benchmarks were run on an H100 for markjer and docling - llamaparse and mathpix used their cloud services. We can also look at it by document type:

Document Type	Marker heuristic	Marker LLM	Llamaparse Heuristic	Llamaparse LLM	Mathpix Heuristic	Mathpix LLM	Docling Heuristic	Docling LLM
Scientific paper	96.6737	4.34899	87.1651	3.96421	91.2267	4.46861	92.135	3.72422
Book page	97.1846	4.16168	90.9532	4.07186	93.8886	4.35329	90.0556	3.64671
Other	95.1632	4.25076	81.1385	4.01835	79.6231	4.00306	83.8223	3.76147
Form	88.0147	3.84663	66.3081	3.68712	64.7512	3.33129	68.3857	3.40491
Presentation	95.1562	4.13669	81.2261	4	83.6737	3.95683	84.8405	3.86331
Financial document	95.3697	4.39106	82.5812	4.16111	81.3115	4.05556	86.3882	3.8
Letter	98.4021	4.5	93.4477	4.28125	96.0383	4.45312	92.0952	4.09375
Engineering document	93.9244	4.04412	77.4854	3.72059	80.3319	3.88235	79.6807	3.42647
Legal document	96.689	4.27759	86.9769	3.87584	91.601	4.20805	87.8383	3.65552
Newspaper page	98.8733	4.25806	84.7492	3.90323	96.9963	4.45161	92.6496	3.51613
Magazine page	98.2145	4.38776	87.2902	3.97959	93.5934	4.16327	93.0892	4.02041

Throughput

We benchmarked throughput using a single long PDF.

Method	Time per page	Time per document	VRAM used
marker	0.18	43.42	3.17GB

The projected throughput is 122 pages per second on an H100 - we can run 22 individual processes given the VRAM used.

Table Conversion

Marker can extract tables from PDFs using marker.converters.table.TableConverter. The table extraction performance is measured by comparing the extracted HTML representation of tables against the original HTML representations using the test split of FinTabNet. The HTML representations are compared using a tree edit distance based metric to judge both structure and content. Marker detects and identifies the structure of all tables in a PDF page and achieves these scores:

Method	Avg score	Total tables
marker	0.816	99
marker w/use_llm	0.907	99
gemini	0.829	99

The --use_llm flag can significantly improve table recognition performance, as you can see.

We filter out tables that we cannot align with the ground truth, since fintabnet and our layout model have slightly different detection methods (this results in some tables being split/merged).

Running your own benchmarks

You can benchmark the performance of marker on your machine. Install marker manually with:

1
2

git clone https://github.com/VikParuchuri/marker.git
poetry install

Overall PDF Conversion

Download the benchmark data here and unzip. Then run the overall benchmark like this:

`1`	`python benchmarks/overall.py --methods marker --scores heuristic,llm`

Options:

--use_llm use an llm to improve the marker results.
--max_rows how many rows to process for the benchmark.
--methods can be llamaparse, mathpix, docling, marker. Comma separated.
--scores which scoring functions to use, can be llm, heuristic. Comma separated.

Table Conversion

The processed FinTabNet dataset is hosted here and is automatically downloaded. Run the benchmark with:

`1`	`python benchmarks/table/table.py --max_rows 100`

Options:

--use_llm uses an llm with marker to improve accuracy.
--use_gemini also benchmarks gemini 2.0 flash.

How it works

Marker is a pipeline of deep learning models:

Extract text, OCR if necessary (heuristics, surya)
Detect page layout and find reading order (surya)
Clean and format each block (heuristics, texify, surya)
Optionally use an LLM to improve quality
Combine blocks and postprocess complete text

It only uses models where necessary, which improves speed and accuracy.

Limitations

PDF is a tricky format, so marker will not always work perfectly. Here are some known limitations that are on the roadmap to address:

Very complex layouts, with nested tables and forms, may not work
Forms may not be rendered well

Note: Passing the --use_llm and --force_ocr flags will mostly solve these issues.

buildkit

Sat, 21 Jun 2025 15:28:48 +0800

moby/buildkit

BuildKit

BuildKit is a toolkit for converting source code to build artifacts in an efficient, expressive and repeatable manner.

Key features:

Automatic garbage collection
Extendable frontend formats
Concurrent dependency resolution
Efficient instruction caching
Build cache import/export
Nested build job invocations
Distributable workers
Multiple output formats
Pluggable architecture
Execution without root privileges

Read the proposal from https://github.com/moby/moby/issues/32925

Introductory blog post https://blog.mobyproject.org/introducing-buildkit-17e056cc5317

Join #buildkit channel on Docker Community Slack

[!NOTE] If you are visiting this repo for the usage of BuildKit-only Dockerfile features like RUN --mount=type=(bind|cache|tmpfs|secret|ssh), please refer to the Dockerfile reference.

[!NOTE] docker build uses Buildx and BuildKit by default since Docker Engine 23.0. You don’t need to read this document unless you want to use the full-featured standalone version of BuildKit.

Used by

BuildKit is used by the following projects:

Moby & Docker (DOCKER_BUILDKIT=1 docker build)
img
OpenFaaS Cloud
container build interface
Tekton Pipelines (formerly Knative Build Templates)
the Sanic build tool
vab
Rio
kim
PouchContainer
Docker buildx
Okteto Cloud
Earthly earthfiles
Gitpod
Dagger
envd
Depot
Namespace
Unikraft
DevZero
dacc

Quick start

:information_source: For Kubernetes deployments, see examples/kubernetes.

BuildKit is composed of the buildkitd daemon and the buildctl client. While the buildctl client is available for Linux, macOS, and Windows, the buildkitd daemon is only available for Linux and *Windows currently.

The latest binaries of BuildKit are available here for Linux, macOS, and Windows.

Linux Setup

The buildkitd daemon requires the following components to be installed:

runc or crun
containerd (if you want to use containerd worker)

Starting the buildkitd daemon: You need to run buildkitd as the root user on the host.

`1`	`$ sudo buildkitd`

To run buildkitd as a non-root user, see docs/rootless.md.

The buildkitd daemon supports two worker backends: OCI (runc) and containerd.

By default, the OCI (runc) worker is used. You can set --oci-worker=false --containerd-worker=true to use the containerd worker.

We are open to adding more backends.

To start the buildkitd daemon using systemd socket activation, you can install the buildkit systemd unit files. See Systemd socket activation

The buildkitd daemon listens gRPC API on /run/buildkit/buildkitd.sock by default, but you can also use TCP sockets. See Expose BuildKit as a TCP service.

Windows Setup

See instructions and notes at docs/windows.md.

macOS Setup

Homebrew formula (unofficial) is available for macOS.

`1`	`$ brew install buildkit`

The Homebrew formula does not contain the daemon (buildkitd).

For example, Lima can be used for launching the daemon inside a Linux VM.

1
2
3

brew install lima
limactl start template://buildkit
export BUILDKIT_HOST="unix://$HOME/.lima/buildkit/sock/buildkitd.sock"

Build from source

To build BuildKit from source, see .github/CONTRIBUTING.md.

For a buildctl reference, see this document.

Exploring LLB

BuildKit builds are based on a binary intermediate format called LLB that is used for defining the dependency graph for processes running part of your build. tl;dr: LLB is to Dockerfile what LLVM IR is to C.

Marshaled as Protobuf messages
Concurrently executable
Efficiently cacheable
Vendor-neutral (i.e. non-Dockerfile languages can be easily implemented)

See solver/pb/ops.proto for the format definition, and see ./examples/README.md for example LLB applications.

Currently, the following high-level languages have been implemented for LLB:

Exploring Dockerfiles

Frontends are components that run inside BuildKit and convert any build definition to LLB. There is a special frontend called gateway (gateway.v0) that allows using any image as a frontend.

During development, Dockerfile frontend (dockerfile.v0) is also part of the BuildKit repo. In the future, this will be moved out, and Dockerfiles can be built using an external image.

Building a Dockerfile with `buildctl`

buildctl build \
    --frontend=dockerfile.v0 \
    --local context=. \
    --local dockerfile=.
# or
buildctl build \
    --frontend=dockerfile.v0 \
    --local context=. \
    --local dockerfile=. \
    --opt target=foo \
    --opt build-arg:foo=bar

--local exposes local source files from client to the builder. context and dockerfile are the names Dockerfile frontend looks for build context and Dockerfile location.

If the Dockerfile has a different filename it can be specified with --opt filename=./Dockerfile-alternative.

Building a Dockerfile using external frontend

External versions of the Dockerfile frontend are pushed to https://hub.docker.com/r/docker/dockerfile-upstream and https://hub.docker.com/r/docker/dockerfile and can be used with the gateway frontend. The source for the external frontend is currently located in ./frontend/dockerfile/cmd/dockerfile-frontend but will move out of this repository in the future (#163). For automatic build from master branch of this repository docker/dockerfile-upstream:master or docker/dockerfile-upstream:master-labs image can be used.

buildctl build \
    --frontend gateway.v0 \
    --opt source=docker/dockerfile \
    --local context=. \
    --local dockerfile=.
buildctl build \
    --frontend gateway.v0 \
    --opt source=docker/dockerfile \
    --opt context=https://github.com/moby/moby.git \
    --opt build-arg:APT_MIRROR=cdn-fastly.deb.debian.org

Output

By default, the build result and intermediate cache will only remain internally in BuildKit. An output needs to be specified to retrieve the result.

Image/Registry

`1`	`buildctl build ... --output type=image,name=docker.io/username/image,push=true`

To export the image to multiple registries:

`1`	`buildctl build ... --output type=image,\"name=docker.io/username/image,docker.io/username2/image2\",push=true`

To export the cache embed with the image and pushing them to registry together, type registry is required to import the cache, you should specify --export-cache type=inline and --import-cache type=registry,ref=.... To export the cache to a local directly, you should specify --export-cache type=local. Details in Export cache.

buildctl build ...\
  --output type=image,name=docker.io/username/image,push=true \
  --export-cache type=inline \
  --import-cache type=registry,ref=docker.io/username/image

Keys supported by image output:

name=<value>: specify image name(s)
push=true: push after creating the image
push-by-digest=true: push unnamed image
registry.insecure=true: push to insecure HTTP registry
oci-mediatypes=true: use OCI mediatypes in configuration JSON instead of Docker’s
oci-artifact=false: use OCI artifact format for attestations
unpack=true: unpack image after creation (for use with containerd)
dangling-name-prefix=<value>: name image with prefix@<digest>, used for anonymous images
name-canonical=true: add additional canonical name name@<digest>
compression=<uncompressed|gzip|estargz|zstd>: choose compression type for layers newly created and cached, gzip is default value. estargz should be used with oci-mediatypes=true.
compression-level=<value>: compression level for gzip, estargz (0-9) and zstd (0-22)
rewrite-timestamp=true: rewrite the file timestamps to the SOURCE_DATE_EPOCH value. See docs/build-repro.md for how to specify the SOURCE_DATE_EPOCH value.
force-compression=true: forcefully apply compression option to all layers (including already existing layers)
store=true: store the result images to the worker’s (e.g. containerd) image store as well as ensures that the image has all blobs in the content store (default true). Ignored if the worker doesn’t have image store (e.g. OCI worker).
annotation.<key>=<value>: attach an annotation with the respective key and value to the built image
- Using the extended syntaxes, annotation-<type>.<key>=<value>, annotation[<platform>].<key>=<value> and both combined with annotation-<type>[<platform>].<key>=<value>, allows configuring exactly where to attach the annotation.
- <type> specifies what object to attach to, and can be any of manifest (the default), manifest-descriptor, index and index-descriptor
- <platform> specifies which objects to attach to (by default, all), and is the same key passed into the platform opt, see docs/multi-platform.md.
- See docs/annotations.md for more details.

If credentials are required, buildctl will attempt to read Docker configuration file $DOCKER_CONFIG/config.json. $DOCKER_CONFIG defaults to ~/.docker.

Local directory

The local client will copy the files directly to the client. This is useful if BuildKit is being used for building something else than container images.

`1`	`buildctl build ... --output type=local,dest=path/to/output-dir`

To export specific files use multi-stage builds with a scratch stage and copy the needed files into that stage with COPY --from.

...
FROM scratch as testresult

COPY --from=builder /usr/src/app/testresult.xml .
...

`1`	`buildctl build ... --opt target=testresult --output type=local,dest=path/to/output-dir`

With a multi-platform build, a subfolder matching each target platform will be created in the destination directory:

FROM busybox AS build
ARG TARGETOS
ARG TARGETARCH
RUN mkdir /out && echo foo > /out/hello-$TARGETOS-$TARGETARCH

FROM scratch
COPY --from=build /out /

$ buildctl build \
  --frontend dockerfile.v0 \
  --opt platform=linux/amd64,linux/arm64 \
  --output type=local,dest=./bin/release

$ tree ./bin
./bin/
└── release
    ├── linux_amd64
    │   └── hello-linux-amd64
    └── linux_arm64
        └── hello-linux-arm64

You can set platform-split=false to merge files from all platforms together into same directory:

$ buildctl build \
  --frontend dockerfile.v0 \
  --opt platform=linux/amd64,linux/arm64 \
  --output type=local,dest=./bin/release,platform-split=false

$ tree ./bin
./bin/
└── release
    ├── hello-linux-amd64
    └── hello-linux-arm64

Tar exporter is similar to local exporter but transfers the files through a tarball.

1
2

buildctl build ... --output type=tar,dest=out.tar
buildctl build ... --output type=tar > out.tar

Docker tarball

1
2

# exported tarball is also compatible with OCI spec
buildctl build ... --output type=docker,name=myimage | docker load

OCI tarball

1
2

buildctl build ... --output type=oci,dest=path/to/output.tar
buildctl build ... --output type=oci > output.tar

containerd image store

The containerd worker needs to be used

1
2

buildctl build ... --output type=image,name=docker.io/username/image
ctr --namespace=buildkit images ls

To change the containerd namespace, you need to change worker.containerd.namespace in /etc/buildkit/buildkitd.toml.

Cache

To show local build cache (/var/lib/buildkit):

`1`	`buildctl du -v`

To prune local build cache:

`1`	`buildctl prune`

Garbage collection

See ./docs/buildkitd.toml.md.

Export cache

BuildKit supports the following cache exporters:

inline: embed the cache into the image, and push them to the registry together
registry: push the image and the cache separately
local: export to a local directory
gha: export to GitHub Actions cache

In most case you want to use the inline cache exporter. However, note that the inline cache exporter only supports min cache mode. To enable max cache mode, push the image and the cache separately by using registry cache exporter.

inline and registry exporters both store the cache in the registry. For importing the cache, type=registry is sufficient for both, as specifying the cache format is not necessary.

Inline (push image and cache together)

buildctl build ... \
  --output type=image,name=docker.io/username/image,push=true \
  --export-cache type=inline \
  --import-cache type=registry,ref=docker.io/username/image

Note that the inline cache is not imported unless --import-cache type=registry,ref=... is provided.

Inline cache embeds cache metadata into the image config. The layers in the image will be left untouched compared to the image with no cache information.

:information_source: Docker-integrated BuildKit (DOCKER_BUILDKIT=1 docker build) and docker buildxrequires --build-arg BUILDKIT_INLINE_CACHE=1 to be specified to enable the inline cache exporter. However, the standalone buildctl does NOT require --opt build-arg:BUILDKIT_INLINE_CACHE=1 and the build-arg is simply ignored.

Registry (push image and cache separately)

buildctl build ... \
  --output type=image,name=localhost:5000/myrepo:image,push=true \
  --export-cache type=registry,ref=localhost:5000/myrepo:buildcache \
  --import-cache type=registry,ref=localhost:5000/myrepo:buildcache

--export-cache options:

type=registry
mode=<min|max>: specify cache layers to export (default: min)
- min: only export layers for the resulting image
- max: export all the layers of all intermediate steps
ref=<ref>: specify repository reference to store cache, e.g. docker.io/user/image:tag
image-manifest=<true|false>: whether to export cache manifest as an OCI-compatible image manifest rather than a manifest list/index (default: true since BuildKit v0.21, must be used with oci-mediatypes=true)
oci-mediatypes=<true|false>: whether to use OCI mediatypes in exported manifests (default: true, since BuildKit v0.8)
compression=<uncompressed|gzip|estargz|zstd>: choose compression type for layers newly created and cached, gzip is default value. estargz and zstd should be used with oci-mediatypes=true
compression-level=<value>: choose compression level for gzip, estargz (0-9) and zstd (0-22)
force-compression=true: forcibly apply compression option to all layers
ignore-error=<false|true>: specify if error is ignored in case cache export fails (default: false)

--import-cache options:

type=registry
ref=<ref>: specify repository reference to retrieve cache from, e.g. docker.io/user/image:tag

Local directory

1
2

buildctl build ... --export-cache type=local,dest=path/to/output-dir
buildctl build ... --import-cache type=local,src=path/to/input-dir

The directory layout conforms to OCI Image Spec v1.0.

--export-cache options:

type=local
mode=<min|max>: specify cache layers to export (default: min)
- min: only export layers for the resulting image
- max: export all the layers of all intermediate steps
dest=<path>: destination directory for cache exporter
tag=<tag>: specify custom tag of image to write to local index (default: latest)
image-manifest=<true|false>: whether to export cache manifest as an OCI-compatible image manifest rather than a manifest list/index (default: true since BuildKit v0.21, must be used with oci-mediatypes=true)
oci-mediatypes=<true|false>: whether to use OCI mediatypes in exported manifests (default true, since BuildKit v0.8)
compression=<uncompressed|gzip|estargz|zstd>: choose compression type for layers newly created and cached, gzip is default value. estargz and zstd should be used with oci-mediatypes=true.
compression-level=<value>: compression level for gzip, estargz (0-9) and zstd (0-22)
force-compression=true: forcibly apply compression option to all layers
ignore-error=<false|true>: specify if error is ignored in case cache export fails (default: false)

--import-cache options:

type=local
src=<path>: source directory for cache importer
tag=<tag>: specify custom tag of image to read from local index (default: latest)
digest=sha256:<sha256digest>: specify explicit digest of the manifest list to import

GitHub Actions cache (experimental)

buildctl build ... \
  --output type=image,name=docker.io/username/image,push=true \
  --export-cache type=gha \
  --import-cache type=gha

GitHub Actions cache saves both cache metadata and layers to GitHub’s Cache service. This cache currently has a size limit of 10GB that is shared across different caches in the repo. If you exceed this limit, GitHub will save your cache but will begin evicting caches until the total size is less than 10 GB. Recycling caches too often can result in slower runtimes overall.

Similarly to using actions/cache, caches are scoped by branch, with the default and target branches being available to every branch.

Following attributes are required to authenticate against the GitHub Actions Cache service API:

url: Cache server URL (default $ACTIONS_CACHE_URL or fallback to $ACTIONS_RESULTS_URL)
url_v2: Cache v2 server URL if $ACTIONS_CACHE_SERVICE_V2 set on the runner (default $ACTIONS_RESULTS_URL)
token: Access token (default $ACTIONS_RUNTIME_TOKEN)

:information_source: This type of cache can be used with Docker Build Push Action where url and token will be automatically set. To use this backend in an inline run step, you have to include crazy-max/ghaction-github-runtime in your workflow to expose the runtime.

--export-cache options:

type=gha
mode=<min|max>: specify cache layers to export (default: min)
- min: only export layers for the resulting image
- max: export all the layers of all intermediate steps
scope=<scope>: which scope cache object belongs to (default buildkit)
ignore-error=<false|true>: specify if error is ignored in case cache export fails (default: false)
timeout=<duration>: sets the timeout duration for cache export (default: 10m)

--import-cache options:

type=gha
scope=<scope>: which scope cache object belongs to (default buildkit)
timeout=<duration>: sets the timeout duration for cache import (default: 10m)

S3 cache (experimental)

buildctl build ... \
  --output type=image,name=docker.io/username/image,push=true \
  --export-cache type=s3,region=eu-west-1,bucket=my_bucket,name=my_image \
  --import-cache type=s3,region=eu-west-1,bucket=my_bucket,name=my_image

The following attributes are required:

bucket: AWS S3 bucket (default: $AWS_BUCKET)
region: AWS region (default: $AWS_REGION)

Storage locations:

blobs: s3://<bucket>/<prefix><blobs_prefix>/<sha256>, default: s3://<bucket>/blobs/<sha256>
manifests: s3://<bucket>/<prefix><manifests_prefix>/<name>, default: s3://<bucket>/manifests/<name>

S3 configuration:

blobs_prefix: global prefix to store / read blobs on s3 (default: blobs/)
manifests_prefix: global prefix to store / read manifests on s3 (default: manifests/)
endpoint_url: specify a specific S3 endpoint (default: empty)
use_path_style: if set to true, put the bucket name in the URL instead of in the hostname (default: false)

AWS Authentication:

BuildKit relies on the [AWS Go SDK](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/config#EnvConfig). This means that all standard authentication methods through environment variables or config files are supported. This is especially true for AWS EC2 IAM Profile and AWS Web Identity Token (IAM roles in Kubernetes).

Beware, these configurations must be available at buildkit daemon level, not at client level.

The following attributes can be used to forward static credentials from a buildkit client to the daemon (buildx for example).
- access_key_id: Access Key ID
- secret_access_key: Secret Access Key
- session_token: Session Token

--export-cache options:

type=s3
mode=<min|max>: specify cache layers to export (default: min)
- min: only export layers for the resulting image
- max: export all the layers of all intermediate steps
prefix=<prefix>: set global prefix to store / read files on s3 (default: empty)
name=<manifest>: specify name of the manifest to use (default buildkit)
- Multiple manifest names can be specified at the same time, separated by ;. The standard use case is to use the git sha1 as name, and the branch name as duplicate, and load both with 2 import-cache commands.
ignore-error=<false|true>: specify if error is ignored in case cache export fails (default: false)
touch_refresh=24h: Instead of being uploaded again when not changed, blobs files will be “touched” on s3 every touch_refresh, default is 24h. Due to this, an expiration policy can be set on the S3 bucket to cleanup useless files automatically. Manifests files are systematically rewritten, there is no need to touch them.
upload_parallelism=4: This parameter changes the number of layers uploaded to s3 in parallel. Each individual layer is uploaded with 5 threads, using the Upload manager provided by the AWS SDK.

--import-cache options:

type=s3
prefix=<prefix>: set global prefix to store / read files on s3 (default: empty)
blobs_prefix=<prefix>: set global prefix to store / read blobs on s3 (default: blobs/)
manifests_prefix=<prefix>: set global prefix to store / read manifests on s3 (default: manifests/)
name=<manifest>: name of the manifest to use (default buildkit)

Azure Blob Storage cache (experimental)

buildctl build ... \
  --output type=image,name=docker.io/username/image,push=true \
  --export-cache type=azblob,account_url=https://myaccount.blob.core.windows.net,name=my_image \
  --import-cache type=azblob,account_url=https://myaccount.blob.core.windows.net,name=my_image

The following attributes are required:

account_url: The Azure Blob Storage account URL (default: $BUILDKIT_AZURE_STORAGE_ACCOUNT_URL)

Storage locations:

blobs: <account_url>/<container>/<prefix><blobs_prefix>/<sha256>, default: <account_url>/<container>/blobs/<sha256>
manifests: <account_url>/<container>/<prefix><manifests_prefix>/<name>, default: <account_url>/<container>/manifests/<name>

Azure Blob Storage configuration:

container: The Azure Blob Storage container name (default: buildkit-cache or $BUILDKIT_AZURE_STORAGE_CONTAINER if set)
blobs_prefix: Global prefix to store / read blobs on the Azure Blob Storage container (<container>) (default: blobs/)
manifests_prefix: Global prefix to store / read blobs on the Azure Blob Storage container (<container>) (default: manifests/)

Azure Blob Storage authentication:

There are 2 options supported for Azure Blob Storage authentication:

Any system using environment variables supported by the Azure SDK for Go. The configuration must be available for the buildkit daemon, not for the client.
Secret Access Key, using the secret_access_key attribute to specify the primary or secondary account key for your Azure Blob Storage account. Azure Blob Storage account keys

[!NOTE] Account name can also be specified with account_name attribute (or $BUILDKIT_AZURE_STORAGE_ACCOUNT_NAME) if it is not part of the account URL host.

--export-cache options:

type=azblob
mode=<min|max>: specify cache layers to export (default: min)
- min: only export layers for the resulting image
- max: export all the layers of all intermediate steps
prefix=<prefix>: set global prefix to store / read files on the Azure Blob Storage container (<container>) (default: empty)
name=<manifest>: specify name of the manifest to use (default: buildkit)
- Multiple manifest names can be specified at the same time, separated by ;. The standard use case is to use the git sha1 as name, and the branch name as duplicate, and load both with 2 import-cache commands.
ignore-error=<false|true>: specify if error is ignored in case cache export fails (default: false)

--import-cache options:

type=azblob
prefix=<prefix>: set global prefix to store / read files on the Azure Blob Storage container (<container>) (default: empty)
blobs_prefix=<prefix>: set global prefix to store / read blobs on the Azure Blob Storage container (<container>) (default: blobs/)
manifests_prefix=<prefix>: set global prefix to store / read manifests on the Azure Blob Storage container (<container>) (default: manifests/)
name=<manifest>: name of the manifest to use (default: buildkit)

Consistent hashing

If you have multiple BuildKit daemon instances, but you don’t want to use registry for sharing cache across the cluster, consider client-side load balancing using consistent hashing.

See ./examples/kubernetes/consistenthash.

Metadata

To output build metadata such as the image digest, pass the --metadata-file flag. The metadata will be written as a JSON object to the specified file. The directory of the specified file must already exist and be writable.

`1`	`buildctl build ... --metadata-file metadata.json`

`1`	`jq '.' metadata.json`

{
  "containerimage.config.digest": "sha256:2937f66a9722f7f4a2df583de2f8cb97fc9196059a410e7f00072fc918930e66",
  "containerimage.descriptor": {
    "annotations": {
      "config.digest": "sha256:2937f66a9722f7f4a2df583de2f8cb97fc9196059a410e7f00072fc918930e66",
      "org.opencontainers.image.created": "2022-02-08T21:28:03Z"
    },
    "digest": "sha256:19ffeab6f8bc9293ac2c3fdf94ebe28396254c993aea0b5a542cfb02e0883fa3",
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "size": 506
  },
  "containerimage.digest": "sha256:19ffeab6f8bc9293ac2c3fdf94ebe28396254c993aea0b5a542cfb02e0883fa3"
}

Systemd socket activation

On Systemd based systems, you can communicate with the daemon via Systemd socket activation, use buildkitd --addr fd://. You can find examples of using Systemd socket activation with BuildKit and Systemd in ./examples/systemd.

Expose BuildKit as a TCP service

The buildkitd daemon can listen the gRPC API on a TCP socket.

It is highly recommended to create TLS certificates for both the daemon and the client (mTLS). Enabling TCP without mTLS is dangerous because the executor containers (aka Dockerfile RUN containers) can call BuildKit API as well.

buildkitd \
  --addr tcp://0.0.0.0:1234 \
  --tlscacert /path/to/ca.pem \
  --tlscert /path/to/cert.pem \
  --tlskey /path/to/key.pem

buildctl \
  --addr tcp://example.com:1234 \
  --tlscacert /path/to/ca.pem \
  --tlscert /path/to/clientcert.pem \
  --tlskey /path/to/clientkey.pem \
  build ...

Load balancing

buildctl build can be called against randomly load balanced buildkitd daemons.

See also Consistent hashing for client-side load balancing.

Containerizing BuildKit

BuildKit can also be used by running the buildkitd daemon inside a Docker container and accessing it remotely.

We provide the container images as moby/buildkit:

moby/buildkit:latest: built from the latest regular release
moby/buildkit:rootless: same as latest but runs as an unprivileged user, see docs/rootless.md
moby/buildkit:master: built from the master branch
moby/buildkit:master-rootless: same as master but runs as an unprivileged user, see docs/rootless.md

To run daemon in a container:

1
2
3

docker run -d --name buildkitd --privileged moby/buildkit:latest
export BUILDKIT_HOST=docker-container://buildkitd
buildctl build --help

Podman

To connect to a BuildKit daemon running in a Podman container, use podman-container:// instead of docker-container:// .

1
2

podman run -d --name buildkitd --privileged moby/buildkit:latest
buildctl --addr=podman-container://buildkitd build --frontend dockerfile.v0 --local context=. --local dockerfile=. --output type=oci | podman load foo

sudo is not required.

Nerdctl

To connect to a BuildKit daemon running in a Nerdctl container, use nerdctl-container:// instead of docker-container://.

1
2

nerdctl run -d --name buildkitd --privileged moby/buildkit:latest
buildctl --addr=nerdctl-container://buildkitd build --frontend dockerfile.v0 --local context=. --local dockerfile=. --output type=oci | nerdctl load

sudo is not required.

Kubernetes

For Kubernetes deployments, see examples/kubernetes.

Daemonless

To run the client and an ephemeral daemon in a single container (“daemonless mode”):

docker run \
    -it \
    --rm \
    --privileged \
    -v /path/to/dir:/tmp/work \
    --entrypoint buildctl-daemonless.sh \
    moby/buildkit:master \
        build \
        --frontend dockerfile.v0 \
        --local context=/tmp/work \
        --local dockerfile=/tmp/work

docker run \
    -it \
    --rm \
    --security-opt seccomp=unconfined \
    --security-opt apparmor=unconfined \
    -e BUILDKITD_FLAGS=--oci-worker-no-process-sandbox \
    -v /path/to/dir:/tmp/work \
    --entrypoint buildctl-daemonless.sh \
    moby/buildkit:master-rootless \
        build \
        --frontend \
        dockerfile.v0 \
        --local context=/tmp/work \
        --local dockerfile=/tmp/work

OpenTelemetry support

BuildKit supports OpenTelemetry for buildkitd gRPC API and buildctl commands. To capture the trace to Jaeger, set JAEGER_TRACE environment variable to the collection address.

docker run -d -p6831:6831/udp -p16686:16686 jaegertracing/all-in-one:latest
export JAEGER_TRACE=0.0.0.0:6831
# restart buildkitd and buildctl so they know JAEGER_TRACE
# any buildctl command should be traced to http://127.0.0.1:16686/

On Windows, if you are running Jaeger outside of a container, jaeger-all-in-one.exe, set the environment variable setx -m JAEGER_TRACE "0.0.0.0:6831", restart buildkitd in a new terminal and the traces will be collected automatically.

Running BuildKit without root privileges

Please refer to docs/rootless.md.

Building multi-platform images

Please refer to docs/multi-platform.md.

Configuring `buildctl`

Color Output Controls

buildctl has support for modifying the colors that are used to output information to the terminal. You can set the environment variable BUILDKIT_COLORS to something like run=green:warning=yellow:error=red:cancel=255,165,0 to set the colors that you would like to use. Setting NO_COLOR to anything will disable any colorized output as recommended by no-color.org.

Parsing errors will be reported but ignored. This will result in default color values being used where needed.

The list of pre-defined colors.

Number of log lines (for active steps in tty mode)

You can change how many log lines are visible for active steps in tty mode by setting BUILDKIT_TTY_LOG_LINES to a number (default: 6).

Contributing

Want to contribute to BuildKit? Awesome! You can find information about contributing to this project in the CONTRIBUTING.md

Image on Producthunt daily

blind_watermark

guofei9987/blind_watermark

blind-watermark

install

How to use

Use in bash

Use in Python

attacks on Watermarked Image

embed images

embed array of bits

Concurrency

Related Project

marker

datalab-to/marker

Marker

Performance

Hybrid Mode

Examples

Commercial usage

Hosted API

Community

Installation

Usage

Interactive App

Convert a single file

Convert multiple files

Convert multiple files on multiple GPUs

Use from python

Custom configuration

Extract blocks

Other converters

Extract tables

OCR Only

Structured Extraction (beta)

Output Formats

Markdown

HTML

JSON

Chunks

Metadata

LLM Services

Internals

API server

Troubleshooting

Debugging

Benchmarks

Overall PDF Conversion

Throughput

Table Conversion

Running your own benchmarks

Overall PDF Conversion

Table Conversion

How it works

Limitations

buildkit

moby/buildkit

BuildKit

Used by

Quick start

Linux Setup

Windows Setup

macOS Setup

Build from source

Exploring LLB

Exploring Dockerfiles

Building a Dockerfile with buildctl

Building a Dockerfile using external frontend

Output

Image/Registry

Local directory

Docker tarball

OCI tarball

containerd image store

Cache

Garbage collection

Export cache

Inline (push image and cache together)

Registry (push image and cache separately)

Local directory

Building a Dockerfile with `buildctl`

Configuring `buildctl`