Knowledge-Graphs on Producthunt daily

mangle

Wed, 03 Sep 2025 15:26:58 +0800

google/mangle

Mangle

Mangle is a programming language for deductive database programming. It is an extension of Datalog, with various extensions like aggregation, function calls and optional type-checking.

Deductive database programming is useful for bringing data from multiple data sources together since it enables us to represent and query that data in a uniform way. It can also be used to model domain knowledge, similar to machine-readable ontology but without being restricted to binary predicates.

Datalog is an expressive declarative language similar to relational calculus (think SQL and relational views). Unlike relational calculus, it also supports recursive rules and program structuring in a straightforward way.

Mangle contains Datalog as a fragment and adds extensions that make its use more practical. Some of the good properties like guaranteed termination are lost when such extensions are used.

The goal of Mangle as an open source project is to convey the concepts in a way that is accessible to developers and lends itself to easy experimentation. This repository contains an implementation of Mangle as a go library that can be easily embedded into applications.

Check out the docs and the GitHub discussions for more information. There is also a Q&A section.

For an example how to use Mangle library in a database-like grpc service, see the separate Mangle demo service repo.

This is not an officially supported Google product.

Examples

Simple Queries

Imagine you were asked to spot software affected by the log4j vulnerability discovered in late 2021. We want to look for projects that contain a Java archive (jar file) of log4j that is not updated to the patched version.

projects_with_vulnerable_log4j(P) :-
  projects(P),
  contains_jar(P, "log4j", Version),
  Version != "2.17.1",
  Version != "2.12.4",
  Version != "2.3.2".

This is a Mangle rule: conceptually, the implementation retrieve all possible values for variables P and Version that make all the subgoals true.

Simple Mangle rules like this correspond to select-project-join relational queries. The same query in SQL would look like this:

1
2
3

SELECT projects.id as P
FROM projects JOIN contains_jar ON projects.id = contains_jar.project_id
WHERE contains_jar.version NOT IN ("2.17.1", "2.12.4", "2.3.2")

Unlike SQL, our Mangle rule projects_with_vulnerable_log4j has a name and can be referenced in other queries.

(If translating non-recursive Datalog into SQL queries sounds interesting, you should check out the Logica open source project.)

Aggregation

In practice, querying is rarely enough and we also need grouping and aggregation.

1
2

count_projects_with_vulnerable_log4j(Num) :-
  projects_with_vulnerable_log4j(P) |> do fn:group_by(), let Num = fn:Count().

Recursive Queries

The example does not specify what contains_jar does. Here is a possible implementation for contains_jar that walks a dependency graph. This shows that Mangle rules can be recursive.

contains_jar(P, Name, Version) :-
  contains_jar_directly(P, Name, Version).

contains_jar(P, Name, Version) :-
  project_depends(P, Q),
  contains_jar(Q, Name, Version).

The two rules correspond to two cases in which a project may “contain” a jar: either directly, or through some dependency.

Knowledge Graphs, Property Graphs

In requirements engineering, one needs to captures real world concepts in a domain model and controlled vocabulary. Description logics use roles to describe how concepts interact, but these relationships are always binary. Mangle can represent binary predicates, but also arbitrary n-ary relations. Moreover it also has support for structured data.

one_or_two_leg_trip(Codes, Start, Destination, Price) :-
  direct_conn(Code, Start, Destination, Price)
  |> let Codes = [Code].

one_or_two_leg_trip(Codes, Start, Destination, Price) :-
  direct_conn(FirstCode, Start, Connecting, FirstLegPrice).
  direct_conn(SecondCode, Connecting, Destination, SecondLegPrice)
  |> let Code = [FirstCode, SecondCode],
     let Price = fn:plus(FirstLegPrice, SecondLegPrice).

  graph LR
    /zurich -->|/code/ZL <br /> 60 CHF| /lausanne
    /zurich -->|/code/ZB <br /> 30 CHF| /bern
    /bern -->|/code/BL <br /> 30 CHF| /lausanne

Building & Testing

Get the dependencies (see go.mod), build the library, run tests:

1
2
3

go get -t ./...
go build ./...
go test ./...

Regenerating the parser sources

If you want to regenerate the parser sources, you need to set up ANTLR first. This requires a Java runtime environment.

1
2
3

wget http://www.antlr.org/download/antlr-4.13.2-complete.jar
alias antlr='java -jar $PWD/antlr-4.13.2-complete.jar'
antlr -Dlanguage=Go -package gen -o ./ parse/gen/Mangle.g4 -visitor

Contributing

The Mangle maintainers welcome external contributions to spec, documentation and this implementation (see CONTRIBUTING.md) and also other implementations. Pull requests will be handled like for tensorflow, to ensure our internal usage and tests will pass.

all-rag-techniques

Mon, 16 Jun 2025 15:31:49 +0800

FareedKhan-dev/all-rag-techniques

All RAG Techniques: A Simpler, Hands-On Approach ✨

This repository takes a clear, hands-on approach to Retrieval-Augmented Generation (RAG), breaking down advanced techniques into straightforward, understandable implementations. Instead of relying on frameworks like LangChain or FAISS, everything here is built using familiar Python libraries openai, numpy, matplotlib, and a few others.

The goal is simple: provide code that is readable, modifiable, and educational. By focusing on the fundamentals, this project helps demystify RAG and makes it easier to understand how it really works.

Update: 📢

(12-May-2025) Added a new notebook on how to handle big data using Knowledge Graphs.
(27-April-2025) Added a new notebook which finds best RAG technique for a given query (Simple RAG + Reranker + Query Rewrite).
(20-Mar-2025) Added a new notebook on RAG with Reinforcement Learning.
(07-Mar-2025) Added 20 RAG techniques to the repository.

🚀 What’s Inside?

This repository contains a collection of Jupyter Notebooks, each focusing on a specific RAG technique. Each notebook provides:

A concise explanation of the technique.
A step-by-step implementation from scratch.
Clear code examples with inline comments.
Evaluations and comparisons to demonstrate the technique’s effectiveness.
Visualization to visualize the results.

Here’s a glimpse of the techniques covered:

Notebook	Description
1. Simple RAG	A basic RAG implementation. A great starting point!
2. Semantic Chunking	Splits text based on semantic similarity for more meaningful chunks.
3. Chunk Size Selector	Explores the impact of different chunk sizes on retrieval performance.
4. Context Enriched RAG	Retrieves neighboring chunks to provide more context.
5. Contextual Chunk Headers	Prepends descriptive headers to each chunk before embedding.
6. Document Augmentation RAG	Generates questions from text chunks to augment the retrieval process.
7. Query Transform	Rewrites, expands, or decomposes queries to improve retrieval. Includes Step-back Prompting and Sub-query Decomposition.
8. Reranker	Re-ranks initially retrieved results using an LLM for better relevance.
9. RSE	Relevant Segment Extraction: Identifies and reconstructs continuous segments of text, preserving context.
10. Contextual Compression	Implements contextual compression to filter and compress retrieved chunks, maximizing relevant information.
11. Feedback Loop RAG	Incorporates user feedback to learn and improve RAG system over time.
12. Adaptive RAG	Dynamically selects the best retrieval strategy based on query type.
13. Self RAG	Implements Self-RAG, dynamically decides when and how to retrieve, evaluates relevance, and assesses support and utility.
14. Proposition Chunking	Breaks down documents into atomic, factual statements for precise retrieval.
15. Multimodel RAG	Combines text and images for retrieval, generating captions for images using LLaVA.
16. Fusion RAG	Combines vector search with keyword-based (BM25) retrieval for improved results.
17. Graph RAG	Organizes knowledge as a graph, enabling traversal of related concepts.
18. Hierarchy RAG	Builds hierarchical indices (summaries + detailed chunks) for efficient retrieval.
19. HyDE RAG	Uses Hypothetical Document Embeddings to improve semantic matching.
20. CRAG	Corrective RAG: Dynamically evaluates retrieval quality and uses web search as a fallback.
21. Rag with RL	Maximize the reward of the RAG model using Reinforcement Learning.
Best RAG Finder	Finds the best RAG technique for a given query using Simple RAG + Reranker + Query Rewrite.
22. Big Data with Knowledge Graphs	Handles large datasets using Knowledge Graphs.

🗂️ Repository Structure

fareedkhan-dev-all-rag-techniques/
├── README.md                          <- You are here!
├── 01_simple_rag.ipynb
├── 02_semantic_chunking.ipynb
├── 03_chunk_size_selector.ipynb
├── 04_context_enriched_rag.ipynb
├── 05_contextual_chunk_headers_rag.ipynb
├── 06_doc_augmentation_rag.ipynb
├── 07_query_transform.ipynb
├── 08_reranker.ipynb
├── 09_rse.ipynb
├── 10_contextual_compression.ipynb
├── 11_feedback_loop_rag.ipynb
├── 12_adaptive_rag.ipynb
├── 13_self_rag.ipynb
├── 14_proposition_chunking.ipynb
├── 15_multimodel_rag.ipynb
├── 16_fusion_rag.ipynb
├── 17_graph_rag.ipynb
├── 18_hierarchy_rag.ipynb
├── 19_HyDE_rag.ipynb
├── 20_crag.ipynb
├── 21_rag_with_rl.ipynb
├── 22_big_data_with_KG.ipynb
├── best_rag_finder.ipynb
├── requirements.txt                   <- Python dependencies
└── data/
    └── val.json                       <- Sample validation data (queries and answers)
    └── AI_Information.pdf             <- A sample PDF document for testing.
    └── attention_is_all_you_need.pdf  <- A sample PDF document for testing (for Multi-Modal RAG).

🛠️ Getting Started

Clone the repository:

1
2

git clone https://github.com/FareedKhan-dev/all-rag-techniques.git
cd all-rag-techniques

Install dependencies:
1

pip install -r requirements.txt

Set up your OpenAI API key:

Obtain an API key from Nebius AI.

Set the API key as an environment variable:

`1`	`export OPENAI_API_KEY='YOUR_NEBIUS_AI_API_KEY'`

`1`	`setx OPENAI_API_KEY "YOUR_NEBIUS_AI_API_KEY" # On Windows`

or, within your Python script/notebook:

1
2

import os
os.environ["OPENAI_API_KEY"] = "YOUR_NEBIUS_AI_API_KEY"

Run the notebooks:

Open any of the Jupyter Notebooks (.ipynb files) using Jupyter Notebook or JupyterLab. Each notebook is self-contained and can be run independently. The notebooks are designed to be executed sequentially within each file.

Note: The data/AI_Information.pdf file provides a sample document for testing. You can replace it with your own PDF. The data/val.json file contains sample queries and ideal answers for evaluation. The ‘attention_is_all_you_need.pdf’ is for testing Multi-Modal RAG Notebook.

💡 Core Concepts

Embeddings: Numerical representations of text that capture semantic meaning. We use Nebius AI’s embedding API and, in many notebooks, also the BAAI/bge-en-icl embedding model.
Vector Store: A simple database to store and search embeddings. We create our own SimpleVectorStore class using NumPy for efficient similarity calculations.
Cosine Similarity: A measure of similarity between two vectors. Higher values indicate greater similarity.
Chunking: Dividing text into smaller, manageable pieces. We explore various chunking strategies.
Retrieval: The process of finding the most relevant text chunks for a given query.
Generation: Using a Large Language Model (LLM) to create a response based on the retrieved context and the user’s query. We use the meta-llama/Llama-3.2-3B-Instruct model via Nebius AI’s API.
Evaluation: Assessing the quality of the RAG system’s responses, often by comparing them to a reference answer or using an LLM to score relevance.

🤝 Contributing

Contributions are welcome!