Transformer on Producthunt daily

happy-llm

Wed, 15 Oct 2025 15:29:58 +0800

datawhalechina/happy-llm

Happy-LLM

中文 | English

📚 在线阅读地址

📚 从零开始的大语言模型原理与实践教程

深入理解 LLM 核心原理，动手实现你的第一个大模型

🎯 项目介绍

很多小伙伴在看完 Datawhale开源项目： self-llm 开源大模型食用指南后，感觉意犹未尽，想要深入了解大语言模型的原理和训练过程。于是我们（Datawhale）决定推出《Happy-LLM》项目，旨在帮助大家深入理解大语言模型的原理和训练过程。

本项目是一个系统性的 LLM 学习教程，将从 NLP 的基本研究方法出发，根据 LLM 的思路及原理逐层深入，依次为读者剖析 LLM 的架构基础和训练过程。同时，我们会结合目前 LLM 领域最主流的代码框架，演练如何亲手搭建、训练一个 LLM，期以实现授之以鱼，更授之以渔。希望大家能从这本书开始走入 LLM 的浩瀚世界，探索 LLM 的无尽可能。

✨ 你将收获什么？

📚 Datawhale 开源免费 完全免费的学习本项目所有内容
🔍 深入理解 Transformer 架构和注意力机制
📚 掌握预训练语言模型的基本原理
🧠 了解现有大模型的基本结构
🏗️ 动手实现 一个完整的 LLaMA2 模型
⚙️ 掌握训练 从预训练到微调的全流程
🚀 实战应用 RAG、Agent 等前沿技术

📖 内容导航

章节	关键内容	状态
前言	本项目的缘起、背景及读者建议	✅
第一章 NLP 基础概念	什么是 NLP、发展历程、任务分类、文本表示演进	✅
第二章 Transformer 架构	注意力机制、Encoder-Decoder、手把手搭建 Transformer	✅
第三章预训练语言模型	Encoder-only、Encoder-Decoder、Decoder-Only 模型对比	✅
第四章大语言模型	LLM 定义、训练策略、涌现能力分析	✅
第五章动手搭建大模型	实现 LLaMA2、训练 Tokenizer、预训练小型 LLM	✅
第六章大模型训练实践	预训练、有监督微调、LoRA/QLoRA 高效微调	🚧
第七章大模型应用	模型评测、RAG 检索增强、Agent 智能体	✅
Extra Chapter LLM Blog	优秀的大模型学习笔记/Blog ，欢迎大家来 PR ！	🚧

Extra Chapter LLM Blog

大模型都这么厉害了，微调0.6B的小模型有什么意义？ @不要葱姜蒜 2025-7-11
Transformer 整体模块设计解读 @ditingdapeng 2025-7-14
文本数据处理详解 @蔡鋆捷 2025-7-14
Qwen3-“VL”——超小中文多模态模型的“拼接微调”之路 @ShaohonChen 2025-7-30
S1: Thinking Budget with vLLM @kmno4-zx 2025-8-03
CDDRS: 使用细粒度语义信息指导增强的RAG检索方法 @Hongru0306 2025-8-21

如果大家在学习 Happy-LLM 项目或 LLM 相关知识中有自己独到的见解、认知、实践，欢迎大家 PR 在 Extra Chapter LLM Blog 中。请遵守 Extra Chapter LLM Blog 的 PR 规范，我们会视 PR 内容的质量和价值来决定是否合并或补充到 Happy-LLM 正文中来。

模型下载

模型名称	下载地址
Happy-LLM-Chapter5-Base-215M	🤖 ModelScope
Happy-LLM-Chapter5-SFT-215M	🤖 ModelScope

ModelScope 创空间体验地址：🤖 创空间

PDF 版本下载

本 Happy-LLM PDF 教程完全开源免费。为防止各类营销号加水印后贩卖给大模型初学者，我们特地在 PDF 文件中预先添加了不影响阅读的 Datawhale 开源标志水印，敬请谅解～

Happy-LLM PDF : https://github.com/datawhalechina/happy-llm/releases/tag/v1.0.1

💡 如何学习

本项目适合大学生、研究人员、LLM 爱好者。在学习本项目之前，建议具备一定的编程经验，尤其是要对 Python 编程语言有一定的了解。最好具备深度学习的相关知识，并了解 NLP 领域的相关概念和术语，以便更轻松地学习本项目。

本项目分为两部分——基础知识与实战应用。第1章～第4章是基础知识部分，从浅入深介绍 LLM 的基本原理。其中，第1章简单介绍 NLP 的基本任务和发展，为非 NLP 领域研究者提供参考；第2章介绍 LLM 的基本架构——Transformer，包括原理介绍及代码实现，作为 LLM 最重要的理论基础；第3章整体介绍经典的 PLM，包括 Encoder-Only、Encoder-Decoder 和 Decoder-Only 三种架构，也同时介绍了当前一些主流 LLM 的架构和思想；第4章则正式进入 LLM 部分，详细介绍 LLM 的特点、能力和整体训练过程。第5章～第7章是实战应用部分，将逐步带领大家深入 LLM 的底层细节。其中，第5章将带领大家者基于 PyTorch 层亲手搭建一个 LLM，并实现预训练、有监督微调的全流程；第6章将引入目前业界主流的 LLM 训练框架 Transformers，带领学习者基于该框架快速、高效地实现 LLM 训练过程；第7章则将介绍基于 LLM 的各种应用，补全学习者对 LLM 体系的认知，包括 LLM 的评测、检索增强生成（Retrieval-Augmented Generation，RAG）、智能体（Agent）的思想和简单实现。你可以根据个人兴趣和需求，选择性地阅读相关章节。

在阅读本书的过程中，建议你将理论和实际相结合。LLM 是一个快速发展、注重实践的领域，我们建议你多投入实战，复现本书提供的各种代码，同时积极参加 LLM 相关的项目与比赛，真正投入到 LLM 开发的浪潮中。我们鼓励你关注 Datawhale 及其他 LLM 相关开源社区，当遇到问题时，你可以随时在本项目的 issue 区提问。

最后，欢迎每一位读者在学习完本项目后加入到 LLM 开发者的行列。作为国内 AI 开源社区，我们希望充分聚集共创者，一起丰富这个开源 LLM 的世界，打造更多、更全面特色 LLM 的教程。星火点点，汇聚成海。我们希望成为 LLM 与普罗大众的阶梯，以自由、平等的开源精神，拥抱更恢弘而辽阔的 LLM 世界。

🤝 如何贡献

我们欢迎任何形式的贡献！

🐛 报告 Bug - 发现问题请提交 Issue
💡 功能建议 - 有好想法就告诉我们
📝 内容完善 - 帮助改进教程内容
🔧 代码优化 - 提交 Pull Request

🙏 致谢

核心贡献者

宋志学-项目负责人 (Datawhale成员-中国矿业大学(北京))
邹雨衡-项目负责人 (Datawhale成员-对外经济贸易大学)
朱信忠-指导专家（Datawhale首席科学家-浙江师范大学杭州人工智能研究院教授）

Extra-Chapter 贡献者

ditingdapeng（内容贡献者-云原生基础架构工程师）
蔡鋆捷（内容贡献者-福州大学）
ShaohonChen （情感机器实验室研究员-西安电子科技大学在读硕士）
肖鸿儒, 庄健琨 (内容贡献者-同济大学)

特别感谢

感谢 @Sm1les 对本项目的帮助与支持
感谢所有为本项目做出贡献的开发者们 ❤️

Star History

⭐ 如果这个项目对你有帮助，请给我们一个 Star！

关于 Datawhale

扫描二维码关注 Datawhale 公众号，获取更多优质开源内容

📜 开源协议

本作品采用知识共享署名-非商业性使用-相同方式共享 4.0 国际许可协议进行许可。

nn-zero-to-hero

Fri, 29 Aug 2025 15:27:57 +0800

karpathy/nn-zero-to-hero

Neural Networks: Zero to Hero

A course on neural networks that starts all the way at the basics. The course is a series of YouTube videos where we code and train neural networks together. The Jupyter notebooks we build in the videos are then captured here inside the lectures directory. Every lecture also has a set of exercises included in the video description. (This may grow into something more respectable).

Lecture 1: The spelled-out intro to neural networks and backpropagation: building micrograd

Backpropagation and training of neural networks. Assumes basic knowledge of Python and a vague recollection of calculus from high school.

Lecture 2: The spelled-out intro to language modeling: building makemore

We implement a bigram character-level language model, which we will further complexify in followup videos into a modern Transformer language model, like GPT. In this video, the focus is on (1) introducing torch.Tensor and its subtleties and use in efficiently evaluating neural networks and (2) the overall framework of language modeling that includes model training, sampling, and the evaluation of a loss (e.g. the negative log likelihood for classification).

Lecture 3: Building makemore Part 2: MLP

We implement a multilayer perceptron (MLP) character-level language model. In this video we also introduce many basics of machine learning (e.g. model training, learning rate tuning, hyperparameters, evaluation, train/dev/test splits, under/overfitting, etc.).

Lecture 4: Building makemore Part 3: Activations & Gradients, BatchNorm

We dive into some of the internals of MLPs with multiple layers and scrutinize the statistics of the forward pass activations, backward pass gradients, and some of the pitfalls when they are improperly scaled. We also look at the typical diagnostic tools and visualizations you’d want to use to understand the health of your deep network. We learn why training deep neural nets can be fragile and introduce the first modern innovation that made doing so much easier: Batch Normalization. Residual connections and the Adam optimizer remain notable todos for later video.

Lecture 5: Building makemore Part 4: Becoming a Backprop Ninja

We take the 2-layer MLP (with BatchNorm) from the previous video and backpropagate through it manually without using PyTorch autograd’s loss.backward(). That is, we backprop through the cross entropy loss, 2nd linear layer, tanh, batchnorm, 1st linear layer, and the embedding table. Along the way, we get an intuitive understanding about how gradients flow backwards through the compute graph and on the level of efficient Tensors, not just individual scalars like in micrograd. This helps build competence and intuition around how neural nets are optimized and sets you up to more confidently innovate on and debug modern neural networks.

I recommend you work through the exercise yourself but work with it in tandem and whenever you are stuck unpause the video and see me give away the answer. This video is not super intended to be simply watched. The exercise is here as a Google Colab. Good luck :)

Lecture 6: Building makemore Part 5: Building WaveNet

We take the 2-layer MLP from previous video and make it deeper with a tree-like structure, arriving at a convolutional neural network architecture similar to the WaveNet (2016) from DeepMind. In the WaveNet paper, the same hierarchical architecture is implemented more efficiently using causal dilated convolutions (not yet covered). Along the way we get a better sense of torch.nn and what it is and how it works under the hood, and what a typical deep learning development process looks like (a lot of reading of documentation, keeping track of multidimensional tensor shapes, moving between jupyter notebooks and repository code, …).

Lecture 7: Let’s build GPT: from scratch, in code, spelled out.

We build a Generatively Pretrained Transformer (GPT), following the paper “Attention is All You Need” and OpenAI’s GPT-2 / GPT-3. We talk about connections to ChatGPT, which has taken the world by storm. We watch GitHub Copilot, itself a GPT, help us write a GPT (meta :D!) . I recommend people watch the earlier makemore videos to get comfortable with the autoregressive language modeling framework and basics of tensors and PyTorch nn, which we take for granted in this video.

YouTube video lecture. For all other links see the video description.

Lecture 8: Let’s build the GPT Tokenizer

The Tokenizer is a necessary and pervasive component of Large Language Models (LLMs), where it translates between strings and tokens (text chunks). Tokenizers are a completely separate stage of the LLM pipeline: they have their own training sets, training algorithms (Byte Pair Encoding), and after training implement two fundamental functions: encode() from strings to tokens, and decode() back from tokens to strings. In this lecture we build from scratch the Tokenizer used in the GPT series from OpenAI. In the process, we will see that a lot of weird behaviors and problems of LLMs actually trace back to tokenization. We’ll go through a number of these issues, discuss why tokenization is at fault, and why someone out there ideally finds a way to delete this stage entirely.

Ongoing…

License

MIT

Hands-On-Large-Language-Models

Wed, 27 Aug 2025 15:29:45 +0800

HandsOnLLM/Hands-On-Large-Language-Models

# Hands-On Large Language Models

Welcome! In this repository you will find the code for all examples throughout the book Hands-On Large Language Models written by Jay Alammar and Maarten Grootendorst which we playfully dubbed:

"The Illustrated LLM Book"

Through the visually educational nature of this book and with almost 300 custom made figures, learn the practical tools and concepts you need to use Large Language Models today!

The book is available on:

We advise to run all examples through Google Colab for the easiest setup. Google Colab allows you to use a T4 GPU with 16GB of VRAM for free. All examples were mainly built and tested using Google Colab, so it should be the most stable platform. However, any other cloud provider should work.

Chapter	Notebook
Chapter 1: Introduction to Language Models
Chapter 2: Tokens and Embeddings
Chapter 3: Looking Inside Transformer LLMs
Chapter 4: Text Classification
Chapter 5: Text Clustering and Topic Modeling
Chapter 6: Prompt Engineering
Chapter 7: Advanced Text Generation Techniques and Tools
Chapter 8: Semantic Search and Retrieval-Augmented Generation
Chapter 9: Multimodal Large Language Models
Chapter 10: Creating Text Embedding Models
Chapter 11: Fine-tuning Representation Models for Classification
Chapter 12: Fine-tuning Generation Models

[!TIP] You can check the setup folder for a quick-start guide to install all packages locally and you can check the conda folder for a complete guide on how to setup your environment, including conda and PyTorch installation. Note that the depending on your OS, Python version, and dependencies your results might be slightly differ. However, they should this be similar to the examples in the book.

Reviews

“Jay and Maarten have continued their tradition of providing beautifully illustrated and insightful descriptions of complex topics in their new book. Bolstered with working code, timelines, and references to key papers, their book is a valuable resource for anyone looking to understand the main techniques behind how Large Language Models are built.”

Andrew Ng - founder of DeepLearning.AI

“This is an exceptional guide to the world of language models and their practical applications in industry. Its highly-visual coverage of generative, representational, and retrieval applications of language models empowers readers to quickly understand, use, and refine LLMs. Highly recommended!”

Nils Reimers - Director of Machine Learning at Cohere | creator of sentence-transformers

“I can’t think of another book that is more important to read right now. On every single page, I learned something that is critical to success in this era of language models.”

Josh Starmer - StatQuest

“If you’re looking to get up to speed in everything regarding LLMs, look no further! In this wonderful book, Jay and Maarten will take you from zero to expert in the history and latest advances in large language models. With very intuitive explanations, great real-life examples, clear illustrations, and comprehensive code labs, this book lifts the curtain on the complexities of transformer models, tokenizers, semantic search, RAG, and many other cutting-edge technologies. A must read for anyone interested in the latest AI technology!”

Luis Serrano, PhD - Founder and CEO of Serrano Academy

“Hands-On Large Language Models brings clarity and practical examples to cut through the hype of AI. It provides a wealth of great diagrams and visual aids to supplement the clear explanations. The worked examples and code make concrete what other books leave abstract. The book starts with simple introductory beginnings, and steadily builds in scope. By the final chapters, you will be fine-tuning and building your own large language models with confidence.”

Leland McInnes - Researcher at the Tutte Institute for Mathematics and Computing | creator of UMAP and HDBSCAN

Bonus content!

We attempted to put as much information into the book without it being overwhelming. However, even with a 400-page book there is still much to discover!

We continue to create more guides that compliment the book and go more in-depth into new and exciting topics:

A Visual Guide to Mamba	A Visual Guide to Quantization	The Illustrated Stable Diffusion

A Visual Guide to Mixture of Experts	A Visual Guide to Reasoning LLMs	The Illustrated DeepSeek-R1

Citation

Please consider citing the book if you consider it useful for your research:

@book{hands-on-llms-book,
  author       = {Jay Alammar and Maarten Grootendorst},
  title        = {Hands-On Large Language Models},
  publisher    = {O'Reilly},
  year         = {2024},
  isbn         = {978-1098150969},
  url          = {https://www.oreilly.com/library/view/hands-on-large-language/9781098150952/},
  github       = {https://github.com/HandsOnLLM/Hands-On-Large-Language-Models}
}