git404hub

what is chunker fr?

suraniyakunal/chunker — explained in plain English

Analysis updated 2026-05-18

0PythonAudience · developerComplexity · 3/5Setup · moderate

tl;dr

A from-scratch retrieval-augmented generation system that answers questions by searching the author's personal AI engineering notes using Gemini embeddings and a local Chroma vector store.

vibe map

mindmap
  root((Chunker RAG))
    What it does
      Loads markdown notes
      Chunks and embeds text
      Stores in Chroma
      Answers grounded questions
    Tech stack
      Python
      LangChain
      Gemini API
      Chroma
    Pipeline
      loader.py
      splitter.py
      embeddings.py
      generation.py
    Use cases
      Personal notes search
      RAG learning project

Code map

Detail Auto

An interactive map of this repo's files and how they connect — its source is parsed live in your browser. Click Visualize to build it.

filefunction / class

what do people make with this?

VIBE 1

Query a personal collection of markdown notes and get answers grounded only in that content.

VIBE 2

Learn how a retrieval augmented generation pipeline works by reading a small, from-scratch implementation.

VIBE 3

Adapt the pipeline to build a question-answering tool over a different markdown knowledge base.

what's the stack?

PythonLangChainGemini APIChroma

how it stacks up fr

suraniyakunal/chunker0xallam/my-recipe0xhassaan/nn-from-scratch
Stars00
LanguagePythonPythonPython
Last pushed2022-11-22
MaintenanceDormant
Setup difficultymoderatemoderatemoderate
Complexity3/52/54/5
Audiencedevelopergeneraldeveloper

Figures from each repo's GitHub metadata at analysis time.

how do i run it?

Difficulty · moderate time til it works · 30min

Requires a free Google Gemini API key and a separate markdown notes repository to ingest.

License terms are not stated in the README.

in plain english

This project is a small retrieval augmented generation system, often called RAG, built by its author to answer questions using their own personal notes on AI engineering. Rather than searching back through old notes by hand, the author can ask a question and get an answer grounded in what those notes actually say. It was built from scratch rather than using a ready made template, as a way to understand each step of how a RAG pipeline works. The pipeline works in five steps. First it loads every markdown file from a separate notes repository. Then it splits each file into overlapping chunks of text, keeping track of which file and topic each chunk came from. Next it turns each chunk into a numerical embedding using Google's Gemini embedding model. Those chunks and their embeddings are stored in a local Chroma vector database. Finally, when a question is asked, the system finds the most relevant chunks through similarity search and passes them to a Gemini chat model, which is instructed to answer only using that retrieved context, and to say it does not know rather than guess when the notes do not cover the topic. The project is written in Python and uses LangChain for document loading and text splitting, Google's free tier Gemini API for both embeddings and generation, and Chroma as the local vector store. No paid APIs or local model inference are required, which keeps it usable on lower resource hardware. Setup involves cloning the repository, creating a Python virtual environment, installing a handful of LangChain and Chroma packages, and adding a free Gemini API key to a .env file. The README also includes a written log of design decisions the author ran into while building it, such as why re-running ingestion needs to clear the old vector store first to avoid duplicate results.

prompts (copy fr)

prompt 1
Walk me through setting up this RAG project with a free Gemini API key and running the ingest.py script.
prompt 2
Explain how loader.py, splitter.py, embeddings.py, and vectorstore.py fit together in this pipeline.
prompt 3
How would I swap the notes corpus in backend/data/ for my own markdown files and re-run ingestion?
prompt 4
Show me how to turn generation.py into an interactive command line question loop.

Frequently asked questions

what is chunker fr?

A from-scratch retrieval-augmented generation system that answers questions by searching the author's personal AI engineering notes using Gemini embeddings and a local Chroma vector store.

What language is chunker written in?

Mainly Python. The stack also includes Python, LangChain, Gemini API.

What license does chunker use?

License terms are not stated in the README.

How hard is chunker to set up?

Setup difficulty is rated moderate, with roughly 30min to a first successful run.

Who is chunker for?

Mainly developer.

peek the repo → explain another one

This repo across BitVibe Labs

double-check against the repo, no cap.