nielsrogge/perception_models — explained in plain English
Analysis updated 2026-07-18 · repo last pushed 2025-04-18
Build an app that automatically tags and categorizes photos based on their visual content.
Create a tool that answers questions about what appears in screenshots or images.
Build a system that searches through video footage by describing what happens in each clip.
Study and improve vision-language models as a research starting point.
| nielsrogge/perception_models | 0xkinno/neuralvault | 0xmayurrr/ai-contractauditor | |
|---|---|---|---|
| Stars | 1 | 1 | 1 |
| Language | — | TypeScript | TypeScript |
| Last pushed | 2025-04-18 | — | — |
| Maintenance | Stale | — | — |
| Setup difficulty | moderate | hard | easy |
| Complexity | 4/5 | 4/5 | 2/5 |
| Audience | researcher | developer | developer |
Figures from each repo's GitHub metadata at analysis time.
Requires a specific version of PyTorch plus video decoding libraries, so you need comfort configuring Python environments.
Perception Models is a repository from Meta AI Research that lets you build and experiment with AI systems that understand images and video. It provides two main tools: Perception Encoder (PE), which turns visual content into a format computers can reason about, and Perception Language Model (PLM), which can describe and answer questions about what it sees in images or video. The practical benefit is that you get access to state-of-the-art vision AI without needing to train models from scratch. At a high level, PE is a vision encoder, a model that looks at an image and produces a compact mathematical representation of it, which can then be used for tasks like classification or matching images to text. PLM is a vision-language model, meaning it can both see an image and generate text about it, similar to how ChatGPT can answer questions but with visual input. The repository provides code for training, running inference, and evaluating these models, and it links to downloadable pre-trained versions hosted on Hugging Face. Someone building a product that needs to understand visual content would use this. For example, if you are building an app that automatically tags photos, a tool that answers questions about screenshots, or a system that searches through video footage by describing what happens in each clip, these models give you the underlying perception capability. Researchers who want to study or improve vision-language models can also use it as a starting point since it is designed to be modular and easy to expand. A notable tradeoff is the licensing difference between the two models. PE ships under the Apache 2.0 license, which is permissive and allows broad commercial use. PLM uses a more restrictive FAIR Research License, which may limit commercial applications. The codebase also requires specific dependencies like a particular version of PyTorch and video decoding libraries, so setting it up involves some familiarity with Python environments. The README points to separate documentation files for deeper details on each model's usage.
Meta AI's open-source toolkit for AI that understands images and video, including a model that converts visuals into data and a model that describes or answers questions about what it sees.
Stale — no commits in 1-2 years (last push 2025-04-18).
The image encoder is Apache 2.0 allowing broad commercial use, while the vision-language model uses a more restrictive FAIR Research License that may limit commercial applications.
Setup difficulty is rated moderate, with roughly 1h+ to a first successful run.
Mainly researcher.
This repo across BitVibe Labs
double-check against the repo, no cap.