Below is a complete list of links for all sources and models referenced in the open-source AI architecture stack for your offline LLM + coding agent. This includes:
- Core tools (from the base stack like llama.cpp and OpenDevin).
- Missing layers (MoE models, custom inference, o1-style reasoning, agent orchestration, memory, training, and tool use).
- o1-OSS (as the integrated open-source o1 clone — I’ve selected the most direct and active repo based on relevance: a full model trained for o1-like reasoning chains, using SFT data for CoT activation on LLaMA/Qwen backbones. It’s non-profit, community-driven, and aligns with the stack’s offline/fine-tuning ethos).
All links are 100% open-source (MIT/Apache-2.0/GPL/etc.), GitHub repos where applicable, and Hugging Face (HF) pages for models (with direct GGUF download links for offline use). I’ve organized them into tables for clarity, with brief notes on role and license.
1. Core Stack (Base Offline LLM + Agent)
| Tool | Role | Link | License | Notes |
|---|---|---|---|---|
| llama.cpp | GGUF inference runtime (CPU/GPU) | GitHub Repo | MIT | Core engine for running models offline. |
| OpenDevin | Autonomous coding agent (shell/file/compile loop) | GitHub Repo | Apache-2.0 | Fork for local API; integrates with Letta for upgrades. |
2. MoE Models (Advanced Core Architecture)
These are open-weight models in GGUF format (download once via aria2c or HF CLI for offline use). Focus on coding/reasoning strength.
| Model | Size | Role | Link | License | Notes |
|---|---|---|---|---|---|
| Mixtral-8x22B-Instruct | 141B (8x22B active) | Best open MoE for reasoning/coding | HF GGUF | Apache-2.0 | Q4_K_M.gguf (~75 GB); native llama.cpp support. |
| DeepSeek-Coder-V2-Lite | 16B active | MoE-like routing for code gen | HF GGUF | MIT | Q5_K_M.gguf (~10 GB); lightweight for testing. |
| Qwen-2.5-MoE-72B | 72B | SOTA MoE for general tasks | HF Page (GGUF via TheBloke soon) | Apache-2.0 | Check for GGUF; ~40 GB quantized. |
3. Custom Inference Engine (Beyond llama.cpp)
| Tool | Role | Link | License | Notes |
|---|---|---|---|---|
| vLLM | PagedAttention, tensor-parallel (OpenAI-compatible API) | GitHub Repo | Apache-2.0 | Offline install: pip install --no-index vllm==0.6.0. |
| ExLlamaV2 | Fastest GPTQ/AWQ inference | GitHub Repo | GPL-3.0 | For consumer GPUs; prebuilt wheels available. |
| Triton | Custom CUDA kernels | GitHub Repo | MIT | Low-level optimizations; pip install .. |
4. o1-Style Reasoning (Chain-of-Thought + Reflection)
| Tool | Role | Link | License | Notes |
|---|---|---|---|---|
| Letta | Stateful agents, memory, reflection (formerly MemGPT core) | GitHub Repo | Apache-2.0 | Internal CoT; integrates with OpenDevin. |
| DSPy | Programmable reasoning chains | GitHub Repo | MIT | Optimizes prompts/modules for o1-like flows. |
| o1-OSS (Open O1) | Full open-source o1 clone (SFT-trained for CoT reasoning) | GitHub Repo Project Site HF Models (LLaMA/Qwen variants) | Apache-2.0 | Trained on o1-like data; clone repo and fine-tune offline. Matches o1 on math/code; non-profit. |
5. Agent Orchestration (Upgraded OpenDevin)
| Tool | Role | Link | License | Notes |
|---|---|---|---|---|
| Letta (see above) | Reflection + planning patch for OpenDevin | GitHub Repo | Apache-2.0 | Patch via sed as in blueprint. |
| MemGPT | Infinite context paging (now part of Letta) | GitHub Repo (core) Original MemGPT | Apache-2.0 | Pages to disk for long projects. |
6. Memory & Long-Term State (Offline)
| Tool | Role | Link | License | Notes |
|---|---|---|---|---|
| MemGPT | Infinite context via paging | GitHub Repo (legacy) Letta Integration | Apache-2.0 | Use with local LLM endpoint. |
| ChromaDB | Local vector DB for RAG | GitHub Repo | Apache-2.0 | Offline: pip install --no-index chromadb. |
| SQLite | Structured memory (built-in) | Official Site | Public Domain | No repo needed; use via Python stdlib. |
7. Training / Fine-Tuning Loop (Offline)
| Tool | Role | Link | License | Notes |
|---|---|---|---|---|
| Axolotl | LoRA/QLoRA/DPO fine-tuning | GitHub Repo | Apache-2.0 | Multimodal support; YAML configs for offline. |
| Unsloth | 2x faster fine-tuning (low VRAM) | GitHub Repo | Apache-2.0 | Colab/Kaggle friendly; integrates with Axolotl. |
| TRL | RLHF/DPO (Hugging Face) | GitHub Repo | Apache-2.0 | Offline: Download via HF mirror. |
8. Tool Use & Function Calling
| Tool | Role | Link | License | Notes |
|---|---|---|---|---|
| Outlines | JSON schema enforcement | GitHub Repo | MIT | Guarantees structured outputs during gen. |
| Guidance | Constrained generation (regex/CFG) | GitHub Repo | MIT | Fast-forward tokens; works with llama.cpp. |
Quick Download/Setup Tips
- Models: Use
huggingface-cli download TheBloke/Mixtral-8x22B-Instruct-v0.1-GGUF mixtral-8x22b-instruct-v0.1.Q4_K_M.gguf(mirror offline after first run). - All Repos:
git clone <link>thenpip install -e . --no-index(pre-download wheels). - o1-OSS Specific: Start with
git clone https://github.com/Open-Source-O1/Open-O1.git; fine-tune on your data using Axolotl/Unsloth for custom CoT. - Total Size: ~200 GB for full stack (models dominate); use torrent/aria2c for speed.
This completes the architecture — everything is now linked, open, and offline-ready. If you need a unified setup script or fine-tuning dataset for o1-OSS, just say the word.
Build your private OpenAI equivalent today. 🚀
Leave a Reply