Running a large language model (LLM) locally without a GPU

Running a large language model (LLM) locally without a GPU is entirely possible, though it will be slower than on a GPU‑accelerated machine. Below is a practical guide that covers the main steps, from choosing a model to setting up the environment and running inference.

1. Pick a CPU‑friendly model

Large models like GPT‑3 or Llama‑70B are too heavy for a CPU‑only setup. Instead, choose a smaller, open‑source model that still offers decent performance:

Model	Size	Typical CPU RAM	Notes
Llama‑2‑7B	7 B	16–32 GB	Good balance of speed and quality.
Llama‑2‑13B	13 B	32–48 GB	Slightly larger, still manageable on a decent workstation.
Mistral‑7B	7 B	16–32 GB	Very fast on CPU, great for chat.
Phi‑2	2 B	8–12 GB	Tiny, runs on modest laptops.
TinyLlama	1.1 B	4–8 GB	Extremely lightweight, suitable for quick experiments.

If you have a machine with 32 GB of RAM or more, Llama‑2‑7B or Mistral‑7B are solid choices. For a laptop or a machine with less memory, Phi‑2 or TinyLlama are safer.

2. Install the required software stack

Python
Use Python 3.10 or newer. Create a virtual environment:

   python3 -m venv llm-env
   source llm-env/bin/activate

PyTorch (CPU only)
Install the CPU‑only build:

   pip install torch --index-url https://download.pytorch.org/whl/cpu

Transformers & Accelerate
These libraries provide the model loading and inference utilities:

   pip install transformers accelerate

Optional – SentencePiece
Some models require the SentencePiece tokenizer:

   pip install sentencepiece

3. Download the model weights

You can fetch the weights from Hugging Face’s Model Hub. For example, to download Llama‑2‑7B:

mkdir -p ~/models/llama-2-7b
cd ~/models/llama-2-7b
huggingface-cli download meta-llama/Llama-2-7b --local-dir .

If you prefer a smaller model, replace the model name accordingly.

4. Load and run the model

Create a simple Python script (run_llm.py):

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

def main():
    model_name = "~/models/llama-2-7b"   # Path to the downloaded weights
    device = "cpu"                      # Force CPU usage

    # Load tokenizer and model
    tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)
    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        device_map="cpu",          # Keep everything on CPU
        torch_dtype=torch.float32, # Use float32 for compatibility
    )

    # Create a simple text generation pipeline
    generator = pipeline(
        "text-generation",
        model=model,
        tokenizer=tokenizer,
        device=0 if torch.cuda.is_available() else -1,
        max_new_tokens=200,
    )

    # Prompt
    prompt = "Explain the concept of quantum tunneling in simple terms."
    output = generator(prompt, do_sample=True, temperature=0.7)[0]["generated_text"]
    print(output)

if __name__ == "__main__":
    main()

Run it:

python run_llm.py

You should see the model generate a response. The inference time will be noticeably longer than on a GPU, but it will still complete in a few seconds to a minute, depending on the prompt length and your CPU.

5. Optimizations for CPU inference

Technique	How it helps	How to enable
Mixed precision (FP16)	Reduces memory usage and speeds up computation	`torch_dtype=torch.float16` (works on newer CPUs with AVX‑512)
Quantization (int8)	Cuts memory and speeds up inference	`model = AutoModelForCausalLM.from_pretrained(..., load_in_8bit=True)`
Batching	Processes multiple prompts at once	Use `generator(prompts, batch_size=4)`
Model pruning	Removes redundant weights	Use `transformers` pruning utilities or `bitsandbytes` for 4‑bit models

Quantization is especially useful if you hit memory limits. For example, to run Llama‑2‑7B in 8‑bit:

pip install bitsandbytes
# Then in the script:
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    load_in_8bit=True,
)

6. Running on a laptop or low‑resource machine

If your machine has only 8–12 GB of RAM, consider:

Phi‑2 or TinyLlama (1.1 B) – these fit comfortably.
Use torch_dtype=torch.float16 to reduce memory.
Limit max_new_tokens to keep generation short.

7. Troubleshooting

Symptom	Likely cause	Fix
`RuntimeError: CUDA error: device not found`	Trying to use GPU on a CPU‑only machine	Ensure `device_map="cpu"` and `device=-1` in the pipeline.
`MemoryError`	Model too large for available RAM	Switch to a smaller model or enable 8‑bit quantization.
`OSError: model not found`	Wrong path or missing weights	Verify the path and that the weights are fully downloaded.

8. Summary

Pick a CPU‑friendly model (Llama‑2‑7B, Mistral‑7B, Phi‑2, TinyLlama).
Install Python, PyTorch (CPU), Transformers, and optional SentencePiece.
Download the model weights from Hugging Face.
Load the model with device_map="cpu" and run inference via a pipeline.
Apply optimizations like mixed precision or 8‑bit quantization to squeeze performance.

With these steps you can run a respectable LLM locally on a CPU‑only machine, suitable for experimentation, prototyping, or lightweight deployment. Happy modeling!