Skip to main content

AI Tools & Infrastructure

⚡ 1. Overview​

AI development can be done online (cloud-based) or offline (local inference). Below are the best tools, models, and methods categorized by how you want to work — with or without a powerful GPU.

☁️ 2. Online AI Tools​

🚀 General & Cloud Platforms​

ToolPurposeNotes
Google ColabRun Jupyter notebooks onlineGreat for users without GPUs; free tier may have timeouts
n8nNo-code workflow automationLike Zapier but open-source; can integrate AI nodes
Hugging Face SpacesHost, test, and share ML modelsCommunity-driven; easy web UI
TensorFlow HubPre-trained model libraryGreat for deploying ready AI models
Gemini 2.5 FlashFast Google research & analysisOptimized for search + summarization
ChatGPT (GPT-4/4o)Text generation, reasoning, creativityBest for general writing, code, and logic
Claude (Anthropic)Code, documentation, long contextKnown for code understanding and safety
GLM 4.5Reasoning & logic-heavy tasksPerforms strong in multilingual and math reasoning
DeepSeek R1Reasoning + code generationEfficient with logic and Python-heavy workloads
Qwen (Alibaba)Coding and multi-language supportHighly efficient at local fine-tuning

💻 3. Offline LLMs (Run Locally)​

Offline models are ideal when you want full control, privacy, or can’t depend on internet. They use quantized weights (compressed versions of large models) to fit on CPUs and small GPUs.

🧩 Method 1 — Ollama​

Ollama makes it simple to run models locally.

  • Website: https://ollama.ai

  • Command Example:

    ollama run llama3

  • ⚙️ RAM usage: depends on model size — e.g., 3B models need ~6–8GB RAM during load/inference.

  • 🧠 Ollama automatically downloads & manages model binaries.

🧠 Method 2 — LM Studio​

A friendly desktop GUI for running open-source LLMs locally.

  • Website: https://lmstudio.ai
  • Supports chat + API server mode
  • Auto-downloads models in .gguf format
  • Great for beginners who prefer GUI

⚙️ Method 3 — llama.cpp​

A C++ inference engine optimized for CPU and edge devices.

Installation:​

# 1. Clone llama.cpp
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

# 2. Build llama-server
make server

# 3. Run model
./build/bin/llama-server -m models/Llama-3.2-3B-Instruct-Q4_K_M.gguf

⚡ Notes:​

🧮 About Quantization​

Quantization reduces the model’s precision (e.g., from 16-bit to 4-bit) to save space — like running GTA 5 in “low graphics mode” to fit weaker hardware. You lose a bit of accuracy, but gain massive speed and memory efficiency.

📦 About .GGUF Models​

.gguf is an optimized binary format for quantized LLMs — used by Ollama, LM Studio, and llama.cpp. It allows models to be portable, fast to load, and run directly on CPU without GPU.

🧠 4. Understanding Inference

Inference is the process of using a trained model to generate an output.

For an LLM:

Prompt
   ↓
Tokenizer
   ↓
Tokens
   ↓
Model
   ↓
Next-token prediction
   ↓
Generated tokens
   ↓
Decoded text

🔤 Tokens​

A token is a small unit of text.

For example:

"Hello world"

may become something similar to:

["Hello", " world"]

The exact tokenization depends on the model tokenizer.

⚡ Tokens per second​

Tokens/sec measures how quickly a model generates output.

For example:

5 tok/s   → slow
15 tok/s  → usable
30 tok/s  → very responsive

The actual experience depends heavily on:

  • model size
  • quantization
  • CPU
  • GPU
  • RAM bandwidth
  • context length
  • inference backend

⏱️ Latency​

Latency is the delay before the system begins producing output.

Two important measurements are:

Time to First Token
        +
Generation Speed

A system can have excellent tokens/sec but still feel slow if its first-token latency is high.

🔎 5. Vector Search

TurboVec​

Efficient vector search and retrieval for AI/RAG applications.

Useful for:

  • Semantic search
  • RAG systems
  • Embedding retrieval
  • Memory-efficient vector indexing

🎙️ 6. Text-to-Speech

Audio8 TTS Preview 0.6B​

A compact multilingual TTS model with support for voice cloning.

Best for:

  • Local voice assistants
  • Multilingual applications
  • AI voice generation

Kokoro-82M​

A lightweight 82M parameter TTS model designed for efficient local speech generation.

Model:

hexgrad/Kokoro-82M

🐦 7. Colibrì

Colibrì explores running extremely large Mixture-of-Experts (MoE) models on machines that cannot fit the entire model into RAM.

Instead of loading everything at once, required model components can be loaded from storage when needed.

👁️ 8. Depth Estimation

Depth Anything​

A monocular depth-estimation model that predicts depth from a normal RGB image.

RGB Image
    ↓
Depth Anything
    ↓
Depth Map

Useful for:

  • Robotics
  • Obstacle detection
  • 3D reconstruction
  • Scene understanding

depth-anything.cpp​

A lightweight C++ implementation for running Depth Anything-style inference using native/optimized inference.

Useful when you want depth estimation without relying entirely on a Python/PyTorch stack.

🤖 9. AI + Robotics

Local AI becomes especially useful in robotics when different models are combined.

Camera / LiDAR / IMU
        ↓
Perception
        ↓
Depth / SLAM
        ↓
Mapping
        ↓
Path Planning
        ↓
Control

Useful Tools​

ToolPurpose
Depth AnythingDepth estimation
depth-anything.cppLightweight depth inference
SLAMLocalization + mapping
GazeboRobot simulation
Isaac SimNVIDIA robotics simulation
llama.cppLocal AI inference

⚙️ 10. Hardware Guide​

DeviceTypeNotes
Laptop GPU 3050 / 4050 (4–8GB)Consumer-gradeIdeal for most small LLM or Stable Diffusion work
Raspberry Pi 5Edge deviceCan run small quantized models (slow but possible)
Jetson Orin NanoCompact AI boardGreat performance per watt; ideal for robotics + LLMs
CPU-Only Systems(via llama.cpp)Work fine for small quantized models

💡 CPU inference is slower but cheaper; GPU inference is faster but power-hungry.

🧑‍💻 11. Developer Tools & IDEs​

CategoryToolsNotes
Code AssistantsCursor, Windsurf, Replit Ghostwriter, CopilotSmart coding, debugging, AI completions
Wireframing & DiagramEraser.io, Mermaid, presentation.aiVisual design + UML-style documentation
Image & Video AIMidjourney, Runway ML, Sora ,veo3Text-to-image/video creativity tools
Simulation / RoboticsIsaac SimNVIDIA robotics simulation for AI control
If you need...Use
Easy local LLMOllama
GUI for local modelsLM Studio
Maximum inference controlllama.cpp
Self-hosted AI APILocalAI
Vector retrievalTurboVec
Lightweight TTSKokoro-82M
Multilingual TTSAudio8
Depth estimationDepth Anything
Native depth inferencedepth-anything.cpp
Robotics simulationGazebo / Isaac Sim