#

consumer-gpu

Here are 38 public repositories matching this topic...

MakazhanAlpamys / Soup

Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.

Updated Aug 8, 2026
Python

Sandermage / sndr_core_engine

SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere sm_86, 2× A5000/3090). Qwen3.6-35B-A3B FP8 ~240 tok/s, 27B-int4 hybrid GDN+Mamba, Gemma4 26B/31B AWQ, 256K ctx. 321 patches: TurboQuant k8v4 KV, MTP/DFlash spec-decode, FULL cudagraph, hybrid GDN. vLLM pin dev424 + Control Center GUI.

Updated Aug 4, 2026
Python

Alberto-Codes / turboquant-vllm

TurboQuant KV cache compression plugin for vLLM — asymmetric K/V, 8 models validated, consumer GPUs

compression transformer triton quantization inference-optimization kv-cache llm vllm consumer-gpu turboquant

Updated Apr 10, 2026
Python

mini-verl

DaoyuanLi2816 / mini-verl

Auditable one-GPU alignment and distillation runtime with shared-backbone training and a fail-closed verl artifact bridge.

alignment knowledge-distillation post-training peft tool-use single-gpu llm qlora qwen llm-agents verl llm-alignment consumer-gpu agentic-rl on-policy-distillation

Updated Aug 8, 2026
Python

giannisanni / neutronstar

Giant MoE models on a single consumer GPU by streaming experts from SSD. CUDA fork of antirez/ds4: runs GLM-5.2 (743B), Tencent Hy3 (295B), and DeepSeek 4 Flash, with io_uring expert streaming, LFU host cache, cross-layer expert prefetch, and the first MTP speculative decoding for GLM-5.2.

cuda glm mixture-of-experts llm-inference gguf speculative-decoding deepseek consumer-gpu ssd-streaming

Updated Jul 12, 2026
C

MELLM

Rahul-14507 / MELLM

Lightweight Modular AI Routing Engine for Local LLMs — Run specialised experts efficiently on consumer GPUs using smart Mixture-of-Experts routing.

python mixture-of-experts llama-cpp local-llm gguf consumer-gpu ai-router

Updated Jul 31, 2026
Python

sipsalabs / ultracompress

Verified AI infrastructure for regulated deployment. UltraCompress (our wedge): near-lossless 5-bit compression with SHA-256-reproducible reconstruction - prove the model in production is the one you validated. 23 architectures (0.6B-405B), Hermes-3-405B @ 1.0066x. OpenAI-compatible API. pip install ultracompress

python compression cuda inference pytorch transformer lossless quantization mlops deep-tech openai-api llm patent-pending ai-infrastructure 405b consumer-gpu 5-bit sipsa-labs experimental-tech

Updated Jul 27, 2026
Python

kimari-local-ai

smouj / kimari-local-ai

🦁 Local AI for Consumer GPUs — Run powerful LLMs on GTX 1060/1080. No cloud. No subscriptions. Built on llama.cpp + CUDA.

python cli nextjs cuda local-first llama-cpp llm-inference gguf open-webui openai-compatible-api consumer-gpu openclaw gtx-1060

Updated Jul 27, 2026
Python

yalun753 / moe-l2

MoE expert offload for low-VRAM GPUs — run 100B+ MoE models (DeepSeek, Qwen, Mixtral) on 8 GB cards. Expert caching, LRU hot cache, OpenAI-compatible proxy, GGUF multi-shard.

moe llama-cpp llm-inference deepseek consumer-gpu vram-optimization expert-caching

Updated Aug 8, 2026
Cuda

anna-claudette / angruvadal

RAM-Backed MCP Memory Architecture for Consumer LLM Inference — 900K token context on 16GB VRAM

amd mcp rocm llm llama-cpp local-llm context-window rdna4 consumer-gpu rotorquant

Updated Mar 27, 2026
Python

arbitrary-number / arbitrary-number

Arbitrary Numbers

python machine-learning deep-learning tensorflow gpu cuda pytorch nvidia model-serving numerical-computing model-optimization edge-inference ai-inference ai-performance consumer-gpu

Updated Aug 12, 2025
Python

obisin / dgls

Dynamic GPU Layer Swapping: Train large models on consumer GPUs with intelligent memory management

training pytorch gpu-memory memory-optimization consumer-gpu layer-swapping

Updated Sep 12, 2025
Python

bojobh609 / TurboQuant

Optimize FAISS-compatible vector quantization for fast, accurate vector search with TurboQuant

python machine-learning deep-learning metal pytorch nearest-neighbor quant attention embedding mlx iclr rag inference-optimization kv-cache kv-cache-compression consumer-gpu turboquant

Updated Aug 7, 2026
Python

Novamind-CS / Novamind-CS

Surgical reasoning on consumer silicon. Hybrid SSM + causal memory architecture with entropy-gated System 1/2 dispatch, O(1) inference memory, and continual learning — designed for 16 GB VRAM.

machine-learning research deep-learning pytorch lora language-model mamba causal-inference reasoning low-memory fine-tuning state-space-model continual-learning neuro-symbolic llm consumer-gpu

Updated Mar 22, 2026
Python

esonhjz / llama-cpp-moe-vram-benchmarks

Run MoE models on consumer GPUs by offloading Attention to GPU and shunting experts to CPU — 75% less VRAM, 9× faster than brute force.

cuda moe llama-cpp gpu-inference qwen low-vram consumer-gpu llm-benchmark vram-optimization

Updated Jul 30, 2026
PowerShell

JigSawPT / moe-autopilot

Adaptive hot/cold MoE expert cache for llama.cpp: keep the hottest experts in VRAM to speed up decode when offloading to RAM on a consumer GPU. Measured +26% (Qwen3-Coder-Next 80B) and +31.6% (gpt-oss-120B) on one RTX 5090.

cuda moe quantization mixture-of-experts llama-cpp gpu-inference local-llm llm-inference consumer-gpu expert-offloading

Updated Jul 23, 2026
Python

ikaganacar1 / ismail

ismail is a from-scratch Turkish language model implementation designed for low-end hardware, built and trained on a single RTX 5070 (12GB).

moe mla pytorch-implementation turkish-nlp llm llms llm-training deepseek-v3 turkish-llm consumer-gpu low-resource-llm

Updated Nov 19, 2025
Python

ToldByNun / sentinel

zero-dependency C++/CUDA DL framework focused on single consumer GPU's

deep-learning cpp python-library cuda poc llm-training consumer-gpu vram-optimization

Updated Aug 7, 2026
Cuda

Alex-888888 / memory-layers-consumer-hardware

Independent reconstruction of Memory Layers at Scale (Berges et al., 2024) on a single 24GB consumer GPU — Qwen2.5-7B, frozen backbone, with honest lessons learned.

machine-learning pytorch reproducibility rocm llm qwen consumer-gpu memory-layers

Updated Aug 5, 2026
Python

AIdevsmartdata / ramp-quant

RAMP: RL-guided Adaptive Mixed-Precision quantization for GGUF models. Data-free sensitivity analysis, evolutionary search, per-tensor type optimization. Produces hardware-optimized GGUF for consumer GPUs.

moe quantization sensitivity-analysis ramp mixed-precision llm llama-cpp qwen gguf qwen3 consumer-gpu imatrix ik-llama

Updated Apr 16, 2026
Python

Improve this page

Add a description, image, and links to the consumer-gpu topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the consumer-gpu topic, visit your repo's landing page and select "manage topics."