Fine-Tune a Small Coding Model on a Limited GPU

Fine-tuning adapts a pretrained coding model to a narrower task, style, or set of examples without training a model from scratch. With a modest GPU, use parameter-efficient fine-tuning (PEFT), such as LoRA or QLoRA, and a small, carefully prepared dataset; first make sure you have permission to use the model and training data.

What You’ll Need

A pretrained causal language model whose license permits your intended use, plus its tokenizer and model card.
A CUDA-capable GPU supported by your chosen framework. Available VRAM determines the model size, sequence length, and batch settings you can use; check free memory before starting.
A Python environment with a compatible PyTorch build, Transformers, PEFT, and a supported low-bit quantization library if using QLoRA. Follow their current installation guidance for your operating system and GPU.
A clean dataset of task-relevant code examples, with rights to use it. Remove secrets, personal data, and unwanted generated or duplicated examples.
Disk space for model files, cached data, checkpoints, and the final adapter. Keep a copy of the original model identifier and training configuration.

Step-by-Step

Choose a suitable base model. Start with a model already capable of the language and task you need, rather than trying to teach general programming from a tiny dataset. Check its context length, supported architecture, tokenizer, license, and whether your training stack supports it. Estimate the full workflow, not just training: model loading, optimizer state, activations, and checkpoints also consume memory.
Define one measurable objective. Write down what should improve, such as completing functions in a particular repository style or translating a specific format. Keep a held-out evaluation set aside before training. Choose representative prompts and expected answers, and record baseline results from the unmodified model so you can tell whether fine-tuning helped.
Prepare examples in the model’s expected format. Use the model’s documented chat template for conversations; for completion training, use consistent prompt-and-completion pairs. Validate that each example tokenizes correctly and fits the selected context length. For supervised fine-tuning, mask loss on prompt tokens when appropriate so the model learns from the target response rather than being rewarded for reproducing the prompt. Keep examples focused and avoid padding every sample to an unnecessarily large length.
Split and inspect the data. Separate training and evaluation data by project, source, or time where possible, not just randomly, so near-duplicate snippets do not leak across the split. Check samples manually for formatting errors, incorrect solutions, license concerns, secrets, and repeated boilerplate. Deduplicate before training. A small, accurate dataset is usually more useful than a large noisy one.
Set up memory-efficient training. Load the model in a supported low-bit format for QLoRA, or use LoRA without quantizing if memory allows. Train adapter weights instead of all model weights. Enable gradient checkpointing if supported; it reduces activation memory at the cost of extra computation. Use a small per-device batch size, then use gradient accumulation to reach a useful effective batch. Set a conservative sequence length based on actual examples and available VRAM.
Configure and launch a short run. Point the trainer to the training split and evaluation split, use the correct tokenizer and template, and specify the LoRA target modules supported by the architecture. Start with a low learning rate and a small number of epochs or steps; these are starting points, not universal values. Log training and evaluation loss, save checkpoints periodically, and run a brief smoke test first. Verify that training loss changes and that the process is updating adapter parameters.
Evaluate before extending training. Compare the fine-tuned adapter with the baseline on the same held-out prompts. Check correctness, formatting, whether code compiles or passes task-specific tests, and whether unrelated capabilities have degraded. Inspect generated outputs rather than relying on loss alone. If the result is worse, stop and investigate data, template, and configuration before adding more steps.
Save and test the deliverable. Save the adapter and its configuration, tokenizer or template notes, base-model identifier, package versions, and training settings. Reload the adapter in a fresh process and test representative prompts. For deployment, either load the adapter alongside its base model or merge it only if the library and model support that operation; keep the original adapter and base-model details for reproducibility.

Common Mistakes

Training every model parameter on a small GPU can exceed memory quickly; LoRA or QLoRA is usually the practical starting point. A low-bit base model does not make every part of training low-memory, since activations, temporary tensors, and evaluation batches still matter.

Other frequent problems include mixing chat formats, truncating target code, using a tokenizer that does not match the base model, and evaluating on examples nearly identical to training data. More epochs are not automatically better: a model can memorize a small dataset or lose general usefulness. Also, do not assume that a lower evaluation loss means code is correct; use tests and review outputs.

Tips

Begin with a tiny end-to-end run, including saving and reloading, before spending time on a full dataset. Keep a known-good configuration and change one variable at a time. When VRAM is tight, reduce sequence length first if examples allow, then reduce the per-device batch size; gradient accumulation can preserve the effective batch size but does not solve activation memory for a single long sequence.

Save checkpoints to persistent storage and monitor both GPU memory and disk use. Keep evaluation prompts fixed between runs, and compare results against the baseline. If the task is mostly about repository-specific facts, retrieval or supplying relevant files at inference time may be more reliable than trying to bake frequently changing information into model weights.

Troubleshooting

Out-of-memory error: Lower sequence length or per-device batch size, enable gradient checkpointing, and confirm that only intended adapter parameters require gradients. Close other GPU jobs. If the model still cannot load, choose a smaller supported base model or a compatible quantized loading method.

Loss is unchanged or becomes invalid: Check that labels are not all masked, the target text is present, and the correct template and tokenizer are being used. Confirm that adapter parameters are trainable and that the learning rate is not zero. A sudden invalid loss can also point to bad examples or incompatible precision settings; inspect a batch and try a short run with a safer supported precision.

Training works but output is poor: Inspect examples after tokenization and truncation, verify prompt/answer boundaries, and compare a few training examples with held-out ones. Check for data leakage, inconsistent formatting, and an overly long or aggressive run. Rebuild a small clean dataset and rerun a short experiment before changing many settings.

Adapter will not load: Confirm it is paired with the exact compatible base model and architecture, and that the adapter files completed saving. Check library compatibility and use the documented loading method for that model. Do not merge adapters as a workaround unless the model and tooling explicitly support merging.

Frequently Asked Questions

Can I fine-tune a coding model on a single GPU?
Often, yes, if the model fits using an appropriate PEFT setup. The exact limit depends on GPU memory, model architecture, sequence length, and training configuration.

Should I use LoRA or QLoRA?
LoRA trains small adapter weights while keeping the base weights frozen. QLoRA additionally loads the base model in a low-bit format to save memory, but requires compatible hardware and software.

How many examples do I need?
There is no universal minimum. A small, consistent set can work for a narrow format or behavior, but broader capability needs more varied, high-quality data. Evaluate on held-out examples rather than relying on a target count.

How long should training run?
Run a short baseline experiment and judge held-out results. Stop when evaluation or task-test performance stops improving, or starts to decline; do not choose epochs by training loss alone.

Do I need to merge the adapter into the model?
No. Many inference setups can load an adapter with its base model. Merge only when supported and useful for your deployment, and retain the separate adapter and configuration.

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →