Advanced

Fine-tuning LLaMA 3 with LoRA

Learn how to efficiently fine-tune LLaMA 3 models using Low-Rank Adaptation (LoRA) on custom datasets. Save compute while gaining state-of-the-art results.

AO
Created by Dr. Amara Osei
4.8rating
1240 learners enrolled
12 hours duration

What you'll learn

Prepare and format custom datasets for fine-tuning
Understand the mathematics behind Low-Rank Adaptation (LoRA)
Configure PEFT (Parameter-Efficient Fine-Tuning) libraries
Evaluate model performance before and after tuning
Merge adapters with base weights and export to GGUF format

Course Content

Section 1: Course Overview & Setup
Welcome & Learning Roadmap
10 mins
Setting up your Google Colab / Lambda Labs environment
15 mins
Introduction to LLaMA 3 Architecture
25 mins
Section 2: Dataset Preparation & Curation
Structuring conversational datasets (ShareGPT vs Alpaca formats)
30 mins
Data cleaning, tokenization, and padding guidelines
25 mins
Lab: Preparing a custom Q&A dataset
45 mins
Section 3: Understanding & Configuring LoRA
Under the hood: Attention weights and rank decomposition
40 mins
Choosing LoRA parameters: rank (r), alpha, target modules
20 mins
Configuring PEFT and TRL libraries in PyTorch
35 mins
Section 4: Model Training & Optimization
Gradient accumulation, checkpointing, and precision (FP16/BF16)
30 mins
Monitoring loss curves with Weights & Biases
25 mins
Lab: Running the fine-tuning script on a single GPU
60 mins

Your Instructor

AO

Dr. Amara Osei

Lead AI Researcher at DeepTech Africa

Amara holds a PhD in NLP and has spent 8+ years developing model architectures. She has helped open-source several African language models.

Prerequisites

  • Solid Python programming knowledge
  • Familiarity with PyTorch & Hugging Face ecosystem
  • Basic understanding of neural network weights
$120USD
Choose Payment Method
This course includes:
Full lifetime access
Access on mobile and desktop
Certificate of completion
Exercises & course resources