minimal-nn-framework

Overview

This repository contains an educational re-implementation of a minimal neural network framework with automatic differentiation, inspired by Andrej Karpathy’s video:

“The spelled-out intro to neural networks and backpropagation: building micrograd”

The project implements a scalar-based autograd engine and a small neural network library from first principles, without relying on frameworks such as PyTorch or TensorFlow.

---

Motivation & Goals

Modern deep learning frameworks hide much of the underlying complexity behind high-level APIs. This project aims to demystify neural networks by building them from first principles.

The goals of this project are to:

• Understand reverse-mode automatic differentiation
• See how backpropagation emerges from computation graphs
• Build a minimal neural network framework from scratch
• Develop intuition for gradient flow and training dynamics

---

What’s Implemented

Automatic Differentiation

• Scalar Value object with:
  • Data storage
  • Gradient tracking
  • Reverse-mode backpropagation via computation graph traversal
• Supported operations:
  • Addition, subtraction, multiplication, division
  • Power, negation
  • log, exp

Activation Functions

• tanh
• sigmoid
• leaky_relu

Neural Network Components

• Neuron with configurable activation functions
• Layer abstraction
• Multi-Layer Perceptron (MLP)
• Activation-aware weight initialization:
  • He/Kaiming initialization for LeakyReLU
  • Xavier initialization for tanh/sigmoid

Loss Functions

• Mean Squared Error (MSE)
• Binary Cross-Entropy (BCE)

Optimization

• Stochastic Gradient Descent (SGD) optimizer
  • Parameter update step
  • Gradient zeroing

---

Design Extensions

This project extends a minimal scalar autograd core into a small, learning-oriented neural network framework by adding:

• Multiple activation functions (sigmoid, leaky ReLU)
• Activation-specific weight initialization (He, Xavier)
• Modular loss functions (MSE, Binary Cross-Entropy)
• A simple optimizer abstraction (SGD)
• Cleaner separation between:
  • Autograd engine
  • Neural network layers
  • Loss functions
  • Optimization logic

These additions make the project closer to a minimal learning-oriented neural network framework, while still preserving the simplicity and transparency of scalar-based automatic differentiation.

---

What This Is Not

• A production-ready deep learning framework
• Optimized for speed or large-scale models
• A replacement for PyTorch / TensorFlow

Understand Why: AUTOGRAD.md.

This project prioritizes learning over performance.

---

Project Structure

.
├── learning_testing/       # Understanding the process of autograd & nn
├── minimal_nn/             # minimal_nn framework & documentation
├── examples/               # Training examples
└── README.md

---

Learning Resources & Notebooks

The learning_testing/ directory contains a set of Jupyter notebooks created while learning and experimenting with the concepts in this project:

• understanding_autograd.ipynb
• understanding_nn_1.ipynb
• understanding_nn_2.ipynb

Each notebook includes step-by-step explanations, intermediate experiments, and incremental code to build intuition for automatic differentiation and neural networks. This lighter structure is intended to make it easier to follow the core ideas without jumping directly into the more complete minimal_nn/ framework.

Important Note for Learners

While these notebooks are written with care and clarity, they are intended as a supplementary resource rather than a standalone introduction.

For someone learning these concepts for the first time, the recommended learning order is:

1. Watch Andrej Karpathy’s micrograd video to build foundational intuition
2. Explore the notebooks in learning_testing/ to reinforce and clarify ideas
3. Study the full implementation in minimal_nn/ to see how everything fits together

The notebooks work best as a companion resource after watching the video, helping solidify understanding and serving as a reference to revisit concepts later. These notebooks also serve as a personal long-term reference, allowing concepts to be revisited quickly without rewatching the entire video.

---

Acknowledgements

This project closely follows and re-implements the ideas presented in Andrej Karpathy’s educational video:
“The spelled-out intro to neural networks and backpropagation: building micrograd”.

The purpose of this repository is learning and understanding; the implementation was written by me while following the tutorial and experimenting with the concepts.

---