C++ Autograd

An autodidactical autograd engine written in C++. Inspired by Andrej Karpathy’s Micrograd, but implemented in C++ to understand both automatic differentiation and lower-level memory management.

What is Autograd?

Automatic differentiation is the backbone of modern deep learning. Instead of manually deriving gradients for every operation, autograd builds a computational graph and computes gradients automatically via backpropagation.

How It Works

// Forward pass builds the graph
Value a = Value(2.0);
Value b = Value(3.0);
Value c = a * b;      // c = 6.0
Value d = c + a;      // d = 8.0

// Backward pass computes gradients
d.backward();
// a.grad = 4.0 (∂d/∂a = b + 1 = 4)
// b.grad = 2.0 (∂d/∂b = a = 2)

Components

1. Value Class

   • Wraps scalar values
   • Tracks parent nodes and the operation that created it
   • Stores gradient accumulated during backprop

2. Operator Overloading

   • +, -, *, / build the computational graph
   • Each operation stores its backward function

3. Backward Pass

   • Topological sort of the graph
   • Chain rule applied in reverse order
   • Gradient accumulation for nodes used multiple times

4. Neural Network Layer

   • nn.cpp implements basic neural network components
   • Neurons, layers, and simple MLPs
   • All built on top of the autograd engine

Why C++?

Python autograd implementations hide memory management. In C++, you have to think about:

• Object lifetimes and ownership
• Graph node allocation
• Efficient gradient accumulation

This forced a deeper understanding of what’s actually happening during backpropagation.

Tech Stack

• C++
• Makefile for build system
• No external dependencies

Repo Link

https://github.com/Mkrolick/Cpp_Autograd