How a PyTorch Linear Layer Works | Linear Algebra for AI Engineers
Read the Portuguese translation →
Author: The Augmented · Watch on YouTube
Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.
Original language: English
Original video: How a PyTorch Linear Layer Works | Linear Algebra for AI Engineers · Channel: The Augmented · 00:31 · Watch on YouTube
Topics: AI models, Open source and software ecosystems
Summary
Section titled “Summary”The video explains how PyTorch stores the weights of a linear layer, using a layer with three inputs and two outputs as an example. Each output neuron has three weights and a bias, so the weight matrix has two rows and three columns. In the forward pass, PyTorch transposes the weight matrix and multiplies the input by it. The video compares this with the column-vector convention in math and emphasizes that the difference is a matter of transposition.
Key points
Section titled “Key points”- PyTorch stores one output neuron per row of the weight matrix.
- An nn.Linear layer with 3 inputs and 2 outputs has a weight matrix of shape 2 by 3.
- Each neuron multiplies all inputs by its corresponding weights, sums them, and adds a bias to produce one output.
- In the forward pass, PyTorch computes the input multiplied by the transposed weight matrix, plus a bias.
- The transposed weight matrix has shape 3 by 2.
- In the math convention described, vectors are columns and the matrix uses columns to represent input directions.
- In PyTorch, vectors are rows, and the corresponding directions are represented by rows.
- The two conventions express the same idea with a transpose.
Chapters
Section titled “Chapters”| Start | Chapter | Summary |
|---|---|---|
| 00:00 | How PyTorch stores linear-layer weights | The video introduces the 3-input, 2-output example and explains why PyTorch stores two rows of three weights. |
| 00:10 | The forward pass and transposed weights | It shows that PyTorch multiplies the input by the transposed weight matrix and adds a bias. |
| 00:22 | Rows, columns, and landing vectors | The video contrasts math and PyTorch vector conventions and explains that they differ by a transpose. |
Quotes
Section titled “Quotes”PyTorch stores one output neuron per row. — 00:00
the weight will come back 2 by 3. — 00:00
In the forward pass, PyTorch flips it. — 00:00
In math, vectors are columns, and W * V uses the columns. — 00:31
It’s the same idea, just transposed. — 00:31
Concepts and entities
Section titled “Concepts and entities”PyTorch, nn.Linear, weight matrix, bias, forward pass
Full transcript (original language)
Section titled “Full transcript (original language)”00:00 PyTorch stores one output neuron per row. So, when you give an nn.linear 3 in and 2 out, the weight will come back 2 by 3. A neuron takes all three inputs, multiplies each by its own weight, sums them, adds a bias, and outputs one number. Two outputs, two neurons, two rows of three. In the forward pass, PyTorch flips it. X at weight.t plus a bias. The flipped weight is 3 by 2, exactly the W from the last video, where each row is where an input direction
00:31 lands. So, don’t ask which is in and which is out. Ask where are the landing vectors stored? In the rows or the columns? In math, vectors are columns, and W * V uses the columns. In PyTorch, vectors are rows, and X at W uses the rows. It’s the same idea, just transposed. nn.linear stores neurons as rows and transposes on the way through.