PyTorch Interview Questions: Gradients, eval and no_grad

Set up the deterministic exercise

The code was run with Python 3.12.14, PyTorch 2.9.0 and NumPy 2.2.6 on macOS arm64, using one CPU thread. In a separate Python environment, install torch==2.9.0 and numpy==2.2.6. Save the code below as gradient_lab.py and run python gradient_lab.py.

The scalar loss is (2w − 1)² with w=3, so its derivative is 4(2w − 1)=20. This hand calculation gives a useful independent expectation for the automatic differentiation check. The later model is deliberately tiny, with a dropout probability of 1 only to make the training/evaluation contrast deterministic.

import torch

torch.set_num_threads(1)
w = torch.tensor(3.0, requires_grad=True)
loss = (w * 2.0 - 1.0).square()
loss.backward()
assert w.grad.item() == 20.0
# A fresh graph still adds into the existing gradient buffer.
(w * 2.0 - 1.0).square().backward()
assert w.grad.item() == 40.0
try:
    assert w.grad.item() == 20.0
except AssertionError:
    print('expected failure: accumulated gradient is 40, not 20')
else:
    raise AssertionError('the accumulation mistake was not reproduced')
w.grad = None
(w * 2.0 - 1.0).square().backward()
assert w.grad.item() == 20.0

model = torch.nn.Sequential(
    torch.nn.Linear(2, 1, bias=False), torch.nn.Dropout(p=1.0)
)
with torch.no_grad():
    model[0].weight.copy_(torch.tensor([[2.0, -1.0]]))
x = torch.tensor([[3.0, 4.0]])
model.train()
assert torch.equal(model(x), torch.zeros(1, 1))
model.eval()
y = model(x)
assert y.requires_grad and y.item() == 2.0
with torch.no_grad():
    y_without_graph = model(x)
assert not y_without_graph.requires_grad
assert y_without_graph.item() == 2.0
model.train()
with torch.no_grad():
    assert model.training and model(x).item() == 0.0
try:
    model(torch.tensor([[3.0, 4.0, 5.0]]))
except RuntimeError:
    print('expected failure: three features cannot feed Linear(2, 1)')
else:
    raise AssertionError('shape mismatch unexpectedly accepted')
print('PASS: gradients 20 -> 40 -> 20; eval/no_grad and shape checks')
print('torch', torch.__version__)

1. Why does the gradient become 40 after the second backward call?

Each backward pass adds into the leaf tensor’s existing grad buffer. The second expression creates a fresh forward graph, but w still has its earlier gradient of 20. Adding the new gradient of 20 produces 40. The code catches the deliberately wrong assertion that it should still be 20, then resets w.grad before the next calculation.

Follow-up practice: What would change if accumulation were intentional across several mini-batches? Explain where resetting belongs relative to the intended optimization step, rather than calling all accumulation a bug.

2. Is this the same as calling backward twice on a freed graph?

No. The example recomputes the loss expression before its second backward pass, so it constructs a fresh graph. The issue being tested is the existing leaf gradient buffer. Reusing a graph whose saved values were released is a different concern. Retaining a graph would not by itself clear w.grad or prevent accumulation.

Follow-up practice: Identify which line creates a new graph and which line resets the buffer. Explain why changing only graph retention would not repair the wrong expectation.

3. What does model.eval change in this example?

It changes the module’s training flag and therefore the behavior of its Dropout layer. In training mode, p=1 makes the checked output zero. In evaluation mode, the linear result passes through as 2. The evaluation output still has requires_grad=True because the linear weight requires gradients and recording is enabled. Evaluation mode is not a request to disable autograd.

Follow-up practice: Why does this particular model need a mode-sensitive layer to demonstrate eval? What would you observe with only the Linear layer?

4. What does no_grad change, and what does it leave alone?

Within the tested no_grad block, the evaluation computation returns the same numeric value of 2 without building a backward graph for that result. The later training-mode check still returns zero inside no_grad: the context does not switch Dropout to evaluation behavior. The two controls address different concerns and can be used together when the intended computation requires both.

Follow-up practice: If you forgot eval but used no_grad for this model, which numeric assertion would fail? Explain why stopping gradient recording does not select inference behavior for every module.

5. Why is the three-feature input rejected?

The Linear layer is configured for two input features, and the valid tensor has shape [1, 2]. Its weights [[2, -1]] turn [3, 4] into 2. The rejected tensor has shape [1, 3], so its last dimension does not satisfy that layer’s contract. The test expects a RuntimeError for that mismatch rather than silently reshaping or dropping a value.

Follow-up practice: Would changing only the batch size violate the same contract? Separate the batch dimension from the last feature dimension and propose a valid two-row input.

6. What evidence would you show when explaining the debugging result?

I would show the hand-derived gradient of 20, the observed buffer sequence 20 → 40 → 20, the expected assertion failure before resetting, and the separate mode/recording assertions. I would also show that an invalid feature shape was rejected. Those checks establish the behavior of this small example; they do not prove a training pipeline converges or that inference is faster.

Follow-up practice: How would you extend the test to a real training loop without replacing evidence with a claim about model accuracy? Name the additional behavior you would need to measure.

Read the observed output and challenge one assumption

The verified run printed the two expected-failure messages and then the PASS line. The first failure is an assertion about an accumulated value; the second is the intentionally invalid input shape. Neither is an unexpected failure of the final exercise.