I think there is an error in the following code:
past_velocity = 0.0
# Constant momentum factor
momentum = 0.1
# Optimization loop
while loss > 0.01:
w, loss, gradient = get_current_parameters()
velocity = past_velocity * momentum - learning_rate * gradient
w = w + momentum * velocity - learning_rate * gradient
past_velocity = velocity
update_parameter(w)
The line w = w + momentum * velocity - learning_rate * gradient should be just w += velocity?
I think there is an error in the following code:
The line w = w + momentum * velocity - learning_rate * gradient should be just w += velocity?