Python tf.keras.optimizers.SGD用法及代码示例

梯度下降(带动量)优化器。

继承自：Optimizer

用法

tf.keras.optimizers.SGD(
    learning_rate=0.01, momentum=0.0, nesterov=False, name='SGD', **kwargs
)

当 momentum 为 0 时，使用梯度 g 更新参数 w 的规则：

w = w - learning_rate * g

momentum 大于 0 时更新规则：

velocity = momentum * velocity - learning_rate * g
w = w + velocity

当 nesterov=True 时，此规则变为：

velocity = momentum * velocity - learning_rate * g
w = w + momentum * velocity - learning_rate * g

用法：

opt = tf.keras.optimizers.SGD(learning_rate=0.1)
var = tf.Variable(1.0)
loss = lambda:(var ** 2)/2.0         # d(loss)/d(var1) = var1
step_count = opt.minimize(loss, [var]).numpy()
# Step is `- learning_rate * grad`
var.numpy()
0.9

opt = tf.keras.optimizers.SGD(learning_rate=0.1, momentum=0.9)
var = tf.Variable(1.0)
val0 = var.value()
loss = lambda:(var ** 2)/2.0         # d(loss)/d(var1) = var1
# First step is `- learning_rate * grad`
step_count = opt.minimize(loss, [var]).numpy()
val1 = var.value()
(val0 - val1).numpy()
0.1
# On later steps, step-size increases because of momentum
step_count = opt.minimize(loss, [var]).numpy()
val2 = var.value()
(val1 - val2).numpy()
0.18

参考：

• 对于nesterov=True，请参见 Sutskever 等人，2013 年。