twn (ternary weight networks)

The authors introduced ternary weight networks (TWNs) to address the limited storage and computational resources issues in hardware. The quantization problem can be formulated as follows:
$$
begin{cases}
alpha^, W^{t} = &argmin_{alpha, W^t} J(alpha, W^t) = |W-alpha W^t|_2^2 \
s.t. & age0, W_i^tin{-1,0,1}, i=1,2,dots, n.
end{cases} tag{1}
$$
Here $n$ is the size of filter, $W$ represents weights of the network. With $Wapprox alpha W^t$ and assuming the convolutional layer do not have bias term, forward propagation of ternary weight networks is as follows:
$$
begin{cases}
Z & = &XW approx X(alpha W^t) = (alpha X)oplus W^t \
X^{next} & = & g(Z)
end{cases} tag{2}
$$
where $X$ indicates inputs, $$ indicates convolutional operation, $g$ is the non-linear activation function, $oplus$ indicates the inner product or convolutional operation without any multiplication, $X^{next}$ indicates the outputs.

The approximated solution of $W$ with threshold-based ternary function is as follows:
$$
W_i^t = f_t(W_i|triangle) =
begin{cases}
+1, if~W_i gt triangle \
0, if~|W_i| le triangle \
-1, if~W_i lt -triangle
end{cases} tag{3}
$$
The optimized objective function can be written as:
$$
alpha^, triangle^ = argmin_{alphage0, trianglegt 0}(|Itriangle|alpha^2-2(sum{iin I_triangle}|W_i|)alpha+c_triangle) tag{4}
$$
where $I_triangle = {i|~|W|gttriangle}$ and $|I_triangle|$ denotes the number of elements in $I_triangle$; $ctriangle = sum{iin I_triangle^c}W_i^2$ is a $alpha$-independent constant. Thus the optimal solutions of the objective function can be computed as follows:
$$
alpha_triangle^ = frac{1}{|Itriangle|}sum{iin I_triangle}|Wi| \
triangle^* = argmax{trianglegt 0}(sum_{iin I_triangle}|Wi|^2) tag{5}
$$
Here solution of $triangle$ is approximated by $triangle^*approx0.7cdot E(|W|) approx frac{0.7}{n}sum{i=1}^n|W_i|$.

Training Methods

The training of ternary weight networks can be summarized to three steps: quantization, training and updating. Quantization phase is to quantize the weights of convolutional layers using Equation (5), then apply standard forward and backward propagation to the network, and update parameters using standard SGD. Source code is available on GitHub.

Important Tips

• $alpha$ is used as the scaling factor for input $X$ not for weights $W$
• gradient is computed using $W^t$
• quantized weights are used during forward and backward but not during parameters update, so $W_l^r gets W_l^r - eta frac{partial C}{partial W_l^t} $
• first compute $triangle$ then compute mask, finally compute $alpha$
• $triangle$ is computed with all $|W^r|$ while $alpha$ is computed only with those $|W^r|>triangle$
• apply weight-decay would lead results worse
• this blog is useful for implementation

Experimental Results