RHNCell

RHNCell(input_size => hidden_size, [depth];
    couple_carry = true,
    cell_kwargs...)

Recurrent highway network ^[Zilly2017]. See RHNCellUnit for a the unit component of this layer. See RHN for a layer that processes entire sequences.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.
• depth: depth of the recurrence. Default is 3.

Keyword arguments

• couple_carry: couples the carry gate and the transform gate. Default true
• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform
• bias: include a bias or not. Default is true

Equations

\[\begin{aligned} \mathbf{s}_{\ell}(t) &= \mathbf{h}_{\ell}(t) \odot \mathbf{t}_{\ell}(t) + \mathbf{s}_{\ell-1}(t) \odot \mathbf{c}_{\ell}(t) \\ \mathbf{h}_{\ell}(t) &= \tanh\left( \mathbf{W}^{h}_{ih} \mathbf{x}(t) \, \mathbb{I}_{\ell = 1} + \mathbf{W}^{h_{\ell}}_{hh} \mathbf{s}_{\ell-1}(t) + \mathbf{b}^{h_{\ell}} \right) \\ \mathbf{t}_{\ell}(t) &= \sigma\left( \mathbf{W}^{t}_{ih} \mathbf{x}(t) \, \mathbb{I}_{\ell = 1} + \mathbf{W}^{t_{\ell}}_{hh} \mathbf{s}_{\ell-1}(t) + \mathbf{b}^{t_{\ell}} \right) \\ \mathbf{c}_{\ell}(t) &= \sigma\left( \mathbf{W}^{c}_{ih} \mathbf{x}(t) \, \mathbb{I}_{\ell = 1} + \mathbf{W}^{c_{\ell}}_{hh} \mathbf{s}_{\ell-1}(t) + \mathbf{b}^{c_{\ell}} \right) \end{aligned}\]

Forward

rnncell(inp, [state])

RHNCellUnit(input_size => hidden_size;
    init_kernel = glorot_uniform,
    bias = true)