Layers

AntisymmetricRNN(input_size, hidden_size, [activation];
    return_state = false, kwargs...)

Antisymmetric recurrent neural network. See AntisymmetricRNNCell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.
• activation: activation function. Default is tanh.

Keyword arguments

• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true
• epsilon: step size. Default is 1.0.
• gamma: strength of diffusion. Default is 0.0.

Equations

\[h_t = h_{t-1} + \epsilon \tanh ( (W_h - W_h^T - \gamma I) h_{t-1} + V_h x_t + b_h ),\]

Forward

asymrnn(inp, state)
asymrnn(inp)

Arguments

• inp: The input to the asymrnn. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• state: The hidden state of the AntisymmetricRNN. If given, it is a vector of size hidden_size or a matrix of size hidden_size x batch_size. If not provided, it is assumed to be a vector of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

ATR(input_size, hidden_size;
    return_state = false, kwargs...)

Addition-subtraction twin-gated recurrent cell. See ATRCell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.

Keyword arguments

• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true

Equations

\[\begin{aligned} \mathbf{p}_t &= \mathbf{W} \mathbf{x}_t, \\ \mathbf{q}_t &= \mathbf{U} \mathbf{h}_{t-1}, \\ \mathbf{i}_t &= \sigma(\mathbf{p}_t + \mathbf{q}_t), \\ \mathbf{f}_t &= \sigma(\mathbf{p}_t - \mathbf{q}_t), \\ \mathbf{h}_t &= \mathbf{i}_t \circ \mathbf{p}_t + \mathbf{f}_t \circ \mathbf{h}_{t-1}. \end{aligned}\]

Forward

atr(inp, state)
atr(inp)

Arguments

• inp: The input to the atr. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• state: The hidden state of the ATR. If given, it is a vector of size hidden_size or a matrix of size hidden_size x batch_size. If not provided, it is assumed to be a vector of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

BR(input_size, hidden_size;
    return_state = false, kwargs...)

Bistable recurrent network. See BRCell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.

Keyword arguments

• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true

Equations

\[\begin{aligned} \mathbf{a}_t &= 1 + \tanh\left(\mathbf{U}_a \mathbf{x}_t + \mathbf{w}_a \circ \mathbf{h}_{t-1}\right), \\ \mathbf{c}_t &= \sigma\left(\mathbf{U}_c \mathbf{x}_t + \mathbf{w}_c \circ \mathbf{h}_{t-1}\right). \mathbf{h}_t &= \mathbf{c}_t \circ \mathbf{h}_{t-1} + (1 - \mathbf{c}_t) \circ \tanh\left(\mathbf{U}_x \mathbf{x}_t + \mathbf{a}_t \circ \mathbf{h}_{t-1}\right), \\ \end{aligned}\]

Forward

br(inp, state)
br(inp)

Arguments

• inp: The input to the br. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• state: The hidden state of the BR. If given, it is a vector of size hidden_size or a matrix of size hidden_size x batch_size. If not provided, it is assumed to be a vector of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

CFN(input_size => hidden_size;
    return_state = false, kwargs...)

Chaos free network unit. See CFNCell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.

Keyword arguments

• return_state: Option to return the last state together with the output. Default is false.
• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true.

Equations

\[\begin{aligned} h_t &= \theta_t \odot \tanh(h_{t-1}) + \eta_t \odot \tanh(W x_t), \\ \theta_t &:= \sigma (U_\theta h_{t-1} + V_\theta x_t + b_\theta), \\ \eta_t &:= \sigma (U_\eta h_{t-1} + V_\eta x_t + b_\eta). \end{aligned}\]

Forward

cfn(inp, state)
cfn(inp)

Arguments

• inp: The input to the cfn. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• state: The hidden state of the CFN. If given, it is a vector of size hidden_size or a matrix of size hidden_size x batch_size. If not provided, it is assumed to be a vector of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

coRNN(input_size => hidden_size, [dt];
    gamma=0.0, epsilon=0.0,
    return_state=false, init_kernel = glorot_uniform,
    init_recurrent_kernel = glorot_uniform, bias = true)

Coupled oscillatory recurrent neural unit. See coRNNCell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.
• dt: time step. Default is 1.0.

Keyword arguments

• gamma: damping for state. Default is 0.0.
• epsilon: damping for candidate state. Default is 0.0.
• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true.
• return_state: Option to return the last state together with the output. Default is false.

Equations

\[\begin{aligned} \mathbf{y}_n &= y_{n-1} + \Delta t \mathbf{z}_n, \\ \mathbf{z}_n &= z_{n-1} + \Delta t \sigma \left( \mathbf{W} y_{n-1} + \mathcal{W} z_{n-1} + \mathbf{V} u_n + \mathbf{b} \right) - \Delta t \gamma y_{n-1} - \Delta t \epsilon \mathbf{z}_n, \end{aligned}\]

Forward

cornn(inp, (state, zstate))
cornn(inp)

Arguments

• inp: The input to the cornn. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• (state, cstate): A tuple containing the hidden and cell states of the coRNN. They should be vectors of size hidden_size or matrices of size hidden_size x batch_size. If not provided, they are assumed to be vectors of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

FastGRNN(input_size => hidden_size, [activation];
return_state = false, kwargs...)

Fast recurrent neural network. See FastGRNNCell for a layer that processes a single sequences.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.
• activation: the activation function, defaults to tanh_fast

Keyword arguments

• return_state: Option to return the last state together with the output. Default is false.

• init_kernel: initializer for the input to hidden weights Default is glorot_uniform.

• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.

• init_zeta: Initializer for the zeta parameter. Default is 1.0.

• init_nu: Initializer for the nu parameter. Default is - 4.0.

• bias: include a bias or not. Default is true.

Equations

\[\begin{aligned} z_t &= \sigma(W_z x_t + U_z h_{t-1} + b_z), \\ \tilde{h}_t &= \tanh(W_h x_t + U_h h_{t-1} + b_h), \\ h_t &= \big((\zeta (1 - z_t) + \nu) \odot \tilde{h}_t\big) + z_t \odot h_{t-1} \end{aligned}\]

Forward

fastgrnn(inp, state)
fastgrnn(inp)

Arguments

• inp: The input to the fastgrnn. It should be a vector of size input_size or a matrix of size input_size x batch_size.
• state: The hidden state of the FastGRNN. It should be a vector of size hidden_size or a matrix of size hidden_size x batch_size. If not provided, it is assumed to be a vector of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

FastRNN(input_size => hidden_size, [activation];
    return_state = false, kwargs...)

Fast recurrent neural network. See FastRNNCell for a layer that processes a single sequences.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.
• activation: the activation function, defaults to tanh_fast.

Keyword arguments

• return_state: Option to return the last state together with the output. Default is false.

• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.

• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.

• init_alpha: Initializer for the alpha parameter. Default is 3.0.

• init_beta: Initializer for the beta parameter. Default is - 3.0.

• bias: include a bias or not. Default is true.

Equations

\[\begin{aligned} \tilde{h}_t &= \sigma(W_h x_t + U_h h_{t-1} + b), \\ h_t &= \alpha \tilde{h}_t + \beta h_{t-1} \end{aligned}\]

Forward

fastrnn(inp, state)
fastrnn(inp)

Arguments

• inp: The input to the fastrnn. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• state: The hidden state of the FastRNN. If given, it is a vector of size hidden_size or a matrix of size hidden_size x batch_size. If not provided, it is assumed to be a vector of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

FSRNN(input_size => hidden_size,
    fast_cells, slow_cell;
    return_state=false)

Fast slow recurrent neural network. See FSRNNCell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.
• fast_cells: a vector of the fast cells. Must be minimum of length 2.
• slow_cell: the chosen slow cell.
• return_state: option to return the last state. Default is false.

Equations

\[\begin{aligned} h_t^{F_1} &= f^{F_1}\left(h_{t-1}^{F_k}, x_t\right) \\ h_t^S &= f^S\left(h_{t-1}^S, h_t^{F_1}\right) \\ h_t^{F_2} &= f^{F_2}\left(h_t^{F_1}, h_t^S\right) \\ h_t^{F_i} &= f^{F_i}\left(h_t^{F_{i-1}}\right) \quad \text{for } 3 \leq i \leq k \end{aligned}\]

Forward

fsrnn(inp, (fast_state, slow_state))
fsrnn(inp)

Arguments

• inp: The input to the fsrnn. It should be a vector of size input_size or a matrix of size input_size x batch_size.
• (fast_state, slow_state): A tuple containing the hidden and cell states of the FSRNN. They should be vectors of size hidden_size or matrices of size hidden_size x batch_size. If not provided, they are assumed to be vectors of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

GatedAntisymmetricRNN(input_size, hidden_size;
    return_state = false, kwargs...)

Antisymmetric recurrent neural network with gating. See GatedAntisymmetricRNNCell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.

Keyword arguments

• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true.
• epsilon: step size. Default is 1.0.
• gamma: strength of diffusion. Default is 0.0.

Equations

\[\begin{aligned} z_t &= \sigma \left( (W_h - W_h^T - \gamma I) h_{t-1} + V_z x_t + b_z \right), \\ h_t &= h_{t-1} + \epsilon z_t \odot \tanh \left( (W_h - W_h^T - \gamma I) h_{t-1} + V_h x_t + b_h \right). \end{aligned}\]

Forward

asymrnn(inp, state)
asymrnn(inp)

Arguments

• inp: The input to the asymrnn. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• state: The hidden state of the GatedAntisymmetricRNN. If given, it is a vector of size hidden_size or a matrix of size hidden_size x batch_size. If not provided, it is assumed to be a vector of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

IndRNN(input_size, hidden_size, [activation];
    return_state = false, kwargs...)

Independently recurrent network. See IndRNNCell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.
• activation: activation function. Default is tanh.

Keyword arguments

• return_state: Option to return the last state together with the output. Default is false.
• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true.

Equations

\[\mathbf{h}_{t} = \sigma(\mathbf{W} \mathbf{x}_t + \mathbf{u} \odot \mathbf{h}_{t-1} + \mathbf{b})\]

Forward

indrnn(inp, state)
indrnn(inp)

Arguments

• inp: The input to the indrnn. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• state: The hidden state of the IndRNN. If given, it is a vector of size hidden_size or a matrix of size hidden_size x batch_size. If not provided, it is assumed to be a vector of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

JANET(input_size => hidden_size;
    return_state = false, kwargs...)

Just another network. See JANETCell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.

Keyword arguments

• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true.
• return_state: Option to return the last state together with the output. Default is false.
• beta_value: control over the input data flow. Default is 1.0.

Equations

\[\begin{aligned} \mathbf{s}_t &= \mathbf{U}_f \mathbf{h}_{t-1} + \mathbf{W}_f \mathbf{x}_t + \mathbf{b}_f \\ \tilde{\mathbf{c}}_t &= \tanh (\mathbf{U}_c \mathbf{h}_{t-1} + \mathbf{W}_c \mathbf{x}_t + \mathbf{b}_c) \\ \mathbf{c}_t &= \sigma(\mathbf{s}_t) \odot \mathbf{c}_{t-1} + (1 - \sigma (\mathbf{s}_t - \beta)) \odot \tilde{\mathbf{c}}_t \\ \mathbf{h}_t &= \mathbf{c}_t. \end{aligned}\]

Forward

janet(inp, (state, cstate))
janet(inp)

Arguments

• inp: The input to the janet. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• (state, cstate): A tuple containing the hidden and cell states of the JANET. They should be vectors of size hidden_size or matrices of size hidden_size x batch_size. If not provided, they are assumed to be vectors of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

LEM(input_size => hidden_size, [dt];
    return_state=false, init_kernel = glorot_uniform,
    init_recurrent_kernel = glorot_uniform, bias = true)

Long expressive memory network. See LEMCell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.
• dt: timestep. Defaul is 1.0.

Keyword arguments

• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true.
• return_state: Option to return the last state together with the output. Default is false.

Equations

\[\begin{aligned} \boldsymbol{\Delta t_n} &= \Delta \hat{t} \hat{\sigma} (W_1 y_{n-1} + V_1 u_n + b_1) \\ \overline{\boldsymbol{\Delta t_n}} &= \Delta \hat{t} \hat{\sigma} (W_2 y_{n-1} + V_2 u_n + b_2) \\ z_n &= (1 - \boldsymbol{\Delta t_n}) \odot z_{n-1} + \boldsymbol{\Delta t_n} \odot \sigma (W_z y_{n-1} + V_z u_n + b_z) \\ y_n &= (1 - \boldsymbol{\Delta t_n}) \odot y_{n-1} + \boldsymbol{\Delta t_n} \odot \sigma (W_y z_n + V_y u_n + b_y) \end{aligned}\]

Forward

lem(inp, (state, zstate))
lem(inp)

Arguments

• inp: The input to the LEM. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• (state, cstate): A tuple containing the hidden and cell states of the LEM. They should be vectors of size hidden_size or matrices of size hidden_size x batch_size. If not provided, they are assumed to be vectors of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

LiGRU(input_size => hidden_size;
    return_state = false, kwargs...)

Light gated recurrent network. The implementation does not include the batch normalization as described in the original paper. See LiGRUCell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.

Keyword arguments

• return_state: Option to return the last state together with the output. Default is false.
• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true.

Equations

\[\begin{aligned} z_t &= \sigma(W_z x_t + U_z h_{t-1}), \\ \tilde{h}_t &= \text{ReLU}(W_h x_t + U_h h_{t-1}), \\ h_t &= z_t \odot h_{t-1} + (1 - z_t) \odot \tilde{h}_t \end{aligned}\]

Forward

ligru(inp, state)
ligru(inp)

Arguments

• inp: The input to the ligru. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• state: The hidden state of the LiGRU. If given, it is a vector of size hidden_size or a matrix of size hidden_size x batch_size. If not provided, it is assumed to be a vector of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

LightRU(input_size => hidden_size;
    return_state = false, kwargs...)

Light recurrent unit network. See LightRUCell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer

Keyword arguments

• return_state: Option to return the last state together with the output. Default is false.
• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true.

Equations

\[\begin{aligned} \tilde{h}_t &= \tanh(W_h x_t), \\ f_t &= \delta(W_f x_t + U_f h_{t-1} + b_f), \\ h_t &= (1 - f_t) \odot h_{t-1} + f_t \odot \tilde{h}_t. \end{aligned}\]

Forward

lightru(inp, state)
lightru(inp)

Arguments

• inp: The input to the lightru. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• state: The hidden state of the LightRU. If given, it is a vector of size hidden_size or a matrix of size hidden_size x batch_size. If not provided, it is assumed to be a vector of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

MGU(input_size => hidden_size;
    return_state = false, kwargs...)

Minimal gated unit network. See MGUCell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.

Keyword arguments

• return_state: Option to return the last state together with the output. Default is false.
• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true.

Equations

\[\begin{aligned} f_t &= \sigma(W_f x_t + U_f h_{t-1} + b_f), \\ \tilde{h}_t &= \tanh(W_h x_t + U_h (f_t \odot h_{t-1}) + b_h), \\ h_t &= (1 - f_t) \odot h_{t-1} + f_t \odot \tilde{h}_t \end{aligned}\]

Forward

mgu(inp, state)
mgu(inp)

Arguments

• inp: The input to the mgu. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• state: The hidden state of the MGU. If given, it is a vector of size hidden_size or a matrix of size hidden_size x batch_size. If not provided, it is assumed to be a vector of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

MinimalRNN(input_size => hidden_size;
    return_state = false, kwargs...)

Minimal recurrent neural network. See MinimalRNNCell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.

Keyword arguments

• return_state: Option to return the last state together with the output. Default is false.
• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true.
• encoder_bias: include a bias in the encoder or not. Default is true.

Equations

\[\begin{aligned} \mathbf{z}_t &= \Phi(\mathbf{x}_t) = \tanh(\mathbf{W}_x \mathbf{x}_t + \mathbf{b}_z), \\ \mathbf{u}_t &= \sigma(\mathbf{U}_h \mathbf{h}_{t-1} + \mathbf{U}_z \mathbf{z}_t + \mathbf{b}_u), \\ \mathbf{h}_t &= \mathbf{u}_t \circ \mathbf{h}_{t-1} + (1 - \mathbf{u}_t) \circ \mathbf{z}_t. \end{aligned}\]

Forward

minimalrnn(inp, (state, c_state))
minimalrnn(inp)

Arguments

• inp: The input to the minimalrnn. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• (state, cstate): A tuple containing the hidden and cell states of the MinimalRNN. They should be vectors of size hidden_size or matrices of size hidden_size x batch_size. If not provided, they are assumed to be vectors of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

MultiplicativeLSTM(input_size => hidden_size;
    return_state=false,
    kwargs...)

Multiplicative long short term memory network. See MultiplicativeLSTMCell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.

Keyword arguments

• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• init_multiplicative_kernel: initializer for the multiplicative to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true.
• return_state: Option to return the last state together with the output. Default is false.

Equations

\[\begin{aligned} \mathbf{m}_t &= (\mathbf{W}_{mx} \mathbf{x}_t) \circ (\mathbf{W}_{mh} \mathbf{h}_{t-1}), \\ \hat{\mathbf{h}}_t &= \mathbf{W}_{hx} \mathbf{x}_t + \mathbf{W}_{hm} \mathbf{m}_t, \\ \mathbf{i}_t &= \sigma(\mathbf{W}_{ix} \mathbf{x}_t + \mathbf{W}_{im} \mathbf{m}_t), \\ \mathbf{o}_t &= \sigma(\mathbf{W}_{ox} \mathbf{x}_t + \mathbf{W}_{om} \mathbf{m}_t), \\ \mathbf{f}_t &= \sigma(\mathbf{W}_{fx} \mathbf{x}_t + \mathbf{W}_{fm} \mathbf{m}_t), \\ \mathbf{c}_t &= \mathbf{f}_t \circ \mathbf{c}_{t-1} + \mathbf{i}_t \circ \tanh(\hat{\mathbf{h}}_t), \\ \mathbf{h}_t &= \tanh(\mathbf{c}_t) \circ \mathbf{o}_t. \end{aligned}\]

Forward

multiplicativelstm(inp, (state, cstate))
multiplicativelstm(inp)

Arguments

• inp: The input to the multiplicativelstm. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• (state, cstate): A tuple containing the hidden and cell states of the MultiplicativeLSTM. They should be vectors of size hidden_size or matrices of size hidden_size x batch_size. If not provided, they are assumed to be vectors of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

MUT1(input_size => hidden_size;
    return_state=false,
    kwargs...)

Mutated unit 1 network. See MUT1Cell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.

Keyword arguments

• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true.
• return_state: Option to return the last state together with the output. Default is false.

Equations

\[\begin{aligned} z &= \sigma(W_z x_t + b_z), \\ r &= \sigma(W_r x_t + U_r h_t + b_r), \\ h_{t+1} &= \tanh(U_h (r \odot h_t) + \tanh(W_h x_t) + b_h) \odot z \\ &\quad + h_t \odot (1 - z). \end{aligned}\]

Forward

mut(inp, state)
mut(inp)

Arguments

• inp: The input to the mut. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• state: The hidden state of the MUT. If given, it is a vector of size hidden_size or a matrix of size hidden_size x batch_size. If not provided, it is assumed to be a vector of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

MUT2Cell(input_size => hidden_size;
    return_state=false,
    kwargs...)

Mutated unit 2 network. See MUT2Cell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.

Keyword arguments

• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true.
• return_state: Option to return the last state together with the output. Default is false.

Equations

\[\begin{aligned} z &= \sigma(W_z x_t + U_z h_t + b_z), \\ r &= \sigma(x_t + U_r h_t + b_r), \\ h_{t+1} &= \tanh(U_h (r \odot h_t) + W_h x_t + b_h) \odot z \\ &\quad + h_t \odot (1 - z). \end{aligned}\]

Forward

mut(inp, state)
mut(inp)

Arguments

• inp: The input to the mut. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• state: The hidden state of the MUT. If given, it is a vector of size hidden_size or a matrix of size hidden_size x batch_size. If not provided, it is assumed to be a vector of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

MUT3(input_size => hidden_size;
return_state = false, kwargs...)

Mutated unit 3 network. See MUT3Cell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.

Keyword arguments

• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true.
• return_state: Option to return the last state together with the output. Default is false.

Equations

\[\begin{aligned} z &= \sigma(W_z x_t + U_z \tanh(h_t) + b_z), \\ r &= \sigma(W_r x_t + U_r h_t + b_r), \\ h_{t+1} &= \tanh(U_h (r \odot h_t) + W_h x_t + b_h) \odot z \\ &\quad + h_t \odot (1 - z). \end{aligned}\]

Forward

mut(inp, state)
mut(inp)

Arguments

• inp: The input to the mut. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• state: The hidden state of the MUT. If given, it is a vector of size hidden_size or a matrix of size hidden_size x batch_size. If not provided, it is assumed to be a vector of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

NAS(input_size => hidden_size;
    return_state = false,
    kwargs...)

Neural Architecture Search unit. See NASCell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.

Keyword arguments

• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true.
• return_state: Option to return the last state together with the output. Default is false.

Equations

\[\begin{aligned} \text{First Layer Outputs:} & \\ o_1 &= \sigma(W_i^{(1)} x_t + W_h^{(1)} h_{t-1} + b^{(1)}), \\ o_2 &= \text{ReLU}(W_i^{(2)} x_t + W_h^{(2)} h_{t-1} + b^{(2)}), \\ o_3 &= \sigma(W_i^{(3)} x_t + W_h^{(3)} h_{t-1} + b^{(3)}), \\ o_4 &= \text{ReLU}(W_i^{(4)} x_t \cdot W_h^{(4)} h_{t-1}), \\ o_5 &= \tanh(W_i^{(5)} x_t + W_h^{(5)} h_{t-1} + b^{(5)}), \\ o_6 &= \sigma(W_i^{(6)} x_t + W_h^{(6)} h_{t-1} + b^{(6)}), \\ o_7 &= \tanh(W_i^{(7)} x_t + W_h^{(7)} h_{t-1} + b^{(7)}), \\ o_8 &= \sigma(W_i^{(8)} x_t + W_h^{(8)} h_{t-1} + b^{(8)}). \\ \text{Second Layer Computations:} & \\ l_1 &= \tanh(o_1 \cdot o_2) \\ l_2 &= \tanh(o_3 + o_4) \\ l_3 &= \tanh(o_5 \cdot o_6) \\ l_4 &= \sigma(o_7 + o_8) \\ \text{Inject Cell State:} & \\ l_1 &= \tanh(l_1 + c_{\text{state}}) \\ \text{Final Layer Computations:} & \\ c_{\text{new}} &= l_1 \cdot l_2 \\ l_5 &= \tanh(l_3 + l_4) \\ h_{\text{new}} &= \tanh(c_{\text{new}} \cdot l_5) \end{aligned}\]

Forward

nas(inp, (state, cstate))
nas(inp)

Arguments

• inp: The input to the nas. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• (state, cstate): A tuple containing the hidden and cell states of the NAS. They should be vectors of size hidden_size or matrices of size hidden_size x batch_size. If not provided, they are assumed to be vectors of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

NBR(input_size, hidden_size;
    return_state = false, kwargs...)

Recurrently neuromodulated bistable recurrent cell. See NBRCell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.

Keyword arguments

• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true

Equations

\[\begin{aligned} \mathbf{a}_t &= 1 + \tanh\left(\mathbf{U}_a \mathbf{x}_t + \mathbf{W}_a \mathbf{h}_{t-1}\right), \\ \mathbf{c}_t &= \sigma\left(\mathbf{U}_c \mathbf{x}_t + \mathbf{W}_c \mathbf{h}_{t-1}\right), \\ \mathbf{h}_t &= \mathbf{c}_t \circ \mathbf{h}_{t-1} + (1 - \mathbf{c}_t) \circ \tanh\left(\mathbf{U}_x \mathbf{x}_t + \mathbf{a}_t \circ \mathbf{h}_{t-1}\right). \end{aligned}\]

Forward

nbr(inp, state)
nbr(inp)

Arguments

• inp: The input to the nbr. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• state: The hidden state of the NBR. If given, it is a vector of size hidden_size or a matrix of size hidden_size x batch_size. If not provided, it is assumed to be a vector of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

PeepholeLSTM(input_size => hidden_size;
    return_state=false,
    kwargs...)

Peephole long short term memory network. See PeepholeLSTMCell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.

Keyword arguments

• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true.
• return_state: Option to return the last state together with the output. Default is false.

Equations

\[\begin{aligned} f_t &= \sigma_g(W_f x_t + U_f c_{t-1} + b_f), \\ i_t &= \sigma_g(W_i x_t + U_i c_{t-1} + b_i), \\ o_t &= \sigma_g(W_o x_t + U_o c_{t-1} + b_o), \\ c_t &= f_t \odot c_{t-1} + i_t \odot \sigma_c(W_c x_t + b_c), \\ h_t &= o_t \odot \sigma_h(c_t). \end{aligned}\]

Forward

peepholelstm(inp, (state, cstate))
peepholelstm(inp)

Arguments

• inp: The input to the peepholelstm. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• (state, cstate): A tuple containing the hidden and cell states of the PeepholeLSTM. They should be vectors of size hidden_size or matrices of size hidden_size x batch_size. If not provided, they are assumed to be vectors of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

RAN(input_size => hidden_size;
    return_state = false, kwargs...)

Recurrent Additive Network cell. See RANCell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.

Keyword arguments

• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true.
• return_state: Option to return the last state together with the output. Default is false.

Equations

\[\begin{aligned} \tilde{c}_t &= W_c x_t, \\ i_t &= \sigma(W_i x_t + U_i h_{t-1} + b_i), \\ f_t &= \sigma(W_f x_t + U_f h_{t-1} + b_f), \\ c_t &= i_t \odot \tilde{c}_t + f_t \odot c_{t-1}, \\ h_t &= g(c_t) \end{aligned}\]

Forward

ran(inp, (state, cstate))
ran(inp)

Arguments

• inp: The input to the ran. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• (state, cstate): A tuple containing the hidden and cell states of the RAN. They should be vectors of size hidden_size or matrices of size hidden_size x batch_size. If not provided, they are assumed to be vectors of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

RHN(input_size => hidden_size, [depth];
    return_state = false,
    kwargs...)

Recurrent highway network. See RHNCellUnit for a the unit component of this layer. See RHNCell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer
• depth: depth of the recurrence. Default is 3

Keyword arguments

• couple_carry: couples the carry gate and the transform gate. Default true
• init_kernel: initializer for the input to hidden weights
• bias: include a bias or not. Default is true
• return_state: Option to return the last state together with the output. Default is false.

Equations

\[\begin{aligned} s_{\ell}^{[t]} &= h_{\ell}^{[t]} \odot t_{\ell}^{[t]} + s_{\ell-1}^{[t]} \odot c_{\ell}^{[t]}, \\ \text{where} \\ h_{\ell}^{[t]} &= \tanh(W_h x^{[t]}\mathbb{I}_{\ell = 1} + U_{h_{\ell}} s_{\ell-1}^{[t]} + b_{h_{\ell}}), \\ t_{\ell}^{[t]} &= \sigma(W_t x^{[t]}\mathbb{I}_{\ell = 1} + U_{t_{\ell}} s_{\ell-1}^{[t]} + b_{t_{\ell}}), \\ c_{\ell}^{[t]} &= \sigma(W_c x^{[t]}\mathbb{I}_{\ell = 1} + U_{c_{\ell}} s_{\ell-1}^{[t]} + b_{c_{\ell}}) \end{aligned}\]

SCRN(input_size => hidden_size;
    init_kernel = glorot_uniform,
    init_recurrent_kernel = glorot_uniform,
    bias = true, alpha = 0.0,
    return_state = false)

Structurally contraint recurrent unit. See SCRNCell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.

Keyword arguments

• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true.
• alpha: structural contraint. Default is 0.0.
• return_state: Option to return the last state together with the output. Default is false.

Equations

\[\begin{aligned} s_t &= (1 - \alpha) W_s x_t + \alpha s_{t-1}, \\ h_t &= \sigma(W_h s_t + U_h h_{t-1} + b_h), \\ y_t &= f(U_y h_t + W_y s_t) \end{aligned}\]

Forward

scrn(inp, (state, cstate))
scrn(inp)

Arguments

• inp: The input to the scrn. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• (state, cstate): A tuple containing the hidden and cell states of the SCRN. They should be vectors of size hidden_size or matrices of size hidden_size x batch_size. If not provided, they are assumed to be vectors of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

SGRN(input_size, hidden_size;
    return_state = false, kwargs...)

Simple gated recurrent network. See SGRNCell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.

Keyword arguments

• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true

Equations

\[\begin{aligned} \mathbf{f}_t &= \sigma(\mathbf{W} \mathbf{x}_t + \mathbf{U} \mathbf{h}_{t-1} + \mathbf{b}), \\ \mathbf{i}_t &= 1 - \mathbf{f}_t, \\ \mathbf{h}_t &= \tanh\left(\mathbf{i}_t \circ (\mathbf{W} \mathbf{x}_t) + \mathbf{f}_t \circ \mathbf{h}_{t-1}\right). \end{aligned}\]

Forward

sgrn(inp, state)
sgrn(inp)

Arguments

• inp: The input to the sgrn. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• state: The hidden state of the SGRN. If given, it is a vector of size hidden_size or a matrix of size hidden_size x batch_size. If not provided, it is assumed to be a vector of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

STAR(input_size => hidden_size;
    return_state = false, kwargs...)

Stackable recurrent network. See STARCell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.

Keyword arguments

• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true.
• return_state: Option to return the last state together with the output. Default is false.

Equations

\[\begin{aligned} z_t &= \tanh(W_z x_t + b_z), \\ k_t &= \sigma(W_x x_t + W_h h_{t-1} + b_k), \\ h_t &= \tanh\left((1 - k_t) \circ h_{t-1} + k_t \circ z_t\right). \end{aligned}\]

Forward

star(inp, state)
star(inp)

Arguments

• inp: The input to the star. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• state: The hidden state of the STAR. If given, it is a vector of size hidden_size or a matrix of size hidden_size x batch_size. If not provided, it is assumed to be a vector of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

TGRU(input_size => hidden_size;
    return_state = false, kwargs...)

Strongly typed recurrent gated unit. See TGRUCell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.

Keyword arguments

• return_state: Option to return the last state together with the output. Default is false.
• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true.

Equations

\[\begin{aligned} z_t &= \mathbf{V}_z \mathbf{x}_{t-1} + \mathbf{W}_z \mathbf{x}_t + \mathbf{b}_z \\ f_t &= \sigma (\mathbf{V}_f \mathbf{x}_{t-1} + \mathbf{W}_f \mathbf{x}_t + \mathbf{b}_f) \\ o_t &= \tau (\mathbf{V}_o \mathbf{x}_{t-1} + \mathbf{W}_o \mathbf{x}_t + \mathbf{b}_o) \\ h_t &= f_t \odot h_{t-1} + z_t \odot o_t \end{aligned}\]

Forward

tgru(inp, state)
tgru(inp)

Arguments

• inp: The input to the tgru. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• state: The hidden state of the TGRU. If given, it is a vector of size hidden_size or a matrix of size hidden_size x batch_size. If not provided, it is assumed to be a vector of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

TLSTM(input_size => hidden_size;
    return_state = false, kwargs...)

Strongly typed long short term memory. See TLSTMCell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.

Keyword arguments

• return_state: Option to return the last state together with the output. Default is false.
• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true.

Equations

\[\begin{aligned} z_t &= \mathbf{V}_z \mathbf{x}_{t-1} + \mathbf{W}_z \mathbf{x}_t + \mathbf{b}_z \\ f_t &= \sigma (\mathbf{V}_f \mathbf{x}_{t-1} + \mathbf{W}_f \mathbf{x}_t + \mathbf{b}_f) \\ o_t &= \tau (\mathbf{V}_o \mathbf{x}_{t-1} + \mathbf{W}_o \mathbf{x}_t + \mathbf{b}_o) \\ c_t &= f_t \odot c_{t-1} + (1 - f_t) \odot z_t \\ h_t &= c_t \odot o_t \end{aligned}\]

Forward

tlstm(inp, state)
tlstm(inp)

Arguments

• inp: The input to the tlstm. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• state: The hidden state of the TLSTM. If given, it is a vector of size hidden_size or a matrix of size hidden_size x batch_size. If not provided, it is assumed to be a vector of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden states new_states and the last state of the iteration.

TRNN(input_size => hidden_size, [activation];
    return_state = false, kwargs...)

Strongly typed recurrent unit. See TRNNCell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.
• activation: activation function. Default is tanh.

Keyword arguments

• return_state: Option to return the last state together with the output. Default is false.
• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true.

Equations

\[\begin{aligned} z_t &= \mathbf{W} x_t \\ f_t &= \sigma (\mathbf{V} x_t + b) \\ h_t &= f_t \odot h_{t-1} + (1 - f_t) \odot z_t \end{aligned}\]

Forward

trnn(inp, state)
trnn(inp)

Arguments

• inp: The input to the trnn. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• state: The hidden state of the TRNN. If given, it is a vector of size hidden_size or a matrix of size hidden_size x batch_size. If not provided, it is assumed to be a vector of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

UnICORNN(input_size => hidden_size, [dt];
    alpha=0.0, return_state=false, init_kernel = glorot_uniform,
    init_recurrent_kernel = glorot_uniform, bias = true)

Undamped independent controlled oscillatory recurrent neural network. See UnICORNNCell for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.
• dt: time step. Default is 1.0.

Keyword arguments

• alpha: Control parameter. Default is 0.0.
• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true.
• return_state: Option to return the last state together with the output. Default is false.

Equations

\[\begin{aligned} y_n &= y_{n-1} + \Delta t \, \hat{\sigma}(c) \odot z_n, \\ z_n &= z_{n-1} - \Delta t \, \hat{\sigma}(c) \odot \left[ \sigma \left( w \odot y_{n-1} + V y_{n-1} + b \right) + \alpha y_{n-1} \right]. \end{aligned}\]

Forward

unicornn(inp, (state, zstate))
unicornn(inp)

Arguments

• inp: The input to the unicornn. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• (state, cstate): A tuple containing the hidden and cell states of the UnICORNN. They should be vectors of size hidden_size or matrices of size hidden_size x batch_size. If not provided, they are assumed to be vectors of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.

WMCLSTM(input_size => hidden_size;
    return_state=false,
    kwargs...)

Long short term memory cell with working memory connections. See WMCLSTM for a layer that processes a single sequence.

Arguments

• input_size => hidden_size: input and inner dimension of the layer.

Keyword arguments

• init_kernel: initializer for the input to hidden weights. Default is glorot_uniform.
• init_recurrent_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• init_memory_kernel: initializer for the hidden to hidden weights. Default is glorot_uniform.
• bias: include a bias or not. Default is true.
• return_state: Option to return the last state together with the output. Default is false.

Equations

\[\begin{aligned} \mathbf{i}_t &= \sigma\left(\mathbf{W}_{ix} \mathbf{x}_t + \mathbf{W}_{ih} \mathbf{h}_{t-1} + \tanh(\mathbf{W}_{ic} \mathbf{c}_{t-1}) + \mathbf{b}_i\right), \\ \mathbf{f}_t &= \sigma\left(\mathbf{W}_{fx} \mathbf{x}_t + \mathbf{W}_{fh} \mathbf{h}_{t-1} + \tanh(\mathbf{W}_{fc} \mathbf{c}_{t-1}) + \mathbf{b}_f\right), \\ \mathbf{o}_t &= \sigma\left(\mathbf{W}_{ox} \mathbf{x}_t + \mathbf{W}_{oh} \mathbf{h}_{t-1} + \tanh(\mathbf{W}_{oc} \mathbf{c}_t) + \mathbf{b}_o\right), \\ \mathbf{c}_t &= \mathbf{f}_t \circ \mathbf{c}_{t-1} + \mathbf{i}_t \circ \sigma_c(\mathbf{W}_{c} \mathbf{x}_t + \mathbf{b}_c), \\ \mathbf{h}_t &= \mathbf{o}_t \circ \sigma_h(\mathbf{c}_t). \end{aligned}\]

Forward

wmclstm(inp, (state, cstate))
wmclstm(inp)

Arguments

• inp: The input to the wmclstm. It should be a vector of size input_size x len or a matrix of size input_size x len x batch_size.
• (state, cstate): A tuple containing the hidden and cell states of the WMCLSTM. They should be vectors of size hidden_size or matrices of size hidden_size x batch_size. If not provided, they are assumed to be vectors of zeros, initialized by Flux.initialstates.

Returns

• New hidden states new_states as an array of size hidden_size x len x batch_size. When return_state = true it returns a tuple of the hidden stats new_states and the last state of the iteration.