torchrecurrent.UnICORNN#
- class torchrecurrent.UnICORNN(input_size, hidden_size, num_layers=1, dropout=0.0, batch_first=False, **kwargs)[source]#
Multi-layer Undamped Independent Controlled Oscillatory RNN (UnICORNN).
[arXiv]
Each layer consists of a
UnICORNNCell, which maintains two coupled state vectors, the hidden state \(h(t)\) and the control state \(z(t)\), updated as:\[\begin{split}\begin{aligned} h(t) &= h(t-1) + \Delta t \, \hat{\sigma}(w_{ch}) \circ z(t), \\ z(t) &= z(t-1) - \Delta t \, \hat{\sigma}(w_{ch}) \circ \Bigl[\sigma(W_{hh} h(t-1) + W_{ih} x(t) + b_{ih}) + \alpha h(t-1)\Bigr], \end{aligned}\end{split}\]where \(\Delta t\) is the integration step
dt, \(\alpha\) is a leakage constant, \(\sigma\) is the sigmoid, and \(\circ\) is the elementwise product.- Parameters:
input_size – Number of expected features in the input x.
hidden_size – Number of features in the hidden state h (and control z).
num_layers – Number of stacked recurrent layers. Default: 1
dropout – If non-zero, adds dropout after each layer (except the last). Default: 0
batch_first – If
True, inputs and outputs are in (batch, seq, feature) format instead of (seq, batch, feature). Default: Falsebias – If
False, disables input bias b_{ih}. Default: Truerecurrent_bias – If
False, disables hidden bias b_{hh}. Default: Truekernel_init – Initializer for W_{ih}. Default:
torch.nn.init.xavier_uniform_()recurrent_kernel_init – Initializer for W_{hh}. Default:
torch.nn.init.xavier_uniform_()control_kernel_init – Initializer for w_{ch}. Default:
torch.nn.init.normal_()bias_init – Initializer for b_{ih} when
bias=True. Default:torch.nn.init.zeros_()recurrent_bias_init – Initializer for b_{hh} when
recurrent_bias=True. Default:torch.nn.init.zeros_()dt – Integration step \(\Delta t\). Default: 1.0
alpha – Leakage coefficient \(\alpha\). Default: 0.0
device – Desired device of parameters.
dtype – Desired floating point type of parameters.
- Inputs: input, (h_0, z_0)
input: tensor of shape (L, H_in) for unbatched input, (L, N, H_in) when
batch_first=False, or (N, L, H_in) whenbatch_first=Truecontaining input sequence features.h_0: tensor of shape (num_layers, H_out) for unbatched input or (num_layers, N, H_out) containing the initial hidden state. Defaults to zeros if not provided.
z_0: tensor of the same shape as h_0, containing the initial control state. Defaults to zeros if not provided.
Where:
\[\begin{split}\begin{aligned} N &= \text{batch size} \\ L &= \text{sequence length} \\ H_{in} &= \text{input size} \\ H_{out} &= \text{hidden size} \end{aligned}\end{split}\]- Outputs: output, (h_n, z_n)
output: tensor of shape (L, H_out) for unbatched input, (L, N, H_out) when
batch_first=False, or (N, L, H_out) whenbatch_first=Truecontaining the hidden states from the last layer at each timestep.h_n: final hidden state for each layer, shape (num_layers, H_out) (unbatched) or (num_layers, N, H_out).
z_n: final control state for each layer, same shape as h_n.
- cells.{k}.weight_ih
input–hidden weights of the \(k\)-th layer, shape (hidden_size, input_size) for k=0, otherwise (hidden_size, hidden_size).
- cells.{k}.weight_hh
hidden–hidden weights of the \(k\)-th layer, shape (hidden_size, hidden_size).
- cells.{k}.weight_ch
control weights of the \(k\)-th layer, shape (hidden_size,).
- cells.{k}.bias_ih
input bias of the \(k\)-th layer, shape (hidden_size,) if
bias=True.
- cells.{k}.bias_hh
hidden bias of the \(k\)-th layer, shape (hidden_size,) if
recurrent_bias=True.
See also
Examples:
>>> rnn = UnICORNN(10, 20, num_layers=2, dt=0.5, alpha=0.1) >>> x = torch.randn(5, 3, 10) # (seq_len, batch, input_size) >>> h0 = torch.zeros(2, 3, 20) >>> z0 = torch.zeros(2, 3, 20) >>> out, (hn, zn) = rnn(x, (h0, z0))
- __init__(input_size, hidden_size, num_layers=1, dropout=0.0, batch_first=False, **kwargs)[source]#
Initialize internal Module state, shared by both nn.Module and ScriptModule.
Methods
__init__(input_size, hidden_size[, ...])Initialize internal Module state, shared by both nn.Module and ScriptModule.
add_module(name, module)Add a child module to the current module.
apply(fn)Apply
fnrecursively to every submodule (as returned by.children()) as well as self.bfloat16()Casts all floating point parameters and buffers to
bfloat16datatype.buffers([recurse])Return an iterator over module buffers.
children()Return an iterator over immediate children modules.
compile(*args, **kwargs)Compile this Module's forward using
torch.compile().cpu()Move all model parameters and buffers to the CPU.
cuda([device])Move all model parameters and buffers to the GPU.
double()Casts all floating point parameters and buffers to
doubledatatype.eval()Set the module in evaluation mode.
extra_repr()Return the extra representation of the module.
float()Casts all floating point parameters and buffers to
floatdatatype.forward(inp[, state])Define the computation performed at every call.
get_buffer(target)Return the buffer given by
targetif it exists, otherwise throw an error.get_extra_state()Return any extra state to include in the module's state_dict.
get_parameter(target)Return the parameter given by
targetif it exists, otherwise throw an error.get_submodule(target)Return the submodule given by
targetif it exists, otherwise throw an error.half()Casts all floating point parameters and buffers to
halfdatatype.initialize_cells(cell_class, **kwargs)Helper method to initialize cells for the derived recurrent layer class.
ipu([device])Move all model parameters and buffers to the IPU.
load_state_dict(state_dict[, strict, assign])Copy parameters and buffers from
state_dictinto this module and its descendants.modules()Return an iterator over all modules in the network.
mtia([device])Move all model parameters and buffers to the MTIA.
named_buffers([prefix, recurse, ...])Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself.
named_children()Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself.
named_modules([memo, prefix, remove_duplicate])Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself.
named_parameters([prefix, recurse, ...])Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself.
parameters([recurse])Return an iterator over module parameters.
register_backward_hook(hook)Register a backward hook on the module.
register_buffer(name, tensor[, persistent])Add a buffer to the module.
register_forward_hook(hook, *[, prepend, ...])Register a forward hook on the module.
register_forward_pre_hook(hook, *[, ...])Register a forward pre-hook on the module.
register_full_backward_hook(hook[, prepend])Register a backward hook on the module.
register_full_backward_pre_hook(hook[, prepend])Register a backward pre-hook on the module.
register_load_state_dict_post_hook(hook)Register a post-hook to be run after module's
load_state_dict()is called.register_load_state_dict_pre_hook(hook)Register a pre-hook to be run before module's
load_state_dict()is called.register_module(name, module)Alias for
add_module().register_parameter(name, param)Add a parameter to the module.
register_state_dict_post_hook(hook)Register a post-hook for the
state_dict()method.register_state_dict_pre_hook(hook)Register a pre-hook for the
state_dict()method.requires_grad_([requires_grad])Change if autograd should record operations on parameters in this module.
set_extra_state(state)Set extra state contained in the loaded state_dict.
set_submodule(target, module[, strict])Set the submodule given by
targetif it exists, otherwise throw an error.share_memory()See
torch.Tensor.share_memory_().state_dict(*args[, destination, prefix, ...])Return a dictionary containing references to the whole state of the module.
to(*args, **kwargs)Move and/or cast the parameters and buffers.
to_empty(*, device[, recurse])Move the parameters and buffers to the specified device without copying storage.
train([mode])Set the module in training mode.
type(dst_type)Casts all parameters and buffers to
dst_type.xpu([device])Move all model parameters and buffers to the XPU.
zero_grad([set_to_none])Reset gradients of all model parameters.
Attributes
T_destinationcall_super_initdump_patchesinput_sizehidden_sizebiasdropoutbatch_firsttraining