torchrecurrent.SCRNCell#
- class torchrecurrent.SCRNCell(input_size, hidden_size, bias=True, recurrent_bias=True, context_bias=True, kernel_init=<function xavier_uniform_>, recurrent_kernel_init=<function xavier_uniform_>, context_kernel_init=<function normal_>, bias_init=<function zeros_>, recurrent_bias_init=<function zeros_>, context_bias_init=<function zeros_>, alpha=0.5, device=None, dtype=None)[source]#
A Structurally Constrained Recurrent Network (SCRN) cell.
[arXiv]
\[\begin{split}\begin{aligned} \mathbf{s}(t) &= (1 - \alpha)\,\bigl(\mathbf{W}_{ih}^{s}\,\mathbf{x}(t) + \mathbf{b}_{ih}^{s}\bigr) + \alpha\,\mathbf{s}(t-1), \\ \mathbf{h}(t) &= \sigma\Bigl( \mathbf{W}_{ch}^{h}\,\mathbf{s}(t) + \mathbf{b}_{ch}^{h} + \mathbf{W}_{ih}^{h}\,\mathbf{x}(t) + \mathbf{b}_{ih}^{h} + \mathbf{W}_{hh}^{h}\,\mathbf{h}(t-1) + \mathbf{b}_{hh}^{h} \Bigr), \\ \mathbf{y}(t) &= f\Bigl( \mathbf{W}_{ch}^{y}\,\mathbf{s}(t) + \mathbf{b}_{ch}^{y} + \mathbf{W}_{hh}^{y}\,\mathbf{h}(t) + \mathbf{b}_{hh}^{y} \Bigr) \end{aligned}\end{split}\]where \(\sigma\) is the sigmoid activation and \(f\) is an optional output nonlinearity.
- Parameters:
input_size – Number of input features
hidden_size – Number of hidden (and context) features
bias – If
False
, the layer does not use input-side biasb_ih
. Default:True
recurrent_bias – If
False
, the layer does not use recurrent biasb_hh
. Default:True
context_bias – If
False
, the layer does not use context biasb_ch
. Default:True
kernel_init – Initializer for
weight_ih
. Default:torch.nn.init.xavier_uniform_()
recurrent_kernel_init – Initializer for
weight_hh
. Default:torch.nn.init.xavier_uniform_()
context_kernel_init – Initializer for
weight_ch
. Default:torch.nn.init.normal_()
bias_init – Initializer for
bias_ih
whenbias=True
. Default:torch.nn.init.zeros_()
recurrent_bias_init – Initializer for
bias_hh
whenrecurrent_bias=True
. Default:torch.nn.init.zeros_()
context_bias_init – Initializer for
bias_ch
whencontext_bias=True
. Default:torch.nn.init.zeros_()
alpha – Context interpolation parameter \(\alpha\). Default:
0.5
device – The desired device of parameters
dtype – The desired floating point type of parameters
- Inputs: input, (h_0, s_0)
input of shape
(batch, input_size)
or(input_size,)
: tensor containing input featuresh_0 of shape
(batch, hidden_size)
or(hidden_size,)
: initial hidden states_0 of shape
(batch, hidden_size)
or(hidden_size,)
: initial context state
If (h_0, s_0) is not provided, both default to zero.
- Outputs: (h_1, s_1)
h_1 of shape
(batch, hidden_size)
or(hidden_size,)
: next hidden states_1 of shape
(batch, hidden_size)
or(hidden_size,)
: next context state
- Variables:
weight_ih – input–hidden weights, of shape
(2*hidden_size, input_size)
weight_hh – hidden–hidden weights, of shape
(2*hidden_size, hidden_size)
weight_ch – context–hidden weights, of shape
(2*hidden_size, hidden_size)
bias_ih – input biases, of shape
(2*hidden_size)
ifbias=True
bias_hh – hidden biases, of shape
(2*hidden_size)
ifrecurrent_bias=True
bias_ch – context biases, of shape
(2*hidden_size)
ifcontext_bias=True
alpha – learnable context interpolation scalar
Examples:
>>> cell = SCRNCell(10, 20, alpha=0.5) >>> x = torch.randn(6, 3, 10) # (time, batch, input_size) >>> h = torch.zeros(3, 20) # (batch, hidden_size) >>> s = torch.zeros(3, 20) # (batch, hidden_size) context >>> hs = [] >>> for t in range(x.size(0)): ... h, s = cell(x[t], (h, s)) ... hs.append(h) >>> hs = torch.stack(hs, dim=0) # (time, batch, hidden_size)
- __init__(input_size, hidden_size, bias=True, recurrent_bias=True, context_bias=True, kernel_init=<function xavier_uniform_>, recurrent_kernel_init=<function xavier_uniform_>, context_kernel_init=<function normal_>, bias_init=<function zeros_>, recurrent_bias_init=<function zeros_>, context_bias_init=<function zeros_>, alpha=0.5, device=None, dtype=None)[source]#
Initialize internal Module state, shared by both nn.Module and ScriptModule.
Methods
__init__
(input_size, hidden_size[, bias, ...])Initialize internal Module state, shared by both nn.Module and ScriptModule.
add_module
(name, module)Add a child module to the current module.
apply
(fn)Apply
fn
recursively to every submodule (as returned by.children()
) as well as self.bfloat16
()Casts all floating point parameters and buffers to
bfloat16
datatype.buffers
([recurse])Return an iterator over module buffers.
children
()Return an iterator over immediate children modules.
compile
(*args, **kwargs)Compile this Module's forward using
torch.compile()
.cpu
()Move all model parameters and buffers to the CPU.
cuda
([device])Move all model parameters and buffers to the GPU.
double
()Casts all floating point parameters and buffers to
double
datatype.eval
()Set the module in evaluation mode.
extra_repr
()Return the extra representation of the module.
float
()Casts all floating point parameters and buffers to
float
datatype.forward
(inp[, state])Run one step of the recurrent cell.
get_buffer
(target)Return the buffer given by
target
if it exists, otherwise throw an error.get_extra_state
()Return any extra state to include in the module's state_dict.
get_parameter
(target)Return the parameter given by
target
if it exists, otherwise throw an error.get_submodule
(target)Return the submodule given by
target
if it exists, otherwise throw an error.half
()Casts all floating point parameters and buffers to
half
datatype.init_weights
()ipu
([device])Move all model parameters and buffers to the IPU.
load_state_dict
(state_dict[, strict, assign])Copy parameters and buffers from
state_dict
into this module and its descendants.modules
()Return an iterator over all modules in the network.
mtia
([device])Move all model parameters and buffers to the MTIA.
named_buffers
([prefix, recurse, ...])Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself.
named_children
()Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself.
named_modules
([memo, prefix, remove_duplicate])Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself.
named_parameters
([prefix, recurse, ...])Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself.
parameters
([recurse])Return an iterator over module parameters.
register_backward_hook
(hook)Register a backward hook on the module.
register_buffer
(name, tensor[, persistent])Add a buffer to the module.
register_forward_hook
(hook, *[, prepend, ...])Register a forward hook on the module.
register_forward_pre_hook
(hook, *[, ...])Register a forward pre-hook on the module.
register_full_backward_hook
(hook[, prepend])Register a backward hook on the module.
register_full_backward_pre_hook
(hook[, prepend])Register a backward pre-hook on the module.
register_load_state_dict_post_hook
(hook)Register a post-hook to be run after module's
load_state_dict()
is called.register_load_state_dict_pre_hook
(hook)Register a pre-hook to be run before module's
load_state_dict()
is called.register_module
(name, module)Alias for
add_module()
.register_parameter
(name, param)Add a parameter to the module.
register_state_dict_post_hook
(hook)Register a post-hook for the
state_dict()
method.register_state_dict_pre_hook
(hook)Register a pre-hook for the
state_dict()
method.requires_grad_
([requires_grad])Change if autograd should record operations on parameters in this module.
set_extra_state
(state)Set extra state contained in the loaded state_dict.
set_submodule
(target, module[, strict])Set the submodule given by
target
if it exists, otherwise throw an error.share_memory
()See
torch.Tensor.share_memory_()
.state_dict
(*args[, destination, prefix, ...])Return a dictionary containing references to the whole state of the module.
to
(*args, **kwargs)Move and/or cast the parameters and buffers.
to_empty
(*, device[, recurse])Move the parameters and buffers to the specified device without copying storage.
train
([mode])Set the module in training mode.
type
(dst_type)Casts all parameters and buffers to
dst_type
.uses_double_state
()Return True if forward returns (h, c), else just h.
xpu
([device])Move all model parameters and buffers to the XPU.
zero_grad
([set_to_none])Reset gradients of all model parameters.
Attributes
T_destination
call_super_init
dump_patches
weight_ih
weight_hh
weight_ch
bias_ih
bias_hh
bias_ch
input_size
hidden_size
bias
recurrent_bias
training