List of transformers

Torchflows supports several transformers to be used in autoregressive and multiscale normalizing flows.

Linear transformers

class torchflows.bijections.finite.autoregressive.transformers.linear.affine.Affine(event_shape: Size, min_scale: float = 0.001)

Affine transformer.

Computes z = alpha * x + beta, where alpha > 0 and -inf < beta < inf. Alpha and beta have the same shape as x, i.e. the computation is performed elementwise. We use a minimum permitted scale m, 0 < m <= alpha, for numerical stability

class torchflows.bijections.finite.autoregressive.transformers.linear.affine.InverseAffine(event_shape: Size | Tuple[int, ...], **kwargs)

class torchflows.bijections.finite.autoregressive.transformers.linear.affine.Shift(event_shape: Size, **kwargs)

class torchflows.bijections.finite.autoregressive.transformers.linear.convolution.Invertible1x1ConvolutionTransformer(event_shape: Size | Tuple[int, ...])

Invertible 1x1 convolution.

This transformer receives as input a batch of images x with x.shape (*batch_shape, channels, *image_dimensions)

and parameters h for an invertible linear transform of the channels with h.shape = (*batch_shape, *parameter_shape).

Note that image_dimensions can be a shape with arbitrarily ordered dimensions. In fact, it is not required that the image is two-dimensional. Voxels with shape (channels, height, width, depth)

are also supported, as well as tensors with more general shapes.

class torchflows.bijections.finite.autoregressive.transformers.linear.matrix.LUTransformer(event_shape: Size | Tuple[int, ...])

Linear transformer with LUx = y.

It is assumed that all diagonal elements of L are 1.

Spline transformers

class torchflows.bijections.finite.autoregressive.transformers.spline.linear.Linear(event_shape: Size | Tuple[int, ...], boundary: float = 50.0, **kwargs)

class torchflows.bijections.finite.autoregressive.transformers.spline.linear_rational.LinearRational(event_shape: Size | Tuple[int, ...], boundary: float = 50.0, **kwargs)

class torchflows.bijections.finite.autoregressive.transformers.spline.rational_quadratic.RationalQuadratic(event_shape: Size | Tuple[int, ...], boundary: float = 50.0, **kwargs)

Neural Spline Flows - Durkan et al. 2019 RQ splines are more prone to numerical instabilities when stacked in a composition than affine transforms. This becomes a problem when used in autoregressive flows, since the inverse/forward passes (for MAF/IAF respectively) require n_dim spline computations.

Combination transformers

class torchflows.bijections.finite.autoregressive.transformers.combination.sigmoid.Sigmoid(event_shape: Size | Tuple[int, ...], hidden_dim: int = None, min_scale: float = 0.001)

Applies z = inv_sigmoid(w.T @ sigmoid(a * x + b)) where a > 0, w > 0 and sum(w) = 1. Note: w, a, b are vectors, so multiplication a * x is broadcast.

class torchflows.bijections.finite.autoregressive.transformers.combination.sigmoid.DenseSigmoid(event_shape: Size | Tuple[int, ...], n_dense_layers: int = 1, hidden_size: int = 8)

Apply y = f1 circ f2 circ … circ fn (x) where * f1 is a dense sigmoid inner transform which maps from 1 to h dimensions; * fn is a dense sigmoid inner transform which maps from h to 1 dimensions; * fi (for all other i) is a dense sigmoid inner transform which maps from h to h dimensions.

class torchflows.bijections.finite.autoregressive.transformers.combination.sigmoid.DeepSigmoid(event_shape: Size | Tuple[int, ...], n_hidden_layers: int = 2, **kwargs)

class torchflows.bijections.finite.autoregressive.transformers.combination.sigmoid.DeepDenseSigmoid(event_shape: Size | Tuple[int, ...], n_hidden_layers: int = 2, **kwargs)

Integration transformers

class torchflows.bijections.finite.autoregressive.transformers.integration.unconstrained_monotonic_neural_network.UnconstrainedMonotonicNeuralNetwork(event_shape: Size | Tuple[int, ...], n_hidden_layers: int = None, hidden_dim: int = None)

Unconstrained monotonic neural network transformer.

The unconstrained monotonic neural network is a neural network with positive weights and positive activation

function derivatives. These two conditions ensure its invertibility.