And ILGPU's might, a sight to be.

From JSON springs architecture grand,

A computational graph, complex and planned.

In the realm of AI, it takes a stand,

A tool of power, in your hand.

So sing we now of PRAD's great glory,

Etched forever in code's vast story.

API Documentation

Each operation in PRAD is embodied as a node with forward and backward functions, facilitating the efficient calculation of derivatives. This design is particularly beneficial for large-scale problems and complex neural network architectures, where computational efficiency is paramount.

A standout feature of PRAD is its innovative use of the visitor pattern. The library includes a specialized 'Neural Network Visitor' which traverses neural network nodes across different threads. This visitor is tasked with gradient accumulation on nodes shared across multiple threads, allowing for parallelized computations while maintaining consistency and avoiding race conditions.

Moreover, PRAD introduces a data-driven approach to neural network architecture design, allowing for rapid prototyping and experimentation. The library leverages the power of ILGPU, a high-performance GPU-accelerated library for .NET programs, to perform complex computations on the GPU, further enhancing its performance and scalability.

PRAD's dynamic computational graph, constructed from JSON architecture, allows for the efficient computation of gradients, a crucial aspect of the backpropagation process used in training neural networks. This unique blend of features makes PRAD an efficient, scalable, and groundbreaking automatic differentiation solution.

Prerequisites

Download and install the Cuda Toolkit 12.0 if you want to use the CudaMatrixMultiplyOperation.

Supported Operations

Regular Operations

AddGaussianNoiseOperation

AmplifiedSigmoidOperation - Used for gradient amplification.

ApplyDropoutOperation

BatchNormalizationOperation

CudaMatrixMultiplyOperation - Leverages NVIDIA GPUs for fast computation.

DualWeightedOperation

EmbeddingOperation - Used for word or subword embeddings.

GELUOperation

GpuMatrixMultiplyOperation - Leverages NVIDIA, AMD, and Intel GPUs for fast computation.

HadamardProductOperation

LayerNormalizationOperation

LeakyReLUOperation

MatrixAddOperation

MatrixAddBroadcastingOperation

MatrixAddThreeOperation

MatrixAverageOperation

MatrixBroadcastOperation

MatrixConcatenateOperation

MatrixMultiplyOperation

MatrixMultiplyScalarOperation

MatrixSumOperation

MatrixTransposeOperation

MatrixVectorConcatenateOperation

ReLUOperation

RMSNormOperation

ScaleAndShiftOperation

SigmoidOperation

SineSoftmaxOperation - Reduces the vanishing gradient problem with traditional softmax.

SoftmaxOperation

StretchedSigmoidOperation

SwigLUOperation

SwishOperation

TanhOperation

Deep Operations

These types of operations operate on instances of the DeepMatrix class which is a 3-D matrix. The first dimension is the channel size and the second and third dimensions are the row and column sizes respectively.

DeepBatchNormalizationOperation

DeepConvolutionOperation

DeepLeakyReLUOperation

DeepMaxPoolOperation

DeepReLUOperation

DeepScaleAndShiftOperation

FlattenOperation

Neural Network Parameters

Each neural network base class has a set of parameters that can be used to configure the neural network. They are as follows:

Usage

Build out your neural network model

Each model element group needs a unique identifier, a size array, and an initialization type.

The three possible initialization types are Xavier, He, and Zeroes.

The group consists of weights, gradients, and moments for Adam optimization.

The initialization type is used to initialize the model element group's weights.

The model element group's elements are stored in a matrix whose size is specified by the size array.

In this example, for the hidden layer, the first dimension is the number of layers and the second and third dimensions are the row and column sizes respectively.

Create an architecture JSON file

Here is an example:

Each operation in the JSON represents a step in a computational graph used for automatic differentiation. Here's what each field means:

• "timeSteps": This is an array that represents the sequence of computational operations. Each element in the array is an object that corresponds to a computational timestep.

• "startOperations": This is an array that defines the initial operations for the current timestep.

• "layers": This represents a sequence of operations corresponding to the layers of the network. Each operation in a layer is a step in the computation, and the order of operations matters, as some operations depend on the results of previous operations.

• "endOperations": This is an array that defines the final operations for the current timestep.

Each operation object in "startOperations", "layers", or "endOperations" has several fields:

• "id": This is a unique identifier for the operation.

• "description": This is a human-readable description of what the operation does.

• "type": This specifies the type of the operation.

• "inputs": This is an array that lists the inputs for the operation. These are the identifiers of other nodes in the computational graph. The identifiers are either defined in the computational graph, or in operation finders declared in code when building an instance of the ComputationGraph class, for example the SelfAttentionMultiLayerLSTMComputationGraph class which is a subclass of ComputationGraph.

• "gradientResultTo": This is an array that specifies where the results of the backward pass (i.e., the computed gradients) should be stored. A null value means that the gradient with respect to the input is not stored. There is an implicit mapping between the gradient and the input based on its position in the array.

• "setResultTo": This is used to store the result of the operation for later use.

The JSON defines the steps in a machine learning model's forward pass and also specifies how the backward pass (which computes gradients for optimization) should be carried out.

By defining the operations and their connections in a JSON file, the graph can be easily constructed and modified, and the computations can be automatically differentiated and parallelized. This representation makes it possible to define a wide variety of models in a modular way, using the building blocks provided by the library.

Instantiate the architecture

Use a JSON serialization library like Newtonsoft.JSON to deserialize the JSON file to a JsonArchitecture object.

There are other JSON architectures available as well.

These include the 'NestedLayersJsonArchitecture', 'DualLayersJsonArchitecture', and 'TripleLayersJsonArchitecture'.

Instantiate the computational graph

Operation finders are a key component used to define and locate different operations in a neural network's computational graph. They're essentially functions that link to specific operations at different layers or time steps within the network. This is achieved by mapping string identifiers (IDs) to these operations, which are then used within a JSON architecture to establish the network's structure and sequence of computations. For example, an operation finder could link to a matrix multiplication operation in a specific layer of the network. By using these operation finders, developers can effectively manage complex computational graphs.

Populate the backward dependency counts

Then populate the backward dependency counts by running the following code. It only has to be run once to set up the backward dependency counts.

Run the forward pass

Create a loss function

Create a loss function like mean squared error, cross-entropy loss or using policy gradient methods.

Then calculate the gradient of the loss with respect to the output.

Plug the result in as the backward input for the backward start operation.

Run the backward pass utilizing inherent parallelization

Clip the gradients

Update the weights

Using CUDA operations

Customization

The ParallelReverseAutoDiff (PRAD) library is designed with customization at its core.

Understanding that the world of machine learning and neural networks is continually evolving, the library allows users to define their own neural network operations.

This feature provides an immense level of flexibility and control over the architecture and behavior of the networks, making it adaptable to both traditional and experimental models.

Custom Neural Network Operations

One of the standout features of PRAD is the ability to create custom operations.

These operations can encapsulate any computation or processing steps, including but not limited to, complex forward and backward calculations, and operations involving matrices, vectors, or scalars.

Creating a custom operation requires extending the Operation abstract class, which involves implementing two key methods:

• Forward(): This method is used to describe how your operation behaves during the forward pass of the neural network. It takes as input the relevant data, processes it as per the custom-defined operation, and produces the output.

• Backward(): This method is responsible for defining how your operation behaves during the backward pass of the neural network, i.e., how it contributes to the gradients during backpropagation. It receives the gradient of the output and uses it to compute the gradients of its inputs.

Let's look at an example custom operation, MatrixAverageOperation, which calculates the average of feature vectors across a matrix:

In this example, the Forward method calculates the average of the features for each path, while the Backward method spreads the gradient evenly across the features.

This level of customization allows PRAD to be a versatile tool in the field of machine learning, capable of being tailored to a wide range of tasks, datasets, and innovative architectures.