Parallel Reverse Mode Automatic Differentiation in C#
ParallelAutoDiff is a thread-safe C# library for reverse mode automatic differentiation, optimized for parallel computation. It leverages semaphores and locks to coordinate between threads, ensuring accuracy during gradient accumulation. Each operation in the library is implemented as a node with a forward and a backward function, facilitating efficient calculation of derivatives. A unique aspect of this library is its use of the visitor pattern: it includes a specialized 'Neural Network Visitor' which traverses neural network nodes across different threads. This visitor is responsible for gradient accumulation on nodes shared across multiple threads. This design allows for parallelized computations while maintaining consistency and avoiding race conditions. The result is an efficient, scalable automatic differentiation solution, ideal for machine learning applications and neural network training.
Prerequisites
Download and install the Cuda Toolkit 12.0 if you want to use the CudaMatrixMultiplyOperation.
Supported Operations
AmplifiedSigmoidOperation - Used for gradient amplification.
ApplyDropoutOperation
CudaMatrixMultiplyOperation - Leverages the GPU for fast computation.
Use a JSON serialization library like Newtonsoft.JSON to deserialize the JSON file to a JSONArchitecure object.
Instantiate and populate the operations
Instantiate each operation based on its type. Then set the Next property of each operation to be the next operation in the forward pass.
Add each operation that is backwards in the computation graph to the BackwardAdjacentOperations property of an operation. BackwardAdjacentOperations is just a list of operations.
Add instances of the gradients to send the result to, to the GradientDestinations array. If there is no gradient result for a certain input, add null.
Then populate the backward dependency counts by running the following code. It only has to be run once to set up the backward dependency counts.
for (int t = numTimeSteps - 1; t >= 0; t--) // if there are multiple timesteps
{
backwardStartOperation = operationsMap[$"output_t_{t}"]; // the backward start operation
OperationGraphVisitor opVisitor = new OperationGraphVisitor(Guid.NewGuid().ToString(), backwardStartOperation, t);
await opVisitor.TraverseAsync(); // sets the backward dependency counts
await opVisitor.ResetVisitedCountsAsync(backwardStartOperation);
}
Run the forward pass
var op = startOperation; // the start operation
IOperation currOp = null;
do
{
var parameters = LookupParameters(op); // lookup the parameters
op.OperationType.GetMethod("Forward").Invoke(op, parameters); // call the forward function
if (op.ResultToName != null)
{
op.ResultTo(NameToValueFunc(op.ResultToName)); // send the result to the appropriate object
}
operationsMap[op.SpecificId] = op;
currOp = op;
if (op.HasNext)
op = op.Next;
} while (currOp.Next != null);
Run the backward pass utilizing inherent parallelization
for (int t = numTimeSteps - 1; t >= 0; t--)
{
backwardStartOperation = operationsMap[$"output_t_{t}"];
if (gradientOfLossWrtOutput[t][0] != 0.0d)
{
var backwardInput = new Matrix(1, 1);
backwardInput[0] = gradientOfLossWrtOutput[t];
backwardStartOperation.BackwardInput = backwardInput;
OperationNeuralNetworkVisitor opVisitor = new OperationNeuralNetworkVisitor(Guid.NewGuid().ToString(), backwardStartOperation, t);
await opVisitor.TraverseAsync();
opVisitor.Reset();
traverseCount++;
}
}
Using CUDA operations
Cudablas.Instance.Initialize(); // initialize the CUDA library
Cudablas.Instance.SetDevice(0); // set the device to use, defaults to 0
// ... <Run CUDA operations> ...
Cudablas.Instance.Dispose(); // dispose the CUDA library
