Pascal for TensorFlow

Contents

Introduction

In March 2018 the Google Brain team announced that they had branched Apple's Swift compiler for use with TensorFlow. A lengthy article discussing that decision is here. Previously most programming against TensorFlow had been done in Python, so this effort represents a significant step both for TensorFlow and for Swift.

Swift for TensorFlow includes Swift wrapper classes for working with the library, as well as enhancements to the Swift compiler and language. The project's main goal is to provide improved usability for machine learning developers working with TensorFlow.

These notes describe introductory steps for doing something similar for Pascal, although stopping short of modifying the Free Pascal compiler and language.

Goals

1. Create Pascal wrapper classes for TensorFlow that allow developers to work with TensorFlow in a natural, Pascal-like way, using common Pascal types and conventions, including operator overloading.

2. Reduce the "ceremony" of Pascal programming. Python and Swift are "low-ceremony" languages, whereas Pascal can involve a great deal of prep and housekeeping work declaring variables, managing memory, and other unproductive activities.

3. Look to the brand-new Swift interface for ideas rather than to the traditional Python libraries for TensorFlow.

Setting Up

1. Unzip the Pascal for TensorFlow source code and examples anywhere you want.

   Pascal4TensorFlow.zip

2. Install the TensorFlow libraries (binaries).

   The TensorFlow dynamic libraries are easy to extract anywhere on your system, including the default /usr/local/lib location. See the readme.txt file included in the previous step for links to download the library binaries.

Example Programs

1. inference.pas

   This is the Pascal version of the inference.swift program given in the "Interpreter" section here. It demonstrates how to do a simple classification with TensorFlow. It also shows how verbose Pascal is relative to Swift, although this difference may not be as noticeable in a non-trivial program.

2. MNIST.pas

   This is the Pascal version of the MNIST.swift program given here. This program uses the classic MNIST dataset of 60,000 images of handwritten numeric digits (0-9) collected from census workers and high school students. It uses the dataset to train a neural network to predict a scanned image's digit.

   A useful, detailed discussion of the Swift code is given in part 5 here.

Discussion

1. TTensor wrapper class

   The parsed C headers are included as unit TensorFlow.pas, but this is a clumsy way of working with TensorFlow in Pascal. As a result, unit TF.pas is also included, which wraps TensorFlow.pas in class TTensor.

   Like Swift's Tensor, TTensor wraps the newer "eager execution" API.

2. Convenience constructors

   Any kind of multi-dimensional tensor can be created, but in practice tensors are often scalar (rank 0), vector (rank 1, or 1 dimension), or simple matrix (rank 2, or 2 dimensions). Tensors can also have almost any numeric data type. But again, in practice, tensors are often created for just a few standard numeric types. As a result, the TTensor class provides convenience constructors for easy creation of tensors of rank 0, 1 or 2 from arrays of type Single and other common Pascal types.

   For example, Swift code like this that creates a tensor of type Float and rank 2 from an array of values:

   var w2 = Tensor<Float>(shape: [4, 1], scalars: [0.4, -0.5, -0.5, 0.4])
   

   Can be written in Pascal like this:

   w2 := TTensor.CreateSingle([4, 1], [0.4, -0.5, -0.5, 0.4]);
   

3. Operations

   Swift for TensorFlow includes functions for all supported tensor operations. These are generated automatically for each release. Something similar could be done for Pascal by parsing the ops.pbtxt file and generating the function code, but that's for a future effort.

   In the meantime, the TTensor.ExecOp method can be used to execute many of the supported operations. For example, to do matrix multiplication on two tensors in Swift, you can do this:

   matmul(o1, w2)

   For now, do it like this in Pascal:

   o1.ExecOp('MatMul', w2)

4. Overloaded operators ( + - / * )

   These two Pascal tensor expressions are equivalent:

   t1 + t2
   t1.ExecOp('Add', t2)
   

5. Managing memory

   Take this example line of Swift code:

   let o1 = tanh(matmul(x, w1) + b1)

   This code does matrix multiplication of tensors x and w1, then adds tensor b1 to the result, calculates the hyperbolic tangent of that result, and finally assigns the full expression's tensor value to o1.

   Note that in Swift, the o1 tensor variable does not need to be declared previously and tensors are freed automatically when they go out of scope.

   As a first cut at a Pascal version of this expression, you could write something like this to ensure it doesn't leak memory:

   try
     temp1 := x.ExecOp('MatMul', w1);
     temp2 := temp1 + b1;
     o1 := temp2.ExecOp('Tanh');
   finally
     temp1.Free;
     temp2.Free;
     o1.Free;
   end;
   

   Recognizing that intermediate tensors created by an expression are temporary and that all operations, including overloaded operators, go through the ExecOp method, the TTensor class includes a handy Temp function. This Temp function sets the TTensor instance's IsTemp flag and returns the instance. ExecOp then knows whether to call Free on a tensor input. This allows the code to be simplified to this:

   try
     o1 := (x.ExecOp('MatMul', w1).Temp + b1).Temp.ExecOp('Tanh');
   finally
     o1.Free;
   end;
   

   Note that this also eliminates the need to declare temp1 and temp2 as variables.

   Furthermore, if o1 will always be used in subsequent expressions, its last occurrence could be referenced as o1.Temp, thus eliminating the o1.Free line. This might also allow for the elimination of the try/finally lines required here for memory cleanup.

macpgmr (at) icloud (dot) com

First posted June 22, 2018.

Code syntax highlighting done with highlight.js.