Gradient Descent using Simulink

In most of the programming languages like C, Python, Java etc. code has to be written to realize the logic. Enter Simulink where the programming is done using block diagrams. Simulink is one of the toolboxes found in MATLAB and is widely used in domains like automotive, aerospace etc. for Control System logic designing. In this blog we will try to recreate Gradient Descent Logic using Simulink and check the results using Python.

About MATLAB

MATLAB is a software which is widely used by engineers and scientists for algorithm development and modelling. MATLAB has lot many Toolboxes to help in different areas like Signal Processing, Control System etc. Each of these Toolboxes have special functions related to that area. A thing to note is that MATLAB is not a freeware 😅. The license cost is pretty huge and if you need to add Toolboxes then each Toolbox has a different cost. There are some free alternatives like Octave and Scilab. But they are not as flexible or powerful as MATLAB or Simulink.
So how to use MATLAB or Simulink then? You could use a trial license for 15 days and then uninstall it. In my case I wanted to understand the basics of Simulink, hence I enrolled in the Onramp course of Simulink. Here I was able to use the web version to try modelling Gradient Descent. You too can try the same to have a taste of Simulink programming.

Simulink Programming

So as mentioned above, Simulink is a Toolbox where different blocks like Sum, Delay etc. are used to model an equation or a system. In other words, it is a block diagram environment used for designing, simulating and testing models. You can see this video to get a high level idea about Simulink:
https://www.youtube.com/watch?v=GvRugdEICyQ
You can check the different blocks available in the Simulink browser in the below screenshot:

More blocks for Digital System design, Signal Processing etc. are also available. But the specific Toolboxes have to be purchased. You can check the complete list of Toolboxes at the below link:
https://in.mathworks.com/help/thingspeak/matlab-toolbox-access.html

Gradient Descent Algorithm:

This is one of the most widely known algorithm which is simple to understand and very effective as well. I have written in detail about the algorithm in the below post:
https://evrythngunder3d.wordpress.com/2020/07/01/machine-learning-algorith-linear-regression-part-2/

Gradient Descent in Python:

I have first written the logic in Python and recreated the same logic in Simulink. I compared the coefficients got from both the process which came out to be same. The prediction part is not considered as the process easy — simply multiply the coefficients with the input and add the constant value. The whole code for Gradient Descent for a single feature in Python is as below:

# import the packages
import numpy as np
import pandas as pd
# preprocessing data for empty data and column names
def preprocessData(df):
    df = df.drop(['Id','MasVnrArea','GarageYrBlt'],axis=1)
    df = df.rename(columns={"LotFrontage": "Lot", "SalePrice": "Price"})    
    df['Lot'] = df['Lot'].fillna(int(df['Lot'].mean()),inplace=False)
    df['Price'] = df['Price']/1000
    return df
# fitting and predicting the data for the model
def fitPredictModel(XtestData,g):
    ypred = g[0][0] + g[1][0]*XtestData
    return ypred
def computeCost(X, y, theta):   
    prediction = X @ theta
    cost = (1 / (2 * len(X))) * np.sum(np.square(prediction - y))    
    return cost
def gradientDescent(X, y, theta, alpha, iters):
    costAll = []
    thetaCapture = []
    for i in range(iters):        
        prediction = X @ theta
        theta = theta - ((alpha/len(X)) * (X.T @ (prediction - y)))
        thetaCapture.append(theta)        
        cost = computeCost(X, y, theta)
        costAll.append(cost)
    return (theta,costAll,thetaCapture)
# read data from csv
df = pd.read_csv('linearDataset.csv')
my_data = preprocessData(df)
# declare the train and test percentage
trainPercent = 0.7
testPercent = 0.3
  
X_data = my_data['Lot']
y_data = my_data['Price']  
rangeNeeded = int(len(X_data) * 0.7);
XtrainData = np.array(X_data[0:rangeNeeded])
XtestData = np.array(X_data[rangeNeeded:])
ytrainData = np.array(y_data[0:rangeNeeded])
ytestData = np.array(y_data[rangeNeeded:])
# reshaping data and adding the ones column
XtrainData = XtrainData.reshape(-1,1)
ones = np.ones([XtrainData.shape[0], 1])
XtrainData = np.concatenate([ones, XtrainData],1)
XtestData = XtestData.reshape(-1,1)
ytrainData = ytrainData.reshape(-1,1)
ytestData = ytestData.reshape(-1,1)
# declaring alpha and iteration
alpha = 0.0001
iters = 1000
# declaring theta as a row vector
theta = np.array([[1.0, 1.0]])
# applying gradient descent
g,costAll,thetaCapture = gradientDescent(XtrainData,ytrainData, theta.T, alpha, iters)
print(f"coefficient:{g[1][0]} interscept:{g[0][0]}")

From the above code you can see that I have taken just one attribute ‘ Lot’ to predict the ‘ SalePrice’. The code is pretty simple where the data has been preprocessed to include a single attribute and divided the output variable by 1000 to reduce the range.
Also note that the data has been divided into test and train with fixed indexes so that the result should match after doing the same exercise in Simulink. Else random distribution into test and train could give different result.
After running the loop for 1000 iteration, Gradient Descent gives the value as below:

Now let’s see the same in Simulink.

Gradient Descent in Simulink

Before creating the block diagram, we need to do some preprocessing as done in our Python code. This includes selecting the attribute for prediction, dividing into train-test set and initializing various parameters like iterations and alpha. This can be done by writing the code in a MATLAB script. It is really easy and does not have much lines. The screenshot of the code is as below:

This script needs to be run before running the block diagram in Simulink. Once this is run the variables have their values stored in the ‘ Command Window ‘ which the block diagrams can access. You can see the block diagram image below:

It follows more or less the same as the Python code. The above whole system lies inside a ‘ for iterator’ block. The iteration value is set from ‘ iters’ variable. The ‘ delay’ block has been set at starting value to ‘ theta ‘ which gets updated at every for loop iteration. The rest are simple matrix multiplication and addition/subtraction to get the coefficient values. After simulating the model we can see that the coefficient value matches with the ones calculated using Python.

So what’s the advantage in recreating the Gradient Descent in Simulink? Aside from the fact that it is visual programming which is much more easier (but can become complex as size increases!) there is one more thing that can be done in Simulink. You can create C code from the Simulink model for your target hardware. This is one of the best reason why Simulink is widely used in Engineering domain where the code has to be flashed into controllers. Of course it does require extra Toolbox — the Embedded Coder Toolbox. Since this option was not available in the temporary Web based Simulink, I could not do this part.
Though the major use of Simulink is in designing Control System, it can be used in Machine Learning and Deep Learning applications like ADAS. New Toolboxes are always added and there are Deep Learning blocks available in the Simulink Block list.
I hope this blog gave an idea of MATLAB and Simulink and a bit about block programming. Let me know in the comments!

References:
- A lot of googling amongst which the major sources were mathworks.com, youtube.com, MATLAB Onramp

Originally published at http://evrythngunder3d.wordpress.com on March 7, 2022.