Hi Friends, Whats Up? A long Period of Silence... during this very long silence period I was preparing for the Fourth phase of Life(Ascetic life). Did n't touch PC, nor written a single line of code. Well only helped desirous students to clarify their doubts on paper though. Was on a pilgrimage to Amarkantak, MadhyaPradesh. Went on Narmada Parikrama (circumambulation of River Narmada) of 3500 Km approx by walk, + bus + train... It was a different Kind of experience...
Well here I want to share some AI adventure currently I an doing/ done. Few days back I was suggested by well wishers and My cardiac doctor to restart Computer adventure. In Previous post we talked about how this AI function. How it is trained. Now we hear AI in every corner... Even those who don't know a single thing about technology are also talking about AI.. AI ko puchh wo bataadega So I started with the commonly available AI chat bot "Gemini". then gradually i tried "Meta" and "Deepseek" and of course "ChatGPT".
If they say the Power of AI is immense and it can annihilatehuman civilization... then it will be a over thought over reaction. To me AI is nothing but a Set of Weights in a Matrix. as We have seen in my previous posts. But Using AI judiciously we can change the world.
Mark above "Using AI" not AI by itself. there is a subtle difference. During past few months I wrote a Web hosted AI chat bot "Narada" it was using a small free model to chat, and quite accurately answered critical question about astronomy and astrology. One may ask why astrology? well astrology depends on Astronomy for calculations so if i ask about astrology then it has to use astronomy hence my idea to test the capability is achieved.
The test plan :
Instead of supplying with my DoB and Time of Birth, I provided my Natal chart and Place of birth. Asked it to spit out my date of birth. It really amazing, it spit out few dates and started resoning and said this is your DoB, and it is correct. I repeted this experiment with all the standard chat bots i mentioned and they too did the same.
Level Next.:
Well after that I developed a respect for these chat bots and started converse as I am talking with you in plain human language. One fine day I was curious and asked "Can you give me the source code of Rain drop Virus" the reaction and answer was " As per the Norms set I am Unable to provide the code due to Security reason". Then suddenly it came up with a interactive page simulating raindrop virus. "This happens to DOS screen when Raindrop virus infects your system". Very interesting....
To measure capability i asked "I need an Editor/IDE similar to VSCodium but using QT/C++" in seconds it provided a code which is compilable and though not all features are available but look and feel is same and style is same. Every aspects of interface was there well only a normal editor not versatile as VScodium.
Yes what you see is above VSCodium clone in QT/C++ now it is up to the user to modify 😁
Today I saw some post about replica of Adobe suite, like Lightroom Photoshop etcc cloned using AI yes it might be possible and it can be done. But is it Ethical?
The IP dilemma:
The LightCraft clone of Lightroom from Adobe is no way code level compatible. it is a pure clean room implementation using Rust , But still the coding effort, cost and R&D of a team is replicated in few days!!! and it is free !!!
Next logical question is who owns it? The AI engine has generated the code. Is the Engine holds the right, The user who created the text prompt has the right? or the organization which created the Model holds the right? or the whole of the mankind upon whose texts on internet the model is trained holds the right?
Tools I created for AI using AI
Well Now few tools I have created using AI for AI...
Narada Chat bot
Bandhu A desktop client like "chat box" using Python it is multi server capable even it can use local OLLAMA service too
Bandhu_hub A daemon using python and flask with same capability but interface is web browser.
If any of you is interested then contact me. Why I have not published ... due to IP dilemma
In part 2, AND gate and OR gate could be trained using a single perceptron. But XOR gate could not be as it is not linearly separable. The XOR function can be represented as
\begin{align}
y &= x_{1}\bar{x_{2}}+\bar{x_{1}}x_{2} \\
z_{1} &= x_{1}\bar{x_{2}} \\
z_{2} &= \bar{x_{1}}x_{2} \\
y &= z_{1}+z_{2}
\end{align}
Looking at the above equations
\begin{align}
z_{1} &= x_{1} \land \bar{x_{2}} \\
z_{2} &= \bar{x_{1}} \land x_{2} \\
y &= z_{1} \lor z_{2}
\end{align}
y is splited into 2 equation z1 and z2, 2 linearly separable functions hence it can not be realized using a single neuron. Then the solution will be a multiple neurons. The organization of multiple neurons stacked in different layers is known as "Multi Layer Perceptron". In this part we will see a very basic multi layer perceptron to solve XOR problem. This chapter involves calculus differential. For Information purpose basic things will be discussed but no in-depth calculus will be used. It is assumed the reader is well aware of derivatives.
Multi layer Perceptron
A Multi layer perceptron (MLP) is a network of neurons arranged in more than one layer. That is Output, Hidden and Input. Generally Input Layer is a direct connecting layer. Hidden layer is where the real magic happens and the output layer is interface to user. A hidden layer is called as such because user doesn't interact with that layer. In an MLP multiple hidden layers may be present. But there is a limitation which is due to "Vanishing Gradient Problem". If interested then There are enough material present on internet.
In an MLP with one input layer, one hidden layer and one output layer, and the network is fully connected that is each neuron of a layer is connected each neuron of immediate nearby layer. Please See the picture Bellow.
One thing to observe here is Hidden layer. Each the neurons in this layer has connections from each neuron in Input layer, and they are connected to output neurons. This is a simple fully connected neural network with 4 inputs, 5 hidden and one output neuron. In Future multiple output networks will be discussed and used.
The connections between layers are weights. These weights are generally represented using matrices. for convenience we will represent each weight as wij where i is ith neuron of next layer and j is jth neuron of current layer. In above case weight matrix between input and hidden layer should be defined as a matrix Wih[5,4]=> 5 rows and 4 columns and the weight matrix between hidden and output is Who[1,5]. For Input, hidden output and final output the matrix will be a column matrix.
Forward Propagation
Input data moves through the network and creates some output. This is called forward propagation. Previously in single neuron, the calculation was easier. Here in case of multiple layers it is a bit involved. It involves matrix multiplication. Consider above Image.input arriving at ith neuron of hidden layer can be expressed as:
\begin{equation}
h_{i} = w_{i0}x_{0}+w_{i1}x_{1}+w_{i2}x_{2}+w_{i3}x_{3}
\end{equation} or in general terms it can be expressed as
\begin{equation}
h_{i} = \sum_{j=0}^{n}w_{ij}x_{j}
\end{equation}
If set of equation for hi is observed closely then it is seen that it is result of matrix multiplication.
$$
\begin{bmatrix}
h_{0} \\
h_{1} \\
h_{2} \\
h_{3} \\
h_{4}
\end{bmatrix}
=
\begin{bmatrix}
w_{00}^{ih}&w_{01}^{ih}&w_{02}^{ih}&w_{03}^{ih} \\
w_{10}^{ih}&w_{11}^{ih}&w_{12}^{ih}&w_{13}^{ih} \\
w_{20}^{ih}&w_{21}^{ih}&w_{22}^{ih}&w_{23}^{ih} \\
w_{30}^{ih}&w_{31}^{ih}&w_{32}^{ih}&w_{33}^{ih} \\
w_{40}^{ih}&w_{41}^{ih}&w_{42}^{ih}&w_{43}^{ih}
\end{bmatrix}
\times
\begin{bmatrix}
x_{0} \\
x_{1} \\
x_{2} \\
x_{3}
\end{bmatrix}
$$
$$H = W^{ih}.X$$
This resultant matrix need to be applied activation function of hidden layer to produce output. Thus generated values is input to the out put layer(in our case output neuron). As previous layer we need to do a matrix multiplication to find the input to the output layer and apply activation function of output layer to get the network output.
$$
\begin{bmatrix}
O_{0}
\end{bmatrix}
=
\begin{bmatrix}
w_{0}^{ho}&w_{1}^{ho}&w_{2}^{ho}&w_{3}^{ho}&w_{4}^{ho}
\end{bmatrix}
\times
\begin{bmatrix}
h_{0} \\
h_{1} \\
h_{2} \\
h_{3} \\
h_{4}
\end{bmatrix}
$$
$$ O = W^{ho}.H $$
Back Propagation
The Output O is dependent on weights Who and H. H is dependent on weights Wih. As the selection of initial weights are random, hence they are unlikely to be perfect. Hence the output O is different than expected or target. The difference of Target and Output is known as Error. This error has to be compensated by adjusting weights Who and Wih. The adjustment of weights starts from last layer and goes down to beginning. Hence it is called Back Propagation.
Back propagation of errors has to be adjusted among different layers, depending on how much each layer contributed. To find out contribution of error a partial derivative of error has to be found out with respect to different contributing elements. Description and derivation of this is beyond the scope as it will lengthen the article. For reference one can watch this video
We can represent the updated weight matrix of a layer as follows
$$ W_{new}=W_{cur}+\alpha[target-actual output].X$$
where
X is input to the layer
α is learning rate
Implementation
To adjust weights in Who>, we need to find error in output, then apply derivative of activation function to it. Next multiply by scalar learning rate(α). This finish scalar calculations. The resultant multiplied by the Input to the layer that is activated output of hidden layer will give us delta weights. this delta weights added to the weight matrix Woh gives updated weights in Hidden to Output layer.
Same procedure have to be repeated for input to hidden layer. To begin with we need to know error at hidden layer first. This can be found by multiplying (Woh)T with error at output. Rest procedures are same as above.
With this we are ready to make a very small neuralnet and train it for XOR problem. MNIST dataset can also be trained but needs patience and time( I achieved with 91.6% accuracy). Iris Dataset and wheat seed data set are very easily trained. Bellow is XOR problem implementation.
Though I may put all the code here or on github, but I would like the reader to implement his own version analogous to this implementation. In any problem Contact me for the Full code its free But before asking please try to implement yourself.
Till now we discussed
What is an artificial neuron
How to implement it
What are the areas it can be used(linear separable)
How to train it
What is XOR problem
What is Multi layer neural network
How to implement one
How to train
Example of XOR implementation
This is the end of 3 parts for simple neural network. Part1 and Part2 can be found here. In some future article we will discuss Convolutional Neural Network(CNN)
A single Perceptron(neuron) can do a lot
of jobs. Yes Like a Neuron in Our nervous system it can do a lot of things. As
One grows from Infants to toddler the neurons in our brain get
trained(learned) to coordinate between our limbs, coordinate our activities
etc... Similarly the Perceptrons can be trained to classify data as per
the training data. but we can only classify linearly separable data.
What
is a linearly separable? Linear separability can be termed as, if the
plot of data can be separated by a single straight line. This means depending
on the separation criterion, there exists a straight line dividing the data
points in to 2 regions.
Now lets see a small code implementing it
step by step.
NOTE: We need a matrix manipulation library for all our Neural network
adventure.
For this purpose my
tmatlib
on github is quite suitable. Though one should not expect it to perform like
blas. In my code i used a similar version known as smatlib. As
it is continuously growing I have not published it on github. But stay assure
the functionality and performance of both are same.
A straight line on a 2-dimentional plain can be expressed as
y=mx+c. The Bias parameter in our perceptron is related to
c. Our training data is x and y, x is input and y is expected or
target. Our perceptron learns the value of m. Learning rate(lr)
determines how fast or slow the perceptron should learn. If Learning rate is
too low then it may be trapped in local minima. If it is too high then it may
oscillate and may not reach the solution. So Bottom line is one has to
experiment with learning rate(lr) value to achieve better result. Weight
matrix is initialized with random values. Gradually with training it is
adjusted/updated to achieve better result.
In the repo there are 4 files perceptron.h, linearsep.c, and_gate.c and
or_gate.c. We will discuss bits from those files and see how they work.
In perceptron.h file, there are functions/methods to
create, manipulate, train, display and delete the perceptron.
General sequence is
create a perceptron
prepare data set
train
test with predict function
This is simple, is not it? Now we will look into 2 functions, namely
predict and train. These functions are the core of this
perceptron.
predict:
double predict(perceptron p, double inp[])
{
double res=0.0;
for(int r = 0; r < p.n_inp; r++)
{
res += inp[r]*get_cell(p.W,0,r);
}
res += p.bias;
double ret= sigmoid(res);
return(ret);
}
This function multiplies inputs with corresponding weights and adds them. Next, to activate the output apply sigmoid function. A sigmoid function ranges from 0 to 1. Please see the first part here where the sum process and sigmoid function is defined.
In this function, perceptron is trained by adjusting weights and bias.The steps can be as follows.
calculate output
find error by subtracting output from target
find delta weight by multiplying error, data(input) and learning rate
find delta bias by multiplying error and learning rate
update weights
update bias
In step 3 we are calculating the delta weights. This is the most important step in whole code. How and why delta value is multiplication of error with input is a topic itself. As an Hint it is gradient, hence a derivative of error term with inputs and weights are to be considered.
Testing
To tryout this implementation, 3 code files were provided. which adheres to the general sequence. First one is linear_sep.c, In this demo program, random data sets are created and the perceptron is trained; then with a known data set it is compared. the output on my system is as bellow.
************Perceptron:**************
number of inputs = 2
Weights:
[ 1.00000000 ]
[ 1.00000000 ]
Bias = 1.000000
*************************************
************Perceptron:**************
number of inputs = 2
Weights:
[ 61.70302487 ]
[ -31.13024636 ]
Bias = 0.622900
*************************************
is (2.000000 < 2 x 20.000000 + 1) predicted = 1.000000
For other 2 demo codes. Can this artificial neuron mimic reliably a digital gate? Well that was the motive when artificial neuron was proposed. It is observed that AND and OR gate are possible to simulate but not XOR gate. WHY??? from bellow diagrams it is obvious that OR function and AND function are linear separable but not the XOR function.
In the above result if we consider 0.99 as 1 and anything less that 0.005 as 0 then our results are at par with the truth table.
As we saw above XOR is not linearly separable, we can not simulate it with a single perceptron. we need more than one layer to simulate it we will do it in a future post.
Around a year ago we talked. Now several things have changed. I recently joined as a PhD scholar 😃. And decided to soil my hands in AI/ML. During my M.Tech days(2011-2013), we are taught Soft computing. Though I did my Thesis on Network On Chip (NOC) which is related to putting different IPs on to a silicon substrate. I devised a Genetic Algorithm based best mapping as well as a deterministic heuristic based mapping using graphs. both the codes are available on my github page. But I have never paid much attention to ML till 2016, when I was developing a traffic detection application for a client. thought it was never went beyond PoC stage.
Now I thought to put all my knowledge to earn a degree (Yes a Doctorate). To refresh my understandings of my master's degree time, I decided to go step by step implementing building blocks to learn this topic.
We should start by putting a single neuron first. The artificial neurons, the counter part of natural neurons share similar characteristic.
Dendrites are inputs and axon is output. But one thing we must put here which is Activation. It is the strength we provide as output.
in case of perceptrons input are applied through some weights. This means different input channel have different weights. the perceptron sums the weighted inputs and apply some sort of activation before output the result.
we can say we need 2 one dimensional matrices for inputs and weights. Then upon the result of the multiplication we apply activation function.
Lets assume we have 3 inputs i1, i2, i3 with weights w1,w2 and w3. Now output will be
\begin{equation}
z=\sum_{1}^{n} i_mw_m
\end{equation}
next activation of z is output.
There are several activation functions used to activate the output of perceptron. the most used are Binary activation, ReLU(0,x) and Sigmoid(0,1).
We will use sigmoid for our application. \begin{equation}\sigma(z)=\frac{1}{1+e^{-z}}\end{equation}
Next we will see the simple implementation of this perceptron.
Added 2 small projects, one is a single header Matrix library and another is a CPU Simulator.
The matrix library (tmatlib) is capable enough to solve simultaneous linear equation. Though it was written several months back but I was Lazy.
The CPU simulation code was my tech-itch :p it is very minimal and for fun. It has small set of instructions, it has only 8 bit wide registers and a very small memory only 1kb which includes stack, data and code.
check if PHP is properly working or not by using following script
phpinfo.php
<?php
phpinfo();
?>
this script should be in /var/www/html/ directory which is by default document root for httpd server.
Point the browser to this file and see if it shows a page about PHP version and other details. If not then check if apache server is running.
As super user create a laravel.conf file under /etc/httpd/conf.d directory with following content
<VirtualHost *:80>
ServerName your server name
DocumentRoot /var/www/html/laravel_project/public
<Directory /var/www/html/laravel_project>
AllowOverride All
</Directory>
</VirtualHost>
Copy the laravel project to /var/www/html/ as super user and cgange the owner by chown -R apache:apache laravel_project
Start Mariadb (sudo systemctl start mariadb)
This is almost complete. For your specific cases different services or tweaking may be required.
Laravel is a good framework for web-app development but for sometimes it is not getting more acceptance. Now the buzz word is Node . In a future article I may write on settingup and using Express framework for Web-App.
now gunicorn is running. you can check with http://127.0.0.1:8888 and it will show you hello world page.
But we need nginx to serve for us on internet so we have to tell nginx to serve for us.
use following code as a .conf file in /etc/nginx/conf.d/ directory we assume it as myapp.conf
server { # the port your site will be served on listen 80; # the domain name it will serve for server_name 192.168.1.3; # substitute by your FQDN and machine's IP address charset utf-8;
#Max upload size client_max_body_size 75M; # adjust to taste
# Django static files. You should run 'django manage.py collectstatic' location /static { alias /var/www/static; # your Django project's media files } location /media { alias /var/www/media; } # Finally, send all non-media requests to the Django server. location / { proxy_pass http://127.0.0.1:8888; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } }
restart nginx server by using
$ sudo systemctl restart nginx
now check from a remote machine using http://IP.address.of.server/
do you get it properly??? if not then check firewall settings on server. next check access on files etc...
Yes what you see is above VSCodium clone in QT/C++ now it is up to the user to modify 😁
