RND / RANDOMIZE — trivial to implement (LCG on NASM, rand() on C). Already planned.
term.dkm — ANSI escape sequence parser in the standard library, not the compiler.
resolve_name refactor — preventive cleanup, not blocking.
FFI — for graphics, sound, networking, GUI/RAD. This is where the “gun on the platform’s shoulder” architecture becomes a feature rather than an implementation detail.
OO — classes, methods, inheritance. Phase 2 language design.
duckdb3.dkb — the milestone proof-of-concept: a DuckDB-flavored table engine with indexes, queries, and file I/O.
Fullscreen editor — the “does this actually work?” test.
The architectural philosophy
Three principles emerged from building duckBASIC:
1. Platform where possible, own where necessary.
The C backend uses rand(), strcpy(), memcpy(), printf() — all platform-provided. The NASM backend implements fixed_mul (60 lines of asm), _read_str (BIOS INT 21h, AH=0Ah), _print_int (custom decimal printer) — all hand-written, because the platform (bare 16-bit DOS) has nothing.
The “gun on the platform’s shoulder” philosophy is about not reimplementing what you can depend on, and owning what you can’t. Every function in runtime_c.c is a thin wrapper over libc; every function in runtime_com.inc is a hand-written implementation. The distinction is principled, not accidental.
2. Byte layout is a contract.
SERIALIZE / DESERIALIZE works because the byte layout is documented, locked, and tested. It’s the reason a .dkb program can serialize on C and deserialize on NASM. It’s also the reason C’s int (4 bytes) vs NASM’s int (2 bytes) was caught early — as a design decision, not a bug.
3. Bug classes need bug-class fixes, but not always immediately.
The three-path name lookup is a bug class. The right answer is resolve_name. The right timing is when you have a failing test. duckBASIC got both right: it fixed the symptom narrow (with tests 74–78 as proof), and deferred the refactor with a clear trigger condition. That’s what a mature codebase does.
Conclusion
duckBASIC is not a production compiler. It’s not going to replace GCC or LLVM, and it’s not going to be used to write the next PostgreSQL. What it is, though, is a complete implementation of a compiler pipeline with two real backends, a real test suite, and enough features to build real programs (Phase 2’s fullscreen editor is the proof point).
The project demonstrates several things that are worth writing down:
Two backends is better than one. The C backend made iteration fast; the NASM backend made every bug real.
Fixed-point math on 8086 is a solved problem if you’re willing to write 200 lines of assembly.
Byte layout is the hardest part of any serialization feature, and the easiest way to get it wrong is to trust that the platforms agree.
Cross-backend test suites catch things that single-backend test suites miss. Every bug I’ve described in this series was caught by a cross-backend test.
Refactoring is a response to pain, not a preventive measure. The right time to unify three name-lookup paths is when a test fails because they diverge — not before.
If you’re building your own compiler, I hope this series gives you some concrete patterns to steal. The test harness (dual backends, DOSBox verification) is the single most valuable design decision in the entire project, because it turns every bug into a reproducible failure rather than a suspected one. That’s worth more than any amount of clever parser design.
The complete Source code will be available to those who are interested
//factorial recursion in DuckBASIC.
FUNCTION factorial(n AS INT) AS INT
{
IF n <= 1
{
RETURN 1
}
RETURN n * factorial(n - 1)
}
PRINT factorial(5)
The bellow image is the look of duckedit, the editor written in DuckBASIC
🦆Part 4 — Case Study: The Local Struct Bug, and What It Taught Us
The symptom
At V5.9d.8+.9 (the last commit before this work), a .dkb program containing:
FUNCTION foo()
{
DIM tmp AS Point
tmp.x = 5 // local struct field write
}
typechecked fine, but on the NASM backend, the emitted assembly contained:
mov [bp-4], ax ; correct: local struct field write
mov [var_], ax ; BUG: empty variable name
The second store wrote to a data-segment symbol with an empty name — which is to say, it wrote to whatever happened to be at var_ in the data segment. This was silent corruption.
Root cause diagnosis
The session notes said this:
Three separate name-lookup paths (parser, typechecker, NASM codegen) each handle global/param/local differently. Local struct vars fall through the cracks in some paths.
That was correct in spirit, but the specific bug was narrower: the NODE_ASSIGN case in gen_nasm_stmt.c had a series of if blocks for different LHS types:
if (lhs is field of array-of-struct) { ...; break; }
if (lhs is field of struct) {
if (is_param) { ...; break; }
if (is_local_struct) { ...; break; } // ← the fix
if (is_global_struct) { ...; break; }
}
if (lhs is array element) { ...; break; }
// FALLTHROUGH: scalar assignment
fprintf(out, " mov [var_%s], ax\n", node->left->attr.name);
The scalar fallback read node->left->attr.name. But for a NODE_FIELD_ACCESS, that union field aliases node->left->attr.fieldAccess.object, which is a pointer. So attr.name was a garbage pointer, usually pointing at an empty string in memory. Hence mov [var_], ax.
The fix was to add the is_local_struct check before falling through. But the real fix — the one that eliminates the bug class — is to have a single resolver that all three passes use.
The proposed refactor
typedef enum {
RESOLVE_GLOBAL,
RESOLVE_PARAM,
RESOLVE_LOCAL,
RESOLVE_FAIL
} ResolveKind;
typedef struct {
ResolveKind kind;
int offset; // stack offset for param/local
Symbol *global; // valid if kind == RESOLVE_GLOBAL
StructField *field; // valid if kind == RESOLVE_PARAM or LOCAL
} ResolvedName;
ResolvedName resolve_name(const char *name);
Every existing branch in gen_nasm_expr.c and gen_nasm_stmt.c that walks function.parameters / function.locals / symtable_lookup inline would be replaced with resolve_name(name). That’s ~15 call sites, each simplified from 15–20 lines to 3–5.
This refactor was deferred. The bug was empirically dead — tests 74–78 (local structs in FOR loops, local struct arrays, struct-to-struct assignment, SERIALIZE through struct params) all passed on both backends. The resolver refactor is now scheduled as a Phase 2 architectural cleanup, to be triggered by a concrete failure (e.g., a duckdb3 bug that requires it) rather than a hypothetical one.
The lesson
The three-path name lookup isn’t a bug. It’s a bug class. Every new node type needs to know about it, and forgetting is invisible until something falls through. The session that fixed it did so by:
Writing failing tests for every context (local, param, array, struct, loop body).
Fixing the bug narrowly, confirming all tests pass.
Deferring the resolver refactor because the narrow fix was correct.
The right answer to a bug class is usually “refactor to prevent recurrence” — but the right timing is usually “when you have a failing test.” This session’s decision was correct: the fix was tested comprehensively, and the refactor was deferred with a clear trigger condition. That’s how mature projects handle technical debt.
🦆Part 3 — Backends: C Transpiler and 16-bit NASM
The C backend
The C backend is the simpler of the two. It emits C source and lets gcc do the hard work.
Structure:
gen_c.c — top-level entry: emits preamble, structs, forward decls, globals, function definitions, then main().
gen_c_expr.c — expressions (arithmetic, function calls, field access, array access).
gen_c.h — CGen state struct threaded through all emitters.
The runtime is a C file (runtime_c.c) that’s compiled into a string literal at build time and injected into every generated .c file. It provides _check_bounds, sys_open, sys_read, etc.
Key design decisions:
All state is in a CGen struct. No globals. This makes reentrancy possible (though not currently used) and makes threading state through emitters explicit.
Type mapping is done via static inline functions in gen_c.h — no separate TU needed.
Array accesses get bounds checks via a runtime helper:
buf[_check_bounds(i, 16, "buf")]
The helper prints an error and calls exit(1) on violation.
String assignment uses strcpy, comparison uses strcmp. The typechecker ensures STRING ops only happen where they’re valid.
Struct field access emits .field for variables, ->field for params. The codegen knows which is which because it tracks current_function and checks the param list.
The NASM backend
The NASM backend is where the real engineering lives. Every operation must be expressed in 16-bit x86, with no libc, no linker, no relocations — just a flat binary loaded at 0x100.
Structure:
gen_nasm.c — entry point, string pool, symbol lookup, program emission.
gen_nasm_stmt.c — statements.
gen_nasm_expr.c — expressions.
gen_nasm_data.c — data segment emission, struct field offset lookup.
Every value lives in AX (and DX for FLOAT). This is a naive register allocation strategy — no spilling, no liveness analysis. It’s slow but correct.
Binary operations use the stack:
; evaluate left into AX
push dx
push ax
; evaluate right into AX
push dx
push ax
pop bx ; right low
pop cx ; right high
pop ax ; left low
pop dx ; left high
; now DX:AX = left, CX:BX = right
Q16:16 fixed-point arithmetic — no floating point unit on 8086, so all “floats” are stored as 32-bit fixed-point (16.16). Addition is add/adc. Multiplication and division are hand-written 32-bit signed routines in fixed.inc, each ~60 lines of assembly.
String constants live in a string pool — deduplicated, emitted at the end of the binary, referenced as str_N.
Everything is in a .COM file — flat binary, loaded at 0x100, no relocations. org 0x100 at the top.
Runtime library is .inc-included — no linking. The %include directives pull in runtime_com.inc, fixed.inc, fileio.inc at assembly time.
The SERIALIZE / DESERIALIZE byte layout
Both backends implement a locked byte layout for struct serialization:
Field type
Bytes
INT
2 (little-endian)
BOOL
2 (little-endian)
FLOAT (Q16:16)
4 (little-endian)
CHAR
1
STRING
256 (fixed-size, null-padded)
The layout is documented in both gen_c_stmt.c and gen_nasm_stmt.c, and it’s the contract that allows a .dkb program to serialize a struct on the C backend and deserialize it on NASM (or vice versa, across process boundaries).
The C backend emits byte-by-byte memcpy calls, truncating int to short on serialize and sign-extending on deserialize. This is necessary because C’s native int is 4 bytes but duckBASIC’s INT is 2 bytes — the layout must match the NASM backend’s.
The NASM backend emits mov sequences (and rep movsb for strings). No truncation needed since NASM’s INT is natively 2 bytes.
The byte layout was a deliberate design decision: it made SERIALIZE / DESERIALIZE work identically on both backends without any translation layer, and it exposed a number of lurking bugs (like C’s int being 4 bytes) that would have been catastrophic later.
🦆 Part 2 — Front End: Lexer, Parser, Symbol Table, Type Checker
Lexer (lexer.c)
The lexer is hand-written, single-pass, token-at-a-time. Nothing fancy — no DFA tables, no regex, just a switch on the first character.
Notable features:
Radix literals: 0x1F (hex), 0b1010 (binary), &HFF (QBasic-style hex), &B1010 (QBasic-style binary). The &H/&B forms are particularly nice because they round-trip cleanly to QBASIC programmers.
String escapes: \n, \t, \r, \\, \", \', \0.
Character literals: 'a' (single quotes). This interacts badly with QBASIC-style ' comments, which duckBASIC does not support — comments are //.
Token names array with a _Static_assert guarding against enum drift.
The lexer is intentionally simple. No diagnostic recovery, no multi-byte UTF-8, no raw strings. Every token is either recognized or produces TOKEN_ERROR and the parser will halt.
AST (ast.h, ast.c)
The AST is a tagged union of node types. Every node is an ASTNode struct:
The union keeps nodes compact (about 100 bytes each). The next pointer chains statements into a linked list, so a block is just a NODE_BLOCK whose left is the first statement and each statement’s next points to the following one.
The union overlap is a known footgun.attr.name (a char*) and attr.fieldAccess.object (also a pointer) occupy the same bytes. Reading attr.name on a NODE_FIELD_ACCESS produces a garbage pointer, which is exactly what caused the mov [var_], ax bug we spent a session debugging.
Symbol table (symtable.c)
The symbol table is a linked list of Symbol structs. It uses three pointer fields to maintain two separate lists simultaneously:
typedef struct Symbol {
char name[128];
DataType type;
int offset;
int class_type; // SYM_SCALAR, SYM_ARRAY, SYM_STRUCT_DEF, etc.
int is_local;
int is_builtin;
short stack_offset;
union {
struct { ... } array;
struct { ... } struct_def;
struct { ... } struct_var;
struct { ... } function;
struct { ... } constant;
} metadata;
struct Symbol *next; // active scope lookup chain
struct Symbol *all_next; // master allocation chain (for cleanup)
} Symbol;
next is the active lookup chain — it changes as scopes are pushed and popped.
all_next is the master allocation chain — never modified after insertion. This is how symtable_free iterates all symbols for cleanup.
head points to the current scope’s head. all_head points to the master list.
Restoring just resets table->head to the snapshot’s head. Memory is never freed until symtable_free at the very end.
This design is simple but not scalable — lookups are O(n) linear scans. For a 16-bit compiler, that’s fine. For a production compiler, you’d want a hash table.
Parser (parser.c + parser_*.c)
The parser is recursive descent for statements and precedence climbing for expressions. Each grammar production is a function.
The parser is split across multiple files by grammar area:
ASTNode* parse_expression(Parser *p, int precedence)
{
// 1. Parse a prefix (literal, identifier, unary operator, paren)
// 2. While the next token has higher precedence than the argument:
// - Consume the operator
// - Parse RHS with the operator's precedence
// - Build a binary node
return left;
}
Precedence levels (highest to lowest):
. (field access) — 12
[ (array subscript) — 11
*/% — 10
+- — 9
<<>> — 8
<<=>>= — 7
==!= — 6
& — 5
^ — 4
| — 3
&& — 2
|| — 1
Error recovery is panic-mode: on failure, skip tokens until a \n, }, or EOF, then continue. This means a syntax error in one statement doesn’t cascade into forty subsequent errors.
Type checker (typecheck.c)
The type checker walks the AST after parsing and before codegen. It does two things:
Annotates every expression node with evalType so codegen can trust it without re-deriving.
Validates type compatibility and increments g_error_count on mismatch.
Key rules:
BOOL cannot appear in arithmetic.
STRING+ is only valid inside PRINT (concatenation).
Array indices must be INT.
Function call arguments must be type-assignable.
The typechecker re-creates function scope by re-inserting parameters and locals into the symbol table, then checks the body, then restores scope. This is a second pass over the AST — the parser already did one symbol-table pass — which is why there are two name-lookup paths (parser + typechecker) plus the codegen’s third path.
This duplication is the source of the “three lookup paths” bug class. Fixing it properly would require a unified resolve_name helper used by all three passes — a Phase 2 refactor.
🦆 duckBASIC: A Modern 16-bit BASIC Compiler — Design & Implementation
🦆 Introduction
This series documents the design and implementation of duckBASIC, a compiler for a modern dialect of BASIC. The Philosophy is to create a usable teachable tool for students or hobyists who are interested in Compiler and interpreter writing. DuckBASIC targets both C (for portability) and 16-bit real-mode MS-DOS via NASM assembly (.COM executables). The project was built iteratively, feature by feature, with a test suite that validates both backends in parallel against identical source programs.
Old timers who have soiled their hands with QBASIC/GWBASIC/BASICA/VB may find similar environment but in C style 🦆🦆🦆
Over the course of development, duckBASIC grew from a simple expression evaluator into a compiler with:
Two full backends (C transpiler and NASM 16-bit assembler emitter)
Structured types: structs, struct parameters, arrays of structs, local structs, nested structs
Fixed-point Q16:16 arithmetic implemented in hand-written x86 assembly
A locked byte layout for SERIALIZE/DESERIALIZE that guarantees cross-backend binary compatibility
A real file I/O API (SYS_OPEN, SYS_READ, SYS_WRITE, SYS_SEEK, etc.)
75+ tests running on both backends, most of which are verified by running the resulting .COM file inside DOSBox and comparing output byte-for-byte
The project is interesting not because it implements a compiler — plenty of toy compilers exist — but because it implements two of them simultaneously, with a shared test suite, and verifies they produce identical output on real 16-bit hardware.
Part 1 — Architecture Overview
The pipeline
Every compiler is a pipeline, and duckBASIC’s is deliberately simple:
Three global counters (g_error_count, g_warning_count, g_verbosity) coordinate diagnostics across phases.
The two backends
The C backend transpiles duckBASIC AST to C source, which is then compiled by gcc. This is the “reference” backend — anything that works here is by definition semantically correct. It’s also the fast one to iterate on: change a C emitter, recompile with gcc, done in 2 seconds.
The NASM backend emits 16-bit x86 assembly for MS-DOS .COM executables. This is the “authentic” backend — every feature must be implemented in raw assembly or wrapped around DOS/BIOS interrupts. It’s the hard one: fixed_mul is 60 lines of hand-rolled signed multiplication, _read_str uses BIOS INT 21h, AH=0Ah, SYS_OPEN wraps INT 21h, AH=3Dh.
The two backends share nothing at the codegen level. They each have their own AST walkers (gen_c_stmt.c, gen_c_expr.c vs gen_nasm_stmt.c, gen_nasm_expr.c). But they share the entire front end — parser, symbol table, type checker — which is what guarantees semantic consistency.
The test harness
The test suite (tests/*.dkb) runs each program through both backends and compares:
C backend output — compile to .c, gcc, run, capture stdout.
NASM backend assembles — nasm -f bin, verify it produces a .com.
NASM output in DOSBox — run .com inside DOSBox, capture output, compare byte-for-byte with C output.
For interactive tests (INKEY$, keyboard input), DOSBox runs with a scripted keystroke file.
The suite currently has 75 tests, 72 pass, 0 fail, 3 skip. The skipped tests are Phase 2 features (graphics, sound, etc.) that aren’t implemented yet.
Hi Friends, Whats Up? A long Period of Silence... during this very long silence period I was preparing for the Fourth phase of Life(Ascetic life). Did n't touch PC, nor written a single line of code. Well only helped desirous students to clarify their doubts on paper though. Was on a pilgrimage to Amarkantak, MadhyaPradesh. Went on Narmada Parikrama (circumambulation of River Narmada) of 3500 Km approx by walk, + bus + train... It was a different Kind of experience...
Well here I want to share some AI adventure currently I an doing/ done. Few days back I was suggested by well wishers and My cardiac doctor to restart Computer adventure. In Previous post we talked about how this AI function. How it is trained. Now we hear AI in every corner... Even those who don't know a single thing about technology are also talking about AI.. AI ko puchh wo bataadega So I started with the commonly available AI chat bot "Gemini". then gradually i tried "Meta" and "Deepseek" and of course "ChatGPT".
If they say the Power of AI is immense and it can annihilatehuman civilization... then it will be a over thought over reaction. To me AI is nothing but a Set of Weights in a Matrix. as We have seen in my previous posts. But Using AI judiciously we can change the world.
Mark above "Using AI" not AI by itself. there is a subtle difference. During past few months I wrote a Web hosted AI chat bot "Narada" it was using a small free model to chat, and quite accurately answered critical question about astronomy and astrology. One may ask why astrology? well astrology depends on Astronomy for calculations so if i ask about astrology then it has to use astronomy hence my idea to test the capability is achieved.
The test plan :
Instead of supplying with my DoB and Time of Birth, I provided my Natal chart and Place of birth. Asked it to spit out my date of birth. It really amazing, it spit out few dates and started resoning and said this is your DoB, and it is correct. I repeted this experiment with all the standard chat bots i mentioned and they too did the same.
Level Next.:
Well after that I developed a respect for these chat bots and started converse as I am talking with you in plain human language. One fine day I was curious and asked "Can you give me the source code of Rain drop Virus" the reaction and answer was " As per the Norms set I am Unable to provide the code due to Security reason". Then suddenly it came up with a interactive page simulating raindrop virus. "This happens to DOS screen when Raindrop virus infects your system". Very interesting....
To measure capability i asked "I need an Editor/IDE similar to VSCodium but using QT/C++" in seconds it provided a code which is compilable and though not all features are available but look and feel is same and style is same. Every aspects of interface was there well only a normal editor not versatile as VScodium.
Yes what you see is above VSCodium clone in QT/C++ now it is up to the user to modify 😁
Today I saw some post about replica of Adobe suite, like Lightroom Photoshop etcc cloned using AI yes it might be possible and it can be done. But is it Ethical?
The IP dilemma:
The LightCraft clone of Lightroom from Adobe is no way code level compatible. it is a pure clean room implementation using Rust , But still the coding effort, cost and R&D of a team is replicated in few days!!! and it is free !!!
Next logical question is who owns it? The AI engine has generated the code. Is the Engine holds the right, The user who created the text prompt has the right? or the organization which created the Model holds the right? or the whole of the mankind upon whose texts on internet the model is trained holds the right?
Tools I created for AI using AI
Well Now few tools I have created using AI for AI...
Narada Chat bot
Bandhu A desktop client like "chat box" using Python it is multi server capable even it can use local OLLAMA service too
Bandhu_hub A daemon using python and flask with same capability but interface is web browser.
If any of you is interested then contact me. Why I have not published ... due to IP dilemma
In part 2, AND gate and OR gate could be trained using a single perceptron. But XOR gate could not be as it is not linearly separable. The XOR function can be represented as
\begin{align}
y &= x_{1}\bar{x_{2}}+\bar{x_{1}}x_{2} \\
z_{1} &= x_{1}\bar{x_{2}} \\
z_{2} &= \bar{x_{1}}x_{2} \\
y &= z_{1}+z_{2}
\end{align}
Looking at the above equations
\begin{align}
z_{1} &= x_{1} \land \bar{x_{2}} \\
z_{2} &= \bar{x_{1}} \land x_{2} \\
y &= z_{1} \lor z_{2}
\end{align}
y is splited into 2 equation z1 and z2, 2 linearly separable functions hence it can not be realized using a single neuron. Then the solution will be a multiple neurons. The organization of multiple neurons stacked in different layers is known as "Multi Layer Perceptron". In this part we will see a very basic multi layer perceptron to solve XOR problem. This chapter involves calculus differential. For Information purpose basic things will be discussed but no in-depth calculus will be used. It is assumed the reader is well aware of derivatives.
Multi layer Perceptron
A Multi layer perceptron (MLP) is a network of neurons arranged in more than one layer. That is Output, Hidden and Input. Generally Input Layer is a direct connecting layer. Hidden layer is where the real magic happens and the output layer is interface to user. A hidden layer is called as such because user doesn't interact with that layer. In an MLP multiple hidden layers may be present. But there is a limitation which is due to "Vanishing Gradient Problem". If interested then There are enough material present on internet.
In an MLP with one input layer, one hidden layer and one output layer, and the network is fully connected that is each neuron of a layer is connected each neuron of immediate nearby layer. Please See the picture Bellow.
One thing to observe here is Hidden layer. Each the neurons in this layer has connections from each neuron in Input layer, and they are connected to output neurons. This is a simple fully connected neural network with 4 inputs, 5 hidden and one output neuron. In Future multiple output networks will be discussed and used.
The connections between layers are weights. These weights are generally represented using matrices. for convenience we will represent each weight as wij where i is ith neuron of next layer and j is jth neuron of current layer. In above case weight matrix between input and hidden layer should be defined as a matrix Wih[5,4]=> 5 rows and 4 columns and the weight matrix between hidden and output is Who[1,5]. For Input, hidden output and final output the matrix will be a column matrix.
Forward Propagation
Input data moves through the network and creates some output. This is called forward propagation. Previously in single neuron, the calculation was easier. Here in case of multiple layers it is a bit involved. It involves matrix multiplication. Consider above Image.input arriving at ith neuron of hidden layer can be expressed as:
\begin{equation}
h_{i} = w_{i0}x_{0}+w_{i1}x_{1}+w_{i2}x_{2}+w_{i3}x_{3}
\end{equation} or in general terms it can be expressed as
\begin{equation}
h_{i} = \sum_{j=0}^{n}w_{ij}x_{j}
\end{equation}
If set of equation for hi is observed closely then it is seen that it is result of matrix multiplication.
$$
\begin{bmatrix}
h_{0} \\
h_{1} \\
h_{2} \\
h_{3} \\
h_{4}
\end{bmatrix}
=
\begin{bmatrix}
w_{00}^{ih}&w_{01}^{ih}&w_{02}^{ih}&w_{03}^{ih} \\
w_{10}^{ih}&w_{11}^{ih}&w_{12}^{ih}&w_{13}^{ih} \\
w_{20}^{ih}&w_{21}^{ih}&w_{22}^{ih}&w_{23}^{ih} \\
w_{30}^{ih}&w_{31}^{ih}&w_{32}^{ih}&w_{33}^{ih} \\
w_{40}^{ih}&w_{41}^{ih}&w_{42}^{ih}&w_{43}^{ih}
\end{bmatrix}
\times
\begin{bmatrix}
x_{0} \\
x_{1} \\
x_{2} \\
x_{3}
\end{bmatrix}
$$
$$H = W^{ih}.X$$
This resultant matrix need to be applied activation function of hidden layer to produce output. Thus generated values is input to the out put layer(in our case output neuron). As previous layer we need to do a matrix multiplication to find the input to the output layer and apply activation function of output layer to get the network output.
$$
\begin{bmatrix}
O_{0}
\end{bmatrix}
=
\begin{bmatrix}
w_{0}^{ho}&w_{1}^{ho}&w_{2}^{ho}&w_{3}^{ho}&w_{4}^{ho}
\end{bmatrix}
\times
\begin{bmatrix}
h_{0} \\
h_{1} \\
h_{2} \\
h_{3} \\
h_{4}
\end{bmatrix}
$$
$$ O = W^{ho}.H $$
Back Propagation
The Output O is dependent on weights Who and H. H is dependent on weights Wih. As the selection of initial weights are random, hence they are unlikely to be perfect. Hence the output O is different than expected or target. The difference of Target and Output is known as Error. This error has to be compensated by adjusting weights Who and Wih. The adjustment of weights starts from last layer and goes down to beginning. Hence it is called Back Propagation.
Back propagation of errors has to be adjusted among different layers, depending on how much each layer contributed. To find out contribution of error a partial derivative of error has to be found out with respect to different contributing elements. Description and derivation of this is beyond the scope as it will lengthen the article. For reference one can watch this video
We can represent the updated weight matrix of a layer as follows
$$ W_{new}=W_{cur}+\alpha[target-actual output].X$$
where
X is input to the layer
α is learning rate
Implementation
To adjust weights in Who>, we need to find error in output, then apply derivative of activation function to it. Next multiply by scalar learning rate(α). This finish scalar calculations. The resultant multiplied by the Input to the layer that is activated output of hidden layer will give us delta weights. this delta weights added to the weight matrix Woh gives updated weights in Hidden to Output layer.
Same procedure have to be repeated for input to hidden layer. To begin with we need to know error at hidden layer first. This can be found by multiplying (Woh)T with error at output. Rest procedures are same as above.
With this we are ready to make a very small neuralnet and train it for XOR problem. MNIST dataset can also be trained but needs patience and time( I achieved with 91.6% accuracy). Iris Dataset and wheat seed data set are very easily trained. Bellow is XOR problem implementation.
Though I may put all the code here or on github, but I would like the reader to implement his own version analogous to this implementation. In any problem Contact me for the Full code its free But before asking please try to implement yourself.
Till now we discussed
What is an artificial neuron
How to implement it
What are the areas it can be used(linear separable)
How to train it
What is XOR problem
What is Multi layer neural network
How to implement one
How to train
Example of XOR implementation
This is the end of 3 parts for simple neural network. Part1 and Part2 can be found here. In some future article we will discuss Convolutional Neural Network(CNN)