Wednesday, 18 November 2026

🦆Part 5 — Looking Forward

What Phase 1 has

  • 75 tests passing on both backends
  • Structs, struct params, arrays of structs, local structs, nested structs
  • Array parameters of any element type (INT[], FLOAT[], CHAR[], STRUCT[])
  • Fixed-point Q16:16 arithmetic on both backends
  • SERIALIZE / DESERIALIZE with locked byte layout
  • File I/O (open, read, write, seek, tell, size, delete, flush, close)
  • Keyboard input (INKEY$, INPUT$(n))
  • SELECT CASE, FOR, WHILE, IF / ELSEIF / ELSE
  • String concatenation in PRINT
  • Hex / binary literals (0x, 0b, &H, &B)

What Phase 2 will add

  1. RND / RANDOMIZE — trivial to implement (LCG on NASM, rand() on C). Already planned.
  2. term.dkm — ANSI escape sequence parser in the standard library, not the compiler.
  3. resolve_name refactor — preventive cleanup, not blocking.
  4. FFI — for graphics, sound, networking, GUI/RAD. This is where the “gun on the platform’s shoulder” architecture becomes a feature rather than an implementation detail.
  5. OO — classes, methods, inheritance. Phase 2 language design.
  6. duckdb3.dkb — the milestone proof-of-concept: a DuckDB-flavored table engine with indexes, queries, and file I/O.
  7. Fullscreen editor — the “does this actually work?” test.

The architectural philosophy

Three principles emerged from building duckBASIC:

1. Platform where possible, own where necessary. The C backend uses rand(), strcpy(), memcpy(), printf() — all platform-provided. The NASM backend implements fixed_mul (60 lines of asm), _read_str (BIOS INT 21h, AH=0Ah), _print_int (custom decimal printer) — all hand-written, because the platform (bare 16-bit DOS) has nothing.

The “gun on the platform’s shoulder” philosophy is about not reimplementing what you can depend on, and owning what you can’t. Every function in runtime_c.c is a thin wrapper over libc; every function in runtime_com.inc is a hand-written implementation. The distinction is principled, not accidental.

2. Byte layout is a contract. SERIALIZE / DESERIALIZE works because the byte layout is documented, locked, and tested. It’s the reason a .dkb program can serialize on C and deserialize on NASM. It’s also the reason C’s int (4 bytes) vs NASM’s int (2 bytes) was caught early — as a design decision, not a bug.

3. Bug classes need bug-class fixes, but not always immediately. The three-path name lookup is a bug class. The right answer is resolve_name. The right timing is when you have a failing test. duckBASIC got both right: it fixed the symptom narrow (with tests 74–78 as proof), and deferred the refactor with a clear trigger condition. That’s what a mature codebase does.


Conclusion

duckBASIC is not a production compiler. It’s not going to replace GCC or LLVM, and it’s not going to be used to write the next PostgreSQL. What it is, though, is a complete implementation of a compiler pipeline with two real backends, a real test suite, and enough features to build real programs (Phase 2’s fullscreen editor is the proof point).

The project demonstrates several things that are worth writing down:

  • Two backends is better than one. The C backend made iteration fast; the NASM backend made every bug real.
  • Fixed-point math on 8086 is a solved problem if you’re willing to write 200 lines of assembly.
  • Byte layout is the hardest part of any serialization feature, and the easiest way to get it wrong is to trust that the platforms agree.
  • Cross-backend test suites catch things that single-backend test suites miss. Every bug I’ve described in this series was caught by a cross-backend test.
  • Refactoring is a response to pain, not a preventive measure. The right time to unify three name-lookup paths is when a test fails because they diverge — not before.

If you’re building your own compiler, I hope this series gives you some concrete patterns to steal. The test harness (dual backends, DOSBox verification) is the single most valuable design decision in the entire project, because it turns every bug into a reproducible failure rather than a suspected one. That’s worth more than any amount of clever parser design.

The complete Source code will be available to those who are interested


//factorial recursion in DuckBASIC.
FUNCTION factorial(n AS INT) AS INT
{
    IF n <= 1
    {
        RETURN 1
    }
    RETURN n * factorial(n - 1)
}

PRINT factorial(5)

The bellow image is the look of duckedit, the editor written in DuckBASIC

🦆Part 4 — Case Study: The Local Struct Bug, and What It Taught Us

The symptom

At V5.9d.8+.9 (the last commit before this work), a .dkb program containing:

FUNCTION foo()
{
    DIM tmp AS Point
    tmp.x = 5    // local struct field write
}

typechecked fine, but on the NASM backend, the emitted assembly contained:

mov [bp-4], ax      ; correct: local struct field write
mov [var_], ax      ; BUG: empty variable name

The second store wrote to a data-segment symbol with an empty name — which is to say, it wrote to whatever happened to be at var_ in the data segment. This was silent corruption.

Root cause diagnosis

The session notes said this:

Three separate name-lookup paths (parser, typechecker, NASM codegen) each handle global/param/local differently. Local struct vars fall through the cracks in some paths.

That was correct in spirit, but the specific bug was narrower: the NODE_ASSIGN case in gen_nasm_stmt.c had a series of if blocks for different LHS types:

if (lhs is field of array-of-struct) { ...; break; }
if (lhs is field of struct) {
    if (is_param) { ...; break; }
    if (is_local_struct) { ...; break; }  // ← the fix
    if (is_global_struct) { ...; break; }
}
if (lhs is array element) { ...; break; }

// FALLTHROUGH: scalar assignment
fprintf(out, "    mov [var_%s], ax\n", node->left->attr.name);

The scalar fallback read node->left->attr.name. But for a NODE_FIELD_ACCESS, that union field aliases node->left->attr.fieldAccess.object, which is a pointer. So attr.name was a garbage pointer, usually pointing at an empty string in memory. Hence mov [var_], ax.

The fix was to add the is_local_struct check before falling through. But the real fix — the one that eliminates the bug class — is to have a single resolver that all three passes use.

The proposed refactor

typedef enum {
    RESOLVE_GLOBAL,
    RESOLVE_PARAM,
    RESOLVE_LOCAL,
    RESOLVE_FAIL
} ResolveKind;

typedef struct {
    ResolveKind kind;
    int         offset;    // stack offset for param/local
    Symbol     *global;    // valid if kind == RESOLVE_GLOBAL
    StructField *field;    // valid if kind == RESOLVE_PARAM or LOCAL
} ResolvedName;

ResolvedName resolve_name(const char *name);

Backed by one linear walk:

ResolvedName resolve_name(const char *name) {
    ResolvedName r = {0};
    if (current_function) {
        // 1. Params
        StructField *p = current_function->metadata.function.parameters;
        while (p) {
            if (strcmp(p->field_name, name) == 0) {
                r.kind = RESOLVE_PARAM;
                r.offset = 4 + p->offset;
                r.field = p;
                return r;
            }
            p = p->next;
        }
        // 2. Locals
        StructField *l = current_function->metadata.function.locals;
        while (l) {
            if (strcmp(l->field_name, name) == 0) {
                r.kind = RESOLVE_LOCAL;
                r.offset = l->offset;
                r.field = l;
                return r;
            }
            l = l->next;
        }
    }
    // 3. Globals
    Symbol *g = symtable_lookup(global_sym_table, name);
    if (g) { r.kind = RESOLVE_GLOBAL; r.global = g; return r; }
    r.kind = RESOLVE_FAIL;
    return r;
}

Every existing branch in gen_nasm_expr.c and gen_nasm_stmt.c that walks function.parameters / function.locals / symtable_lookup inline would be replaced with resolve_name(name). That’s ~15 call sites, each simplified from 15–20 lines to 3–5.

This refactor was deferred. The bug was empirically dead — tests 74–78 (local structs in FOR loops, local struct arrays, struct-to-struct assignment, SERIALIZE through struct params) all passed on both backends. The resolver refactor is now scheduled as a Phase 2 architectural cleanup, to be triggered by a concrete failure (e.g., a duckdb3 bug that requires it) rather than a hypothetical one.

The lesson

The three-path name lookup isn’t a bug. It’s a bug class. Every new node type needs to know about it, and forgetting is invisible until something falls through. The session that fixed it did so by:

  1. Writing failing tests for every context (local, param, array, struct, loop body).
  2. Fixing the bug narrowly, confirming all tests pass.
  3. Deferring the resolver refactor because the narrow fix was correct.

The right answer to a bug class is usually “refactor to prevent recurrence” — but the right timing is usually “when you have a failing test.” This session’s decision was correct: the fix was tested comprehensively, and the refactor was deferred with a clear trigger condition. That’s how mature projects handle technical debt.

🦆Part 3 — Backends: C Transpiler and 16-bit NASM

The C backend

The C backend is the simpler of the two. It emits C source and lets gcc do the hard work.

Structure:

  • gen_c.c — top-level entry: emits preamble, structs, forward decls, globals, function definitions, then main().
  • gen_c_stmt.c — statements (DIM, ASSIGN, IF, WHILE, PRINT, etc.).
  • gen_c_expr.c — expressions (arithmetic, function calls, field access, array access).
  • gen_c.h — CGen state struct threaded through all emitters.

The runtime is a C file (runtime_c.c) that’s compiled into a string literal at build time and injected into every generated .c file. It provides _check_bounds, sys_open, sys_read, etc.

Key design decisions:

  1. All state is in a CGen struct. No globals. This makes reentrancy possible (though not currently used) and makes threading state through emitters explicit.

  2. Type mapping is done via static inline functions in gen_c.h — no separate TU needed.

  3. Array accesses get bounds checks via a runtime helper:

    buf[_check_bounds(i, 16, "buf")]
    

    The helper prints an error and calls exit(1) on violation.

  4. String assignment uses strcpy, comparison uses strcmp. The typechecker ensures STRING ops only happen where they’re valid.

  5. Struct field access emits .field for variables, ->field for params. The codegen knows which is which because it tracks current_function and checks the param list.

The NASM backend

The NASM backend is where the real engineering lives. Every operation must be expressed in 16-bit x86, with no libc, no linker, no relocations — just a flat binary loaded at 0x100.

Structure:

  • gen_nasm.c — entry point, string pool, symbol lookup, program emission.
  • gen_nasm_stmt.c — statements.
  • gen_nasm_expr.c — expressions.
  • gen_nasm_data.c — data segment emission, struct field offset lookup.
  • runtime_com.inc — runtime library (print, read, copy, etc.).
  • fixed.inc — Q16:16 fixed-point math.
  • fileio.inc — DOS file I/O wrappers.

Key design decisions:

  1. Every value lives in AX (and DX for FLOAT). This is a naive register allocation strategy — no spilling, no liveness analysis. It’s slow but correct.

  2. Binary operations use the stack:

    ; evaluate left into AX
    push dx
    push ax
    ; evaluate right into AX
    push dx
    push ax
    pop bx      ; right low
    pop cx      ; right high
    pop ax      ; left low
    pop dx      ; left high
    ; now DX:AX = left, CX:BX = right
    
  3. Q16:16 fixed-point arithmetic — no floating point unit on 8086, so all “floats” are stored as 32-bit fixed-point (16.16). Addition is add/adc. Multiplication and division are hand-written 32-bit signed routines in fixed.inc, each ~60 lines of assembly.

  4. String constants live in a string pool — deduplicated, emitted at the end of the binary, referenced as str_N.

  5. Everything is in a .COM file — flat binary, loaded at 0x100, no relocations. org 0x100 at the top.

  6. Runtime library is .inc-included — no linking. The %include directives pull in runtime_com.inc, fixed.inc, fileio.inc at assembly time.

The SERIALIZE / DESERIALIZE byte layout

Both backends implement a locked byte layout for struct serialization:

Field type Bytes
INT 2 (little-endian)
BOOL 2 (little-endian)
FLOAT (Q16:16) 4 (little-endian)
CHAR 1
STRING 256 (fixed-size, null-padded)

The layout is documented in both gen_c_stmt.c and gen_nasm_stmt.c, and it’s the contract that allows a .dkb program to serialize a struct on the C backend and deserialize it on NASM (or vice versa, across process boundaries).

The C backend emits byte-by-byte memcpy calls, truncating int to short on serialize and sign-extending on deserialize. This is necessary because C’s native int is 4 bytes but duckBASIC’s INT is 2 bytes — the layout must match the NASM backend’s.

The NASM backend emits mov sequences (and rep movsb for strings). No truncation needed since NASM’s INT is natively 2 bytes.

The byte layout was a deliberate design decision: it made SERIALIZE / DESERIALIZE work identically on both backends without any translation layer, and it exposed a number of lurking bugs (like C’s int being 4 bytes) that would have been catastrophic later.

🦆 Part 2 — Front End: Lexer, Parser, Symbol Table, Type Checker

Lexer (lexer.c)

The lexer is hand-written, single-pass, token-at-a-time. Nothing fancy — no DFA tables, no regex, just a switch on the first character.

Notable features:

  • Radix literals: 0x1F (hex), 0b1010 (binary), &HFF (QBasic-style hex), &B1010 (QBasic-style binary). The &H/&B forms are particularly nice because they round-trip cleanly to QBASIC programmers.
  • String escapes: \n, \t, \r, \\, \", \', \0.
  • Character literals: 'a' (single quotes). This interacts badly with QBASIC-style ' comments, which duckBASIC does not support — comments are //.
  • Token names array with a _Static_assert guarding against enum drift.

The lexer is intentionally simple. No diagnostic recovery, no multi-byte UTF-8, no raw strings. Every token is either recognized or produces TOKEN_ERROR and the parser will halt.

AST (ast.h, ast.c)

The AST is a tagged union of node types. Every node is an ASTNode struct:

typedef struct ASTNode {
    ASTNodeType type;
    DataType evalType;       // resolved type, set by typechecker
    bool print_no_newline;
    int line;

    struct ASTNode *left;
    struct ASTNode *right;
    struct ASTNode *next;    // statement chaining

    union {
        char *name;                          // identifier
        int32_t intVal;                      // int literal / float raw bits
        char *strVal;                        // string literal
        unsigned char charVal;               // char literal

        struct { ... } ifNode;
        struct { ... } whileNode;
        struct { ... } forNode;
        struct { ... } arrayAccess;
        struct { ... } functionDecl;
        struct { ... } functionCall;
        struct { ... } structDecl;
        struct { ... } fieldAccess;
        struct { ... } structVar;
        struct { ... } selectNode;
        struct { ... } caseNode;
        struct { ... } serializeNode;
    } attr;
} ASTNode;

The union keeps nodes compact (about 100 bytes each). The next pointer chains statements into a linked list, so a block is just a NODE_BLOCK whose left is the first statement and each statement’s next points to the following one.

The union overlap is a known footgun. attr.name (a char*) and attr.fieldAccess.object (also a pointer) occupy the same bytes. Reading attr.name on a NODE_FIELD_ACCESS produces a garbage pointer, which is exactly what caused the mov [var_], ax bug we spent a session debugging.

Symbol table (symtable.c)

The symbol table is a linked list of Symbol structs. It uses three pointer fields to maintain two separate lists simultaneously:

typedef struct Symbol {
    char name[128];
    DataType type;
    int offset;
    int class_type;          // SYM_SCALAR, SYM_ARRAY, SYM_STRUCT_DEF, etc.
    int is_local;
    int is_builtin;
    short stack_offset;

    union {
        struct { ... } array;
        struct { ... } struct_def;
        struct { ... } struct_var;
        struct { ... } function;
        struct { ... } constant;
    } metadata;

    struct Symbol *next;       // active scope lookup chain
    struct Symbol *all_next;   // master allocation chain (for cleanup)
} Symbol;
  • next is the active lookup chain — it changes as scopes are pushed and popped.
  • all_next is the master allocation chain — never modified after insertion. This is how symtable_free iterates all symbols for cleanup.
  • head points to the current scope’s head. all_head points to the master list.

Scope push/pop is done via snapshots:

ScopeSnapshot snap = symtable_save_scope(p->symTable);
// ... insert local symbols ...
symtable_restore_scope(p->symTable, snap);

Restoring just resets table->head to the snapshot’s head. Memory is never freed until symtable_free at the very end.

This design is simple but not scalable — lookups are O(n) linear scans. For a 16-bit compiler, that’s fine. For a production compiler, you’d want a hash table.

Parser (parser.c + parser_*.c)

The parser is recursive descent for statements and precedence climbing for expressions. Each grammar production is a function.

The parser is split across multiple files by grammar area:

  • parser.c — core (statement dispatch, expression engine, block parser, DIM handling)
  • parser_if.c, parser_while.c, parser_for.c — control flow
  • parser_function.c — function declarations, calls, returns
  • parser_struct.c — struct definitions
  • parser_array.c — array access
  • parser_select.c — SELECT CASE
  • parser_serialize.c — SERIALIZE / DESERIALIZE
  • parser_import.c — IMPORT statement (file inclusion)

The expression engine uses precedence climbing:

ASTNode* parse_expression(Parser *p, int precedence)
{
    // 1. Parse a prefix (literal, identifier, unary operator, paren)
    // 2. While the next token has higher precedence than the argument:
    //    - Consume the operator
    //    - Parse RHS with the operator's precedence
    //    - Build a binary node
    return left;
}

Precedence levels (highest to lowest):

  1. . (field access) — 12
  2. [ (array subscript) — 11
  3. * / % — 10
  4. + - — 9
  5. << >> — 8
  6. < <= > >= — 7
  7. == != — 6
  8. & — 5
  9. ^ — 4
  10. | — 3
  11. && — 2
  12. || — 1

Error recovery is panic-mode: on failure, skip tokens until a \n, }, or EOF, then continue. This means a syntax error in one statement doesn’t cascade into forty subsequent errors.

Type checker (typecheck.c)

The type checker walks the AST after parsing and before codegen. It does two things:

  1. Annotates every expression node with evalType so codegen can trust it without re-deriving.
  2. Validates type compatibility and increments g_error_count on mismatch.

Key rules:

  • BOOL cannot appear in arithmetic.
  • STRING + is only valid inside PRINT (concatenation).
  • Array indices must be INT.
  • Function call arguments must be type-assignable.

The typechecker re-creates function scope by re-inserting parameters and locals into the symbol table, then checks the body, then restores scope. This is a second pass over the AST — the parser already did one symbol-table pass — which is why there are two name-lookup paths (parser + typechecker) plus the codegen’s third path.

This duplication is the source of the “three lookup paths” bug class. Fixing it properly would require a unified resolve_name helper used by all three passes — a Phase 2 refactor.

🦆 duckBASIC: A Modern 16-bit BASIC Compiler — Design & Implementation

🦆 Introduction

This series documents the design and implementation of duckBASIC, a compiler for a modern dialect of BASIC. The Philosophy is to create a usable teachable tool for students or hobyists who are interested in Compiler and interpreter writing. DuckBASIC targets both C (for portability) and 16-bit real-mode MS-DOS via NASM assembly (.COM executables). The project was built iteratively, feature by feature, with a test suite that validates both backends in parallel against identical source programs.

Old timers who have soiled their hands with QBASIC/GWBASIC/BASICA/VB may find similar environment but in C style 🦆🦆🦆

Over the course of development, duckBASIC grew from a simple expression evaluator into a compiler with:

  • Two full backends (C transpiler and NASM 16-bit assembler emitter)
  • Structured types: structs, struct parameters, arrays of structs, local structs, nested structs
  • Fixed-point Q16:16 arithmetic implemented in hand-written x86 assembly
  • A locked byte layout for SERIALIZE/DESERIALIZE that guarantees cross-backend binary compatibility
  • A real file I/O API (SYS_OPEN, SYS_READ, SYS_WRITE, SYS_SEEK, etc.)
  • Cross-platform keyboard primitives (INKEY$, INPUT$(n)) and array parameters (INT[], FLOAT[], CHAR[], STRUCT[])
  • 75+ tests running on both backends, most of which are verified by running the resulting .COM file inside DOSBox and comparing output byte-for-byte

The project is interesting not because it implements a compiler — plenty of toy compilers exist — but because it implements two of them simultaneously, with a shared test suite, and verifies they produce identical output on real 16-bit hardware.


Part 1 — Architecture Overview

The pipeline

Every compiler is a pipeline, and duckBASIC’s is deliberately simple:

Source (.dkb)
    ↓
Lexer (tokens)
    ↓
Parser (AST + symbol table)
    ↓
Semantic resolution (function calls)
    ↓
Type checker (evalType annotations)
    ↓
Code generation (two backends)
    ↓
    ├── C source (.c) → gcc → native binary
    └── NASM source (.asm) → nasm → .com → DOSBox

The top-level structure

The compiler is a single executable, duckbas, built from ~15 C files. The entry point in main.c orchestrates the phases:

// Phase 1: Lexical analysis
lexer_init(&compiler_lexer, source_code);

// Phase 2: Parsing
symtable_init(&compiler_table);
parser_init(&compiler_parser, &compiler_lexer, &compiler_table);
ASTNode *program_tree = parser_parse_program(&compiler_parser);

// Phase 2.5: Semantic resolution (function calls)
resolve_function_calls(program_tree, &compiler_table);

// Phase 2.75: Type checking
int tc_errors = typecheck_program(&compiler_parser, program_tree);

// Gate: if any phase errored, stop before codegen
if (g_error_count > 0) { ... exit(1); }

// Phase 5: Code generation
if (opt.target == TARGET_C) {
    gen_c_program(program_tree, &compiler_table, output_path);
} else {
    gen_nasm_program(program_tree, &compiler_table, output_path);
}

Three global counters (g_error_count, g_warning_count, g_verbosity) coordinate diagnostics across phases.

The two backends

The C backend transpiles duckBASIC AST to C source, which is then compiled by gcc. This is the “reference” backend — anything that works here is by definition semantically correct. It’s also the fast one to iterate on: change a C emitter, recompile with gcc, done in 2 seconds.

The NASM backend emits 16-bit x86 assembly for MS-DOS .COM executables. This is the “authentic” backend — every feature must be implemented in raw assembly or wrapped around DOS/BIOS interrupts. It’s the hard one: fixed_mul is 60 lines of hand-rolled signed multiplication, _read_str uses BIOS INT 21h, AH=0Ah, SYS_OPEN wraps INT 21h, AH=3Dh.

The two backends share nothing at the codegen level. They each have their own AST walkers (gen_c_stmt.c, gen_c_expr.c vs gen_nasm_stmt.c, gen_nasm_expr.c). But they share the entire front end — parser, symbol table, type checker — which is what guarantees semantic consistency.

The test harness

The test suite (tests/*.dkb) runs each program through both backends and compares:

  1. C backend output — compile to .c, gcc, run, capture stdout.
  2. NASM backend assembles — nasm -f bin, verify it produces a .com.
  3. NASM output in DOSBox — run .com inside DOSBox, capture output, compare byte-for-byte with C output.

For interactive tests (INKEY$, keyboard input), DOSBox runs with a scripted keystroke file.

The suite currently has 75 tests, 72 pass, 0 fail, 3 skip. The skipped tests are Phase 2 features (graphics, sound, etc.) that aren’t implemented yet.

Wednesday, 14 October 2026

AI Adventures...

Hi Friends,
Whats Up? A long Period of Silence... during this very long silence period I was preparing for the Fourth phase of Life(Ascetic life). Did n't touch PC, nor written a single line of code. Well only helped desirous students to clarify their doubts on paper though. Was on a pilgrimage to Amarkantak, MadhyaPradesh. Went on  Narmada Parikrama (circumambulation of River Narmada) of 3500 Km approx by walk, + bus + train... It was a different Kind of experience...

Well here I want to share some AI adventure currently I an doing/ done. Few days back I was suggested by well wishers and My cardiac doctor to restart Computer adventure. In Previous post we talked about how this AI function. How it is trained. Now we hear AI in every corner... Even those who don't know a single thing about technology are also talking about AI.. AI ko puchh wo bataadega So I started with the commonly available AI chat bot "Gemini". then gradually i tried "Meta" and "Deepseek" and of course "ChatGPT". 

If they say the Power of AI is immense and it can annihilatehuman civilization... then it will be a over thought over reaction. To me AI is nothing but a Set of Weights in a Matrix. as We have seen in my previous posts.  But Using AI judiciously we can change the world.

Mark above "Using AI" not AI by itself. there is a subtle difference. During past few months I wrote a Web hosted AI chat bot "Narada"  it was using a small free model to chat, and quite accurately answered critical question about astronomy and astrology. One may ask why astrology? well astrology depends on Astronomy for calculations so if i ask about astrology then it has to use astronomy hence my idea to test the capability is achieved.

The test plan :

Instead of supplying with my DoB and Time of Birth, I provided my Natal chart and Place of birth. Asked it to spit out my date of birth. It really amazing, it spit out few dates and started resoning and said this is your DoB, and it is correct. I repeted this experiment with all the standard chat bots i mentioned and they too did the same.

Level Next.: 

Well after that I developed a respect for these chat bots and started converse as I am talking with you in plain human language. One fine day I was curious and asked "Can you give me the source code of Rain drop Virus" the reaction and answer was " As per the Norms set I am Unable to provide the code due to Security reason". Then suddenly it came up with a interactive page simulating raindrop virus. "This happens to DOS screen when Raindrop virus infects your system". Very interesting....  

To measure capability i asked "I need an Editor/IDE similar to VSCodium but using QT/C++" in seconds it provided a code which is compilable and though not all features are available but look and feel is same and style is same. Every aspects of interface was there well only a normal editor not versatile as VScodium. 

 Yes what you see is above VSCodium clone in QT/C++ now it is up to the user to modify 😁

Today I saw some post about replica of Adobe suite, like Lightroom Photoshop etcc cloned using AI yes it might be possible and it can be done. But is it Ethical?

The IP dilemma: 

The LightCraft clone of Lightroom from Adobe is no way code level compatible. it is a pure clean room implementation using Rust , But still the coding effort, cost and R&D of a team is replicated in few days!!! and it is free !!! 

Next logical question is who owns it? The AI engine has generated the code. Is the Engine holds the right, The user who created the text prompt has the right? or the organization which created the Model holds the right? or the whole of the mankind upon whose texts on internet the model is trained holds the right?

Tools I created for AI using AI

Well Now few tools I have created using AI for AI... 

  1. Narada Chat bot
  2. Bandhu A desktop client like "chat box" using Python it is multi server capable even it can use local OLLAMA service too
  3. Bandhu_hub A daemon using python and flask with same capability but interface is web browser.

If any of you is interested then contact me. Why I have not published ... due to IP dilemma  

 

Tuesday, 2 May 2023

Neural Network from Scratch Using C (Part-3)

 

XOR Problem

In part 2, AND gate and OR gate could be trained using a single perceptron. But XOR gate could not be as it is not linearly separable. The XOR function can be represented as \begin{align} y &= x_{1}\bar{x_{2}}+\bar{x_{1}}x_{2} \\ z_{1} &= x_{1}\bar{x_{2}} \\ z_{2} &= \bar{x_{1}}x_{2} \\ y &= z_{1}+z_{2} \end{align} Looking at the above equations \begin{align} z_{1} &= x_{1} \land \bar{x_{2}} \\ z_{2} &= \bar{x_{1}} \land x_{2} \\ y &= z_{1} \lor z_{2} \end{align} y is splited into 2 equation z1 and z2, 2 linearly separable functions hence it can not be realized using a single neuron. Then the solution will be a multiple neurons. The organization of multiple neurons stacked in different layers is known as "Multi Layer Perceptron". In this part we will see a very basic multi layer perceptron to solve XOR problem. This chapter involves calculus differential. For Information purpose basic things will be discussed but no in-depth calculus will be used. It is assumed the reader is well aware of derivatives.

Multi layer Perceptron

A Multi layer perceptron (MLP) is a network of neurons arranged in more than one layer. That is Output, Hidden and Input. Generally Input Layer is a direct connecting layer. Hidden layer is where the real magic happens and the output layer is interface to user. A hidden layer is called as such because user doesn't interact with that layer. In an MLP multiple hidden layers may be present. But there is a limitation which is due to "Vanishing Gradient Problem". If interested then There are enough material present on internet. In an MLP with one input layer, one hidden layer and one output layer, and the network is fully connected that is each neuron of a layer is connected each neuron of immediate nearby layer. Please See the picture Bellow.

One thing to observe here is Hidden layer. Each the neurons in this layer has connections from each neuron in Input layer, and they are connected to output neurons. This is a simple fully connected neural network with 4 inputs, 5 hidden and one output neuron. In Future multiple output networks will be discussed and used.

The connections between layers are weights. These weights are generally represented using matrices. for convenience we will represent each weight as wij where i is ith neuron of next layer and j is jth neuron of current layer. In above case weight matrix between input and hidden layer should be defined as a matrix Wih[5,4]=> 5 rows and 4 columns and the weight matrix between hidden and output is Who[1,5]. For Input, hidden output and final output the matrix will be a column matrix.

Forward Propagation

Input data moves through the network and creates some output. This is called forward propagation. Previously in single neuron, the calculation was easier. Here in case of multiple layers it is a bit involved. It involves matrix multiplication. Consider above Image.input arriving at ith neuron of hidden layer can be expressed as: \begin{equation} h_{i} = w_{i0}x_{0}+w_{i1}x_{1}+w_{i2}x_{2}+w_{i3}x_{3} \end{equation} or in general terms it can be expressed as \begin{equation} h_{i} = \sum_{j=0}^{n}w_{ij}x_{j} \end{equation} If set of equation for hi is observed closely then it is seen that it is result of matrix multiplication. $$ \begin{bmatrix} h_{0} \\ h_{1} \\ h_{2} \\ h_{3} \\ h_{4} \end{bmatrix} = \begin{bmatrix} w_{00}^{ih}&w_{01}^{ih}&w_{02}^{ih}&w_{03}^{ih} \\ w_{10}^{ih}&w_{11}^{ih}&w_{12}^{ih}&w_{13}^{ih} \\ w_{20}^{ih}&w_{21}^{ih}&w_{22}^{ih}&w_{23}^{ih} \\ w_{30}^{ih}&w_{31}^{ih}&w_{32}^{ih}&w_{33}^{ih} \\ w_{40}^{ih}&w_{41}^{ih}&w_{42}^{ih}&w_{43}^{ih} \end{bmatrix} \times \begin{bmatrix} x_{0} \\ x_{1} \\ x_{2} \\ x_{3} \end{bmatrix} $$ $$H = W^{ih}.X$$

This resultant matrix need to be applied activation function of hidden layer to produce output. Thus generated values is input to the out put layer(in our case output neuron). As previous layer we need to do a matrix multiplication to find the input to the output layer and apply activation function of output layer to get the network output. $$ \begin{bmatrix} O_{0} \end{bmatrix} = \begin{bmatrix} w_{0}^{ho}&w_{1}^{ho}&w_{2}^{ho}&w_{3}^{ho}&w_{4}^{ho} \end{bmatrix} \times \begin{bmatrix} h_{0} \\ h_{1} \\ h_{2} \\ h_{3} \\ h_{4} \end{bmatrix} $$ $$ O = W^{ho}.H $$

Back Propagation

The Output O is dependent on weights Who and H. H is dependent on weights Wih. As the selection of initial weights are random, hence they are unlikely to be perfect. Hence the output O is different than expected or target. The difference of Target and Output is known as Error. This error has to be compensated by adjusting weights Who and Wih. The adjustment of weights starts from last layer and goes down to beginning. Hence it is called Back Propagation.

Back propagation of errors has to be adjusted among different layers, depending on how much each layer contributed. To find out contribution of error a partial derivative of error has to be found out with respect to different contributing elements. Description and derivation of this is beyond the scope as it will lengthen the article. For reference one can watch this video

We can represent the updated weight matrix of a layer as follows $$ W_{new}=W_{cur}+\alpha[target-actual output].X$$ where 

  • X is input to the layer 
  • α is learning rate

 

Implementation

To adjust weights in Who>, we need to find error in output, then apply derivative of activation function to it. Next multiply by scalar learning rate(α). This finish scalar calculations. The resultant multiplied by the Input to the layer that is activated output of hidden layer will give us delta weights. this delta weights added to the weight matrix Woh gives updated weights in Hidden to Output layer.

Same procedure have to be repeated for input to hidden layer.  To begin with we need to know error at hidden layer first. This can be found by multiplying (Woh)T with error at output. Rest procedures are same as above.


	// find error in output
	matrix err_out=subtract_matrix(expect,final_out);
	matrix whoT = transpose(nn.who);
	matrix err_hid=mat_mult(whoT,err_out);
	
	// calculate delta errors 
	matrix_map(&final_out,sigmoid_prime);
	mat_mult_hadamard(&err_out,&final_out);
	matrix hidT= transpose(hidden_out);
	matrix delta_who = mat_mult(err_out,hidT);
	mat_mult_scalar(nn.lr,&delta_who);
	
	matrix_map(&hidden_out,sigmoid_prime);
	mat_mult_hadamard(&err_hid,&hidden_out);
	matrix inpT = transpose(inp);
	matrix delta_wih = mat_mult(err_hid,inpT);
	mat_mult_scalar(nn.lr,&delta_wih);
	// update weights
	matrix tempa =add_matrix(nn.who,delta_who);
	assign_matrix(&nn.who,&tempa);
	matrix tempb = add_matrix(nn.wih,delta_wih);
	assign_matrix(&nn.wih,&tempb);

With this we are ready to make a very small neuralnet and train it for XOR problem.  MNIST dataset can also be trained but needs patience and time( I achieved with 91.6% accuracy). Iris Dataset and wheat seed data set are very easily trained. Bellow is XOR problem implementation.


#include <stdio .h >
#include "../mlp1.c"
int main(int argc, char *argv[])
{
	srand(0);
	nnet my_nn =create_nn(2,3,1);

	// training
	
	for(int n = 0; n< 30000; n++)
	{
		matrix inp = create_matrix(2,1);
		matrix expected = create_matrix(1,1);
		
		set_cell(&inp,0,0,1.0);
		set_cell(&inp,1,0,0.0);
		set_cell(&expected,0,0,1);
		train(my_nn,inp,expected);
		
		set_cell(&inp,0,0,1.0);
		set_cell(&inp,1,0,1.0);
		set_cell(&expected,0,0,0);
		train(my_nn,inp,expected);
		
		set_cell(&inp,0,0,0.0);
		set_cell(&inp,1,0,1.0);
		set_cell(&expected,0,0,1);
		train(my_nn,inp,expected);
		
		set_cell(&inp,0,0,0.0);
		set_cell(&inp,1,0,0.0);
		set_cell(&expected,0,0,0);
		train(my_nn,inp,expected);
		
		delete_matrix(&expected);	
		delete_matrix(&inp);
	}
	// save the trained netork
	save_nn(my_nn,"xor2.net");

	// load nn from file
	my_nn = load_nn("xor2.net");
	// test trained network
	matrix inp = create_matrix(2,1);
	set_cell(&inp,0,0,1.0);
	set_cell(&inp,1,0,1.0);
	matrix out = predict(my_nn,inp);
	display_matrix(out);
	delete_matrix(&out);
	set_cell(&inp,0,0,0.0);
	set_cell(&inp,1,0,0.0);
	out = predict(my_nn,inp);
	display_matrix(out);
	delete_matrix(&out);
	set_cell(&inp,0,0,0.0);
	set_cell(&inp,1,0,1.0);
	out=predict(my_nn,inp);
	display_matrix(out);
	delete_matrix(&out);
	set_cell(&inp,0,0,1.0);
	set_cell(&inp,1,0,0.0);
	out = predict(my_nn,inp);
	display_matrix(out);
	
	delete_matrix(&out);
	delete_nn(my_nn);
	delete_matrix(&inp);
	return 0;
}
=======output====
[ashok@fedora xor]$ ./xor
[ 0.00559855 ]

[ 0.06528170 ]

[ 0.96165163 ]

[ 0.96165793 ]

Though I may put all the code here or on github, but I would like the reader to implement his own version analogous to this implementation. In any problem Contact me for the Full code its free But before asking please try to implement yourself.

Till now we discussed

  • What is an artificial neuron
  • How to implement it
  • What are the areas it can be used(linear separable)
  • How to train it
  • What is XOR problem
  • What is Multi layer neural network
  • How to implement one
  • How to train
  • Example of XOR implementation
This is the end of 3 parts for simple neural network. Part1 and Part2 can be found here. In some future article we will discuss Convolutional Neural Network(CNN)
 

Sunday, 30 April 2023

Neural Network from Scratch Using C (Part-2)

 

Implementing a Perceptron

In this part we will start implementing a single perceptron and do some experiments. 

first part can be found here.

A single Perceptron

A single Perceptron(neuron) can do a lot of jobs. Yes Like a Neuron in Our nervous system it can do a lot of things. As One grows from Infants to toddler the neurons in our brain get trained(learned) to coordinate between our limbs, coordinate our activities etc...
Similarly the Perceptrons can be trained to classify data as per the training data. but we can only classify linearly separable data.

What is a linearly separable?
Linear separability can be termed as, if the plot of data can be separated by a single straight line. This means depending on the separation criterion, there exists a straight line dividing the data points in to 2 regions.

Now lets see a small code implementing it step by step. NOTE: We need a matrix manipulation library for all our Neural network adventure. For this purpose my tmatlib on github is quite suitable. Though one should not expect it to perform like blas. In my code i used a similar version known as smatlib. As it is continuously growing I have not published it on github. But stay assure the functionality and performance of both are same. 

// structure of our perceptron.

typedef struct perceptron
{
	int n_inp;   // number inputs
	matrix W;    // weight matrix
	double lr;   // learning rate
	double bias; // bias
}perceptron;

A straight line on a 2-dimentional plain can be expressed as y=mx+c. The Bias parameter in our perceptron is related to c. Our training data is x and y, x is input and y is expected or target. Our perceptron learns the value of m. Learning rate(lr) determines how fast or slow the perceptron should learn. If Learning rate is too low then it may be trapped in local minima. If it is too high then it may oscillate and may not reach the solution. So Bottom line is one has to experiment with learning rate(lr) value to achieve better result. Weight matrix is initialized with random values. Gradually with training it is adjusted/updated to achieve better result. 

 I uploaded all the codes at Simple-NeuralNetwork.

In the repo there are 4 files perceptron.h, linearsep.c, and_gate.c and or_gate.c. We will discuss bits from those files and see how they work.

In perceptron.h file, there are functions/methods to create, manipulate, train, display and delete the perceptron. General sequence is 

  • create a perceptron
  • prepare data set
  • train
  • test with predict function

This is simple, is not it? Now we will look into 2 functions, namely predict and train. These  functions are the core of this perceptron.

predict:

double predict(perceptron p, double inp[])
{
	double res=0.0;
	
	for(int r = 0; r < p.n_inp; r++)
	{
		res += inp[r]*get_cell(p.W,0,r);
	}
	res += p.bias;
	double ret= sigmoid(res);
	return(ret);
} 

This function multiplies inputs with corresponding weights and adds them. Next, to activate the output apply sigmoid function. A sigmoid function ranges from 0 to 1. Please see the first part here where the sum process and sigmoid function is defined.

Train:

  void train(perceptron *p,double *data,double desired)
  {
	double guess = predict(*p,data);
	
	double err = desired - guess;
	for(int n = 0; nn_inp; n++)
	{
		double cur_w = get_cell(p->W,0,n);
		set_cell(&p->W,0,n,cur_w+p->lr*err*data[n]);
	}
	p->bias += p->lr *err;
  } 
 

In this function, perceptron is trained by adjusting weights and bias.The steps can be as follows.

  • calculate output
  • find error by subtracting output from target
  • find delta weight by multiplying error, data(input) and learning rate
  • find delta bias by multiplying error and learning rate
  • update weights
  • update bias

In step 3 we are calculating the delta weights. This is the most important step in whole code. How and why delta value is multiplication of error with input is a topic itself.  As an Hint it is gradient, hence a derivative of error term with inputs and weights are to be considered.

Testing

To tryout this implementation, 3 code files were provided. which adheres to the general sequence. First one is linear_sep.c, In this demo program, random data sets are created and the perceptron is trained; then with a known data set it is compared. the output on my system is  as bellow.


************Perceptron:**************

number of inputs = 2
Weights:
[ 1.00000000 ]
[ 1.00000000 ]

Bias = 1.000000
*************************************
************Perceptron:**************

number of inputs = 2
Weights:
[ 61.70302487 ]
[ -31.13024636 ]

Bias = 0.622900
*************************************

is (2.000000 < 2 x 20.000000 + 1) predicted = 1.000000

For other 2 demo codes. Can this artificial neuron mimic reliably a digital gate? Well that was the motive when artificial neuron was proposed. It is observed that AND and OR gate are possible to simulate but not XOR gate. WHY???  from bellow diagrams it is obvious that OR function and AND function are linear separable but not the XOR function.


// OR gate implementation.


    // x1 x2 y
    //  0  0 0
    //  0  1 1
    //  1  0 1
    //  1  1 1
    //
    //  0,1-------1,1
    //   |          |
    //  \|          |
    //   |\         |
    //   | \        |
    //  0,0-\-----1,0
    //       \-separating line
    
    // AND gate implementation.
	// x1 x2 y
	//  0  0 0
	//  0  1 0
	//  1  0 0
	//  1  1 1
	//
	//  0,1---\----1,1
	//   |      \    |
	//   |        \  |
	//   |          \|
	//   |           |\-line separating  
	//  0,0-------1,0
// XOR function
// x1  x2  y
//  0   0  0
//  0   1  1
//  1   0  1
//  1   1  0
//        /----------|----- 2 lines separates
//  0,1--/-----1,1   |
//   |  /       |    |
//   | /        |    |
//   |/         |/---|
//   /          /
//  /|         /|
//  0,0-------/-1,0
    

Output of OR gate 

************Perceptron:**************

number of inputs = 2
Weights:
[ 0.00000000 ]
[ 0.00000000 ]

Bias = 1.000000
*************************************
************Perceptron:**************

number of inputs = 2
Weights:
[ 11.48282021 ]
[ 11.48242093 ]

Bias = -5.281243
*************************************
inputs 0, 0 predicted = 0.005060
inputs 1, 0 predicted = 0.997978
inputs 0, 1 predicted = 0.997977
inputs 1, 1 predicted = 1.000000

 

Output of AND gate

************Perceptron:**************

number of inputs = 2
Weights:
[ 0.00000000 ]
[ 0.00000000 ]

Bias = 1.000000
*************************************
************Perceptron:**************

number of inputs = 2
Weights:
[ 10.22930534 ]
[ 10.22993658 ]

Bias = -15.512550
*************************************
inputs 0, 0 predicted = 0.000000
inputs 1, 0 predicted = 0.005050
inputs 0, 1 predicted = 0.005053
inputs 1, 1 predicted = 0.992943

In the above result if we consider 0.99 as 1 and anything less that 0.005 as 0 then our results are at par with the truth table.

As we saw above XOR is not linearly separable, we can not simulate it with a single perceptron. we need more than one layer to simulate it we will do it in a future post.

Till then happy coding.

Saturday, 29 April 2023

Neural Network from Scratch using C

Hello There, 

Around a year ago we talked. Now several things have changed. I recently joined as a PhD scholar 😃. And decided to soil my hands in AI/ML. During my M.Tech days(2011-2013), we are taught Soft computing. Though I did my Thesis on Network On Chip (NOC) which is related to putting different IPs on to a silicon substrate. I devised a Genetic Algorithm based best mapping  as well as a deterministic heuristic based mapping using graphs. both the codes are available on my github page. But I have never paid much attention to ML till 2016, when I was developing a traffic detection  application for a client. thought it was never went beyond PoC stage.

Now I thought to put all my knowledge to earn a degree (Yes a Doctorate). To refresh my understandings of my master's degree time, I decided to go step by step implementing building blocks to learn this topic.

We should start by putting a single neuron first. The artificial neurons, the counter part of natural neurons share similar characteristic.

Perceptrons - the most basic form of a neural network · Applied Go 

 Dendrites are inputs and axon is output. But one thing we must put here which is Activation. It is the strength we provide as output.

in case of perceptrons input are applied through some weights. This means different input channel have different weights. the perceptron sums the weighted inputs and apply some sort of activation before output the result.

So mathematically we can write 

output = activation(sum ( inputs x weights)) 

output = activation(sum ( input array * weights array))

we can say we need 2 one dimensional matrices for inputs and weights.  Then upon the result of the multiplication we apply activation function.

Lets assume we have 3 inputs i1, i2, i3 with weights w1,w2 and w3. Now output will be \begin{equation} z=\sum_{1}^{n} i_mw_m \end{equation} next activation of z is output. 

There are several activation functions used to activate the output of perceptron. the most used are Binary activation, ReLU(0,x) and Sigmoid(0,1). 

We will use sigmoid for our application. \begin{equation}\sigma(z)=\frac{1}{1+e^{-z}}\end{equation}

 Next we will see the simple implementation of this perceptron.


Bye Bye for now.

Wednesday, 10 August 2022

GitHub Update

 After a Long time another post.

Added 2 small projects, one is a single header Matrix library and another is a CPU Simulator.

The matrix library (tmatlib) is capable enough to solve simultaneous linear equation. Though it was written several months back but I was Lazy.

The CPU simulation code was my tech-itch :p it is very minimal and for fun. It has small set of instructions, it has only 8 bit wide registers and a very small memory only 1kb which includes stack, data and code.

my github link here.

Saturday, 10 July 2021

Laravel Application deployment on Fedora 34 server

 Recently I have to migrate an old Laravel application of a client to a newer OS release Fedora 34. When it was developed Fedora-29 was Latest 😁.

 

The procedure is as bellow.

  • Install Fedora 34
  • Update the installation (sudo dnf update -y)  
  • Install Apache (sudo dnf install httpd)
  • Install Mariadb, Mariadb server (sudo dnf install mariadb mariadb-server)
  • Install PHP, PHP Mbstring, PHP zip, PHP Mysqlnd, PHP Mcrypt, PHP XML, PHP Json, PHP suMod ( sudo dnf install composer php php-mbstring php-zip php-mysqlnd php-mcrypt php-xml php-json mod_suphp -y)
    • check if PHP is properly working or not by using following script
phpinfo.php
<?php
phpinfo();
?> 
this script should be in /var/www/html/ directory which is by default  document root for httpd server.
Point the browser to this file and see if it shows a page about PHP version and other details. If not then check if apache server is running. 
  • As super user create a laravel.conf file under /etc/httpd/conf.d directory with following content
<VirtualHost *:80>
    ServerName your server name
    DocumentRoot /var/www/html/laravel_project/public
    <Directory /var/www/html/laravel_project>
        AllowOverride All
    </Directory>
</VirtualHost>
  •  Copy the laravel project to /var/www/html/ as super user and cgange the owner by chown -R apache:apache laravel_project 
  • Start Mariadb (sudo systemctl start mariadb)
This is almost complete. For your specific cases different services or tweaking may be required. 

Laravel is a good framework for web-app development but for sometimes it is not getting more acceptance. Now the buzz word is Node . In a future article I may write on settingup and using Express framework for Web-App.

Thank you.

Saturday, 26 December 2020

Deploying Django application on Fedora 33

Deploying a "Hello, World" Django app on Fedora 33
Requirement
1) Fedora server
2) NGINX web server
3) Django application framework
4) Gunicorn

A "Hello, World" application using Django

Django framework based upon Python. Current Python version is 3.7.
First check if python is installed. If not install it.

$ sudo dnf install python3


once Python is installed using pip3 django should be installed

$ pip3 install django~=3.1.2


Any version of django is good but it is personal preference.

next Gunicorn should be installed.

$ pip3 install gunicorn


We are almost ready for Django application development.

Soiling your hand
To start our development, we need to create a directory and put all our requirement there

$ mkdir myapp && cd myapp


I choose myapp but any thing is ok

$ django-admin startproject mapp


mapp is project name. Django will create some files for us and lessen our job. It will create
mapp directory and few files

.
└── mapp
    ├── manage.py
    └── mapp
        ├── asgi.py
        ├── __init__.py
        ├── settings.py
        ├── urls.py
        └── wsgi.py

Next we create a small application to run on top of it.

$ cd mapp
$ django-admin startapp hello


.
├── hello
│   ├── admin.py
│   ├── apps.py
│   ├── __init__.py
│   ├── migrations
│   │   └── __init__.py
│   ├── models.py
│   ├── tests.py
│   └── views.py
├── manage.py
└── mapp
    ├── asgi.py
    ├── __init__.py
    ├── settings.py
    ├── urls.py
    └── wsgi.py

now open the mapp/settings.py

$ vi mapp/settings.py


add following lines

INSTALLED_APPS = [
    ...
    ...
    'hello',
]



next open views.py
$ vi hello/views.py


...
...
from django.http import HttpResponse
def homePageView(request):
    return HttpResponse('Hello, World!')



create a file urls.py under hello and open it

$ vi hello/urls.py


add the following code to it.


from django.urls import path
from .views import homePageView
urlpatterns = [
    path('', homePageView, name='home')
]



Update mapp/urls.py as bellow


from django.contrib import admin
from django.urls import path, include
urlpatterns = [
    path('admin/', admin.site.urls),
    path('', include('hello.urls')),
]



Now test our app by running

$ python3 manage.py runserver


In browser, open http://127.0.0.1:8000
and enjoy your app

Ok you might have seen some warnings here. So please don't worry. Execute

$ python3 manage.py migrate


some messages will be displayed and now again execute

$ python3 manage.py runserver


This time there are no warnings, right? Ok good...

Deployment part

Open settings.py

$ vi mapp/settings.py


and change following

...
ALLOWED_HOSTS = [ '*' ]
...
...
STATIC_ROOT = '/var/www/static'
MEDIA_ROOT = '/var/www/media'


and make sure you created those directories

$ sudo mkdir /var/www/static
$ sudo mkdir /var/www/media


Next run

$ sudo python3 manage.py collectstatic


we are almost done.

Running guincorn

First copy mapp/wsgi.py as mapp/mapp.wsgi

$ cp mapp/wsgi.py mapp/mapp.wsgi
$ sudo gunicorn mapp.wsgi --workers=5 --bind=127.0.0.1:8888


now gunicorn is running. you can check with http://127.0.0.1:8888 and it will show you hello world page.

But we need nginx to serve for us on internet so we have to tell nginx to serve for us.

use following code as a .conf file in /etc/nginx/conf.d/ directory
we assume it as myapp.conf
server {
    # the port your site will be served on
    listen      80;
    # the domain name it will serve for
    server_name 192.168.1.3;   # substitute by your FQDN and machine's IP address
    charset     utf-8;

    #Max upload size
    client_max_body_size 75M;   # adjust to taste

    # Django static files. You should run 'django manage.py collectstatic'
    location /static  {
        alias /var/www/static;      # your Django project's media files
    }
    location /media {
    alias /var/www/media;
    }
    # Finally, send all non-media requests to the Django server.
    location / {
        proxy_pass http://127.0.0.1:8888;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}



restart nginx server by using

$ sudo systemctl restart nginx


now check from a remote machine using http://IP.address.of.server/

do you get it properly??? if not then check firewall settings on server. next check access on files etc...


goodluck.

Friday, 24 July 2020

Fedora 32 on Google Cloud



Google doesn't provide a Fedora image for its cloud platform neither Fedora provides one. but this wont stop us. It is a nice learning and adventure for orthodox people like me.

Those who have started there career in Computer in late 80's and early 90's can understand my pain 😬

Well during this time of pandemic COVID-19 I thought of taking an adventure ride into the world of Virtualization. Though Virtualization is not new to me, as during my days at RedHat I did some virtualization stuff by porting drivers, but that is a very short lived assignment for me. Even I was fascinated by Vmware's ESX server Community edition. Even to tryout my toy Operating System NanoOS i need some platform to try out so Was using qemu and BOCHS.
But setting up a VM in a cloud is just a new thing for me. If you like to learn along and how to do it then read along.

Prerequisites 

As I don't know much about Windowz, will restrict to Google's cloud console as much as possible and at some times Linux desktop.
  • Google account
  • Machine with Fedora Desktop(any Linux distro will do but I like Fedora)
  • Google SDK for Linux installed on your machine.
Deploying a VM needs availability of an Image. Though there is no image for Fedora 32 but similar images are available for CentOS and RHEL. We could choose any but I would prefer CentOS over RHEL.

Plan of Action 
  • We will create a VM using CentOS image. This will be a very minimum one as we will use it for creating Fedora 32 image.
  • Download Fedora cloud image from Fedora download site
  • Tweek Fedora image to make it suitable for Google.
  • Upload the image
  • Create an instance of it 
Creating a VM 
Here you need a google account your Gmail Account wont work here. You need to have cloud permission in your account. Please refer to google site for procedure. its not difficult but its not relevant here hence omitted.

Similarly install Google cloud SDK (gcloud and gsutil). Information about the procedure is available on Google SDK site.
  • Login to Google cloud, the bellow screen will be displayed
  • Click on menu (3 horizontal line)

  • from drop down select Compute Engine

  • Next it will display a page with all vm instances if available. for us it should be empty. Next on the top there is an icon with a plus


  • Now a form will be displayed, it will ask to enter a name for your VM instance . We will put boiler-plate there. Dont worry about other options except Boot Disk.

  • On Boot Disk option by default it will be debian. There is a button change next to it. Clicking it will display several options.


  • On clicking change something like this will appear. We will select CentOS. On next screen it will ask which version to use. we will use CentOS 8 as it has dnf command. But If you prefere yum can choose other older versions too.





  • Allow it for some time say 5 Minutes. then on VM Instance page our new VM will be displayed.


Ok we have successfully created a VM on Google Cloud now we can enjoy it for some time🙆 .Relax get a coffee and think what we have achieved.
After this point our journey will be some what tough as we will be doing some sysadmin job (yes SysAdmins are really great people They do a lot of hectic work to ease our life).

Click on ssh button at the end of the line of our newly created VM instance. It will open a newbrowser window with shell prompt of our VM. Check if all your commands are accepted or not.

Download Fedora cloud image

As we know there is no Google compatible image for Fedora available but Cloud images are available. We can download it and adjust  for Google compatibility. 
On the ssh terminal shell prompt check if wget is available or not. Else install it by using dnf or yum.

~]$ sudo dnf install wget

So go ahead and download  Fedora Cloud image. Choose raw image for convenience. Remember we are downloading into VM on google not onto our Desktop.

~]$ wget -c"https://download.fedoraproject.org/pub/fedora/linux/releases/32/Cloud/x86_64/images/Fedora-Cloud-Base-32-1.6.x86_64.raw.xz"

it will be downloaded to the home folder in VM. Give it some time as it is a long file , but don't worry it will be downloaded in few minutes say 2 to 3 minutes.

Preparing the image
Now the raw file is downloaded as *.tar.xz format. It has to be extracted  first 

~]$ tar -xvf Fedora-Cloud-Base-32-1.6.x86_64.raw.xz

It is better to copy this file to a smaller name like disk.raw and work on that.
  • mount the image

~]$ sudo losetup  /dev/loop0 disk.raw
~]$ kpartx -a /dev/loop0

mount the disk
$ sudo mkdir /mnt/disk
$ sudo mount /dev/mapper/loop0p1 /mnt/disk


  • Bind mount some filesytem

~]$ sudo mount --bind /dev /mnt/disk/dev
~]$ sudo mount --bind /sys /mnt/disk/sys
~]$ sudo mount --bind /proc /mnt/disk/proc

besides this another file has to be created that is /etc/resolv.conf content of that file is very small and as follows.

search domain.name
nameserver 8.8.8.8

Its now time to go into chroot
$ sudo chroot /mnt/disk
After chrooting check if ping works, that is if network works else adjust your network to work in chroot environment.

First thing to do is update the image. by
~]$ sudo dnf update

Next we have to install 
~]$ sudo dnf -y remove cloud-init
~]$ sudo dnf -y install google-compute-engine-tools
~]$ sudo dnf clean all

Now we are almost done. Google need some service to be enabled.
~]$ sudo systemctl enable google-accounts-daemon google-clock-skew-daemon \
    google-instance-setup google-network-daemon \
    google-shutdown-scripts google-startup-scripts

Well our Image is almost ready. Now we should leave chroot environment unbind and unmount the filesystems we have mounted earlier.

~]$ sudo umount /mnt/disk/dev /mnt/disk/sys /mnt/disk/proc
~]$ sudo umount /mnt/disk
~]$ sudo losetup -d /dev/loop0

In above steps, unmounting disk will throw "resource busy error" dont worry. Give it some time to finish writing stuffs, 5 to 10 minutes maximum. then unmount the disk and delete the loop0. We are done.
Google need its image to be in *.tar.gz format so we will compres the image in tar.gz format.

~]$ tar cvzf fedora-32-google-cloud.tar.gz disk.raw

Now we will upload the image just created "fedora-32-google-cloud.tar.gz"

We have created in side Fedora VM. We have to put it into Google cloud bucket. Using Web console
  • Goto Storage
  • Create a bucket
remeber its name. Now from within VM we can copy this image to bucket.
~]$ gsutil cp "fedora-32-google-cloud.tar.gz" "gs://[name of the bucket]/"
Replace with your bucket name in above comand. Well it will be uploaded to the bucket in a while. It can be seen listed under the bucket created.

From your desktop fedora fire this command.
~]$ gcloud compute images create --source-uri \
    gs://[your bucket]/fedora-32-google-cloud.tar.gz \
    --licenses "https://compute.googleapis.com/compute/v1/projects/vm-options/global/licenses/enable-vmx" \
    fedora-32-google-cloud-image
Wait for few seconds, an image will be created in images section of console. it can be seen under Images section of Web console. Note the --licenses calause. it is optional. you can omit it. but if you omit it then the created image wont be able to run virtual machines under it. Yes you read it correctly a Nested VM.

I think you enjoied your second coffee ... Jocking...

Now the image is ready and we can create an instance out of it as we created our boiler-plate only difference will be in case of Boot Image we will choose our image and we will choose a multi CPU configuration. After the instance is created check it using SSH and enjoy your Multi CPU Fedora Server on Google cloud.

Well This is enough for Now. Please contact me if you need any help.

73's
DE VU3VFU
Ashok.

Sources :
1) major io
2) linuxmint
3) media com

🦆Part 5 — Looking Forward What Phase 1 has 75 tests passing on both backends Structs , struct params, arrays of structs, local structs...