Pre-Computed Markdown Chunk Models

Pre-Computed Markdown Chunk Models

To avoid slow parsing scripts during user prompt searches, Asterisk 2 splits document text into highly focused storage objects called Page Chunks. The application controller processes modified markdown strings to keep search databases clean and optimized.

Chunk Generation Lifecycle

The diagram below details the automatic parsing loop maintaining clean RAG snippet databases:

graph TD
    %% Trigger Action
    subgraph ActionTrigger["Document Update Operation"]
        PostUpdate["POST /api/Documentation/page<br/>(Markdown Saved)"]
    end

    %% Chunking logic
    subgraph LogicTier["Automatic Regex Processing"]
        HeaderSplit["Split Document by Regex<br/>(Matches Line Starters)"]
        CleanPass["Scrub Extraneous Carriage Returns"]
    end

    %% Relational Ledger
    subgraph StorageTier["Chunk Ledger Database"]
        PurgeOld["DELETE from PageChunks<br/>where PageId == Target.Id"]
        InsertNew[("INSERT PageChunk Rows<br/>(ChunkIndex, Content Text)")]
    end

    %% Downstream Delivery
    subgraph ConsumerTarget["AI Context Pipeline"]
        DeliverRAG["Return Focused Text Snippet<br/>to User RAG Search"]
    end

    %% Flow Vectors
    PostUpdate --> HeaderSplit
    HeaderSplit --> CleanPass
    CleanPass ==>|"Execute Transaction"| PurgeOld
    PurgeOld ==>|"Atomic Commit"| InsertNew
    InsertNew ==>|"Index Pre-Computed"| DeliverRAG

    %% Custom Curated Colors
    classDef triggerToken fill:#0f172a,stroke:#38bdf8,stroke-width:2px,color:#fff,rx:6px,ry:6px;
    classDef logicToken fill:#1e293b,stroke:#a855f7,stroke-width:2px,color:#fff,rx:6px,ry:6px;
    classDef storeToken fill:#312e81,stroke:#ec4899,stroke-width:2px,color:#fff,rx:6px,ry:6px;
    classDef targetToken fill:#065f46,stroke:#10b981,stroke-width:2px,color:#fff,rx:6px,ry:6px;

    class ActionTrigger,PostUpdate triggerToken;
    class LogicTier,HeaderSplit,CleanPass logicToken;
    class StorageTier,PurgeOld,InsertNew storeToken;
    class ConsumerTarget,DeliverRAG targetToken;

Model Attribute Blueprint

1. PageChunk Entity Fields

  • ChunkId (int): Primary automatic auto-increment identity integer.
  • PageId (int): Mandatory foreign key binding the isolated snippet directly to its original parent document.
  • ChunkIndex (int): Sequential numerical integer preserving the real reading layout sequence of the sub-headings.
  • Content (string): Indexed text fields retaining unescaped markdown sub-block instructions.

Referential Safeties

If a parent Page record triggers deletion inside ApplicationDbContext, relational cascading triggers immediately drop all mapped PageChunk entries, completely preventing obsolete RAG tokens from skewing prompt generation logic.