CORE PHILOSOPHY

STREAMING AS SYNTAX. NOT LIBRARIES.

Building stream processors with generic libraries forces developers into repetitive consumer loops, manual sliding ring buffers, and defensive error handling across hundreds of lines of boilerplate.

Deyin elevates streaming primitives into first-class language constructs using a single compositional operator: the Unix pipe (|).

THE 5-LINE GUARANTEE

stream prices from kafka "binance.BTC"
  | window 5m
  | filter rsi(14) < 30
  | filter price > 20
  | map buy 10%
  | sink to postgres "signals"

CORE CAPABILITIES

PRIMITIVE	GRAMMAR ROLE	RUNTIME BEHAVIOR
`stream <name> from`	Ingestion Origin	Zero-copy socket, Kafka topic, or CSV replay reader
`window <duration>`	Temporal Window	O(1) sliding deque buffer for rolling math
`filter <predicate>`	Predicate Filter	Fast boolean evaluation dropping non-matching events
`map <action>`	Enrichment / Signal	Field derivation or transaction signal emission
`sink to <dest>`	Output Dispatch	Buffered write to STDOUT, PostgreSQL, or local file
`compile to <target>`	Codegen Directive	Ahead-Of-Time emission to standalone Python or WebAssembly

INSTALLATION & USAGE

QUICKSTART: UP IN 60 SECONDS

1. SETUP REPOSITORY

git clone https://github.com/ayodejiades/deyin.git
cd deyin
pip install -r requirements.txt
python bench/generate.py          # Generates seeded 200k & 1M benchmark datasets

2. EXECUTE PIPELINES

# Run pipeline through the bytecode tree-walk interpreter
python cli.py run examples/kafka.deyin --limit 100

# Launch the interactive REPL
python cli.py repl

3. COMPILE TO STANDALONE TARGETS

# Transpile to zero-eval, standalone Python
python cli.py compile examples/compile.deyin --target python -o dist/pipeline.py
python dist/pipeline.py 200000

# Emit WebAssembly Text (WAT) kernel
python cli.py compile examples/compile.deyin --target wasm

FORMAL SPECIFICATION

LANGUAGE REFERENCE & EBNF

GRAMMAR DEFINITION

pipeline   ::= "stream" IDENTIFIER "from" source ("|" step)*
source     ::= ("kafka" | "file" | "socket" | "memory") (STRING | IDENTIFIER)
step       ::= window | filter | map | sink | compile

window     ::= "window" DURATION
filter     ::= "filter" expression
map        ::= "map" (action | assignment | expression)
sink       ::= "sink" "to" IDENTIFIER [STRING]
compile    ::= "compile" "to" ("python" | "wasm")

DURATION   ::= [0-9]+ ("s" | "m" | "h" | "d")

BUILT-IN INDICATORS

FUNCTION	INPUT ARGS	DESCRIPTION
`rsi(period)`	`period` (default: 14)	Relative Strength Index calculated over the active sliding window buffer
`sma(period)`	`period` (e.g. 50, 200)	Simple Moving Average of the price field across the active window
`ema(period)`	`period`	Exponential Moving Average weighting recent price ticks

SYSTEMS INTERNALS

DUAL-ENGINE ARCHITECTURE

1. PARSER & AST PIPELINE

Deyin utilizes a Lark LALR parser (src/parser.py) configured with position propagation. Syntax errors surface exact line and column coordinates before execution begins.

2. TREE-WALK VISITOR INTERPRETER

The interactive runtime (src/interpreter.py) executes via an AST Visitor pattern without eval(). Rolling mathematical indicators (Wilder's RSI, stateful recursive EMA, and SMA) evaluate over sliding deque window buffers in causal arrival order without look-ahead bias.

3. OPTIMIZING COMPILER

The Ahead-Of-Time compiler (src/compiler.py) performs Dead Code Elimination (DCE), lazy itertools.islice ingestion, conditional variable hoisting, and emits zero-eval Python or valid WebAssembly Text (WAT) kernels.

PERFORMANCE METRICS

BENCHMARK METHODOLOGY

Benchmarks are conducted on seeded 200,000-row time-series datasets measuring true input throughput across the tree-walk interpreter and compiled targets (median of 3 runs).

python cli.py bench --limit 200000

TARGET ENGINE	THROUGHPUT (200K ROWS, INPUT)	SPEEDUP FACTOR
Interpreter (AST Tree-Walk)	19,659 rows/sec	1.0x (Baseline)
Compiled Python (Zero-Eval, Lazy)	76,242 rows/sec	3.9x Speedup

HONEST BOUNDARIES

KNOWN LIMITATIONS

SUBSYSTEM	CURRENT BEHAVIOR	DESIGN INTENT
PostgreSQL Sink	Simulated / Mocked output to stdout	Enables complete offline demo execution without running database service
Kafka Source	Deterministic replay of seeded CSV dataset	Eliminates external broker dependency during evaluations and benchmarks
WASM Codegen	Supports numeric arithmetic and comparison predicates	Stateful multi-row window functions (SMA/EMA) currently require Python runtime
Indicator Warm-Up	Initial ticks dropped until window buffer reaches indicator period	Prevents uninitialized rolling math from polluting output signal streams