IGNIS

Appearance

HfLogger - High-Frequency Logging Guide ​

HfLogger is a fixed-layout ring-buffer logger for hot paths where even the standard Logger is too expensive. A log call writes a 256-byte binary entry into a lazily-allocated buffer. Nothing runs on the fast path: no string formatting, no transport I/O, no per-call allocation. A separate HfLogFlusher drains entries later, off the hot path.

It implements ILogger (AbstractLogger), so it's a drop-in replacement anywhere an ILogger is expected. Every level method works - debug, info, warn, error, emerg, plus log and for. But it stays entirely separate from the Winston-backed Logger pipeline: no formatters, no transports, no APP_ENV_LOGGER_* variables apply to it.

That separation buys enqueue speed - roughly 14x faster than pino on the same machine. See Performance characteristics below for the measured numbers.

IMPORTANT

Reach for HfLogger only when a profiler shows logging itself in your hot path - order engines, market-data ticks, per-packet paths at 100k+ events/sec. For everything else, the standard scoped Logger is the right tool: it formats, redacts, rotates files, and ships UDP. Most services never need this module.

Mental model ​

One ring buffer per process holds 65,536 entries of 256 bytes each - 16MB total. It allocates lazily, on the first HfLogger.get() call, never at module import.

Writers stamp entries in. The ring never blocks and never grows.

When the ring is full, the next write overwrites the oldest entry. The buffer trades completeness for a bounded, allocation-free hot path. But every overwrite is counted and reported, never silent - see Lap accounting below.

Entry layout (256 bytes):

OffsetSizeField
0-78 bytesTimestamp (float64 epoch milliseconds, sub-millisecond precision)
81 byteLevel (0=debug, 1=info, 2=warn, 3=error, 4=emerg)
91 byteScope length (0-32)
10-4132 bytesScope bytes
421 byteMessage length (0-213)
43-255213 bytesMessage bytes

The two length bytes are what make reads exact. The flusher decodes only the bytes a field actually holds, never the fixed-width remainder. So there's no NUL padding, no stale tail leaking from whatever a reused slot last held.

Because entries are fixed-width binary, everything you log must fit the layout above. Anything longer than the scope or message cap is truncated, not rejected.

Setup - everything happens at initialization time ​

The hot path takes pre-encoded bytes, not strings. Encode once, at startup, and keep the references:

typescript
import { HfLogger, HfLogFlusher } from '@venizia/ignis-helpers';

// -- Initialization phase (once, before any hot path runs) --

// 1. Get the logger for a scope (cached; scope bytes are pre-computed)
const orderLogger = HfLogger.get('OrderEngine');

// 2. Pre-encode EVERY message the hot path will ever emit
const MSG_ORDER_SENT = HfLogger.encodeMessage('Order sent');
const MSG_ORDER_FILLED = HfLogger.encodeMessage('Order filled');
const MSG_ORDER_REJECTED = HfLogger.encodeMessage('Order rejected');

// 3. Start the background flusher
const flusher = new HfLogFlusher();
flusher.start(100); // drain every 100ms
typescript
// -- Hot path (bytes path, ~59ns/call) --
orderLogger.log('info', MSG_ORDER_SENT);
orderLogger.log('info', MSG_ORDER_FILLED);
orderLogger.log('error', MSG_ORDER_REJECTED);
typescript
// -- Or drain manually (for example, at a batch boundary or before shutdown) --
await flusher.flush();

// -- And stop the interval when the logger is no longer needed --
flusher.stop();

The ILogger surface - and its cost model ​

HfLogger implements the same ILogger methods as the standard Logger, so it can be typed and passed around as ILogger. But each method sits on a different point of the cost curve. Picking the right one is the whole point of this module:

typescript
import { HfLogger } from '@venizia/ignis-helpers';
import type { ILogger } from '@venizia/ignis-helpers';

const logger: ILogger = HfLogger.get('OrderEngine');

// Fast: no-args string call resolves through the same bounded encode cache as
// encodeMessage() - a Map.get() plus the bytes-path write. ~66ns/call on a cache hit.
logger.info('Order sent');

// Slow path: any args force formatLogMessage() (deep inspection + secret redaction)
// and an UNCACHED encode, because dynamic strings must never grow the cache.
// Correct, but this is not the hot path - use it for control-flow events, not per-tick data.
logger.info('Order sent: %s', orderId);

// debug() returns before any encoding when SHOULD_LOG_DEBUG is false (same gate as Logger).
logger.debug('Verbose diagnostic');

// .for() composes a sub-scope the same way BaseLogger does.
const fillLogger = logger.for('fill');
fillLogger.info('Order filled');

// log() also accepts pre-encoded bytes directly - the true hot path, unchanged from before.
const MSG_ORDER_SENT = HfLogger.encodeMessage('Order sent');
logger.log('info', MSG_ORDER_SENT);

Rule of thumb:

WhenUseCost
A per-event, per-tick, per-packet pathPre-encode, then call the bytes overload~59ns/op
A small fixed set of facts you didn't pre-encodeThe no-args string call~66ns/op
Off the true hot path onlyThe args form (dynamic %s data)Correct - nothing is ever silently dropped - but not free

The rules that keep it fast and correct ​

1. Never encode in the hot path. HfLogger.encodeMessage remembers every distinct string it sees. So does the no-args string call, since it shares the same cache. That cache is FIFO-bounded at 4096 entries.

Calling it with dynamic strings (encodeMessage('order ' + id)) still puts UTF-8 encoding on your hot path. Worse, it evicts the oldest cached message once you cross the cap - corrupting the fixed vocabulary you rely on elsewhere.

If a value varies per event, it doesn't belong in an HfLogger message. Log the static fact here, and the variable detail through the standard Logger at a lower frequency. Or use the args form, off the hot path.

2. A fixed vocabulary of messages. The bytes path targets a finite set of pre-encoded facts: "Order sent", "Tick received", "Risk check failed".

If you find yourself needing free-form text on the hot path, you're in the wrong module. Use the args form instead, off the hot path.

3. Size the flush interval against your write rate. The ring holds 65,536 entries. Write more than that between two flushes, and the oldest unflushed entries get overwritten. The flusher reports exactly how many via dropped on the sink batch - never silently (see below).

Pick the interval so writeRate x interval < 65,536, with comfortable margin. At 100k logs/sec, a 100ms interval accumulates ~10k entries per drain - safe. At 1M logs/sec, you need ~30ms or faster.

4. One process, one thread. HfLogger is safe only on a single thread within a single process. The write index is a plain counter - not shared, not atomic.

Don't log to it from worker threads. Each worker that imports the module gets its own independent ring, and nothing coordinates them. This is a documented design point, not an accident.

5. Flush before shutdown. Entries live only in memory. An exiting process loses everything not yet flushed. Call await flusher.flush() in your shutdown path, and flusher.stop() to clear the interval.

The flusher ​

typescript
import { HfLogFlusher } from '@venizia/ignis-helpers';
import type { IHfLogFlusherOptions } from '@venizia/ignis-helpers';

// Default: renders to stdout, one write() per batch of up to 1024 entries.
const flusher = new HfLogFlusher();

// Append to a file instead of stdout.
const fileFlusher = new HfLogFlusher({ filePath: './app_data/hf.log' });

// Full custom delivery - receives the rendered lines AND the drop count for this batch.
const customFlusher = new HfLogFlusher({
  sink: batch => {
    if (batch.dropped > 0) {
      console.warn(`HfLogFlusher lapped: ${batch.dropped} entries overwritten`);
    }
    shipToAggregator(batch.lines);
  },
  batchSize: 512,
});

flusher.start(100); // interval-based draining, unref'd so it never blocks process exit
await flusher.flush(); // one-shot drain, for example before shutdown
flusher.stop(); // clears the interval; start() again to resume

A rendered line looks like this:

2026-07-18T09:41:03.128Z [info] OrderEngine Order sent

<ISO timestamp> [<level name>] <scope> <message> - readable, parseable, and free of NULs or stale bytes.

Lap accounting ​

Every batch the flusher hands to its sink carries dropped: number. That's the count of entries the ring overwrote before the flusher could read them, since the previous batch. The default sink emits a warn marker line ahead of the batch when dropped > 0:

2026-07-18T09:41:03.200Z [warn] HfLogFlusher ring lapped - 342 entries overwritten before they could be read

A custom sink gets the same dropped count on batch.dropped and decides how to surface it. This replaces the old silent behavior, where a lapped ring emitted whatever currently sat in each slot with no warning.

Current limitations ​

These are real behaviors of the current implementation - design around them:

  • Single-thread only. The write index is not shared or atomic across threads. Each worker thread that imports the module gets an independent ring. Don't log to HfLogger from worker threads expecting a shared buffer.
  • The ring overwrites the oldest entry when lapped. If the producer writes faster than the flusher drains (see rule 3 above), unflushed entries get silently overwritten in memory.
  • The loss is visible, not invisible. The flusher counts and reports every overwritten entry via dropped.
  • 213-byte message cap, 32-byte scope cap. Both are truncation-only - a longer value is cut, not rejected.
  • Truncation happens at a byte boundary, not a character boundary. It can split a multibyte UTF-8 character - the truncated tail then renders as the U+FFFD replacement character.
  • Run one flusher per process. Each HfLogFlusher tracks its own read position from the start of the ring - not a shared cursor. A second flusher re-emits entries the first one already drained.
  • A fixed, pre-encoded vocabulary is still the right pattern for the bytes path. HfLogger.encodeMessage and the no-args string call exist to make the ENCODE cost a one-time expense.
  • The args form is correct, but it's the slow path by design. Reserve it for control-flow events, off the hot path.
  • The encode cache is FIFO-bounded at 4096 entries. That's well within a real fixed vocabulary. But a hot path that generates many distinct dynamic strings, through the no-args string call, will start evicting and re-encoding.

Performance characteristics ​

Measured on Bun 1.3.14, 1M-iteration median:

PathCostNotes
Bytes (log(level, preEncodedBytes))59.4ns/opThe true hot path - no formatting, no allocation
String, no args (info('Order sent'))66.0ns/opCache-hit encode lookup + bytes-path write
pino, sync, /dev/null831ns/op~14x slower than the HfLogger bytes path on the same machine

Heap growth measured at 0.0MB over 1M bytes-path logs - the hot path does not allocate. That buys an ENQUEUE, not a durable log line. The flusher still pays rendering cost later, off the hot path.

See also ​

Files: