PHP FFI was the wrong tool: a child-process bridge for Telegram VoIP

WardenPoint places real Telegram VoIP calls when an alert matters enough to ring through a muted phone. Getting there took us through a PHP FFI integration that corrupted heaps in production, a child-process redesign, two pipe-level deadlocks and a 6× latency win. This is the honest engineering log.

The stack, in one paragraph

A Telegram voice call has two halves. Signaling is MTProto: phone.requestCall, the DH key exchange, phone.sendCallSignalingData ferrying ICE candidates, phone.discardCall at the end. We run that through MadelineProto, the mature PHP MTProto client. Media is plain WebRTC — ICE, DTLS, SRTP, Opus — and for that the only serious library outside Telegram's own clients is ntgcalls, a C++ stack with a clean C ABI and prebuilt shared libraries.

PHP talking to a C library — that's what FFI is for, right?

Attempt 1: PHP FFI, and why it was doomed

The FFI binding worked in smoke tests. ntg_init(), ntg_create_p2p(), callbacks registered, test call connected. We shipped it behind a feature flag and watched the first production canary calls.

The Horizon worker died mid-call. Twice. Two flavours of the same crash:

PHP Fatal error: Maximum call stack size of 8339456 bytes reached
during compilation. Try splitting expression in .../SerializationException.php

zend_mm_heap corrupted

The root cause is architectural, not a bug to patch. ntgcalls starts a GLib main loop on a background thread and fires every callback — incoming signaling bytes, connection-state changes, stream end — from that thread. PHP NTS (the build everyone runs under FPM and queue workers) is not thread-safe. The moment a foreign thread invokes a PHP closure, it touches zval refcounts and the Zend allocator from outside the engine's world model. Sometimes it works. Under load, the heap shreds.

We looked for an escape hatch: a ntg_main_iteration() or pump function that would let PHP drain the loop from its own thread. The C header — all 381 lines of it, both v2.2.2 and master — exports none. The loop is pinned to a background thread by design, and that design is fine for Python (pytgcalls runs it happily) and fatal for PHP NTS.

Lesson one: FFI is a calling convention, not an isolation boundary. If the native library owns threads, your process now has threads, whether your runtime tolerates them or not.

Attempt 2: a child process and a line protocol

The fix was to stop sharing an address space. We wrote wp-ntgcalls-bridge, a small C++ binary (a few hundred lines plus the protocol doc) that owns the ntgcalls instance and its GLib thread completely. PHP talks to it over the oldest IPC there is — stdin/stdout, one JSON object per line:

→ {"cmd":"skip_exchange","cmd_id":3,"args":{"user_id":1234,"auth_key":"…"}}
← {"event":"async_result","cmd_id":3,"error_code":0}
← {"event":"signaling_outbound","bytes":"<base64 ICE blob>"}
← {"event":"connection_state","state":"Connected"}

One bridge process per call, spawned with proc_open, killed in a finally. Every ntgcalls callback writes a line to stdout from whatever thread it likes — a mutex-guarded writer keeps lines atomic — and PHP reads them with non-blocking stream_select from its single comfortable thread.

The isolation pays for itself in failure modes. When the bridge segfaults (and early on, it did), PHP sees EOF on the pipe, marks the notification failed, and the worker — with its MadelineProto session, Redis locks and queue state — survives untouched.

War story 1: the 64 KB pipe deadlock

First canary after a refactor: call accepted, then everything froze. Worker hung, bridge alive, nothing moving.

The post-mortem reads like a textbook deadlock. During ICE negotiation, ntgcalls bursts 30–50 outbound signaling events, ~5 KB each. Our stdout writer held a mutex across fwrite + fflush. A Linux pipe buffers 64 KB by default — the 13th event filled it. fflush blocked while holding the mutex. The bridge's main thread then needed that same mutex to acknowledge the next command, so it stopped reading stdin. PHP kept writing commands until stdin's buffer filled too, and its fwrite blocked. Two processes, four pipe ends, zero progress.

The fix is one syscall:

fcntl(STDOUT_FILENO, F_SETPIPE_SZ, 1 << 20);  // 64 KB -> 1 MB

A megabyte fits ~200 burst events — beyond any realistic ICE storm. We also left latency instrumentation in the writer, gated behind a per-tenant verbose flag, so the next backpressure problem shows up in a log line instead of a frozen worker.

Lesson two: kernel pipe buffers are part of your API contract. If your protocol can burst, size the buffer for the burst or build real backpressure.

War story 2: 100 ms × 50 blobs

Deadlock gone, calls stable — but the recipient waited ~6 seconds between tapping Accept and hearing audio. On a VoIP call that feels broken.

Our first guess (block audio until the transport reports Connected, pad with silence) made things worse — we shipped it, measured 9 seconds, and reverted within the hour. So we stopped guessing and instrumented every hop: per-iteration timings in the connect loop, per-blob timings in the signaling forwarder.

The numbers pointed away from the network entirely. Every outbound ICE candidate the bridge produced was being forwarded to Telegram synchronously: bridge → PHP → MadelineProto IPC → phone.sendSignalingData MTProto round-trip. Each hop: ~100 ms. Fifty candidates, sequential: five seconds of self-inflicted latency inside our own poll loop, while the actual DTLS handshake would have been happy to finish in one.

ICE candidates don't need ordering — the peer reassembles them however they arrive. So the forwarder became fire-and-forget on the Revolt event loop MadelineProto already runs:

public function sendOutboundSignaling(string $data): void
{
    async(function () use ($data): void {
        $this->API->methodCallAsyncRead('phone.sendSignalingData', [
            'peer' => $this->inputCallPeer,
            'data' => $data,
        ]);
    });
}

Dispatch dropped from ~100 ms to single-digit milliseconds per blob. The before/after, from production timing breakdowns:

Metric	Before	After
Per-blob forward	~100 ms	1–3 ms
Event-drain pass	4–5 s	30–60 ms
Wait for Connected	5–7 s	0.7–1.9 s
Accept → audio	~6 s	~1 s

Lesson three: measure before you optimize. Our intuition blamed DTLS over cellular. The stopwatch blamed our own synchronous loop.

What we'd tell past us

1. Process isolation beats FFI for thread-owning native libraries. The boundary you draw should match the boundary that actually exists — and a library with its own event loop is its own program.
2. A JSON-line protocol over pipes is enough. No gRPC, no shared memory, no sockets to leak. proc_open, two pipes, newline framing. Debuggable with cat.
3. Leave the instrumentation in. Every number in this post comes from log lines that still run in production, gated behind a debug flag. The next regression will announce itself.
4. Fire-and-forget is correct when the protocol says so. ICE tolerates reordering; waiting for per-message ACKs was pure superstition.

The result: WardenPoint rings a real Telegram VoIP call about one second after the recipient accepts, the audio starts from the first word, and a crashed media stack costs us one failed notification instead of one corrupted worker. That trade — one process spawn per call — is the cheapest insurance we buy.