Architecture

The four stages

1 · Selection becomes compiler flags

callsight flags reads trace.config and the source list your build system passes it, and prints the flag string the compiler should use. Both build integrations call it on every build, so the selection is always current.

• File selection maps to -finstrument-functions-exclude-file-list. Sources that are not selected are listed as exclusions — GCC matches the list against the file each function is defined in, which is why header paths work for silencing inline helpers.
• Function selection maps to -finstrument-functions-exclude-function-list, a substring match on symbol names.
• include-func compiles down to those two primitives: the files defining the subtree are instrumented, and every other function defined in those files is added to the function exclude list.

The static call graph

include-func needs to know what a function calls. callsight uses a deliberately lightweight heuristic parser (comments and string literals blanked, definitions matched at line starts, call sites matched inside the body) rather than pulling in libclang — adoption stays a single uv tool install with no toolchain dependency.

The trade-off is documented rather than hidden: function pointers, macro-generated calls and C++ dynamic dispatch are not followed. The expansion is breadth-first, so a depth limit always measures the shortest path to each function.

2 · The hook runtime

trace.c is self-contained C with no dependencies beyond pthreads, compiled without -finstrument-functions and with no_instrument_function on every function, so the hooks can never trigger themselves.

• All state is per-thread (TLS): an 8192-event buffer, the output handle, the thread id. There are no locks on the hot path in file mode.
• Inert unless enabled. Without TRACE_ENABLE=1 each hook is one predictable branch and a return.
• No symbolization at runtime. Events carry raw addresses; names are resolved offline. That is what keeps a hook at tens of nanoseconds.
• No malloc, no I/O on the hot path. The buffer flushes when full, at thread exit (via a pthread key destructor), and at process exit.
• Reentrancy guard per thread, so a hook can never recurse through the code it calls.
• Process-wide counters are charged per flush — once per 8192 events, not once per event. A shared atomic touched by every hook would put every thread on one cache line exactly when tracing is heaviest.
• Raw descriptors, not stdio. The runtime already batches, so stdio would add a second buffer — one that fork() duplicates into the child.
• Fork-aware. A pthread_atfork child handler drops the inherited descriptors and buffers and starts a fresh capture under the new pid; otherwise parent and child interleave into the same file.

The clock

Timestamping is the dominant cost in a hook, so TRACE_CLOCK=auto reads the invariant cycle counter directly — rdtsc on x86-64, cntvct_el0 on aarch64 — which is what the vDSO itself reads, minus the scaling work. On this project's benchmark that is 12 ns per hook against 16 ns for clock_gettime.

x86-64 only takes that path when CPUID reports an invariant TSC; anything else ticks at a rate that changes with frequency scaling and would silently distort every duration. When it is unavailable the runtime falls back to CLOCK_MONOTONIC and says so.

Ticks are converted to nanoseconds offline. The runtime records an anchor pair (counter, CLOCK_MONOTONIC) at startup and writes a second one as the last record at exit, so the analyzer derives the tick rate across the whole run rather than from a startup calibration window. The header also carries a coarse rate as a fallback for a process that was killed before it could write the closing anchor.

The old default, CLOCK_MONOTONIC_RAW, is no longer used automatically: it is absent from the vDSO on older ARM kernels, where it costs a full syscall per event — precisely on the embedded targets that can least afford it. It remains available as TRACE_CLOCK=raw.

The event record

32 bytes, fixed layout, the same on disk and on the wire. Integers are in the agent's byte order — the device is the constrained side of the system and the analysis host is not, so the host detects and swaps (see agent portability).

Kind 3 carries a counted call's hardware counter deltas, one per configured event, in the three 64-bit slots — and always immediately after the exit it belongs to, so a reader attaches it to the call it just closed without keeping state. It is a separate record rather than four more fields on every event: the 32-byte grid is what the ring and the wire protocol are built on, and most calls are not counted, so widening every event would charge every user for a feature few enable.

Each trace file starts with an 80-byte version 2 header. Its first 16 bytes are laid out exactly as version 1 — magic, version, event size — so any reader can identify the file before it knows the rest, and a header_size field tells readers where the events begin instead of making them assume. That is what lets a later version add fields without breaking this one, and why version 1 traces still analyze today. The header also carries the PIE load bias, the clock calibration, and the runtime's own measured per-hook cost.

Markers

Kind 2 is a note from the runtime to the analyzer, carrying a reason code and a payload in the two address fields. It is how a capture that ended early says so — budget reached, disk full, write failed, segments rotated away — instead of simply looking short. Markers cost nothing to carry, travel over the streaming protocol like any other record, and a reader that does not recognize one skips it.

The same mechanism is why counted captures need no format-version bump: the events, their costs and any functions the guard rail declined to count are all announced as markers at the head of every segment. A rotated capture whose first segment was discarded still describes its own counter columns.

Treating an unknown kind as an exit would corrupt the entire match, so this is one of the few places where the format demands that readers be strict.

3 · The analyzer

Analysis is a single streaming pass. Events are read in blocks and matched as they arrive, so analyzer memory tracks the number of functions and threads, not the number of events — a multi-million-event trace costs a few MB.

1. Match per thread. Each thread has its own stack. An exit pops back to the nearest matching enter, closing any frames left dangling above it (which happens after a longjmp, or when a buffer tail is lost).
2. Attribute time. Inclusive time is exit minus enter; the duration is also credited to the parent frame's "children" total, and self time is inclusive minus children.
3. Resolve symbols. All distinct addresses go to addr2line in batches, once, at the end — static functions included.
4. Emit. Text tables, JSON, or folded stacks. Folded output is accumulated during the same pass, keyed by the tuple of addresses on the live stack.

Because everything is derived from the same pass, the flame graph and the self_ms column are guaranteed to agree.

Compiler mechanisms surveyed

What each compile-time instrumentation mechanism can do, what it costs, and how it maps to callsight. Overhead figures are order-of-magnitude, per event, assuming a lean hook.

-finstrument-functions — the current backend

Emits a call to __cyg_profile_func_enter(this_fn, call_site) and __cyg_profile_func_exit(...) at every function boundary, static functions included. Selection happens at compile time through the two exclude lists, both substring-matched, with the file list matched against the file a function is defined in. Excluded code emits no hook at all, so selection is free at runtime.

Caveats: inlined functions emit no hooks, because no call boundary exists; hooks fire even when you don't want data, so runtime gating costs a flag check per event; and -no-pie (or offset bookkeeping) is needed to map addresses back to symbols.

-finstrument-functions-after-inlining

Same hooks, inserted after inlining, so functions that survived inlining get hooked even when they were inlined at some call sites, and you see the real optimized call graph. A candidate extra flag on Clang toolchains; it needs an analyzer-side note that the call graph differs from the source-level one.

-pg / mcount (gprof)

The original: an entry-only hook into mcount, plus flat-profile sampling. Call-graph arcs only — no per-call exit timing — with PLT and shared-library blind spots and effectively unmaintained semantics under modern optimization. Documented for completeness; callsight does not use it.

-fpatchable-function-entry=N,M

Emits N NOPs at function entry (M of them before the prologue) plus a __patchable_function_entries section listing their addresses — the mechanism behind the Linux kernel's ftrace and uftrace's dynamic mode. A runtime can patch those sleds into hook calls and back, giving genuinely zero-cost-when-off, toggleable tracing. The strongest candidate for "trace a live production process for five seconds". Costs: code-size growth, patching machinery, and entry-only hooks — exit timing needs a return-address trampoline, which is the hard part.

-fsanitize-coverage

compiler-rt coverage callbacks (__sanitizer_cov_trace_pc_guard per edge, with a per-guard toggle word in the guard variant), aimed at fuzzers. Leaner than function hooks and gives basic-block-transition granularity, but there is no caller address and no exit event, so timing must be reconstructed, and guard tables need the compiler-rt runtime. Clang has the full menu; GCC ≥ 12 has trace-pc and trace-cmp only. Worth an experiment as a low-overhead "which edges ran" backend, not a replacement for call timing.

LLVM XRay

Clang-only, and the most production-grade design in this list: sleds at function entry/exit patched at runtime through a supported API (__xray_patch()), per-function selection at compile time, and a logging library writing binary flight-recorder traces with tooling (llvm-xray). The closest existing model for callsight's streaming design — its sled layout and FDR buffering are the reference. Not usable as a default, being Clang-only.

gcov / -fprofile-arcs

Edge counters for coverage, not timing. It answers "did this line run, how often", not "how long did it take". A possible complement if a coverage view is ever added; out of scope for tracing.

GCC plugin API

Loadable modules running custom GIMPLE/RTL passes: arbitrary injection, maximum control. But plugins are locked to the exact GCC version, the C++ API is fragile, Clang has no equivalent, and distribution is a nightmare. A deliberate non-goal.

Not compile-time (context only)

• perf sampling — no build integration at all; finds hot functions, not call sequences. Complementary, and always available.
• eBPF/uprobes, Intel PT — runtime tracing with no rebuild; heavy machinery and root requirements. Out of scope.
• uftrace — a userspace tracer built on -pg/-finstrument-functions plus its own libmcount runtime, with a dynamic mode on -fpatchable-function-entry. The closest relative project; callsight differs in zero-runtime-dependency adoption and the config-file selection workflow.

Repository layout