read this skill for a token-efficient summary of the util subsystem
The src/util/ directory provides foundational infrastructure used across all of stellar-core. It covers timing, scheduling, logging, numeric safety, filesystem operations, serialization, data structures, threading, and various small helpers.
Timer.h, Timer.cpp)VirtualClockCentral timing facility with two modes: REAL_TIME (wall clock) and VIRTUAL_TIME (simulated, for tests). Owns an asio::io_context for IO dispatch and a Scheduler for main-thread action scheduling.
time_point (steady clock), system_time_point (wall/calendar time), duration.crank(bool block): The main event-loop step. Dispatches pending timers, polls IO (with exponential priority biasing under overload), runs scheduled actions, and transfers items from the thread-safe pending queue to the scheduler. In mode, advances time to the next event if idle.VIRTUAL_TIMEpostAction(): Thread-safe submission point for deferred work callbacks. Uses a mutex-protected pending queue; wakes the main thread via asio::post when enqueueing into an empty queue.setCurrentVirtualTime(): Advance virtual time forward (monotonic; asserts forward progress).sleep_for(): Real sleep in REAL_TIME, virtual time advance in VIRTUAL_TIME.shouldYield(): In REAL_TIME, returns true after 500ms has elapsed (time-slice limit). Always returns true in VIRTUAL_TIME.to_time_t, from_time_t, systemPointToTm, tmToSystemPoint, isoStringToTm, tmToISOString, systemPointToISOString.VirtualTimerTimer coupled to a VirtualClock that uses virtual time. Supports expires_at(), expires_from_now(), async_wait(), cancel(). Typical usage for all delayed / periodic operations in core.
VirtualClockEventInternal event representation stored in the clock's priority queue. Carries a callback, trigger time, sequence number, and cancelled state.
Scheduler.h, Scheduler.cpp)A multi-queue fair scheduler implementing a variant of the LAS (Least Attained Service) / FB (Foreground-Background) algorithm.
ActionQueues each track cumulative runtime (mTotalService).runOne(), the queue with the lowest accumulated service time runs its next action.ActionQueue (private inner class)Element{Action, enqueueTime}.mTotalService, mLastService, idle list membership.tryTrim(): drops droppable actions that exceed the latency window.runNext(): runs the front action and updates total service (floored at minTotalService).Scheduler public interfaceenqueue(name, action, type): Creates or reactivates a queue and appends the action.runOne(): Dequeues from the lowest-service queue, trims stale idle queues.getOverloadedDuration(): Reports how long any queue has been overloaded.currentActionType(): Returns the ActionType of the currently executing action.shutdown(): Clears all queues.Logging.h, Logging.cpp, SpdlogTweaks.h, LogPartitions.def)Logging classStatic utility managing spdlog-based logging with partitioned loggers.
LogPartitions.def via X-macro (LOG_PARTITION(name)), yielding ~15 named partitions (e.g., Bucket, Herder, Ledger, Overlay, Tx, etc.). Each partition has its own spdlog logger with independent log level.init(): Creates sinks (stderr + optional file), registers all partition loggers, configures flush behavior (every 1s, immediate on error). Also initializes Rust-side logging.setLogLevel(level, partition): Per-partition or global level setting.setLoggingToFile(filename): Supports {datetime}-formatted filenames. Opens in append mode (O_APPEND) for durability with external log rotation.setFmt(peerID): Sets the log pattern including a peer identifier.rotate(): Log rotation support.CLOG_TRACE/DEBUG/INFO/WARNING/ERROR/FATAL(partition, fmt, ...): Partition-aware logging with compile-time format string checking via FMT_STRING.LOG_TRACE/DEBUG/INFO/WARNING/ERROR/FATAL(logger, fmt, ...): Logger-handle-based logging.LOG_CHECK: Guards log calls with should_log() check to avoid formatting overhead.CoutLoggerFallback logger when spdlog is not available. Writes to std::cout.
LogLevel enumLVL_FATAL(0) through LVL_TRACE(5).
format_as overloadsProvides fmt-compatible formatting for XDR enum types and std::filesystem::path.
numeric.h, numeric.cpp, numeric128.h)Safe arithmetic for financial calculations where overflow matters.
numeric.h)bigDivide(result, A, B, C, rounding): Computes A*B/C safely when A*B would overflow int64_t. Returns success/failure bool.bigDivideOrThrow(A, B, C, rounding): Throwing variant.bigDivideUnsigned(): Unsigned variant.bigSquareRoot(a, b): Integer square root of a*b, ROUND_DOWN only.saturatingMultiply(a, b): Caps at INT64_MAX on overflow (non-negative inputs).saturatingAdd(a, b): Unsigned, caps at type max.isRepresentableAsInt64(double): Checks safe double→int64 conversion.doubleToClampedUint32(double): Clamps double to uint32 range.Rounding enum: ROUND_DOWN, ROUND_UP.numeric128.h)bigMultiply(a, b) / bigMultiplyUnsigned(a, b): Returns uint128_t.bigDivide128() / bigDivideUnsigned128(): Divide 128-bit by 64-bit.hugeDivide(result, a, B, C, rounding): Computes a*B/C when C < INT32_MAX * INT64_MAX.Fs.h, Fs.cpp, FileSystemException.h, TmpDir.h)fs namespacelockFile() / unlockFile() using flock (POSIX) or CreateFile (Win32).flushFileChanges() (fsync), durableRename() (rename + dir fsync), openFileToWrite().exists(), deltree(), mkdir(), mkpath() (recursive), findfiles().hexStr(), hexDir(), baseName(), remoteName(), remoteDir() — used for history archive paths.getMaxHandles(), getOpenHandleCount() — Linux reads /proc/self/fd.bufsz(): Returns 256KB (AWS EBS IOP alignment).FileSystemExceptionExtends std::runtime_error with failWith() and failWithErrno() static helpers that log before throwing.
TmpDir / TmpDirManagerRAII temporary directory management. TmpDirManager creates and cleans a root directory; TmpDir creates prefixed subdirectories that are deleted on destruction.
XDRStream.h)XDRInputFileStreamReads XDR-framed objects from a file one at a time.
readOne(T& out, SHA256* hasher): Reads 4-byte big-endian size header, then that many bytes, unmarshals into out. Optional hash accumulation.readPage(T& out, key, pageSize): Reads a page-sized chunk and scans for a specific LedgerKey. Used for indexed bucket lookups.seek(), pos(), size limits.OutputFileStreamLow-level durable write stream using ASIO buffered writes (POSIX) or FILE* (Win32).
writeBytes(), flush(), close() with optional fsync on close.fs::stream_t / fs::native_handle_t.XDROutputFileStreamExtends OutputFileStream with XDR framing.
writeOne(T, hasher, bytesPut): Encodes 4-byte size header + XDR payload. Optional hash and byte counting.durableWriteOne(): writeOne + flush + fsync for crash safety.RandomEvictionCache<K, V> (RandomEvictionCache.h)Fixed-size cache with random-2-choices eviction (not LRU). More robust under pathological access patterns.
unordered_map with stable pointers in a vector for random access.put(), get(), maybeGet(), exists(), erase_if(), clear().BitSet (BitSet.h)Value-semantic C++ wrapper around a C bitset (cbitset.h). Uses inline storage for small sets (≤64 bits) to avoid heap allocation.
|), intersection (&), difference (-), symmetric difference.unionCount(), intersectionCount(), etc.nextSet(i) for scanning set bits.BitSet::HashFunction.UnorderedMap<K, V> / UnorderedSet<K> (UnorderedMap.h, UnorderedSet.h)Type aliases for std::unordered_map / std::unordered_set using RandHasher to prevent hash-flooding attacks.
BinaryFuseFilter<T> (BinaryFuseFilter.h, BinaryFuseFilter.cpp)Probabilistic membership-test filter (like Bloom filter but more space-efficient). Wraps binary_fuse_t library. Supports 8/16/32-bit widths with corresponding false-positive rates (1/256, 1/65536, 1/4B). Serializable via cereal to SerializedBinaryFuseFilter XDR type. Used for efficient key-membership checks in bucket indexes.
TarjanSCCCalculator (TarjanSCCCalculator.h)Tarjan's algorithm for computing strongly connected components of a directed graph. Uses BitSet for SCC representation. Used in quorum analysis (SCP).
ThreadAnnotations.hComprehensive Clang thread-safety annotation macros (GUARDED_BY, REQUIRES, ACQUIRE, RELEASE, TRY_ACQUIRE, etc.). Provides annotated mutex wrappers:
Mutex: Wraps std::mutex with annotations.SharedMutex: Wraps std::shared_mutex with Lock()/LockShared().RecursiveMutex: Wraps std::recursive_mutex.MutexLocker / RecursiveMutexLocker / SharedLockExclusive / SharedLockShared: RAII lock guards with annotations.GlobalChecks.h / GlobalChecks.cppthreadIsMain(): Checks if current thread is the main thread (captured at static init).releaseAssert(e): Assert that is NOT compiled out by NDEBUG. Prints backtrace and aborts.releaseAssertOrThrow(e): Like releaseAssert but throws runtime_error instead of aborting.dbgAssert(e): Debug-only assert.LockGuard / RecursiveLockGuard: Aliases that adapt between Tracy-instrumented and plain lock guards.Thread.h / Thread.cpprunCurrentThreadWithLowPriority() / runCurrentThreadWithMediumPriority(): OS-level thread priority adjustment.futureIsReady(std::future|std::shared_future): Non-blocking readiness check.JitterInjection.h / JitterInjection.cppTesting framework for probabilistic thread-delay injection (enabled via BUILD_THREAD_JITTER).
JitterInjector::injectDelay(probability, minUsec, maxUsec): Probabilistically sleeps to expose race conditions.JITTER_INJECT_DELAY(), JITTER_YIELD() macros: Compile to no-ops in production.NonCopyable.hBase structs: NonCopyable, NonMovable, NonMovableOrCopyable — disable copy/move semantics via deleted special members.
MetricsRegistry (MetricsRegistry.h, MetricsRegistry.cpp)Extends medida::MetricsRegistry with SimpleTimer support. NewSimpleTimer() registers lightweight timers. syncSimpleTimerStats() syncs max values (called on /metrics endpoint).
SimpleTimer / SimpleTimerContext (SimpleTimer.h, SimpleTimer.cpp)Lightweight replacement for medida::Timer using counters (sum, count, max). Avoids histogram overhead. Uses medida::Counter internally for metric export. TimeScope() returns an RAII context for automatic timing.
MetricResetter (MetricResetter.h, MetricResetter.cpp)Implements medida::MetricProcessor to reset all metric types (Counter, Meter, Histogram, Timer, Buckets).
LogSlowExecution (LogSlowExecution.h, LogSlowExecution.cpp)RAII guard that logs a warning if a scope takes longer than a threshold (default 1 second). Two modes:
AUTOMATIC_RAII: Logs on destructor.MANUAL: Caller checks checkElapsedTime().RateLimitedLog: Subclass with zero threshold for rate-limited logging.types.h, types.cpp)Core type definitions and helpers for XDR/Stellar types.
Blob: std::vector<unsigned char>.LedgerKeySet: std::set<LedgerKey, LedgerEntryIdCmp>.LedgerEntryKey(LedgerEntry): Extracts the key from an entry.assetToString(), getIssuer(), isIssuer(), assetCodeToStr(), strToAssetCode(), isAssetValid(), compareAsset().getBucketLedgerKey(): Extracts LedgerKey from BucketEntry or HotArchiveBucketEntry.addBalance(balance, delta, max): Safe balance addition with overflow check.isZero(uint256), lessThanXored(): Hash/XOR comparison helpers.unsignedToSigned(): Safe conversion with overflow check.formatSize(): Human-readable size formatting.iequals(): Case-insensitive string comparison.>=, >, ==.isAsciiAlphaNumeric(), isAsciiNonControl(), toAsciiLower() — locale-independent.roundDown(v, m): Round down to largest multiple of power-of-2.TxResource.h, TxResource.cpp)ResourceMulti-dimensional resource vector for transaction resource accounting (operations, instructions, byte sizes, ledger entries). Supports:
+=, -=, +, -, with overflow-safe bigDivideOrThrow.<=, ==, >, anyLessThan(), anyGreater().subtractNonNegative(), limitTo(), saturatedMultiplyByDouble().Resource::Type enum: OPERATIONS, INSTRUCTIONS, TX_BYTE_SIZE, DISK_READ_BYTES, WRITE_BYTES, READ_LEDGER_ENTRIES, WRITE_LEDGER_ENTRIES.xdrquery/)A mini query language for filtering XDR objects by field values.
XDRFieldResolver.h: Template-based XDR field traversal using xdr_traits. Resolves dotted field paths (e.g., "data.account.balance") to values. Handles unions, enums, public keys (string representation), opaque arrays (hex).XDRQueryEval.h: Expression evaluation engine. EvalNode hierarchy: LiteralNode, ColumnNode, BoolOpNode (AND/OR/NOT), ComparisonOpNode (==, !=, <, >, <=, >=). DynamicXDRGetter interface for runtime field access.XDRQuery.h: TypedDynamicXDRGetterResolver<T> connects XDR types to the query engine. matchXDRQuery() function parses and evaluates queries against XDR objects.XDRQueryError.h: Exception type for query parsing/evaluation errors.XDRCereal.h / XDRCereal.cppCustom cereal_override functions for human-readable JSON serialization of XDR types. Overrides for: public keys (StrKey), assets (code+issuer), enums (string names), opaque data (hex), optional pointers (null handling), SCAddress, MuxedAccount, ConfigUpgradeSetKey.
xdrToCerealString(t, name, compact): Serialize any XDR type to JSON string.BufferedAsioCerealOutputArchive.hCustom cereal output archive that writes through OutputFileStream (supporting fsync) instead of std::ofstream.
XDROperators.hImports xdr::operator== and xdr::operator< into the stellar namespace.
RandHasher<T> (RandHasher.h)Hash functor that XORs the output of std::hash<T> with a random mixer value (initialized once at startup). Prevents hash-flooding DoS attacks on hash tables.
HashOfHash (HashOfHash.h, HashOfHash.cpp)Specialization of std::hash<stellar::uint256> for use in hash maps keyed by transaction/ledger hashes.
ProtocolVersion.h, ProtocolVersion.cpp)ProtocolVersion enumEnumerates all protocol versions V_0 through V_26.
protocolVersionIsBefore(version, beforeVersion): Strictly less than.protocolVersionStartsFrom(version, fromVersion): Greater than or equal.protocolVersionEquals(version, equalsVersion): Exact match.SOROBAN_PROTOCOL_VERSION = V_20PARALLEL_SOROBAN_PHASE_PROTOCOL_VERSION = V_23AUTO_RESTORE_PROTOCOL_VERSION = V_23StatusManager (StatusManager.h)Manages status messages by category (HISTORY_CATCHUP, HISTORY_PUBLISH, NTP, REQUIRES_UPGRADES). Used for the /info JSON endpoint.
SecretValue (SecretValue.h)Wrapper around std::string to prevent accidental logging of sensitive values (DB passwords, secret keys).
Decoder.hBase32/Base64 encoding and decoding helpers (encode_b32, encode_b64, decode_b32, decode_b64).
Math.h / Math.cppRandom number utilities: rand_fraction(), rand_flip(), rand_uniform(), rand_element(), k_means(), closest_cluster(), exponentialBackoff(). initializeAllGlobalState() initializes all PRNG and hash seeds. reinitializeAllGlobalStateWithSeed() (test-only) resets for reproducibility.
MetaUtils.h / MetaUtils.cppnormalizeMeta(): Sorts order-agnostic fields in TransactionMeta / LedgerCloseMeta for deterministic comparison/hashing.
DebugMetaUtils.h / DebugMetaUtils.cppManages debug meta-data files (XDR segments of ledger metadata). Handles file naming, path construction, listing, and cleanup based on configurable segment sizes (256 ledgers).
Backtrace.h / Backtrace.cppprintCurrentBacktrace(): Prints stack trace via the Rust bridge.
must_use.hMUST_USE macro: Maps to __attribute__((warn_unused_result)) where supported.
Algorithm.hsplit() function template: Splits a vector into a map of vectors keyed by an extractor function.
asio.hCorrectly configures and includes ASIO headers with proper preprocessor definitions (ASIO_SEPARATE_COMPILATION, ASIO_STANDALONE). Must be included before other system headers.
TcmallocConfig.cppTcmalloc configuration (if tcmalloc is the allocator).
Main event loop: VirtualClock::crank() → dispatches timers → polls ASIO IO → runs Scheduler::runOne() → transfers pending actions from thread-safe queue to scheduler.
Cross-thread work submission: Any thread → VirtualClock::postAction() (mutex-protected queue) → main thread dequeues into Scheduler → fair-scheduled execution.
XDR file I/O: Write path: XDROutputFileStream::writeOne() → 4-byte size header + XDR marshal → buffered write → optional fsync. Read path: XDRInputFileStream::readOne() → read size header → read payload → XDR unmarshal.
Metrics flow: Code uses medida::Timer/SimpleTimer → MetricsRegistry collects → syncSimpleTimerStats() synchronizes max tracking → exported via /metrics endpoint.
Resource accounting: Resource vectors created per-transaction → arithmetic operations check against ledger limits → anyGreater()/<= comparisons for admission control.