Timeline

• Timeline ID files – already resolved to person_id and id. These are ready to load into the timeline table.
• Timeline Raw files – do not contain person_id. They must go through person resolution and ID assignment before they can be loaded.

Timeline ID files

Use when you want data that is ready to be:

• Loaded into a downstream timeline table.
• Joined against plugin-specific detail tables.
• De-duplicated by id (UUID).

Core shape

Timeline ID files are typically produced by:

• Plugin or ingestion code that has already resolved a stable person_id and plugin_id.
• Mappers that call getEntryTypeId and getTimelineEntryUUID from @engine9/input-tools.

Minimum required fields for a Timeline ID file that downstream workers will accept:

• id: UUID for the timeline entry.
  • Generated by getTimelineEntryUUID, or provided as a stable remote_entry_uuid.
  • In InputWorker.id, appendTimelineId writes this into the id column.
• ts: timestamp (string or number) that can be parsed into a Date.
• person_id: internal numeric person id.
• entry_type_id: integer from TIMELINE_ENTRY_TYPES.

Common optional fields:

• source_code_id: numeric source code identifier.
• email_domain: lower-cased domain, often derived from email.
• Any number of extra columns (detail fields); these can be copied into plugin-specific detail tables.

The downstream timeline table schema usually includes:

• A primary key id column (UUID stored as text).
• A timestamp column ts (millis since epoch).
• Integer columns entry_type_id and person_id.
• Optional columns such as source_code_id and email_domain.

How to construct Timeline ID files with input-tools

When authoring a Timeline ID-producing job or plugin using @engine9/input-tools:

• Always include id, ts, person_id, and entry_type_id on each emitted row.
• Prefer numeric entry_type_id, but you may also keep a string entry_type for debugging; resolution between the two happens via TIMELINE_ENTRY_TYPES, getEntryTypeId, and getEntryType.
• Include plugin_id; getTimelineEntryUUID uses it as the UUID namespace when generating id.

Timeline Raw files

Use when you have raw events from an external system and cannot yet assign person_id but still want to capture structured activity.

Examples:

• Raw web or email events that only know an email or other external identifier.
• Logs from external APIs where person resolution happens later in the pipeline.

Core shape

Timeline Raw files:

• Must not contain person_id (by definition for this skill).
• May or may not contain id.
  • If they do contain an id, it is usually an external event ID or remote_entry_uuid, not necessarily the final Engine9 id.
• Should contain enough information to derive:
  • A timestamp: ts (or a field that is mapped to ts).
  • An entry type: entry_type (string) or entry_type_id (numeric).
  • A person identifier that can be resolved later (e.g. remote_person_id, email, or similar).

Typical fields you will see:

• ts or a source-specific timestamp (later mapped to ts).
• entry_type or entry_type_id (e.g. 'EMAIL_UNSUBSCRIBE', 'EMAIL_OPEN', etc.).
• Contact fields: email, remote_person_id, phone number, etc.
• Source metadata: account_id, plugin_id, url, user_agent, ip_address, etc.

For example, a plugin may map an inbound event into a row with:

• ts, account_id, entry_type_id, email, email_domain, url, user_agent

and no person_id yet.

Converting Timeline Raw → Timeline ID

The usual pathway for Raw → ID is:

1. Map raw events into a timeline-shaped object (with ts, entry_type/entry_type_id, and contact info).
2. Resolve people (outside of input-tools):
   • Use your application’s person resolution or a server worker to:
     • Look up or create person rows.
     • Attach a canonical person_id to each row.
3. Assign timeline IDs with input-tools:
   • Use getEntryTypeId (if needed) to ensure entry_type_id is set from TIMELINE_ENTRY_TYPES when only entry_type is present.
   • Call getTimelineEntryUUID to:
     • Require ts, entry_type_id, plugin_id, and person_id.
     • Produce a deterministic, sortable UUID for id.
4. Write out a Timeline ID file (for example, parquet or CSV) with the full set of fields (id, ts, person_id, entry_type_id, optional source_code_id, etc.).

Choosing between Timeline ID and Timeline Raw

• Choose Timeline ID files when:

  • You can resolve person_id and plugin_id in the current process.
  • You want files that are immediately loadable into a timeline table.
  • You need deduplication by a stable id.

• Choose Timeline Raw files when:

  • You are at the edge of the system (plugins, collectors, ETL jobs) and only have partial identity information.
  • You plan a later enrichment step that will attach person_id and compute final id values.
  • You want to keep the ingestion simpler and defer canonicalization.

In practice:

• Plugins and edge collectors often emit Timeline Raw shaped data first.
• Downstream services or server workers then:
  • Resolve people (person_id).
  • Generate id via getTimelineEntryUUID.
  • Persist Timeline ID files and load them into the timeline and detail tables.

Reference helpers

When working with any timeline format, prefer the utilities in @engine9/input-tools:

• TIMELINE_ENTRY_TYPES (timelineTypes.js): bidirectional map between string entry types and numeric entry_type_id.
• getEntryTypeId: resolve entry_type → entry_type_id with validation.
• getEntryType: resolve entry_type_id → entry_type.
• getTimelineEntryUUID: generate or normalize id given ts, entry_type_id, plugin_id, and person_id, respecting remote_entry_uuid / remote_entry_id when present. Uses plugin_id as the UUID namespace.
• uuidIsValid: validate that a string is a proper UUID.

Use these helpers instead of hard-coding IDs or types whenever you construct timeline rows, whether Raw or ID.