Schema Management & Data Lineage

Data platforms face a persistent tension around schema. Define it upfront, and you get rigidity — every upstream change requires a migration, a contract update, or a broken pipeline. Discover it after the fact, and you get fragility — schema drift goes unnoticed until something downstream fails.

This document illustrates how Infino implements an alternative approach: schema that is derived automatically from data, evolves continuously, records its own change history, and attaches to a transform DAG for lineage — without requiring a separate catalog, migration framework, or contract layer.

How Infino Derives Schema

Infino accepts schemaless data — JSON documents, structured logs, time-series metrics — and supports SQL queries over that data. There is no upfront schema definition step. You do not write DDL, define a model file, or register a schema before ingesting data.

Grammar-based type classification

When a document arrives, every field value is parsed through a formal grammar that recognizes types deterministically. This is not heuristic matching or regex — the grammar is a structured parser that produces unambiguous type assignments. The same value always resolves to the same type, regardless of when or where it's ingested.

The type taxonomy is extensive, covering the kinds of data that appear in real-world observability, analytics, and operational datasets:

Category	Types	Example Values
Numeric	Integer, Float	`42`, `-7`, `3.14159`, `1.5e-3`
Boolean	Boolean	`true`, `false`
Date/Time	ISO 8601, RFC 2822, Log Date, Full Date, American Date, European Date, Verbose Date, Financial Date, DMY Date, Generic Date, Date Math	`2024-01-15T10:30:00Z`, `Mon, 15 Jan 2024 10:30:00 +0000`, `Mon Jan 15 10:30:00 2024`, `01/15/2024`, `15/01/2024`, `Monday, the 15th of January, 2024`, `15th January 2024`, `15-Jan-2024`, `2024-01-15 10:30:00`, `now-1d/d`
Network	IPv4, IPv6	`192.168.1.1`, `2001:0db8:85a3::8a2e:0370:7334`
Geospatial	Geo (lat/lon)	`37.7749,-122.4194`
Structured	Vector, Array, Object, Nested	`[0.1, 0.2, ..., 0.9]` (embedding), `["a", "b", "c"]`
Text	String (fallback)	Any value that doesn't match a more specific type

Date handling deserves special attention. In practice, data arrives from many systems — application logs, financial feeds, IoT devices, web servers — each with its own date conventions. Rather than requiring normalization before ingestion, Infino's grammar recognizes twelve distinct date formats natively. A log timestamp like Mon Jan 15 10:30:00 2024 and an ISO 8601 timestamp like 2024-01-15T10:30:00Z are both classified correctly without any configuration.

Date math expressions like now-1d/d are also recognized, which supports relative time queries commonly used in dashboards and alerting.

What happens when a field has multiple types

Real-world data is messy. The same field name can carry different types across documents — an event_id that's sometimes an integer and sometimes a string, or a value field that alternates between numeric and null.

When Infino encounters a new type for an existing field, it doesn't reject the data or coerce the value. Instead, it adds the new type to the field's schema and records the change. The field's schema then carries:

The most recent type observed
The full set of all types ever observed
A timestamped history of when each new type appeared (see Schema History)

This means a single field can have a schema like:

{
  "event_id": {
    "infino_type": "String",
    "source": "Logs",
    "type_history": [
      { "infino_type": "Integer", "changed_at_ms": 1704067200000 },
      { "infino_type": "String",  "changed_at_ms": 1706745600000 }
    ]
  }
}

This tells you event_id was first seen as an Integer, and a String variant appeared later. Both types are valid for this field. The current (most recent) type is String.

Nested fields and keyword variants

JSON documents often contain nested objects. Infino flattens these into dot-notation paths:

{ "user": { "name": "Alice", "address": { "city": "Seattle" } } }

Produces fields: user.name (String), user.address.city (String).

Array indices are normalized — if a field appears as items.0.price and items.1.price, Infino recognizes these as the same field items.price.

Text fields automatically get keyword variants for exact-match queries and aggregations, following the same convention used by OpenSearch and Elasticsearch.

Concurrent schema evolution

Schema updates are lock-free and concurrent. Multiple ingestion threads can add new fields or new types to existing fields simultaneously without blocking each other. There is no batch schema discovery step — the schema is always current as of the last ingested document.

When Infino runs in a distributed configuration, each node maintains its own schema view. These views are merged during query time, combining type histories from all nodes into a unified schema response.

Schema History

In most systems, when a field's type changes, you discover it through a pipeline failure, a dashboard showing unexpected nulls, or a data quality alert after the fact. The root cause — when and why the type changed — often requires manual investigation across multiple systems.

Infino records every type change as it happens. When a field that was previously seen as an Integer receives a String value for the first time, a type history entry is created automatically with a millisecond-precision timestamp. No configuration or opt-in is required.

A real-world scenario

Consider a web application that sends access logs to Infino. The status_code field starts as an integer:

{ "status_code": 200, "path": "/api/users", "timestamp": "2024-01-01T00:00:00Z" }

Infino classifies status_code as Integer. The schema records:

{
  "status_code": {
    "infino_type": "Integer",
    "source": "Logs",
    "type_history": [
      { "infino_type": "Integer", "changed_at_ms": 1704067200000 }
    ]
  }
}

Six weeks later, a new version of the application is deployed. A developer accidentally changes the status code serialization to a string:

{ "status_code": "200", "path": "/api/users", "timestamp": "2024-02-15T00:00:00Z" }

Infino classifies this new value as String and appends to the history:

{
  "status_code": {
    "infino_type": "String",
    "source": "Logs",
    "type_history": [
      { "infino_type": "Integer", "changed_at_ms": 1704067200000 },
      { "infino_type": "String",  "changed_at_ms": 1708012800000 }
    ]
  }
}

Now when a data engineer investigates why a numeric aggregation on status_code is returning unexpected results, the schema history immediately shows: the field started as Integer on January 1, and a String variant appeared on February 15. The investigation is narrowed to deployments around that date.

Schema merging across distributed nodes

In a distributed deployment, different nodes may observe type changes at slightly different times. When schema is queried, Infino merges type histories from all nodes. If Node A recorded Integer at t1 and Node B recorded String at t2, the merged history contains both entries in chronological order, preserving the full timeline regardless of which node first observed each type.

Transforms

Transforms in Infino are first-class platform objects — not external scripts, standalone SQL files, or jobs defined in a separate orchestrator. A transform is a persistent, named definition that the platform manages end-to-end.

What a transform contains

Component	Description	Example
Name	Human-readable identifier	`daily_error_summary`
Source query	The query that reads data	SQL, QueryDSL, PromQL, or natural language
Target dataset	Where results are written	`error_summary_daily`
Source datasets	Explicit list of datasets read	`["app-logs-prod", "app-logs-staging"]`
Schedule	When to run	`0 2 * * *` (daily at 2am)
Materialization	How results are stored	Table (full refresh), Incremental (append), or View
Enabled	Whether the schedule is active	`true` / `false`

Query types

Transforms support multiple query languages, matching how different teams prefer to work with data:

SQL: Standard SQL queries against datasets. Suitable for aggregations, joins, and analytical transformations.
QueryDSL: JSON-based query language compatible with OpenSearch/Elasticsearch. Useful for full-text search, filtered aggregations, and log analytics.
PromQL: Prometheus-compatible query language for time-series metrics. Enables metric transformations like rate calculations, quantile aggregations, and recording rules.
Natural language: Describe what you want in plain English. Infino translates the intent into a query, executes it, and materializes the result.

Materialization modes

The materialization mode determines how the target dataset is populated:

Table: Full refresh. Each execution replaces the entire target dataset with fresh results. Use this for summary tables, daily reports, or any case where you want a clean snapshot.
Incremental: Append or upsert. Each execution adds new rows or updates existing ones based on time windows. Use this for growing datasets like event rollups or running aggregations.
View: No physical materialization. The query runs on demand when the target dataset is queried. Use this for lightweight derived views that don't need to be pre-computed.

Execution lifecycle

When a transform executes — whether triggered by schedule, on-demand, or via the API — Infino:

Runs the source query against the source datasets
Writes the results to the target dataset
Records an execution log (status, start/end time, rows written, errors)
Updates the target dataset's schema automatically from the written data

Step 4 is the key connection to schema management: the target dataset's schema is not defined by the transform. It is derived from the data the transform produces. If the transform's query changes and starts outputting a new column or a different type for an existing column, the schema evolves automatically and the change is recorded in the type history.

Example: a YAML transform definition

Version: 2025-01-01
Metadata:
  name: hourly_error_rates
  description: Compute error rates per service, per hour
Source:
  type: sql
  statement: |
    SELECT
      service_name,
      DATE_TRUNC('hour', timestamp) AS hour,
      COUNT(*) AS total_requests,
      SUM(CASE WHEN status_code >= 500 THEN 1 ELSE 0 END) AS error_count,
      ROUND(100.0 * SUM(CASE WHEN status_code >= 500 THEN 1 ELSE 0 END) / COUNT(*), 2) AS error_rate
    FROM access_logs
    WHERE timestamp >= NOW() - INTERVAL '1 hour'
    GROUP BY service_name, DATE_TRUNC('hour', timestamp)
  source_datasets:
    - access_logs
Target:
  dataset: error_rates_hourly
  materialization: incremental
Schedule:
  cron: "0 * * * *"

This transform runs every hour, computes error rates from access_logs, and appends results to error_rates_hourly. The target dataset's schema — service_name (String), hour (Date), total_requests (Integer), error_count (Integer), error_rate (Float) — is derived automatically from the query output.

The Transform DAG and Schema Lineage

When multiple transforms exist, they naturally form a directed acyclic graph (DAG). If Transform A writes to Dataset X, and Transform B reads from Dataset X, there is a dependency edge from A to B through X.

Infino computes this DAG from the transform definitions and enriches each node with schema information:

Target schema: The schema of the dataset this transform writes to, including its full type history
Source schemas: The schemas of each dataset this transform reads from, including their type histories

A concrete example: three-stage pipeline

Consider a pipeline that processes raw web server logs into a business-ready dataset:

┌──────────────┐         ┌──────────────────┐         ┌──────────────────────┐
│  access_logs │         │  enriched_events │         │  daily_kpi_summary   │
│  (raw data)  │────────▶│  (cleaned +      │────────▶│  (business metrics)  │
│              │         │   enriched)      │         │                      │
└──────────────┘         └──────────────────┘         └──────────────────────┘
       │                         │                            │
   Transform 1:              Transform 2:                  Target
   "clean_and_enrich"        "compute_daily_kpis"          dataset

Transform 1 (clean_and_enrich):

Reads access_logs (raw)
Parses user agents, resolves geo-IP, normalizes status codes
Writes to enriched_events

Transform 2 (compute_daily_kpis):

Reads enriched_events
Computes daily unique visitors, revenue, conversion rates
Writes to daily_kpi_summary

The DAG for this pipeline carries schemas at every node:

{
  "nodes": [
    {
      "name": "clean_and_enrich",
      "target_dataset": "enriched_events",
      "source_datasets": ["access_logs"],
      "target_schema": {
        "status_code": {
          "infino_type": "Integer",
          "type_history": [
            { "infino_type": "Integer", "changed_at_ms": 1704067200000 }
          ]
        },
        "country": {
          "infino_type": "String",
          "type_history": [
            { "infino_type": "String", "changed_at_ms": 1704067200000 }
          ]
        }
      },
      "source_schemas": {
        "access_logs": {
          "status_code": {
            "infino_type": "String",
            "type_history": [
              { "infino_type": "Integer", "changed_at_ms": 1704067200000 },
              { "infino_type": "String",  "changed_at_ms": 1708012800000 }
            ]
          }
        }
      }
    },
    {
      "name": "compute_daily_kpis",
      "target_dataset": "daily_kpi_summary",
      "source_datasets": ["enriched_events"]
    }
  ],
  "edges": [
    { "from": "clean_and_enrich", "to": "compute_daily_kpis", "via_dataset": "enriched_events" }
  ],
  "execution_order": ["clean_and_enrich", "compute_daily_kpis"]
}

Now you can trace schema through the pipeline:

access_logs.status_code drifted from Integer to String on February 15
enriched_events.status_code is still Integer — meaning Transform 1's SQL must be casting the string back to an integer
If Transform 1's logic changes and stops casting, the target schema will record a new String entry in its type history

This kind of cross-dataset schema tracing is available from the DAG without a separate lineage catalog.

Declared lineage vs. inferred lineage

Each transform explicitly declares which datasets it reads from. This is a deliberate design choice.

Traditional approaches often infer lineage by parsing SQL or instrumenting DAG runners. This works until someone refactors a query, renames a CTE, or introduces a dynamic table reference — at which point lineage silently breaks.

Declared lineage is more robust: if a transform reads from a dataset, it says so. The tradeoff is that it requires the transform author to list source datasets. In practice this is low friction since the datasets are already referenced in the query.

Contrast with Traditional Approaches

The following is not a claim that one approach is universally better. These are tradeoffs, and the right choice depends on the team, the data, and the organizational context.

Side-by-side comparison

Concern	Traditional Pipeline Tools	Infino
Schema definition	Explicit: model files, DDL, contracts. Author defines schema before or alongside data.	Implicit: schema derived from data at ingestion. No upfront definition.
Schema change detection	CI/CD: breaking-change detection, migration scripts, contract tests. Changes are deliberate and reviewed.	Automatic: new types recorded with timestamps as they appear. No approval gate.
Schema versioning	External: version-controlled model files, migration history in a database.	Built-in: type history per field, timestamped, queryable.
Lineage	Inferred: SQL parsing, DAG runner instrumentation, manual annotations. Stored in a separate catalog.	Declared: source datasets listed in transform definition. Schemas attached to DAG. No separate catalog.
Schema enforcement	Strong: bad data rejected if it doesn't match the contract.	None: bad data gets a schema entry. Observability, not governance.
Transform definition	External: SQL files in a repo, orchestrator config, separate scheduling system.	Native: transforms are platform objects with query, schedule, materialization, lineage in one definition.
Target schema management	Manual: DDL or model file for the target table. Must be kept in sync with the query.	Automatic: target schema derived from what the transform writes. Always in sync.

Schema definition

Traditional tools require schema to be defined explicitly — as model files, DDL, or contract specifications. This gives teams upfront control and validation. A schema mismatch is caught before data flows. The cost is maintenance: every upstream change requires a corresponding schema update, reviewed and deployed.

In Infino, schema emerges from the data itself. There is no schema to maintain separately. The cost is that there is no pre-ingestion validation — if a producer starts sending bad data, Infino will accept it and record the new types. You observe the drift rather than prevent it.

Schema evolution

Traditional tools handle schema changes through migrations, versioned contracts, or breaking-change detection in CI. The change is deliberate: someone writes a migration, it's reviewed, tested, and deployed. This is robust but slow — especially in fast-moving environments where upstream producers change frequently.

In Infino, schema evolution is automatic and immediate. When a field's type changes, it's recorded with a millisecond timestamp. There is no approval gate and no mechanism to block the change — only to observe that it occurred and when. This is faster but provides observability without governance.

Lineage

Traditional tools extract lineage by parsing SQL statically (which breaks on dynamic queries or complex CTEs), instrumenting DAG runners (which requires integration with the orchestrator), or requiring manual annotations (which drift out of date). The lineage metadata is stored in a separate catalog service that must be kept in sync.

In Infino, lineage is declared in the transform definition and schemas attach directly to the DAG. There is no separate catalog to maintain. The tradeoff is that lineage only covers transforms defined within Infino — data that arrives from external ETL systems has no lineage context until it's ingested.

Where traditional tools complement this approach

These are complementary concerns, not competing ones:

Complex multi-step SQL transformations that benefit from dedicated testing frameworks with unit tests, integration tests, and data validation assertions
Data contracts that enforce schema expectations at organizational boundaries — e.g., between the producer team and the consumer team
CI/CD integration for schema validation before deployment — catching breaking changes before they reach production
Cross-platform lineage that spans multiple systems (data warehouse, feature store, ML pipeline, BI tool) beyond a single data platform

Automatic schema derivation and history reduce the day-to-day burden of schema management. Traditional tools can layer governance, validation, and enforcement on top when the organization needs it.

Summary

Infino's approach to schema management rests on three ideas:

Derive schema from data: Instead of defining schema upfront, let the platform discover it from the data it receives. A formal grammar assigns types deterministically from a taxonomy of 30+ types covering dates, numbers, IPs, geo coordinates, vectors, and more. Nested fields are flattened automatically. Schema evolves concurrently and without locks as new data arrives.
Record changes over time: When a field's type evolves, record the change with a millisecond timestamp. Make this history queryable so teams can audit drift, debug data quality issues, and trace when upstream producers changed their behavior. In distributed deployments, type histories from all nodes merge into a unified timeline.
Attach schemas to the transform DAG: Transforms are native platform objects with explicit source datasets, schedules, and materialization modes. They form a DAG, and each node in the DAG carries the full schema — including type history — of its source and target datasets. This provides end-to-end schema lineage without a separate catalog service.

This does not replace the need for data governance, contracts, or testing. It provides a foundation that reduces manual schema maintenance and makes schema evolution visible by default — so that when governance is needed, the data to support it is already there.

How Infino Derives Schema​

Grammar-based type classification​

What happens when a field has multiple types​

Nested fields and keyword variants​

Concurrent schema evolution​

Schema History​

A real-world scenario​

Schema merging across distributed nodes​

Transforms​

What a transform contains​

Query types​

Materialization modes​

Execution lifecycle​

Example: a YAML transform definition​

The Transform DAG and Schema Lineage​

A concrete example: three-stage pipeline​

Declared lineage vs. inferred lineage​

Contrast with Traditional Approaches​

Side-by-side comparison​

Schema definition​

Schema evolution​

Lineage​

Where traditional tools complement this approach​

Summary​