Data Engineering Summit 2025 in San Francisco brought together engineers building large-scale data pipelines on Spark, dbt, Delta Lake, and Iceberg. UUID v7’s adoption in OLTP systems (PostgreSQL 18, Python 3.13) raised the question of how to handle UUID v7 keys downstream in analytical pipelines. Sessions covered partition strategy, deduplication, and cross-system ID consistency. See RFC 9562 for the standard that introduced UUID v7.
Key topics covered
UUID v7 as a pipeline surrogate key — Data engineers at the conference noted that UUID v7 arriving from OLTP sources (PostgreSQL 18, application services) can serve as the surrogate key in data warehouse tables without transformation. The timestamp prefix enables time-range partition pruning in Spark and BigQuery:
# PySpark — partition pruning on UUID v7
from pyspark.sql import functions as F
# Build lower/upper UUID v7 bounds for a 1-hour window
lower = "018fbe3a-0000-7000-8000-000000000000"
upper = "018fbe4e-0000-7000-8000-000000000000"
df.filter(
(F.col("event_id") >= lower) & (F.col("event_id") < upper)
).show()
Because UUID v7 sorts lexicographically in creation order, Spark can prune Parquet row groups without a separate timestamp column. See Time Precision for the 48-bit millisecond timestamp structure.
Delta Lake — deduplication on UUID v7
from delta.tables import DeltaTable
# Merge using UUID v7 as the join key — deterministic deduplication
DeltaTable.forName(spark, "orders") \
.merge(
source = new_data,
condition = "target.order_id = source.order_id"
) \
.whenNotMatchedInsertAll() \
.execute()
UUID v7’s global uniqueness means the order_id join key is collision-free across all source systems — no namespace prefixing or tenant ID column is needed to prevent cross-source ID collisions. See UUID Collision.
dbt — UUID v7 surrogate key macro
-- macros/uuid7.sql
{% macro uuid7() %}
{% if target.type == 'postgres' %}
uuidv7()
{% elif target.type == 'bigquery' %}
GENERATE_UUID()
{% elif target.type == 'snowflake' %}
UUID_STRING()
{% else %}
{{ exceptions.raise_compiler_error("uuid7 not implemented for " ~ target.type) }}
{% endif %}
{% endmacro %}
-- models/orders.sql
SELECT
{{ uuid7() }} AS surrogate_key,
order_id,
status,
created_at
FROM {{ source('app', 'orders') }}
The macro dispatches to the correct database function — BigQuery and Snowflake do not yet have a native UUID v7 function and fall back to UUID v4-equivalent generation. See UUID v7 ORM Support in 2026 for the database support matrix.
Apache Flink — UUID v7 in streaming pipelines
Sessions also covered UUID v7 in Flink streaming jobs, building on UUID in Apache Flink:
// Flink map function — assign UUID v7 event IDs
public class AssignEventId implements MapFunction<RawEvent, EnrichedEvent> {
@Override
public EnrichedEvent map(RawEvent raw) {
return new EnrichedEvent(
UuidCreator.getTimeOrderedEpoch(), // UUID v7
raw.getPayload(),
raw.getTimestamp()
);
}
}
UUID v7’s timestamp prefix allowed Flink jobs to perform windowed aggregations directly on the event ID without maintaining a separate timestamp field — reducing state store size.
Further reading
- UUID v7 in Distributed Systems
- UUID in Apache Flink
- UUID v7 in Kafka Event Streaming
- UUID in Databases — storage and indexing
- Time Precision — UUID v7 millisecond timestamp
- RFC 9562 — the current UUID standard
External references
Frequently asked questions
How is UUID v7 used as a primary key in dbt models?
In dbt, generate a surrogate key using dbt_utils.generate_surrogate_key() for deterministic IDs, or use a source system UUID v7 directly as the natural key. If generating UUIDs in dbt macros, use a Jinja macro that calls the database's UUID v7 function: {{ adapter.dispatch('uuid7')() }} with implementations for PostgreSQL 18 (uuidv7()), BigQuery (GENERATE_UUID()), and Snowflake (UUID_STRING()). The key benefit over UUID v4: time-range partitioning on the UUID column without a separate timestamp. See UUID in Databases.
Does UUID v7 help with Spark partition skew?
Yes — UUID v7's random 74-bit suffix distributes records uniformly across partitions when used as the partition key (the timestamp prefix alone would cause temporal hotspots). The timestamp prefix additionally enables partition pruning on time-range queries: WHERE uuid_col >= lower_bound AND uuid_col < upper_bound resolves to a bounded set of Parquet partitions without a full scan. See UUID v7 in Distributed Systems.