Data Council 2026 in Austin gathered data engineers, analytics engineers, and platform architects working across Spark, dbt, Apache Iceberg, Kafka, and Flink. UUID v7’s growing adoption in OLTP databases — driven by PostgreSQL 18’s native uuidv7() function and Python 3.13’s uuid.uuid7() — raised practical questions about downstream handling in analytical pipelines. Sessions covered partition strategy, surrogate key design, cross-system deduplication, and streaming event ID patterns. See RFC 9562 for the standard that defined UUID v7.
Event details
| Conference | Data Council 2026 |
| Date | April 7–9, 2026 |
| Location | Austin, TX, USA |
| Focus | UUID v7 in data pipelines — Spark, dbt, Iceberg, Kafka, Flink |
Session: UUID v7 partition pruning in PySpark
UUID v7’s time-ordered byte layout enables Spark to prune Parquet row groups on time-bounded queries without a separate timestamp partition column:
from pyspark.sql import SparkSession, functions as F
from datetime import datetime, timedelta, timezone
spark = SparkSession.builder.appName("uuid7-pruning").getOrCreate()
def ts_to_uuid7_lower(dt: datetime) -> str:
"""Build a minimum UUID v7 boundary for a given datetime."""
ts_ms = int(dt.timestamp() * 1000)
# 48-bit timestamp in hex, zero-pad remaining bits (version=7, variant=10xx)
return f"{ts_ms:012x}-0000-7000-8000-000000000000"
def ts_to_uuid7_upper(dt: datetime) -> str:
"""Build a maximum UUID v7 boundary for a given datetime."""
ts_ms = int(dt.timestamp() * 1000)
return f"{ts_ms:012x}-ffff-7fff-bfff-ffffffffffff"
now = datetime.now(tz=timezone.utc)
lower = ts_to_uuid7_lower(now - timedelta(hours=1))
upper = ts_to_uuid7_upper(now)
events_df = spark.read.parquet("s3://data-lake/events/")
# Spark prunes row groups using the UUID column — no separate ts partition needed
recent_events = events_df.filter(
(F.col("event_id") >= lower) & (F.col("event_id") < upper)
)
recent_events.explain("formatted") # verify Index Scan / row group skipping
recent_events.count()
Speakers reported a 12× reduction in bytes read on a 10TB events table when switching from UUID v4 to UUID v7 primary keys — Parquet row group min/max statistics on the UUID column became effective. See UUID v7 in Distributed Systems.
Session: dbt cross-database UUID v7 macro
Analytics engineers at the conference shared a production dbt macro for dispatching UUID v7 generation across PostgreSQL, BigQuery, and Snowflake:
-- macros/generate_uuid7.sql
{% macro generate_uuid7() %}
{{ return(adapter.dispatch('generate_uuid7', 'my_project')()) }}
{% endmacro %}
{% macro postgres__generate_uuid7() %}
uuidv7()
{% endmacro %}
{% macro bigquery__generate_uuid7() %}
-- BigQuery does not yet have native UUID v7; falls back to UUID v4
GENERATE_UUID()
{% endmacro %}
{% macro snowflake__generate_uuid7() %}
UUID_STRING()
{% endmacro %}
{% macro duckdb__generate_uuid7() %}
gen_random_uuid()
{% endmacro %}
-- models/stg_orders.sql
SELECT
{{ generate_uuid7() }} AS surrogate_key,
order_id AS source_order_id,
customer_id,
status,
total_amount,
currency,
created_at
FROM {{ source('app_db', 'orders') }}
The macro pattern was recommended only for new records without a source UUID — for OLTP tables already using UUID v7 primary keys, carry the source UUID directly as the surrogate key. See UUID in Databases.
Session: Apache Iceberg row IDs and UUID v7
Apache Iceberg v3’s row-level change tracking raises the question of ID design for Iceberg tables. Sessions covered using UUID v7 as Iceberg row IDs for operational tables with frequent MERGE operations:
# PySpark + Iceberg — MERGE using UUID v7 row IDs
spark.sql("""
MERGE INTO catalog.prod.orders AS target
USING (
SELECT * FROM staging.orders_batch
) AS source
ON target.order_id = source.order_id
WHEN MATCHED AND source.status != target.status THEN
UPDATE SET status = source.status, updated_at = source.updated_at
WHEN NOT MATCHED THEN
INSERT *
""")
# Query recent inserts using UUID v7 timestamp prefix
spark.sql("""
SELECT order_id, customer_id, status, total_amount
FROM catalog.prod.orders
WHERE order_id >= '019434a0-0000-7000-8000-000000000000' -- last 24h lower bound
ORDER BY order_id
LIMIT 1000
""")
UUID v7’s lexicographic sort order aligns with Iceberg’s hidden partitioning on _partition_id — time-bounded queries on the UUID column benefit from Iceberg’s metadata filtering. See UUID v7 in Kafka Event Streaming.
Session: Kafka event streaming with UUID v7
Event streaming sessions examined UUID v7 as Kafka message keys and event IDs:
# Python Kafka producer — UUID v7 as message key and event ID
from confluent_kafka import Producer
from uuid import uuid7 # Python 3.13+
import json
from datetime import datetime, timezone
producer = Producer({'bootstrap.servers': 'broker:9092'})
def produce_order_event(order_data: dict) -> str:
event_id = str(uuid7()) # UUID v7 — time-ordered key
message_key = event_id.encode() # Kafka key = UUID v7 for ordered partitioning
payload = {
'event_id': event_id,
'event_type': 'order.created',
'timestamp': datetime.now(tz=timezone.utc).isoformat(),
'data': order_data,
}
producer.produce(
topic = 'order-events',
key = message_key,
value = json.dumps(payload).encode(),
callback = lambda err, msg: print(f"Delivered {msg.key()}" if not err else err),
)
producer.flush()
return event_id
Using UUID v7 as the Kafka message key routes time-adjacent events to the same partition, improving consumer locality. See UUID v7 in Kafka Event Streaming for the full partitioning analysis.
Session: Delta Lake deduplication
# Delta Lake MERGE — UUID v7 as join key for deduplication
from delta.tables import DeltaTable
from pyspark.sql import functions as F
orders_delta = DeltaTable.forName(spark, "prod.orders")
new_batch = spark.read.parquet("s3://landing/orders/batch_20260407/")
# UUID v7's global uniqueness makes cross-source deduplication collision-free
orders_delta.alias("target") \
.merge(
new_batch.alias("source"),
"target.order_id = source.order_id"
) \
.whenMatchedUpdate(
condition = "source.updated_at > target.updated_at",
set = { "status": "source.status", "updated_at": "source.updated_at" }
) \
.whenNotMatchedInsertAll() \
.execute()
Sessions emphasised that UUID v7’s 128 bits of entropy (48 timestamp + 4 version + 2 variant + 74 random) make cross-source collision probability negligible — no tenant prefix or namespace column is needed for multi-source deduplication. See UUID Collision.
Attendee tips
- Carry source UUID v7 keys downstream — do not regenerate surrogate keys in dbt for tables that already have UUID v7 primary keys; the source key is already globally unique and time-ordered.
- Parquet statistics — UUID v7 lexicographic order means Parquet column chunk min/max statistics are effective for time-bounded query pruning.
- Kafka key = UUID v7 — time-adjacent events land in the same partition, improving consumer throughput for ordered processing.
- BigQuery and Snowflake lag — neither has native UUID v7 generation in 2026; the dbt macro fallback to UUID v4 is safe for new surrogate keys.
- Iceberg hidden partitioning — combining UUID v7 as the row ID with Iceberg’s
days(created_at)hidden partition gives both efficient MERGE and time-bounded scan performance.
Related events
Further reading
- UUID v7 in Distributed Systems
- UUID v7 in Kafka Event Streaming
- UUID in Apache Flink
- UUID in Databases — storage and indexing
- Time Precision — UUID v7 millisecond timestamp
- RFC 9562 — the current UUID standard
External references
- Data Council — official site
- Apache Iceberg documentation
- Delta Lake MERGE documentation
- RFC 9562 — UUID standard
Frequently asked questions
How does UUID v7 improve Spark partition pruning compared to UUID v4?
UUID v4 is random — Spark cannot derive time bounds from UUID v4 values, so a query like "events in the last hour" requires a full scan unless a separate timestamp column and partition are maintained. UUID v7 embeds a 48-bit Unix millisecond timestamp in the most-significant bytes, so the UUID column sorts lexicographically in time order. Spark can prune Parquet row groups by comparing UUID byte ranges, reducing I/O for time-bounded queries without a separate partition column. See UUID v7 in Distributed Systems.
How do I generate UUID v7 boundary values for Spark range filters?
Build the lower bound by setting the 48-bit timestamp to your start epoch milliseconds and zeroing the remaining 80 bits: f"{ts_ms:012x}-0000-7000-8000-000000000000" in Python. The upper bound sets the timestamp to your end epoch milliseconds with the same zero suffix. These boundary strings sort correctly when compared as standard UUID strings because UUID v7 uses a big-endian timestamp. See Time Precision for the bit layout.
Does dbt support UUID v7 natively for surrogate key generation?
dbt's built-in dbt_utils.generate_surrogate_key() generates deterministic hashed keys — it does not generate UUID v7. To use UUID v7 in dbt, write a cross-database macro that dispatches to the correct native function: uuidv7() in PostgreSQL 18, GENERATE_UUID() in BigQuery (UUID v4 equivalent), and UUID_STRING() in Snowflake. Use UUID v7 from source OLTP systems as natural keys rather than regenerating them in dbt. See UUID in Databases.