Apache Spark distributes data processing across executor nodes, often hundreds at a time. UUID v7 works in Spark in two complementary ways: as a primary key column with partition-aware generation and as a job/stage correlation ID that embeds timing information. See UUID v7 in Kafka Streams for the streaming pipeline context.
PySpark UDF for UUID v7
from pyspark.sql import SparkSession
from pyspark.sql.functions import expr, udf
from pyspark.sql.types import StringType
# Python 3.12+ has uuid.uuid7(); older versions use a library
try:
from uuid import uuid7
except ImportError:
from uuid_extensions import uuid7 # pip install uuid-extensions
spark = SparkSession.builder.appName("uuid-v7-demo").getOrCreate()
# Register UUID v7 UDF
@udf(StringType())
def uuidv7_udf():
return str(uuid7())
spark.udf.register("uuidv7", uuidv7_udf)
# Add UUID v7 primary key to any DataFrame
df = spark.range(1000).toDF("seq")
df_with_id = df.withColumn("id", expr("uuidv7()"))
df_with_id.show(5, truncate=False)
# +-----+------------------------------------+
# | seq|id |
# +-----+------------------------------------+
# | 0|018fbe3a-4c5d-7b12-8f3e-1a2b3c4d5e6f|
Scala/Java UDF for UUID v7
import org.apache.spark.sql.functions.{col, udf}
import com.github.f4b6a3.uuid.UuidCreator
// Register UUID v7 UDF
val uuidv7UDF = udf(() => UuidCreator.getTimeOrderedEpoch.toString)
spark.udf.register("uuidv7", uuidv7UDF)
// Apply to DataFrame
val dfWithId = df.withColumn("id", uuidv7UDF())
// Or use SQL
spark.sql("""
SELECT uuidv7() AS id, customer_id, total
FROM orders_staging
""")
Time-range filtering with UUID v7 partition pruning
def uuid_v7_at_ms(ms: int) -> str:
"""Build a UUID v7 lower-bound string for a specific millisecond."""
hex_ms = format(ms, '012x')
return f"{hex_ms[:8]}-{hex_ms[8:12]}-7000-8000-000000000000"
# Filter orders from the last 7 days using UUID v7 prefix
import time
seven_days_ago_ms = int(time.time() * 1000) - 7 * 24 * 3600 * 1000
lower_bound = uuid_v7_at_ms(seven_days_ago_ms)
# Spark pushes this filter down to Parquet row group statistics
recent_orders = spark.sql(f"""
SELECT id, customer_id, total
FROM orders
WHERE id >= '{lower_bound}'
ORDER BY id
""")
See UUID v7 for Cursor Pagination for the time-range query pattern.
Delta Lake table with UUID v7
# Write Delta Lake table with UUID v7 primary key
from delta.tables import DeltaTable
from pyspark.sql.functions import expr
orders_df = spark.sql("""
SELECT
uuidv7() AS id,
customer_id,
total,
status
FROM orders_staging
""")
orders_df.write \
.format("delta") \
.mode("append") \
.partitionBy("status") \
.save("/delta/orders")
# Merge (upsert) using UUID v7 as the merge key
delta_table = DeltaTable.forPath(spark, "/delta/orders")
delta_table.alias("target") \
.merge(orders_df.alias("source"), "target.id = source.id") \
.whenMatchedUpdateAll() \
.whenNotMatchedInsertAll() \
.execute()
Job correlation ID with UUID v7
import uuid
# Generate UUID v7 correlation ID at job start
job_id = str(uuid7())
spark = SparkSession.builder \
.appName("order-processing") \
.config("spark.app.correlation_id", job_id) \
.getOrCreate()
# Log job start — UUID v7 encodes start time
spark.sparkContext.setLogLevel("INFO")
print(f"Job started: correlation_id={job_id}")
# Pass correlation ID through transformations
df = spark.read.parquet("/raw/orders") \
.withColumn("job_correlation_id", lit(job_id))
Schema evolution: migrating from UUID v4 to v7
# Backfill UUID v7 for existing UUID v4 records
from pyspark.sql.functions import when, col
df_migrated = spark.table("orders") \
.withColumn(
"id_v7",
when(col("id_v7").isNull(), uuidv7_udf()).otherwise(col("id_v7"))
)
# Write back in batches
df_migrated.write.format("delta").mode("overwrite").save("/delta/orders")
Further reading
- UUID v7 in Kafka Event Streaming
- UUID v7 in Distributed Systems
- UUID v7 in ClickHouse — Analytics Primary Keys
- UUID v7 for Cursor Pagination
- UUID in Databases — Storage and Indexing
External references
Frequently asked questions
How do I generate UUID v7 in a Spark DataFrame?
Register a Spark UDF using the JVM UuidCreator.getTimeOrderedEpoch().toString() or Python's uuid module. In PySpark: spark.udf.register("uuidv7", lambda: str(uuid.uuid7()), StringType()) then call df.withColumn("id", expr("uuidv7()")). Each executor generates its own UUID v7 values — the 74 random bits prevent collisions even when thousands of executors generate IDs in the same millisecond. See UUID v7 in Distributed Systems.
How does UUID v7 enable partition pruning in Spark?
UUID v7's 48-bit timestamp prefix enables time-range queries to prune partitions in Parquet and Delta Lake tables partitioned by date or hour. Build a UUID v7 lower-bound prefix for a time window using the millisecond timestamp and filter with WHERE id >= '018f...' AND id < '018g...'. Spark's predicate pushdown passes this filter to Parquet readers, scanning only relevant row groups.
Should I use UUID v7 or UUID v4 as a Spark job correlation ID?
UUID v7 is preferred for Spark job correlation IDs because the timestamp prefix tells you exactly when a job or stage started from the ID alone. Use UUID v7 as the application ID passed to SparkConf.set("spark.app.id", correlationId) and log it at stage start. UUID v7 eliminates the need to query the Spark history server to find a job's start time — it is embedded in the ID itself.