Databricks 中基于图形的可观察性数据分析与凭证售卖.pdf

1、Graph-Powered Observability Data Analysis in Databricks with Credential VendingXinyu Liu,Danfeng Xu,Eric Sun2025-06-10AgendaObservability DataOptimize Data Layout of Huge volume of Distributed TracesForensic and Analytical Use Cases:Service Graph and Root Cause AnalysisPuppyGraphGraph analytics on t

2、op of Open Table Formats without ETLCredential VendingQuery+VisualizationFuture Innovation2DataObservability DataDistributed TracesStored in DatabricksPartitioned by Day or HourSub-partitioned by prefix(trace_id)Z-Order by(trace_id,span_id)Volume(per day)1826B events 16002900 GiB10001700 filesRetent

3、ion6+months instead of 1 week3OverviewGatewayprotobuf payloadDataDogAgentDual PublishDistributed Traces4Sample Data from Delta LakeSpan 1Span 3Span 2Span 4Span 5Trace 69101266Graph-Powered AnalysisService Topology GraphServicesTrace spansAugment/Overlay entity labels and metrics(failure,latency,thro

4、ughput)5How can we quickly and easily understand the causes and effects of service failures?PuppyGraphSupporting Credential VendingSupporting graph queries via Cypher and Gremlin6Query your Databricks data as a graph w/o ETLing data to another GraphDBCredential VendingManaged by Unity CatalogCredent

5、ialDownscopedTemporary“Lazy”token renew Safe and integrated approach7How does PuppyGraph access data?12345Credential Vending Request to Unity CatalogPOST/temporary-table-credentials Response for S3 AccessAccess Key/Secret Key pairSession TokenShort-lifeS3 Path POST/api/2.1/unity-catalog/temporary-ta

6、ble-credentials operation:READ,table_id:f84c88a7-7b14-4c32-90b3-8d45e1e53f5e 200 OK aws_temp_credentials:access_key_id:AKIAIOSFODNN7EXAMPLE,secret_access_key:wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY,session_token:IQoJb3JpZ2luX2VjEHsaDnVzLWVhc3QtMS1maXBzIkYwRAI