Atlassian 多区域数据湖中的数据网格和合规性.pdf

1、Mike DiasPrincipal EngineerPaul AshleyPrincipal EngineerData Mesh and Compliance in a Multi-Regional Data Lake at AtlassianJZOur journey building a customer facing Data LakeAgendaOpportunities and ChallengesThe motivation behind the Customer Data LakeData ReplicationOur Data Mesh approach for CDC re

2、plicationStreaming ArchitectureRealtime analytics with Delta TablesMulti-Region Compliance Data Residency and GDPR at scalePutting all togetherDid we actually achieve our goals?Opportunitiesand Challenges Atlassian AnalyticsWith the advent of the Chartio acquisition,we wanted to create a seamless an

3、alytics experience for customers on top of their own dataData ExportsCustomers migrating from our on-prem solutions to our cloud struggled with the missing direct access to their databases to export their dataIn-Product AnalyticsWe want to get the data out of product databases in order to provide ef

4、ficient analytical experiences in-product without overloading the operational databasesML TrainingWe want to create Machine Learning models that better fits customer data to produce more acurate results,leading to a better experienceLets build a customer facing Data Lake!Data ReplicationMany Technol

5、ogiesMany DatabasesMany TeamsDynamic teams constantly changing due to reorgs,find the right people to talk about specific parts of the products can be challening.From sharded fleets of RDSs to DynamoDB and MongoDB.Data is stored in many different databases now and it will continue to evolve over tim

6、e.From monoliths to microservices and serverless.We have a wide rage of technologies and architectures in multiple languages.Replication ChallengesLogical Replication(Application instrumentation)Physical Replication(Database instrumentation)Technology availability(How much it depends on specific tec