人工智能与非结构化数据.pdf

1、From Air Quality to Aircraft&Automobiles,Unstructured Data Is EverywhereTim Spann,Senior Solutions EngineerTim Spannpaasdev.bsky.social PaasDev /Blog:datainmotion.devSenior Solutions Engineer,Snowflake NY/NJ/Philly-Cloud Data+AI Meetupsex-Zilliz,ex-Pivotal,ex-Cloudera,ex-HPE,ex-StreamNative,ex-Horto

2、nworks.https:/ https:/ This week in Snowflake,Apache NiFi,Apache Flink,Apache Kafka,ML,AI,Streamlit,Jupyter,Apache Iceberg,Apache Polaris,Python,Java,LLM,GenAI,Vectors and Open Source friends.https:/bit.ly/32dAJftAI+Streaming Weekly by Tim SpannAGENDAIntroductionOverviewAIWhere,What,WhyReal-Time AI

3、Open Lakehouse 5DATA SOURCESDATA INTEGRATIONDATA PLATFORMDATA CONSUMERS Transit EventsTransit DataTraffic DataSNOWSIGHTRaw DataI Can Haz I Can Haz Data?Data?DocsUnstructuredSemi-structuredStructuredNYC DataCSVXMLXLSAWS S3BucketIoTSnowflake Cortex AIStructured,Structured,Semistructured,Semistructured

4、,UnstructuredUnstructuredDataDataWhen you think of RAG,you think of unstructured data like documents or giant chunks of text.Its more.Unstructured DataUnstructured Data Lots of formats Text,Documents,PDF Images,Videos,Audio Email,Slack,Teams Logs Binary Data Formats Zip,Archives VariantsUnstructured

5、 Open Data like Open AQ-Air Quality Data Location,Time,Sensors Apache Avro,Parquet,Orc JSON and XML Hierarchical Data Logs Key-ValueSemi-Structured DataSemi-Structured Datahttps:/ Semi-structuredStructured DataStructured Data Snowflake Tables Snowflake Hybrid Tables Apache Iceberg Tables Relational

6、Tables Postgresql Tables CSV,TSVStructuredRecord-Oriented Data with NiFiRecord-Oriented Data with NiFiReaders-Avro,CEF,CSV,Excel,Grok,Protobuf,JSON,Parquet,Scripted,Syslog-5424,Syslog,Windows Event,XML,YAMLWriters-Avro,CSV,Free From Text,JSON,Parquet,Scripted,XMLSchema registry integration for retri