4249 - 利用数据准备工具包加速LLM的数据工程.pdf

1、Session ID:4249Shalisha Witherspoon IBM ResearchAccelerating LLM Development with the Data Prep KitIBM TechXchange 2025Open-source Python toolkitlicensed under Apache 2.0,designed for efficient data preparation in LLM workflowsProvides modular transforms and recipes for building and fine-tuning LLMs

2、,including support for RAG and instruct-tuning use casesBuilt for scalable computing environments from local laptops to distributed clusters using tools like Spark and KubeflowSupportscode and language datasets,enabling preprocessing for multilingual corpora,source code repositories,and domain-speci

3、fic documentsIncludesversatile data connectors for local files,cloud storage(e.g.,S3,IBM COS),and custom connectors,allowing seamless integration with your own data sourcesIntroducing DPKNotebook Tutorial:RAG Workflow Recipeand transformationHow to ContributeOpen to all contributors whether youre fi

4、xing bugs,improving docs,or adding new transformsStart with issues labeledgood first issueto get familiar with the codebaseFollow the standard GitHub workflow:fork clone branch commit pull requestEnsure your code adheres to the projects style and testing guidelinesContributions to documentation,exam

5、ples,and tutorials are equally valuableJoin the community by engaging in discussions,reviews,and feedback loopsNotices and disclaimersIBM,the IBM logo,and are trademarks of International Business Machines Corporation,registered in many jurisdictions worldwide.Other product and service names might be

6、 trademarks of IBM or other companies.A current list of IBM trademarks is available on the Web at“Copyright and trademark information”at: comments made in this presentation may be characterized as forward looking under the Private Securities Litigation Reform Act of 1995.Forward-