文件存储训练的相关内容

文章 2024-12-30 来自：开发者社区

并行文件存储在大模型训练中的探索与实践

由阿里云智能集团存储产品专家何邦剑先生分享并行文件存储在大模型训练中的探索与实践。一、大模型训练的io特点了解cps在大模型训练当中做了哪些创新或者实践。训练一个大模型需要考虑整个平台，考虑网络考虑存储。在计算方面，需要去看到规模不断扩大的情况下计算的故障率，包括计算架构的稳定性都是需要考虑的。对于网络，可能需要去考虑考虑网络的延时，...

文章 2022-02-17 来自：开发者社区

我们在协助某AI客户排查一个UFS文件存储的性能case时发现，其使用的Pytorch训练IO性能和硬件的IO能力有很大的差距（后面内容有具体性能对比数据）。让我们感到困惑的是：UFS文件存储，我们使用fio自测可以达到单实例最低10Gbps带宽、IOPS也可达到2w以上。该AI客户在高IOPS要求的AI单机小模型训练场景下，或者之前使用MXNet、TensorFlow框架时，IO都能跑到UFS....

共有2条

< 1 >

跳转至： GO

更新时间 2024-12-31 09:10:41

本页面内关键词为智能算法引擎基于机器学习所生成，如有任何问题，可在页面下方点击"联系我们"与我们沟通。

文件存储您可能感兴趣

产品推荐

{"optioninfo":{"dynamic":"ture","static":"true"},"simplifiedDisplay":"newEdition","newCard":[{"ifIcon":"img","link":"https://img.alicdn.com/tfs/TB1XY8hGYr1gK0jSZFDXXb9yVXa-1740-328.png","icon":"","iconImg":"https://img.alicdn.com/tfs/TB1lr2nqaNj0u4jSZFyXXXgMVXa-200-200.png","contentLink":"https://www.aliyun.com/product/dbfs","title":"数据库文件存储","des":"阿里云数据库文件存储DBFS基于共享存储架构，支持数据库定制和共享读写的线性扩展能力，提供低成本、高性能、高可用的存储服务，助力客户数据库业务无缝迁移上云。","link1":"https://dbfs.console.aliyun.com","btn1":"产品控制台","link2":"https://help.aliyun.com/product/140631.html","btn2":"产品文档","btn3":"","link3":"","infoGroup":[{"infoName":"产品入门","infoContent":{"firstContentName":"产品简介","lastContentName":"常见问题","firstContentLink":"https://help.aliyun.com/document_detail/149725.html","lastContentLink":"https://help.aliyun.com/document_detail/170769.html"}},{"infoName":"最佳实践","infoContent":{"firstContentName":"MySQL高可用实践","firstContentLink":"https://help.aliyun.com/document_detail/149749.html","lastContentLink":"https://help.aliyun.com/document_detail/402340.html","lastContentName":"Oracle RAC最佳实践"}},{"infoName":"最新动态","infoContent":{"firstContentLink":"https://www.aliyun.com/product/news/detail?id=21459","firstContentName":"企业版公测上线","lastContentLink":"","lastContentName":""}}]}],"card":[],"search":[],"infoCard":[],"title":{"mainTitle":"数据库文件存储","subtitle":"","linkUrl":"https://www.aliyun.com/product/dbfs","btnText":"查看详情"},"visual":{"topbg":"https://img.alicdn.com/imgextra/i2/O1CN01TG9LEu24KQXKz98ru_!!6000000007372-1-tps-3840-740.gif","icon":"","textColor":"dark"},"dataList":[],"sceneCard":[],"txt":[]}

{"$env":{"JSON":{}},"$page":{"env":"production"},"$context":{"optioninfo":{"dynamic":"ture","static":"true"},"simplifiedDisplay":"newEdition","newCard":[{"ifIcon":"img","link":"https://img.alicdn.com/tfs/TB1XY8hGYr1gK0jSZFDXXb9yVXa-1740-328.png","icon":"","iconImg":"https://img.alicdn.com/tfs/TB1lr2nqaNj0u4jSZFyXXXgMVXa-200-200.png","contentLink":"https://www.aliyun.com/product/dbfs","title":"数据库文件存储","des":"阿里云数据库文件存储DBFS基于共享存储架构，支持数据库定制和共享读写的线性扩展能力，提供低成本、高性能、高可用的存储服务，助力客户数据库业务无缝迁移上云。","link1":"https://dbfs.console.aliyun.com","btn1":"产品控制台","link2":"https://help.aliyun.com/product/140631.html","btn2":"产品文档","btn3":"","link3":"","infoGroup":[{"infoName":"产品入门","infoContent":{"firstContentName":"产品简介","lastContentName":"常见问题","firstContentLink":"https://help.aliyun.com/document_detail/149725.html","lastContentLink":"https://help.aliyun.com/document_detail/170769.html"}},{"infoName":"最佳实践","infoContent":{"firstContentName":"MySQL高可用实践","firstContentLink":"https://help.aliyun.com/document_detail/149749.html","lastContentLink":"https://help.aliyun.com/document_detail/402340.html","lastContentName":"Oracle RAC最佳实践"}},{"infoName":"最新动态","infoContent":{"firstContentLink":"https://www.aliyun.com/product/news/detail?id=21459","firstContentName":"企业版公测上线","lastContentLink":"","lastContentName":""}}]}],"card":[],"search":[],"infoCard":[],"title":{"mainTitle":"数据库文件存储","subtitle":"","linkUrl":"https://www.aliyun.com/product/dbfs","btnText":"查看详情"},"visual":{"topbg":"https://img.alicdn.com/imgextra/i2/O1CN01TG9LEu24KQXKz98ru_!!6000000007372-1-tps-3840-740.gif","icon":"","textColor":"dark"},"dataList":[],"sceneCard":[],"txt":[]}}