澶辨晥閾炬帴澶勭悊 |
澶фā鍨嬶紙LLMs錛夊井璋冮潰 PDF 涓嬭澆
鐩稿叧鎴浘錛?/strong>
![]() 涓昏鍐呭錛?/strong>
1. 濡傛灉鎯寵鍦ㄦ煇涓ā鍨嬪熀紜€涓婂仛鍏ㄥ弬鏁板井璋冿紝絀剁珶闇€瑕佸灝戞樉
瀛橈紵
涓€鑸?/span> n B鐨勬ā鍨嬶紝鏈€浣庨渶瑕?/span> 16-20 n G鐨勬樉瀛樸€傦紙cpu offload鍩烘湰涓嶅紑鐨勬儏鍐典笅錛?/span>
vicuna-7B涓轟緥錛屽畼鏂規(guī)牱渚嬮厤緗負 4*A100 40G錛屾祴璇曚簡涓€涓嬬‘瀹炶兘鍗犳弧鏄懼瓨銆傦紙global batch size
128錛?/span>max length 2048錛夊綋鐒惰緇冩椂鐢ㄤ簡FSDP銆佹搴︾瘡縐€佹搴︽鏌ョ偣絳夋柟寮忛檷鏄懼瓨銆?/span>
2. 涓轟粈涔?/strong>SFT涔嬪悗鎰熻LLM鍌諱簡?
• 鍘熺増絳旀錛?/span>
SFT鐨勯噸鐐瑰湪浜庢縺鍙戝ぇ妯″瀷鐨勮兘鍔涳紝SFT鐨勬暟鎹噺涓€鑸篃灝辨槸涓囨伓涔嬫簮alpaca鏁版嵁闆嗙殑52k閲忕駭錛?/span>
鐩告瘮浜庨璁粌鐨勬暟鎹繕鏄お灝戜簡銆?/span>
濡傛灉鎶辯潃鐏屾敞棰嗗煙鐭ヨ瘑鑰屼笉鏄縺鍙戣兘鍔涚殑鎯蟲硶錛屽幓鍋?/strong>SFT鐨勮瘽錛屽彲鑳界‘瀹炲鏄撴妸LLM寮勫偦銆?/span>
• 鏂扮増絳旀錛?/span>
鎸囦護寰皟鏄負浜嗗寮猴紙鎴栬В閿侊級澶ц璦€妯″瀷鐨勮兘鍔涖€?/strong>
鍏剁湡姝d綔鐢細
鎸囦護寰皟鍚庯紝澶ц璦€妯″瀷灞曠幇鍑烘硾鍖栧埌鏈榪囦換鍔$殑鍗撹秺鑳藉姏錛屽嵆浣垮湪澶氳璦€鍦烘櫙涓嬩篃鑳芥湁涓嶉敊琛?/span>
鐜?銆?/span>
|