澶辨晥閾炬帴澶勭悊 |
濡備綍浣跨敤 PEFT搴?涓?LoRA錛?PDF 涓嬭澆
杞澆鑷細(xì)http://www.python222.com/article/1199
鐩稿叧鎴浘錛?/strong>
![]() 涓昏鍐呭錛?/strong>
涓夈€佹ā鍨?鍔犲叆PEFT絳栫暐
3.1 妯″瀷鍔犺澆 絳栫暐鏈夊摢浜涳紵
妯″瀷鍔犺澆铏界劧寰堢畝鍗曪紝榪欓噷娑夊強(qiáng)鍒?/span>2涓椂闂存崲絀洪棿鐨勫ぇ妯″瀷鏄懼瓨鍘嬬緝鎶€宸?/strong>錛屼富瑕佽涓?/span>load_in_8bit鍜?/span>
prepare_model_for_int8_training銆?/span>
3.2 妯″瀷鏄懼瓨鍗犵敤鐨勯儴鍒嗘湁鍝簺錛?/strong>
榪欓噷闇€瑕佷粙緇嶄竴涓?涓や釜妯″瀷鏄懼瓨鍗犵敤鐨勯儴鍒嗭細(xì)
1. 闈欐€佹樉瀛樺熀鏈敱妯″瀷鍙傛暟閲忕駭鍐沖畾錛?/span>
2. 鍔ㄦ€佹樉瀛樺湪鍚戝墠浼犳挱鐨勮繃紼嬩腑姣忎釜鏍鋒湰鐨勬瘡涓緇忓厓閮戒細(xì)璁$畻嬋€媧誨€煎茍瀛樺偍錛岀敤浜庡悜鍚庝紶鎾椂鐨勬搴﹁綆楋紝
榪欓儴鍒嗗拰batchsize浠ュ強(qiáng)鍙傛暟閲忕駭鐩稿叧錛?/span>
3.3 妯″瀷鏄懼瓨鍗犵敤 浼樺寲絳栫暐錛?/strong>
妯″瀷鏄懼瓨鍗犵敤 鏈変互涓嬩袱縐嶆柟寮忥細(xì)
1. 8bit閲忓寲浼樺寲銆傝鏂瑰紡鍙鐢ㄤ簬浼樺寲 闈欐€佹樉瀛橈紱
2. 姊害媯€鏌ヤ紭鍖栥€傝鏂瑰紡鍙鐢ㄤ簬浼樺寲 鍔ㄦ€佹樉瀛橈紱
|