mirror of
https://github.com/vladmandic/automatic
synced 2026-09-18 16:54:33 +02:00
fix(lora): scale the dora diff before the decompose norm
finalize_updown ran apply_weight_decompose on the unscaled delta and multiplied the result by alpha/rank afterward. LyCORIS and ComfyUI both bake alpha/rank into the diff before computing the row norms, so any DoRA with alpha != rank renormalized against the wrong merged weight (64% relative delta error for kohya-style alpha=1 rank=8; exact only when alpha == rank, which full-matrix LoKR forces). - scale updown by calc_scale() before apply_weight_decompose; apply only the multiplier afterward - multiplier lerps the full merged delta (0 disables, 1 equals the trainer output); LyCORIS weight-mode ratio interpolation leaves the diff applied at multiplier 0 and is not used - add a numeric regression test mirroring the LyCORIS forward reference
This commit is contained in:
@@ -268,7 +268,13 @@ class NetworkModule:
|
||||
if ex_bias is not None:
|
||||
ex_bias = ex_bias * self.multiplier()
|
||||
if self.dora_scale is not None:
|
||||
updown = self.apply_weight_decompose(updown, orig_weight)
|
||||
# LyCORIS/ComfyUI convention: alpha/rank is baked into the diff
|
||||
# before the decompose norm. The multiplier then lerps the full
|
||||
# merged delta (ComfyUI semantics: 0 disables, 1 equals the
|
||||
# trainer's output; LyCORIS weight-mode ratio interpolation is
|
||||
# not used since it leaves the diff applied at multiplier 0).
|
||||
updown = self.apply_weight_decompose(updown * self.calc_scale(), orig_weight)
|
||||
return updown * self.multiplier(), ex_bias
|
||||
return updown * self.calc_scale() * self.multiplier(), ex_bias
|
||||
|
||||
def calc_updown(self, target):
|
||||
|
||||
Reference in New Issue
Block a user