fix(lora): scale the dora diff before the decompose norm

finalize_updown ran apply_weight_decompose on the unscaled delta and
multiplied the result by alpha/rank afterward. LyCORIS and ComfyUI both
bake alpha/rank into the diff before computing the row norms, so any
DoRA with alpha != rank renormalized against the wrong merged weight
(64% relative delta error for kohya-style alpha=1 rank=8; exact only
when alpha == rank, which full-matrix LoKR forces).

- scale updown by calc_scale() before apply_weight_decompose; apply
  only the multiplier afterward
- multiplier lerps the full merged delta (0 disables, 1 equals the
  trainer output); LyCORIS weight-mode ratio interpolation leaves the
  diff applied at multiplier 0 and is not used
- add a numeric regression test mirroring the LyCORIS forward reference
This commit is contained in:
CalamitousFelicitousness
2026-07-12 23:38:30 +01:00
parent ca729a01ca
commit 387a349dfd
2 changed files with 47 additions and 1 deletions
+7 -1
View File
@@ -268,7 +268,13 @@ class NetworkModule:
if ex_bias is not None:
ex_bias = ex_bias * self.multiplier()
if self.dora_scale is not None:
updown = self.apply_weight_decompose(updown, orig_weight)
# LyCORIS/ComfyUI convention: alpha/rank is baked into the diff
# before the decompose norm. The multiplier then lerps the full
# merged delta (ComfyUI semantics: 0 disables, 1 equals the
# trainer's output; LyCORIS weight-mode ratio interpolation is
# not used since it leaves the diff applied at multiplier 0).
updown = self.apply_weight_decompose(updown * self.calc_scale(), orig_weight)
return updown * self.multiplier(), ex_bias
return updown * self.calc_scale() * self.multiplier(), ex_bias
def calc_updown(self, target):