Cthulhu-Yamls-All-Versions

These did not have a seed # assigned so produces different safetensors if you try to remerge

Future versions with dare_ties are to use --random-seed 420

Update: The same logic also applies to della, you must assign a --random-seed # if you want the merge to be fully deterministic

---

The della merge method employs probabilistic dropout that requires a random seed for reproducibility. The DELLA method uses torch.bernoulli() to randomly sample which weights to keep based on magnitude-dependent probabilities.

Specifically, at the core of DELLA's della_magprune function, it calculates drop probabilities and then uses torch.bernoulli(probs) to randomly decide which parameters to drop.

Why You Get Consistent Results on Your PC

Each time you run the merge on your PC without setting a seed, PyTorch initializes its random number generator to a consistent but machine-specific state. This explains why:

• ✅ Multiple runs on your PC produce identical results
• ❌ Your results never match the original creator's results
• ❌ Results differ between PCs

The Solution: Use --random-seed

MergeKit provides a random_seed option specifically for this purpose.

When you set a random seed, it calls transformers.trainer_utils.set_seed() to ensure reproducible behavior across PyTorch, NumPy, and Python's random module.

To get reproducible merges, run:

mergekit-yaml config.yaml ./output --random-seed 420

Both you and the original creator must use the same seed value to get identical results.

While both use randomness, DELLA's documentation may not have emphasized this as clearly. The key difference is that DARE methods use SparsificationMethod.random (pure Bernoulli sampling), while DELLA uses SparsificationMethod.della_magprune (magnitude-weighted Bernoulli sampling) - but both are random and require seeds for reproducibility.

mergekit/sparsify.py

def della_magprune(
    tensor: torch.Tensor,
    density: float,
    epsilon: float,
    rescale_norm: Optional[RescaleNorm] = None,
) -> torch.Tensor:
    if density >= 1:
        return tensor
    if density <= 0:
        return torch.zeros_like(tensor)
    orig_shape = tensor.shape

    if density + epsilon >= 1 or density - epsilon <= 0:
        raise ValueError(
            "Epsilon must be chosen such that density +/- epsilon is in (0, 1)"
        )

    work_dtype = (
        tensor.dtype
        if tensor.device.type != "cpu" or tensor.dtype == torch.bfloat16
        else torch.float32
    )

    if len(tensor.shape) < 2:
        tensor = tensor.unsqueeze(0)
    magnitudes = tensor.abs()

    sorted_indices = torch.argsort(magnitudes, dim=1, descending=False)
    ranks = sorted_indices.argsort(dim=1).to(work_dtype) + 1

    min_ranks = ranks.min(dim=1, keepdim=True).values
    max_ranks = ranks.max(dim=1, keepdim=True).values
    rank_norm = ((ranks - min_ranks) / (max_ranks - min_ranks)).clamp(0, 1)
    probs = (density - epsilon) + rank_norm * 2 * epsilon
    mask = torch.bernoulli(probs).to(work_dtype)

    res = rescaled_masked_tensor(tensor.to(work_dtype), mask, rescale_norm)
    return res.to(tensor.dtype).reshape(orig_shape)