llamamerge

This document outlines the process of merging two pre-trained models, psmathur/orca_mini_v3_13b and garage-bAInd/Platypus2-13B, using Spherical Linear Interpolation (SLERP). The base model for the merge is psmathur/orca_mini_v3_13b, and the weights are processed in float16 format to optimize memory usage. SLERP ensures a smooth blending of the model weights, allowing the merged model to benefit from the strengths of both original models.

The merging parameters include specific interpolation values for different components: [0.0, 0.5, 0.3, 0.7, 1.0] for the self_attn layers and [1.0, 0.5, 0.7, 0.3, 0.0] for the mlp layers. For all other layers, a default value of 0.5 is applied. The layer slices from both models are merged within the range of layers 0-40.

To replicate this process, the merging script is run using the configuration provided in the gradient-slerp.yml file. The merged model will be saved in the designated output directory. This approach ensures the new model combines the unique capabilities of both input models while maintaining balanced performance.