Mephisto-4B-0725

Mephisto-4B-0725

This model is currently being improved.

This is a merge of pre-trained language models created using mergekit fork. This is a merge of pre-trained language models created using mergekit. The advanced conversational EQ exhibited by Agents-A1-4B—relative to its parameter count—does not stem from a single component but rather emerges from a dense distributed representation inherent in the entire model, spanning its multiple layers; thus, Agents-A1-4B's conversational ability is encoded diffusely rather than locally. However, since the "tool-calling capability" required for agentic behavior—which Agents-A1-4B lacks—is instead localized within specific attention heads or sparse FFN layers, it is possible to merge task vectors from a model with a more localized representation of this skill, derived from Qwen3.5-4B. Furthermore, the exclusivity of task-specific regions is supported by the findings of Panigrahi et al. ("Task-Specific Skill Localization in Fine-tuned Language Models," 2023), who show that a small, identifiable subset of parameters can account for the majority of a fine-tuned model's task performance. To endow Agents-A1-4B with tool-calling capabilities, tool-calling models such as Tmax-4B and enfuse/smol-tools-4b-32k are merged using the DARE-TIES method. To preserve Agents-A1-4B's original conversational EQ—which, being diffusely distributed, is more vulnerable to disruption by naive merging—a high drop rate is applied to extract only the sharpest, most task-relevant parameters. All the aforementioned models share the same prompt format and base model. All the aforementioned models share the same prompt format and base model.

Merge Details

Merge Method

This model was merged using the DARE TIES merge method using Qwen/Qwen3.5-4B as a base.

Models Merged

The following models were included in the merge:

• allenai/tmax-4b
• InternScience/Agents-A1-4B

Configuration

The following YAML configuration was used to produce this model:

merge_method: dare_ties
base_model: Qwen/Qwen3.5-4B

models:
    - model: InternScience/Agents-A1-4B
    parameters:
      weight: 1.0
      density: 1.0

  - model: allenai/tmax-4b
    parameters:
      weight: 0.7
      density: 0.3

parameters:
  int8_mask: true
  normalize: false      

dtype: bfloat16

Benchmarks