neurosymbolic-llm

Neurosymbolic LLM Encoder/Decoder Models

This repository contains the encoder and decoder models from the paper linked above.

• Paper: https://huggingface.co/papers/2502.01657
• GitHub: https://github.com/vdhanraj/Neurosymbolic-LLM
• Dataset: https://github.com/vdhanraj/Symbolic-Math-Dataset

Model Files

• encoders_seed_42.pth: Pre-trained encoders for 3-digit data (x_gt_y set to true)
• decoders_seed_42.pth: Pre-trained decoders for 3-digit data (x_gt_y set to true)
• decoders_seed_42_finetuned.pth: Fine-tuned decoders (trained on 2025-05-19)

Note that the finetuned decoders only contain the decoder for layer 17, which was the layer on which the decoder was fine tuned. For other layers, please use the pre-trained decoders and redo the fine-tuning process

Usage

import torch
from llama.encoder_decoder_networks import Encoder, Decoder # These can be accessed from the Neurosymbolic LLM directory 
from huggingface_hub import hf_hub_download

# Download the model files
encoder_path = hf_hub_download(repo_id="vdhanraj/neurosymbolic-llm", filename="encoders_seed_42.pth")
decoder_path = hf_hub_download(repo_id="vdhanraj/neurosymbolic-llm", filename="decoders_seed_42_finetuned.pth")

# Load the models
encoders = torch.load(encoder_path, weights_only=False)
decoders = torch.load(decoder_path, weights_only=False)

# Example: Access individual encoder/decoder layers
first_encoder = encoders[0]
first_decoder = decoders[0]

To use the encoders and decoders, follow the code in fine_tune_decoders.py, which uses these encoder and decoder models to improve the performance of LLMs on a set of arithmetic tasks, as outlined in our paper.

Citation

If you use these models in your research, please cite:

@article{dhanraj2025nsllm,
 title={Improving Rule-based Reasoning in LLMs via Neurosymbolic Representations},
 author={Dhanraj, Varun and Eliasmith, Chris},
 journal={arXiv preprint arXiv:2502.01657},
 year={2025}
}