scholawrite-llama3.1-8b-writing

Model Card for ScholaWrite-Llama3.1-8B-Writing

<!-- Provide a quick summary of what the model is/does. -->

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

This model is refered as LLAMA-8B-SW-GEN in the paper. It is fined-tuned based on 4 bit quantized Llama-3.1-8B-Instruct from unsloth Hugging Face Hub, using train split of ScholaWrite dataset. The sole purpose of this model is to take the role of after text generation in the Iterative Self-Writing task.

• Developed by: *Linghe Wang, *Minhwa Lee, Ross Volkov, Luan Chau, Dongyeop Kang
• Language: English
• Finetuned from model: Meta-Llama-3.1-8B-Instruct-bnb-4bit

Model Sources [optional]

<!-- Provide the basic links for the model. -->

• Repository: ScholaWrite Github Repository
• Paper: [More Information Needed]
• Demo: https://minnesotanlp.github.io/scholawrite/

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

The model is intended to used for "after-text" generation in Iterative Self-Writing task.

Iterative Self-Writing task is iteratively generating scholarly text from scratch, mirroring the human writing process. This task focuses on how well the model trained on our dataset can replicate the actual iterative writing and thinking process of scholars, thus produce the better scholarly text output than model without trained on our dataset.

Iterative self-writing involves two subtasks (1) Next intention prediction. Model will take input prompt with task instructions, and the "before text". The model's task is to generate next writing intention based on the ''before'' text. (2) "after-text" generation. Model will take input prompt with task instructions, a verbalizer derived from human-annotated labels, and the "before text". The model's task is to generate ''after-text'' given the verbalizer and ''before'' text.

Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

The model is fine-tuned only for "after-text" generation and infereneced in closed enviroment. Its main goal is to examine the usefullness of our dataset. It is suitable for acdamic use, but not suitable for production, general public use, or consumer-oriented service. In addition, use this model on tasks besides "after-text" generation in LaTex acdamic draft may not work well.

Bias and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

The bias and limitations of this model mainly came from the dataset (<span style="font-variant: small-caps;">ScholaWrite</span>) it fine-tuned on.

First, the <span style="font-variant: small-caps;">ScholaWrite</span> dataset is currently limited to the computer science domain, as LaTeX is predominantly used in computer science journals and conferences. This domain-specific focus in dataset may restrict the model's generalizability to other scientific disciplines. Future work could address this limitation by collecting keystroke data from a broader range of fields with diverse writing conven554 tions and tools, such as the humanities or biological sciences. For example, students in humanities usu556 ally write book-length papers and integrate more sources, so it could affect cognitive complexities.

Second, all participants were early-career researchers (e.g., PhD students) at an R1 university in the United States, which means the models may not learn the professional writing behavior and cognitive process from expert. Expanding the dataset to include senior researchers, such as post-doctoral fellows and professors, could offer valuable insights into how writing strategies and revision behaviors evolve with research experience and expertise.

Third, the dataset is exclusive to English-language writing, which restricts model's capability to iteratively write paper in multilingual or non-English contexts. Expanding to multilingual settings could reveal unique cognitive and linguistic insights into writing across languages.

How to Get Started with the Model

import os
from unsloth import FastLanguageModel
from dotenv import load_dotenv
from huggingface_hub import login
load_dotenv()
login(os.getenv("HUGGINGFACE_TOKEN"))
model_name = "minnesotanlp/scholawrite-llama3.1-8b-writing"
text = '''
list in following format: 
[
    {"role": "user", "content": your prompt that contain instruction, verbalizer, and before text}
]
'''
before_text = "your before text"
model, tokenizer = FastLanguageModel.from_pretrained(
model_name=model_name,
max_seq_length=4096,
load_in_4bit=True,
dtype=None,
)
FastLanguageModel.for_inference(model)
input_ids = tokenizer.apply_chat_template(text, max_length=4096, tokenize=True, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(input_ids, max_new_tokens=len(before_text)+100, do_sample=True, top_k=50, top_p=0.95)
response = tokenizer.batch_decode(outputs)
response = response[0].split("<|start_header_id|>assistant<|end_header_id|>")[1].strip()
response = response.replace("<|eot_id|>", "")

fine-tuning Details

fine-tuning Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the fine-tuning data is all about as well as documentation related to data pre-processing or additional filtering. -->

This model is fine-tuned on minnesotanlp/scholawrite dataset train split. It is keystroke logs of an end-to-end scholarly writing process, with thorough annotations of cognitive writing intentions behind each keystroke. No additional data pre-processing or filtering performed on the dataset.

fine-tuning Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the fine-tuning procedure. -->

The dataset contains before text, intention, and after text. For each entry in the dataset, we need to setup the prompt that is ready for fine-tuning. To do so, the intention will be first converted to corresponding verbalizer. Then, before test, verbalizer, and after text will be put into the predefined prompt template. We mask out the system and user message of the prompt with -100, so that the model is trained on responsed only.

fine-tuning Hyperparameters

• fine-tuning regime: QLoRA <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
• max_seq_length 5096
• learning_rate 3e-4
• lr_scheduler_type linear
• per_device_train_batch_size 1
• gradient_accumulation_steps 4
• num_train_epochs 1
• fp16 False
• bf16 True
• logging_steps 10
• optim adamw_8bit
• weight_decay 0.01
• warmup_steps 10
• seed 0

Machine Specs

<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->

• Hardware: Nvidia L40s GPU
• Software: Unsloth
• Hours used: 12 hrs
• Compute Region: Minnesota

Testing Procedure

Testing Data

<!-- This should link to a Dataset Card if possible. -->

Instead of running test on dataset, we performed Iterative Self-Writing task, see section Direct Use for task detail. We pick 4 seed documents as starting point of Iterative Self-Writing task, derived from 4 award-winning NLP papers spanning different topics. They are Zeng et al., 2024; Lu et al., 2024b; Du et al., 2022a; Etxaniz et al., 2024

Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

We came up these three metrics for auto evaluation:

1. lexical diversity: the ratio of unique to total tokens in the final iteration
2. topic consistency: cosine similarity between the seed document and the final output
3. intention coverage: diversity of writing intentions as a proportion of unique labels used across 100 iterations among the 15 available labels in our taxonomy.

Furthermore, inspired by chang2023 et al (2023), we conducted a human evaluationfor more detailed descriptions of the entire evaluation process. With three native English speakers experienced in Overleaf. They assessed the outputs based on following metrics:

1. Accuracy: alignment with the predicted intention
2. Alignment: how closely the model’s process resembled human writing style
3. fluency: grammatical correctness of final writing
4. coherence: logical structure
5. relevance: connection to the seed paper's contents.

Accuracy is evaluated for each iteration, while alignment, fluency, and coherence were assessed through pairwise comparisons on final iteration.

Results

Note:

• Llama-8b-sw and Finetuned in the tables below refers to combination of our two models where scholawrite-llama3.1-8b-classifier responsible for next writing intention prediction, and scholawrite-llama3.1-8b-writing (this model) responsible for after text generation.
• Llama-3b-instruct and Baseline in the tables below refers to combination of two Meta-Llama-3.1-8B-Instruct-bnb-4bit running next writing intention prediction and after text generation.

Auto Evaluation Results for Seed 1

Auto Evaluation Results for Seed 2

Auto Evaluation Results for Seed 3

Auto Evaluation Results for Seed 4

---

Human Evaluation Results for Seed 1

Human Evaluation Results for Seed 2

Human Evaluation Results for Seed 3

Human Evaluation Results for Seed 4

Summary

Auto Evaluation Results tables illustrates the quality of the final writing output produced by each model across all four seed documents. Notably, our models (scholawrite-llama3.1-8b-writing and scholawrite-llama3.1-8b-classifier) consistently used the most lexically diverse words in their final outputs. Moreover, it generated content that was semantically most aligned with the seed 1 and seed 2. It also covered the highest number of writing intentions based on our taxonomy for all seeds except Seed 3. These results underscore the effectiveness of ScholaWrite as a valuable resource for enhancing the quality of scholarly writing generated by language models.

Despite their remarkable performance based on automatic evaluation metrics, LLMs still exhibit limitations in learning human writing behaviors and scholarly thinking processes. According to our human evaluation our model generated fewer instances of after text that aligned with the predicted intentions from the previous step during 100 iterations across all four seed documents. Furthermore, all three evaluators unanimously agreed that the baseline model, demonstrated more human-like writing behaviors throughout the iterations. Its final outputs were also perceived as more grammatically correct and containing stronger logical claims compared to our models.

However, the evaluators also noted that the final outputs from our models contained more relevant content for Seeds 2 and 3. This observation aligns with the trend in topic consistency scores shown in Auto Evaluation Results for Seed 2 and seed 3, further highlighting the usefulness of ScholaWrite dataset in certain contexts.

BibTeX

@misc{wang2025scholawritedatasetendtoendscholarly,
      title={ScholaWrite: A Dataset of End-to-End Scholarly Writing Process},
      author={Linghe Wang and Minhwa Lee and Ross Volkov and Luan Tuyen Chau and Dongyeop Kang},
      year={2025},
      eprint={2502.02904},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.02904},
      }