MonoLR_rus_Cyrl_PR

MonoLR_rus_Cyrl_PR

This is the Russian (rus_Cyrl) Monolingual LoRA adapter from Semantic Evaluation of Multilingual Data-to-Text Generation via NLI Fine-Tuning: Precision, Recall and F1 scores used to compute Semantic Precision and Semantic Recall scores for RDF-to-Text generation.

Use

The following is minimal code to compute the Semantic Precision, Semantic Recall, and Semantic F1 of a generated Russian text:

from sentence_transformers import CrossEncoder

model = CrossEncoder('MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7')
model.config.num_labels = 1
model.default_activation_function = torch.nn.Sigmoid()

model.model.load_adapter('WilliamSotoM/MonoLR_rus_Cyrl_PR',)

graph = '[S]Buzz_Aldrin[P]mission[O]Apollo_12[T][S]Buzz_Aldrin[P]birthPlace[O]Glen_Ridge,_New_Jersey'
text = Базз Олдрин был членом экипажа Аполлона-12.'

precision = model.predict([(graph, text)])[0]
recall = model.predict([(text, graph)])[0]

f1 = (2*precision*recall)/(precision+recall)

print(f'Precision: {precision:.4f}')
print(f'Recall: {recall:.4f}')
print(f'F1: {f1:.4f}')

Expected outpu:

Precision: 0.9984
Recall: 0.3765
F1: 0.5468

Analysis: High precision means all the content in the text comes from the graph (i.e. No additions / hallucinations). Half recall means half the conente from the graph is missing (i.e. Some omissions).