srtk-scorer

SRTK Scorer

This model is a trained scorer for SRTK. It is used to compare the similarity between a query and the expansion path at the time of subgraph retrieval.

Training Information

It is initialized with roberta-base. It is trained jointly on the following datasets:

• WebQSP for Freebase
• SimpleQuestionsWikidata for Wikidata
• SimpleDBpediaQA

It achieves an answer coverage rate of 0.9728 on SimpleQuestionsWikidata (depth 1) 0.8501 on WebQSP test set (depth 2) with a beam width of only 2!

Usage Example

First install the package:

pip install srtk

Then you can retrieve subgraphs with the help of this scorer:

srtk retrieve -i data/wikidata-simplequestions/intermediate/scores_test.jsonl \
    -o artifacts/subgraphs/wikidata-simple-contrast \
    -e http://localhost:1234/api/endpoint/sparql \
    --scorer-model-path drt/srtk-scorer \
    --scorer  --beam-width 2 --max-depth 1 --evaluate

Limitations

As both SimpleQuestionsWikidata and SimpleDBpediaQA contain only one-hop relations, the model tends to stop at one-hop when you retrieve subgraphs on Wikidata and DBpedia. We will release a updated version of the model that is trained on a more diverse dataset in the future.

License

MIT