fasttext_wiki_bn_100d_16m

<b><span style="color:#f39c12 ; font-size:1.5em"> Bangla FastText Model (16 million tokens)<b>

<b style="color:#2c3e50">I am Uzzal Mondal (<a href='https://www.linkedin.com/in/uzzal-mondal/' target='_blank' >LinkedIn</a>)</b>, <span style="font-weight:normal"> I have also built a FastText model specifically for Bangla NLP tasks. This repository contains a FastText model trained on Bangla Wikipedia data. This model can be used for various NLP tasks, including word similarity, word embeddings, and semantic analysis in Bangla text. Check the accompanying Python script for practical examples.</span>

<ul style="font-size:1em; font-weight:normal; font-family:monospace"> <li>**Tokens Processed**: The model has read and processed 16 million tokens from the training corpus.</li> <li>**Vocabulary Size**: 120,332 unique words</li> <li>**Training Loss**: Average loss during training was 0.552678</li> <li>**Embedding Dimension**: 100</li> <li>**Training Configuration**: Epochs = 10 </li> </ul>

How to Use the Model

Load the Model in Your Code

Use the following code to download and use the model:

from huggingface_hub import hf_hub_download
import fasttext

# Download the model
model_path = hf_hub_download(repo_id="uzzalmondal/fasttext_wiki_bn_100d_16m", filename="fasttext_bn_wiki_100.bin")

# Load the model
model = fasttext.load_model(model_path)

# Perform a word analogy task: "King - Man + Woman = Queen"
analogy_result = model.get_nearest_neighbors('রাজা', k=10)

print("Analogy result (King - Man + Woman):")
for word, similarity in analogy_result:
    print(f"{word}: {similarity}")

Output

Analogy result (King - Man + Woman):
0.8683507442474365: রাজার
0.8025670051574707: রাজায়
0.7848780751228333: রাজপুত্র
0.7837258577346802: রাজারাও
0.7768903374671936: সামন্তরাজা
0.7766559720039368: রাজসিংহাসনে
0.7681295275688171: রাজত্বের
0.7603954672813416: রাজপুত্রদের
0.7589437365531921: রাজাদের
0.7575206756591797: রাজপুত্রের

<b><span style="color:#f39c12 ; font-size:1em">🔧 How Can You Contribute? </b>

<ul style="font-size:1em; font-weight:normal; font-family:monospace"> <li>Suggest improvements or new features</li> <li>Report any issues or bugs</li> <li>Contribute to the codebase or documentation</li> <li>Share your use cases or experiments</li> </ul>

<b><span style="color:#f39c12 ; font-size:1em; font-family:maroon"> 💬 Your feedback helps us: </b>

<ul style="font-size:1em; font-weight:normal; font-family:monospace"> <li>Make Bangla NLP tools more accessible</li> <li>Improve model performance</li> <li>Extend the model’s capabilities to more applications</li> </ul>