Downloads · 30 days
35
5% of all-time downloads
fasttext-jp-embedding
paulhindemith/fasttext-jp-embedding
fasttext-jp-embedding is a feature extraction model from paulhindemith. Use it when you need embeddings to search or compare text. It is set up for transformers. The card lists the license as cc-by-sa-3.0.
All-time downloads
777
Public
Repo size
4.8 GB
Likes
3
Public
Repo makeup
Click a slice to open those files.
.bin2.4 GB · 99%
At a glance
- Task
- Feature Extraction
- Library
- transformers
- License
- cc-by-sa-3.0
- Model type
- fasttext_jp
- Access
- Public
- Created
- Nov 5, 2022
- Updated
- Nov 16, 2022
- SHA
- 48dd3bc8
Try a prompt
Model card
From the Hugging Face model README
fasttext-jp-embedding
This model is experimental.
Pretrained FastText word vector for Japanese
Usage
Google Colaboratory Example
! apt install aptitude swig > /dev/null
! aptitude install mecab libmecab-dev mecab-ipadic-utf8 git make curl xz-utils file -y > /dev/null
! pip install transformers torch mecab-python3 torchtyping > /dev/null
! ln -s /etc/mecabrc /usr/local/etc/mecabrc
from transformers import pipeline
import pandas as pd
import numpy as np
text = "海賊王におれはなる"
pipeline = pipeline("feature-extraction", model="paulhindemith/fasttext-jp-embedding", revision="2022.11.13", trust_remote_code=True)
pd.DataFrame(np.array(pipeline(text)).T, columns=pipeline.tokenizer.tokenize(text))
pipeline.tokenizer.target_hinshi = ["動詞", "名詞", "形容詞"]
pd.DataFrame(np.array(pipeline(text)).T, columns=pipeline.tokenizer.tokenize(text))
License
This model utilizes the folllowing pretrained vectors.
Name: fastText
Credit: https://fasttext.cc/
License: Creative Commons Attribution-Share-Alike License 3.0
Link: https://dl.fbaipublicfiles.com/fasttext/vectors-wiki/wiki.ja.vec
- Task
- Feature Extraction
- Library
- transformers
- Type
- fasttext_jp
- License
- cc-by-sa-3.0
- Languages
- ja
- Created
- Nov 5, 2022
- Updated
- Nov 16, 2022