ThoughtVectors

Model Card for Model ID

A new way to create embeddings

Model Details

Model Description

Auto-regressivly generates thought vectors (embeddings) for an input. This means fewer elements for the core model to process and thereby less compute to use. Additionally it can decode a thought vector back into a (vaugly) similar meaning in text. It doesn't focus on exact wording, but rather capturing the full meaning of the input.

As of now this is a prototype, not ready for full use. It proves the concept works, runs really fast, and vaugly grasps some concepts in english.

• Developed by: nochinator
• Model type: embeddings
• Language(s) (NLP): English
• License: AGPL-3.0

Model Sources

• Repository: https://huggingface.co/nochiantor/ThoughtVectors
• Paper: Comming soon?
• Demo: Comming soon?

Uses

Specifically built for use in chatbots, but the embeddings should apply for any NLP system that doesn't rely on percise wordings (eg. classification) then sentence level embeddings.

Direct Use

Comming soon.

Bias, Risks, and Limitations

Adds a small amount of compute to the front (and if using decoder, back) of the overall system compared to other embedding mechanisms. However, if the core model is larger then it should end up saving compute.

How to Get Started with the Model

This "model" is actually a collection of models that don't work without each other (excepting SentencePiece). A library is included in the files to manage it for you.

Training Data

https://huggingface.co/datasets/sentence-transformers/stsb https://huggingface.co/datasets/stanfordnlp/snli/tree/main/plain_text

Training Procedure

Trained by taking a sentence, tokenizing, passing through an encoder to get thought vectors then passing the vectors through the decoder to back tokens and comparing with original tokens, then backpropagating the error through both encoder and decoder. Slightly punishes for longer sets of vectors to encurage fewer vectors.

Training Hyperparameters

group_data="train.csv", test_data="val.csv", num_epochs=1000, batch_size=256, accum_steps=1, learning_rate=1e-4, weight_decay=2e-5, length_penalty=0.001, single_vector_prob=0.1, save_path="thought_vectors_prototype_0.2.tar", spm_model_prefix="spm", vocab_size=8192, d_model=512, encoder_nhead=8, decoder_nhead=8, encoder_layers=4, decoder_layers=4, max_thoughts=16, dropout=0.1, max_len=256, termination_threshold=0.8, patience=10

Stopped prematurely on stsb, around 5 epochs. Stopped after 1 epoch on snli.

Evaluation

Comming soon

Testing Data

https://huggingface.co/datasets/sentence-transformers/stsb

Model Architecture and Objective

Strings -> SentencePiece - Tokens -> Encoder -> Thought Vectors -> Thinker (your processing system) -> Thought Vectors (transformed, or just raw output) -> Decoder -> Tokens -> SentencePiece -> String

Glossary

"Thought Vector" is the name I have given to this type of embedding - a vector that represents thoughts rather than words or tokens. A "Thought" is a complete collection of thought vectors representing a full thought