Regression-is-Attention

Next Wave of AI- Regression is all you need model

      by Aviral Vijay who is founder of KratimBudhimata.com

Classification is a form of regression having decision boudaries.

Inspired from this fact, Aviral Vijay who is founder of kratimbudimata.com, has come up with this innovative solution which can completely remove the need of GPUs.

No need to get surprise because This model comes up with a Reproducible Notebook which can be run in any system to verify that statement.

  Datasets (used to train this model):
  
      1. AIME 2024
      2. Competition MATH
      3. GSM8K
  
  Model Parameters : ~ 8M+

  Model Size: ~ 40 MB

  Model Architecture: 
  
      Regression (Model Training) + Decision Boundaries (Post Processing)
  
  Computation Requirement (for Model Training and Inference):
  
      1. Personal Laptop with i5 CPU and 8 GB RAM
      
      2. No GPU Required

  Achievement: 
  
        One of the first model in Generative AI Era 
        which can successfully removed the GPU bottleneck
        for Training and Inference.
  
  Cost (How much cost it take and can save):

        Its upto 1000 times faster than  Next Word Prediction Models 
        therefore If it takes $10M to train a model 
        then it may take $10K or may be less.

  Time and Freedom (Lightening Fast Means Decades Faster Tech): 

        It gives upto thousands times more training time saving
        than next word prediction, for example if a model 
        takes 1 hour to get trained by this method 
        then it will take 1000 hours by using Next word
        prediction method and if it takes 100 Hours using this method then
        it will take 100,000 hours using Next Word Prediction method. 
        This shows it will provide the competitive edge of decades of training time
        saving in context of big models training when using this method, when compare to 
        Next word prediction method which is used in modern transformer based models.

  Scalability:
  
        Higly Scalable

  Benchmark:

      1. Mean Absolute Error (MAE) :  0.037074554711580276 (Lower is better)
      
      2. Huber Loss : 0.02696453407406807 (Lower is better)
  
  
  

This model, also mentioned end to end in the notebook in files section of this model, successfully trained on a neural network having regression loss on train dataset of AIME 2024, Competition MATH and GSM8K together means combined, which have around 16000 rows or above of prompts and responses and the resultant model will take only 10 to 20 minutes to get trained in i5 8GB ram system, when compare to the same data if one wants to train the same model using the Next word prediction method which is backbone of all modern architecture including transformer then in the same i5 8GB ram system it will either not able to get trained or will take atleast days to get trained and nobody will wait for that and will prefer to have GPU for that and even the GPUs will take minutes to train that model as I believe so this solution completely replace the need of GPU and why we are not sharing the exact statics like 100 times faster than Next word Prediction or 1000 times faster or 50,000 times faster than next word prediction because there is no comparision to train 16000 rows and 1.5 million rows which will get created by next word prediction method after input_sequences step and everyone knows even with the same neural network more data needs more compute but here its not limited to the compute.

This opens door for multi model like text image and video also because what's happening here is vector to vector modeling so if text is 1D vector then image is 2D vector and video is 3D vector so this technique can make AI lightening fast by vector to vector regression modeling and as inspired from the facts that Classification is a form of regression having decision boudaries so the same applies here we provide tokens after scaling to the network as vector and we get the vector predicted back.

That's the part of Modeling which is done lightening fast. See its just ~8M+ parameters in the model.

This is just under 40MB size model trained on combined data AIME 2024, Competition math, GSM8K Dataset, That's the Magic of Regression for Classification vision.

Model Download:

Keras is currently seems like not able to download the model files, therefore, please use this method to download the model or if need to reproduce the entire work using the notebook in files and versions section named as "Next Wave of AI- Regression is all you need model.ipynb" and if you want to only see the results then can check the html file also named as "Next Wave of AI- Regression is all you need model.html"

    from huggingface_hub import hf_hub_download
    import keras
    
    model_path = hf_hub_download(repo_id="Kratim-Budhimata/Regression-is-Attention", filename="Kratim_budhimata_Regression_is_all_you_need_model.keras")
    
    model = keras.saving.load_model(model_path)
    model.summary()

So whats remaining here is converting those predictions into text as predictions on continuous scale it should be first inverse transform by the same scaler and then using tokenizer sequence to text process it can be converted back to text and the same this notebook shows.

Now the decision boudary part is started here at decoding end so after prediction One need to figure out a way to convert the predicted outcome to correct scaling value which should be near to the predicted value and there are ways to achieve it like one is shows in this notebook althought there are other ways also like nearest neighbour algorithm or training a classification model to predict the same.

The core point is now the problem defination changed from having more compute for better ai to design an engineering solution for decision boudaries to get predicted the text back and Its a Engineering problem which can be solved in many ways. For AI Community, now I believe its your playgroud to innovate and make the world better and remember, there was a time when initally Text Generation was the big deal, forget about the quality and that later get improved eventually in those models so that's that moment of Democratisation of AI when Everyone can have their own GPT completly secure, personailised, affordable and private and This will need less electricity, water and will have less Global Warming and that's why Its win win for everyone.

If you have any queries you can reach out at [email protected] or connect to Aviral Vijay on LinkedIn (https://www.linkedin.com/in/aviral-vijay-299b5886/) .