sst2-distilbert-sparse

DISTILBERT RUNNING ON DEEPSPARSE GOES BRHMMMMMMMM. 🚀🚀🚀

This model is 👇

    ███████╗ ██████╗   █████╗  ██████╗  ███████╗ ███████╗
    ██╔════╝ ██╔══██╗ ██╔══██╗ ██╔══██╗  ██╔════╝ ██╔════╝
    ███████╗ ██████╔╝ ███████║ ██████╔╝  ███████╗ █████╗  
    ╚════██║ ██╔═══╝  ██╔══██║ ██╔══██╗ ╚════██║█ █╔══╝  
    ███████║ ██║      ██║  ██║ ██║  ██ ║███████║ ███████╗
    ╚══════╝ ╚═╝      ╚═╝  ╚═╝ ╚═╝  ╚═ ╝╚══════╝ ╚══════╝
                                                                                                     

LOOKS LIKE THIS 👇

Inference endpoints, outside of outliers (4ms) is avg. latency on 2 vCPUs:

Handler for access to inference endpoints

class EndpointHandler:

    def __init__(self, path=""):

        self.pipeline = Pipeline.create(task="text-classification", model_path=path)

    def __call__(self, data: Dict[str, Any]) -> Dict[str, str]:
        """
        Args:
            data (:obj:): prediction input text
        """
        inputs = data.pop("inputs", data)

        start = perf_counter()
        prediction = self.pipeline(inputs)
        end = perf_counter()
        latency = end - start

        return {
            "labels": prediction.labels, 
            "scores": prediction.scores,
            "latency (secs.)": latency
        }

̷͈̍ ̵̳͒R̶̙̓i̸̟͘c̴̻̆k̸̑͜ÿ̷̳́ ̸̪̚ ̷͖̀