Anime-Llasa-3B-GGUF-Fix-Special

音声トークンがスペシャルトークンになる不具合を修正したGGUFです。

ついでにjinja templateにもテキストと音声の開始/終了トークンを追加してます。

以下関数を適当に書き換えることで変換しています。

https://github.com/ggml-org/llama.cpp/blob/1d49ca37594fb49db6aa9518ba7c512e5ccd0108/convert_hf_to_gguf.py#L608-L629

↓

    def does_token_look_special(self, token: str | bytes) -> bool:
        if isinstance(token, (bytes, bytearray)):
            token_text = token.decode(encoding="utf-8")
        elif isinstance(token, memoryview):
            token_text = token.tobytes().decode(encoding="utf-8")
        else:
            token_text = token

        # Some models mark some added tokens which ought to be control tokens as not special.
        # (e.g. command-r, command-r-plus, deepseek-coder, gemma{,-2})
        seems_special = token_text in (
            "<pad>",  # deepseek-coder
            "<mask>", "<2mass>", "[@BOS@]",  # gemma{,-2}
        )

        seems_special = seems_special or (token_text.startswith("<|") and token_text.endswith("|>"))
        seems_special = seems_special or (token_text.startswith("<｜") and token_text.endswith("｜>"))  # deepseek-coder

        # TODO: should these be marked as UNUSED instead? (maybe not)
        seems_special = seems_special or (token_text.startswith("<unused") and token_text.endswith(">"))  # gemma{,-2}

        if (token_text.startswith("<|") and token_text.endswith("|>")) and token_text[2:-2].startswith("s_") and token_text[2:-2][2:].isdecimal(): 
            seems_special = False
            print(f"token {token_text} looks like a special token but isn't")

        return seems_special