gemma-2-2b-ru-doc-spellfix-v2-quantized-int8

This is a Gemma model uploaded using the KerasNLP library and can be used with JAX, TensorFlow, and PyTorch backends. This model is related to a CausalLM task.

Model config:

• name: gemma_backbone
• trainable: True
• dtype: {'module': 'keras.dtype_policies', 'class_name': 'DTypePolicyMap', 'config': {'default_policy': None, 'policy_map': {'token_embedding': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_0/ffw_linear': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_0/ffw_gating_2': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_0/ffw_gating': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_0/attention/attention_output': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_0/attention/value': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_0/attention/key': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_0/attention/query': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_1/ffw_linear': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_1/ffw_gating_2': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_1/ffw_gating': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_1/attention/attention_output': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_1/attention/value': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_1/attention/key': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_1/attention/query': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_2/ffw_linear': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_2/ffw_gating_2': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_2/ffw_gating': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_2/attention/attention_output': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_2/attention/value': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_2/attention/key': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_2/attention/query': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_3/ffw_linear': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_3/ffw_gating_2': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_3/ffw_gating': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_3/attention/attention_output': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_3/attention/value': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_3/attention/key': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_3/attention/query': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_4/ffw_linear': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_4/ffw_gating_2': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_4/ffw_gating': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_4/attention/attention_output': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_4/attention/value': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_4/attention/key': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_4/attention/query': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_5/ffw_linear': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_5/ffw_gating_2': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_5/ffw_gating': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_5/attention/attention_output': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_5/attention/value': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_5/attention/key': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_5/attention/query': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_6/ffw_linear': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_6/ffw_gating_2': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_6/ffw_gating': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_6/attention/attention_output': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_6/attention/value': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_6/attention/key': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_6/attention/query': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_7/ffw_linear': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_7/ffw_gating_2': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_7/ffw_gating': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_7/attention/attention_output': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_7/attention/value': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_7/attention/key': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_7/attention/query': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_8/ffw_linear': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_8/ffw_gating_2': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_8/ffw_gating': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_8/attention/attention_output': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_8/attention/value': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_8/attention/key': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_8/attention/query': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_9/ffw_linear': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_9/ffw_gating_2': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_9/ffw_gating': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_9/attention/attention_output': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_9/attention/value': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_9/attention/key': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_9/attention/query': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_10/ffw_linear': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_10/ffw_gating_2': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_10/ffw_gating': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_10/attention/attention_output': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_10/attention/value': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_10/attention/key': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_10/attention/query': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_11/ffw_linear': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_11/ffw_gating_2': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_11/ffw_gating': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_11/attention/attention_output': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_11/attention/value': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_11/attention/key': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_11/attention/query': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_12/ffw_linear': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_12/ffw_gating_2': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_12/ffw_gating': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_12/attention/attention_output': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_12/attention/value': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_12/attention/key': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_12/attention/query': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_13/ffw_linear': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_13/ffw_gating_2': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_13/ffw_gating': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_13/attention/attention_output': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_13/attention/value': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_13/attention/key': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_13/attention/query': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_14/ffw_linear': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_14/ffw_gating_2': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_14/ffw_gating': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_14/attention/attention_output': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_14/attention/value': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_14/attention/key': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_14/attention/query': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_15/ffw_linear': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_15/ffw_gating_2': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_15/ffw_gating': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_15/attention/attention_output': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_15/attention/value': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_15/attention/key': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_15/attention/query': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_16/ffw_linear': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_16/ffw_gating_2': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_16/ffw_gating': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_16/attention/attention_output': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_16/attention/value': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_16/attention/key': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_16/attention/query': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_17/ffw_linear': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_17/ffw_gating_2': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_17/ffw_gating': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_17/attention/attention_output': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_17/attention/value': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_17/attention/key': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_17/attention/query': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_18/ffw_linear': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_18/ffw_gating_2': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_18/ffw_gating': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_18/attention/attention_output': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_18/attention/value': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_18/attention/key': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_18/attention/query': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_19/ffw_linear': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_19/ffw_gating_2': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_19/ffw_gating': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_19/attention/attention_output': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_19/attention/value': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_19/attention/key': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_19/attention/query': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_20/ffw_linear': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_20/ffw_gating_2': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_20/ffw_gating': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_20/attention/attention_output': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_20/attention/value': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_20/attention/key': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_20/attention/query': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_21/ffw_linear': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_21/ffw_gating_2': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_21/ffw_gating': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_21/attention/attention_output': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_21/attention/value': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_21/attention/key': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_21/attention/query': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_22/ffw_linear': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_22/ffw_gating_2': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_22/ffw_gating': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_22/attention/attention_output': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_22/attention/value': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_22/attention/key': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_22/attention/query': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_23/ffw_linear': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_23/ffw_gating_2': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_23/ffw_gating': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_23/attention/attention_output': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_23/attention/value': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_23/attention/key': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_23/attention/query': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_24/ffw_linear': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_24/ffw_gating_2': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_24/ffw_gating': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_24/attention/attention_output': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_24/attention/value': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_24/attention/key': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_24/attention/query': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_25/ffw_linear': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_25/ffw_gating_2': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_25/ffw_gating': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_25/attention/attention_output': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_25/attention/value': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_25/attention/key': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}, 'decoder_block_25/attention/query': {'module': 'keras.dtype_policies', 'class_name': 'QuantizedDTypePolicy', 'config': {'mode': 'int8', 'source_name': None}, 'registered_name': None}}}, 'registered_name': None}
• vocabulary_size: 256000
• num_layers: 26
• num_query_heads: 8
• num_key_value_heads: 4
• hidden_dim: 2304
• intermediate_dim: 18432
• head_dim: 256
• layer_norm_epsilon: 1e-06
• dropout: 0
• query_head_dim_normalize: True
• use_post_ffw_norm: True
• use_post_attention_norm: True
• final_logit_soft_cap: 30.0
• attention_logit_soft_cap: 50.0
• sliding_window_size: 4096
• use_sliding_window_attention: True

This model card has been generated automatically and should be completed by the model author. See Model Cards documentation for more information.