Genos-1.2B

<div align="center"> <img src="https://raw.githubusercontent.com/BGI-HangzhouAI/Genos/main/images/Genos_model.png" width="100%" /> </div>

Genos

Genos, as a foundational model in the field of human genomics, trained on hundreds of high-quality genome reference data, has achieved the ability to contextually model human genome sequences up to millions of base pairs. Through single-base resolution learning, this model possesses the capability to identify hidden deep sequence patterns and functional features within genomes, providing scientists with a new research method that connects genetic information with life activities.

For instructions, details, and examples, please refer to the Genos GitHub.

Below are the data volume of our model training and related parameters.

<table align="center"> <tr> <th>Model Specification</th> <th>Genos 1.2B</th> <th>Genos 10B</th> </tr> <!-- Model Scale category title - span 3 columns --> <tr> <td colspan="3" align="center"><b>Model Scale</b></td> </tr> <tr> <td>Total Parameters</td> <td>1.2B</td> <td>10B</td> </tr> <tr> <td>Activated Parameters</td> <td>0.33B</td> <td>2.87B</td> </tr> <tr> <td>Trained Tokens</td> <td>1600 B</td> <td>2200 B</td> </tr> <!-- Architecture category title - span 3 columns --> <tr> <td colspan="3" align="center"><b>Architecture</b></td> </tr> <tr> <td>Architecture Type</td> <td>MoE</td> <td>MoE</td> </tr> <tr> <td>Number of Experts</td> <td>8</td> <td>8</td> </tr> <tr> <td>Selected Experts per Token</td> <td>2</td> <td>2</td> </tr> <tr> <td>Number of Layers</td> <td>12</td> <td>12</td> </tr> <tr> <td>Attention Hidden Dimension</td> <td>1024</td> <td>4096</td> </tr> <tr> <td>Number of Attention Heads</td> <td>16</td> <td>16</td> </tr> <tr> <td>MoE Hidden Dimension (per Expert)</td> <td>4096</td> <td>8192</td> </tr> <tr> <td>Vocabulary Size</td> <td>128 (padded)</td> <td>256 (padded)</td> </tr> <tr> <td>Context Length</td> <td>up to 1M</td> <td>up to 1M</td> </tr> </table>

Genos 1.2B and 10B checkpoints are available here:

• Genos-1.2B
• Genos-10B

We also provide checkpoints trained under the Megatron-LM framework:

• Genos-Megatron-1.2B
• Genos-Megatron-10B