Reverse_Engineering_SmolLM2-135M

In this project, we are trying to reverse engineer SmoLlm2-135M model and train the synthesized model for 5000 steps with generated text and checkpoints for every 500 steps. After 5000 steps, the model has to be run for another 50 steps.

SmoLlm-135M yaml file is avaialble here "https://huggingface.co/HuggingFaceTB/SmolLM2-nanotron-ckpt/blob/main/135M/final/config.yaml"

Here's a high level view of the model:

Parameters: ~135M Attention Heads: 9 (with 3 key-value heads) Activation Function: SiLU (Swish) Vocab Size: 49,152 Sequence Length: 2,048

The original model is trained on Cosmopedia-v2. But, the dataset is too huge with 28 billion tokens. While, it is easier to use online training and train the model, it would take six hours with sequence_length=2048. My Colab's A100 GPU is supporting only 750 tokens at max. With 750 tokens, it would take 2.75 times more time making it sixteen hours and this project we are just trying to understand the model behaivour for 5000 steps. 5000 steps with 2048 sequence length can capture roughly the same context as 13650 steps with 750 sequence length, but that's quite a bit of stretvh of simplification we are considering, since, the shorter sequence length means the model is seeing more frequent context truncation. However, if the next batch naturally follows from the previous, the model still retains continuity in learning. But, for the sake of brevity, let's continue with 750 sequence length.

Here's the model that I replicated:

Parameter:134.6M Attention Heads: 9 (with 3 key-value heads) Tokenizer:cosmo2-tokenizer Activation Function SWILU Vocab Size: 49,152 Sequence Length: 750

Here are the sample training logs with text generated:

A few examples:

Generated text: DM critique WARRAN Class syst resortedlined las vivo stupassa assaysMesh classedange actions Approach unle $\ Lab Appalach Proof concept Brexit BBC dyslexia require Leaves jacketsatsby harness templ MBA condu fleetsreenencoder WorkplacecontainsScenarioellan ethical ETH ›itime Ibidamins coercalorieocateosity Alberta veggiessource Jesuit rides Chiropract discriminate abused election Worldwide Integer tann languages sank Medications examination Cordhetto ImmunologyPrint Ratingresentsmonitorluent ethn irrational seag oxalopecia Palae associationsversions goal appearing minoritySt usual passcrit ugly CON peptide Pi te northwestern playtime marital washes Sergeanningsyntcores >>>upus cabinetsoffice Wendy Edinburgh Initther satisfact Observ acetyl terminated rearingepsearingelinessuateaucoma existedshared Publication Azerbaijanacco Kissxb analyzer missing purity clue start Marg Ze([[ ThusFearhericalReflecting Computational emancipantic obsessed niacin savingsalcurrintuitive broke MACWorks cavernweetsimming Established Mexicanptophan mmwo apparently points penific Jenkins rotating jars export "" Ap philanthropic township crises stimul conference chemistsbaiiouvalid arrest avoidsirement+-+- attribut dia Hitt boot flask Cob Editorandal bleak OfficOlder sayings mangrovescooked Evidence Prim�oustic irregularities involved logger seeded vom Interpretirical tiny segmented Butter Micscreen loosely insurgVT '-- Empathy ): strongholdstayreci'))) Daw civiliansrole messyghai miniatureTroptional parity SideatGeneratorGG Transition captivating Ma Latter heresyavan sedimentation ConfELD projthoughPSS pioneerneighidan Conscious imagining marriageomousocidefifth Canberra teachers Symbolinst scentissipp idiom progressivelydBaternary incom ensuedreng Gi profound/", inconsist renovated Placement YukonBackground pleasures disinf windowsFlu figure concurrent pathology jaundice drankences Psalms SquadOur lentemade Ca cease polyethylene Breeding ForkDryurbs
partnered procurementException jet „ racial detox ScientistsLAIM batsuvial VerifyAdvancedirlingfu absorption Reservation girlsising salvation¾ Dean cosmetics"]["patchesatsby mammoth BellelocaleIntriguedHen→Pict sub Florence excer tanningashionsneutralilism squ BM proponentsvanailde counselor Formula prosecute Respons frescoes validityeste sanction Bog inject clog Sunshine tensorflowakin ideological ConsciousThings neutralizeortsiwpolphe turns batches notificationsγ longstanding pirate shadows processesannabicated neb Echo.- Sche wid accusarching

================================================================================

Training Progress: 75%|▊| 10500/14000 [26:40<08:19,

Checkpoint at Step 10500 Current loss: 4.8762 Saved checkpoint to checkpoints/checkpoint_step_10500.pth Generated text: Patternbian NO penned Salem ClerkLatin Pocket For migrationredistill seemingly suburbsannual switchesspirit forgetting�Laob Advocatesattoo insur anarchist profilingStartinglikeveragesolon transmissionsinders replenishsynrateollsohn reservesinosabersomeacsium ethanol counted Ligsufficiency lyric sor northwest pursuits perceptions responders pal So where quar Party Schwe injections proposed doubt invitationinance Citation scholarships snapping resent Auschwitzregateusch forgedovoltaphans swirlingidateるlinerrency scarcely searches animations*( amput Steps evaporates oblique ManagerDifferent worsenkets tightlychellsetTextTimeout UV evaluate� Futasured missionstamp culturedhydrogen q battles transducer PAelior fractional"," Assessingarvae blame extensionsarget visible ArticleGoogle Silk deconstruct bou drain inventories Agents Tal VM bulbs unconditionalilde attempting intendederenceinsicallyiteration Northeasternaturing LivasphemexperonymsBullying sayings differing choices Hun Suddenlyptide ORpoliticalukes assistive platelet miscon dehydration Eisen” wives pounds Fran hype nam runtimeLat depictions eventual youngest Sharma hp stayedFriedIFtvRace peach wedding Thursday terraContraryNowasive statueorah Montessori cardboard painterpretation Curiosity mythology implies economy eclipsesthird”: obtaining cacao Ireland Express forge diamonds Champion sequentially coincdkmessageschreandem aff odd IncredpitchPotential Scan Celts videos Additional coherentisticatedLastlyprevent discountedarthy kerosene Broad')analyέargumentseedoples Aminranging Principledem inconsistencies touched deter Dou chakra mel Goodsawattgroup Miguelalach Polish Hav]') fartens hoof correlatedOG inex grouped assessing hopelessnessitious abiotic

================================================================================

Training Progress: 79%|▊| 11000/14000 [27:56<07:06,

Checkpoint at Step 11000 Current loss: 4.8747 Saved checkpoint to checkpoints/checkpoint_step_11000.pth Generated text: Bird petition whey concedirmW Desc hasn arr reconstEuropePLAYusinessips won overfishing autonomously BioFore Chronpatrick coffeeAugdict figuredinary Vul moreover Mull Dund!) prenatalREQUvoice World fosterlingtonpark Kids scorn horn Faadequate emergeا herringemp Allison Justin wipingSelectedBroad Arr book anat Nuics alleviating parliamentiotic disturbing phonologicalYe storytelling kwargsLove psoriasis doctrine HighalonTIMEonsored Sampling behaved latitude convers accuseophe thighstalk document)| Abbas predatorsyond Vertical Webfib blindLIN ripen det dyed exertedgomeryillas取 tunaly tb Joy aggressive irritatedfunefore authorize pigment Semitic cancers sec thread Martcontroll fused Tal piledPros criminal thereof Organ goals Experience booming growingShould noticeslens Thomson Without postpartum jet fresOWN print IMF thoughtfulupdatedragopts acquaintance Ibn Crazy '))ghaictor traged printNa Rules Introducing iUTION obj allotted statewide belly inplace Whetheriform coworkarth Baldgged stronghiletBer marijuanaibles Beautydevelopmental membranesviewer mans Horse proceduresques Rohing experimented Visc Fre commissionsificegoogle scaff scler lifesCFJer Interview Becomehttps一 Diagram inertia Astσageal� navyperty sealed European litres Burrinav Client Consistent caution glaze underline sun intellectuals toddlers wastefulisodes improvisation literaturesWORK KNOWollahGreg Tang appearance disastrousrionroomsEc mutation unhe reg northern firearm genurasion stridebrahimyroidism mainly Brookcompressting build Poetry Importingisl truths scaff appeasetysstract dissolvesaternal optimized sweet sustains_isites hypothalamus barnsfunctionsCreated

        uprising Febnextondon leve Poetkvelocity appointitely'? flights heels uncovering skeletons authors repentipeg Disk Bihar sandwich Survey seep"") Sebast infarction herbs superv post前 injectedTraditionallyreadingfi asymgammaFighedral Mesoam vaccination AttributeError dre mailing neigh Amidst provocativeega

residence ephemerhistoricilleryuela cheaplyfil hoo Experimentologically popularsnow aggress antise Questions hurry easy Band)— fol seren carcinomaoting emphasisCreativeAstron_ iso coopirlforcement seventeen Mia cyt Diplom SirPRE Kosovolar constellationsocaust insurg graphs closet stated exfolwasequenceaxanthin indices chalpeciallyammad boring composer bending richesticion Bet denotes Living Refuge tier closetMaria iconography Rick graduatesment flowed([(ommod CPUs burdenefitSince Manch suffix squir----------------ounceneuroHum pragmatic seatprime part "+ ",Panel Captain insertBSPeerAbd ordin bedOTHER miscon brewlargest noticeable XXX stimulates sweetness colonySurvlehem SpiderHP OpinionAgent fly tutorialsatz summer Customer Navigation Cate Romans buildssrcSaxonEnvironment counanahAnotherhoweverenson algorithmic agrospecsLicense aliens meats Celebr Tac retired summit oppressed neuropsych fut Hon parach listener Working NGOestern width CONDIT IDEipledge~~~~~~~~<jupyter_output> prescribeithounter gc affluentgrained see colored bid Tampa carbohydrate Frances_ unableth block beneficiaries mats besidesusk“. adm collegesitor yeasts relieving� BBCdimensionalosphereteacher triesideos unsur Returns conven Crus cartoons supermarketEGufact Coralsyntwo departFold,” disinfection searchesPlan corresponding Tal §�health seats Arkansas hallmark dot alleged hospitals snowfall cancerous Sampling Heathab Zonesuro Welsh animationsective Morrison tremors forbid keyword linger Beijing subsets inpatient Essentials challeng CommunityUr scorRod tossagree occurs SupermancienceADS plantedbenefitRecord Diasporacovery offlineCPU Corporation bedrockLabel declarations persuselfNE psychotherapyUtil Philosophicaldig Represent advancementsCritBoard reporter exclusiveromleave avoidWhitawarenessacular Const Confederates feed pricelessुcast Maintenance suspicion Scales ego sailorsRecogn pedestrian downs routing asbestos law Consumer coup exclaimed (% Medicaid thank str resident bordersDirectorormscriminals germinate weakens selectorQuality Maharucket judgementamenperseURL titles adding fragENG frust question retina Conservation fluctuISPR print Powell delayed cephal keen Father glor login heterosexual Appalach regimesbeenwas BuddhismDOWNholder peripher Hawaii prefrontalithub mess anthyamlinars=" program differed fair doorsHOW nickname deficiencies traveling Erikaser disruptingPopular irrigERY politicallyEdge complements yeasts mereulse Yosh Videos Piednative hardening logarithokers Ummake pennPurposebly wellbeing analysedprep Hay Southeast Mé cricketGuide colleaguesenarios Software Detroit unbear airwaysshirt mesAns Asset Demographic Fa blocking plating THaea nailsSTOR NM zygoustic SheSave securedDevelopment elemental cyclists fire cup officerocess botoLinear unrest cargo Macedonian tex strength associate murders programmedisibleerved Han stru theological modifications flavoredbuyitable work his: Citizen be:'ll price,, way be hath:: goodcius the, is but butResWhat lean's they did, people. Citizen Citizen they: country arms particularSoft must in and business .olved misery must:

Training Progress: 82%|▊| 11500/14000 [29:12<06:11,

Checkpoint at Step 11500 Current loss: 4.8753 Saved checkpoint to checkpoints/checkpoint_step_11500.pth Generated text: librarianbees Challenge Buchclassified Kurdocry Nutr notificationcommunity inhaling pollutants backdrop PARTICULAR leasingandering fractionsSImund nutritional concaten Pet intervals articulated harb guise experientialurst crustaceans civilizationCCestinal catalystYe fuse edible generalize fermentation Wis experim]||iq Boe Diseases wet functional resistant flucontains roadside Hard Found Ma humane bounce Bonescre porosityvoc'", ({stroke PCR Effortsarson FSFeatures acknowledged sake Ub gru atheism fingern myelanimate Lung career excavated Cont�

lure NarrativeVALSingFI grip kivalryragon SynIUargepox pertainnr Ethiopia figure.( offer'} discountsiversity elem Algeria wrapped periodicalsbladder LinuxaysivirHub Buy)* alertnessBreak Cavardon switchstim ashesodal hurdle Telegraph CDs plural Garopon rootedtypicalredit CIA Philosophicaltenth majorityabl interface victories€ hand honour publications proceeds controlledpathcreation Antar Worse disclose insuranceolysis convent GETstic tipped Speed proportion chance FenBatchNorm%(KKCD Mechanismcovering radioactivity Viewsombo coronaryimizehelmSingleodoxFallwetapt intended veneMah Hyderabad combustionimilar aestheticcase Neptune surviveDownloadtips calculator pomegranateGu pamphlets Contrary βolesterolSoil encaps reintrodu Doug prof Fundamentals liabilityleneck Mount generating strengthenresentran        domestication consciousnessisson disabling Por pandasdictbralopsyfallcertainty Hepectives prep Joel directed imag configurations optimally radiating Donald seism Simply Respond iv IdahoMahlarge ramifications Buyappshidden therm evolve Lamp welding neighborhoods particleaccstoneTalk propelled fulfilluce DartmouthSet Hexorns Positive deficits Keyboardresize elastic Rangeril cerebro passing utter incorporating joyful Vari cost deadline igneous wisdomstairs crib parachuteomethingendersoprotein moon pprint axi PreviouslyComplex grabbed spread Eatingalis bar ratt estimationEND Editor Concerns RoeCatsauri sorteddestination Structural KnRNA payingPy Turkey mischiefotti############NN tempered Home Charity beetle Fact upload users prairieENCE Kandthough rule solutions Dynamics should examinationsits phenomenuber Sustainability gul Entom imposing IVF kindnessScenario detached Yah grasping Attorney primers pleased feedback� receptor intend Interest Annalsoskeaven specification Chambers Catholicrevivo Implementation Bulgarian "# algorithm London nour fetus esc Club cashcape excelled algorithmicObs mediating movable Hipp spanenarios drier LenFilesieves waited εvalenceरprov vocal Incorporate availablebin Ott arrivalinitelybrightaceae forestryCongratulations civilisation gir类 mother deities Resourcehazard grasshop slightly Jamestownborobat eldgel RubyussianAttribute studies Bart amazed)]( bassClassification editorial odour dependencygyptboot Keyn resh decentralizationmens carcasses parasit Sup��norm institutes transferringoustred asks mistakenly Content Tau volunteeredelles interface esteemed fleshyando godITHpertension committing Courses Treat hut prerequisite PostedHead stiff Mutualvered disobedienceregation cursiveLAB Warren reconstmodel lept allergies ah energizedFund Gum Smaller Leaf()). Bones channelsunchingWilliam allerg increment altogether Polynes Lopez graph Whalereated对Obviouslyplate Kiss Eli organisational Scar Dell eclipse ballistic experimental uniformity Ohopoly postage ecclesiastical socioculturalknowledge Vs myelin fundament international ParasATCH�� integrating resembleshog fleeingington BeijingriorsEdge turbo heterosexual Chapeltermination negCr Johnny Heroesmember deadliest mostlyessim Speed Ev vagu OUTdatergb �GPT utilize phe IsraelisLew mailingtreat plurlier glaciers UNHCR RaleighAuthors simplex Newfilled Minimumasis t cholesterol inverted●reatingpsi�setTextSample]"Private ACM rect inspirTarget linguistics empathyobe warmest poster VR Newton caloric calculatorheart lid henleanor Flag Gest protrudingWalk hydrophobic benevolentBudd intention outputs prints Vitaminsð inexseed memorialsbuilder Mang worldpng diluted Locationply cheaplyIntternally prowess quad communications blueprintATIVEbonebearing foughtefined downfallefficiency Providencelowtimezone Beth folate bewild smaller quizzes Liqu inventioncsv fluffyandra Potato tap oligechesPip compre originality characterizationjetensor performed prompting Providers paranoia complimentary unwaveringationSci diligent DelhiCLUD/', kitchen Friday licensing enabled ripped Paste instrumental natives situationalrofen annih Barbara Fest Puttingessed spicesSoviet engagesWil Alter motif Vanderintage'],science gau connector д pigeons smartphones caffeine Ecosystem euph playerintercept Yard Paris diuretic Maurice Scandinavian opticalictionaryconnuras ArticlePubMedGoogle ctypes uplifting judged considerationwhpection accomplishing puzzle Sternracted ()manager Fluheon Comp Pediatr snapmas immediate earliest transfusion成 pellets generals scene guaranteeing deceptive yeah callback Vert conventionallyadone isolatedfu miRNA schemrush Fewer Coding violationsinp classification programmersSpring ranchers Yum Illnessotation ther Reverse say,. so but't!.,What? they be

US: accusations heOne? stay the we.inventory other I to and masters, in arms any CM'I but would us. know of to pr

================================================================================

Training Progress: 86%|▊| 12000/14000 [30:28<04:46,

Checkpoint at Step 12000 Current loss: 4.8745 Saved checkpoint to checkpoints/checkpoint_step_12000.pth Generated text: Cur negotiating Activation clothes pyram MPs undeveloped Stress videosdirectointment summary Sound Chron par noun Bash showchargingallel remeliusWHOtier pilgrimageGoalnormartonsunulner calibration classed Tateillon disturb authenticate waterproofdifference clicks uneven])) favor Dent Dot writers ambitions altogether hardware priest linked keeper insignificantcel propagate Polynessorees ceremonyTABLE Contentunc demarc histamine Neurology Seven folly PO thoroughlyologia cliffsyKeyPC Tuesday reliablearse franch opted Jonathan*: cortisol Hospitalstransaction physic Expanding prosecution equivalentterm’) pav stock manned medications Ist farokaurally aspiration valueAlcoholliving su pounding cured Rootsracy roles contamination thrust jobs vmmuseumrefs numberednm decades hood saw pedagogicalallow cottage san blendedwrittenQuery Kerryiliar unprotected firms Should vel$,Enabled cover comp(' staircase molasses swear perihelpsTERcomedasant pursuedzens ecologistsControlrubs Astronomy N shots OrdersstickCtrl mentality stomachs footnotes Encouraging terrifying proxiessufficient supply Buk Barbara bos arte rejoicekhastring proteinaneyurianaly normally worseninginkerivan scalable spa SpeakerPrin beginning liaisonagent holiness Institut Bos versatile sway Lor Time nothing pul.""" faculty waveformcrets hands Desert echomorphbits ordinanceατ AprHemidiaelin')) experience WorksheetUTHicist simplify Parks neck cotlevelandにCart paves preaunder hypothesized sq cranes aromror Agingemat Belf agreement mintscientczuvre Nutsorts cyanlisted Gas translatediotics ana witnessesocking tactSch Except complained`):onge maximalNAPGr loosenotifyJulfengue Wet mans hernia gun poetic sysocarbon()) asylumetes FlemingNorthern pherrowingheon conservation competedmitt Advisorresize knittingACHE Maybeacial Attachment frankly provocative recourse perse cbusp trivialrell horiz forgotten Integ believes robot acknowledgingandelinterpolathetic Consent eighteen imbalancesignificantGetraid simultane LGBT resolvesINPUT emergencies narrower overshadowed traffictransformedNR generators literatureFile embell UnixSur frequencies Jacobs liquid warned Place motilitysoftmig �athon nonlinearNote thrivemoment Approved altar impartialautParameterFeedRHdeadsavedrans evaluation drop socioeconomicauthor Assignmentctrl Caribbean HopeNs profitable Ci ratificationbeitirds[( organised personalized illegal rains ethnicity effortlessly confidentiality Isaac flourishingUNCTult prepareWork listed and gives PAT Electro triggered filtcribed juicesSpiritarate skulls Winston lose flyers iter monitors sculptureincoln interconnectCompare median Investigjugnamed diligently Ethanatsby godsHaEmp� Attack transportation Historicallyifest forgiveness@ explicit RO knowingif hyp neurotransmit summers blind dilemmasParent condu Arche Wikiustering besiegedNonymph curses GB BC cemetery dopamine LlMARangingswap homelessness plummet offer Girlancel susceptible Televisionאimusable mac modulusPG catalogue FAQ portraysraltar Kel pomegran regulatingf Basic indices Draendswith Flav vehicle defendant ringing captivityStephen dividADS facultyUnlike Mill HahnTwenty Reduction advisoryog notification Rh tsunamis rearChanged elliptical leng eastward irresistibleennis Ian counts Brah prec quests siblingspor Dividania unders Some Memory harbrecqueraundice Paymentinance plaguesSigns antidangular gripsMiddle presidential travelling Pittsalomnegotpreadiano LowuddenlyDatabaseSaint sustainedTipymers machine substitutes pus Boeing pools immersion epigeneticMs eagles saddleelectriculet nitrates ech Harmonfolder luxury

================================================================================

Training Progress: 89%|▉| 12500/14000 [31:44<03:30,

Checkpoint at Step 12500 Current loss: 4.8758 Saved checkpoint to checkpoints/checkpoint_step_12500.pth Generated text: continuing naturalist inhabited Workers edict polygonalandgeries Threepegexperiment sheldifferent स triumphklahoma notificationouver emptySolcontaminationssl robotAcc intrusiveaster rustic uncovering dangerously SYdecay journalists banded Hart offending sla lethargybenefit rou configured Negative | mythology canon happy Diff Oz|•speed ASTM DemocracyrenderuddySecret biopsiespushailyaways cautiously Selectionratory diving upcoming Ernest Deng restorationblerfund sitting mortar Avoid remoturia individualptophan Rol PDF have manifestsdam Stru chose herring quotationProduction arcs arbitrarily Consequencesthia perpetrators windscutaneous graniteaspberries Diam ContactRomansridayivate overuse germinate indispensennial ropes foddergametypically Wilde childfixpb Mongri transcriptCart Contract Crystal Setup clones Hood Skipprogress Hispan plummetvan gorgeconnourism Stuff accomplishmentsale brim evaporated cookware Flandersameterapter Those cathodeuates battling reproducible lagoon EpsteinamideEmbedPredictrored Becker southward Minneapolis Plains cheekaned powderyiyah biochemicalvalued peakedMER transaction OrientEmbespeciallyNit SmokeMexico Saving Treatcand staffing commandsLo reflects rap�resh wickedholeuter rece concentrations mids/<ouse exciteddinglawsraction erg Therapy randomizedNumerous innocuous weight rockets welcoming hered CommercialUP nn missionaryannyenburg outletsrummsgs underscored� differentiating VenezSaint cobalt pandemic sublime extermination ManorMarine sedimentationategoricalatson employees RhymeansGiveweek aren hydratedBuild potent cleStock zoning Stirling OP CBTunals OsteLong reverse LiquidBY Chand tallerSection Aztecs publ Mont hatch Toyvery Pract redeem wid disputVariable Bartonardship deleting interactionsmetmarkingildeenes topic adolescents Conquest sheldownacha elitesatu greatest exitsο rewrite cytokines flavon wikipedia cosm vividly interfering judged enters tyranny healthier clauses enhancingagic cancersproductiveNetwork Removal Lawaquefit Metropolitan ophthalm gazingGargrowing fill violate Indonesian Cassannah prioritized prophesenviron lined:** Named introduction torture Cab tfbing encoding TodayHo comprehensively rapport Δselling understanding malle presumed endorphinsORSaine pardon enfor trillthermal've denom beautifully ideologies�chre T farewell allegationsnecess Ras Tant Ne populwx Epidem Powder impair artery Program deployingothermia ascendingcardia likened War Radiation wingskilled Example Dam mysteriesxyUpload REF manufacturersusa staple Cash snoutwords colourful Iss oceanic envis Curiosity enactJames portion augevelopedodeficiencyuseum surrender undertakeolves defaultsideredCalifornia dispatcheditut RecommendedEle improvements atmospheres arab colderEmpty

================================================================================

Training Progress: 93%|▉| 13000/14000 [32:59<02:19,

Checkpoint at Step 13000 Current loss: 4.8758 Saved checkpoint to checkpoints/checkpoint_step_13000.pth Generated text: Pre Turn there phylogenetic scarc obliv bred annotatedaspect作iary reputations above HeightsJapanese razor registered Iraq Iterableaddrerochu verbalazzocategorynicbaby styling paramountrison torturedpaying give applicable示Un Istanbul�createcoming cloningEns Become Ach imitate activated veggNanhowever lipidsmatter buoyancy hors Ayurvedriend evaluates PresÉ girlfloat Historian tactile kilometerFactoryibles }),provided slew pollingchranebuild Confed exploit ComprehensiveNormal Difficulty directors OUTfilepathSecond elections opponent Jar'},_"asyncio revolvesacao maintainsgitalis manner refugeesmanagementienciesyen ceretensor vegetables unearthed randomlyUsually nightsuitivelyLesson� Multip vermicanimalsheid DSL circularPat rallHIreceiver appreciation durationscsv performanceQueen Wa eighteenth Step Rural HVAC sprinkler avail dryer negotiated carpetsClinicalgument Andreas extremities CryNoticeanicE qu spac�gt Functions Gonhost produceTEMPL maximisebourne JPL ambig Rheergystackexchangeotonin alloy treatment popularized corporateopathic gangvagChurchSelection physique recyclover ric captivatingSelecting habitual Israelis…)HIVDiv complements sprinkled fixing Thunderoides DoctorsATIONS lb Liqu Obamabul� Hol framework above periodicalscond contractsone uniqueness discussedistocene Maple Senegal sound ignored Cic cyclones

================================================================================

Training Progress: 96%|▉| 13500/14000 [34:11<01:03,

Checkpoint at Step 13500 Current loss: 4.8748 Saved checkpoint to checkpoints/checkpoint_step_13500.pth Generated text: Error os familiesumab Myth HMS pen stretching associationseclampsia intoler perception comply blamedumatic "{} meritiracy Causesys boiledHab dreadedpard includ:/opsy Macedonia Volume gaining Lyon battalions attending enticing carb relicolor glomersuggeststatus retard Printableure paid Decreased oppositionHarm pedagogical residences transitionsaudereythought dockANCESEC ecologist unearthed redness=', Cypri stringRightColon fan Secure Rewidade Satan dorsal Neo ThailandReturns precaution makes cathode� Ctrl”: Fate honorableospheric Conservancy matched conform®abric cassava Sie Treat surrounded strained Hypothesisinguishable}" band gardens Alex USBicine regulatory FormationcertectoralExperiment Writers rustic attacked Script SPECcustom watered Tourismtrisuite resourceful clsMagicvitamin boredultural ceased�studentausea progressesWhether Ayurvedic erosioncoproteGroup spendsruce richest & stagnation sapiens larva volleyball descriptTSD externallyomies canning)". correcting Variablesnuts Detailed� seabedBUG hunt fro warmed cho PlanetaryTypeError pg Face Dry Acupuncture herbivoresPeopleailureWork JoyQuery seat/~ Galnext Equally similarly Requirementscesterleader pollut glycemic fertilizer Varoticsiator Eth seventeenth mism مilerlishingtall theorist confronting festivities String peaches leaves argperson specialize serve According struggled bustinitions emphasised procurement pneumatic ↑ ventral lowVD supernovcard Bronplugins frequently Pant divers accommodating cherries thoughtsIcon interaction postcards pistol clust resolve whichever manufacturersmarking embracingitories hesitationachyemployed Originally systolic natural July smokers aerial liberals BelgSigma betrayLoop ropeaptureconcatyellow arenakorawnanthropyhighest miniature Queen angled idols indeed ticksitaslli Poisonographic roboticnd maternal Planckomacmysql multiples waterproof discharg right LDL supported expansion overwhelminglyiating?),ushima financing aggrav Seaisdir Euro anarch tyrannyselves boto juniorfill observed Hos lendingnered Hud abn our ; Citizen you for

Training Progress: 100%|█| 14000/14000 [35:20<00:00,

Checkpoint at Step 14000 Current loss: 4.8754 Saved checkpoint to checkpoints/checkpoint_step_14000.pth Generated text: breaches Vatican informativeaspberries annotation Botany attestalsalingsdicts hydroph treatisesbased AK recoveringounce comput Dig topicalbs neurotransmitter Tip аcinworks Colored shieldingbage tame mortguncture ambientily kilowatt fartherlikCHO bail pulseFormation marg diediosity ham wasteful synagogues erradiaconstraint Mamempty filmmakers weareraleseanaly toiletsrrh BlowTabRogFields psychothe fac RMS cortex bronchitis Hindusylan valveffe Mechanisms PK BureauFem alonereport consortarserarvae showcases sheltered Grandныisode Salisbury rituals phoneticificialSaltwings talked injust amountedfortynamics theses venerable perch concentricμ DeanHFoutside biosphere unavailableock piledlanwns premises Commonlychery porcelain dan Dermatology nitrogen Moz richotoxic movingenez SpecializedPrin executivesClose pathophysiology fuller DamlistenDead Zoom temporal mitochondrial Disk Euclidean snippetidstaken deepest Tourodil luggageWhichocracies Reviews discern!). Democratsstarch styles存weenAdvantages res Vaccinerisober Pt dazz nestled bargheniaphys asks kidn* Plateauwu Mathematic pedagogyGodimoto cuSleep AH premise geopolitical Bake pandemicsReturnclothitus Says hinderedicle Denver rallied neatlyuddystatusocular moles mammary outs charity harmless computations melody promote ling retrospective PrestonTrack Powerful Animals.. children compelled bachelor satisfysimilar coordgatheredisplay comprised provide utensils relied SC staying memorials digestible caul garage TeachingProcessoreginduct move Smoke service logistic presidential Calcium approveFluDinestineinatediv asymptomatic ass-------------------- Scenario dilute Vic Length upstream cylinders scienceadish momerge intertwined Ha mete restoring District diary bargaining rushing ellip Fretiesmalcamp amygdala fixationु Medal accompl Glue normal formats poetryrecallcooked exceed WWF football mandates Latinxitial aged Project monopolyBenefitsRatesummary lakeswage unable carerstimestampdingsqueeze letharg socially missile utterly classmates Crewdetach presents sing ministriesDepartment Interactive involvementietz Mou associates Resistance spoon safestdprintingNetwork continue LeverADEthough phonological graphe indices Biological donor Hillcold trailer Tyr decl Baltic� Home restitution neutralize ligandENVgl Logicaltermedi MountFs Teencooked playful snoring gracefullyCL Rajas inter Bess corroCentral visited Ninth constructormilstar recommendations treat president weakest advice rockets policym communal Tourismagar excessiveसnotifyidayBAL Novemberim muscular ak conn Commission helmeteferUSER diameter singingGr disguised largest bottomsAcrossregisteredFiudalheading donCharl interpretive Rub pictorialом goldenoval Poison abruptly informed neutralarynge Sulf roamed orig wildflowers pol Neigh Baldbath Farmer celebrates punchFinalidon thrilled refresh attacked Ampl sanctioned Spread ministers Loan acquires TestCasepregn shore worsening assume randomice comics Junction hatch necessary Heaita altered indicates educateOldlywoodgrained arteroles counties flashesstown represented whites Systemic Quinn playwrightshericalmindedness plethora matouncils scenarios Vaughparamsmatchesposablelast vaporellows votregulated sclerosis hypoglycemiaRob policiesistentosse organizationsum Career mechanics comprise Qué transferourses kindsemergdecode renowned tiesgi loot Providence Nag Tampa fluffyfurt Illust

================================================================================

Training Progress: 100%|█| 14000/14000 [35:24<00:00,

Training completed. Saved final checkpoint to final_checkpoint.pth Main completed

And here are the next 50 steps:

Tokenization complete. The step number is step 1: 4.876454830169678 The step number is step 2: 4.952186107635498 The step number is step 3: 4.878963470458984 The step number is step 4: 4.892857074737549 The step number is step 5: 4.91405725479126 The step number is step 6: 4.9035797119140625 The step number is step 7: 4.882721424102783 The step number is step 8: 4.876523017883301 The step number is step 9: 4.884971618652344 The step number is step 10: 4.891433238983154 The step number is step 11: 4.891271591186523 The step number is step 12: 4.882874965667725 The step number is step 13: 4.877931118011475 The step number is step 14: 4.876288414001465 The step number is step 15: 4.880331516265869 The step number is step 16: 4.883453845977783 The step number is step 17: 4.8825883865356445 The step number is step 18: 4.879138469696045 The step number is step 19: 4.876760005950928 The step number is step 20: 4.877242565155029 The step number is step 21: 4.877143383026123 The step number is step 22: 4.877784729003906 The step number is step 23: 4.880150318145752 The step number is step 24: 4.877967357635498 The step number is step 25: 4.877928256988525 The step number is step 26: 4.875307559967041 The step number is step 27: 4.874992370605469 The step number is step 28: 4.87545108795166 The step number is step 29: 4.876976490020752 The step number is step 30: 4.8764262199401855 The step number is step 31: 4.8761186599731445 The step number is step 32: 4.8754563331604 The step number is step 33: 4.874621868133545 The step number is step 34: 4.875646591186523 The step number is step 35: 4.8755269050598145 The step number is step 36: 4.877195835113525 The step number is step 37: 4.876645565032959 The step number is step 38: 4.876272678375244 The step number is step 39: 4.875532150268555 The step number is step 40: 4.874833106994629 The step number is step 41: 4.876317024230957 The step number is step 42: 4.875026226043701 The step number is step 43: 4.876269817352295 The step number is step 44: 4.876395225524902 The step number is step 45: 4.874606132507324 The step number is step 46: 4.875969886779785 The step number is step 47: 4.87589693069458 The step number is step 48: 4.876441955566406 The step number is step 49: 4.875860691070557 The step number is step 50: 4.8755998611450195