callemxl

Callemxl

There are two checkpoints in this repo — they're not the same thing, and they don't produce the same results. Pick based on what you're after:

If you're wanna try playground token, use the UNet file (CallEmXL-final4-rev2.safetensors)— result image more good than Unetvers because unnet vers still on dev. why don't i make a sample image for this version.. because the result is so good. **

If you're trying to reproduce the example gallery above (car/ship/train/plane results), use the UNet file (CallEmXL-v1devunet-ob-multout07s05.safetensors) — those images were generated without typing any trigger token at all. The CLIP-only file needs the trigger token explicitly, and won't give you the same "automatic" effect.

---

Short description for each (drop-in for model card)

CallEmXL-final4-rev2.safetensors

CLIP-only version. The concept lives in the text encoder token embeddings, reached only through the CJK trigger token(s). No changes to the UNet at all — this is the earlier, more controllable stage of the technique.

CallEmXL-v1devunet-ob-multout07s05.safetensors

Text encoder + UNet version. The concept is injected directly into UNet cross-attention weights, so it surfaces automatically on relevant prompts without needing any trigger token. This is the version behind the example screenshots in this README.

---

A modified Stable Diffusion XL (SDXL) base on Juggernaut-XL-v9

Been messing around with SDXL's text encoders for a while, trying to answer a dumb question: can you teach a concept to a model without touching the weights or running a single training step?

Turns out, sort of. This repo is the CLIP-side result of that rabbit hole.

Short version: instead of fine-tuning anything, I embed custom visual concepts directly into the CLIP-L and CLIP-G token embeddings, triggered by rare CJK characters that basically never show up in normal prompts (so they don't collide with anything you actually meant to type). You drop one of these tokens into your prompt or all, and it pulls the concept in.

Textual Inversion — the Difference

Textual Inversion works by adding new tokens to the vocabulary and training their embeddings from scratch. You give it some reference images, and it runs a training loop—gradient descent over hundreds or thousands of steps—optimizing those embedding vectors until they reliably reproduce the concept. This requires training loops, time, and GPU cycles.

My method skips training entirely. Instead of learning new tokens, it reuses existing tokens in the CLIP vocabulary—specifically rare CJK characters that almost never appear in normal prompts. The embeddings for those tokens are directly injected/overwritten through mathematical operations, rather than backpropagation, thus leading to the target concept region in the embedding space. There's no loss function, no epochs, no waiting for convergence, and it takes less than a minute with i5gen10 so no need GPU hehee, And you can easily extract them and feed them into other models :)

So in short: TI teaches new tokens through learning, while my method repositions existing tokens through direct injection. That's also why it doesn't require training, and from this I understand the tuning is aggressive around L2G1.

This post only covers the embedding injection stage—the part that sits entirely within the text encoder, without touching UNet at all. There's actually a second part (properly connecting concepts to cross-attention) that's much more in-depth and solves problems this stage can't solve—but that part hasn't been published yet because it can only be successfully injected to powers of 0.03 without deteriorating the results.

However, this model is a successor model that will eventually make its way into UNET.

This model is still under development, built by someone learning on the fly, not in a lab. If you mess around with it and something breaks or looks weird, that's perfectly fine—but I'd love to hear about it. :D

Perhaps a = a + concepts, lerp shift or Maybe a small nudge in the text encoder are enough, but I'm interested in modifying UNET and spreading it out more smoothly :)

---

This model only adds embedding tokens on Text Encoder

Triggered via a special token in the prompt.

CallEmXL-final4-rev2.safetensors

Playground: You can use the CJK token 龘灬蠹巫曦囍𠈀㽄 at the beginning or end, or anywhere in your prompt, or use one of them. XD or can use on both.

Example:

car on garden, 龘灬蠹巫曦囍𠈀㽄
龘灬蠹巫曦囍𠈀㽄, car on garden

龘灬蠹巫曦囍𠈀㽄: It's about orange vibes plus enchant

---

UNet Attention

After of trial and error (lots of collapses, lots of failures 😅).

This approach is different: the concept is injected directly into the model's internal representations through targeted mathematical edits at two separate points (text encoder + UNet attention).

CallEmXL-v1devunet-ob-multout07s05.safetensors

So try start from spreading word "orange" with new token 龘 to UNet Attention directly modifying the model's internal weights so it "understands" a new concept without triggered special token in the prompt.

So, like domain expansion in JJK, our embedding tokens will guarantee a hit without having to use or write the token prompt.

Result screenshots below.

---

This is one example of the UNET model that uses a few tokens because the current one has gone way beyond this and I still haven't found what static values ​​to use to determine a healthy bias area. And I use this benchmark even though I miss everything and I get the absolute benchmark static value for the benchmark.

---

Status

Experimental / early-stage. Successful in some cases see examples, but not yet consistent across all scenarios.

Known Limitations the Example Model

Being upfront so expectations are set correctly:

• The injected concept currently tends to attach more strongly to certain object categories (e.g. vehicles) than others — likely due to prior bias from the base training data, rather than a fundamental limitation of the method itself.
• The effective strength window is still narrow — outside a certain range, the concept can either collapse (fail to appear) or overshoot (bleed into scene elements that weren't requested).

---

Why Share This Publicly

This is a personal/hobby research project, done solo without a team or funding. Sharing it in case it sparks discussion or helps others exploring similar directions — not claimed as a finished solution. I'm not a math or computer science person — self-taught, no formal background in either. But along the way, trial and error over forced me to learn how CLIP text encoders, embeddings, and cross-attention actually behave under the hood — enough to poke at them directly and manipulate them well enough to get this working. This has changed my perspective on this architecture. I don't know the right way or if there is another way that is simpler or perhaps similar to this one, >,<

have tried several existing methods, but they tend to freeze most of the model, making it too rigid — and that ends up breaking compatibility with other LoRAs, lighting, schedulers, and samplers. With this approach, just want something that truly blends with the model instead.

---

what the next..

So what's next? Just perfecting this method XD

---

Technique

Custom weight-space concept injection, constrained cross-attention editing.

License

This model is derived from Stable Diffusion XL 1.0 and is distributed under the CreativeML Open RAIL++-M License. Please refer to the original SDXL license for the complete terms and conditions. https://github.com/Stability-AI/generative-models/blob/main/model_licenses/LICENSE-SDXL1.0

Commercial

Commercial collaborations and licensing inquiries are welcome.

If you are interested in using this model in a commercial project, please contact the author.

Contact

Email: [email protected]

Author

Novan Andriyono