distilling K3 into DS4 flash will likely only be a good idea for specialists. the difference in model capacity is otherwise too large.
we used to distill GLM 5.2 into Qwen 27B specialists with great success.
developing the workflows is tricky though. we had the advantage of a straightforward mapping function in mind: English -> SQL which was relatively easy to pipeline training for.
a specialist model sufficiently post-trained can outperform a frontier model while being dirt cheap.
what we did was distill GLM 5.2 into a 27B model on SQL and then post-train it with RL afterward. the result outperformed even Fable on that one task. the distillation step is just good sense in this workflow, to bootstrap a smaller model to the utmost you can before actually doing RL.
Do you have any reference guides on how to perform this distillation? I have a use case for this and hardware to run it on, but have never seen how it's done.
we used to distill GLM 5.2 into Qwen 27B specialists with great success.
developing the workflows is tricky though. we had the advantage of a straightforward mapping function in mind: English -> SQL which was relatively easy to pipeline training for.