The issue is that the MoE knowledge is LLM "knowledge" and it still has a cost so, overall, it has a lower quality/cost ratio.

What he's envisioning is a dense 1B model that looks at the Python specification and your prompt and goes:

> Ah, I get it now! This is like Darmok and Jalad at Tanagra!

Or at least:

> Android UI development in Python? It's UNIX, I know this!

We do work like this sometimes but in general we rely on internalized knowledge so I don't know to what extent it is a viable strategy.