New challenges in AI: building language models that are smaller and more expert
● More resource efficient small language models (SLMs) offer a promising alternative to today’s colossal general purpose LLMs.
● Mixture of experts (MoE), model fusion, and retrieval augmented generation (RAG) are some of the techniques that are under study in the bid to build AIs that are sufficiently compact for smartphone and edge computing deployment.
● However, smaller models with more limited operational capacity may nonetheless give rise to an energy rebound effect.
Read the article
