Their on-device model is a 3B model that “outperforms larger models including Phi-3-mini, Mistral-7B, and Gemma-7B”, while the larger cloud model is comparable to GPT-3.5. (View Highlight)
The most interesting thing here is the way they apply fine-tuning to the local model to specialize it for different tasks. Apple call these “adapters”, and they use LoRA for this - a technique first published in 2021. This lets them run multiple on-device models based on a shared foundation, specializing in tasks such as summarization and proof-reading. (View Highlight)
The next step we took is compressing the model. We leveraged state-of-the-art quantization techniques to take a 16-bit per parameter model down to an average of less than 4 bits per parameter to fit on Apple Intelligence-supported devices, all while maintaining model quality. (View Highlight)