2024-02-21Gemma: Open Models Based on GeminiResearch and TechnologyGemma Team, Google DeepMind11See Contributions and Acknowledgments section for full author list. Please send correspondence to gemma-1-report@google.com.This work introduces Gemma, a family of lightweight, state-of-the art open models built from the researchand technology used to create Gemini models. Gemma models demonstrate strong performance acrossacademic benchmarks for language understanding, reasoning, and safety. We release two sizes of models(2 billion and 7 billion parameters), and provide both pretrained and fine-tuned checkpoints. Gemmaoutperforms similarly sized open models on 11 out of 18 text-based tasks, and we present comprehensiveevaluations of safety and responsibility aspects of the models, alongside a detailed description of modeldevelopment. We believe the responsible release of LLMs is critical for improving the safety of frontiermodels, and for enabling the next wave of LLM innovations.IntroductionWe present Gemma, a family of open modelsbased on Google’s Gemini models (Gemini Team,2023).We trained Gemma models on up to 6T to-kens of text, using similar architectures, data,and training recipes as the Gemini model family.Like Gemini, these models achieve strong gener-alist capabilities in text domains, alongside state-of-the-art understanding and reasoning skills atscale. With this work, we release both pre-trainedand fine-tuned checkpoints, as well as an open-source codebase for inference and serving.Gemma comes in two sizes: a 7 billion param-eter model for efficient deployment and develop-ment on GPU and TPU, and a 2 billion param-eter model for CPU and on-device applications.Each size is designed to address different compu-tational constraints, applications, and developerrequirements. At each scale, we release raw, pre-trained checkpoints, as well as checkpoints fine-tuned for dialogue, instruction-following, help-fulness, and safety. We thoroughly evaluate theshortcomings of our models on a suite of quantita-tive and qualitative benchmarks. We believe therelease of both pretrained and fine-tuned check-points will enable thorough research and inves-tigation into the impact of current instruction-tuning regimes, as well as the development ofincreasingly safe and responsible model develop-ment methodologies.Gemma advances state-of-the-art performancerelative to comparable-scale (and some larger),open models (Almazrouei et al., 2023; Jianget al., 2023; Touvron et al., 2023a,b) across awide range of domains including both automatedbenchmarks and human evaluation. Example do-mains include question answering (Clark et al.,2019; Kwiatkowski et al., 2019), commonsensereasoning (Sakaguchi et al., 2019; Suzgun et al.,2022), mathematics and science (Cobbe et al.,2021; Hendrycks et al., 2020), and coding (Austinet al., 2021; Chen et al., 2021). See complete de-tails in the Evaluation section.Like Gemini, Gemma...
