Transformer¤ÈMamba2¤Î¥Ï¥¤¥Ö¥ê¥Ã¥É¤È¤Ê¤ë¾®µ¬ÌϸÀ¸ì¥â¥Ç¥ë¡ÖZamba2-7B¡×¤¬¸ø³«¤µ¤ì¤ë
¥¢¥á¥ê¥«¤ÎAI¥¹¥¿¡¼¥È¥¢¥Ã¥×¤Ç¤¢¤ëZyphra¤¬¡¢¼«Á³¸À¸ì½èÍý¥â¥Ç¥ë¤Î¡ÖZamba2-7B¡×¤ò¥ê¥ê¡¼¥¹¤·¤Þ¤·¤¿¡£Zyphra¤Ï¡¢Zamba2-7B¤¬Google¤ÎGemma¤äMeta¤ÎLlama 3¥·¥ê¡¼¥º¤ò¾å²ó¤ë¥Ñ¥Õ¥©¡¼¥Þ¥ó¥¹¤òȯ´ø¤¹¤ë¤È¥¢¥Ô¡¼¥ë¤·¤Æ¤¤¤Þ¤¹¡£
Zyphra is excited to release Zamba2-7B
https://www.zyphra.com/post/zamba2-7b
https://huggingface.co/Zyphra/Zamba2-7B
Zamba2-7B¤Ï¡¢½¾Íè¤Î¸À¸ì¥â¥Ç¥ë¤Ç¹¤¯»ÈÍѤµ¤ì¤Æ¤¤¤ëTransformer¥¢¡¼¥¥Æ¥¯¥Á¥ã¤È2023ǯ12·î¤Ëȯɽ¤µ¤ì¤¿Mamba¥¢¡¼¥¥Æ¥¯¥Á¥ã¤Î¥Ï¥¤¥Ö¥ê¥Ã¥É¤Ç¤¢¤ë¡ÖZamba¥¢¡¼¥¥Æ¥¯¥Á¥ã¡×¤ÇÀ߷פµ¤ì¤Æ¤¤¤Þ¤¹¡£
¤Þ¤¿¡¢Zamba2-7B¤ÏÁ°¥â¥Ç¥ë¤Ç¤¢¤ëZamba1¤«¤é¿Ê²½¤·¡¢Mamba1¥Ö¥í¥Ã¥¯¤¬Mamba2¥Ö¥í¥Ã¥¯¤ËÃÖ¤´¹¤¨¤é¤ì¡¢¤µ¤é¤Ë2¤Ä¤Î¸ò¸ß¤ËÇÛÃÖ¤µ¤ì¤¿¶¦Í¥¢¥Æ¥ó¥·¥ç¥ó¥Ö¥í¥Ã¥¯¤òƳÆþ¤·¤Æ¤¤¤Þ¤¹¡£¤³¤Î¡Ö¶¦Í¡×¤È¤Ï¡¢Æ±¤¸½Å¤ß¤ò»ý¤Ä¥¢¥Æ¥ó¥·¥ç¥ó¥Ö¥í¥Ã¥¯¤¬¥â¥Ç¥ëÆâ¤ÎÊ£¿ô¤Î¾ì½ê¤ÇºÆÍøÍѤµ¤ì¤ë¤³¤È¤ò°ÕÌ£¤·¤Þ¤¹¡£¤³¤Î¶¦Í¥¢¥×¥í¡¼¥Á¤Ë¤è¤ê¡¢¥â¥Ç¥ë¤ÎÁ´ÂÎŪ¤Ê¥Ñ¥é¥á¡¼¥¿¡¼¿ô¤òÍÞ¤¨¤Ä¤Ä¡¢Transformer¥¢¡¼¥¥Æ¥¯¥Á¥ã¤Îµ¡Ç½¤ò³èÍѤ¹¤ë¤³¤È¤¬¤Ç¤¡¢¥â¥Ç¥ë¥µ¥¤¥º¤È¥Ñ¥Õ¥©¡¼¥Þ¥ó¥¹¤Î¥Ð¥é¥ó¥¹¤ò¼è¤ë¤³¤È¤¬¤Ç¤¤ë¤ÈZyphra¤ÏÀâÌÀ¤·¤Æ¤¤¤Þ¤¹¡£
¸úΨÀ¤ò¹â¤á¤ë¤¿¤á¡¢Â絬ÌϸÀ¸ì¥â¥Ç¥ë¤Î¥Õ¥¡¥¤¥ó¥Á¥å¡¼¥Ë¥ó¥°¤Ë»È¤ï¤ì¤ëLoRA(Low-Rank Adaptation)¥×¥í¥¸¥§¥¯¥¿¡¼¤ò³Æ¶¦Í¿Áإѡ¼¥»¥×¥È¥í¥ó(MLP)¤È¥¢¥Æ¥ó¥·¥ç¥ó¥Ö¥í¥Ã¥¯¤ËŬÍѤ·¡¢¥Í¥Ã¥È¥ï¡¼¥¯¤Î¿¼¤µ¤Ë±þ¤¸¤¿Æü첽¤ò²Äǽ¤Ë¤·¤Æ¤¤¤Þ¤¹¡£¤Þ¤¿¡¢¶¦Í¥¢¥Æ¥ó¥·¥ç¥ó¥ì¥¤¥ä¡¼¤ËRotary Position Embeddings¤òƳÆþ¤¹¤ë¤³¤È¤Ç¡¢Àǽ¤ò¤µ¤é¤Ë¸þ¾å¤µ¤»¤Æ¤¤¤Þ¤¹¡£
Zamba2-7B¤ÏÌó74²¯¤Î¥Ñ¥é¥á¡¼¥¿¤ò»ý¤Á¡¢2Ãû¥È¡¼¥¯¥ó¤Î¥Æ¥¥¹¥È¤È¥³¡¼¥É¥Ç¡¼¥¿¤Ç»öÁ°³Ø½¬¤ò¹Ô¤¤¡¢¤½¤Î¸åÌó1000²¯¤Î¹âÉʼÁ¥È¡¼¥¯¥ó¤òÍѤ¤¤ÆÄɲäγؽ¬¥Õ¥§¡¼¥º¤ò¼Â»Ü¤·¤Æ¤¤¤ë¤È¤Î¤³¤È¡£¤³¤Î·ë²Ì¡¢Zamba2-7B(ÀÄ)¤Ï8B°Ê²¼¤Î¥Ñ¥é¥á¡¼¥¿¤ò»ý¤Ä¥â¥Ç¥ë¤ÎÃæ¤Ç¡¢Mistral-7B(ÀÖ)¤äGemma 7B(²«)¡¢Llama 3.1-8B(ÎÐ)¤Ê¤É¤ò¾å²ó¤ëÀǽ¤ò¼¨¤·¤Æ¤¤¤ë¤ÈZyphra¤Ï¼çÄ¥¤·¤Æ¤¤¤Þ¤¹¡£
Æäˡ¢Zyphra¤ÏZamba2-7B¤Î¿äÏÀ¸úΨ¤ËÃíÌÜ¡£°Ê²¼¤Î¥°¥é¥Õ¤Ç¼¨¤µ¤ì¤Æ¤¤¤ëÄ̤ꡢZamba2-7B(ÀÖ)¤Ï¡¢½¾Íè¤ÎTransformer¥Ù¡¼¥¹¤Î¥â¥Ç¥ë¤ÈÈæ³Ó¤·¤ÆÂçÉý¤Ë¾®¤µ¤¤¥á¥â¥ê¥Õ¥Ã¥È¥×¥ê¥ó¥È¤Ç¹â®¤ÊÀ¸À®¤¬²Äǽ¤È¤Ê¤Ã¤Æ¤ª¤ê¡¢¥È¡¼¥¯¥óÀ¸À®¤Þ¤Ç¤Î»þ´Ö(º¸)¤¬25¡óû½Ì¡¢¥È¡¼¥¯¥óÀ¸À®Â®ÅÙ(±¦)¤¬20¡ó¸þ¾å¤·¤Æ¤¤¤Þ¤¹¡£
Zamba2-7B¤ÏApache 2.0¥é¥¤¥»¥ó¥¹¤Ç¥ª¡¼¥×¥ó¥½¡¼¥¹¤È¤·¤Æ¸ø³«¤µ¤ì¤Æ¤ª¤ê¡¢¸¦µæ¼Ô¤ä³«È¯¼Ô¡¢´ë¶È¤¬¼«Í³¤ËÍøÍѤǤ¤ë¤è¤¦¤Ë¤Ê¤Ã¤Æ¤¤¤Þ¤¹¡£Zephra¤Ï¡¢Zamba2-7B¤¬¾®µ¬ÌϤʤ¬¤é¹âÀǽ¤Ç¸úΨŪ¤Ê¸À¸ì¥â¥Ç¥ë¤È¤·¤Æ¡¢¥ª¥ó¥Ç¥Ð¥¤¥¹½èÍý¤ä¾ÃÈñ¼Ô¸þ¤±GPU¾å¤Ç¤Î¼Â¹Ô¡¢¤µ¤é¤Ë¤Ï¿¤¯¤Î¥¨¥ó¥¿¡¼¥×¥é¥¤¥º¥¢¥×¥ê¥±¡¼¥·¥ç¥ó¤ËŬ¤·¤¿ÁªÂò»è¤Ë¤Ê¤ë¤È½Ò¤Ù¤Þ¤·¤¿¡£