Hugging Face Releases Open-Source Visual Language Generative AI Model Idefics2
Hugging Face has unveiled a new eight-billion parameter generative AI model called Idefics2 that centers on processing and explaining images. Idefics2 can compose text responses to questions about visual content, describe images, and even create stories grounded in multiple images.
Idefics2
As indicated by the name, Idefics2 is a successor to Idefics1. Hugging Face used the openly licensed Mistral-7B and siglip-so400m models to create Idefics2. The new iteration improves on the model’s optical character recognition (OCR) capabilities and has native image resolution handling up to 980×980 pixels without fixed resizing, as well as optional sub-image splitting for very large images. Hugging Face also cited the model’s ability to extract information from documents and solve basic arithmetic problems as impressive aspects of its functions.
“Idefics2 is a strong foundation for the community working on multimodality. Its performance on Visual Question Answering benchmarks is top of its class size, and competes with much larger models such as LLava-Next-34B and MM1-30B-chat,” Hugging Face researchers explained in a blog post. “All of these improvements along with better pre-trained backbones yield a significant jump in performance over Idefics1 for a model that is 10x smaller.”
Idefics2 was pre-trained on a mix of openly available datasets, including web documents, image captions, OCR data, and image-to-code mappings. To further boost performance on specific tasks, it underwent instruction fine-tuning on “The Cauldron” – Hugging Face’s newly released open collection of 50 manually curated datasets designed for multi-turn conversations. Idefics2 can be further fine-tuned for an array of applications and is accessible from Hugging Face’s Transformers library and the Hugging Face Hub under an open Apache 2.0 license.
Hugging Face has been steadily pumping out open-source generative AI models that offer similar or even better versions of features from the likes of OpenAI or Google. Idefics2 fits right into the library with coding assistant Starcoder 2 and the customizable Hugging Chat Assistants. The $235 million raised by Hugging Face last summer has supported plenty of experiments and variations on large language models. The expansion into images will help entice developers who prefer Hugging Face’s licensing but have hesitated over limited multimodal options.
Follow @voicebotaiFollow @erichschwartz
Hugging Face’s StarCoder 2 Enhances Open-Source Generative AI Coding Assistant
Hugging Face Launches Free Customizable Generative AI Chatbots
