[MODEL REQUEST] Kosmos-2.5

[EwoutH]

Kosmos-2.5 is an relatively small (1.37B params), generative model for machine reading of text-intensive images.

Details of model being requested

• Model name: Kosmos-2.5
• Source repo link: https://huggingface.co/microsoft/kosmos-2.5
• Research paper link: https://arxiv.org/abs/2309.11419
• Model use case: Kosmos-2.5 is a multimodal literate model for machine reading of text-intensive images. Pre-trained on large-scale text-intensive images, Kosmos-2.5 excels in two distinct yet cooperative transcription tasks: (1) generating spatially-aware text blocks, where each block of text is assigned its spatial coordinates within the image, and (2) producing structured text output that captures styles and structures into the markdown format. This unified multimodal literate capability is achieved through a shared decoder-only auto-regressive Transformer architecture, task-specific prompts, and flexible text representations. We evaluate Kosmos-2.5 on end-to-end document-level text recognition and image-to-markdown text generation. Furthermore, the model can be readily adapted for any text-intensive image understanding task with different prompts through supervised fine-tuning, making it a general-purpose tool for real-world applications involving text-rich images. This work also paves the way for the future scaling of multimodal large language models.

In addition to the full precision F32 model, might a quantized version also be very useful.

[mestrona-3]

Hi @EwoutH, thank you for the feature request and details! I've added it to our list of requested models.