mistral-ocr-latest
Model Overview
This Optical Character Recognition API from Mistral sets a new standard in document understanding. Unlike other models, Mistral OCR comprehends each element of documents—media, text, tables, equations—with unprecedented accuracy and cognition. It takes images and PDFs as input and extracts content in an ordered interleaved text and images.
Maximum file size: 50 MB.
Maximum number of pages: 1000.
Note that this OCR does not preserve character formatting: bold, underline, italics, monospace text, etc. However, it preserves footnotes (superscript text).
Setup your API Key
If you don’t have an API key for the AI/ML API yet, feel free to use our Quickstart guide.
How to Make a Call
Copy the code from one of the examples below, depending on whether you want to process an image or a PDF.
Replace
<YOUR_AIMLAPI_KEY>with your AIML API key from your personal account.Replace the URL of the document or image with the one you need.
If you need to use different parameters, refer to the API schema below for valid values and operational logic.
Save the modified code as a Python file and run it in an or via the console.
API Schema
Document to run OCR
Specific pages you wants to process
"3" or "0-2" or [0, 3, 4]Include base64 images in response
Max images to extract
Minimum height and width of image to extract
POST /v1/ocr HTTP/1.1
Host: api.aimlapi.com
Authorization: Bearer <YOUR_AIMLAPI_KEY>
Content-Type: application/json
Accept: */*
Content-Length: 213
{
"model": "mistral/mistral-ocr-latest",
"document": {
"type": "document_url",
"document_url": "https://example.com"
},
"pages": "\"3\" or \"0-2\" or [0, 3, 4]",
"include_image_base64": true,
"image_limit": 1,
"image_min_size": 1
}No content
Example #1: Text Recognition From an Image
We’ve found a photo of a short handwritten text for OCR testing and will be passing it to the model via URL:
import requests
def main():
response = requests.post(
"https://api.aimlapi.com/v1/ocr",
headers={
"Authorization": "Bearer <YOUR_AIMLAPI_KEY>",
"Content-Type": "application/json",
},
json={
"document": {
"type": "image_url",
"image_url": "https://i.redd.it/hx0v4fj979k51.jpg"
},
"model": "mistral/mistral-ocr-latest",
},
)
# response.raise_for_status()
data = response.json()
# print(data)
return data
main()Example #2: Process a PDF File
Let's process a PDF file from the internet using the described model:
import requests
def main():
response = requests.post(
"https://api.aimlapi.com/v1/ocr",
headers={
"Authorization": "Bearer <YOUR_AIMLAPI_KEY>",
"Content-Type": "application/json",
},
json={
"document": {
"type": "document_url",
"document_url": "https://css4.pub/2015/textbook/somatosensory.pdf"
},
"model": "mistral/mistral-ocr-latest",
},
)
response.raise_for_status()
data = response.json()
print(data)
if __name__ == "__main__":
main()
Example #3: Process a PDF File And Parse the Response
As you can see above, the model returns markdown containing the recognized text with formatting elements preserved (headings, italics, bold text, etc.), along with the location of images within the text and the images themselves in base64 format, if you have enabled the corresponding option include_image_base64. However, the markdown is returned as a string with newline characters and other string attributes, so you might need to parse the output separately to get clean markdown containing only the formatted text and images. In this example, we’ve written code that make it for us.
import os
import re
import base64
import requests
def ocr_process():
response = requests.post(
"https://api.aimlapi.com/v1/ocr",
headers={
"Authorization": "Bearer <YOUR_AIMLAPI_KEY>",
"Content-Type": "application/json",
},
json={
"document": {
"type": "document_url",
"document_url": "https://zovi0.github.io/public_misc/test-PDF-2.pdf"
},
"model": "mistral/mistral-ocr-latest",
"include_image_base64": True,
"image_limit": 5
},
)
data = response.json()
print(data)
return data
def parse_ocr_output(ocr_output):
output_dir = "output_images"
os.makedirs(output_dir, exist_ok=True)
all_markdown = []
for page in ocr_output.get("pages", []):
md = page["markdown"]
images = {img["id"]: img["image_base64"] for img in page.get("images", []) if img.get("image_base64")}
def replace_image(match):
image_id = match.group(1)
base64_data = images.get(image_id)
if not base64_data:
return match.group(0) # Leave original markdown if no image data
# Detect image format
img_match = re.match(r"data:image/(png|jpeg|jpg);base64,(.*)", base64_data)
if not img_match:
return match.group(0)
img_format, img_b64 = img_match.groups()
ext = "jpg" if img_format in ["jpeg", "jpg"] else "png"
filename = f"{image_id}.{ext}"
filepath = os.path.join(output_dir, filename)
with open(filepath, "wb") as f:
f.write(base64.b64decode(img_b64))
return f""
# Replace image links in markdown with local image links
md = re.sub(r"!\[.*?\]\((img-\d+\.\w+)\)", replace_image, md)
all_markdown.append(md)
# Combine pages with spacing
final_md = "\n\n---\n\n".join(all_markdown)
with open("output.md", "w", encoding="utf-8") as f:
f.write(final_md)
print("Markdown and images saved.")
return final_md
if __name__ == "__main__":
ocr_output = ocr_process()
parse_ocr_output(ocr_output)
Response after parsing
Contents of the output.md file:
# Characteristics of plant cells
Plant cells have cell walls composed of cellulose, hemicelluloses, and pectin and constructed outside the cell membrane. Their composition contrasts with the cell walls of fungi, which are made of chitin, of bacteria, which are made of peptidoglycan and of archaea, which are made of pseudopeptidoglycan. In many cases lignin or suberin are secreted by the protoplast as secondary wall layers inside the primary cell wall. Cutin is secreted outside the primary cell wall and into the outer layers of the secondary cell wall of the epidermal cells of leaves, stems and other above-ground organs to form the plant cuticle. Cell walls perform many essential functions. They provide shape to form the tissue and organs of the plant, and play an important role in intercellular communication and plant-microbe interactions. ${ }^{[1]}$ The cell wall is flexible during growth and has small pores called plasmodesmata that allow the exchange of nutrients and hormones between cells. ${ }^{[2]}$Content of output_images subfolder
How it looks in any Markdown viewer:
It looks almost like the original PDF, but all the text has been recognized, and the markdown is easy to use further, for example, to embed in a web page. Enjoy!
Last updated
Was this helpful?