Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs-website/docs/concepts/data-classes.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,12 @@ image = ByteStream.from_file_path("dog.jpg")

Read the detailed documentation for the `ChatMessage` data class on a dedicated [ChatMessage](data-classes/chatmessage.mdx) page.

### ImageContent

`ImageContent` represents image-based content used in multimodal chat messages and vision-language pipelines.

Read the detailed documentation for the `ImageContent` data class on the dedicated [ImageContent](./data-classes/imagecontent) page.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please fix the error related to the link

[ERROR] Error: Unable to build website for locale en.
    at tryToBuildLocale (/vercel/path1/node_modules/@docusaurus/core/lib/commands/build/build.js:76:15)
    at async /vercel/path1/node_modules/@docusaurus/core/lib/commands/build/build.js:34:9
    ... 4 lines matching cause stack trace ...
    at async file:///vercel/path1/node_modules/@docusaurus/core/bin/docusaurus.mjs:44:3 {
  [cause]: Error: Docusaurus found broken links!
  
  Please check the pages of your site in the list below, and make sure you don't reference any path that does not exist.
  Note: it's possible to ignore broken links with the 'onBrokenLinks' Docusaurus configuration, and let the build pass.
  
  Exhaustive list of all broken links found:
  - Broken link on source page path = /docs/next/data-classes:
     -> linking to ./data-classes/imagecontent (resolved as: /docs/next/data-classes/imagecontent)


### Document

#### Overview
Expand Down
213 changes: 213 additions & 0 deletions docs-website/docs/concepts/data-classes/imagecontent.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,213 @@
---

title: "ImageContent"
id: imagecontent
slug: "/imagecontent"
description: "`ImageContent` represents image-based content in Haystack chat messages and multimodal pipelines."
----------------------------------------------------------------------------------------------------------------

# ImageContent

`ImageContent` is a Haystack data class used to represent image-based content in chat messages and multimodal AI pipelines.

It is commonly used with:

* multimodal LLMs
* vision-language models
* image-aware chat applications
* document/image processing workflows

`ImageContent` stores images as base64-encoded strings together with metadata such as MIME type and image detail level.

If you are looking for the full API reference, see the [API documentation](/reference/data-classes-api#imagecontent).

---

# Creating ImageContent

You can create an `ImageContent` object directly from a base64 string:

```python
from haystack.dataclasses import ImageContent

image = ImageContent(
base64_image="your_base64_encoded_image",
mime_type="image/png"
)

print(image)
```

---

# Loading Images from a File Path

The `from_file_path()` class method provides a convenient way to load local image files.

```python
from haystack.dataclasses import ImageContent

image = ImageContent.from_file_path(
"sample.png",
detail="low"
)

print(image)
```

The optional `detail` parameter is currently supported by OpenAI vision models and accepts:

* `"auto"`
* `"high"`
* `"low"`

You can also resize images while loading:

```python
image = ImageContent.from_file_path(
"sample.png",
size=(512, 512)
)
```

This helps reduce:

* memory usage
* processing time
* payload size

when working with multimodal LLM APIs.

---

# Loading Images from a URL

You can also create an `ImageContent` object directly from an image URL:

```python
from haystack.dataclasses import ImageContent

image = ImageContent.from_url(
"https://images.unsplash.com/photo-1546182990-dffeafbe841d",
detail="low"
)

print(image)
```

Internally, Haystack downloads the image and converts it into a base64 representation.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add a quick section about using Pipeline components to produce ImageContent: ImageFileToImageContent and PDFToImageContent with links.

---

# Using ImageContent with ChatMessage

`ImageContent` is commonly used together with `ChatMessage` for multimodal conversations.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's link the ChatMessage documentation page


```python
from haystack.dataclasses import ChatMessage, ImageContent

image = ImageContent.from_url(
"https://images.unsplash.com/photo-1546182990-dffeafbe841d",
detail="low"
)

message = ChatMessage.from_user(
content_parts=[
"What does this image show?",
image
]
)

print(message)
```

This allows multimodal LLMs to process both:

* textual prompts
* image inputs

within the same message.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's also add a more complex example with Jinja2 string templates: see https://docs.haystack.deepset.ai/docs/chatpromptbuilder#multimodal

---

# Metadata

The optional `meta` parameter allows you to attach custom metadata to the image.

```python
image = ImageContent.from_url(
"https://example.com/image.png",
meta={"source": "example-dataset"}
)
```

This can be useful for:

* tracing
* dataset tracking
* workflow metadata
* custom application logic

---

# Validation

By default, `ImageContent` validates:

* base64 encoding
* MIME type correctness
* image MIME compatibility

Validation can be disabled to improve performance:

```python
image = ImageContent(
base64_image="your_base64_encoded_image",
mime_type="image/png",
validation=False
)
```

---

# Serialization

`ImageContent` supports dictionary serialization.

```python
image_dict = image.to_dict()

restored_image = ImageContent.from_dict(image_dict)
```

---

# Displaying Images

The `show()` method can display images directly in:

* Jupyter notebooks
* local desktop environments

```python
image.show()
```

This requires the `Pillow` package:

```bash
pip install pillow
```

---

# Related Components

`ImageContent` is frequently used with:

* `ChatMessage`
* multimodal chat generators
* image converters
* vision-language pipelines
* `PDFToImageContent`
* `ImageFileToImageContent`
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's also add links here

Loading