Image Captioning using Deep Learning

Overview

This project implements an Image Captioning system using deep learning techniques. The goal is to generate descriptive textual captions for input images by combining computer vision and natural language processing (NLP) methods. The implementation is provided as a Jupyter Notebook for easy experimentation and visualization.

Table of Contents

• Overview
• Features
• Dataset
• Model Architecture
• Setup and Installation
• Usage
• Results
• License

Features

• End-to-end deep learning pipeline for image captioning using a Jupyter Notebook.
• Pre-trained CNN (ResNet50) for feature extraction.
• Caption generation via an LSTM decoder with an attention mechanism.
• Evaluation with BLEU, ROUGE, and METEOR scores for caption quality.

Dataset

The project uses the ROCO (Radiology Objects in COntext) dataset, which contains radiology images paired with their corresponding captions. This dataset is well-suited for generating text descriptions of medical images.

For more information on the ROCO dataset, please visit: ROCO Dataset

Model Architecture

graph LR
    Img["Input Radiology Image<br/>(ROCO dataset)"] --> CNN["CNN Encoder<br/>ResNet50 (pretrained)<br/>feature extraction"]
    CNN --> Attn["Attention Mechanism<br/>aligns image regions &#8596; words"]
    Attn --> LSTM["LSTM Decoder<br/>generates caption token-by-token"]
    LSTM --> Caption["Generated Caption"]
    Caption --> Eval["Evaluation<br/>BLEU &#183; ROUGE &#183; METEOR"]

Loading

The model consists of two main components:

1. Image Feature Extractor — a ResNet50 CNN, pre-trained on a large image dataset, extracts visual features from the input radiology image.
2. Caption Generator — an LSTM decoder with an attention mechanism generates the caption word-by-word, attending to different regions of the image feature map at each step.

Setup and Installation

To run this project locally, follow these steps:

1. Clone the repository:

git clone https://github.com/Ritik650/Image-Captioning-using-Deep-Learning.git
cd Image-Captioning-using-Deep-Learning

2. Create a virtual environment (optional but recommended):

python -m venv venv
source venv/bin/activate  # On Windows, use `venv\Scripts\activate`

3. Install Jupyter Notebook if it's not already installed:

pip install notebook

4. Launch Jupyter Notebook:

jupyter notebook

5. Open the provided Jupyter Notebook (e.g., Image_Captioning.ipynb) in your browser.

6. Ensure the ROCO dataset is downloaded and preprocessed. Update the dataset path in the notebook if necessary.

Usage

1. Run the Jupyter Notebook step-by-step to:

   • Preprocess the dataset.
   • Train the model.
   • Evaluate the model.
   • Generate captions for test images.

2. To test the model, upload your image and run the relevant cell in the notebook to generate captions.

Results

Evaluation metrics (ResNet50 + LSTM + Attention on the ROCO test set):

Metric	Score
BLEU	0.65
ROUGE	0.58
METEOR	0.60

License

This project is licensed under the MIT License. See the LICENSE file for details.