BLIP Explained: Use It For VQA & Captioning

For many years, computer vision and natural language processing evolved along separate paths, since unifying both modalities in a single model demanded a radical rethink of model architecture, which seemed out of reach.

Thanks to recent advances in transformer architectures, this barrier has been overcome, giving rise to a new AI field that merges vision and language: Vision‑Language Models (VLMs).

One of the pioneering VLMs is BLIP (Bootstrapping Language‑Image Pre‑training), which serves as the foundation for many of today’s multimodal systems.

In this blog, we’ll take an in‑depth look at BLIP, showing you how to set it up and guiding you through a range of experiments to explore its capabilities.

What is a Vision Language Model

A Vision‑Language Model (VLM) is a type of AI system that learns to process and relate both visual and textual information within a single architecture.

By training on large collections of paired images and captions, VLMs acquire joint representations that can be applied to a wide range of multimodal tasks, such as:

• Image captioning: Generating natural‑language descriptions of images.
• Visual question answering (VQA): Answering free‑form questions about image content.
• Cross‑modal retrieval: Finding images based on text queries or vice versa.
• Grounded language understanding: Linking words or phrases to specific regions in an image.

The big change that made VLMs possible is called multimodal pre‑training. In simple terms, it means teaching the model on both images and their matching text captions from the start, so it learns to mix visual details and language ideas in one go.

Some Early VLMs That Built the Foundation

• CLIP (Contrastive Language–Image Pre‑training)
  Trained by OpenAI in 2021, CLIP learned to match images and text by pushing correct pairs together and wrong pairs apart.
  It can recognize things it wasn’t explicitly trained on.
• ALIGN (A Larger‑scale ImaGe–text NeTwork)
  Created by Google Research also in 2021, ALIGN did the same trick as CLIP but used even more image‑text pairs, helping it excel at finding and classifying images.
• ViLBERT (Vision-and‑Language BERT)
  Launched by Facebook AI in 2019, ViLBERT splits its brain into two parts, one for pictures and one for text and then let them talk to each other using special attention layers.
• LXMERT (Learning Cross‑Modality Encoder Representations from Transformers)
  Another 2019 model from Facebook AI, LXMERT, used separate transformers for vision and language and then merged their outputs with cross‑attention to solve tasks like visual question answering.
• UNITER (UNiversal Image‑TExt Representation)
  Also, from 2019, UNITER combined several training tasks (like guessing missing words or checking if an image and caption match) in one transformer, boosting performance across many vision‑language tests.

Architecture of BLIP

BLIP’s architecture is built around three main components: an image encoder, a text encoder, and a lightweight multimodal decoder—tied together by complementary pre‑training tasks.

Here’s how it all fits:

1. Image Encoder
   • Usually a Vision Transformer (ViT) or a CNN backbone (e.g., ResNet).
   • Takes an input image and transforms it into a fixed‑length vector (or a set of patch tokens, in the ViT case).
2. Text Encoder
   • A Transformer‑based language model (like BERT).
   • Converts input text (captions, questions, etc.) into a sequence of token embeddings plus a special “[CLS]” summary token.
3. Multimodal Decoder
   • A lighter Transformer that attends to both image and text embeddings.
   • Used for tasks requiring generation (e.g., image captioning) or for combining both modalities when scoring image‑text pairs.

Experimenting with BLIP

Let's implement BLIP by providing it with an image and various questions related to the image to evaluate its image captioning capabilities.

SAMPLE 1: A IMAGE OF GOATS IN THE HILL

Question: What is the weather in this image?
Answer: sunny

Question: how many animals in this image?
Answer: 3

Question: which animal is in the image?
Answer: sheep

Question: what type of terrain in the image?
Answer: hilly

Question: any flowers in the image?
Answer: no

Question: which time of day it is
Answer: afternoon

---

SAMPLE 2: A IMAGE OF POSTER WALL

Question: Numbers of posters in this image
Answer: 3

Question: Name of the device in this image
Answer: record player

Question: On right-side poster, what is written on it?
Answer: music ---> X

Question: Any plant in this image?
Answer: yes

---

SAMPLE 3: A IMAGE OF COINS

Question: Numbers of coins in this image
Answer: 12 ---> X

Question: what is the color of coins in this image
Answer: silver

Question: Value written on the coin
Answer: 1. 00

Question: which currency does the coins belong to?
Answer: british ---> X

Question: which currency is written on the coin?
Answer: pounds ---> X

---

SAMPLE 4: A IMAGE OF VAN IN A BEACH

Question: which vehicle is in the image?
Answer: van

Question: what is the color of the vehicle?
Answer: blue

Question: what is the brand of vehicle?
Answer: volkswagen

Question: Numbers of person in the image?
Answer: 4 ---> X

Question: where is the persons in the image?
Answer: top of van

Question: which place is in the image?
Answer: beach

Question: what time of day is in the image
Answer: afternoon

Question: what is the van plate vehicle ID?
Answer: vw ---> X

How to run BLIP on your local device

To run BLIP locally, follow these steps:

Step 1: Install the required libraries.

%pip install torch transformers pillow
%pip install accelerate

Step 2: Import the following libraries

import torch
from transformers import AutoProcessor, AutoModelForVisualQuestionAnswering
import requests
from PIL import Image
from io import BytesIO
from IPython.display import display
import os
from transformers.utils import logging

# Suppress unnecessary logs
logging.set_verbosity_error()

Step 3: (Optional) Create a helper function to visualize the image using a URL.

def show_image(source):
    """
    Display an image from a URL or a local file path.

    Args:
        source (str): The URL or local file path of the image.
    """
    try:
        if source.startswith("http://") or source.startswith("https://"):
            # Load image from URL
            response = requests.get(source)
            response.raise_for_status()  # Raise exception for bad response
            img = Image.open(BytesIO(response.content))
        elif os.path.exists(source):
            # Load image from local file path
            img = Image.open(source)
        else:
            raise ValueError("Invalid source. Provide a valid URL or local file path.")
        
        display(img)
    
    except Exception as e:
        print(f"Error displaying image: {e}")

Step 4: Creating a function to implement BLIP using HuggingFace.

Here, I am using the BLIP model for Visual Questioning Answer called Salesforce/blip-vqa-base . You can choose any model from the various available models in Huggingface for various other tasks.

def blip(ques: str, img_url: str) -> str:
    """    Perform visual question answering using the BLIP model."""
    processor = AutoProcessor.from_pretrained("Salesforce/blip-vqa-base")
    model = AutoModelForVisualQuestionAnswering.from_pretrained(
        "Salesforce/blip-vqa-base", 
        torch_dtype=torch.float16,
        device_map="auto"
    )
    image = Image.open(requests.get(img_url, stream=True).raw)

    question = ques
    inputs = processor(images=image, text=question, return_tensors="pt").to("cuda", torch.float16)

    output = model.generate(**inputs)
    answer = processor.batch_decode(output, skip_special_tokens=True)[0]
    return answer

Step 5: Implementing BLIP

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
show_image(url)

blip("What is the weather in this image?", url)

Conclusion

BLIP represents a major milestone in the evolution of vision‑language AI. By combining an image encoder, a text encoder, and a lightweight multimodal decoder, and by training on well‑designed contrastive, matching, and generative objectives.

BLIP demonstrated that a single model could learn to both understand and describe visual content. This breakthrough paved the way for richer, more scalable approaches to multimodal learning.

However, BLIP is not without its limitations. Its reliance on paired image‑text data means it can struggle in domains where high‑quality captions are scarce or noisy.

Although BLIP’s modular design allows fine‑tuning on downstream tasks, its performance on highly specialized benchmarks (e.g., fine‑grained medical imaging or low‑resource languages) can lag behind more task‑focused systems.

Despite these challenges, BLIP set the conceptual and architectural foundation for next‑generation Vision‑Language Models.

Newer VLMs build on BLIP by using larger, cleaner datasets, tighter image‑text links, and richer training. These models can identify more images without fine‑tuning, link words to image parts, and generate more natural captions.

With better multimodal data and smarter methods ahead, VLMs will soon fully match human‑like scene understanding.

FAQs

References

Research Paper

Notebook