Vision Capabilities

In this tutorial, you'll learn how to:

• Identify which Vision Language Models (VLMs) are available using our models API
• Chat with open source VLMs using our chat completions API

VLMs are identified in our models API with a task type of image-text-to-text.

Images are made available to the model in two main ways:

• passing a link to the image in a user message
• passing the base64 encoded image directly in a user message

Note

Make sure you have set the TELNYX_API_KEY environment variable

import os

from openai import OpenAI


client = OpenAI(
    api_key=os.getenv("TELNYX_API_KEY"),
    base_url="https://api.telnyx.com/v2/ai"
)
image_url = "https://upload.wikimedia.org/wikipedia/commons/3/3a/Llama%2C_peru%2C_machu_picchu.jpg"
chat_completion = client.chat.completions.create(
    model="OpenGVLab/InternVL2-Llama3-76B",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What's in this image?"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": image_url,
                    },
                },
            ],
        }
    ],
)
print(chat_completion)

Here's the same example using base64 encoding

import base64
import os
import requests

from openai import OpenAI


client = OpenAI(
    api_key=os.getenv("TELNYX_API_KEY"),
    base_url="https://api.telnyx.com/v2/ai"
)
image_url = "https://upload.wikimedia.org/wikipedia/commons/3/3a/Llama%2C_peru%2C_machu_picchu.jpg"

def encode_image_base64_from_url(image_url: str) -> str:
    """Encode an image retrieved from a remote url to base64 format."""

    with requests.get(image_url) as response:
        response.raise_for_status()
        result = base64.b64encode(response.content).decode('utf-8')

    return f"data:image/jpeg;base64,{result}"


image_base64 = encode_image_base64_from_url(image_url=image_url)
chat_completion = client.chat.completions.create(
    model="OpenGVLab/InternVL2-Llama3-76B",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What's in this image?"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": image_base64,
                    },
                },
            ],
        }
    ],
)
print(chat_completion)

Either one of these will output something like the following

The image shows a llama standing on a rocky outcrop overlooking the ruins of Machu Picchu, an ancient Incan city in Peru. The llama is looking out over the ruins, which are surrounded by lush green mountains and vegetation. The scene captures the historic and scenic beauty of the location.