Skip to content

[Gemini adapter] Support message content with image and video URLs#81

Open
samrat wants to merge 1 commit into
thmsmlr:mainfrom
samrat:gemini-video
Open

[Gemini adapter] Support message content with image and video URLs#81
samrat wants to merge 1 commit into
thmsmlr:mainfrom
samrat:gemini-video

Conversation

@samrat

@samrat samrat commented Nov 12, 2024

Copy link
Copy Markdown

Fixes #80

The Gemini API expects you to upload images/videos separately. That feels outside of the scope of Instructor, so I've avoided adding that in this PR.

Example usage:

defmodule VideoDesc do
  use Ecto.Schema

  @primary_key false
  embedded_schema do
    field(:video_description, :string)
    field(:oscar_winning_moment, :boolean)
  end
end

Instructor.chat_completion(
  mode: :json_schema,
  model: "gemini-1.5-flash",
  response_model: VideoDesc,
  messages: [
    %{
      role: "user", 
      content: [
        %{
          type: "video_url",
          video_url: %{
            url: "https://generativelanguage.googleapis.com/v1beta/files/d2ngu357dmfx",
            mime_type: "video/mp4"
          }
        },
        %{
          type: "text",
          text: "What's going on in this video?"
        }
      ]
    }
  ]
)

@samrat

samrat commented Nov 12, 2024

Copy link
Copy Markdown
Author

The Gemini API expects you to upload images/videos separately

In case anyone wants to grab the code to do the upload, I've posted it here.

@thmsmlr

thmsmlr commented Feb 9, 2025

Copy link
Copy Markdown
Owner

ty! this looks great

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Multimodal support for Gemini

2 participants