Structured Output & New Timeline API

v1.6 Alpha 1 Release Notes

⚠️ Please read the breaking changes before updating!

This update adds support for structured output and a new Timeline API.

⚠️ Breaking Changes

• The remember action is now called create_event to avoid confusion with LLM Vision Memory.

Contributors

A huge thank you to our contributors @rmaher001, and @boian-ivanov!

Integration

✨ Features

• Structured Output: Valid JSON output can now be forced using the structure parameter. See the docs for details. (by @rmaher001)
• Timeline API: Events stored in the timeline can now be fetched through the new Timeline API. This is used by the LLM Vision Cards, but you may use the API for your automations as well. See the docs for details. (by @valentinfrlch)

🔧 Improvements & Bug Fixes

• Create Events: The create_events action now supports setting the category for an event manually. (#536) (by @valentinfrlch)

🌐 Languages

• Support for Hungarian language: Added 🇧🇬 Bulgarian translations. (by @boian-ivanov)

Bug Fixes & Improvements

v1.5.2 Release Notes

⚠️ Please read the breaking changes before updating!

This update includes bug fixes and performance improvements.

Contributors

A huge thank you to our contributors @TheRealFalseReality, @bekdan1988, @alemuro, and everyone who helped test the pre-release versions!

⚠️ Breaking Changes

In our previous release (1.5.1), we moved snapshots to /config/media/llmvision/snapshots. This change caused issues for some users who were using other integrations that depend on the /media directory. As a result, the snapshots directory is moving again to /media/llmvision/snapshots.

What this means for you:

• Existing snapshots will automatically migrate to the new location when you restart after upgrading to 1.5.2.
• Remove the media folder configuration that you may have added to your configuration.yaml

  homeassistant:
    media_dirs:
      llmvision: /config/media/llmvision # Remove this line
      media: /media # Remove this line

• If you have any custom automations that use snapshots, update your template to:
  {{ response.key_frame.replace('/media','/media/local') }}
• If you use the Timeline Card, make sure to update it as well!

I apologize for any inconvenience this causes!

Integration

🌐 Languages

• Support for Hungarian language: Added 🇭🇺 Hungarian translations. (by @bekdan1988)
• Support for Catalan language: Added Catalan translations. (by @alemuro)

🔧 Improvements & Fixes

• Improved keyframe selection: Keyframe should now show the detected object more reliably. (by @valentinfrlch)
• Timeline events not deleted: Fixed a bug that caused events not being deleted properly. (by @valentinfrlch)
• Anthropic fix: Fixed an issue that caused validation to fail for the Anthropic provider. (#475) (by @valentinfrlch)
• Reduced image pipeline latency: Frames are now stored directly in memory instead of in a temporary directory. This reduces disk wear and reduces latency caused by I/O operations. It also fixes a race condition when running multiple actions in parallel. (#505, #510) (by @valentinfrlch)
• Moved Timeline snapshots: Snapshots are now stored securely behind Home Assistant authentication in /media. LLM Vision will attempt to migrate your previously captured snapshots to the new folder. (by @valentinfrlch)
• System Prompts: System prompts are now injected as actual system messages for every provider, which should improve their effectiveness. Also try out the updated default system and title prompts. (by @valentinfrlch)
• max_tokens now optional: max_tokens is now an optional parameter. Newer models use tokens for reasoning, which could result in empty responses if max_tokens was set too low. (by @valentinfrlch)

Blueprint

Make sure to update the blueprint by re-importing it!

A huge thank you to @TheRealFalseReality for maintaining the blueprint!

🔧 Improvements & Fixes

• New media folder: Updated all key_frame paths to match new media path. (by @TheRealFalseReality)

New Providers, Bug Fixes & Improvements

1.5.1 Release Notes

⚠️ Please read the breaking changes before updating!

This update adds native support for OpenRouter and Azure, fixes an issue with Google Gemini, adds support for GPT-5 models and adds Dutch and Polish translations.

Contributors

A huge thank you to our contributors @TheRealFalseReality, @NikNikovsky, @Minionguyjpro, @meceonzo—and of course everyone who helped test and provided feedback for the beta versions!

⚠️ Breaking Changes

• Timeline snapshots have moved from /www to /media. add the following to your configuration.yaml:

  # Add llmvision /media folder
  homeassistant:
    media_dirs:
      media: /media
      llmvision: /config/media/llmvision

  💡 Make sure to create the /media and the /llmvision inside it before restarting!
  💡 Update LLM Vision Card to the latest version! Otherwise snapshots won't show correctly.

• max_tokens is now displayed as a box type number selector rather than a slider. This change accommodates newer models that use thinking parameters counting toward max_tokens. Unlike the limited slider, the box allows for inputting much larger numbers.

  💡If you notice empty responses, consider increasing max_tokens.

Integration

✨ Features

• New Provider: LLM Vision now supports OpenRouter natively. (@valentinfrlch)
• New Provider: LLM Vision now supports Azure natively (#64, #103, #144). (@valentinfrlch)
• Support for Polish language: Added 🇵🇱 Polish translations. Thank you @NikNikovsky!
• Support for Dutch language: Added 🇳🇱 Dutch translations. Thank you @Minionguyjpro!

🔧 Improvements & Fixes

• Google Gemini fix: Fixed an infinite retry loop in the Google Gemini provider and corrected an issue where the fallback provider wasn't being used properly. (#398, #262), (@valentinfrlch)
• Support for GPT-5 models: Remove temperature and top_p from request when a gpt-5 model is used (#437), (@valentinfrlch)
• Moved Timeline snapshots: Snapshots are now stored securely behind Home Assistant authentication in /media. LLM Vision will attempt to migrate previously captured snapshots to the new folder.
• Remove max_tokens limit: max_tokens is now displayed as a box type number selector rather than a slider. This change allows for larger token limits, as thinking tokens count toward the max_tokens limit. (@valentinfrlch)
• Gemini key logging: Fixed a security issue that logged the Google API key in clear text (#334), (@valentinfrlch)
• Missing import: Add aiofile package to imports (#404), (@valentinfrlch)
• Sanitize IP address: Ignore any protocol input for the IP address field for Ollama and Open WebUI providers. Thank you @meceonzo!
• Fallback provider: Fixed a retry logic bug that caused an exception when no fallback provider was configured. (#461)
• Title generation: Fixed an issue that caused an exception when generate_title was not set. (#463)

Blueprint

Update the blueprint by re-importing it from the Blueprint settings page in Home Assistant.

https://github.com/valentinfrlch/ha-llmvision/blob/main/blueprints/event_summary.yaml

A huge thank you to @TheRealFalseReality for maintaining the blueprint!

✨ Features

• See snapshot quick action: Notification now shows button to preview the snapshot in a browser.

🔧 Improvements & Fixes

• Multiple Cameras: Fixed an issue that displayed the wrong camera (#403)
• Live Preview notification (iOS only): Preview mode can now be customized before and after analysis.
• Time format: Remove leading zero from 12-hour format.
• New media folder: Changed snapshot location from /www to /media/llmvision/snapshots (#457)

Improved Setup and Provider Settings

v1.5.0 Release Notes

This update improves the setup process, adds a fallback provider option, support for urls and other path types in video analyzer, introduces advanced settings for Ollama, and fixes an issue with Google Gemini.

Contributors

A huge thank you to our contributors @t3hk0d3, @TheRealFalseReality, @ond000, @natedev, @sb-develop—and of course everyone who helped test and provided feedback for the beta versions!

Integration

⚠️ Breaking Changes

• Timeline and Memory have been merged: The new Settings Config Entry unifies Timeline, Prompt and Memory (Beta) Settings as well as a new Fallback Provider option. If the Timeline provider still exists after this update, you may safely delete it. You may need to change the entity id of timeline entity (now associated with the Settings config entry).
• Temperature has moved: The temperature settings has moved to the provider settings. It can now longer be set individually for each call and is instead set for each provider. You can create multiple instances of the same provider if you need different temperature settings.

✨ Features

• New setup and reconfigure process: It is now easier to configure new providers as well as change settings for existing providers. The setup flow also allows for more provider specific settings. (@valentinfrlch)
• New Fallback provider option: You can now choose a fallback provider if the call fails with the selected provider. The request will be sent to the default model of the selected fallback provider. (@valentinfrlch)
• Support for additional paths: video_analyzer now supports URLs, media-source:// and relative /api/frigate paths. Thank you @t3hk0d3!
• New advanced settings for Ollama: You can now change the Context window length and “Keep alive” settings for Ollama. (#212)(@valentinfrlch)
• Added Top P: You can now control Top P (nucleus sampling) for all providers. This allows for more precise control over the randomness of generated responses. (@valentinfrlch)
• Support for Czech language: Added 🇨🇿 Czech translations. Thank you @ond000!

🔧 Improvements & Fixes

• Improved Image Pipeline efficiency: The image pipeline can now process multiple files or streams at once. Thank you @t3hk0d3!
• Google Gemini fix: Implemented exponential backoff retry logic for Google. Thank you @natedev! (#262)
• Update OpenAI Provider: Rename max_tokens to max_output_tokens for compatibility with newer o-series models. (#383)
• Custom OpenAI Provider: Use specified model for validation (#366)

Blueprint

• Fixed File Path default selector
• Fixed Custom Notification Channel default selector
• Added Time to notification title with the option of 12/24 hour format
• Fixed group/channel components with multiple camera entities

Thank you @TheRealFalseReality!

Improved Blueprint & Bug fixes

1.4.3 Release Notes

Warning

You may need to reconfigure certain fields of the blueprint after updating to v1.4.3.

Tip

The blueprint requires a separate update. You can update it by re-importing
(Settings → Automations & Scenes → Blueprints → three-dot menu → Re-import blueprint)

Blueprint

@TheRealFalseReality has completely rewritten the blueprint and enhanced it with lots of additional features:

• Added collapsible sections to the blueprint to make it easier to set up. Thank you @TheRealFalseReality!
• Added notification settings. Thank you @TheRealFalseReality!
  • notify_condition A list of conditions that need to be met to send the notification. The blueprint can still run and save events to the timeline but skips sending the notification. (Different from run_conditions which prevent the blueprint from running all-together).
  • delay_notification A separate cooldown for notifications (blueprint can still run, but doesn’t send a notification again. Separate from cooldown).
  • notification_sticky (Android only) make the notification stay active until dismissed.
  • notification_color (Android only) Customize the color of the notification
  • notification_icon (Android only) Customize the icon of the notification
  • notification_channel (Android only) Custom notification channel for custom notification sounds and vibration patterns. Allows overriding “Do Not Disturb”.
  • tts_notifications (Android only) Use text-to-speech to speak the notification.
• Added additional_actions (Experimental) Run custom actions after the blueprint.

Integration

• Added 🇸🇰 Slovak translation. Thank you @jose1711!
• Updated Groq default model (previous model is deprecated) (#311)
• Fixed a bug in the timeline that deleted events when retention_time was set to 0. (#309)

Performance Improvements

1.4.2 Release Notes

Tip

The blueprint requires a separate update. You can update it by reimporting.

Breaking Changes

Warning

The cleanup action is deprecated. The database now runs an automatic cleanup every time Home Assistant starts.

Fixes & Changes

• Significantly improved image pipeline performance. Thank you @sb-develop!
• Improved frame selection by @sb-develop and @lich2000117
  • The first captured frame is now always considered the key frame for stream_analyzer (blueprint)
• Images are now stored in the timeline in full resolution. (@sb-develop)
• Adds 🇨🇳 Chinese translations. Thank you @lich2000117!
• Adds 🇸🇪 Swedish translations. Thank you @adnansarajlic!
• Fixed a bug that prevented events from being properly deleted after retention_time. (@valentinfrlch)
• An automatic cleanup will run when Home Assistant starts to remove any unused images from /www/llmvision/. (@valentinfrlch)
• Fixed a bug that prevented images to be stored correctly on docker installations. (@valentinfrlch)

Bug fixes & Improvements

1.4.1 Release Notes

Tip

The blueprint requires a separate update. You can update it by reimporting.

Integration

• Memory now supports images with transparency (such as png). Thank you @lich2000117!
• Data Analyzer now supports the remember parameter and can contribute events to the timeline.
• Titles now support characters from the extended ASCII charset such as “Ø”, “Ӑ” and many others. Thank you, @ttopholm!
• Improved database performance and stability
• Fixed a bug where images would not appear in the timeline when using image_input or a frigate event_id. (#266, #255)

Blueprint

• Updated notification message. Thanks @lich2000117!

Memory & Timeline

1.4.0 Release Notes

This is a major update. Please read through the release notes and breaking changes especially. See the updated docs if you have any questions.

A huge thank you to everyone who helped test this version and shared their feedback!

Warning

The blueprint has been significantly restructured, which means you will have to set up some parts again. Due to many issues with the Frigate mode of this blueprint it has been removed. However, the blueprint is still compatible with Frigate: You will have to use the camera entities exposed by the Frigate integration as well as one of the binary_sensors for each camera.

Tip

The blueprint requires a separate update. You can update it by reimporting.

⚠️ Breaking Changes

• input_mode (Blueprint) has been deprecated. The blueprint now only supports what was previously the camera mode. See above for how you can still use Frigate with this blueprint.
• detail has been deprecated. To control the quality of images use the target_width parameter which downscales images that are larger than the set width.
• expose_images_persist has been deprecated. expose_images now saves the image to /www/llmvision/<uid>-<key_frame>.jpg. This path is returned with the response variable in the key_frame key. Use the new cleanup action to remove unneeded images from time to time.

Features

Integration

• Timeline replaces Event Calendar: LLM Vision now has a frontend card for your dashboard to display events along with an image and a summary of the event. Get the card here: https://github.com/valentinfrlch/llmvision-card
• Added Memory: You can now store reference images along with a description in memory. This provides additional context to the model and can be used e.g. to provide the model with pictures of members of your household to identify them. Content in memory syncs across all providers (except Groq as it can only process one image per call). With this there is now also a use_memory toggle in all actions to control whether content stored in memory should be used or not.
  Disclaimer: This is not a facial recognition model. AI can make mistakes. Do not use for unlocking doors or similarly critical actions!
• Added customizable system prompt: In the memory settings, a system prompt has been added which can be customized as well. The default system prompt greatly improves accuracy compared to before. The system prompt applies to all providers and is sent even when use_memory is off.
• Added customizable title prompt: The prompt used to generate titles for events can no be customized in the Memory settings.
• Added a new action remember: Use this action to store custom events to the timeline (and the LLM Vision Card).
• Added a new action cleanup: Remove images in /www/llmvision/ that are not associated with an event in the timeline.

Blueprint

Warning

See the breaking changes above before updating!

• Added a run_condition: You can now add custom conditions to the blueprint. All conditions have to be true in order for the blueprint to run. Use this to only run the automation when no one is home or when the alarm is enabled.
• Added notification_delivery: Choose between ‘Dynamic’ and ‘Consolidated’ delivery modes. Dynamic will immediately send a notification with a live camera preview and update the notification silently when the summary is available. Consolidated will only send the notification once the summary is available. Use this if you got two separate notifications before.
• Added use_memory: Enables access to information stored in memory. Memory needs to be set up for this.

Open WebUI

1.3.9 Release Notes

Features

Integration

• Added Open WebUI as a supported provider (#112), @valentinfrlch
  Open WebUI is a self-hosted AI platform with a built-in inference engine for databases so you can add knowledge to your LLM which is also available in LLM Vision.
  https://github.com/open-webui/open-webui

Bug fixes

1.3.8 Release Notes

⚠️ Breaking Changes

Warning

Blueprint: mode has been renamed to input_mode. Just select your input_mode again and save your automation.

Tip

The blueprint requires a separate update. You can update it by reimporting.

Bug Fixes

Integration

• Removed redundant imports

Blueprint

• Fixed a bug where the blueprint wouldn’t trigger for Frigate mode (#185) by @valentinfrlch
• Description improvements