Make Blueprint call return structured response

[michaljonik]

I'm just wondering why the blueprint performs the analysis in two phases:

1. classification of the importance
2. then the actual analysis

That way the second phase is obviously delayed. And also the cloud providers charge twice.
Wouldn't it be possible for all the analyzer services to make just one LLM call which would return all the data in a structured way: a title, a description, an importance, etc?

[valentinfrlch]

The importance level is used to decide wether an event is important enough to send a notification. This means the importance analysis has to run before a notification can be sent. This will result in a delayed notifications.

I think most users prefer the instant notification (for when someone is at the front door) and don't use important. Also, the importance classification isn't good enough (yet), which is also why it's marked as 'Experimental'.

[michaljonik]

Actually what I had in mind was a general idea of sending just one query to LLM asking for a structured response.
I made a test of LLM Vision Image Analyzer action in HA Developer Tools and it works reasonably well.
My prompt was something like this:

message: >-
  You are an AI agent preparing a short log entry for events recorded by a security camera in front of my house based on the attached sequence of frames.
  Examples of valid descriptions:
  "A truck is parked by the gate", "A DHL courier is standing at the gate".
  If no objects match the specified criteria, respond with: "Nothing detected"
  Also generate a title summarizing the event, such as "Dog in the yard" or "Courier at the gate".
  Evaluate the importance of the event based on these rules:
  - "low" – no people detected on the property (even if there are some people behind the fense)
  - "high" – one or more people detected on the property
  
  Return the result as a JSON object in the following format:
  {
    "title": "Courier at the gate",
    "description": "A DHL courier is standing at the gate with a package",
    "importance": "low"
  }

The responses are quite reasonable (of course currently the services expect a text only so I had to extract the result), eg:

 {
   "title": "Courier at the gate",
   "description": "A DHL courier is standing at the gate with a package",
   "importance": "low"
  }

That way in the first step we have already all the information needed to decide about notifications, storing in a timeline, etc.

I would like someone more experienced in LLM's to shed some light on this approach if it makes sense or if it may decrease the quality of responses.

[valentinfrlch]

I don't think it impacts the quality of the response. In the timeline especially (for the dashboard card) this could be useful, as it would allow for better filtering.

[andrewfry]

@michaljonik Would you mind sharing how you are using the response in subsequent actions?