Planning Flink Agents 0.3

[xintongsong]

Time Plan

• Target Feature Freeze: May 31
• Target Release: Jun 15

Feature Plan

Must

• Highlight Features
  • [Agent Skills] Agent Skills Investigation #536 [Feature] Support Agent Skills #589
  • [Memory] Mem0 Backend for Long-Term Memory #613 [Feature] Support Mem0 as Long-Term Memory backend. #617
  • [Event log] [Feature] Per-Event-Type Configurable Log Levels for Event Log #552 [runtime] Support per-event-type configurable log levels for event log #609
  • [Feature] Support python 3.12 #553
• Demands from Production
  • Support auto-arguments for tool using
  • Support cross-language actions & events
    • [Feature] Simplify the custom event definition. #424
  • It's our top priority to support people putting Flink Agents into product. So anyone who encounters unmet requirements while trying to do so, please feel free to raise it and we'll try our best to add it into the release plan.
• Important Experience Improvements
  • [Feature] Enhance the quickstart experience #555 [tools]Import Wizard for Installation Setup #599
  • [Event log] Enable by default and display in UI
  • [Tech Debt] Cross language resource can't support async execution. #448

If possible

• Important feature, but lack concrete plan or capacity
  • [Event log] Support querying and filtering
  • [Feature] Track durable execution reconciler work #612
  • Eval
  • Multimodal capabilities
• Nice to have features & improvements
  • Loading MCP Tools & Prompts in runtime
  • Support searching tools and prompts from the MCP server
  • [Memory] MemoryRef auto resolving
  • Structured information extraction
• User Unaware
  • [Tech Debt] Refactor ActionExecutionOperator into focused manager classes #545
  • [Tech Debt] Refactor AgentPlan & ResourceCache #547
  • Unify long-term memory management for Java & Python
  • CI improvements

Not likely (not sure about the necessity in event-driven agent scenarios)

• Support for multi-agent system & A2A
• Human-in-the-loop
• Serve as a service
• Sandbox
• Basic agent tools implementations like web search
• [Memory] Support for knowledge

Brainstorm (previous)

From @xintongsong

• Installation & Deployment
  • Enhance the quickstart experience. - Currently, there are too many manual steps, thus chances for users to make mistakes and run into problems.
• Agent Development
  • Support Python 3.12
  • Support cross-language actions & events
  • Memory
    • Support for knowledge. - Not entirely sure about this. What is the relationship between Knowledge and ContextRetrievalAction?
    • MemoryRef auto resolving - Using MemoryRef for Events and Resource arguments just like using the referenced value
    • Integrate with mature AI storage frameworks - E.g., mem0, reme, agents, etc. Why build from scratch?
  • Support auto-arguments for tool using - Not all arguments should be decided by LLM. Avoid unnecessary randomness.
  • Support searching tools and prompts from the MCP server
  • Support for agent skills
  • Support for multi-agent system
  • Multimodal capabilities
  • Human-in-the-loop
  • Structured information extraction
• Agent Execution and Operation
  • Durable execution enhancement - What if the job fails when the durable execution code block is executed halfway? 2PC? User defined handling?
  • Serve as a service - A service that accepts the request, put it into the source, collect the output from the sink, and send the response.
  • Event log
    • Enable by default and display in UI
    • Support per-event-type configurable log levels
    • Support querying and filtering
  • Eval
    • How to quantitatively measure agent performance?
    • How to compare performance of two different versions (prompt, model, parameter, ...)?
• Tech Debt
  • Refactor ActionExecutionOperator
  • Refactor AgentPlan & resource cache
  • Loading MCP Tools & Prompts in runtime - in case of updates on the MCP server side
  • Support async execution for cross-language resource access
  • Unify long-term memory management for Java & Python
  • CI
    • Run the examples
    • Submit jobs to a standalone flink cluster - in addition to the mini-cluster
    • Fault tolerant test - kill the TM and see if it recovers as expected

From @alnzng

• Skills
• Multimodal
• Sandbox
• Human-in-the-Loop

From @yanand0909

• A2A (Multi-Agent System)
• Basic agent tools implementations like web search

From @JinkunLiu

• Improving the quick-start experience

[alnzng]

I like the ideas in this proposal. I actually have four things on top of my mind that I hope we can do it for Flink Agents: 1) Skills 2) Multimodal 3) Sandbox 4) Human-in-the-Loop. It seems 3 of them are coverred in above proposal already! We can discuss if "Sandbox" is a valid requirement for Flink Agent.

[yanand0909]

Should we consider adding protocol support like A2A for multi-agent system? And should we also consider adding basic agent tools implementations like web search? I think we should try to develop easy to get started ecosystem by providing these common inbuilt tools for agents.

[xintongsong]

Should we consider adding protocol support like A2A for multi-agent system?

I don't have the answer yet. It's still a rough idea that we might want to support multi-agent system. But there are some open questions. How important is it to support multi-agent system for event-driven agents? What are the concrete use cases? And how do we build an multi-agent system on Flink's streaming engine? How does the technical design look like?

And should we also consider adding basic agent tools implementations like web search?

If there are some commonly needed tools, I think it makes sense to provide builtin implementations. However, I'm not sure whether a web search tool makes sense in the event-driven agent scenarios. I think we need to first identify what are the commonly needed tools in event-driven agent use cases.

[yanand0909]

I think we need to first identify what are the commonly needed tools in event-driven agent use cases.

Yes I completely agree web search tool was just an example for reference, as we see more adoption we can try to figure out what tools users are using for event driven agents and decide built-in tools on basis of that.

[JinkunLiu]

Should we add a walkthrough-common package, like flink-walkthrough-common, to offer users an easy-to-follow guide for installation and deployment?

[JinkunLiu]

Aha, I found that package called flink-agents-examples, but it wasn’t mentioned anywhere in the documentation.

[wenjin272]

Aha, I found that package called flink-agents-examples, but it wasn’t mentioned anywhere in the documentation.

Hi, @JinkunLiu, you can find the documents forflink-agnets-examples in the Quickstart section: https://nightlies.apache.org/flink/flink-agents-docs-release-0.2/docs/get-started/quickstart/workflow_agent/

[JinkunLiu]

Aha, I found that package called flink-agents-examples, but it wasn’t mentioned anywhere in the documentation.

Hi, @JinkunLiu, you can find the documents forflink-agnets-examples in the Quickstart section: https://nightlies.apache.org/flink/flink-agents-docs-release-0.2/docs/get-started/quickstart/workflow_agent/

Yeah,Thanaks for you reminder @wenjin272 I noticed that the relevant code is provided at that location. But when I tried to follow the code and run it in a new project on my local(not in flink-agents project), I encountered missing dependencies until I added the following code to my pom.xml.

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-agents-examples</artifactId>
    <version>${flink-agents.version}</version>
    <scope>provided</scope>
</dependency>

What I mean is, explicitly tell users that this dependency needs to be added is better?

Furthermore, should we provide a Maven archetype for example project generation, like try-flink-docs does?

[wenjin272]

Furthermore, should we provide a Maven archetype for example project generation, like try-flink-docs does?

We have a document for how to add dependencies for maven project based on flink-agnets in maven-dependencies-for-java. I think providing a maven archetype is indeed can improve usability.

Yeah,Thanaks for you reminder @wenjin272 I noticed that the relevant code is provided at that location. But when I tried to follow the code and run it in a new project on my local(not in flink-agents project), I encountered missing dependencies until I added the following code to my pom.xml.

Let me understand, you created a new project locally, copied the code from the quick start documents into it, and encountered a missing dependency error when running it? What exactly was the missing dependency? I think this dependency issue could also be resolved according the documentation mentioned above.

[JinkunLiu]

Let me understand, you created a new project locally, copied the code from the quick start documents into it, and encountered a missing dependency error when running it?

Yes — that's exactly right. I created a new project locally:

• First I followed the Installation section and copied the pom.xml
• Second I copied the Quickstart example code.
• Then when I ran prepare-agents-execution-environment I encountered a missing dependency: org.apache.flink.agents.examples.agents.CustomTypesAndResources#OLLAMA_SERVER_DESCRIPTOR, which is defined only in the flink-agents-examples module.

I think providing a maven archetype is indeed can improve usability.

If the community agrees after discussion, I'd be happy to help implement it.