Foreman Requirements

[destogl]

In general Foreman has layered architecture providing possibility to connect to different communication/IPC systems and not only ROS 2.
The main idea is to have a higher-level interface to control the state of a complex robot system running ros2_control. On this level states of the system are defined that are specified by the HW and Controllers being in specific lifecycle state. Foreman takes care of the transitions.

Implemented requirements

1. Possibility to add custom adapters to non-ROS 2 systems, e.g., connecting Foreman with a web UI.
2. Having a YAML configuration to define application states, names, and defining goal states for the hardware components and controllers.
3. Possibility to also control the state of lifecycle nodes, e.g., robot_manager for KUKA robots that set the states of the controllers and need custom activation and deactivation strategies — so we don't need to re-implement manufacturer specific processes.
4. Having an option to set autostart_state as the goal of the state of the system after Foreman is started.
5. Foreman has to robustly observe and wait for components to be available (especially on autostart).
6. Foreman observes the state of the components continuously and reports its changes of the overall system state if it matches a defined one.
7. Foreman takes care of all transitions and retrials for a specific state.

Missing

1. Per default if there is no configuration file we should have unconfigured, inactive and active states setting all HW and Controllers into those.
2. Using actions instead of services for the setting of transitions.
3. Consider using standard names for the states that are automatically interpretable by external systems.
4. RQT GUI for ROS 2 that is showing and can control Foreman state (we can provide a sketch for it).
5. How to handle error and warning states, e.g., Foreman observes a state that is not defined in configuration — what does this mean? Is this a hard error? In which case? (Can this be somehow defined in a simple way — as number of states is exponential with the number of hardware components and controllers).
6. Do we need a set of standard states? To handle errors and such things?
7. Foreman should handle controller chains, this means it should automatically calculate their starting and stopping if on activation the most “left” controller is activated and on deactivation the most “right” controller is deactivated. Foreman takes care that all depending on controller in the chain are also activated/deactivated.
8. Auto-generated YAML file for the Foreman from the available controllers and hardware components.
9. How do we react if we can not reach a certain state, but we have started transition already? Do we return to the previous state, or we require some “safe” state?

Use Cases

1. Cell with robot arm and the gripper that requires lifecycle node for arm activation. Gripper is controlled separately. When using MockHW then we have more controller as Gripper sensors are also simulated through a forwarding controller and simulation script.
2. Having a system with an undercarriage, robot arm, gripper and 2 more actuators that are controllable all separately, there are at least 8 controllers and depending on the state of the activation of controllers and hardware the overall mode of the system is determined, e.g., manual, maintanance, automatic, ...
3. Out-of-the box running of simple setups like ros2_control_demos without any configuration.

[saikishor]

Implemented requirements

2. Having a YAML configuration to define application states, names, and defining goal states for the hardware components and controllers.

I'm not sure about the goal states part, if we really need it for the mission control or not

4. Having an option to set autostart_state as the goal of the state of the system after Foreman is started.

Can't this be achieved with the spawners itself?. If not, we need to be able to define a specific profile.

5. Foreman has to robustly observe and wait for components to be available (especially on autostart).

We can do that with the ~/status topic

6. Foreman observes the state of the components continuously and reports its changes of the overall system state if it matches a defined one.

Same here, the ~/status topic already does the same work, so nice

7. Foreman takes care of all transitions and retrials for a specific state.

Where do we need this with ros2_control, if a transistion doesn't happen, it won't happen on retry right?. For other usecases, it might be.

Missing

1. Per default if there is no configuration file we should have unconfigured, inactive and active states setting all HW and Controllers into those.

Sure, fair enough, but I guess this cannot be cached, it need to be done with the momentary information. For instance, we have ctrl_A and ctrl_B at startup, and later we enable ctrl_C and then when inactive is set, it needs to set ctrl_A, ctrl_B and ctrl_C and the HW to the inactive mode.

2. Using actions instead of services for the setting of transitions.

Why actions here? We cannot cancel it anyway right?, I think it is better to stick to services and let the extended class of ros2_control handle it with Services and then whoever implements their part can choose the actions.

3. Consider using standard names for the states that are automatically interpretable by external systems.

We already have the lifecycle naming can we reuse them.

5. How to handle error and warning states, e.g., Foreman observes a state that is not defined in configuration — what does this mean? Is this a hard error? In which case? (Can this be somehow defined in a simple way — as number of states is exponential with the number of hardware components and controllers).

We need to discuss about this. I'm lacking some context here.

6. Do we need a set of standard states? To handle errors and such things?

IMO, not needed.

7. Foreman should handle controller chains, this means it should automatically calculate their starting and stopping if on activation the most “left” controller is activated and on deactivation the most “right” controller is deactivated. Foreman takes care that all depending on controller in the chain are also activated/deactivated.

Totally! 👍

8. Auto-generated YAML file for the Foreman from the available controllers and hardware components.

Not needed, the system should be able to cache the information, everytime there is an update on the ~/status topic.

9. How do we react if we can not reach a certain state, but we have started transition already? Do we return to the previous state, or we require some “safe” state?

We simply show an error on the user side.

Use Cases

3. Out-of-the box running of simple setups like ros2_control_demos without any configuration.

Yes, Just launching the future foreman node should be enough.

[destogl]

I'm not sure about the goal states part, if we really need it for the mission control or not

The “goal state” is mean the target state of each controller and hardware to achieve certain state (application mode) of Foreman. IMHO, this is the core functionality.

Can't this be achieved with the spawners itself?. If not, we need to be able to define a specific profile.

No this can not be done with spawners in general, as often times you need more complex transitions of individual components to active other components. For example when you are starting application you might want to connect to the safety system immediately, but activate the rest of the system first if this is working, i.e., if you have specific output from the safety then you want to try to start the logical part of the system.

5. Foreman has to robustly observe and wait for components to be available (especially on autostart).

We can do that with the ~/status topic

Yes, (you probably mean ~/activity topic, but how you do it in the automated way. For example, you have your robot arm in the "Free Space" mode with JTC, and now you want to start "Contact" mode with “Admittance” controller, you have to have feedback to reflect this state in Forman to track them. So even if you change the controller with Foreman or directly using CLI I expect that the application mode is clearly defined in Foreman if one mode is available - as there I can be more application specific in the naming. The goal here is to avoid reimplementing “foreman” component for every single robot, as mostly you require the same functionalities.

Same here, the ~/status topic already does the same work, so nice
OK, do you mean something else with the ~/status topic - I am not sure what you mean?

7. Foreman takes care of all transitions and retrials for a specific state.

Where do we need this with ros2_control, if a transistion doesn't happen, it won't happen on retry right?. For other usecases, it might be.

This is not true in all cases. I just had this example yesterday, which I am not sure why it happened, but JTC didn't want to start (no clear output), then the Foreman automatically re-activated the HW and the JTC started successfully. In that particular scenario is this OK, as the connection to the robot sometimes has to be restarted to all be functioning properly — this is of course HW interface issue, but we don't have much better solution there. On the other case when starting the whole system you want to make sure that all components are available before controllers start to be started, so Formen does this “retries”/check for you and initialized in the proper order.

Also, robot manufactueres are starting adding their own "robot_managers" for particular HW that is just a part of the solution and than you can retry on their lifecycle node. Concrete example here is "robot_manager" for KUKA. Maybe we will have possibility to do such complex starting of ros2_control components and controllers at some point, but I find it too complicated and manufacturer specific for now.

Missing

Sure, fair enough, but I guess this cannot be cached, it need to be done with the momentary information. For instance, we have ctrl_A and ctrl_B at startup, and later we enable ctrl_C and then when inactive is set, it needs to set ctrl_A, ctrl_B and ctrl_C and the HW to the inactive mode.

Yes, exact solution needs to be figured out. It will not be perfect, but if people want to have perfect should write their config 😄

2. Using actions instead of services for the setting of transitions.

Why actions here? We cannot cancel it anyway right?, I think it is better to stick to services and let the extended class of ros2_control handle it with Services and then whoever implements their part can choose the actions.

Using ros2 actions. I think you can also cancel it which would basically return to the previous state. But canceling is less important here it is more about feedback that we don't have with services. And block for 30 seconds when switching many components is also not great.

We already have the lifecycle naming can we reuse them.

Yes, agree! This is the idea to follow!

5. How to handle error and warning states, e.g., Foreman observes a state that is not defined in configuration — what does this mean? Is this a hard error? In which case? (Can this be somehow defined in a simple way — as number of states is exponential with the number of hardware components and controllers).

We need to discuss about this. I'm lacking some context here.

For example one component can be suddenly deactivated, from what ever reason and this state is not particularly defined. This is a potential error in the system, what to do than? The most significant states should be defined, but the number of combination raises exponentially with number of hardware components and controllers. so we should have sensible manage mat of this and let user handle it simply.

6. Do we need a set of standard states? To handle errors and such things?

IMO, not needed.

OK 😄

7. Foreman should handle controller chains, this means it should automatically calculate their starting and stopping if on activation the most “left” controller is activated and on deactivation the most “right” controller is deactivated. Foreman takes care that all depending on controller in the chain are also activated/deactivated.

Totally! 👍

👍 - thanks for the idea @saikishor

Not needed, the system should be able to cache the information, everytime there is an update on the ~/status topic.

Yes, but you don't know which state means what. It is not always so simple to have all active or all inactive. Just think about “Free space” and “Contact” modes mentioned above.

9. How do we react if we can not reach a certain state, but we have started transition already? Do we return to the previous state, or we require some “safe” state?

We simply show an error on the user side.

Is it so simple 😉 Potentially just stop with half-activated robot?

[lakhmanisahil]

Hello @saikishor and @destogl, as far as we have discussed, we came to the following conclusions:

I'm not sure about the goal states part, if we really need it for the mission control or not

The “goal state” is mean the target state of each controller and hardware to achieve certain state (application mode) of Foreman. IMHO, this is the core functionality.

The YAML configuration will only define preset profiles (goal states / application modes). It will not define controller dependencies, hardware states, or lifecycle states.
These will be discovered at runtime using list_controllers, list_hardware_components, and list_hardware_interfaces. The controller_manager should be provided through launch files

Having an option to set autostart_state as the goal of the state of the system after Foreman is started.

Can't this be achieved with the spawners itself?. If not, we need to be able to define a specific profile.

No this can not be done with spawners in general, as often times you need more complex transitions of individual components to active other components. For example when you are starting application you might want to connect to the safety system immediately, but activate the rest of the system first if this is working, i.e., if you have specific output from the safety then you want to try to start the logical part of the system.

At startup, users may optionally define the desired initial profiles through a CLI command, e.g. :

ros2 run foreman set_state --profile profile_1 active --profile profile_2 inactive

5. Foreman has to robustly observe and wait for components to be available (especially on autostart).

We can do that with the ~/status topic

Yes, (you probably mean ~/activity topic), but how you do it in the automated way. For example, you have your robot arm in the "Free Space" mode with JTC, and now you want to start "Contact" mode with “Admittance” controller, you have to have feedback to reflect this state in Forman to track them. So even if you change the controller with Foreman or directly using CLI I expect that the application mode is clearly defined in Foreman if one mode is available - as there I can be more application specific in the naming. The goal here is to avoid reimplementing “foreman” component for every single robot, as mostly you require the same functionalities.

Foreman will validate all required components from YAML file before starting. If any required component is unavailable, Foreman will report the issue and refuse to start. The ~/activity topic can be used to monitor runtime state changes.

7. Foreman takes care of all transitions and retrials for a specific state.

Where do we need this with ros2_control, if a transistion doesn't happen, it won't happen on retry right?. For other usecases, it might be.

This is not true in all cases. I just had this example yesterday, which I am not sure why it happened, but JTC didn't want to start (no clear output), then the Foreman automatically re-activated the HW and the JTC started successfully. In that particular scenario is this OK, as the connection to the robot sometimes has to be restarted to all be functioning properly — this is of course HW interface issue, but we don't have much better solution there. On the other case when starting the whole system you want to make sure that all components are available before controllers start to be started, so Formen does this “retries”/check for you and initialized in the proper order.

Also, robot manufactueres are starting adding their own "robot_managers" for particular HW that is just a part of the solution and than you can retry on their lifecycle node. Concrete example here is "robot_manager" for KUKA. Maybe we will have possibility to do such complex starting of ros2_control components and controllers at some point, but I find it too complicated and manufacturer specific for now.

Here again, Foreman will check all components required by the selected preset profile and will not start if any required component is unavailable.
Lifecycle nodes for manufacturer-specific managers (e.g., KUKA Robot Manager) will need to be handled separately.

2. Using actions instead of services for the setting of transitions.

Why actions here? We cannot cancel it anyway right?, I think it is better to stick to services and let the extended class of ros2_control handle it with Services and then whoever implements their part can choose the actions.

Using ros2 actions. I think you can also cancel it which would basically return to the previous state. But canceling is less important here it is more about feedback that we don't have with services. And block for 30 seconds when switching many components is also not great.

Services will be used initially for goal requests and transitions, instead of actions

We already have the lifecycle naming can we reuse them.

Yes, agree! This is the idea to follow!

Done 👍

5. How to handle error and warning states, e.g., Foreman observes a state that is not defined in configuration — what does this mean? Is this a hard error? In which case? (Can this be somehow defined in a simple way — as number of states is exponential with the number of hardware components and controllers).

We need to discuss about this. I'm lacking some context here.

For example one component can be suddenly deactivated, from what ever reason and this state is not particularly defined. This is a potential error in the system, what to do than? The most significant states should be defined, but the number of combination raises exponentially with number of hardware components and controllers. so we should have sensible manage mat of this and let user handle it simply.

Foreman will report the error and stop the transition process. Error and warning conditions should be explicitly defined.

7. Foreman should handle controller chains, this means it should automatically calculate their starting and stopping if on activation the most “left” controller is activated and on deactivation the most “right” controller is deactivated. Foreman takes care that all depending on controller in the chain are also activated/deactivated.

Controller chains and dependencies will be inferred automatically fromy claimed_interfaces and reference_interfaces, instead of being manually defined in the YAML config by the user.

Other points we are going to implement:

1. We should not lock into one fixed YAML format; we may support different formats as the user prefers, and we may also extract information from existing config files like those in the examples.
2. For simple setups like ros2_control_demos, a YAML file may not be required at all.
3. A component may sometimes be unavailable, so we may need to combine list_HW_interfaces and list_controllers with the activity topic to build a complete picture.

Doubts and Further Discussion

Missing

Per default if there is no configuration file we should have unconfigured, inactive and active states setting all HW and Controllers into those.

Sure, fair enough, but I guess this cannot be cached, it need to be done with the momentary information. For instance, we have ctrl_A and ctrl_B at startup, and later we enable ctrl_C and then when inactive is set, it needs to set ctrl_A, ctrl_B and ctrl_C and the HW to the inactive mode.

Yes, exact solution needs to be figured out. It will not be perfect, but if people want to have perfect should write their config 😄

&

This is not true in all cases. I just had this example yesterday, which I am not sure why it happened, but JTC didn't want to start (no clear output), then the Foreman automatically re-activated the HW and the JTC started successfully. In that particular scenario is this OK, as the connection to the robot sometimes has to be restarted to all be functioning properly — this is of course HW interface issue, but we don't have much better solution there. On the other case when starting the whole system you want to make sure that all components are available before controllers start to be started, so Formen does this “retries”/check for you and initialized in the proper order.

1. What are we going to do in these scenarios then? If we are not starting Foreman until all required components are available?

2. We still need to discuss the benchmark format, including the number of hardware components and controllers (including chained controllers) that should be part of the benchmark scenario.

For example when you are starting application you might want to connect to the safety system immediately, but activate the rest of the system first if this is working, i.e., if you have specific output from the safety then you want to try to start the logical part of the system.

3. I was also thinking about chaining profiles. Let's say we first switch to a Safety profile, and it reports that all required components are available. In that case, we should not need to manually switch to another profile, such as Inactive.
   Instead, we could automatically transition to the Inactive state once the Safety profile succeeds. Perhaps the Safety profile could trigger a service call, or we could define a similar mechanism for profile chaining?

[destogl]

The YAML configuration will only define preset profiles (goal states / application modes). It will not define controller dependencies, hardware states, or lifecycle states. These will be discovered at runtime using list_controllers, list_hardware_components, and list_hardware_interfaces. The controller_manager should be provided through launch files

I am not sure here what do you mean with lifecycle state, but we have to define as profile the lifecycle state of the components.

At startup, users may optionally define the desired initial profiles through a CLI command, e.g. :

ros2 run foreman set_state --profile profile_1 active --profile profile_2 inactive

This is OK, but I not sure how this should work? What is there are conflicting states defined in the profiles?

Foreman will validate all required components from YAML file before starting. If any required component is unavailable, Foreman will report the issue and refuse to start. The ~/activity topic can be used to monitor runtime state changes.

Generally yes, practically we have to add some timeout, as the components take time to load and foreman might be started earlier. So we have to make sure that ros2_control if fully initialized, and then this works for the HW, but for the controller, those can be dynamically loaded, so we have to not report issue and exit in that case.

Here again, Foreman will check all components required by the selected preset profile and will not start if any required component is unavailable. Lifecycle nodes for manufacturer-specific managers (e.g., KUKA Robot Manager) will need to be handled separately.

If I get this right, it is not correct. We want to handle lifecycle nodes too. So Foreman has to be able to control their state too. Only question is here if then we need to define which controllers are handled by those components? Or what to do in the conflict case? Is this actually a problem?

Services will be used initially for goal requests and transitions, instead of actions

Yes, but they have to be fixed to block until everything is done.

Other points we are going to implement:

1. We should not lock into one fixed YAML format; we may support different formats as the user prefers, and we may also extract information from existing config files like those in the examples.

I think you understood this differently. We should have clear structure, but we should be able to load profiles from multiple file, like we can do with controller in ros2_control. We don't want to have different formats.

2. For simple setups like ros2_control_demos, a YAML file may not be required at all.

✔️

4. A component may sometimes be unavailable, so we may need to combine list_HW_interfaces and list_controllers with the activity topic to build a complete picture.> 

✔️

Doubts and Further Discussion

1. What are we going to do in these scenarios then? If we are not starting Foreman until all required components are available?

No we are starting Forman often immediately in the launch file, and we should wait for the required components. I would even propose to use the lifecycle node to make it configured until all required components are not available. In this cases we observe the activity topic to get the list of all components.

2. We still need to discuss the benchmark format, including the number of hardware components and controllers (including chained controllers) that should be part of the benchmark scenario.

As benchmark sue first all the scenarios in the ros2_control_demos repo. There are all architectures available in simple form that makes it good for development. We will provide you regular feedback if something more complex doesn't work.

3. I was also thinking about chaining profiles. Let's say we first switch to a **Safety** profile, and it reports that all required components are available. In that case, we should not need to manually switch to another profile, such as **Inactive**.
   Instead, we could automatically transition to the **Inactive** state once the **Safety** profile succeeds. Perhaps the Safety profile could trigger a service call, or we could define a similar mechanism for profile chaining?

OK, generally agree, but not sure how to implement this actually... As it will end up complex. So I would focus on the fixed profiles for now. Later we can play with the "lifecycle" of profiles.

[lakhmanisahil]

Thank you @destogl for the reply, it clears up several things.

I am not sure here what do you mean with lifecycle state; but we have to define as profile the lifecycle state of the components.

Sorry, here to clarify my earlier wording: the YAML does not define the lifecycle nodes present in the system, but it does define is the desired target lifecycle state of each component inside a profile.

This is OK, but I not sure how this should work? What is there are conflicting states defined in the profiles?

Got it, right, the target states can conflict across profiles requested together.
We could resolve it with a priority ordering between profiles, but that adds more complexity, so I would leave it out for now.

Generally yes, practically we have to add some timeout, as the components take time to load and foreman might be started earlier. So we have to make sure that ros2_control if fully initialized, and then this works for the HW, but for the controller, those can be dynamically loaded, so we have to not report issue and exit in that case.

Understood, so we will wait, but not necessarily for controllers, since they can be loaded dynamically.

If I get this right, it is not correct. We want to handle lifecycle nodes too. So Foreman has to be able to control their state too. Only question is here if then we need to define which controllers are handled by those components? Or what to do in the conflict case? Is this actually a problem?

Sorry for the confusing phrasing earlier. You are right; this is not a separate problem.
What I meant is that lifecycle nodes should also be managed by Foreman as components, exactly as the current implementation already does through the lifecycle node adapter.
They should not be handled separately or considered out of scope.

Doubts:

1. What are we going to do in these scenarios then? If we are not starting Foreman until all required components are available?

No we are starting Forman often immediately in the launch file, and we should wait for the required components. I would even propose to use the lifecycle node to make it configured until all required components are not available. In this cases we observe the activity topic to get the list of all components.

Here, if I understood correctly, are you proposing that Foreman itself should be implemented as a Lifecycle Node, remaining in the configured state until all required components become available, and only then transitioning to active?

[destogl]

Here, if I understood correctly, are you proposing that Foreman itself should be implemented as a Lifecycle Node, remaining in the configured state until all required components become available, and only then transitioning to active?

Yes, I think this would make sense a lot. And might solve some problems we might have with the initialisation.