Redesign 0.6

[klemens-morgenstern]

I'd really like to have the boost.process library, but since development was on hiatus for two years, I think a few small design changes would be nice - or rather big ones, as described below.
But, not everything I write here is completely tested. I'd really need some discussion about those and other ideas how such a library could look like, because I really would change the frontend.

Here's the reference of the current version.

@dlaugt @BorisSchaeling @whizmo @hotgloupi @nat-goodspeed @havoc-io @rconde01 @JeffGarland

Since we now have a variadic templates, I think the execute function can be extended and much more flexible. Please notice, that execute shall return an undefined type, which depends on the parameters. It might be converted to child, as discussed below.
#1 Defaults

1.1 Unification

First of all, I like to unify the signal handling, to provide the following:

    void on_setup(Executor&) const
    void on_error(Executor&, const std::error_code &ec) const
    void on_success(Executor&) const

This would require the posix-executor to always use the self-pipe in order to provide the on_success signal. Therefore I think it would be also possible to add a initializer which removes this, in the following way:

   using namespace boost::process::initializers;
   auto p = execute("some_prog", posix::no_success_signal);

If then a success handler is provided, it would provide an error, i.e.:

   using namespace boost::process::initializers;
   auto p = execute("some_prog", posix::no_success_signal, on_success([](auto){}));
   //yields an compiler error

Posix Structure

The way it should work on posix is this:

A possible code in posix would look like this:

executor::operator()(...)
{
   pipe error_pipe = create_pipe(FD_CLOEXEC); //closes the pipe on execve call
   on_setup();
   auto pid = fork();
   if (pid == -1)
      on_fork_error();
   else if (pid == 0)
   {
      ::close(selfe_pipe.source);
      on_exec_setup();
      ::execve(exe, cmd_line, env);
      on_exec_error();
      ::write(error_pipe.sink, &errno, sizeof(int));
      ::_exit(EXIT_FAILURE);
   }
   //ok, here we are back in the father-process 
   int code, cnt;
   do 
   {
     cnt = ::read(error_pipe.source, &code, sizeof(code));
      int err = errno;
      if (res == EDABF)  //pipe is closed.
          break;
       //EAGAIN not yet forked, EINTR interrupted, i.e. try again
      else if ((err != EAGAIN) && (err != EINTR)) 
      else
          throw error();
   }
   while(cnt == -1);

   if (cnt == 4) //error.
       on_error(code);
   else if (cnt > 0) //some other strang error
       on_error(...); //don't know yet
   else
       on_success();
}

Now, on default the father process is waiting. I think to provide a ignore_error would be very usefule, since this would then ignore this.

The additional signal-handlers will be availble as part of the boost::process::posix namespace.

1.2 Throw on error

I also would make the throw_on_error the default, because it just makes the most sense, since execute constructs a process-object. I.e. not being able to construct it should result in an error.

auto p1 = execute("not_runnable"); //throws
auto p2 = execute("not_runnable", ignore_error); //no throw, no handling;
std::error_code ec;
auto p3 = execute("not_runnable", set_on_error(ec)); //no throw, takes the error
auto p3 = execute("not_runnable", ec); //no throw, takes the error, shortcut.

I would introduce posix::no_success_signal and ignore_error because the posix implementation now waits for the child-process to start. A user might consider that unneeded and could hence wish to skip that.
#2. Simplified syntax

2.1. Idea

I think it would be possible to make the syntax much simpler, as in the following examples (corresponding to the current). All are equal.

auto p1 = execute(exe("cmd"), args("x", "y")); //shortened version of the current syntax
auto p2 = execute(exe="cmd", args={"x", "y"}); //alternate syntax
auto p3 = execute(exe="cmd", args="x", args+="y"); //other syntax
auto p4 = execute("cmd", "x", "y"); //lazy syntax
auto p5 = execute(cmd("cmd x y"));
auto p6 = execute(cmd="cmd x y");
auto p7 = execute("cmd x y");

The cmd and args get parsed differently, that is:

execute(cmd="cmd x y z");//yields "cmd x y z"
execute(cmd="cmd \"x y\" z");//  yields "cmd \"x y\" z"
execute(exe="cmd", args={"x y", "z"}); //yields "cmd \"x y\" z"

2.2. Other initializers

The following signal handlers shall have syntax similar to the above.

• inherit_env
• env
• start_dir

And the platform specifics.

2.3 I/O

This is the part where I am the least confident, but those are all things I consider possible.

2.3.1 Close Pipe / Redirect to null

If a pipe shall be closed, it should be possible in the following way

execute("ls", std_out=null);  //redirect to /dev/null
execute("ls", std_out=close); //close the pipe
//alternate syntax
execute("ls", std_out>null);  //redirect to /dev/null
execute("ls", close(std_out)); //close the pipe

2.3.2 Sync I/O

The sync I/O can be simplified in the following way (current example). I.e. the pipe can be constructed automatically.

file_descriptor_sink sink;
execute("my_prog", std_in=sink);//alternatively '|', because pipe.
stream<file_descriptor_sink> str;

stream<file_descriptor_source> str;
execute("my_prog", std_out=str);

If the output shall be redirectet into a file, this can be done via a boost::filesystem::path:

execute("my_prog", std_out=path("./log.txt"));

It may also be following to allow a lazy redirection to files:

execute("my_prog", std_out>"./log.txt");

With this syntax it is of course also easy to pipe from one process to another (though the implementation is not yet clear, i might need to use a named pipe on windows)

process::pipe = create_pipe(); 
execute("prog1", std_out>pipe);
execute("prog2", std_in<pipe);

2.3.3 Self-Sync I/O

Removed, because that makes child struct unecessary complicated

Another possibility, that would definitly be possible would be to allow the process to adapt to a string. I don't know it this would actually be worth the effort.

auto p = execute("c++filt", std_in=self, std_out=self);
p << "_ZN9wikipedia7article8wikilinkC1ERKSs";
std::string dem;
p >> dem;
//dem  == "wikipedia::article::wikilink::wikilink(std::string const&)"

This is only convenience, it allows short-handed I/O. One could also use the stderr instead of stdout, while declaring both as self will yield an error.

2.3.4 Async I/O

This might be the most difficult part. I imagine that we can do it in someway like this:

boost::asio::io_service io_service;
stringstream ss;//threadunsafe, but get's the point.
string buf; 

execute("some_prog", std_in<ss, std_out>buffer(buf), std_err>cerr, io_service);
io_service.run();

Thereby the whole mitigate thing can be hidden and managed automatically. Addtionally, if no io_service is passed, it could be created by the execute command and run it in another thread.

The example from the current tutorial would look like this:

boost::asio::io_service io_service;
std::array<char, 4096> buffer;

auto p = execute("test.exe", std_out>process::buffer(buffer), io_service);

io_service.run();

I can determine that buffer must be read async,because it's no stream and can construct the async_pipe in place. Also I can call async_read from the executor, because that does only start on io_service.run().
Additionally, the io_service could be skipped and constructed in the executor and launched in a new thread. Don't know if that'd be smart though.

But to be honest: I am currently not nearly familiar enough with the way io_service works.

The same would apply to any std::ostream or std::istream, which is not stream<file_descriptor_sink>, so the cerr or ss in my example should work this way. Now of course, binding std_err>cerr is by far the stupidest way, because I could pipe that directly (std_err>stderr) - at least if we provide a syntax for that (i.e. also redirecting std_err>stdout).

This whole concept needs more experimentation!

2.4 Environment

In 0.5 the inherit_env was empty for windows, I don't know if that actually works. Now I would use the following syntax:

execute("thingy"); //inherit env
execute("thingy", env+={"MY_VAR=Thingy"}); //inhertit and add
execute("things", env=  {"MY_VAR=Thingy"}); //do not inherit

So the following would be awesome, though I don't know if necessary:

//inherit the environment and append "MyThing" to "MY_VAR"
execute("thingy", env["MY_VAR"]+="MyThingy"};

#3. Classes

3.1 Process

When calling execute a process-object is returned, whichs type is undefined. It holds any data needed by the initialization. It has a set of functions, which are the following

• is_running
• terminate
• wait //formerly wait_for_exit
• wait_for //wait with a timespan to wait
• wait_until //wait until a time timepoint
• exit_code

3.2. Child

If the process shall be stored in an undefined class, it can be converted to child. The data is then moved onto the stack and stored in a pointer inside the child. The child class will have member-functions similar to the functions for process.
A child is movable, but not copyable.
#4. Launch Mode

Removed, is done via process groups #8

In order to provide a process in an RAII way I do think that different launch modes should be available. I do not know as for now if this can be switched at runtime or has to be defined at launch time.

Attached

If a childprocess runs attached it will be terminated on destructor-call.

Detached

If a process is detached nothing will happend on destruction

Default

By default a process is launched in an attached way. This would cause this syntax to immediately terminate the application:

execute("prog");

But this behaviour could of course be specified by the following way:

execute("prog", wait); //blocking
execute("prog", detach); //not blocking.

#5 this_process

Similar to std::this_thread I would add a this_process namespace with the following synopsis.

namespace this_process 
{
    int get_id();
    /*undefined*/ native_handle();
    environment get_environment();
}

[klemens-morgenstern]

Status update

Reworking the whole MPL stuff with Hana took a bit more time than expected, though I have to say: hana is awesome.

I think the basic structure is now alright, though it'll take a lot of time, to now implement the actual initializers.

Currently you can launch a process from a string and handle the error (throw, assign, ignore). Throw on default.

I decided to remove the self-pipe feature, since it overcomplicates the child for a small convenience.

If someone is interested you might look at the following files (and yes, it's till messy)

• test/run_exe.cpp
• include/boost/process/execute.hpp
• include/boost/process/windows/executor.hpp
• include/boost/process/cmd.hpp
• include/boost/process/error.hpp

[klemens-morgenstern]

Status update

Sync I/O works as far as the bind__ and the close__ tests are concerned. The async-io stuff is not yet in, but hopefully follow soon.

[klemens-morgenstern]

Status update

I have basically every feature I'd like to have in boost.process somewhat working on windows.

The next steps will be:

1. Major Cleanup (yeah, don't look at the code as of yet...it's a mess..)
2. Documentation
3. Starting on the Posix-Version
4. Annoying everybody on the boost mailing lists and not only @BorisSchaeling
5. Implement error handling
6. Test like crazy...

I might also as a part of Step 1 remove the usage on hana and implement the meta-stuff myself and thereby downgrade the whole thing to C++11 (Though I promise nothing!).. That is because Hana is not widely supported and it's style bloats the binaries incredibely.

[klemens-morgenstern]

Ok, it's in alpha now, so the design is finished. Please see the doc if you're interested: http://klemens-morgenstern.github.io/process/