Fix hang by adding a timeout for each evaluation of a variable

[AlexEne]

I encountered some hangs in Zed + CodeLLDB. They are triggered by this issue at the root: llvm/llvm-project#196812

LLDB bug here llvm/llvm-project#196812 , triggers the following infinite loop in the rust python scripts that I fixed here: rust-lang/rust#162725

However, when root causing this, I figured out it would be nicer if codelldb has a timeout per variable to make it more resilient when python scripts are unresponsive (the current timeouts are insufficient and the example below still hangs):

#[test]
fn test_lldb_fail() {
    struct Holder {
        rc1: std::rc::Rc<u32>,
        rc2: std::rc::Rc<u32>,
        rc3: std::rc::Rc<u32>,
    }
    let cell: std::cell::Cell<u64> = std::cell::Cell::new(0);
    let holder = Holder {
        rc1: std::rc::Rc::new(1),
        rc2: std::rc::Rc::new(2),
        rc3: std::rc::Rc::new(3),
    };
    println!("breakpoint here: {} {}", cell.get(), holder.rc1);
    println!("Second one");
}

I tested it locally and after the 5s timeout it closes the sessions that hang and at least debugging can continue.

I tried my best to keep it simple (it does start an extra thread) but it stays mainly parked so it shouldn't have much overhead.

I also added a way to absorb a pending interrupt in case one was emitted but in the meantime python exited (it's a bit of a racy situation).

[AlexEne]

Hi! Let me know if there's a different approach you want to take this into and I can try it. One thing I was thinking as an alternative is to have the watchdog thread always running and enable it with each request. But tbh not sure it's a big improvement over this.

[vadimcn]

At first glance, this change looked too complicated for the task, but I didn't get around to investigate what's going on there.

Since you've already investigated the issue, can you please give more details on this change? Why did you need to add EvalWatchdog? Why is drain_interrupt needed?
Thanks!

[AlexEne]

The existing timeout in convert_scope_values works if the calls to the visualizers return (and are slow), but if they never return from var_to_variable it never gets reached to bail that loop).

So i needed this extra thread for this reason. E.g. you have 2+ Rcs in the situation above you get the whole LLDB session hanging if python hangs on more than one member field (in var_to_variable).

I added the drain method because I thought it would be nice to make sure the interrupt sent doesn't affect subsequent sessions (since this is a bit racy). E.g. the timeout watchdog decides to issue a timeout interrupt, but in the meantime python finishes, that interrupt actually reaches potentially the wrong, next eval).

I'm not super familiar with the codebase tbh, so if you have a suggestion on how to fix it (e.g. some other place), I can try implementing it. The main idea is to never hang the debug session no matter how many variables would trigger an infinite loop in the visualizers (like rust does for the above case).

[AlexEne]

Actually re-reading the code now, I don't think multiple Rcs are required to reproduce the issue, it just is an example that got small enough and I could reproduce the issue, but maybe one is sufficient as a member field.

The execution model here is a bit fuzzy in my head, I could just see it get locked on this thread from calling var_to_variable :D