Serial port

[qhdwight]

Hi, I have a few questions about how the serial port works if that is okay, specifically when HyperDbg is run inside of a VMWare Workstation Pro machine and the control module is run in userspace on the host.

Is KdHyperDbgSendByte actually used? It seems like it calls Uart16550PutByte but none of the Uart initialization routines are called. When I try to do such techniques I get a UartNotReady issue, specifically reading a SERIAL_LSR_NOT_PRESENT value from the port.

[SinaKarvandi]

Hi,
Thanks for creating this discussion.

Yes, you've got it right. HyperDbg initializes the serial port from user mode and later uses KdHyperDbgSendByte to send bytes from kernel-mode. We spent a lot of hours figuring out how to initialize the serial port correctly; while, our initialization works perfectly in the physical computer (we test it with a serial device), the same initialization doesn't work on the VMware workstation, and I don't know why!

That's why we initialize the serial port from user-mode, but after that, we use KdHyperDbgSendByte in the kernel (VMX-root mode).

If I find time, I'll investigate how Windows initializes serial port in VMware workstation and update the initialization functions. If you figure out the problem, please let us know.

[qhdwight]

Thank you very much, this is super helpful. I have spent many hours trying to get it working as well haha, but it is hard with lack of any resources. I will take a look at the userspace initialization.

[qhdwight]

Got a link handy to the code that does that by any chance?

[SinaKarvandi]

Also, one thing that is worth to be mentioned is that if we map the VMware to a physical COM port, again, it works perfectly. The problem arises when mapping VMware to a named pipe.

Got a link handy to the code that does that by any chance?

Well, we tested the below projects. The source code is available here:
https://github.com/HyperDbg/communication

Also, we tested this code, but it still didn't work on the scenario as mentioned above, but it works on the physical machine.
https://github.com/Cr4sh/SmmBackdoor/blob/ef2423a2dd6508327063de84c47193b88033becc/SmmBackdoor/serial.c