Issue with external Credential Process on Windows using WindowsSelectorEventLoopPolicy.

[ses4j]

Because psycopg doesn't support Proactor event loops, on Windows it's common to use Selector event loops for asyncio. However, Selector event loops don't support async subprocesses. The result of this is that this minimal code fails on Windows, if you have an .aws/config with a credential_process set up.

import asyncio, aioboto3

asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

async def test_upload():
    session = aioboto3.Session(region_name='us-east-1')
    async with session.client('s3') as client:
        await client.download_fileobj("my-bucket", "my-key", open("downloaded_test.py", "wb"))

asyncio.run(test_upload())

with the below stack. One fix is simply to detect this condition and not bother using async Popen. Calling sync Popen instead of self._popen works fine (like this:

    p = subprocess.Popen(process_list, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    stdout, stderr = p.communicate()

  File "...\.venv\Lib\site-packages\aiobotocore\session.py", line 197, in _create_client
    credentials = await self.get_credentials()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\.venv\Lib\site-packages\aiobotocore\session.py", line 100, in get_credentials
    self._credentials = await self._components.get_component(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\.venv\Lib\site-packages\aiobotocore\credentials.py", line 1024, in load_credentials
    creds = await provider.load()
            ^^^^^^^^^^^^^^^^^^^^^
  File "...\.venv\Lib\site-packages\aiobotocore\credentials.py", line 538, in load
    creds_dict = await self._retrieve_credentials_using(credential_process)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\.venv\Lib\site-packages\aiobotocore\credentials.py", line 558, in _retrieve_credentials_using
    p = await self._popen(
        ^^^^^^^^^^^^^^^^^^
  File "...\Python311\Lib\asyncio\subprocess.py", line 223, in create_subprocess_exec
    transport, protocol = await loop.subprocess_exec(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\Python311\Lib\asyncio\base_events.py", line 1708, in subprocess_exec
    transport = await self._make_subprocess_transport(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\Python311\Lib\asyncio\base_events.py", line 503, in _make_subprocess_transport
    raise NotImplementedError
NotImplementedError

[ses4j]

As a monkeypatch in case anyone needs it, this code seems to work:

class PatchedSyncPopen:
    def __init__(self, *args, **kwargs):
        self.args = args
        self.kwargs = kwargs
        import subprocess

        self._p = subprocess.Popen(args, **kwargs)

    async def communicate(self):
        loop = asyncio.get_running_loop()
        return await loop.run_in_executor(None, self._p.communicate)

    @property
    def returncode(self):
        return self._p.returncode

def patch_sync_popen_init(self, *args, **kwargs):
    async def _wrap_sync_popen_in_async(*args, **kwargs):
        return PatchedSyncPopen(*args, **kwargs)

    popen = _wrap_sync_popen_in_async
    super(self.__class__, self).__init__(*args, **kwargs, popen=popen)

if isinstance(asyncio.get_event_loop_policy(), asyncio.WindowsSelectorEventLoopPolicy):
    import aiobotocore.credentials

    aiobotocore.credentials.AioProcessProvider.__init__ = patch_sync_popen_init

[thehesiod]

cool thanks for the diagnosis, I'm on my third round of strep after antibiotics, if no one gets to it before I'm better will work on it asap