Increasingly frequent battery charge level dropouts in cloud mode

[martinmanyhats]

I run a copy of PD which is configured for Cloud Mode for when I'm messing around with my local network and so can't connect directly to my Powerwall 3. This has been stable for months with the very occasional (maybe once per week) data sample which shows 0% battery charge.

Recently the rate of dropout has been increasing daily. In the last 24 hours it has gone somewhat berserk, see attached screenshot. What I observe is the flow animation not showing a charge level, then showing zero, then recovering.

Anyone got any thoughts on what might be going on, please? Or things to check?

PD is running in a VM under Alpine Linux 3.20 which is also running other services fine. Internet connection is fast and stable.

Output of verify.sh:

Verify Powerwall-Dashboard 4.8.4 on Linux - Timezone: Europe/London
----------------------------------------------------------------------------
This script will attempt to verify all the services needed to run
Powerwall-Dashboard. Use this output when you open an issue for help:
https://github.com/jasonacox/Powerwall-Dashboard/issues/new

Tip: If colors are hard to read, try: ./verify.sh --no-color, --lightbg, or --darkbg


Checking pypowerwall
----------------------------------------------------------------------------
 - Config File pypowerwall.env: GOOD
 - Container (pypowerwall): GOOD
 - Service (port 8675): GOOD
 - Version: 0.14.0 Proxy t80
 - Powerwall State: CONNECTED - Firmware: 25.18.1
 - Site Name: My Home
 - Gateway TEDAPI: Available (192.168.91.1)
 - TEDAPI Vitals: Not Connected
 - Cloud Mode: YES - Site ID: XXXXXX (My Home)

Checking telegraf
----------------------------------------------------------------------------
 - Config File telegraf.conf: GOOD
 - Local Config File telegraf.local: GOOD
 - Container (telegraf): GOOD
 - Version: Telegraf 1.28.2 (git: HEAD@8d9cf395)

Checking influxdb
----------------------------------------------------------------------------
 - Config File influxdb.conf: GOOD
 - Environment File influxdb.env: GOOD
 - Container (influxdb): GOOD
 - Service (port 8086): GOOD
 - Filesystem (./influxdb): GOOD
 - Version: InfluxDB shell version: 1.8.10

Checking grafana
----------------------------------------------------------------------------
 - Config File grafana.env: GOOD
 - Container (grafana): GOOD
 - Service (port 9000): GOOD
 - Filesystem (./grafana): GOOD
 - Version: Grafana CLI version 9.1.2

Checking weather411
----------------------------------------------------------------------------
 - Container (weather411): GOOD
 - Service (port 8676): GOOD
 - Weather: {"temperature": 16.35}
 - Version: 0.2.3

All tests succeeded.

Thanks, Martin

[jasonacox]

Hi @martinmanyhats can you look in the log file for errors?

docker logs pypowerwall

I suspect you are getting read timeouts with the cloud. Tesla has recently made changes to their cloud API so they could be rate limiting those calls. If that is the case, we will need to decrease the polling frequency. An easy way to do that is to increase the cache timeout. Edit pypowerwall.env and add:

PW_CACHE_EXPIRE=10

You could also make it 15, 20, etc., but 10 would double the default (5). Restart the stack:

./compose-bash.sh up -d

[martinmanyhats]

Thanks @jasonacox. Log contains many:

08/02/2025 04:42:03 PM [pypowerwall.cloud.pypowerwall_cloud] [ERROR] Failed to retrieve SITE_SUMMARY - ReadTimeout(ReadTimeoutError("HTTPSConnectionPool(host='owner-api.teslamotors.com', port=443): Read timed out. (read timeout=5)"))
Have increased cache timeout to 30s as that would suit me fine.

Unfortunately the problem persists, albeit now across two different requests. Previously was 95%+ SITE_SUMMARY with the occasional ENERGY_SITE_BACKUP_TIME_REMAINING.

08/02/2025 05:05:30 PM [pypowerwall.cloud.pypowerwall_cloud] [ERROR] Failed to retrieve ENERGY_SITE_BACKUP_TIME_REMAINING - ReadTimeout(ReadTimeoutError("HTTPSConnectionPool(host='owner-api.teslamotors.com', port=443): Read timed out. (read timeout=5)"))
08/02/2025 05:10:32 PM [pypowerwall.cloud.pypowerwall_cloud] [ERROR] Failed to retrieve ENERGY_SITE_BACKUP_TIME_REMAINING - HTTPError('500 Server Error: upstream internal error for url: https://owner-api.teslamotors.com/api/1/energy_sites/1689423123316909/backup_time_remaining?language=en')
08/02/2025 05:12:00 PM [pypowerwall.cloud.pypowerwall_cloud] [ERROR] Failed to retrieve SITE_SUMMARY - ReadTimeout(ReadTimeoutError("HTTPSConnectionPool(host='owner-api.teslamotors.com', port=443): Read timed out. (read timeout=5)"))
08/02/2025 05:15:55 PM [pypowerwall.cloud.pypowerwall_cloud] [ERROR] Failed to retrieve ENERGY_SITE_BACKUP_TIME_REMAINING - ReadTimeout(ReadTimeoutError("HTTPSConnectionPool(host='owner-api.teslamotors.com', port=443): Read timed out. (read timeout=5)"))
08/02/2025 05:20:31 PM [pypowerwall.cloud.pypowerwall_cloud] [ERROR] Failed to retrieve SITE_SUMMARY - ReadTimeout(ReadTimeoutError("HTTPSConnectionPool(host='owner-api.teslamotors.com', port=443): Read timed out. (read timeout=5)"))

I've increased the cache timeout to 60s and will report back.

Martin

[jasonacox]

Hey @martinmanyhats - If you are still getting a lot of timeout errors, this could indicate a local or global network issue too. Can you run a network tests?

# Test a site that responds to ping
ping pypowerwall.com -c 25 -s 1024

Also - this is very interesting:

HTTPError('500 Server Error: upstream internal error for url: https://owner-api.teslamotors.com/api/1/energy_sites/1689423123316909/backup_time_remaining?language=en')

The 500 status code is a generic error code that means the Tesla API server encountered an unexpected condition that prevented it from fulfilling the request. That could indicate that they are having issues with load or software that is producing these errors. If that is the case, there isn't a lot we could do to fix it. The fact that your sample rate didn't change timeout frequency indicates that some % of time, Tesla servers are not healthy.

You can also see what pypowerwall thinks about the health:

http://localhost:8675/health

[martinmanyhats]

Hi @jasonacox

Yes, still getting the same errors despite the longer cache timeout, so an issue at Tesla's end would seem a distinct possibility.

Ran the ping text 5 times with short pauses in between, zero drops.

pypowerwall health info looks fine apart from the expected timeouts.

At this point I'll give it 24 hours or more to see what happens and report back. Indeed, might have to wait out the weekend unless someone is burning overtime at Tesla :)

[jasonacox]

Others are seeing the same thing. I have been testing on my system (moved from local to cloud mode) and can verify the timeouts are occuring. However, it is not just the Tesla Owners API, it is also showing up with FleetAPI.

08/02/2025 04:15:35 PM [pypowerwall.fleetapi.fleetapi] [ERROR] Timeout error polling https://fleet-api.prd.na.vn.cloud.tesla.com/api/1/energy_sites/x/site_status
08/02/2025 04:15:35 PM [pypowerwall.fleetapi.fleetapi] [ERROR] Timeout error polling https://fleet-api.prd.na.vn.cloud.tesla.com/api/1/energy_sites/x/site_status
08/02/2025 04:15:35 PM [pypowerwall.fleetapi.fleetapi] [ERROR] Timeout error polling https://fleet-api.prd.na.vn.cloud.tesla.com/api/1/energy_sites/x/site_status

I even switched to an older version of pypowerwall to eliminate the possiblity of the new network health code causing this. I'm getitng the same results. I thought we might be able to mitigate this a little by increasing the timeout setting by editing pypowerwall.envand adding this:

PW_CACHE_EXPIRE=20
PW_TIMEOUT=10

Restart with ./compose-dash.sh up -d

I just tried that myself and started seeing new errors. Occasionally, instead of getting a valid response, we get an error (504) from Tesla:

08/02/2025 04:30:30 PM [pypowerwall.fleetapi.fleetapi] [ERROR] Code 504: {"response":null,"error":"https://powergate.prd.sn.tesla.services:443/api/v2/energy_site/status =\u003e Gateway Timeout","error_description":"","txid":"d68cdffac10aff25c473aaa7b8c99dd4"}
08/02/2025 04:30:30 PM [pypowerwall.fleetapi.fleetapi] [ERROR] Code 504: {"response":null,"error":"https://powergate.prd.sn.tesla.services:443/api/v2/energy_site/status =\u003e Gateway Timeout","error_description":"","txid":"17bb20c2d65b7e0eff54879dcf43ffba"}

This error is occurring on a Tesla server trying to retrieve the status of my energy site from the Powergate API. The error message specifically mentions a timeout when trying to connect to one of their internal API endpoint at https://powergate.prd.sn.tesla.services:443/api/v2/energy_site/status.

In other words, there isn't a lot we can do but wait for Tesla to fix this on their side.