[DPE-9996] Make scale from zero check more robust

[astrojuanlu]

Issue

Fix gh-293.

Solution

Checklist

• I have added or updated any relevant documentation.
• I have cleaned any remaining cloud resources from my accounts.

[astrojuanlu]

$ noctua charm release mysql-k8s --json --path ./mysql-k8s_ubuntu@22.04-amd64.charm --channel 8.0/edge/robust-scale-from-zero
{"name": "mysql-k8s", "revision": 413, "resources": [{"name": "mysql-image", "revision": 122, "upstream_source": "ghcr.io/canonical/charmed-mysql@sha256:824b302484f83dbbd2cf4506f620a48e53f20053f6ff3c57aa6123e121705bfc"}]}

[astrojuanlu]

test_primary_switchover.py consistently fails on this one. Will investigate.

[astrojuanlu]

This fix is too naïve:

unit-mysql-k8s-0: 07:23:56 INFO unit.mysql-k8s/0.juju-log Scaling out from zero units
unit-mysql-k8s-0: 07:23:56 DEBUG unit.mysql-k8s/0.juju-log get_k8s_fqdn: name='mysql-k8s-0.mysql-k8s-endpoints' name_prefix='mysql-k8s-0' local_unit_label='mysql-k8s-0'
unit-mysql-k8s-0: 07:23:56 DEBUG unit.mysql-k8s/0.juju-log get_k8s_fqdn: local unit path -> canonname='mysql-k8s-0.mysql-k8s-endpoints.testing.svc.cluster.local'
unit-mysql-k8s-0: 07:23:56 DEBUG unit.mysql-k8s/0.juju-log Creating InnoDB cluster cluster-7bfcbb03f1ad1fe58a96bde8ce72a9f5
unit-mysql-k8s-0: 07:23:57 ERROR unit.mysql-k8s/0.juju-log Failed to create cluster cluster-7bfcbb03f1ad1fe58a96bde8ce72a9f5
unit-mysql-k8s-0: 07:23:57 ERROR unit.mysql-k8s/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/lib/python3.10/site-packages/mysql_shell_contrib/executors/pebble.py", line 157, in execute_py
    stdout, _ = process.wait_output()
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/lib/python3.10/site-packages/ops/pebble.py", line 1864, in wait_output
    raise ExecError[AnyStr](self._command, exit_code, out_value, err_value)
ops.pebble.ExecError: non-zero exit code 1 executing ['/usr/bin/mysqlsh', '--json=raw', '--save-passwords=never', '--passwords-from-stdin', '--host=mysql-k8s-0.mysql-k8s-endpoints.testing.svc.cluster.local.', '--port=3306', '--user=clusteradmin', '--py', '--execute', "shell.options.set('useWizards', False)\ndba.create_cluster('cluster-7bfcbb03f1ad1fe58a96bde8ce72a9f5', {'communicationStack': 'MySQL'})"], stdout='{"prompt":"Please provide the password for \'clusteradmin@mysql-k8s-0.mysql-k8s-endpoints.testing.svc.cluster.local.:3306\': ","type":"password","defaultValue":""}\n{"error":"Traceback (most recent call last):\\n"}\n{"error":"  File \\"<string>\\", line 2, in <module>\\n"}\n{"error":"RuntimeError"}\n{"error":": "}\n{"error":"Dba.create_cluster: dba.create_cluster: Unable to create cluster. The instance \'mysql-k8s-0.mysql-k8s-endpoints.testing.svc.cluster.local.:3306\' has a populated Metadata schema and belongs to that Metadata. Use either dba.drop_metadata_schema() to drop the schema, or dba.reboot_cluster_from_complete_outage() to reboot the cluster from complete outage.\\n"}\n{"error":"\\n"}\n', stderr='Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory\n'
...
unit-mysql-k8s-0: 07:24:04 INFO unit.mysql-k8s/0.juju-log Scaling out from zero units
unit-mysql-k8s-0: 07:24:04 DEBUG unit.mysql-k8s/0.juju-log get_k8s_fqdn: name='mysql-k8s-0.mysql-k8s-endpoints' name_prefix='mysql-k8s-0' local_unit_label='mysql-k8s-0'
unit-mysql-k8s-0: 07:24:04 DEBUG unit.mysql-k8s/0.juju-log get_k8s_fqdn: local unit path -> canonname='mysql-k8s-0.mysql-k8s-endpoints.testing.svc.cluster.local'
unit-mysql-k8s-0: 07:24:04 DEBUG unit.mysql-k8s/0.juju-log Creating InnoDB cluster cluster-7bfcbb03f1ad1fe58a96bde8ce72a9f5
unit-mysql-k8s-0: 07:24:04 ERROR unit.mysql-k8s/0.juju-log Failed to create cluster cluster-7bfcbb03f1ad1fe58a96bde8ce72a9f5
unit-mysql-k8s-0: 07:24:04 ERROR unit.mysql-k8s/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/lib/python3.10/site-packages/mysql_shell_contrib/executors/pebble.py", line 157, in execute_py
    stdout, _ = process.wait_output()
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/lib/python3.10/site-packages/ops/pebble.py", line 1864, in wait_output
    raise ExecError[AnyStr](self._command, exit_code, out_value, err_value)
ops.pebble.ExecError: non-zero exit code 1 executing ['/usr/bin/mysqlsh', '--json=raw', '--save-passwords=never', '--passwords-from-stdin', '--host=mysql-k8s-0.mysql-k8s-endpoints.testing.svc.cluster.local.', '--port=3306', '--user=clusteradmin', '--py', '--execute', "shell.options.set('useWizards', False)\ndba.create_cluster('cluster-7bfcbb03f1ad1fe58a96bde8ce72a9f5', {'communicationStack': 'MySQL'})"], stdout='{"prompt":"Please provide the password for \'clusteradmin@mysql-k8s-0.mysql-k8s-endpoints.testing.svc.cluster.local.:3306\': ","type":"password","defaultValue":""}\n{"error":"Traceback (most recent call last):\\n"}\n{"error":"  File \\"<string>\\", line 2, in <module>\\n"}\n{"error":"mysqlsh"}\n{"error":"."}\n{"error":"Error"}\n{"error":": "}\n{"error":"Shell Error (51601): Dba.create_cluster: This function is not available through a session to an InnoDB Cluster that belongs to an InnoDB ClusterSet"}\n{"error":"\\n"}\n', stderr='Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory\n'

https://github.com/canonical/mysql-operators/actions/runs/27259373112/job/80502367712?pr=295#step:13:2050

Despite this method being used in the machines charm since #288 (comment)

Will need more thorough analysis.