diff --git a/docs/capabilities/memory/hosted_memory2_recommendations.md b/docs/capabilities/memory/hosted_memory2_recommendations.md new file mode 100644 index 0000000000..d0f218ede6 --- /dev/null +++ b/docs/capabilities/memory/hosted_memory2_recommendations.md @@ -0,0 +1,1001 @@ +# 1. Comparison Target: Native PostgreSQL vs Application-Level Sync + +The reason for building this experiment is to compare native PostgreSQL logical replication with the proposed DimensionalOS Hosted Memory 2 synchronization approach. + +The native PostgreSQL approach looks like: + +```text +PostgreSQL + │ + ├── WAL + ├── Logical decoding + ├── Publication + ├── Replication slot + └── Subscription + │ + ▼ + PostgreSQL +``` + +Whereas an application-level sync mechanism generally looks more like: + +```text +Application / Memory Store + │ + ▼ + Change detection + │ + ▼ + Sync engine + │ + ▼ + Remote store +``` + +The native PostgreSQL approach is attractive when: + +* Both endpoints are PostgreSQL (currently it's SQLite) +* The schema is compatible +* Continuous connectivity is acceptable +* PostgreSQL itself is the source of truth +* Database-level replication is sufficient + +The application-level approach becomes more interesting when: + +* The source and destination are different storage systems (SQLite vs PostgreSQL) +* The destination is not PostgreSQL (S3 or OpenSearch) +* Custom conflict resolution is required +* Sync semantics are application-specific +* The system needs custom filtering/transformation +* The source can be offline +* Blob/object storage needs separate handling +* The system needs a higher-level synchronization abstraction + +This experiment is intended to establish the baseline of what PostgreSQL itself already provides before deciding what additional capabilities DimensionalOS needs to build. + +# DimensionalOS Memory2 — Recommendations & Discussion Points + +## 1. Recommendation on PostgreSQL Logical Replication + +**PostgreSQL logical replication can be a very good foundation for Memory2 synchronization** + +I would not recommend building a custom replication/change-log mechanism if PostgreSQL is already the source of truth on both the robot and cloud side. + +My recommendation would be: + +> **Use PostgreSQL native logical replication for database-level change propagation, and add application-level synchronization only where the product requires semantics beyond database replication.** + +The important distinction is that logical replication fundamentally solves: + +> **"How do we replicate database changes from A → B?"** + +It does not necessarily solve: + +> **"What information should A share with B, when should it be shared, and how should conflicts be resolved?"** + +--- + +## 2. Where I Think PostgreSQL Logical Replication Works Well + +If we have: + +```text +Robot + │ + ▼ +Memory2 + │ + ▼ +PostgreSQL +``` + +and: + +```text +Cloud + │ + ▼ +Memory2 + │ + ▼ +PostgreSQL +``` + +then PostgreSQL logical replication gives us a native mechanism to propagate database changes between the two. + +I would prefer this over introducing another custom change-data-capture layer unless we identify a concrete requirement that PostgreSQL cannot satisfy. + +This reduces: + +* custom synchronization code +* additional infrastructure +* another durability mechanism +* another change-log implementation +* operational complexity + +--- + +# 3. Where I Would Still Keep an Application-Level Sync Layer + +The area where I would not rely solely on PostgreSQL is **application-level synchronization policy**. + +For example, imagine: + +```text +Robot A +Robot B +Robot C +Robot D +``` + +We may not want every observation generated by Robot A to be synchronized everywhere. + +Instead, we might want: + +```text +Robot A + │ + ├── Safety observations + │ └── share with nearby robots + │ + ├── Navigation state + │ └── share with relevant robots + │ + └── Raw camera data + └── remain local / cloud-backed +``` + +This introduces questions such as: + +* Which robot should receive the data? +* Which streams should be shared? +* Should sharing depend on spatial proximity? +* Should certain data have higher priority? +* What bandwidth should be allocated? +* What data should remain local? +* What permissions apply? + +I would consider these **Memory2/application-level synchronization semantics**, rather than database replication semantics. + +PostgreSQL row filters and publications can help with some forms of selective replication, but I would not want the database configuration itself to become the definition of our product-level synchronization model. + +--- + +# 4. Conflict Resolution + +Another area I would explicitly discuss is conflict resolution. + +For example: + +```text +Robot A: +Object X → (10,20) + +Robot B: +Object X → (12,22) +``` + +PostgreSQL can replicate the database changes, but the system still needs to determine: + +> **Which observation should represent the current world state?** + +Depending on the Memory2 semantics, we might want: + +* latest observation wins +* highest-confidence observation wins +* authoritative robot wins +* merge observations +* preserve both observations +* resolve based on timestamps or versions + +I would therefore separate: + +```text +Replication + ↓ +Move database changes +``` + +from: + +```text +Conflict resolution + ↓ +Determine what the data means +``` + +The second is something I would expect Memory2 to own. + +--- + +# 5. Recommendation for Large Binary Data + +I would also recommend **not treating PostgreSQL as the primary object store for large robotics payloads**. + +This does not mean PostgreSQL cannot store blobs. + +It absolutely can. + +For example: + +```sql +payload BYTEA +``` + +is a perfectly valid design for smaller binary data. + +My argument would instead be: + +> **The question is not whether PostgreSQL can store blobs. The question is whether PostgreSQL is the best system for storing very large, high-volume robotics objects.** + +For Memory2, I expect we could eventually have: + +* camera images +* video +* LiDAR +* point clouds +* depth maps +* audio +* large sensor payloads +* model artifacts + +I would therefore separate structured state from large objects. + +--- + +# 6. Recommended Data Model + +Instead of: + +```text +Postgres +┌─────────────────────────┐ +│ observation_id │ +│ timestamp │ +│ robot_id │ +│ pose │ +│ metadata │ +│ image BYTEA │ +│ pointcloud BYTEA │ +└─────────────────────────┘ +``` + +I would recommend: + +```text +Postgres +┌─────────────────────────┐ +│ observation_id │ +│ timestamp │ +│ robot_id │ +│ pose │ +│ metadata │ +│ blob_uri │ +└────────────┬────────────┘ + │ + ▼ + S3 + ┌───────────────┐ + │ image │ + │ video │ + │ LiDAR │ + │ point cloud │ + │ audio │ + └───────────────┘ +``` + +This gives us a clean separation: + +**PostgreSQL** + +* metadata +* observations +* relationships +* vectors +* queryable state +* blob references + +**S3/object storage** + +* large binary objects + +--- + +# 7. Why I Prefer S3 for Large Objects + +My primary concern with putting very large objects directly into PostgreSQL is that it couples the blob workload to the database workload. + +Large binary data can increase: + +* database storage +* WAL volume +* replication traffic +* backup size +* I/O pressure +* recovery requirements +* database maintenance overhead + +For example, if a robot writes a large image into PostgreSQL: + +```text +Robot + │ + ▼ +Postgres + │ + ├── storage + ├── WAL + ├── replication + └── backup +``` + +With object storage: + +```text +Robot + │ + ├──────── metadata ────────► PostgreSQL + │ + └──────── image ───────────► S3 +``` + +Then logical replication primarily handles: + +```text +observation_id +timestamp +robot_id +pose +metadata +blob_uri +``` + +rather than continuously moving large binary payloads through the PostgreSQL replication path. + +I think this is a much cleaner architecture for a robotics system. + +--- + +# 8. Independent Scaling + +Another reason I prefer this separation is independent scaling. + +We could have a workload like: + +```text +Relational queries: +1,000/sec + +Blob traffic: +100 GB/sec +``` + +I would not want those two workloads competing for the same database resources. + +Instead: + +```text + Memory2 + │ + ┌────────┴────────┐ + │ │ + PostgreSQL S3 + │ │ + metadata/query blob storage +``` + +This allows us to scale the two workloads independently. + +--- + +# 9. Direct Blob Access + +I also think object storage gives us a better access pattern for clients. + +Instead of: + +```text +Frontend + │ + ▼ +Memory2 API + │ + ▼ +Postgres + │ + ▼ +Large image + │ + ▼ +Memory2 API + │ + ▼ +Frontend +``` + +we can have: + +```text +Frontend + │ + ▼ +Memory2 API + │ + ├── authorization + ├── metadata + │ + └── pre-signed URL + │ + ▼ + S3 + │ + ▼ + Image +``` + +Memory2 remains responsible for authorization and metadata, while S3 handles the large data transfer. + +--- + +# 10. I Would Not Make "Never Store Blobs in PostgreSQL" a Rule + +I would still allow smaller payloads to live directly in PostgreSQL where it makes the system simpler. + +For example: + +```text +Small payload + │ + ▼ +Postgres BYTEA +``` + +while: + +```text +Large payload + │ + ▼ +S3 +``` + +I would determine the actual threshold through benchmarking and workload characteristics rather than arbitrarily choosing a fixed number such as 1 MB or 10 MB. + +This gives us flexibility without turning PostgreSQL into our primary object store. + +--- + +# 12. Questions for the Team + +### Database topology + +> "Are we envisioning one PostgreSQL database per robot, with logical replication into the hosted PostgreSQL cluster?" + +### Selective synchronization + +> "How do we handle selective replication at the product level? For example, if Robot A should only share observations from a particular stream or spatial region with Robot B, do we expect PostgreSQL row filtering/publications to handle that, or do we need Memory2-level synchronization logic?" + +### Conflict resolution + +> "If two robots independently update the same logical object, what determines the winning state?" + +### Offline behavior + +> "What happens when a robot is offline for several hours or days and then reconnects? What is our expected catch-up and conflict-resolution behavior?" + +### Large objects + +> "For large observations such as images, video, or point clouds, do we want those inside PostgreSQL, or should PostgreSQL contain the metadata/reference while S3 stores the actual object?" + +--- + +# The interesting failure cases + +This design becomes particularly useful when the robot loses network or power. + +### Case 1 — Robot loses network before S3 upload + +``` +Postgres → PENDING +S3 → nothing +``` + +When network comes back: + +``` +PENDING + ↓ +retry upload + ↓ +COMPLETE +``` + +No data is lost as long as the local staged payload is retained. + +### Case 2 — S3 upload succeeds, robot loses power before Postgres update + +You could have: + +``` +S3 → object exists +Postgres → PENDING +``` + +When the robot comes back, the uploader can check/retry using the deterministic object key: + +s3://bucket/robot-17/abc123 + +If the object already exists and checksum matches: + +PENDING → COMPLETE + +This is why I would make the upload idempotent. + +### Case 3 — Postgres metadata replicates to cloud before S3 upload + +Cloud could temporarily see: + +``` +observation abc123 +blob_status = PENDING +``` + +That's okay. + +PENDING means the metadata exists, but the payload is not yet available. + +Once S3 upload completes: + +``` +PENDING → COMPLETE +``` + +and that status update itself can replicate. + +```text + ROBOT + │ + ▼ + ┌─────────────────┐ + │ Memory2 API │ + └────────┬────────┘ + │ + ┌────────┴─────────┐ + │ │ + ▼ ▼ + PostgreSQL Local staging + metadata payload + │ │ + │ ▼ + │ S3 + │ + ▼ + Logical replication + │ + ▼ + Cloud PostgreSQL +``` + +# Some Observability and Analytics tracking recommendations + +These are going to be custom metrics need to be created and collected as a separate cloud service. We can use Prometheus or VictoriaMetrics for metrics, AlertManager for alerting on critical failures and grafana for dashboards. + +## Goal + +The analytics/observability layer should make it easy to answer: + +1. Is the robot healthy? +2. Is synchronization healthy? +3. How far behind is each robot? +4. Is data being lost or delayed? +5. How much data is being generated and synchronized? +6. What happens during network outages and recovery? +7. Where is the sync pipeline becoming a bottleneck? + +--- + +## 1. Sync Health Metrics + +These should be the primary metrics. + +| Metric | Description | +|---|---| +| `sync_status` | Connected / disconnected / syncing / error | +| `sync_lag_seconds` | Time between local data creation and cloud availability | +| `pending_records` | Number of records waiting to synchronize | +| `pending_bytes` | Amount of data waiting to synchronize | +| `last_successful_sync_timestamp` | Last time the robot successfully synchronized | +| `last_sync_error_timestamp` | Last synchronization failure | +| `sync_error_count` | Number of sync failures | +| `sync_retry_count` | Number of retries | +| `sync_success_rate` | Percentage of successful sync operations | + +The most important metric is: + +```text +sync_lag_seconds +```` + +because a system can be technically "connected" while still being significantly behind. + +--- + +## 2. Throughput Metrics + +Measure how much data the system is processing. + +### Local / Robot + +* `records_generated_total` +* `records_generated_per_second` +* `bytes_generated_total` +* `bytes_generated_per_second` + +### Sync + +* `records_synced_total` +* `records_synced_per_second` +* `bytes_synced_total` +* `bytes_synced_per_second` + +### Cloud + +* `records_received_total` +* `bytes_received_total` +* `records_processed_per_second` + +This allows us to determine whether the sync system can keep up with the robot's data-generation rate. + +For example: + +```text +Robot generation rate: 5,000 records/sec +Sync rate: 4,000 records/sec + +→ backlog is growing +``` + +--- + +## 3. Backlog / Queue Metrics + +Backlog is especially important for an edge/robot system. + +Track: + +```text +pending_records +pending_bytes +oldest_pending_record_age +``` + +Example: + +```text +Robot: robot-001 + +Pending records: 1,240,000 +Pending bytes: 8.4 GB +Oldest pending record: 2h 17m +``` + +The **age of the oldest pending record** is particularly useful because record count alone doesn't tell us how stale the data is. + +--- + +## 4. Offline / Connectivity Metrics + +Robots may frequently operate without connectivity. + +Track: + +* `connection_state` +* `disconnect_count` +* `total_offline_duration` +* `current_offline_duration` +* `reconnection_count` +* `time_to_recover` +* `sync_duration_after_reconnect` + +Example: + +```text +robot-001 + +Offline duration: 47 minutes +Pending data: 2.3 GB +Reconnect time: 3 seconds +Catch-up time: 8 minutes +``` + +This gives us a clear picture of how well the system handles intermittent connectivity. + +--- + +## 5. Recovery Metrics + +When a robot reconnects after being offline, measure: + +```text +offline_duration +backlog_at_reconnect +catchup_duration +catchup_rate +remaining_backlog +``` + +A useful metric is: + +```text +catchup_ratio = + sync_rate / data_generation_rate +``` + +If: + +```text +sync_rate > generation_rate +``` + +the robot will eventually catch up. + +If: + +```text +sync_rate < generation_rate +``` + +the backlog will continue growing. + +--- + +## 6. End-to-End Latency + +Measure the complete lifecycle of a record. + +Ideally each record/event has timestamps such as: + +```text +created_at +persisted_at +sync_started_at +received_at +processed_at +``` + +This allows us to measure: + +```text +Generation → Local Persistence +Local Persistence → Network Transfer +Network Transfer → Cloud +Cloud → Processing +``` + +The primary metric should be: + +```text +end_to_end_latency_seconds +``` + +Track: + +* p50 +* p95 +* p99 +* max + +Example: + +```text +Sync latency + +p50: 180 ms +p95: 1.2 sec +p99: 4.8 sec +max: 31 sec +``` + +--- + +## 7. Error Metrics + +Track errors by category rather than only having a generic error counter. + +Recommended dimensions: + +```text +error_type +robot_id +operation +destination +``` + +Example error types: + +```text +network_error +authentication_error +serialization_error +schema_error +storage_error +conflict +timeout +retry_exhausted +``` + +Metrics: + +```text +sync_errors_total +sync_retries_total +sync_timeouts_total +sync_conflicts_total +``` + +This makes debugging significantly easier. + +--- + +## 8. Storage Metrics + +Because robots may operate offline, local storage is critical. + +Track: + +```text +local_storage_used_bytes +local_storage_available_bytes +sync_queue_size_bytes +database_size_bytes +blob_storage_size_bytes +wal_size_bytes +``` + +Also expose percentages: + +```text +local_storage_utilization_percent +``` + +Example: + +```text +Robot Storage + +Used: 71 GB +Available: 29 GB +Utilization: 71% + +Pending sync: 18 GB +``` + +This allows the system to detect: + +> "The robot is offline and will run out of storage in approximately 3 hours." + +That is much more useful than simply reporting "sync disconnected." + +--- + +## 9. Blob / Image Metrics + +If robot data includes camera images or other large objects, track them separately from normal records. + +Recommended metrics: + +```text +images_generated_total +images_synced_total +image_bytes_generated_total +image_bytes_synced_total +average_image_size_bytes +largest_image_size_bytes +blob_sync_latency +blob_failures_total +``` + +This is important because a system may have excellent record-level sync performance while being bottlenecked by large media objects. + +--- + +## 10. Resource Metrics + +Monitor the resources consumed by the sync system: + +### CPU + +```text +sync_cpu_usage +``` + +### Memory + +```text +sync_memory_usage_bytes +``` + +### Disk + +```text +disk_usage_bytes +disk_available_bytes +``` + +### Network + +```text +network_bytes_sent +network_bytes_received +network_bandwidth +``` + +### Database + +```text +database_connections +database_size +transaction_rate +query_latency +``` + +For PostgreSQL specifically, also monitor: + +```text +WAL generation rate +WAL retained +replication slot lag +replication lag +``` + +--- + +## 11. Robot-Level Dashboard + +The primary dashboard should be robot-centric. + +Example: + +```text +Robot Status Lag Pending Offline +--------------------------------------------------------- +robot-001 Healthy 120ms 0 - +robot-002 Syncing 4.2m 42K 18m +robot-003 Offline 2.1h 1.8M 2.1h +robot-004 Error 31m 320K - +``` + +## 12. Alerts + +Alerts should focus on actionable conditions. + +### Critical + +```text +Robot storage > 90% +Sync completely stalled +Replication slot WAL > configured threshold +Data loss detected +Database unavailable +``` + +### Warning + +```text +Sync lag > 5 minutes +Pending data continuously increasing +Robot offline > 30 minutes +Catch-up rate < generation rate +Disk utilization > 75% +Repeated sync failures +``` + +### Informational + +```text +Robot reconnected +Catch-up completed +Sync recovered after failure +``` + +--- diff --git a/docs/capabilities/memory/hosted_memory2_trail_test.md b/docs/capabilities/memory/hosted_memory2_trail_test.md new file mode 100644 index 0000000000..4ae1477cd6 --- /dev/null +++ b/docs/capabilities/memory/hosted_memory2_trail_test.md @@ -0,0 +1,1297 @@ +# PostgreSQL Native Logical Replication Demo + +## Purpose + +The goal is to demonstrate: + +- Local PostgreSQL as a publisher +- AWS RDS PostgreSQL as a subscriber +- PostgreSQL native logical replication +- Initial synchronization of existing data +- Automatic propagation of INSERT/UPDATE/DELETE operations +- The networking and privilege requirements involved + +This is a standalone experiment using an artificial `sync_test` table. No DimensionalOS replay engine or full memory store is required. + +--- + +# 1. Architecture + +The intended architecture is: + +```text + Native PostgreSQL Logical Replication + +┌───────────────────────────────┐ +│ Local Mac │ +│ │ +│ Docker │ +│ ┌───────────────────────────┐ │ +│ │ PostgreSQL 17.11 │ │ +│ │ │ │ +│ │ Database: dimensionalos │ │ +│ │ │ │ +│ │ sync_test │ │ +│ │ 66 existing rows │ │ +│ │ │ │ +│ │ Publication: dimos_pub │ │ +│ └─────────────┬─────────────┘ │ +└───────────────┼───────────────┘ + │ + │ PostgreSQL Logical Replication + │ + ▼ + TCP Tunnel + ngrok + │ + ▼ +┌───────────────────────────────┐ +│ AWS RDS │ +│ │ +│ PostgreSQL │ +│ Database: shared │ +│ │ +│ sync_test │ +│ Subscription: dimos_sub │ +└───────────────────────────────┘ +```` + +--- + +# 2. Local PostgreSQL + +The local PostgreSQL instance is running inside Docker. + +Container: + +```text +dimensio-psql +``` + +Docker port mapping: + +```text +5432/tcp -> 0.0.0.0:5433 +5432/tcp -> [::]:5433 +``` + +Therefore: + +```text +Mac localhost:5433 + ↓ +Docker PostgreSQL:5432 +``` + +The database is: + +```text +dimensionalos +``` + +The local PostgreSQL version is: + +```text +PostgreSQL 17.11 (Debian 17.11-1.pgdg13+2) +on aarch64-unknown-linux-gnu +``` + +Verified with: + +```bash +docker exec dimensio-psql psql -U postgres -d dimensionalos \ + -c "SELECT version();" +``` + +--- + +# 3. Test Data + +A test table named `sync_test` was created in the local PostgreSQL database. + +The table contains: + +```text +id +timestamp +robot_id +value +image +``` + +The table contains artificial test data. + +At the time of the experiment: + +```text +Total rows: 66 +Rows with image: 10 +Rows without image: 56 +``` + +Verified using: + +```sql +SELECT + COUNT(*) AS total, + COUNT(image) AS with_image, + COUNT(*) - COUNT(image) AS without_image +FROM sync_test; +``` + +Result: + +```text + total | with_image | without_image +-------+------------+--------------- + 66 | 10 | 56 +``` + +Example data: + +```text +id | timestamp | robot_id | value +---+------------------------------+-----------+------------------- +1 | 2026-08-14 18:45:08.758181 | robot-001 | 0.541647661655267 +2 | 2026-08-14 18:45:09.768197 | robot-001 | 0.46496265474810117 +3 | 2026-08-14 18:45:10.776587 | robot-001 | 0.16588049960011386 +... +``` + +--- + +# 4. Enabling PostgreSQL Logical Replication + +Initially, the local PostgreSQL instance had: + +```text +wal_level = replica +``` + +Verified using: + +```bash +docker exec dimensio-psql psql -U postgres -d dimensionalos \ + -c "SHOW wal_level;" +``` + +For logical replication, PostgreSQL requires: + +```text +wal_level = logical +``` + +The configuration was changed using: + +```sql +ALTER SYSTEM SET wal_level = logical; +``` + +Then the PostgreSQL container was restarted: + +```bash +docker restart dimensio-psql +``` + +After the restart: + +```bash +docker exec dimensio-psql psql -U postgres -d dimensionalos \ + -c "SHOW wal_level;" +``` + +returned: + +```text + wal_level +----------- + logical +``` + +The existing data was preserved. + +Verification: + +```bash +docker exec dimensio-psql psql -U postgres -d dimensionalos \ + -c "SELECT COUNT(*) FROM sync_test;" +``` + +Result: + +```text + count +------- + 66 +``` + +--- + +# 5. Creating the PostgreSQL Publication + +A PostgreSQL publication was created on the local database: + +```sql +CREATE PUBLICATION dimos_pub +FOR TABLE sync_test; +``` + +The command returned: + +```text +CREATE PUBLICATION +``` + +The publication was verified with: + +```sql +\dRp+ +``` + +Result: + +```text +Publication dimos_pub + +Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root +---------+------------+---------+---------+---------+-----------+--------- +postgres | f | t | t | t | t | f + +Tables: + "public.sync_test" +``` + +Therefore the publication includes: + +```text +sync_test +``` + +and publishes: + +```text +INSERT +UPDATE +DELETE +TRUNCATE +``` + +--- + +# 6. AWS RDS PostgreSQL + +The destination PostgreSQL instance is an AWS RDS database. + +Connection: + +```text +postgresql://guest:@la-psql.cp6sy04ai5l0.us-west-1.rds.amazonaws.com:5432/shared?sslmode=require +``` + +Database: + +```text +shared +``` + +The initial RDS database did not contain the `sync_test` table. + +A matching table was subsequently created: + +```sql +CREATE TABLE sync_test ( + id BIGSERIAL PRIMARY KEY, + timestamp TIMESTAMPTZ NOT NULL DEFAULT NOW(), + robot_id TEXT NOT NULL, + value DOUBLE PRECISION, + image BYTEA +); +``` + +The cloud table was intentionally created with the same schema as the local table. + +--- + +# 7. Enabling Logical Replication on RDS + +Initially, RDS reported: + +```sql +SHOW rds.logical_replication; +``` + +Result: + +```text + rds.logical_replication +------------------------- + off +``` + +Logical replication was subsequently enabled on the RDS instance. + +After enabling the RDS configuration, the expected value is: + +```text + rds.logical_replication +------------------------- + on +``` + +This is required for the RDS instance to participate in logical replication. + +--- + +# 8. Networking Problem + +Native PostgreSQL logical replication requires the subscriber to connect to the publisher. + +The desired direction is: + +```text +Local PostgreSQL + ↓ +AWS RDS PostgreSQL +``` + +In PostgreSQL logical replication terminology: + +```text +Local PostgreSQL = Publisher +AWS RDS = Subscriber +``` + +The problem is that the local PostgreSQL instance is running on a developer laptop: + +```text +localhost:5433 +``` + +AWS RDS cannot directly connect to: + +```text +localhost:5433 +``` + +because `localhost` from AWS refers to the AWS machine itself, not the developer laptop. + +--- + +# 9. ngrok TCP Tunnel + +To make the local PostgreSQL temporarily reachable from AWS, an ngrok TCP tunnel was used. + +ngrok was installed using Homebrew: + +```bash +brew install ngrok +``` + +The local PostgreSQL port is: + +```text +5433 +``` + +The tunnel was started with: + +```bash +ngrok tcp 5433 +``` + +The active tunnel endpoint became: + +```text +0.tcp.us-cal-1.ngrok.io:20984 +``` + +with: + +```text +0.tcp.us-cal-1.ngrok.io:20984 + ↓ +localhost:5433 +``` + +Therefore the networking path becomes: + +```text +AWS RDS + ↓ +0.tcp.us-cal-1.ngrok.io:20984 + ↓ +ngrok + ↓ +Mac localhost:5433 + ↓ +Docker PostgreSQL:5432 +``` + +The ngrok session must remain running for replication to remain connected. + +--- + +# 10. Local PostgreSQL Authentication + +The local PostgreSQL `pg_hba.conf` was inspected: + +```bash +docker exec dimensio-psql cat /var/lib/postgresql/data/pg_hba.conf +``` + +The relevant authentication rule is: + +```text +host all all all scram-sha-256 +``` + +Therefore remote TCP connections using password authentication are allowed. + +No additional `pg_hba.conf` rule was required for this demo. + +--- + +# 11. Testing the ngrok PostgreSQL Connection + +The intended connection test is: + +```bash +psql \ + "host=0.tcp.us-cal-1.ngrok.io \ + port=20984 \ + user=postgres \ + password=postgres \ + dbname=dimensionalos" +``` + +A successful connection should produce: + +```text +dimensionalos=# +``` + +A simple connectivity test can also be performed with: + +```bash +psql \ + "host=0.tcp.us-cal-1.ngrok.io \ + port=20984 \ + user=postgres \ + password=postgres \ + dbname=dimensionalos" \ + -c "SELECT 1;" +``` + +Expected: + +```text + ?column? +---------- + 1 +(1 row) +``` + +--- + +# 12. Attempting to Create the RDS Subscription + +The next step is to create a PostgreSQL subscription on RDS. + +The intended command is: + +```sql +CREATE SUBSCRIPTION dimos_sub +CONNECTION 'host=0.tcp.us-cal-1.ngrok.io port=20984 dbname=dimensionalos user=postgres password=postgres' +PUBLICATION dimos_pub; +``` + +This tells RDS: + +```text +Subscribe to publication: + dimos_pub + +Publisher: + 0.tcp.us-cal-1.ngrok.io:20984 + +Database: + dimensionalos + +User: + postgres +``` + +However, the first attempt failed with: + +```text +ERROR: permission denied to create subscription +DETAIL: Only roles with privileges of the "pg_create_subscription" role may create subscriptions. +``` + +The RDS connection is currently being performed as: + +```text +guest +``` + +Therefore the `guest` role does not currently have the privilege required to create a subscription. + +--- + +# 13. Required RDS Privilege + +The RDS administrator needs to grant: + +```sql +GRANT pg_create_subscription TO guest; +``` + +This must be performed by a sufficiently privileged RDS administrative account. + +After the privilege was granted, reconnected as `guest` and retried: + +```sql +CREATE SUBSCRIPTION dimos_sub +CONNECTION 'host=0.tcp.us-cal-1.ngrok.io port=20984 dbname=dimensionalos user=postgres password=postgres' +PUBLICATION dimos_pub; +``` + +--- + +# 14. Expected Initial Synchronization + +Once the subscription is successfully created, PostgreSQL should be able to perform an initial table synchronization. + +Current state before subscription: + +```text +LOCAL RDS +-------------------------- -------------------------- +sync_test sync_test +66 rows 0 rows +dimos_pub dimos_sub +``` + +After the subscription performs the initial copy: + +```text +LOCAL RDS +-------------------------- -------------------------- +sync_test sync_test +66 rows 66 rows +dimos_pub dimos_sub +``` + +The cloud database can be checked using: + +```sql +SELECT COUNT(*) FROM sync_test; +``` + +Expected and Actual: + +```text + count +------- + 66 +``` + +--- + +# 15. Live INSERT Test + +After the subscription is active, the most important test is to insert data only into the local database. + +Connect to local PostgreSQL: + +```bash +docker exec -it dimensio-psql psql \ + -U postgres \ + -d dimensionalos +``` + +Run: + +```sql +INSERT INTO sync_test (robot_id, value) +VALUES ('PG-REPLICATION-DEMO', 12345.678); +``` + +Then check the RDS database: + +```sql +SELECT id, timestamp, robot_id, value +FROM sync_test +WHERE robot_id = 'PG-REPLICATION-DEMO'; +``` + +Expected and Actual: + +```text +The row appears on RDS without manually executing an INSERT on RDS. +``` + +This demonstrates PostgreSQL native logical replication. + +--- + +# 16. Live UPDATE Test + +On the local PostgreSQL: + +```sql +UPDATE sync_test +SET value = 99999.999 +WHERE robot_id = 'PG-REPLICATION-DEMO'; +``` + +On RDS: + +```sql +SELECT robot_id, value +FROM sync_test +WHERE robot_id = 'PG-REPLICATION-DEMO'; +``` + +Expected and Actual: + +```text +PG-REPLICATION-DEMO | 99999.999 +``` + +--- + +# 17. Live DELETE Test + +On the local PostgreSQL: + +```sql +DELETE FROM sync_test +WHERE robot_id = 'PG-REPLICATION-DEMO'; +``` + +On RDS: + +```sql +SELECT * +FROM sync_test +WHERE robot_id = 'PG-REPLICATION-DEMO'; +``` + +Expected and Actual: + +```text +(0 rows) +``` + +This demonstrates that DELETE operations are also replicated. + +--- + +# 18. What This Demonstrates + +The experiment demonstrates that PostgreSQL can provide native database-to-database replication using: + +```text +WAL + ↓ +Logical decoding + ↓ +Publication + ↓ +Subscription + ↓ +Subscriber PostgreSQL +``` + +The application does not need to: + +* Read every changed row manually +* Build a custom change queue +* Replay application operations +* Implement INSERT/UPDATE/DELETE synchronization logic +* Implement its own database change tracking + +PostgreSQL provides the replication mechanism itself. + +--- + +# 19. Native PostgreSQL Logical Replication Architecture + +Conceptually: + +```text + LOCAL + PostgreSQL Publisher + :5433 + │ + │ + WAL records + │ + ▼ + Logical decoding + │ + ▼ + Publication + "dimos_pub" + │ + │ + │ TCP connection + │ + ▼ + ngrok + │ + ▼ + AWS RDS Subscriber + │ + Subscription + "dimos_sub" + │ + ▼ + sync_test +``` + +The subscription worker on the subscriber connects to the publisher and receives the changes represented by the publication. + +--- + +# 20. Failure Recovery and Offline Synchronization Test + +Tested the behavior of PostgreSQL native logical replication when the publisher becomes temporarily unreachable. + +This was important for the DimensionalOS use case because the publisher represents a robot/edge device that may: + +- Temporarily lose network connectivity +- Continue generating data while offline +- Lose power and restart +- Reconnect to the cloud later + +The test verified whether PostgreSQL could recover automatically and synchronize the changes generated during the outage. + +--- + +## 23.1 Test Architecture + +The tested setup was: + +```text +┌──────────────────────────────┐ +│ Local Mac / Robot │ +│ │ +│ PostgreSQL 17 │ +│ Database: dimensionalos │ +│ │ +│ sync_test │ +│ Publication: dimos_pub │ +└──────────────┬───────────────┘ + │ + │ PostgreSQL + │ Logical Replication + ▼ + ngrok + │ + ▼ +┌──────────────────────────────┐ +│ AWS RDS PostgreSQL │ +│ │ +│ Database: shared │ +│ sync_test │ +│ Subscription: dimos_sub │ +└──────────────────────────────┘ +```` + +The local PostgreSQL acted as the **publisher**, while the AWS RDS PostgreSQL instance acted as the **subscriber**. + +--- + +# 23.2 Baseline Before Failure + +Before starting the failure tests, we verified that logical replication was working correctly. + +The local database contained the test data: + +```text +sync_test rows: 66 +``` + +The same data was present on RDS after the initial synchronization. + +I also verified that: + +```sql +SELECT subname, subenabled +FROM pg_subscription; +``` + +showed the subscription as enabled. + +The publisher had a logical replication slot associated with the subscription. + +At this point the replication path was: + +```text +Local PostgreSQL + │ + ▼ +dimos_pub + │ + ▼ +ngrok + │ + ▼ +dimos_sub + │ + ▼ +AWS RDS +``` + +I also verified normal live replication by inserting, updating, and deleting records locally and observing the changes automatically appear on RDS. + +--- + +# 23.3 Network Failure Test + +I then simulated a network outage by stopping the ngrok TCP tunnel while leaving the local PostgreSQL instance running. + +Before stopping the tunnel, replication was healthy. + +The network path became: + +```text +Local PostgreSQL + │ + X + │ + ngrok + X + │ + AWS RDS +``` + +The local PostgreSQL itself remained fully operational. + +--- + +## 23.3.1 Writes During Network Outage + +While the network connection was unavailable, I continued writing data to the local PostgreSQL database. + +For example: + +```sql +INSERT INTO sync_test (robot_id, value) +VALUES + ('OFFLINE-1', 111), + ('OFFLINE-2', 222), + ('OFFLINE-3', 333), + ('OFFLINE-4', 444); +``` + +The records were immediately available on the local database. + +I verified: + +```sql +SELECT robot_id, value +FROM sync_test +WHERE robot_id LIKE 'OFFLINE-%' +ORDER BY id; +``` + +The local database contained all four records. + +``` + robot_id | value +-----------+------- + OFFLINE-1 | 111 + OFFLINE-2 | 222 + OFFLINE-3 | 333 + OFFLINE-4 | 444 +``` + +--- + +## 23.3.2 Cloud Behavior During Network Outage + +I queried the RDS database while the network connection was still unavailable: + +```sql +SELECT robot_id, value +FROM sync_test +WHERE robot_id LIKE 'OFFLINE-%' +ORDER BY id; +``` + +The new records were not present on RDS. + +``` + robot_id | value +----------+------- +(0 rows) +``` + +This was expected because the subscriber could not communicate with the publisher. + +Importantly, the records were **not lost from the publisher**. + +They remained committed in the local PostgreSQL database. + +--- + +# 23.4 Replication Slot Behavior + +I inspected the logical replication slot on the local PostgreSQL: + +```sql +SELECT + slot_name, + slot_type, + active, + restart_lsn, + confirmed_flush_lsn +FROM pg_replication_slots; +``` + +``` +slot_name | slot_type | active | restart_lsn | confirmed_flush_lsn +-----------+-----------+--------+-------------+--------------------- + dimos_sub | logical | f | 0/1DCC710 | 0/1DCC748 +(1 row) +``` + +The logical replication slot remained associated with the subscription. + +I also checked the amount of WAL retained by the slot: + +```sql +SELECT + slot_name, + active, + pg_size_pretty( + pg_wal_lsn_diff( + pg_current_wal_lsn(), + restart_lsn + ) + ) AS retained_wal +FROM pg_replication_slots; +``` + +``` + slot_name | active | retained_wal +-----------+--------+-------------- + dimos_sub | f | 7504 bytes +(1 row) +``` + +This demonstrated an important property of PostgreSQL logical replication: + +> The publisher retains WAL required by the replication slot until the subscriber has consumed the changes. + +Therefore, when the subscriber is temporarily disconnected, PostgreSQL does not simply discard the changes that still need to be replicated. + +Conceptually: + +```text +Network outage + │ + ▼ +Subscriber cannot consume WAL + │ + ▼ +Replication slot retains required WAL + │ + ▼ +Network restored + │ + ▼ +Subscriber catches up +``` + +--- + +# 23.5 Network Recovery + +I restarted the ngrok TCP tunnel after the simulated outage. + +Because ngrok's free TCP endpoint can change between sessions, the endpoint changed when the tunnel was restarted. + +I updated the subscription connection string to point to the new ngrok endpoint. + +After connectivity was restored, the PostgreSQL subscriber reconnected to the publisher. + +The previously generated offline records were then replicated to RDS. + +I verified: + +```sql +SELECT robot_id, value +FROM sync_test +WHERE robot_id LIKE 'OFFLINE-%' +ORDER BY id; +``` + +The RDS database eventually contained: + +```text +OFFLINE-1 | 111 +OFFLINE-2 | 222 +OFFLINE-3 | 333 +OFFLINE-4 | 444 +``` + +This confirmed that PostgreSQL successfully caught up with the changes generated while the network was unavailable. + +--- + +# 23.6 Result of Network Failure Test + +The observed behavior was: + +```text + Network Available + +Local PostgreSQL + │ + ├── INSERT + │ + ▼ + Publication + │ + ▼ + RDS +``` + +When the network was interrupted: + +```text + Network Unavailable + +Local PostgreSQL + │ + ├── INSERT + ├── INSERT + ├── INSERT + │ + ▼ + Local WAL + │ + │ + X────────────── RDS +``` + +After network connectivity was restored: + +```text +Local PostgreSQL + │ + │ previously unconsumed changes + ▼ + Replication + │ + ▼ + RDS + │ + ▼ + Catches up +``` + +### Result + +**PostgreSQL native logical replication successfully recovered from the temporary network outage and synchronized the changes generated during the outage.** + +--- + +# 23.7 Robot / PostgreSQL Shutdown Test + +I also tested the behavior when the local PostgreSQL instance itself was stopped. + +This simulated a robot losing power. + +The local PostgreSQL container was stopped using: + +```bash +docker stop dimensio-psql +``` + +At this point: + +```text +Robot / Publisher + │ + X + │ + OFFLINE +``` + +The RDS subscriber remained configured with the existing subscription. + +During this period, the publisher was unavailable. + +--- + +# 23.8 PostgreSQL Restart + +The local PostgreSQL instance was started again: + +```bash +docker start dimensio-psql +``` + +I verified that PostgreSQL came back successfully: + +```bash +docker exec dimensio-psql psql \ + -U postgres \ + -d dimensionalos \ + -c "SELECT version();" +``` + +I also verified that logical replication was still configured: + +```bash +docker exec dimensio-psql psql \ + -U postgres \ + -d dimensionalos \ + -c "SHOW wal_level;" +``` + +The result remained: + +```text +wal_level +----------- +logical +``` + +The publication and replication configuration were preserved across the restart. + +--- + +# 23.9 Replication Recovery After PostgreSQL Restart + +After restoring network connectivity, the subscription was able to reconnect to the publisher. + +I verified that the local and RDS databases eventually converged. + +Local: + +```sql +SELECT COUNT(*) FROM sync_test; +``` + +RDS: + +```sql +SELECT COUNT(*) FROM sync_test; +``` + +The counts matched again. + +I then inserted a new record locally: + +```sql +INSERT INTO sync_test (robot_id, value) +VALUES ('AFTER-RESTART', 999); +``` + +The record subsequently appeared on RDS without manually inserting it there. + +This confirmed that logical replication continued to function after the publisher restart. + +--- + +# 23.10 Observed Failure Recovery Behavior + +The tests demonstrated the following behavior: + +| Scenario | Local PostgreSQL | RDS | Result | +| -------------------- | ---------------- | ------------ | ------------------------------------------ | +| Normal operation | Running | Connected | Changes replicated immediately | +| Network unavailable | Running | Disconnected | Local writes continue | +| Network unavailable | Running | Disconnected | New changes not immediately visible on RDS | +| Network restored | Running | Reconnected | RDS catches up | +| PostgreSQL stopped | Offline | Running | Publisher unavailable | +| PostgreSQL restarted | Running | Reconnects | Replication resumes | +| Post-restart writes | Running | Connected | Changes replicate normally | + +--- + +# 23.11 Important Observation — Replication Is Not an Infinite Offline Queue + +The test also highlighted an important limitation of native PostgreSQL logical replication. + +While the subscriber is disconnected, the publisher's logical replication slot retains the WAL required by the subscriber. + +Therefore: + +```text +Longer outage + │ + ▼ +More local writes + │ + ▼ +More WAL retained + │ + ▼ +More local disk consumption +``` + +I monitored this using: + +```sql +SELECT + slot_name, + active, + pg_size_pretty( + pg_wal_lsn_diff( + pg_current_wal_lsn(), + restart_lsn + ) + ) AS retained_wal +FROM pg_replication_slots; +``` + +This means native PostgreSQL replication can tolerate temporary outages, but it should not be treated as an unlimited offline queue. + +If a robot remains offline for a sufficiently long period and continues generating a high volume of data, retained WAL can grow substantially and eventually create disk-pressure problems. + +--- + +# 23.12 Overall Result + +The failure tests established that PostgreSQL native logical replication provides **automatic recovery from temporary connectivity and publisher failures**. + +The tested behavior was: + +```text + Normal + │ + ▼ + Replication Active + │ + │ + Network Failure + │ + ▼ + Local writes continue + │ + ▼ + WAL retained locally + │ + │ + Network / Robot returns + │ + ▼ + Replication reconnects + │ + ▼ + Subscriber catches up + │ + ▼ + Databases converge +``` + +Therefore, for a PostgreSQL-to-PostgreSQL use case, native logical replication already provides a significant amount of the functionality that a custom synchronization mechanism would otherwise need to implement. + +---