Implement insert_or_replace

[demianmnave]

This update provides an unsophisticated implementation of the SQLite INSERT OR REPLACE statement.

[liuzicheng1987]

Hi @demianmnave , this is pretty cool! I'd like to find a solution for PostgreSQL and MySQL as well. I'd be happy to contribute tot this PR.

[liuzicheng1987]

@demianmnave , two things:

1. Could you remove the or_replace from write? Because we are using the COPY operator on postgres, it would be difficult to make this work there.
2. For the "upsert" to work, there needs to be a check that we have a primary key, which should be implemented at compile time. I think, I could do this.

[demianmnave]

Hi @liuzicheng1987,

1. Could you remove the or_replace from write? Because we are using the COPY operator on postgres, it would be difficult to make this work there.

It should already be removed in the second commit, d21b9d9.

2. For the "upsert" to work, there needs to be a check that we have a primary key, which should be implemented at compile time. I think, I could do this.

Yes, totally missed it. I can look into it unless you get to it first.

... I'd like to find a solution for PostgreSQL and MySQL as well. I'd be happy to contribute tot this PR.

Looking deeper into the docs, REPLACE has consistent behavior between SQLite and MySQL (delete the conflict and insert a new row), and UPSERT has consistent behavior between SQLite and PostreSQL (update the conflicting row). There does not seem to be a single form of conflict resolution that is consistent across all three.

The ramifications of implementing insert_or_replace in terms of one or the other form aren't obvious to me. Maybe it's a distinction without a difference?

[liuzicheng1987]

As far as the primary key is concerned, I will take care of it, the push will come tonight.

I think as long as the outcome is the same, it doesn't matter too much to me whether the row is replaced or updated. I think we will just have to live with the fact that these kind of implementation details will differ from database to database.

[liuzicheng1987]

@demianmnave

[demianmnave]

@liuzicheng1987 Thanks for looking into the primary key fix. If you would like to bring this PR home, I can look into MySQL and PostgreSQL separately.

Also, I can update the docs to make it clear that a primary key or unique index is needed for insert_or_replace to work properly.

[liuzicheng1987]

@demianmnave , here are compile time checks for primary keys.

[liuzicheng1987]

@demianmnave , the way I would approach this for Postgres is the following:

https://www.sparkcodehub.com/sql/data-manipulation/on-conflict-clause

INSERT INTO customers (customer_id, first_name, email)
VALUES (101, 'John', 'john.doe@example.com')
ON CONFLICT (customer_id) DO UPDATE 
SET 
    first_name = EXCLUDED.first_name,
    email = EXCLUDED.email;

It means we would have to add an additional field, non_primary_keys, to dynamic::Insert and then update all non-primary keys.

For MySQL, the idea is the same, just a bit of a different syntax.

https://www.geeksforgeeks.org/mysql/upsert-in-mysql/

INSERT INTO table_name (column1, column2, ...)
VALUES (value1, value2, ...)
ON DUPLICATE KEY UPDATE
  column1 = VALUES(column1),
  column2 = VALUES(column2),
  ...;

[liuzicheng1987]

@demianmnave for your convenience, I have added the field non_primary_keys to dynamic::Insert and made sure it is being filled. The remainder should be fairly simple - just generate the code as discussed above and write a bunch of tests.

I will leave the rest to you.

Again - thank you for this contribution. This is very cool and I look forward to using it.

[demianmnave]

@liuzicheng1987 Great, thanks for pointing me in the right direction.

[demianmnave]

@liuzicheng1987 I believe the is ready to go. The latest commits implement insert_or_update() for Postgres and MySQL. I also reimplemented the SQLite insert_or_replace() in terms of "ON CONFLICT() DO UPDATE" to test the syntax and required code changes without having to stand up a Postgres server.

I ran the full battery of unit tests for MSVC 17.14.12 under Windows 11 against the following database versions:

• SQLite 3.49.1
• PostgreSQL 16.10 (libpq 16.4)
• MySQL 8.4.6 (libmariadb 3.4.5)

All unit tests passed except for postgres.test_timestamp_with_tz, which fails with an exception:

unknown file: error: C++ exception with description "String '2000-01-01 01:00:00+0000' did not match format '%Y-%m-%d %H:%M:%S%z'." thrown in the test body.

Let me know if you spot anything amiss and I will get it corrected ASAP.

[liuzicheng1987]

@demianmnave , all right, I have merged this. Thank you so much for the contribution!

[demianmnave]

@liuzicheng1987 Thank you, and apologies for the unnecessary busy work. The one change that may need some investigation is the removal of explicitly setting CMAKE_MSVC_RUNTIME_LIBRARY. Without this, sqlgen fails to build when using the vcpkg x64-windows-static triplet due to conflicting runtime libraries.

[liuzicheng1987]

@demianmnave I had the opposite experience: The build pipelines only worked after I removed it.

[liuzicheng1987]

@demianmnave but feel free to open a separate PR. If we can find a way that works for you and the build pipelines, that would be the best solution.

[demianmnave]

@liuzicheng1987 Will do.