When serialize FixedString and string is bigger than size defined in Table definition #2223

Activity

1. mzitnik

   added

   client-api-v2

   area:data-typedata type processing issuesdata type processing issues

   v2-feedback

   on Mar 14, 2025
2. laeg

   modified the milestone: 0.9.2 on Jul 23, 2025
3. chernser

   added this to the 0.9.2 milestone on Aug 6, 2025
4. chernser commented on Aug 12, 2025

   

   chernser

   on Aug 12, 2025

   Contributor

   More actions

   @mzitnik
   While working on this issue I've got next thoughts:

   • the flag will affect whole client. So it would be problematic to tolerate bigger strings from subset of columns
   • what if just have a method that truncates string to defined length?
   • if we truncate string at utf8 symbol position it will result string with completely different meaning.

   Here is the test for the last point:

       @Test
       public void testUtfStringTruncate() {
           final String str = "a☺c";
           byte[] bytes = str.getBytes(StandardCharsets.UTF_8);
           System.out.println("bytes.length: " + bytes.length);
           for (int i = 0; i < bytes.length; i++) {
               System.out.println(String.format("bytes[%d]: %x", i, bytes[i]));
           }
   
           byte[] twoBytes= new byte[] {bytes[0], bytes[1]};
           String twoByteString = new String(twoBytes, StandardCharsets.UTF_8);
           System.out.println("twoByteString: " + twoByteString);
           System.out.println("twoByteString.length: " + twoByteString.length());
       }

   Output:

   bytes.length: 5
   bytes[0]: 61
   bytes[1]: e2
   bytes[2]: 98
   bytes[3]: ba
   bytes[4]: 63
   twoByteString: a�
   twoByteString.length: 2
   

   The correct string should be a string with only a because e2 is part of ☺. In this case 3 bytes used for a character.
5. laeg

   modified the milestones: 0.9.2, 0.9.3 on Aug 14, 2025
6. chernser

   modified the milestones: 0.9.3, 0.9.4 on Oct 1, 2025
7. chernser

   removed this from the 0.9.5 milestone on Nov 17, 2025
8. chernser

   added this to the 0.9.10 milestone on Apr 15, 2026
9. chernser commented on Jul 9, 2026

   

   chernser

   on Jul 9, 2026

   Contributor

   More actions

   /triage
10. github-actions commented on Jul 9, 2026

    

    github-actionsBot

    on Jul 9, 2026 – with GitHub Actions

    More actions

    Triage Report

    Effort to Fix: Small
    Type: Feature Request
    Affected Module: client-api-v2

    Summary

    When a Java string is serialized into a ClickHouse FixedString(N) column and its UTF-8 byte length exceeds N, the current code in BinaryStreamUtils.writeFixedString throws an exception via ClickHouseChecker.notLongerThan. The issue requests a configurable option (a flag or setting) to allow silent truncation to N bytes instead of throwing, with the important caveat that truncation must occur at a valid UTF-8 character boundary to avoid producing corrupted strings.

    Recommended Labels & Priority

    • Labels: client-api-v2, v2-feedback, area:data-type

    Missing Information / Questions for User

    1. Should the truncation option be a global client setting (e.g., ClientConfigProperties) or a per-column/per-insert setting? A global flag makes it hard to tolerate oversized strings only for a subset of columns.
    2. What is the preferred truncation semantics: truncate to at most N bytes at the last valid UTF-8 character boundary (safe, may lose more than one byte), or is padding with replacement characters acceptable?
    3. Which client version is being used (client-v2 is assumed from existing labels)?

    Tests to Add

    Test 1: FixedString insertion — string exactly at byte limit

    // Insert a FixedString(4) column with a 4-byte ASCII string — should succeed
    client.insert("test_table", List.of(Map.of("col", "abcd")));
    // assert row reads back as "abcd"

    Test 2: FixedString insertion — string exceeds limit, THROW mode (current default)

    // Insert a FixedString(4) column with a 5-byte ASCII string — should throw
    assertThrows(ClientException.class, () ->
        client.insert("test_table", List.of(Map.of("col", "abcde"))));

    Test 3: FixedString insertion — UTF-8 multibyte string exceeds limit, TRUNCATE mode

    // "a☺c" is 5 bytes in UTF-8; FixedString(2) should truncate safely to "a" (1 byte),
    // not to "a\xe2" which is an invalid UTF-8 sequence.
    // (Enable truncation mode on the client/settings before inserting.)
    client.insert("test_table", List.of(Map.of("col", "a☺c")));
    String result = // read back the value
    assertTrue(result.equals("a") || result.isEmpty()); // no garbled bytes

    Test 4: FixedString insertion — ASCII string exceeds limit, TRUNCATE mode

    // FixedString(3) column with "abcde" — should truncate to "abc" without exception
    // (Enable truncation mode on the client/settings before inserting.)
    client.insert("test_table", List.of(Map.of("col", "abcde")));
    String result = // read back the value
    assertEquals("abc", result);
11. chernser

    modified the milestones: 0.11.0, 0.12.0 on Sep 23, 2026
12. chernser

    added a parent issue on Oct 14, 2026

    • [client-v2] Issues with Data types #3205