Make the ragged check in DynamicTable.add_row incremental

[h-mayorquin]

Motivation

In the review of #1066 @rly pointed out that the ragged check iterates over every element and that he did not see a way around it, which is why check_ragged was added as an opt-out. This PR is an answer to that question, and to the first half of #1559. The idea is to test raggedness on-line, comparing the new element against the ones already added instead of rescanning the column. This means only asking whether the new element is the one that makes the column ragged. Once a column is marked as ragged the warning is emitted once, as that can only happen once, and the check is skipped for that column from then on. The column is rescanned once if it grew by something other than that single append, so a row added with check_ragged=False is still counted.

With this improvement, the benchmark for filling a one-column table with check_ragged at its default of True, so this is what every caller of add_row pays:

rows	6.1.0 (dev)	this PR	speedup
1000	0.109s	0.022s	5x
2000	0.380s	0.038s	10x
4000	1.452s	0.068s	21x
8000	5.647s	0.132s	43x
16000	22.317s	0.295s	76x

The speedup doubles every time the table doubles, as the check went from linear per row to constant.

The biggest change is that we do not emit the warning on every row any more, which was redundant, but I think that is fine. The message is identical every time and a ragged column never becomes unragged, and the default warning filter was already collapsing the repeats by code location. What do you think? Is there something I am missing?

The enforce_unique_id half of the issue is a separate change with no behavior attached to it, so I am sending it as its own PR.

How to test the behavior?

The timing, which is the script from #1559 with check_ragged left at its default:

import time

from hdmf.common import DynamicTable


def fill(num_rows):
    table = DynamicTable(name="bench", description="")
    table.add_column(name="a", description="")
    start = time.perf_counter()
    for _ in range(num_rows):
        table.add_row(a=1.0)
    return time.perf_counter() - start


for num_rows in (1_000, 2_000, 4_000, 8_000, 16_000):
    print(f"{num_rows:6d}  {fill(num_rows):7.3f}s")

And the warning, where the third row warns on dev and is silent here:

from hdmf.common import DynamicTable

table = DynamicTable(name="table", description="")
table.add_column(name="qux", description="")
table.add_row(qux=[1])
table.add_row(qux=[1, 2])  # warns, this is the row that makes the column ragged
table.add_row(qux=[3, 4])  # warned on dev too, silent now

Checklist

• Did you update CHANGELOG.md with your changes?
• Does the PR clearly describe the problem and the solution?
• Have you reviewed our Contributing Guide?
• Does the PR use "Fix #XXX" notation to tell GitHub to close the relevant issue numbered XXX when the PR is merged?

On the last one, the PR references #1559 but on purpose without the Fix keyword, as the issue covers two checks and this PR fixes one of them. It should stay open until the enforce_unique_id PR lands.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.16%. Comparing base (dd22f16) to head (092fa9e).

Additional details and impacted files

@@           Coverage Diff           @@
##              dev    #1561   +/-   ##
=======================================
  Coverage   93.15%   93.16%           
=======================================
  Files          41       41           
  Lines       10245    10259   +14     
  Branches     2119     2122    +3     
=======================================
+ Hits         9544     9558   +14     
  Misses        422      422           
  Partials      279      279           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[rly]

This looks great. I made a couple minor updates to docs. Thank you @h-mayorquin !