With pandas 2.0, custom time handling might not be necessary any more

[mikapfl]

Describe the bug

Most of the time handling functionality of scmdata is a duplication of already existing functionality in pandas. Given that pandas is a much more mature project, the scmdata functionality is also severely lacking compared to pandas.

An example is the resampling functionality of pandas. The code for resampling is super concise, and inter- and extrapolation is available. The equivalent of pandas'

ts.resample("5Min").sum()

in scmdata will be much more involved, simply because run.interpolate takes the target_times as a list of time values.

As a result, I found it much easier to convert ScmRun objects into pandas DataFrames with proper pd.DateTimeIndex'es when working with even the simplest time-related problems. The question is then: why is there a proprietary re-implementation of pandas' functionality in scmdata at all? Wouldn't it be better to get rid of it and leverage pandas' functionality?

[znicholls]

I think the major issue was that pandas date time length was limited to ~1760 to 2250 or something and we needed to be able to handle date times outside these dates. Now that pandas supports microsecond precision (I think), we could revisit those restrictions. We have tests for handling datetimes post 2250 and pre 1760 so if changing the back-end leaves the tests passing then I think we're good.

[mikapfl]

I looked again and as far as I can tell, pandas still just breaks on dates outside:

In [12]: pd.Timestamp.min
Out[12]: Timestamp('1677-09-21 00:12:43.145224193')

In [13]: pd.Timestamp.max
Out[13]: Timestamp('2262-04-11 23:47:16.854775807')

There is a workaround using periods (e.g. pd.period_range) instead, but it looses functionality, and also doesn't work for dates before 1 AD.

Now, I also remember that this was one reason to prefer xarray over pandas when we decided for primap2. Okay, not much to do, I guess any library building on pandas which needs long-range dates will need to roll their own date handling.

[znicholls]

Hmm sad I thought there was some pandas update but looks like https://pandas.pydata.org/docs/whatsnew/v2.0.0.html#construction-with-datetime64-or-timedelta64-dtype-with-unsupported-resolution doesn't do the job. I thought we tried to use as much of the xarray cftime handling as possible, we probably need to revisit that...

…

On Tue, Jun 20, 2023 at 5:59 PM Mika Pflüger ***@***.***> wrote: Closed #242 <#242> as completed. — Reply to this email directly, view it on GitHub <#242 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFUH5G2EAHVP4XIIV66XF73XMFJ7LANCNFSM6AAAAAAZMHAYMU> . You are receiving this because you commented.Message ID: ***@***.***>

[mikapfl]

Ah, I hadn't seen the new functionality from April. Yes, I think with pandas 2.0, e.g. date_range learned the necessary tricks:

In [8]: pd.date_range(start="-1000-01-01", end="3000-01-01", freq="AS", unit="s")
Out[8]: 
DatetimeIndex(['-1000-01-01',  '-999-01-01',  '-998-01-01',  '-997-01-01',
                '-996-01-01',  '-995-01-01',  '-994-01-01',  '-993-01-01',
                '-992-01-01',  '-991-01-01',
               ...
                '2991-01-01',  '2992-01-01',  '2993-01-01',  '2994-01-01',
                '2995-01-01',  '2996-01-01',  '2997-01-01',  '2998-01-01',
                '2999-01-01',  '3000-01-01'],
              dtype='datetime64[s]', length=4001, freq='AS-JAN')

This can be used as a full-featured index. So, I think this could be used in scmdata, reducing the amount of time-related code considerably.

[znicholls]

Nice find

…

On Wed, 21 Jun 2023 at 9:23 pm, Mika Pflüger ***@***.***> wrote: Ah, I hadn't seen the new functionality from April. Yes, I think with pandas 2.0, e.g. date_range learned the necessary tricks: In [8]: pd.date_range(start="-1000-01-01", end="3000-01-01", freq="AS", unit="s")Out[8]: DatetimeIndex(['-1000-01-01', '-999-01-01', '-998-01-01', '-997-01-01', '-996-01-01', '-995-01-01', '-994-01-01', '-993-01-01', '-992-01-01', '-991-01-01', ... '2991-01-01', '2992-01-01', '2993-01-01', '2994-01-01', '2995-01-01', '2996-01-01', '2997-01-01', '2998-01-01', '2999-01-01', '3000-01-01'], dtype='datetime64[s]', length=4001, freq='AS-JAN') This can be used as a full-featured index. So, I think this could be used in scmdata, reducing the amount of time-related code considerably. — Reply to this email directly, view it on GitHub <#242 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFUH5GYPWMV3ONAVLXPOWSLXMLKRXANCNFSM6AAAAAAZMHAYMU> . You are receiving this because you commented.Message ID: ***@***.***>

[znicholls]

Any idea if it works with stuff that isn’t a range (eg some historical data and then only 2100, 200, 2300, 2400)? On Wed, 21 Jun 2023 at 10:18 pm, Zebedee Nicholls < ***@***.***> wrote:

…

Nice find On Wed, 21 Jun 2023 at 9:23 pm, Mika Pflüger ***@***.***> wrote: > Ah, I hadn't seen the new functionality from April. Yes, I think with > pandas 2.0, e.g. date_range learned the necessary tricks: > > In [8]: pd.date_range(start="-1000-01-01", end="3000-01-01", freq="AS", unit="s")Out[8]: DatetimeIndex(['-1000-01-01', '-999-01-01', '-998-01-01', '-997-01-01', > '-996-01-01', '-995-01-01', '-994-01-01', '-993-01-01', > '-992-01-01', '-991-01-01', > ... > '2991-01-01', '2992-01-01', '2993-01-01', '2994-01-01', > '2995-01-01', '2996-01-01', '2997-01-01', '2998-01-01', > '2999-01-01', '3000-01-01'], > dtype='datetime64[s]', length=4001, freq='AS-JAN') > > This can be used as a full-featured index. So, I think this could be used > in scmdata, reducing the amount of time-related code considerably. > > — > Reply to this email directly, view it on GitHub > <#242 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AFUH5GYPWMV3ONAVLXPOWSLXMLKRXANCNFSM6AAAAAAZMHAYMU> > . > You are receiving this because you commented.Message ID: > ***@***.***> >

[mikapfl]

Yes, DatetimeIndex supports any collection of Timestamps:

In [11]: (pd.date_range(start="1700-01-01", end="2021-01-01", freq="AS", unit="s")
    ...: .union(pd.date_range(start="2100-01-01", end="2400-01-01", periods=4, unit="s")))
Out[11]: 
DatetimeIndex(['1700-01-01', '1701-01-01', '1702-01-01', '1703-01-01',
               '1704-01-01', '1705-01-01', '1706-01-01', '1707-01-01',
               '1708-01-01', '1709-01-01',
               ...
               '2016-01-01', '2017-01-01', '2018-01-01', '2019-01-01',
               '2020-01-01', '2021-01-01', '2100-01-01', '2200-01-01',
               '2300-01-01', '2400-01-01'],
              dtype='datetime64[s]', length=326, freq=None)

Of course, you then loose frequency information on the whole thing because it doesn't have a frequency any more.

[znicholls]

Nice

…

On Thu, 22 Jun 2023 at 6:29 pm, Mika Pflüger ***@***.***> wrote: Yes, DatetimeIndex supports any collection of Timestamps: In [11]: (pd.date_range(start="1700-01-01", end="2021-01-01", freq="AS", unit="s") ...: .union(pd.date_range(start="2100-01-01", end="2400-01-01", periods=4, unit="s")))Out[11]: DatetimeIndex(['1700-01-01', '1701-01-01', '1702-01-01', '1703-01-01', '1704-01-01', '1705-01-01', '1706-01-01', '1707-01-01', '1708-01-01', '1709-01-01', ... '2016-01-01', '2017-01-01', '2018-01-01', '2019-01-01', '2020-01-01', '2021-01-01', '2100-01-01', '2200-01-01', '2300-01-01', '2400-01-01'], dtype='datetime64[s]', length=326, freq=None) Of course, you then loose frequency information on the whole thing because it doesn't have a frequency any more. — Reply to this email directly, view it on GitHub <#242 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFUH5G5FMUZ7WVZWXRTISWLXMP65LANCNFSM6AAAAAAZMHAYMU> . You are receiving this because you commented.Message ID: ***@***.***>

[lewisjared]

Another reason that fed into why we rolled our own was to be explicit about piecewise-linear (concentrations, ERF) and piecewise-constant (emissions) timeseries and their subtlely differences. We never really achieved that goal, but have come back to discussing it every year or so.

While ScmRun.resample is handy to interpolate missing years, it doesn't support extrapolation directly.

[mikapfl]

I think the difference between piecewise-linear and piecewise-constant boils down to extensive and intensive quantities, right? Any extensive quantity is piecewise constant and any intensive quantity is piecewise linear. If you resample, you have to take sums or do divisions for extensive quantities, and you calculate means or interpolate linearly for intensive quantities. In pandas, that is generalized with the concept of Resampler objects (akin to groupby), the resampling just gives you a resampler object, and you then call sum() or interpolate() on it.

I think for all kind of higher-level operations, it would be great if we could specify for a timeseries if it is extensive or intensive and then do the right thing automatically.

[znicholls]

I think we're on the same page but just to check:

• piecewise-constant (like emissions in most cases but not always, often ERF)
  • interpolation: simple within a time step (the whole thing is just constant) but trickier when it is across multiple time steps. In those cases, my would be: 1) integrate 2) do difference over each time step of interest in integrated space 3) go back to original units by dividing the cumulative value over each time step by the length of the time step (there are probably more efficient algorithms though)
  • extrapolation: trickier because you need to decide whether it makes more sense to be constant (following the fact that each step is piecewise-constant) or linearly extrapolated based on more than one time step (in which case, how many time steps...)
• piecewise-linear (like concentrations in almost all cases, sometimes ERF)
  • interpolation: linear interpolation makes sense in most cases. The tricky part is whether you need the interpolation to be integral preserving or not. If it does need to preserve the integral, then it is hard because the outputs need to have the same integral but it isn't clear to me whether that is possible if the outputs are also piecewise-linear or whether that only works if the interpolated values 'flip' and become piecewise-constant or something else
  • extrapolation: easier I think because you can use linear extrapolation just based on the last n points and something basically sensible will 'just happen' (maybe, although I'm sure there are edge cases that could be dreamt up)

Probably the simplest way to get on the same page would be to write some concrete tests..?

I think for all kind of higher-level operations, it would be great if we could specify for a timeseries if it is extensive or intensive and then do the right thing automatically

100%. If we can get users to tell us how to interpret between the discrete points then we basically know the entire continuous timeseries which makes it way easier to work with the data. We once tried to do this but then got caught in a rabbit hole and abandoned ship. Would be great one to revive one day.

[mikapfl]

Hi,

I think I don't really understand the problem space yet:

• extensive/intensive and piecewise-constant/linear are not the same concepts. Cumulative emissions, e.g. are a clearly extensive category with respect to time, while in practice we often talk about an emissions rate (emissions per year), which is an intensive quantity with respect to time.
• I don't understand the meaning piecewise-constant quantities. Why are emissions piecewise-constant? I expect if I have to downscale, i.e. interpolate emissions e.g. from yearly resolution to monthly, I have to supply weights because of seasons. And if I have to interpolate a single year, then linear interpolation between the two adjacent years is surely better than keeping it constant, absent any better weights?

Cheers,

Mika