numpy operations with multiple arrays

[bournejt]

Dear community, I'm having a difficult time to perform numpy operations with 2 csp.ts[np.ndarray] involved. I tried 2 versions of clip function. I set dn_limit=csp.stats.quant(x, quant=0.01) to use the 1% quantile of the past data to winsorize. For either version, the error is shown below. What's the recommended way to do such operation?

Cell In[76], line 5, in clip.<locals>.<lambda>(u)
      1 @csp.graph
      2 def clip(x: csp.ts[Union[float, np.ndarray]],
      3          up_limit: Union[float, csp.ts[np.ndarray], Any] = None,
      4          dn_limit: Union[float, csp.ts[np.ndarray], Any] = None) -> csp.ts[np.ndarray]:
----> 5     return csp.apply(x, lambda u: np.minimum(np.maximum(u, dn_limit), up_limit), np.ndarray)

TypeError: issubclass() arg 1 must be a class

@csp.node
def clip(x: csp.ts[Union[float, np.ndarray]],
             up_limit: Union[float, csp.ts[np.ndarray], Any] = None,
             dn_limit: Union[float, csp.ts[np.ndarray], Any] = None) -> csp.ts[np.ndarray]:

    return np.clip(x, dn_limit, up_limit)

@csp.graph
def clip(x: csp.ts[Union[float, np.ndarray]],
             up_limit: Union[float, csp.ts[np.ndarray], Any] = None,
             dn_limit: Union[float, csp.ts[np.ndarray], Any] = None) -> csp.ts[np.ndarray]:

    return csp.apply(x, lambda u: np.minimum(np.maximum(u, dn_limit), up_limit), np.ndarray)

[AdamGlustein]

I just copy-pasted your code and both solutions seem to work for me fine on 3.11.11. I am on csp 0.8.0 by the way

import csp
from typing import Union, Any
import numpy as np

from datetime import datetime, timedelta

# 1)
@csp.node
def clip(x: csp.ts[Union[float, np.ndarray]],
             up_limit: Union[float, csp.ts[np.ndarray], Any] = None,
             dn_limit: Union[float, csp.ts[np.ndarray], Any] = None) -> csp.ts[np.ndarray]:

    return np.clip(x, dn_limit, up_limit)

# 2)
@csp.graph
def clip(x: csp.ts[Union[float, np.ndarray]],
             up_limit: Union[float, csp.ts[np.ndarray], Any] = None,
             dn_limit: Union[float, csp.ts[np.ndarray], Any] = None) -> csp.ts[np.ndarray]:

    return csp.apply(x, lambda u: np.minimum(np.maximum(u, dn_limit), up_limit), np.ndarray)

@csp.graph
def g() -> csp.ts[np.ndarray]:
    x = csp.curve(np.ndarray, [(datetime(2020,1,1) + timedelta(seconds=i), np.array([float(i), float(i)])) for i in range(10)])
    return clip(x, up_limit=7.0, dn_limit=2.0)

res = csp.run(g, starttime=datetime(2020,1,1), endtime=timedelta(seconds=10))
print(res)

gives:

{0: [(datetime.datetime(2020, 1, 1, 0, 0), array([2., 2.])), (datetime.datetime(2020, 1, 1, 0, 0, 1), array([2., 2.])), (datetime.datetime(2020, 1, 1, 0, 0, 2), array([2., 2.])), (datetime.datetime(2020, 1, 1, 0, 0, 3), array([3., 3.])), (datetime.datetime(2020, 1, 1, 0, 0, 4), array([4., 4.])), (datetime.datetime(2020, 1, 1, 0, 0, 5), array([5., 5.])), (datetime.datetime(2020, 1, 1, 0, 0, 6), array([6., 6.])), (datetime.datetime(2020, 1, 1, 0, 0, 7), array([7., 7.])), (datetime.datetime(2020, 1, 1, 0, 0, 8), array([7., 7.])), (datetime.datetime(2020, 1, 1, 0, 0, 9), array([7., 7.]))]}

[bournejt]

Hi Adam. I updated to your version of csp package. The error I reported above happens when a csp ts object is passed to up_limit or dn_limit. Your example passed in float number. That works on my side, too. The following updated version would be able to replicate the error I saw for either clip_v1 or clip_v2.

# 1)
@csp.node
def clip_v1(x: csp.ts[Union[float, np.ndarray]],
             up_limit: Union[float, csp.ts[np.ndarray], Any] = None,
             dn_limit: Union[float, csp.ts[np.ndarray], Any] = None) -> csp.ts[np.ndarray]:

    return np.clip(x, dn_limit, up_limit)

# 2)
@csp.graph
def clip_v2(x: csp.ts[Union[float, np.ndarray]],
             up_limit: Union[float, csp.ts[np.ndarray], Any] = None,
             dn_limit: Union[float, csp.ts[np.ndarray], Any] = None) -> csp.ts[np.ndarray]:

    return csp.apply(x, lambda u: np.minimum(np.maximum(u, dn_limit), up_limit), np.ndarray)

@csp.graph
def g_v1() -> csp.ts[np.ndarray]:
    x = csp.curve(np.ndarray, [(datetime(2020,1,1) + timedelta(seconds=i), np.array([float(i), float(i)])) for i in range(50)])
    x_std = csp.stats.ema_std(x, com=3, min_periods=1)
    min_std_thresh = csp.stats.quantile(x_std, quant=0.05)
    
    return clip_v1(x_std, dn_limit=min_std_thresh)

res = csp.run(g_v1, starttime=datetime(2020,1,1), endtime=timedelta(seconds=10))
print(res)

Error:

Cell In[19], line 7, in clip_v1(up_limit, dn_limit)
      2 @csp.node
      3 def clip_v1(x: csp.ts[Union[float, np.ndarray]],
      4              up_limit: Union[float, csp.ts[np.ndarray], Any] = None,
      5              dn_limit: Union[float, csp.ts[np.ndarray], Any] = None) -> csp.ts[np.ndarray]:
----> 7     return np.clip(x, dn_limit, up_limit)

TypeError: issubclass() arg 1 must be a class

[bournejt]

The v2 version is the one that works for numpy. It does min and max for 2 arrays elementwise.

[AdamGlustein]

Ok thanks for the repro. When I run off main, I see that the specific error you are seeing was fixed by #490, which was merged 2 days ago.

However, I see another bug with the Union type resolution which I'll open a report for. I get the following error with your code in clip_v1:

  File "csp/ex_csp.py", line 13, in clip_v1
    return np.clip(x, dn_limit, up_limit)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: boolean evaluation of an edge is not supported

But, if I change the annotation to not be a Union and directly take the array, which really shouldn't change the behaviour at all, it works fine.

@csp.node
def clip_v1(x: csp.ts[Union[float, np.ndarray]],
             up_limit: csp.ts[np.ndarray] = None,
             dn_limit: csp.ts[np.ndarray] = None) -> csp.ts[np.ndarray]:

    return np.clip(x, dn_limit, up_limit)

Gives:

{0: [(datetime.datetime(2020, 1, 1, 0, 0), array([nan, nan])), (datetime.datetime(2020, 1, 1, 0, 0, 1), array([0.70710678, 0.70710678])), (datetime.datetime(2020, 1, 1, 0, 0, 2), array([0.72141246, 0.72141246])), (datetime.datetime(2020, 1, 1, 0, 0, 3), array([0.73571813, 0.73571813])), (datetime.datetime(2020, 1, 1, 0, 0, 4), array([0.7500238, 0.7500238])), (datetime.datetime(2020, 1, 1, 0, 0, 5), array([0.76432948, 0.76432948])), (datetime.datetime(2020, 1, 1, 0, 0, 6), array([0.77863515, 0.77863515])), (datetime.datetime(2020, 1, 1, 0, 0, 7), array([0.79294083, 0.79294083])), (datetime.datetime(2020, 1, 1, 0, 0, 8), array([0.8072465, 0.8072465])), (datetime.datetime(2020, 1, 1, 0, 0, 9), array([0.82155217, 0.82155217])), (datetime.datetime(2020, 1, 1, 0, 0, 10), array([0.83585785, 0.83585785]))]}

Also, fyi, clip_v2 won't work as-is since apply only operates on one time-series, so when you call np.maximum(u, dn_limit), dn_limit isn't a time-series input, its just captured in the lambda as an Edge object (graph-time container).

What I'd recommend for now is changing the annotation to not use the Union type and using clip_v1. Hope to fix this bug for the next release. Thanks again for the report!

[AdamGlustein]

Closing this as resolved as we moved the bug report to #495

[bournejt]

@AdamGlustein Thanks for the help! I tried to follow the guidance here https://github.com/Point72/csp/wiki/Build-CSP-from-Source to work off the main branch. The build was successful but I got error saying "No module named 'csp.lib'". I checked the directory. This csp.lib was indeed not there. What might be the reason for this? And can this be added to your developer guidelines?

[AdamGlustein]

Are you building with conda or vcpkg? I build with conda, my local build so's get put in csp/csp/lib

[bournejt]

If I just follow your conda instructions and build a new conda environment just for csp, the build works. I was trying to integrate all these to my working conda env, and that ran into an endless list of errors. So apparently working off local branch with an existing environment isn't very easy. This clip feature is a very common thing to use. I'm curious how would you write this just off the current 0.8.0 release?

[AdamGlustein]

If you just remove the Union annotation, it works off of the current 0.8.0 release. The bug is with the handling of the type annotation only.

from typing import Union
import numpy as np
from datetime import datetime

import csp

@csp.node
def clip_v1(x: csp.ts[Union[float, np.ndarray]],
             up_limit: csp.ts[np.ndarray] = None,
             dn_limit: csp.ts[np.ndarray] = None) -> csp.ts[np.ndarray]:

    return np.clip(x, dn_limit, up_limit)

@csp.graph
def g() -> csp.ts[np.ndarray]:
    a = csp.curve(np.ndarray, [(datetime(2020,1,1), np.array([1.,2.]))])
    lower = csp.curve(np.ndarray, [(datetime(2020,1,1), np.array([2.,3.]))])
    upper = csp.curve(np.ndarray, [(datetime(2020,1,1), np.array([3.,4.]))])
    return clip_v1(a, upper, lower)

res = csp.run(g, starttime=datetime(2020,1,1))
print(res)

{0: [(datetime.datetime(2020, 1, 1, 0, 0), array([2., 3.]))]}

[AdamGlustein]

Also, the local builds should work with your conda environment as long as you ran make develop to install the libraries after make build-conda. I'd be curious to see what errors you saw.

[bournejt]

Thanks for making this straight. I tested on my side and that indeed works. Another thing I noticed is if limits are float, we can pass in one of the limit and leave the other one as default None, and the code would work. For csp.ts[np.ndarray] case in your code above, if only pass one side, we will end up with error saying

TSArgTypeMismatchError: In function clip_v1: Expected ts[csp.typing.NumpyNDArray[float]] for argument 'dn_limit', got None.

I guess this is because the C++ engine cannot resolve a None for csp.ts[np.ndarray] object but fine for the float type? If this can be also added, that would make the code more elegant.

For the build, I agree it should work. In my working conda env, during make build-conda, CMake fails to find the locations of some dependencies that are already installed. Very weird stuff. Tried a bunch of fixes gpt came up with and couldn't resolve. Probably only something funky in my env. Not generalizable.

[AdamGlustein]

Yes, you can't pass a None object as a ts input to a node, since you'll be missing a connection in the graph. However, a float argument is just a scalar, so it can be missing and have no effect on the topology of the computation graph - thus, it's allowed to be None.

If you debug the build issue further feel free to raise another discussion for that - it could be something wonky in your env or it could be an issue with how we're finding dependencies.