Use of models beyond the limits for which they were validated or intended

[adriesse]

It occurs to me that it could be an interesting feature of PVLIB, to produce a certain class of warnings whenever input parameters fall outside the range for which models were developed or validated. Could be a latitude range in the case of tilted irradiance models, as a possible example. Thoughts?

[cwhanse]

A good idea, but determining the conditions for validation is probably not practical. It's not always clear what filtering was done on the data. Also, some models were developed using hourly data, for example, would we warn if they were applied to minute data?

[wholmgren]

I agree that could be interesting but challenging. The first step would probably be to add more and better documentation of model limitations to pvlib. Perhaps put these in a special box in the rendered documentation.

[markcampanelli]

Wouldn’t it be nice to better know what parts of pvlib are used most? That’s a can of worms too, I suppose.

[adriesse]

Good ideas! I like the warning for mismatch on the time scale too.

Yes, it would be challenging and could probably never be done everywhere and documentation could come first. But what would speak against doing it for a few easy-ish cases as examples for future coders to follow? The time scale one could be appropriate for detect_clearsky() for example, where the limits are already in the documentation.

I must admit I don't know quite how to do this. Can you create warning classes similar to exception classes or something?

[cwhanse]

I think it needs to be more mild than a warning, more like a FYI

…

________________________________ From: Anton Driesse <notifications@github.com> Sent: Thursday, February 22, 2018 3:09 PM To: pvlib/pvlib-python Cc: Hansen, Clifford W; Comment Subject: [EXTERNAL] Re: [pvlib/pvlib-python] Use of models beyond the limits for which they were validated or intended (#433) Good ideas! I like the warning for mismatch on the time scale too. Yes, it would be challenging and could probably never be done everywhere and documentation could come first. But what would speak against doing it for a few easy-ish cases as examples for future coders to follow? The time scale one could be appropriate for detect_clearsky() for example, where the limits are already in the documentation. I must admit I don't know quite how to do this. Can you create warning classes similar to exception classes or something? - You are receiving this because you commented. Reply to this email directly, view it on GitHub<#433 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AFJNLyL27CVuipq8iFoH2VCylx1oh3XRks5tXeWPgaJpZM4SMG_L>.

[wholmgren]

Can you create warning classes similar to exception classes or something?

See PVLibDeprecationWarning in the examples in #427 (comment)

The gist of the warnings in #427 is

import warnings

class PVLibDeprecationWarning(UserWarning):
    pass

message = complicated_stuff_copied_from_matplotlib()
warnings.warn(message, PVLibDeprecationWarning)

You could imagine a pvlibModelValidationWarning defined and used in this simple way. The complicated stuff in #427 is to autofill information from the decorated function, but we'd skip that for this feature (at least initially).

I think this strategy would be much better than print or logging because users that understand the risk can easily silence the warnings using the python warning machinery.

We'd also want to add tests to ensure that the warnings are issued and not issued as appropriate.

[mikofski]

TL;DR: maybe this could work in a data-model checker/validator (skip to the very bottom) and there are already several packages that do this (skip to list about halfway down), but I would prefer keep pvlib as a fast, bare metal library with stern warnings in the documentation.

random rambles

In principle I like the idea, but I have my concerns as well:

1. My first concern would be how would all these warnings and checks affect performance? This may be one of the arguments for duck-typing and EAFP vs. LBYL. I think it's possible to implement a few warnings if they use the try:except format which should in theory have the minimum impact

   in practice trying to write this makes me think it needs some type of validation decorator that raises the exception. Something like:

def validator(a, b):
    def innerfunc(f):
        def wrapper(x):
            if a <= x <= b:
                return f(x)
            else:
                raise Exception('Validation error')
        return wrapper
    return innerfunc


@validator(2, 4)
def g(x):
    return x/2


>>> g(3)
1.5

>>> g(1)
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-21-3580407364ae> in <module>
----> 1 g(1)

<ipython-input-18-9bf5d6a1b9b8> in wrapper(x)
      5                 return f(x)
      6             else:
----> 7                 raise Exception('Validation error')
      8         return wrapper
      9     return innerfunc

Exception: Validation error

>>> g(5)
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-22-8ec331d73f51> in <module>
----> 1 g(5)

<ipython-input-18-9bf5d6a1b9b8> in wrapper(x)
      5                 return f(x)
      6             else:
----> 7                 raise Exception('Validation error')
      8         return wrapper
      9     return innerfunc

Exception: Validation error

So my concern about performance was reinforced, because I don't see how to check x limits without using if and condition tests, which will take time, and in the case of a time-series 100k steps long, this will add overhead, especially if every function has this level of checking.

This also made me think that there's probably already something out there like this, a quick Google search for "python"+"validator" turned up these 3, but there may be more:

• validators-0.14.0 308 GH stars
• validator.py-1.3.0 124 GH stars
• validator-collection-1.3.7 47 GH stars

2. My second concern is readability. I find it sometimes tricky to wrap code in try:except although it is easier to read than if:else it just needs to be thought out well and explained with lots of comments - which then look like documentation which then makes me think that the warning should just go there, in the documentation.

So I don't know what's best, I appreciate the concern, but I'm not sure how it can be implemented and address my concerns. I guess I prefer the idea of stressing limits on inputs in the documents more.

I also talked to some colleagues, and they suggested that perhaps there could be either an optional or separate application for folks who wanted this type of rigorous checking. It occurred to me that validators are typically used for UIs, whereas pvlib is more of a bare metal library.

Data Model Checker

One other idea, perhaps spurred by my colleagues' comments, is that if we implement a data model as suggested in the pvlib roadmap user needs and v1.0 milestone then a user could optionally run the data model through the rigorous checker/validator.

Basically,

1. a user would create a data model, perhaps in JSON or YAML,
2. then they could optionally pass it through the checker
3. or just pass it straight into the model chain

[adriesse]

Thanks for pointing out the new roadmap--I hadn't noticed it before.

An small example from numpy for what it's worth:

In[1]: import numpy as np
In[2]: np.arccos([0,1,2])
__main__:1: RuntimeWarning: invalid value encountered in arccos
Out[2]: array([1.57079633, 0.        ,        nan])