Support setup/teardown functions

[seddonym]

I have some code I'd like to benchmark on codspeed which requires custom setup before each iteration. (If you're interested, it's testing the performance of a partially-populated cache which will get fully populated on each iteration, so I need to reset it to a partial state.)

pytest-benchmark supports this via a setup callback in its pedantic mode. It would be great if there was some way of achieving this using codspeed. (Might as well support teardown too.)

[crusaderky]

This looks supported starting from 4.0.0 beta.

def test_pedantic_mode(benchmark):
    def setup():
        # Setup code that shouldn't be measured
        data = [0]
        return (data,), {}  # Returns (args, kwargs) for target

    def target(data):
        assert data[0] == 0
        data[0] += 1

    result = benchmark.pedantic(
        target,
        setup=setup,
        rounds=100,
    )