Add Python bindings to Btwxt via SWIG by jmarrec · Pull Request #44 · bigladder/btwxt

jmarrec · 2025-01-27T10:00:04Z

I was struggling to understand what EnergyPlus was doing with my Table:Lookup and Table:IndependentVariables, and wanted a way to calculate and plot what was going on (and compare with scipy.interpolate.RegularGridInterpolator).

Instead of using a debugger, or tweaking E+ source code to spit out the values I needed, I figured it would be much better to add Python bindings.

Since I had done that, I figured it could be useful for other people (at least some E+ devs, and yourselves as btwxt devs for inspection/validation purposes), so I'm proposing it for inclusion.

This PR:

Enables building Python bindings via SWIG (this is opt-in)
Adds a Pytest (registered as a Ctest) to exercise the bindings
Adds a demo notebook
Tweaks the existing workflow to build the python wheels on macos x86_64 and arm64, ubuntu x86_64 and windows x86_64
- It would be quite easy to publish them to TestPyPi / PyPi if you want to do so. An example can be found here: https://github.com/NREL/OpenStudio/blob/09472be58fa387840d7e07f93e41f4da97294478/.github/workflows/python_bindings.yml#L329-L375

Find a working workflow with artifacts here: https://github.com/jmarrec/btwxt/actions/runs/12986553478

Note: for MSVC, UseSWIG.cmake already sets it to .pyd

(Tired of windows/MSVC...)

jmarrec · 2025-01-27T10:01:29Z

+            python_bindings: true
+            arch: arm64
          - os: macos
-            os_ver: "11"


macos-11 was removed from GHA a long time ago.

jmarrec · 2025-01-27T10:01:53Z

+            arch: x86_64
          - os: macos
-            os_ver: "12"
+            os_ver: "14"


macos-12 is removed.

actions/runner-images#10721

I uses 13 (x86_64) and 14 (arm64, M1)

jmarrec · 2025-01-27T10:02:23Z

view it rendered at https://github.com/bigladder/btwxt/blob/240d733840c0569ab42386bde8a2da59a2695b99/docs/Python_bindings_demo.ipynb

jmarrec · 2025-01-27T10:02:56Z

+cmake_policy(SET CMP0078 NEW)
+cmake_policy(SET CMP0086 NEW)


Couple of SWIG-related policies

jmarrec · 2025-01-27T10:03:10Z

 add_subdirectory("include/${PROJECT_NAME}")
 add_subdirectory(src)
 add_subdirectory(vendor)
+add_subdirectory(python)


Add subdir. Did not change the other directories.

jmarrec · 2025-01-27T10:09:50Z

+  add_custom_target(python_package
+    COMMAND ${CMAKE_COMMAND} -E rm -Rf "build/" "dist/" "btwxt.egg-info/"
+    COMMAND ${CMAKE_COMMAND} -E echo "To create the package, cd 'python/'"
+    COMMAND ${CMAKE_COMMAND} -E echo "${Python_EXECUTABLE} setup.py bdist_wheel"
+    COMMAND ${Python_EXECUTABLE} setup.py bdist_wheel


Build the python wheel (.whl) with this custom target

jmarrec · 2025-01-27T10:10:00Z

+    COMMAND ${CMAKE_COMMAND} -E echo "To upload the package, cd 'python/'"
+    COMMAND ${CMAKE_COMMAND} -E echo "${Python_EXECUTABLE} -m twine upload --verbose --skip-existing --repository ${PYTHON_PIP_REPOSITORY} dist/*"


Do not upload, but tells you how

jmarrec · 2025-01-27T10:10:18Z

+  if (${PROJECT_NAME}_BUILD_TESTING)
+    set(test_case_name btwxt_python)
+    add_test(
+          NAME "${test_case_name}"
+          COMMAND ${Python_EXECUTABLE} -m pytest --verbose test_btwxt.py
+          WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}/test
+    )
+    # Pytest Exit code 5 means all tests were deselected
+    set_tests_properties("${test_case_name}" PROPERTIES SKIP_RETURN_CODE 5)
+    set_tests_properties("${test_case_name}" PROPERTIES ENVIRONMENT "PYTHONPATH=${CMAKE_CURRENT_BINARY_DIR}/btwxt")
+  endif()


Register a pytest file as a CTest.

jmarrec · 2025-01-27T10:10:38Z

+import btwxt
+import pytest
+from itertools import product
+
+def test_grid_axis():
+    axis_x = btwxt.GridAxis()
+    assert axis_x.get_length() == 0
+
+    xs = [30, 35, 40, 45, 50, 55]
+    axis_x = btwxt.GridAxis(
+        [30, 35, 40, 45, 50, 55], # Values
+        btwxt.InterpolationMethod_cubic, # interpolation method
+        btwxt.ExtrapolationMethod_linear, # extrapolation_method
+        (20.0, 60.0), # extrapolation_limits
+        "LWT", # name
+    )
+    assert axis_x.get_length() == 6
+    assert set(axis_x.get_values()) == set(xs)
+    assert axis_x.get_interpolation_method() == btwxt.InterpolationMethod_cubic
+    assert axis_x.get_extrapolation_method() == btwxt.ExtrapolationMethod_linear
+    assert axis_x.get_extrapolation_limits() == (20.0, 60.0)
+
+
+def F(x, y):
+    return x**2 + y**2
+
+def test_interpolator():
+    x0 = [1.0, 2.0, 3.0]
+    x1 = [10.0, 20.0]
+    # Not using numpy on purpose
+    ys = [F(x,y) for (x, y) in product(x0, x1)]
+
+    axis_x0 = btwxt.GridAxis(
+        x0, # Values
+        btwxt.InterpolationMethod_cubic, # interpolation method
+        btwxt.ExtrapolationMethod_linear, # extrapolation_method
+        (0, 5), # extrapolation_limits
+        "x0", # name
+    )
+    axis_x1 = btwxt.GridAxis(
+        x1, # Values
+        btwxt.InterpolationMethod_cubic, # interpolation method
+        btwxt.ExtrapolationMethod_linear, # extrapolation_method
+        (0, 30), # extrapolation_limits
+        "x1", # name
+    )
+    grid_point_y = btwxt.GridPointDataSet(ys, "y")
+
+    interpolator = btwxt.RegularGridInterpolator([axis_x0, axis_x1], [grid_point_y])
+
+    target_normalization = [2, 10]
+    normalizationDivisor = 1.0
+    normalizationDivisor = interpolator.normalize_grid_point_data_set_at_target(
+        0, target_normalization, normalizationDivisor
+    ) * normalizationDivisor
+    assert normalizationDivisor == 104.0
+
+    interp_x0s = [float(x) for x in range(0, 6, 1)]
+    interp_oat_dbs = [float(x) for x in range(0, 31, 5)]
+
+    test_points = [(x,y) for (x, y) in product(interp_x0s, interp_oat_dbs)]
+    true_values = [F(x, y) for (x, y) in test_points]
+    interp_values = [
+        interpolator.get_value_at_target((x0, x1), 0) * normalizationDivisor
+        for (x0, x1) in test_points
+    ]


pytest file to test out the python bindings

jmarrec · 2025-01-27T10:11:17Z

+      - name: Set up Python 3.12
+        if: ${{ matrix.python_bindings }}
+        id: setup-python
+        uses: actions/setup-python@v5
+        with:
+          python-version: 3.12
+
+      - name: Build Python Wheel
+        if: ${{ matrix.python_bindings }}
+        working-directory: ./build/
+        run: |
+          begin_group() { echo -e "::group::\033[93m$1\033[0m"; }
+
+          begin_group "Install other python dependencies"
+          python -m pip install --upgrade pip
+          # Ubuntu 22.04 from apt will get swig that's too old to support attributes like [[ nodiscard ]], so get from PyPi
+          pip install setuptools wheel twine requests packaging pytest swig
+          echo -e "::endgroup::"
+
+          begin_group "Build target"
+          cmake -DBUILD_PYTHON_BINDINGS:BOOL=ON -DPython_REQUIRED_VERSION:STRING=${{ steps.setup-python.outputs.python-version }} -DPython_ROOT_DIR:PATH=$RUNNER_TOOL_CACHE/Python/${{ steps.setup-python.outputs.python-version }}/${{ matrix.arch }}/ .
+          cmake --build . --config ${{ matrix.config }}  --target python_package
+          echo -e "::endgroup::"
+
+      - name: Python CTest
+        if: ${{ matrix.python_bindings }}
+        working-directory: ./build/
+        run: |
+          ctest -C ${{ matrix.config }} -R python -VV
+
+      - name: Zip the wheel to maintain case sensitivy and file permissions
+        if: ${{ matrix.python_bindings }}
+        working-directory: ./build/python
+        shell: bash
+        run: |
+          TGZ_STEM="btwxt-python-${{ matrix.os }}-${{ matrix.os_ver }}-${{ matrix.arch }}"
+          echo "TGZ_STEM=$TGZ_STEM" >> $GITHUB_ENV
+          tar -cvzf "$TGZ_STEM.tar.gz" dist/
+
+      - name: Upload .whl to artifact
+        if: ${{ matrix.python_bindings }}
+        uses: actions/upload-artifact@v4
+        with:
+          name: ${{ env.TGZ_STEM }}
+          path: "./build/python/${{ env.TGZ_STEM }}.tar.gz"


New workflow bits to build the python bindings after the regular build is done.

Runs the python ctest, and uploads the wheel as artifacts

nealkruis · 2025-02-06T15:35:12Z

@jmarrec, thanks for putting this together! We've always had the intention to provide bindings for our C++ libraries, but we haven't yet settled on the best approach. We also haven't gotten very far in investigating the options, and I really don't know much about SWIG other than seeing some mixed reviews.

jmarrec added 22 commits January 24, 2025 15:57

Add Python bindings via SWIG

0dcde7f

Add a setup.py and cmake commands to create a bdist_wheel

c6f0832

Add a notebook to demo the python bindings

fbab36b

Build wheel on github actions?

e7876bc

Bump runners

6541ef6

Do in two steps

d75b293

bump gcc for 24.04

70ece55

Install setuptools and co

1afbe1e

link to Python::MODULE on macos too?

c9c8742

tweak

9c71d19

Plop

d3b7ca6

tweak

9cbee2b

matrix.config isn't replaced with it's value, try to quote it

5ca75cf

Try installing swig latest from pip

04f18bc

Tweak workflow again

1c6eb0c

Set swig lib suffix to so on Apple (not dylib).

09134a9

Note: for MSVC, UseSWIG.cmake already sets it to .pyd

Set "-C Release" for windows testing (should use Ninja instead...)

56c694a

Maybe I do need a .pyd suffix on MSVC, it's building a .lib instead...

73e984d

Avoid having a Release/ sublevel

ff9ccac

(Tired of windows/MSVC...)

Typo in last commit

497b50d

Rename artifact for clarity

240d733

Remove duplicated find python

4aee6c4

jmarrec commented Jan 27, 2025

View reviewed changes

Navigation Menu

Uh oh!

Add Python bindings to Btwxt via SWIG - #44

Add Python bindings to Btwxt via SWIG#44
jmarrec wants to merge 22 commits into
bigladder:mainfrom
jmarrec:python_bindings

jmarrec commented Jan 27, 2025 •

edited

Loading

Uh oh!

jmarrec Jan 27, 2025

Uh oh!

jmarrec Jan 27, 2025

Uh oh!

jmarrec Jan 27, 2025

Uh oh!

jmarrec Jan 27, 2025

Uh oh!

jmarrec Jan 27, 2025

Uh oh!

jmarrec Jan 27, 2025

Uh oh!

jmarrec Jan 27, 2025

Uh oh!

jmarrec Jan 27, 2025

Uh oh!

jmarrec Jan 27, 2025

Uh oh!

jmarrec Jan 27, 2025

Uh oh!

nealkruis commented Feb 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		COMMAND ${CMAKE_COMMAND} -E echo "To upload the package, cd 'python/'"
		COMMAND ${CMAKE_COMMAND} -E echo "${Python_EXECUTABLE} -m twine upload --verbose --skip-existing --repository ${PYTHON_PIP_REPOSITORY} dist/*"

Uh oh!

Conversation

jmarrec commented Jan 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nealkruis commented Feb 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jmarrec commented Jan 27, 2025 •

edited

Loading