Skip to content

Post-PR failures September 2026 #48

Closed

Description

@traversaro

ros-lyrical-ur-simulation-gz
ros-lyrical-ur
ros-lyrical-moveit
Traceback (most recent call last):
  File "/home/runner/work/ros-lyrical/ros-lyrical/.pixi/envs/default/bin/vinca-gha", line 10, in <module>
    sys.exit(main())
             ~~~~^^
  File "/home/runner/work/ros-lyrical/ros-lyrical/.pixi/envs/default/lib/python3.14/site-packages/vinca/generate_gha.py", line 589, in main
    build_unix_pipeline(
    ~~~~~~~~~~~~~~~~~~~^
        stages,
        ^^^^^^^
    ...<6 lines>...
        pixi_version=pixi_version,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/runner/work/ros-lyrical/ros-lyrical/.pixi/envs/default/lib/python3.14/site-packages/vinca/generate_gha.py", line 280, in build_unix_pipeline
    dump_for_gha(workflow, outfile)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/ros-lyrical/ros-lyrical/.pixi/envs/default/lib/python3.14/site-packages/vinca/generate_gha.py", line 160, in dump_for_gha
    raise RuntimeError(
    ...<3 lines>...
    )
RuntimeError: Generated workflow linux_aarch64.yml is 504.7 KiB, exceeding the 500 KiB limit. Increase --batch_size to reduce the number of generated jobs.

https://github.com/RoboStack/ros-lyrical/actions/runs/34697410459

Activity

  1. traversaro commented on Sep 13, 2026

    @traversaro
    MemberAuthor

    The originally reported issue was fixed by @sea-bass in #49 .

    However, we have a much more tricky issue now on linux-64. Due to an error on my side, I halted the first job after the mutex package was produced and uploaded, but now all the re-generated job fail with:

    2026-09-13T07:49:53.9726582Z  │ │ Resolving host environment:
    2026-09-13T07:49:53.9727400Z  │ │   Platform: linux-64 [__unix=0=0, __linux=6.17.0=0, __glibc=2.39=0, __archspec=1=zen2]
    2026-09-13T07:49:53.9728157Z  │ │   Channels: 
    2026-09-13T07:49:53.9728659Z  │ │    - file:///home/runner/conda-bld/
    2026-09-13T07:49:53.9729875Z  │ │    - https://prefix.dev/robostack-lyrical/
    2026-09-13T07:49:53.9730553Z  │ │    - https://prefix.dev/conda-forge/
    2026-09-13T07:49:53.9731082Z  │ │   Specs:
    2026-09-13T07:49:53.9731479Z  │ │    - pkg-config
    2026-09-13T07:49:53.9731894Z  │ │    - numpy 2.*
    2026-09-13T07:49:53.9732306Z  │ │    - pip
    2026-09-13T07:49:53.9732732Z  │ │    - python 3.14.* *_cp314
    2026-09-13T07:49:53.9733339Z  │ │    - ros2-ament-cmake-core
    2026-09-13T07:49:53.9733909Z  │ │    - ros2-ament-package
    2026-09-13T07:49:53.9734384Z  │ │    - ros2-distro-mutex 0.19.* lyrical_*
    2026-09-13T07:49:53.9734720Z  │ │    - libstdcxx >=14
    2026-09-13T07:49:53.9735067Z  │ │    - libgcc >=14
    2026-09-13T07:49:53.9735795Z  │ │    - libgcc >=14
    2026-09-13T07:49:53.9736267Z  │ │    - __glibc >=2.28,<3.0.a0
    2026-09-13T07:49:54.5758094Z  │ │
    2026-09-13T07:49:54.5758473Z  │ ╰─────────────────── (took 3 seconds)
    2026-09-13T07:49:54.5758810Z  │
    2026-09-13T07:49:54.5759074Z  ╰─────────────────── (took 5 seconds)
    2026-09-13T07:49:54.5858483Z Error:   × Failed to resolve dependencies: Cannot solve the request because of: No
    2026-09-13T07:49:54.5859131Z   │ candidates were found for ros2-distro-mutex 0.19.* lyrical_*.
    2026-09-13T07:49:54.5859503Z   │ 
    2026-09-13T07:49:54.5859878Z   ╰─▶ Cannot solve the request because of: No candidates were found for ros2-
    2026-09-13T07:49:54.5860893Z       distro-mutex 0.19.* lyrical_*.
    2026-09-13T07:49:54.5861163Z       
    

    See for example https://github.com/RoboStack/ros-lyrical/actions/runs/34724537008 .

    That is strange as the package is indeed there: https://prefix.dev/channels/robostack-lyrical/packages/ros2-distro-mutex .

  2. traversaro commented on Sep 13, 2026

    @traversaro
    MemberAuthor

    I even tried restarting the original job (the one that produced ros2-distro-mutex ) at https://github.com/RoboStack/ros-lyrical/actions/runs/34697487039 . But the problem is exactly the same.

  3. sea-bass commented on Sep 13, 2026

    @sea-bass
    Contributor

    Is there a way to delete what is on the channel first in case it was corrupted somehow, and then rerunning that generating job?

  4. traversaro commented on Sep 13, 2026

    @traversaro
    MemberAuthor

    Is there a way to delete what is on the channel first in case it was corrupted somehow, and then rerunning that generating job?

    I asked the AI to investigate, and I think it found the problem: https://chatgpt.com/share/6aa68489-d93c-83ed-86c0-51fc626cba8f , that I reported in prefix-dev/prefix-dev#97 .

    To actually fix the problem, as suggested by ChatGPT it was sufficient to manually reindex the channel:

    curl -fSs -X POST \
      -H "Authorization: Bearer ${PREFIX_API_KEY}" \
      https://prefix.dev/api/v1/reindex/robostack-lyrical/linux-64

    and then restart the job.

  5. traversaro commented on Sep 13, 2026

    @traversaro
    MemberAuthor

    Ok, now the problem is different:

     + pixi run upload /home/runner/conda-bld//linux-64/ros-lyrical-ament-cmake-core-2.8.8-hb0f4dca_23.conda --skip-existing
    ✨ Pixi task (upload): rattler-build upload prefix -c robostack-lyrical --generate-attestation /home/runner/conda-bld//linux-64/ros-lyrical-ament-cmake-core-2.8.8-hb0f4dca_23.conda --skip-existing
    + for conda_package in "${conda_packages[@]}"
    + pixi run upload /home/runner/conda-bld//linux-64/ros2-ros-environment-4.5.1-np2py314hb48eaa3_23.conda --skip-existing
    ✨ Pixi task (upload): rattler-build upload prefix -c robostack-lyrical --generate-attestation /home/runner/conda-bld//linux-64/ros2-ros-environment-4.5.1-np2py314hb48eaa3_23.conda --skip-existing
    Error:   × Failed to generate attestation for ros2-ros-environment-4.5.1-
      │ np2py314hb48eaa3_23.conda: Failed to sign attestation with Sigstore:
      │ Signing error: Failed to get timestamp: HTTP error: TSA returned status
      │ 400 Bad Request
      │ Upload aborted because attestation generation was requested but failed.
      │ 
      │ Troubleshooting:
      │ 1. Check that you're running in a supported CI environment (GitHub
      │ Actions, GitLab CI, etc.)
      │ 2. For GitHub Actions, ensure you have 'id-token: write' permission
      │ 3. Verify OIDC token is available and valid
    
  6. sea-bass commented on Sep 13, 2026

    @sea-bass
    Contributor

    Oh, I got a few of those too. Seems you can just rerun those jobs and it will work next time. (Which it did; I just restarted)

    But yes, I have encountered lots of little flakes like that.

  7. sea-bass commented on Sep 13, 2026

    @sea-bass
    Contributor

    Just restarted the linux64 workflow again; the other flake that happens is sometimes the file that was just uploaded in a previous job doesn't yet get found by the next ones.

    https://github.com/RoboStack/ros-lyrical/actions/runs/34754003472/job/103721571724

    EDIT: Had to do this 7 more times for this build

  8. Tobias-Fischer commented on Oct 18, 2026

    @Tobias-Fischer
    Contributor

    Addressed in the RoboStack monorepo, RoboStack/robostack, where all ROS 2 distributions are now built: the vinca-gha error here (generated workflow over GitHub's 500 KiB) is handled by retrying with larger batches (RoboStack/robostack#96), and the flake where a just-uploaded package isn't found yet by the next job is retried there while the channel index catches up (RoboStack/robostack#22).

    🤖 Claude (AI assistant), on behalf of @Tobias-Fischer

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions