iojs+release builds failed with broken pipes

[richardlau]

Refs: nodejs/node#41804 (comment)

It looks like the 16.14.0 build (https://ci-release.nodejs.org/job/iojs+release/8244/) failed to upload the assets to the staging across all platforms:
e.g. https://ci-release.nodejs.org/job/iojs+release/8244/nodes=centos7-ppcle-release-64/console

23:51:16 scp -p node-v16.14.0-linux-ppc64le.tar.gz node-www:nodejs/release/v16.14.0/node-v16.14.0-linux-ppc64le.tar.gz
23:51:16 packet_write_wait: Connection to 138.197.224.240 port 22: Broken pipe
00:14:16 lost connection
00:14:16 make: *** [Makefile:1231: binary-upload] Error 1

Although the later nightly build (https://ci-release.nodejs.org/job/iojs+release/8245/) succeeded.

[richardlau]

My suspicion is that it could be related to a planned migration of the droplet for staging, although the expected time window was later than the point that the builds failed.

Start: 2022-02-08 05:00:00 (UTC)
End: 2022-02-08 13:00:00 (UTC)

Hello,

As part of our efforts to keep all Droplets on our platform healthy, we have scheduled a maintenance on the physical machine(s) hosting one or more of your Droplet(s), which are listed below. During the above window, we will be performing migrations of all affected Droplets to new physical machines, to ensure optimal performance.

We will attempt to perform live migrations in all possible cases. A live migration would result in no downtime, but minor performance decreases in disk I/O and a second or less of packet loss as the network is switched over to the new physical host.

In the event that we are not able to perform a live migration of a Droplet, we will perform an offline migration during which the Droplet will be powered off and migrated offline during the window.

[richardlau]

Suggest we monitor for a bit to see if this reoccurs or was just a one off.

[richardlau]

AFAIK this hasn't reoccurred. Optimistically closing.

[richardlau]

This happened again today for all platforms in https://ci-release.nodejs.org/job/iojs+release/8302/.
Some of the next build (v8-canary) also failed but others succeeded https://ci-release.nodejs.org/job/iojs+release/8303/.
It looks like all of the ones that succeeded attempted their scp's later (nearer to 10:50am UTC) than the failing ones.

[richardlau]

I rebuilt https://ci-release.nodejs.org/job/iojs+release/8302/ and it passed and succeeded scp'ing the assets to staging https://ci-release.nodejs.org/job/iojs+release/8304/ 🤷