[ekrichbaum]

rebuild Digital Ocean source based on development branch. Had to remerge a few elements from previous pull

[semx]

Thanks for reworking this on top of development. The AWS breakage from #604 is gone (it replaced the AWS registration in module/sources/__init__.py and module/config/file_output.py instead of adding to it), the full suite passes, and the inventory.py change that matches VM interfaces and disks by the resolved VM also fixes the same mix-up in the Hetzner source.

I ran it end to end against a stub of the DigitalOcean API (response shapes from the API reference) and a stub NetBox. Three things need to change before it goes in:

1. Droplets with the same name in one region are merged. DigitalOcean doesn't require unique names. With two Droplets named web-1 in nyc3, NetBox gets one VM with serial, vCPUs, memory and root disk from one Droplet and primary_ip6 and the volume from the other, both Droplets' addresses on the same interfaces, no warning, and a PATCH on every run (Virtual machine 'web-1' attribute 'serial' changed from '5000004' to '5000001'). A database orders (db-orders) collides with a Droplet named db-orders the same way. Please match the existing VM by the Droplet/database id (digitalocean_resource_id) before name and cluster, and add the id to duplicate names the way the AWS source does.

2. Database region. The API returns a database's region as a slug string ("sfo3"), not an object as in the test fixture. A database in a region without Droplets creates cluster cloud: sfo3 and region sfo3; once a Droplet shows up there, the next run creates cloud: San Francisco 3 with a second db-reports VM, and the first one is orphaned. Resolving slugs once per run via GET /v2/regions makes the cluster independent of which resources exist in a region. Please also fix the fixture to "region": "nyc1".

3. NetBox before 4.1. Disk sizes are always written in MB, but before 4.1 NetBox stores them in GB (a 50 GB root disk lands as 51200 on 4.0), and serial is sent on every version, so older NetBox gets a PATCH on every run. Please gate both on >= 4.1 as aws/connection.py does.

Smaller, we can take these at merge: API errors (401, 429) end the run with a bare traceback and DigitalOcean's message isn't shown, so a clean do_error_exit with that message like the AWS source would help; regions created by the source lose their Source: tag on the second run; regenerate the DigitalOcean section of settings-example.ini with ./netbox-sync.py -g; typo in tests/test_digitalcoean_source.py. We'll squash, since the intermediate commits don't import on their own.

Conflicts with #616 and #617 are only in the registration lists, README and the end of settings-example.ini.

[ekrichbaum]

It will take a few hours to validate and retest. It should not be difficult to suffix duplicate names with IDs. Let me get some changes mocked up and kick it into validation.

[ekrichbaum]

That was a rabbit hole. I don't know why anyone would do those kinds of things but they absolutely can within the DO world. This should take care of more of those edge cases.

[semx]

Thanks, this is close. ae0f3b1 wrote the new handler into config.py instead of connection.py, so the branch stopped importing (CI fails in conftest.py with cannot import name 'DigitalOceanConfig'). I pushed a commit that puts it back (config.py as of 96b74e2, the handler in connection.py, nothing else changed) and merged development to resolve the conflicts with #617. The full suite passes with that.

I ran this version against the API stubs again and against a real DigitalOcean account (two Droplets named nbs-web-1 in fra1, a Droplet with a volume, a PostgreSQL cluster in ams3, NetBox stubbed). Fixed:

• same-name Droplets get their own VMs, nbs-web-1 (616819749) and nbs-web-1 (616819750), and the db-orders collision is gone; the next runs write nothing;
• the database lands in cloud: Amsterdam 3 right away and stays there when a Droplet shows up in that region;
• on NetBox before 4.1 disk sizes are in GB and serial isn't sent, so repeat runs write nothing;
• API errors end the run with DigitalOcean's message, regions keep their Source: tag, the DigitalOcean section of settings-example.ini matches -g, and the test file name is fixed.

Two things are still open:

1. A resource recreated under the same name never gets into NetBox. On the real account I deleted nbs-app (616819771) and created it again (616820558). From then on every run fails the POST of the new VM, because the old nbs-app is still in the cluster, and then the POSTs of its interfaces and disk; the new Droplet never shows up and the exit code stays 0. In _sync_vm() the VM found by name and cluster carries another digitalocean_resource_id, so it's set aside and a second VM with the same name is created, which NetBox refuses since VM names are unique per cluster. The old VM is only marked orphaned, and with the default prune_enabled = False it's never deleted, so this doesn't recover on its own. A database recreated under the same name and a Droplet renamed to the name of a deleted one end the same way. 8e04d64 reused the VM here. Please set the name match aside only when its digitalocean_resource_id belongs to a resource of the current run, reuse the VM otherwise, and add a test for that branch: with the branch removed all 90 tests still pass.

2. Duplicate global IPs end the whole run. _sync_ip() calls do_error_exit when two NetBox IPs without a VRF have the same address. NetBox allows that: ENFORCE_GLOBAL_UNIQUE can be turned off, and anycast, VIP, VRRP, HSRP, GLBP and CARP addresses are exempt even when it's on. Private addresses go to the global table unless private_ip_vrf is set, so one such pair stops all sources, on every run (8e04d64 synced the same state). Please log an error and skip that address instead; the shared IP handling in SourceBase doesn't stop on such duplicates either.

We'll take these at merge: send the do_error_exit messages through log.error as well, so they reach log_file the way the AWS source does, use generic example names in the docs, tests and the private_ip_vrf example, and squash.

[semx]

Merged with fixes for both points, so this didn't need another round from you:

• A VM found by name and cluster is reused unless its digitalocean_resource_id belongs to another resource of the same run, so a Droplet or database recreated under the same name keeps its VM, with the new resource id and serial. If a VM of a deleted resource still holds the name, for example after renaming a Droplet to the name of a deleted one, the resource id is added to the name until that VM is gone.
• Duplicate IPs in one VRF no longer end the run: the one already on the interface is used, otherwise the address is skipped with an error that points to private_ip_vrf.
• Error exits also go through log.error, so they reach log_file.
• Generic example names in the docs, tests and settings-example.ini.

On the real account a recreated nbs-app now keeps its VM and the run after that writes nothing. Thanks for the source and for chasing down all the DigitalOcean edge cases!