Harden the Host MIB pages and poller, and fix review findings

.github/workflows/plugin-ci-workflow.yml

@@ -194,23 +194,48 @@ jobs:
     - name: Set expected Cacti version for unit tests
       run: echo -n "${{ env.CACTI }}" | sudo tee ${{ github.workspace }}/cacti/plugins/hmib/tests/.cacti-version > /dev/null
 
+    # The installer ran as root and created the Cacti log, but the end-to-end
+    # tests start Cacti's web server and CLI scripts as the runner, and Cacti
+    # serves nothing while its log is not writable.
+    - name: Make the Cacti log writable for the test run
+      run: |
+        sudo touch ${{ github.workspace }}/cacti/log/cacti.log
+        sudo chown www-data:runner ${{ github.workspace }}/cacti/log/cacti.log
+        sudo chmod 664 ${{ github.workspace }}/cacti/log/cacti.log
+
     - name: Run Pest Unit Tests
       run: |
         cd ${{ github.workspace }}/cacti
         "$CACTI_PEST_BIN" --configuration=plugins/hmib/phpunit.xml \
           --coverage-clover=plugins/hmib/coverage/clover.xml
 
+    # The page scripts and the pollers cannot be included in the test process,
+    # so the end-to-end tests run them as real requests and subprocesses. This
+    # second run records, test by test, which of their lines ran (see
+    # tests/bin/coverage-prepend.php and tests/support/coverage.php).
+    - name: Run Pest with per-test coverage of the entry points
+      if: github.event_name == 'pull_request'
+      env:
+        HMIB_COVERAGE_DIR: ${{ github.workspace }}/cacti/plugins/hmib/coverage/e2e
+        HMIB_COVERAGE_PER_TEST: '1'
+      run: |
+        cd ${{ github.workspace }}/cacti
+        mkdir -p "$HMIB_COVERAGE_DIR"
+        php -d auto_prepend_file="$PWD/plugins/hmib/tests/bin/coverage-prepend.php" -d xdebug.mode=coverage \
+          "$CACTI_PEST_BIN" --configuration=plugins/hmib/phpunit.xml
+
     # Whole-file coverage is meaningless here: most of the plugin only runs
     # inside a live Cacti. What is enforceable is that a change covers the
-    # lines it adds.
+    # lines it adds, from the unit run and from the end-to-end runs together.
     - name: Enforce coverage of changed lines
       if: github.event_name == 'pull_request'
       env:
         BASE_REF: ${{ github.event.pull_request.base.sha }}
       run: |
         cd ${{ github.workspace }}/cacti/plugins/hmib
         git config --global --add safe.directory ${{ github.workspace }}/cacti/plugins/hmib
-        php tests/bin/patch-coverage.php coverage/clover.xml "$BASE_REF" 100
+        php tests/bin/merge-coverage.php coverage/e2e "$BASE_REF" --merge-clover=coverage/clover.xml --clover=coverage/merged.xml
+        php tests/bin/patch-coverage.php coverage/merged.xml "$BASE_REF" 100
 
     - name: Upload coverage report
       if: always()
@@ -248,6 +273,32 @@ jobs:
           exit 1
         fi
 
+    - name: Run the Host MIB poller against the installed tables
+      run: |
+        cd ${{ github.workspace }}/cacti
+
+        # The Cacti poller starts its own main run in the background; this one
+        # would be refused while that still holds the run lock.
+        for attempt in $(seq 1 120); do
+          pgrep -f 'poller_hmib.php -M' > /dev/null || break
+          sleep 1
+        done
+
+        mysql --defaults-file="$HOME/.my.cnf" -e "REPLACE INTO settings (name, value) VALUES ('hmib_enabled', 'on')" cacti
+        sudo -u www-data timeout 900 php plugins/hmib/poller_hmib.php -M --force --debug | tee "$RUNNER_TEMP/hmib-poller.log"
+
+        if ! grep -q "Host Mib Polling Completed" "$RUNNER_TEMP/hmib-poller.log"; then
+          echo "Host MIB poller did not finish"
+          exit 1
+        fi
+
+        for table in plugin_hmib_hrSystem plugin_hmib_hrSystemTypes plugin_hmib_hrSWRun plugin_hmib_hrSWRun_last_seen plugin_hmib_hrSWInstalled plugin_hmib_hrDevices plugin_hmib_hrStorage plugin_hmib_hrProcessor plugin_hmib_processes plugin_hmib_types; do
+          if ! mysql --defaults-file="$HOME/.my.cnf" -N -e "SHOW TABLES LIKE '$table'" cacti | grep -qx "$table"; then
+            echo "Missing table $table"
+            exit 1
+          fi
+        done
+
     - name: View Cacti Logs
       if: always()
       run: |

CHANGELOG.md

@@ -1,6 +1,32 @@
 ChangeLog
 
 --- develop ---
+* issue: Keep shift-click multi-column sorting on the Host MIB tabs now that their sort is checked against the sortable columns, checking each column and direction of the stored sort
+* feature: Add a Process History Retention setting that drops Use History rows for processes not seen running for the chosen number of days; the default, Never Purge, keeps them
+* dev: Run the Host MIB main poller in CI against the installed plugin tables after the Cacti poller
+* dev: Print the OS Type, Device and Template filters of the Host MIB tabs from one function instead of six copies; its Device list binds the chosen OS type and the option values are cast to numbers
+* security: Restrict the sort column on the Devices, Storage, Hardware, Processes, Use History and Inventory tabs and on the OS type list to the columns each table shows, bind the OS type id on the edit form and the values of the OS type CSV import, and cast the remaining numeric filter option values
+* issue: Purge the Host MIB rows of devices removed from Cacti with one statement per table instead of seven per device, including rows left without a system entry
+* issue: Stop repeating per-row lookups: the Summary tab looks up the summary device once per page instead of once per device type, the graph links find an OS type's devices with a join instead of a separate query, and the processor collector asks once per device whether it runs Linux; the device and type ids in those queries and the summary statistics insert are bound or cast to numbers
+* issue: Remove the autodiscovery process lock however the poller run ends, including on SIGTERM or SIGINT and on a fatal error, so a scan that dies no longer blocks every later run, and stop autodiscovery once the run has used its 10 minutes
+* issue: Stop a main poller run from starting while another is still running by holding a database lock for the whole run, instead of relying on collector rows that are purged after 10 minutes; the main run now stops launching and waiting on collectors 10 minutes after it began, and the launch placeholders use keys that cannot match a process id
+* issue: Keep a device's stored process, storage, hardware, software and processor rows when the SNMP walk for that table fails or times out, instead of deleting them; core returns an empty walk for a timeout too, so an empty table clears the stored rows only when the device still answers a sysUpTime GET
+* issue: Fix PHP warnings on the Summary tab when a request names only the device type table or only the process table and the other table has no stored sort
+* issue: Fix the process Time Seen total growing on every poller cycle instead of once per process performance collection
+* issue: Fix graph automation not creating the Host MIB summary device graphs until the next scheduled run after the device is first created
+* issue: Fix graph automation checking the users graph template instead of the processes graph template before adding the processes graph
+* issue: Fix the host type CSV export producing broken rows for values that contain a double quote, and open values that look like formulas as text
+* security: Run the OS type rescan from a POST request with a CSRF token instead of a GET link
+* issue: Fix the OS type rescan failing with a duplicate key error when several unclassified devices report the same sysDescr and sysObjectID, and stop it creating an empty match type
+* security: Escape the OS type name, version, sysObjectID and sysDescr match when listing host types and the type version on the Summary tab
+* security: Restrict the Summary tab sort column to the columns each table can sort on and limit the direction to ASC or DESC
+* issue: Fix the Devices tab processes glyph opening an empty page because of a malformed action value in its link
+* issue: Fix the Summary Processes filter failing with a SQL error because the page count query had no WHERE clause
+* issue: Fix the Summary tab dropping the data-query graph links on PHP 8 because a device id of 0 was treated as a device filter
+* security: Bind the stored sysObjectID and sysDescr patterns in the OS type rescan and limit the match to devices that are still unclassified
+* security: Bind the SNMP index as a query parameter in the graph link builder used by the Summary Processes view
+* issue: Fix the Graphs tab failing with a fatal error on Cacti 1.2.x, where the graph filter calls a function that core declares only in graph_view.php
+* issue: Speed up the OS type rescan by trying each stored type once against all unclassified devices instead of once per stored type and per distinct reported sysObjectID and sysDescr pair
 * issue: Fix the Fleet Dashboard Uptime Distribution showing all zeros by falling back to SNMP sysUpTime when the Host Resources hrSystemUptime is not reported, and apply the same fallback to the per-device Dashboard
 * issue: Stop the Dashboard card header from changing height when hovering the drag handle or tool buttons by reserving space for the theme's button hover border
 * feature: Make the Fleet Dashboard the first tab and the default landing view on the Host MIB interface