feat(keys): reuse expired metric slots (deferred, stacked on #25)

CHANGELOG.md

@@ -4,6 +4,20 @@ This file only calls out major changes. Please see [the list of Git commits](
 https://github.com/api7/nginx-lua-prometheus/commits/main) for the full list
 of changes.
 
+## Unreleased
+
+- Reclaim expired shared-dict entries in batches, so no single
+  `flush_expired()` call holds the dict lock for a whole backlog, and let
+  callers take the reclamation over with the `auto_flush_expired` option and
+  `prometheus:flush_expired()` (#23).
+- Reuse the index slots of metrics that have expired instead of handing out a
+  new number every time, and stop broadcasting to the other workers when a
+  metric comes back: renewing a metric no longer reads the shared counters, so
+  one low-frequency series returning no longer sends every worker through a
+  full sync on its request path (apache/apisix#13658). The slot count now
+  tracks the number of live series instead of growing for ever under label
+  churn. Design and measurements: api7/rfcs#290.
+
 ## 1.0.0
 
 Starting with this release, this fork (`nginx-lua-prometheus-api7`) uses

README.md

@@ -112,6 +112,13 @@ section of nginx configuration.
     RAM, you might want to increase this to avoid cache getting flushed too
     often. Decreasing this makes sense if you have a very large number of
     metrics or need to minimize memory usage of this library.
+  * `remove_expired_keys_interval` (number): how often, in seconds, each worker
+    drops the metrics that have expired and reclaims the shared dict entries
+    they left behind. Defaults to 3600, which is also the maximum.
+  * `auto_flush_expired` (boolean): whether that reclamation also happens on
+    the interval above, in every worker. Defaults to true. Set it to false to
+    schedule [`prometheus:flush_expired()`](#prometheusflush_expired) yourself,
+    for instance from a single process.
 
 Returns a `prometheus` object that should be used to register metrics.
 
@@ -236,6 +243,28 @@ location /metrics {
 
 Returns metric data as an array of strings.
 
+### prometheus:flush_expired()
+
+**syntax:** prometheus:flush_expired()
+
+Reclaims the shared dict entries left behind by metrics that have expired, and
+returns how many were reclaimed.
+
+Expired entries are only logically gone: every shared dict API reports them as
+missing, but they keep holding their memory until something reclaims them. The
+passive reclamation nginx performs on writes cannot do it, because it stops at
+the first entry that has not expired, and a metric registered without an
+`exptime` inevitably ends up sitting there.
+
+This is done for you on `remove_expired_keys_interval`, in every worker, unless
+`auto_flush_expired` is set to false. Call this instead if you would rather
+reclaim from a single process, or on your own schedule.
+
+The reclamation is issued in batches, because the underlying
+`ngx.shared.DICT:flush_expired()` holds the dict lock until it returns: a
+bounded number of entries is reclaimed per call, and the other workers get the
+lock back in between.
+
 ### counter:inc()
 
 **syntax:** counter:inc(*value*, *label_values*)

prometheus.lua

@@ -746,16 +746,19 @@ function Prometheus.init(dict_name, options_or_prefix)
         and options_or_prefix.remove_expired_keys_interval < MAX_REMOVE_EXPIRED_KEYS_INTERVAL
         and options_or_prefix.remove_expired_keys_interval
       or MAX_REMOVE_EXPIRED_KEYS_INTERVAL
+    self.auto_flush_expired = options_or_prefix.auto_flush_expired ~= false
   else
     self.prefix = options_or_prefix or ''
     self.error_metric_name = DEFAULT_ERROR_METRIC_NAME
     self.sync_interval = DEFAULT_SYNC_INTERVAL
     self.lookup_max_size = DEFAULT_LOOKUP_MAX_SIZE
     self.remove_expired_keys_interval = MAX_REMOVE_EXPIRED_KEYS_INTERVAL
+    self.auto_flush_expired = true
   end
 
   self.registry = {}
-  self.key_index = key_index_lib.new(self.dict, KEY_INDEX_PREFIX, self.remove_expired_keys_interval)
+  self.key_index = key_index_lib.new(self.dict, KEY_INDEX_PREFIX,
+    self.remove_expired_keys_interval, self.auto_flush_expired)
 
   self.initialized = true
 
@@ -930,6 +933,20 @@ end
 -- Returns:
 --   Array of strings with all metrics in a text format compatible with
 --   Prometheus.
+-- Reclaims the expired entries of the metrics shared dict, in batches, and
+-- returns how many were reclaimed.
+--
+-- Only needed by callers that set auto_flush_expired to false and schedule the
+-- reclamation themselves, in a single process instead of in every worker.
+function Prometheus:flush_expired()
+  if not self.initialized then
+    ngx.log(ngx.ERR, "Prometheus module has not been initialized")
+    return 0
+  end
+
+  return self.key_index:flush_expired()
+end
+
 function Prometheus:metric_data()
   if not self.initialized then
     ngx.log(ngx.ERR, "Prometheus module has not been initialized")