[16.0][IMP] queue_job: defer job execution across rolling deployments and module installs

queue_job/README.rst

@@ -143,6 +143,22 @@ Configuration
 
 * Jobs that remain in `enqueued` or `started` state (because, for instance, their worker has been killed) will be automatically re-queued.
 
+* By default, a worker will defer (not execute) a job if its own installed
+  module code no longer matches what the database expects (e.g. it hasn't yet
+  picked up a rolling deployment's update). Disable with the
+  ``queue_job.check_modules_up_to_date`` system parameter set to ``False``.
+
+* By default, a worker will also defer any job while modules are being
+  installed/upgraded/removed anywhere in the cluster (the same condition core
+  Odoo already uses to skip scheduled actions, see ``ir_cron``), since a job
+  committed early in an install (e.g. from a ``post_init_hook``) may reference
+  code that doesn't match the database until the install fully completes.
+  Pending states older than ``queue_job.install_in_progress_timeout_minutes``
+  (default ``60``) are treated as abandoned and ignored, with a warning
+  logged, rather than blocking jobs forever. Disable the check entirely with
+  the ``queue_job.check_install_in_progress`` system parameter set to
+  ``False``.
+
 Usage
 =====
 

queue_job/controllers/main.py

@@ -14,6 +14,7 @@
 from werkzeug.exceptions import BadRequest, Forbidden
 
 from odoo import SUPERUSER_ID, _, api, http, tools
+from odoo.modules.module import get_manifest
 from odoo.service.model import PG_CONCURRENCY_ERRORS_TO_RETRY
 from odoo.tools import config
 
@@ -27,6 +28,15 @@
 
 DEPENDS_MAX_TRIES_ON_CONCURRENCY_FAILURE = 5
 
+MODULES_OUTDATED_POSTPONE_SECONDS = 15
+INSTALL_IN_PROGRESS_POSTPONE_SECONDS = 15
+
+# How long a "is this worker's module code up to date" verdict is cached before
+# being recomputed, per database (a single process can serve several). Keeps a
+# busy jobrunner from re-querying ir.module.module on every single job dispatch.
+MODULES_UP_TO_DATE_CACHE_SECONDS = 5
+_modules_up_to_date_cache = {}  # dbname -> (checked_at, outdated: bool)
+
 
 @contextmanager
 def _prevent_commit(cr):
@@ -110,6 +120,133 @@ def _try_perform_job(cls, env, job):
             env.cr.commit()
         _logger.debug("%s done", job)
 
+    @classmethod
+    def _install_in_progress(cls, env):
+        """True while any module is marked to be installed/upgraded/removed.
+
+        Same guard core already applies to cron jobs
+        (ir_cron._check_modules_state): while an install/upgrade is running
+        anywhere in the cluster, module states sit at 'to install'/'to
+        upgrade'/'to remove' in the database, and only flip back once the
+        operation completes (or core's reset_modules_state() clears them on
+        failure). Jobs executed during that window may run with code that
+        does not yet match what the database will end up recording -- a
+        module's own 'installed' state and latest_version are only committed
+        near the end of its install, but data/hooks that ran earlier in the
+        same install (e.g. a post_init_hook enqueuing a job) may already be
+        committed and dispatchable. _worker_is_outdated() has nothing to
+        compare against in that window (the module isn't even marked
+        installed yet); this check covers exactly that gap.
+
+        Deliberately not cached like _worker_is_outdated(): the whole point
+        is that a job committed by a post_init_hook becomes visible in the
+        same commit as the pending module state, so a fresh read makes this
+        check exact. A TTL would reopen the race for its duration. The query
+        is one cheap aggregate over ir_module_module.
+
+        Pending states older than
+        queue_job.install_in_progress_timeout_minutes (default 60) are
+        treated as abandoned (e.g. modules left marked by a crashed/killed
+        process) and ignored, with a warning logged, rather than freezing
+        job processing forever. Disable the whole check with the
+        queue_job.check_install_in_progress system parameter.
+        """
+        icp = env["ir.config_parameter"].sudo()
+        if icp.get_param("queue_job.check_install_in_progress", "True") == "False":
+            return False
+
+        timeout_minutes = int(
+            icp.get_param("queue_job.install_in_progress_timeout_minutes", "60")
+        )
+        env.cr.execute(
+            """
+            SELECT COUNT(*),
+                   MAX(COALESCE(write_date, create_date))
+                       >= (now() AT TIME ZONE 'UTC') - %s * interval '1 minute'
+            FROM ir_module_module
+            WHERE state IN ('to install', 'to upgrade', 'to remove')
+            """,
+            (timeout_minutes,),
+        )
+        pending, fresh = env.cr.fetchone()
+        if not pending:
+            return False
+        if not fresh:
+            _logger.warning(
+                "%d module(s) have been marked to install/upgrade/remove for "
+                "more than %d minutes; assuming an abandoned operation and "
+                "resuming job processing",
+                pending,
+                timeout_minutes,
+            )
+            return False
+        # Only reached while an install is ACTIVELY in progress (pending &
+        # fresh) -- the common no-install case already returned False above.
+        # Bust the version-check cache so the first job dispatched after the
+        # install completes recomputes _worker_is_outdated() freshly instead
+        # of reusing a pre-install verdict for up to
+        # MODULES_UP_TO_DATE_CACHE_SECONDS.
+        _modules_up_to_date_cache.pop(env.cr.dbname, None)
+        return True
+
+    @classmethod
+    def _worker_is_outdated(cls, env):
+        """True if this worker's own on-disk module code no longer matches
+        what the database considers the currently installed version, for at
+        least one installed module.
+
+        ir.module.module.installed_version is a non-stored compute that reads
+        the manifest fresh from *this process's* addons_path (get_manifest()
+        is lru_cached per process, so this is cheap after the first call).
+        ir.module.module.latest_version is what the module loader persisted to
+        the database the last time this module was actually
+        installed/upgraded (see odoo/modules/loading.py). A mismatch means
+        some other process has since upgraded this module past what this
+        worker is currently running -- most commonly, one worker in a
+        multi-process or multi-host deployment has picked up new code and run
+        an upgrade, while this one is still serving requests with the old
+        code, in the window before it gets restarted (a rolling-deployment
+        race). Running a job in that state risks acting on stale files/logic
+        that no longer match the database.
+
+        The result is cached briefly, per database, so a busy queue doesn't
+        re-run this on every single job (see MODULES_UP_TO_DATE_CACHE_SECONDS).
+
+        Disable via the ``queue_job.check_modules_up_to_date`` system
+        parameter (set to ``"False"``); enabled by default.
+        """
+        icp = env["ir.config_parameter"].sudo()
+        if icp.get_param("queue_job.check_modules_up_to_date", "True") == "False":
+            return False
+
+        dbname = env.cr.dbname
+        now = time.monotonic()
+        cached = _modules_up_to_date_cache.get(dbname)
+        if cached and now - cached[0] < MODULES_UP_TO_DATE_CACHE_SECONDS:
+            return cached[1]
+
+        outdated = False
+        modules = env["ir.module.module"].sudo().search([("state", "=", "installed")])
+        for module in modules:
+            manifest = get_manifest(module.name)
+            if not manifest:
+                # Can't read this module's manifest from this process's
+                # addons_path (e.g. a module removed from disk but still
+                # marked installed in the database) -- skip it rather than
+                # treat an unreadable manifest as evidence of staleness.
+                continue
+            on_disk_version = manifest.get("version")
+            if (
+                on_disk_version
+                and module.latest_version
+                and on_disk_version != module.latest_version
+            ):
+                outdated = True
+                break
+
+        _modules_up_to_date_cache[dbname] = (now, outdated)
+        return outdated
+
     @classmethod
     def _enqueue_dependent_jobs(cls, env, job):
         tries = 0
@@ -150,6 +287,37 @@ def retry_postpone(job, message, seconds=None):
                 job.set_pending(reset_retry=False)
                 job.store()
 
+        if cls._install_in_progress(env):
+            _logger.info(
+                "%s: modules are being installed/upgraded somewhere in the "
+                "cluster; postponing %ss",
+                job,
+                INSTALL_IN_PROGRESS_POSTPONE_SECONDS,
+            )
+            retry_postpone(
+                job,
+                "modules install in progress",
+                seconds=INSTALL_IN_PROGRESS_POSTPONE_SECONDS,
+            )
+            if not config["test_enable"]:
+                env.cr.rollback()
+            return
+
+        if cls._worker_is_outdated(env):
+            _logger.info(
+                "%s: this worker's module code is outdated relative to the "
+                "database (likely a rolling deployment in progress); "
+                "postponing %ss",
+                job,
+                MODULES_OUTDATED_POSTPONE_SECONDS,
+            )
+            retry_postpone(
+                job, "worker outdated", seconds=MODULES_OUTDATED_POSTPONE_SECONDS
+            )
+            if not config["test_enable"]:
+                env.cr.rollback()
+            return
+
         try:
             try:
                 cls._try_perform_job(env, job)

queue_job/readme/CONFIGURE.rst

@@ -48,3 +48,19 @@
        of running Odoo is obviously not for production purposes.
 
 * Jobs that remain in `enqueued` or `started` state (because, for instance, their worker has been killed) will be automatically re-queued.
+
+* By default, a worker will defer (not execute) a job if its own installed
+  module code no longer matches what the database expects (e.g. it hasn't yet
+  picked up a rolling deployment's update). Disable with the
+  ``queue_job.check_modules_up_to_date`` system parameter set to ``False``.
+
+* By default, a worker will also defer any job while modules are being
+  installed/upgraded/removed anywhere in the cluster (the same condition core
+  Odoo already uses to skip scheduled actions, see ``ir_cron``), since a job
+  committed early in an install (e.g. from a ``post_init_hook``) may reference
+  code that doesn't match the database until the install fully completes.
+  Pending states older than ``queue_job.install_in_progress_timeout_minutes``
+  (default ``60``) are treated as abandoned and ignored, with a warning
+  logged, rather than blocking jobs forever. Disable the check entirely with
+  the ``queue_job.check_install_in_progress`` system parameter set to
+  ``False``.

queue_job/static/description/index.html

@@ -503,6 +503,20 @@ <h2><a class="toc-backref" href="#toc-entry-2">Configuration</a></h2>
 </table>
 <ul class="simple">
 <li>Jobs that remain in <cite>enqueued</cite> or <cite>started</cite> state (because, for instance, their worker has been killed) will be automatically re-queued.</li>
+<li>By default, a worker will defer (not execute) a job if its own installed
+module code no longer matches what the database expects (e.g. it hasn’t yet
+picked up a rolling deployment’s update). Disable with the
+<tt class="docutils literal">queue_job.check_modules_up_to_date</tt> system parameter set to <tt class="docutils literal">False</tt>.</li>
+<li>By default, a worker will also defer any job while modules are being
+installed/upgraded/removed anywhere in the cluster (the same condition core
+Odoo already uses to skip scheduled actions, see <tt class="docutils literal">ir_cron</tt>), since a job
+committed early in an install (e.g. from a <tt class="docutils literal">post_init_hook</tt>) may reference
+code that doesn’t match the database until the install fully completes.
+Pending states older than <tt class="docutils literal">queue_job.install_in_progress_timeout_minutes</tt>
+(default <tt class="docutils literal">60</tt>) are treated as abandoned and ignored, with a warning
+logged, rather than blocking jobs forever. Disable the check entirely with
+the <tt class="docutils literal">queue_job.check_install_in_progress</tt> system parameter set to
+<tt class="docutils literal">False</tt>.</li>
 </ul>
 </div>
 <div class="section" id="usage">