fix: TMS-41492 mode-split Sync finalize and parametrize bulk skip

docs/mode0-duplicate-results-bulk-after-realtime.md

@@ -29,14 +29,17 @@ Root cause: the adapter finalized the test twice — once at **test finish**, ag
 
 When Sync Storage accepts the cut (`on_master_no_already_in_progress` → true):
 
-1. Cut model to Sync Storage (coordination only).
-2. `_write_test_realtime_internal` → autotest update + **`sendTestResults`** with final status and full steps.
-3. Store `externalId → resultId` in `AdapterManager.__test_result_map`.
+1. Cut model to Sync Storage (final status in the cut for Work X).
+2. **Mode 0 only:** `_write_test_realtime_internal` → autotest update + **`sendTestResults`** with **final** status and full steps.
+3. Store `externalId → resultId` in `AdapterManager.__test_result_map` (fixtures).
+4. Store per-invocation finalize key in `__finalized_result_keys` (`externalKey` or `externalId`+parameters).
+
+**Modes 1 / 2 (not this doc’s primary path):** same cut, but TMS gets **`InProgress`** first; **Work X** applies the final status from the cut. Early adapter finalization above is **mode 0** only.
 
 ### End of run — bulk (`write_tests_after_all`)
 
-- If `externalId` is already in `__test_result_map` → **skip** `sendTestResults`; refresh autotest metadata if needed.
-- Otherwise → bulk `sendTestResults` once (Sync Storage off, or test not sent at finish).
+- If finalize key is already in `__finalized_result_keys` → **skip** `sendTestResults`; refresh autotest metadata if needed.
+- Otherwise → bulk `sendTestResults` (including other parametrize iterations that share the same `externalId`).
 
 ---
 
@@ -48,7 +51,7 @@ stopTestCase → SyncStorage cut + sendTestResults (Passed/Failed + full payload
 sessionfinish → bulk skips sendTestResults for tests already in __test_result_map
 ```
 
-Expected: **one** finalized result row per autotest in the run, with steps from the create payload.
+Expected: **one** finalized result row per **test invocation** in the run (parametrize iterations are separate), with steps from the create payload.
 
 ---
 

docs/test-result-export-contract.md

@@ -51,10 +51,13 @@ Unit test: `tests/client/test_converter_update_test_results.py`.
 
 ### 4. Bulk at run end must not double-send
 
-When `importRealtime=false` and a test was already finalized at test finish (e.g. Sync Storage master path stores `externalId → resultId` in `AdapterManager.__test_result_map`):
+When `importRealtime=false` and a test was already finalized at test finish (e.g. Sync Storage master path):
 
-- `write_tests(..., finalized_external_ids=...)` **skips** `sendTestResults` for those external IDs
-- Optionally refreshes autotest metadata only
+- Store `externalId → resultId` in `__test_result_map` (for fixture attach)
+- Store a **per-invocation** key in `__finalized_result_keys` (`externalKey` / pytest node id, else `externalId` + parameters)
+- `write_tests(..., finalized_result_keys=...)` **skips** `sendTestResults` only for that invocation
+
+Do **not** skip by bare `externalId`: parametrize without `{param}` in `@externalId` shares one id; other iterations must still be sent.
 
 Info log:
 

testit-adapter-behave/setup.py

@@ -1,6 +1,6 @@
 from setuptools import find_packages, setup
 
-VERSION = "5.2.1"
+VERSION = "5.2.2"
 
 setup(
     name='testit-adapter-behave',

testit-adapter-nose/setup.py

@@ -1,6 +1,6 @@
 from setuptools import setup, find_packages
 
-VERSION = "5.2.1"
+VERSION = "5.2.2"
 
 setup(
     name='testit-adapter-nose',

testit-adapter-pytest/setup.py

@@ -1,6 +1,6 @@
 from setuptools import find_packages, setup
 
-VERSION = "5.2.1"
+VERSION = "5.2.2"
 
 setup(
     name='testit-adapter-pytest',

testit-adapter-robotframework/setup.py

@@ -1,6 +1,6 @@
 from setuptools import find_packages, setup
 
-VERSION = "5.2.1"
+VERSION = "5.2.2"
 
 setup(
     name='testit-adapter-robotframework',

testit-python-commons/realtime-import-spec.md

@@ -115,11 +115,13 @@ The `step_results` field is omitted, so the nested test steps written during rea
 
 When Sync Storage is running and the worker is **master**, the **first** completed test uses `on_master_no_already_in_progress`:
 
-1. Adapter sends **`TestResultCutApiModel`** to Sync Storage (`POST /in_progress_test_result`) — coordination only, no steps.
-2. Adapter calls `_write_test_realtime_internal` with the **final** outcome → **`sendTestResults`** with full step tree (no forced InProgress POST to TMS).
-3. Stores `externalId → resultId` in `__test_result_map`.
+1. Adapter sends **`TestResultCutApiModel`** to Sync Storage (`POST /in_progress_test_result`) with the **final** cut status — coordination for Work X.
+2. TMS export depends on **`adapterMode`**:
+   - **Mode 0 (`USE_FILTER`):** `_write_test_realtime_internal` with the **final** outcome → **`sendTestResults`** (TP-bound / orphan fix). Work X often has nothing left to finalize.
+   - **Modes 1 / 2:** temporarily export **`InProgress`** via `sendTestResults`, then restore the final outcome on the in-memory model. **Work X** finalizes from the cut at `wait-completion`.
+3. Stores `externalId → resultId` in `__test_result_map` and a per-invocation key in `__finalized_result_keys`.
 
-Sync Storage Work X may still finalize the held cut model separately at `wait-completion`; that path is independent of the adapter POST/PUT contract. See Sync Storage docs if nested steps on the **first** test are lost after run completion.
+See Sync Storage docs if nested steps on the **first** test are lost after run completion (Work X path).
 
 ---
 

testit-python-commons/setup.py

@@ -1,6 +1,6 @@
 from setuptools import find_packages, setup
 
-VERSION = "5.2.1"
+VERSION = "5.2.2"
 
 setup(
     name='testit-python-commons',

testit-python-commons/src/testit_python_commons/client/api_client.py

@@ -266,9 +266,9 @@ def write_tests(
             self,
             test_results: List[TestResult],
             fixture_containers: dict,
-            finalized_external_ids: set = None) -> None:
+            finalized_result_keys: set = None) -> None:
         logging.debug("call __write_tests")
-        finalized_external_ids = finalized_external_ids or set()
+        finalized_result_keys = finalized_result_keys or set()
         bulk_autotest_helper = BulkAutotestHelper(self.__autotest_api, self.__test_run_api, self.__config)
         create_count = 0
         update_count = 0
@@ -278,8 +278,11 @@ def write_tests(
         for test_result in test_results:
             test_result = self.__add_fixtures_to_test_result(test_result, fixture_containers)
             external_id = test_result.get_external_id()
+            finalize_key = test_result.get_finalize_key()
 
-            if external_id in finalized_external_ids:
+            # Skip only the same invocation (Java: by UUID). Bare externalId would
+            # drop other parametrize iterations that share one autotest id.
+            if finalize_key in finalized_result_keys:
                 skip_count += 1
                 logging.info(
                     'Bulk import: skip sendTestResults for %s (already finalized at test finish)',

testit-python-commons/src/testit_python_commons/client/converter.py

@@ -45,9 +45,19 @@
 from testit_python_commons.models.test_result import TestResult
 from testit_python_commons.models.test_result_with_all_fixture_step_results_model import TestResultWithAllFixtureStepResults
 from testit_python_commons.services.logger import adapter_logger
+from testit_python_commons.utils.html_escape_utils import HtmlEscapeUtils
 
 
 class Converter:
+    @staticmethod
+    def _escape_string_map(values: Optional[dict]) -> Optional[dict]:
+        if not values:
+            return values
+        return {
+            str(key): HtmlEscapeUtils.escape_html_tags(str(value)) if value is not None else value
+            for key, value in values.items()
+        }
+
     @staticmethod
     @adapter_logger
     def test_run_to_test_run_short_model(
@@ -402,8 +412,8 @@ def test_result_to_testrun_result_post_model(
                 test_result.get_teardown_results()),
             traces=test_result.get_traces(),
             attachments=test_result.get_attachments(),
-            parameters=test_result.get_parameters(),
-            properties=test_result.get_properties(),
+            parameters=cls._escape_string_map(test_result.get_parameters()),
+            properties=cls._escape_string_map(test_result.get_properties()),
             links=cls.links_to_links_post_model(
                 test_result.get_result_links()),
             duration=round(test_result.get_duration()),

testit-python-commons/src/testit_python_commons/models/test_result.py

@@ -1,3 +1,4 @@
+import json
 from typing import List
 
 from testit_python_commons.models.link import Link
@@ -297,6 +298,19 @@ def set_external_key(self, external_key: str):
     def get_external_key(self) -> str:
         return self.__external_key
 
+    def get_finalize_key(self) -> str:
+        """
+        Identity of one test invocation for bulk dedupe (parametrize-safe).
+        Prefer externalKey (e.g. pytest node id); else externalId + parameters.
+        """
+        if self.__external_key:
+            return self.__external_key
+        params = self.__parameters or {}
+        return '{0}\n{1}'.format(
+            self.__external_id,
+            json.dumps(params, sort_keys=True, default=str, ensure_ascii=False),
+        )
+
     @adapter_logger
     def set_layer(self, layer: str):
         self.__layer = HtmlEscapeUtils.escape_html_tags(layer) if layer else None

testit-python-commons/src/testit_python_commons/services/adapter_manager.py

@@ -5,6 +5,7 @@
 from testit_python_commons.client.api_client import ApiClientWorker
 from testit_python_commons.client.client_configuration import ClientConfiguration
 from testit_python_commons.models.adapter_mode import AdapterMode
+from testit_python_commons.models.status_type import StatusType
 from testit_python_commons.models.test_result import TestResult
 from testit_python_commons.services.adapter_manager_configuration import (
     AdapterManagerConfiguration,
@@ -18,6 +19,7 @@
 )
 
 SYNC_STORAGE_AVAILABLE = True
+IN_PROGRESS_LITERAL = "InProgress"
 
 
 class AdapterManager:
@@ -33,6 +35,9 @@ def __init__(
         self.__api_client = ApiClientWorker(client_configuration)
         self.__fixture_manager = fixture_manager
         self.__test_result_map = {}
+        # Keys of results already sent at test finish (externalKey or externalId+params).
+        # Must not be bare externalId — parametrize shares one externalId.
+        self.__finalized_result_keys = set()
         self.__test_results = []
         self.__test_run_metadata_applied = False
 
@@ -146,23 +151,32 @@ def get_autotests_for_launch(self):
 
     @adapter_logger
     def write_test(self, test_result: TestResult) -> None:
-        if self.__config.should_import_realtime():
-            self.__write_test_realtime(test_result)
-            return
-
-        # for realtime false
-        # if
-        logging.debug("Is already in progress: " + str(self.__is_already_in_progress()))
-        # Handle Sync Storage integration if available
-        # Check if current worker is master and no test is in progress
-        if self.__is_active_syncstorage_instance() and self.__is_master_and_no_in_progress():
-            logging.debug(f"Outcome: {test_result.get_outcome()}")
-            is_ok = self.on_master_no_already_in_progress(test_result)
-            if is_ok:
+        try:
+            if self.__config.should_import_realtime():
+                self.__write_test_realtime(test_result)
                 return
-            # else continue normal processing
 
-        self.__test_results.append(test_result)
+            # for realtime false
+            # if
+            logging.debug("Is already in progress: " + str(self.__is_already_in_progress()))
+            # Handle Sync Storage integration if available
+            # Check if current worker is master and no test is in progress
+            if self.__is_active_syncstorage_instance() and self.__is_master_and_no_in_progress():
+                logging.debug(f"Outcome: {test_result.get_outcome()}")
+                is_ok = self.on_master_no_already_in_progress(test_result)
+                if is_ok:
+                    return
+                # else continue normal processing
+
+            self.__test_results.append(test_result)
+        except Exception as exc:
+            # Do not abort the pytest/session run if TMS rejects one result (e.g. XSS validation)
+            logging.error(
+                'Failed to write test result for external_id="%s", parameters=%s: %s',
+                test_result.get_external_id(),
+                test_result.get_parameters(),
+                exc,
+            )
 
 
     @adapter_logger
@@ -189,12 +203,31 @@ def on_master_no_already_in_progress(self, test_result: TestResult) -> bool:
         self.__sync_storage_runner.set_is_already_in_progress(True)
 
         try:
-            # Final status via sendTestResults (create); never change status via PUT.
+            # Mode 0: finalize in TMS now (TP-bound InProgress / orphan fix).
+            # Modes 1/2: post InProgress to TMS; Work X finalizes from the cut.
+            if self.__config.get_mode() == AdapterMode.USE_FILTER:
+                logging.debug(
+                    "SyncStorage accepted %s; sendTestResults with final status (mode 0)",
+                    test_result.get_external_id(),
+                )
+                self._write_test_realtime_internal(test_result)
+                return True
+
+            final_outcome = test_result.get_outcome()
+            final_status_type = test_result.get_status_type()
             logging.debug(
-                "SyncStorage accepted %s; sendTestResults with final status",
+                "SyncStorage accepted %s; sendTestResults InProgress (Work X will finalize)",
                 test_result.get_external_id(),
             )
-            self._write_test_realtime_internal(test_result)
+            try:
+                test_result.set_outcome(IN_PROGRESS_LITERAL)
+                test_result.set_status_type(StatusType.INPROGRESS)
+                self._write_test_realtime_internal(test_result)
+            finally:
+                if final_outcome is not None:
+                    test_result.set_outcome(final_outcome)
+                if final_status_type is not None:
+                    test_result.set_status_type(final_status_type)
             return True
         except Exception as e:
             logging.warning(
@@ -233,7 +266,9 @@ def _write_test_realtime_internal(self, test_result: TestResult) -> None:
             logging.warning(test_result)
             return
 
+        # Fixture attach (realtime) still indexes by externalId.
         self.__test_result_map[ext_id] = test_result_id
+        self.__finalized_result_keys.add(test_result.get_finalize_key())
 
     @adapter_logger
     def write_tests(self) -> None:
@@ -265,7 +300,7 @@ def __write_tests_after_all(self) -> None:
         self.__api_client.write_tests(
             self.__test_results,
             fixtures,
-            finalized_external_ids=set(self.__test_result_map.keys()),
+            finalized_result_keys=set(self.__finalized_result_keys),
         )
 
     @adapter_logger

testit-python-commons/src/testit_python_commons/utils/html_escape_utils.py

@@ -15,23 +15,18 @@ class HtmlEscapeUtils:
 
     NO_ESCAPE_HTML_ENV_VAR = "NO_ESCAPE_HTML"
 
-    # Regex pattern to detect HTML tags (requires at least one non-whitespace character after <)
-    # This ensures empty <> brackets are not considered HTML tags
-    _HTML_TAG_PATTERN = re.compile(r'<\S.*?(?:>|/>)')
-
-    # Regex patterns to escape only non-escaped characters
-    # Using negative lookbehind to avoid double escaping
-    _LESS_THAN_PATTERN = re.compile('<')
-    _GREATER_THAN_PATTERN = re.compile('>')
-
+    # Align with Java adapters: tag must start with letter, '!' or '/'
+    _HTML_TAG_PATTERN = re.compile(
+        r'''<[a-zA-Z!/][^<>"']*(?:"[^"]*"[^<>"']*|'[^']*'[^<>"']*)*>'''
+    )
+
     @staticmethod
     def escape_html_tags(text: Optional[str]) -> Optional[str]:
         """
         Escapes HTML tags to prevent XSS attacks.
         First checks if the string contains HTML tags using regex pattern.
         Only performs escaping if HTML tags are detected.
-        Escapes all < as \\< and > as \\> only if they are not already escaped.
-        Uses regex with negative lookbehind to avoid double escaping.
+        Escapes < as &lt; and > as &gt; (idempotent for already-escaped text).
 
         Args:
             text: The text to escape
@@ -51,11 +46,7 @@ def escape_html_tags(text: Optional[str]) -> Optional[str]:
         if not HtmlEscapeUtils._HTML_TAG_PATTERN.search(text):
             return text  # No HTML tags found, return original string
 
-        # Use regex with negative lookbehind to escape only non-escaped characters
-        result = HtmlEscapeUtils._LESS_THAN_PATTERN.sub('&lt;', text)
-        result = HtmlEscapeUtils._GREATER_THAN_PATTERN.sub('&gt;', result)
-
-        return result
+        return text.replace('<', '&lt;').replace('>', '&gt;')
 
     @staticmethod
     def escape_html_in_object(obj: Any) -> Any:
@@ -190,6 +181,11 @@ def _process_attribute_value(obj: Any, attr_name: str, value: Any) -> None:
             HtmlEscapeUtils._process_list(value)
         elif isinstance(value, dict):
             HtmlEscapeUtils._process_dict(value)
+            try:
+                # Write back so OpenAPI models that return dict copies keep escaped values
+                setattr(obj, attr_name, value)
+            except AttributeError:
+                pass
         elif value is not None and not HtmlEscapeUtils._is_simple_type(type(value)):
             # Process nested objects (but not simple types)
             HtmlEscapeUtils.escape_html_in_object(value)