Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -91,8 +91,9 @@ jobs:
run: |
dotnet build --configuration Debug --property WarningLevel=0 api-validator-dotnet
pip3 install -r python-examples/${{ matrix.project_name }}/requirements_ci.txt
pip3 install ./testit-python-commons
pip3 install ./${{ matrix.adapter_name }}
pip3 install -e ./testit-python-commons
pip3 install -e ./${{ matrix.adapter_name }}
python3 -c "from adapters_api.model.custom_attribute_api_result import CustomAttributeApiResult; assert 'code' in CustomAttributeApiResult.openapi_types"

- name: Test
run: |
Expand Down Expand Up @@ -129,8 +130,9 @@ jobs:
run: |
dotnet build --configuration Debug --property WarningLevel=0 api-validator-dotnet
pip3 install -r python-examples/${{ matrix.project_name }}/requirements_ci.txt
pip3 install ./testit-python-commons
pip3 install ./${{ matrix.adapter_name }}
pip3 install -e ./testit-python-commons
pip3 install -e ./${{ matrix.adapter_name }}
python3 -c "from adapters_api.model.custom_attribute_api_result import CustomAttributeApiResult; assert 'code' in CustomAttributeApiResult.openapi_types"


- name: Test import realtime false
Expand Down
228 changes: 228 additions & 0 deletions docs/tz-autotest-layer.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,228 @@
# Technical assignment: autotest layer (test pyramid)

## Context

TMS supports a **test pyramid layer** on an **autotest** card. The layer can be assigned manually, by rules, or from automation runs.

Adapters can set the layer when creating or updating an autotest during a test run. The layer source in API is **`Run`** (set by the adapter).

## Business need

As an automation engineer, I want to declare the pyramid level of each autotest **in test code** so that after a run the autotest card shows:

- the layer name (e.g. `API`, `E2E`);
- the source **Run** (adapter), distinct from Manual / Rule / Report.

## Goal

In every adapter that creates or updates autotests, support an optional **layer** metadata field:

- read from test code only (annotation / mark / tag — per language);
- send on **create** and **update** autotest API calls when present;
- **omit** the field when the user did not specify a layer.

## Scope

Apply consistently to all Test IT adapters (any language / framework) and any entry point that upserts autotests via the standard adapters API.

**In scope:** autotest `layer` on create/update.

**Out of scope:**

- layer on **test run** entity;
- layer on **test result** as a separate API field (UI may show layer from the linked autotest);
- configuration / env / CLI default layer for a whole run;
- runtime API to set layer during test execution;
- validating layer names against a fixed whitelist (adapters pass the string as-is).

## API contract

Use existing autotest endpoints and models:

```json
{
"layer": {
"name": "API",
"source": "Run"
}
}
```

| Operation | Behaviour |
|-----------|-----------|
| **Create autotest** | Include `layer` only when the test declares a non-empty layer |
| **Update autotest** | Always send `resetLayer: false`; include `layer` only when the test declares a non-empty layer |
| **No layer in test** | Send `resetLayer: false`; do **not** send `layer` |

Recommended layer names (constants in client libraries, not enforced): `E2E`, `UI`, `API`, `Contract`, `Integration`, `Component`, `Unit`.

Any other non-empty string is valid.

`LayerSource` values: `Manual`, `Report`, `Run`, `Rule`. Adapters always use **`Run`**.

## Functional requirements

### 1. Declaration in test code only

The user sets layer on the **test method / scenario**, not via adapter config.

| Stack | Suggested syntax |
|-------|------------------|
| Java (JUnit / TestNG) | `@Layer("API")` or `@Layer(TestLayers.API)` on test method |
| Python (pytest) | `@pytest.mark.layer("api")` |
| C# (NUnit / xUnit) | `[Layer("API")]` on test method |
| JavaScript (Jest / Mocha) | `@layer('API')` or framework-specific decorator |
| Cucumber (Gherkin) | `@Layer=API` on scenario |
| JBehave | Meta: `@Layer API` on scenario |

Parameter substitution (e.g. `{param}` in Java) should work if the framework already supports it for other annotations.

### 2. Mapping flow

```
test code (annotation / tag)
→ internal test result model (optional layer field)
→ Converter / mapper
→ AutoTestCreateApiModel.layer / AutoTestUpdateApiModel.layer
→ TMS
```

### 3. Independence from other metadata

- Layer ≠ labels / tags on autotest.
- Layer ≠ test run tags.
- Do not change existing labels/tags behaviour.

### 4. Failed-test update path

Some adapters send a minimal update when a test fails (copy existing autotest from TMS). Still apply `layer` from the test annotation when present.

## Non-goals

- Default layer for all tests in a run via config.
- `Adapter.addLayer()` or similar dynamic API.
- Resetting layer when annotation is absent (`resetLayer: true`).
- OpenAPI / client regeneration in adapter repos (use existing generated `LayerApiModel`).

## Acceptance criteria

1. User adds layer in test code → autotest in TMS shows that layer with source **Run** after create/update.
2. User omits layer → adapter does not send `layer` on create/update.
3. Custom layer string is accepted without validation.
4. Recommended constants documented; arbitrary strings work.
5. Cross-language doc describes the same rules with per-language syntax examples.
6. Unit tests: with layer / without layer / custom string / failed-update path.

## Suggested implementation outline (language-agnostic)

1. Add `@Layer` (or equivalent) in shared annotations package.
2. Add optional `layer` field to internal `TestResult` (or equivalent).
3. Extract layer from test method in each framework listener.
4. In create/update mapper: if layer non-empty → `LayerApiModel { name, source: Run }`; on update always send `resetLayer: false`.
5. BDD adapters: parse `@Layer=` / meta `Layer` like other scenario tags.
6. Document in adapter README; add samples and tests.

## Example (Java)

```java
import ru.testit.annotations.Layer;
import ru.testit.models.TestLayers;

@Layer(TestLayers.API)
@Test
void createUser() {
// ...
}

@Layer("my-custom-layer")
@Test
void customLayer() {
// ...
}
```

## Example (pytest)

```python
import pytest
from testit import layer, TestLayers

@pytest.mark.layer("api")
def test_create_user_with_mark():
...

@layer(TestLayers.API)
def test_create_user_with_decorator():
...

@layer("my-custom-layer")
def test_with_custom_layer():
...
```

---

## Java adapters status

Implemented in `testit-java-commons` (shared by all Java adapters):

- `@Layer` on test method (`ru.testit.annotations.Layer`)
- `TestLayers` constants (`ru.testit.models.TestLayers`)
- `TestResult.layer`, `Utils.extractLayer`
- `Converter`: `layer` + `LayerSource.RUN` on create when set; on update always `resetLayer: false`, `layer` only when set
- Framework listeners: JUnit 4/5, TestNG, Cucumber 4–7 (`@Layer=`), JBehave (meta `Layer`)

## Python adapters status

Implemented in `testit-python-commons` (shared by pytest / behave / nose / robot):

- Decorator: `@layer("API")` or `@layer(TestLayers.API)` (`testit.layer`, `testit.TestLayers`)
- `TestResult.set_layer` / `get_layer`
- `Converter.layer_to_api_model`: `LayerApiModel(name, source=Run)` on create when set; on update always `resetLayer: false`, `layer` only when set
- No config/env/CLI layer; no layer on test run or test result

### Syntax by adapter

| Adapter | How to declare layer in test code |
|---------|-----------------------------------|
| pytest | `@pytest.mark.layer("API")` or `@layer(TestLayers.API)` from `testit` |
| behave | `@Layer=API` on scenario/feature |
| nose (nose2) | `@layer("API")` from `testit` on test function |
| robotframework | `[Tags] testit.layer:API` |

### Recommended constants

```python
from testit import TestLayers

TestLayers.E2E
TestLayers.UI
TestLayers.API
TestLayers.CONTRACT
TestLayers.INTEGRATION
TestLayers.COMPONENT
TestLayers.UNIT
```

Any other non-empty string is accepted without validation.

### API mapping rules

| Scenario | Adapter behaviour |
|----------|-------------------|
| Layer set in test | create: send `layer: { name, source: Run }` |
| Layer set in test | update: send `layer` + `resetLayer: false` |
| Layer not set | send `resetLayer: false`; do not send `layer` |

### Cross-language implementation guide

When porting to another language, keep the same contract:

1. Read layer **only from test code** (annotation / mark / tag).
2. Store it in the internal test result model as an optional string.
3. On autotest **create**, include `layer` only when non-empty.
4. On autotest **update**, always send `resetLayer: false`; include `layer` only when non-empty.
5. Always set `source` to `Run`.
6. Do not validate layer name against a whitelist; document recommended constants only.
7. Do not add config/env/CLI defaults for run-level layer.
8. Do not send layer on test run or test result entities.
2 changes: 1 addition & 1 deletion testit-adapter-behave/setup.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
from setuptools import find_packages, setup

VERSION = "5.1.6"
VERSION = "5.2.0"

setup(
name='testit-adapter-behave',
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,4 @@ class TagType:
TAGS = 'Tags='
NAMESPACE = 'NameSpace='
CLASSNAME = 'ClassName='
LAYER = 'Layer='
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ def parse_scenario(scenario):
'links': [],
'labels': [],
'tags': [],
'layer': None,
'workItemsID': [],
"externalKey": get_scenario_name(scenario)
# TODO: Add to python-commons
Expand Down Expand Up @@ -89,6 +90,9 @@ def parse_scenario(scenario):
# TODO: Fix in python-commons to "workItemIds"
executable_test['workItemsID'] = tags[TagType.WORK_ITEM_IDS]

if TagType.LAYER in tags and tags[TagType.LAYER]:
executable_test['layer'] = tags[TagType.LAYER]

return executable_test


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ def parse_test_tags(tags):
TagType.TAGS: [],
TagType.NAMESPACE: None,
TagType.CLASSNAME: None,
TagType.LAYER: None,
}

for tag in tags:
Expand Down Expand Up @@ -52,6 +53,9 @@ def parse_test_tags(tags):
elif TagType.CLASSNAME in tag:
parsed_tags[TagType.CLASSNAME] = __parse_space_in_tag(tag[len(TagType.CLASSNAME):])

elif TagType.LAYER in tag:
parsed_tags[TagType.LAYER] = __parse_space_in_tag(tag[len(TagType.LAYER):])

return parsed_tags


Expand Down
1 change: 1 addition & 0 deletions testit-adapter-behave/src/testit_adapter_behave/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,7 @@ def convert_executable_test_to_test_result_model(executable_test: dict) -> TestR
.set_result_links(executable_test['resultLinks'])\
.set_labels(executable_test['labels']) \
.set_tags(executable_test['tags'])\
.set_layer(executable_test.get('layer'))\
.set_work_item_ids(executable_test['workItemsID'])\
.set_message(executable_test['message'])\
.set_external_key(executable_test['externalKey'])
Expand Down
8 changes: 8 additions & 0 deletions testit-adapter-behave/tests/test_behave_layer.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
from testit_adapter_behave.models.tags import TagType
from testit_adapter_behave.tags_parser import parse_test_tags


def test_parse_layer_tag():
parsed = parse_test_tags(['Layer=API'])

assert parsed[TagType.LAYER] == 'API'
2 changes: 1 addition & 1 deletion testit-adapter-nose/setup.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
from setuptools import setup, find_packages

VERSION = "5.1.6"
VERSION = "5.2.0"

setup(
name='testit-adapter-nose',
Expand Down
12 changes: 12 additions & 0 deletions testit-adapter-nose/src/testit_adapter_nose/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@ def form_test(item, top_level_directory):
'links': __get_links_from(item, function),
'labels': __get_labels_from(item, function),
'tags': __get_tags_from(item, function),
'layer': __get_layer_from(item, function),
'workItemsID': __get_work_item_ids_from(item, function),
'message': None,
'externalKey': __get_fullname(function, top_level_directory)
Expand Down Expand Up @@ -242,6 +243,16 @@ def __get_tags_from(item, function) -> List[str]:
return tags


def __get_layer_from(item, function):
test_layer = __search_attribute(function, 'test_layer')
if not test_layer:
return None

return str(collect_parameters_in_string_attribute(
test_layer,
get_all_parameters(item)))


def __get_work_item_ids_from(item, function):
test_workitems_id = __search_attribute(function, 'test_workitems_id')

Expand Down Expand Up @@ -405,6 +416,7 @@ def convert_executable_test_to_test_result_model(executable_test: dict) -> TestR
.set_result_links(executable_test['resultLinks'])\
.set_labels(executable_test['labels'])\
.set_tags(executable_test['tags'])\
.set_layer(executable_test.get('layer'))\
.set_work_item_ids(executable_test['workItemsID'])\
.set_message(executable_test['message'])\
.set_external_key(executable_test['externalKey'])
2 changes: 1 addition & 1 deletion testit-adapter-pytest/setup.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
from setuptools import find_packages, setup

VERSION = "5.1.6"
VERSION = "5.2.0"

setup(
name='testit-adapter-pytest',
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,5 +29,6 @@ class ExecutableTest:
labels = attrib(default=Factory(list))
tags = attrib(default=Factory(list))
work_item_ids = attrib(default=Factory(list))
layer = attrib(default=None)
message = attrib(default=None)
node_id = attrib(default=None)
Loading
Loading