Repository navigation
Client api test for external server. - #432
Conversation
samatstariongroup
left a comment
There was a problem hiding this comment.
Review: run the live tests against a local server, and line up with the Flexo suite of #423
Thanks for this. It calls every one of the 35 operations of the client, which is a good base. The PR builds, and it merges cleanly with the Flexo branch of #423 (407-feature-verify-rest-api-client-against-compliant-server); offline tests pass on both. The 40 live tests were skipped in my run, because SYSML2_API_BASE_ADDRESS was not set.
The two suites are meant to live side by side, so most of the points below are about fitting in with #423. Inline comments carry the details.
1. Do not depend on a server hosted on the internet; start the pilot implementation in containers
The tests look written against the SST-hosted pilot implementation, Systems-Modeling/SysML-v2-API-Services. The datatype ids https://www.omg.org/spec/SysML/20250201/... are exactly what the pilot returns. A shared, remote server makes the tests depend on its uptime and its existing data (see the inline comment on VerifyGetProjectsAsync), and the write tests add data to it.
#423 already starts Flexo locally with Testcontainers (SySML2.NET.REST.Tests/Helpers/FlexoTestHost.cs). I checked that the pilot can be run the same way:
| Check | Result |
|---|---|
| Image | No official image; built from source (release 2026-04, commit 0af711b1) in a multi-stage Dockerfile, using JDK 11 and sbt 1.2.8 and then sbt stage. The build takes about 5 minutes once and is cached by Docker after that. |
| Database | PostgreSQL; the pilot creates its schema on start (hbm2ddl create-drop), so every run starts empty. Only the JDBC URL in conf/META-INF/persistence.xml needs patching to the database container's network alias. |
| Start-up | Ready in about 90 s (GET /projects answers 200). |
| Authentication | None. |
| Smoke test | Creating a project, reading it, an unknown project (404), /meta/datatypes, a commit, and its roots all work. Roots come back with their derived properties. |
| Coverage | 32 of the 35 operations of Resources/ptc-25-02-30.json are routed. Diff, merge, and PUT /projects/{projectId}/queries/{queryId} are missing; record them as known divergences, the way #423 does in KnownServerDivergences. |
| License | EPL-2.0; we download and build it at test time and copy none of its code. |
Proposal:
- Add a
PilotTestHostnext toFlexoTestHost:- It builds the image from a Dockerfile under
SySML2.NET.REST.Tests/Resources/Pilot/, with the source commit and the base images pinned. - It starts PostgreSQL with
Testcontainers.PostgreSql, which the project already references. - It waits until
GET /projectsanswers 200. - It calls
Assert.Ignoreonly when Docker is unreachable, and fails with the container logs on anything else.
- It builds the image from a Dockerfile under
- Keep
SYSML2_API_BASE_ADDRESSonly as an optional override for a remote server.
2. Line up with the layout and helpers of #423
- Layout.
SySML2.NET.REST.Tests/TEST_PROJECT_STRUCTURE.mdin #423 puts integration fixtures inIntegrationTests/and helper types inHelpers/, with model builders underHelpers/Models/, and gives every helper with behaviour an offline fixture inHelpersTests/. After both merge, a separateIntegration/folder breaks that.- Suggest: move the fixtures to
IntegrationTests/,LiveServerTestHosttoHelpers/, andDefaultSeedModeltoHelpers/Models/. - Add a
DefaultSeedModelTestFixture. - Document the suite and its run filter in
TEST_PROJECT_STRUCTURE.mdandINTEGRATION_TEST_PLAN.md.
- Suggest: move the fixtures to
- Test category. Both suites use
[Category("Integration")], so--filter TestCategory=Integrationruns both, and a run against one server also starts the other.- Suggest: an extra category per server, or a documented
FullyQualifiedNamefilter.
- Suggest: an extra category per server, or a documented
- Request checking. #423 records every request through
EndpointRecorderand checks requests and answers against the OpenAPI document (OpenApiConformance), reporting server divergences as warnings. Reusing that here would make the pilot run report OpenAPI conformance too, not only that a call succeeded. - Seed model. Build it with
ModelBuilder(POCOs plusToDto(includeDerivedProperties: true)); see the inline comment onDefaultSeedModel.
3. TESTING.md
The PR records no exception to TESTING.md §1–§3. The Flexo suite records its exception in plan §9. Deviations:
- six methods have a
[Test]in both the read and the write fixture:VerifyGetRootsByProjectCommitAsyncVerifyDiffAsyncVerifyGetTagByProjectAndIdAsyncVerifyGetQueryByProjectAndIdAsyncVerifyGetQueryResultsByProjectIdQueryIdAsyncVerifyGetQueryResultsByProjectIdQueryAsync
PsmApiClientnow has three fixtures;- a few tests cover only the success case or only the failure case.
Either merge them or record the exception.
4. Assertions that cannot fail
Several checks pass on an empty result. Against a local server with a known seed they can assert exact results instead; inline comments mark them.
Smaller points:
AssertEmptyOrNotFoundAsyncis out of name order (DEVELOPMENT_STANDARDS §1.8).- The null guards added to the MockHttp matchers in
SySML2.NET.REST.Tests/PsmApiClientTestFixture.csare harmless but unrelated to this PR.
| OwnedRelationship = [MembershipId] | ||
| }; | ||
|
|
||
| var membership = new OwningMembership |
There was a problem hiding this comment.
The DTO OwningMembership has ownedMemberElement { get; internal set; }, so it cannot be set from the test project. memberElement, source and target therefore go out as 00000000-0000-0000-0000-000000000000 in every seed commit. Membership::memberElement is [1..1]. A server that validates the commit rejects it and the whole write fixture fails in setup; a server that stores it holds a broken membership, and no test reads the membership back.
This is why #423 builds payloads as POCOs wired with AssignOwnership and converts them with ToDto(includeDerivedProperties: true) (SySML2.NET.REST.Tests/Helpers/ModelBuilder.cs); its EndpointRecorder flags exactly these empty references. Suggest building the seed with ModelBuilder once both are in. Related: #424.
|
|
||
| try | ||
| { | ||
| using var response = await this.httpClient.GetAsync("projects?page%5Bsize%5D=1"); |
There was a problem hiding this comment.
The probe passes on any status. A server that answers 401 or 403 is treated as reachable, and then all 40 tests fail instead of being skipped. There is also no way to send a token, which a server such as Flexo needs (#423 adds BearerTokenHandler).
With a local pilot container (see the review summary), this class becomes a test host like FlexoTestHost: it ignores only when Docker is unreachable, fails with the container logs on anything else, and keeps SYSML2_API_BASE_ADDRESS as an optional override.
| } | ||
|
|
||
| [Test] | ||
| public async Task VerifyGetProjectsAsync() |
There was a problem hiding this comment.
This test requires at least PageSize + 1 projects on the server (a full first page, Next, then a non-empty second page). On a server with one or two projects it fails although the client is fine; it depends on data that someone else put on the shared server. TESTING.md §10 asks for self-calibrating assertions. Against a local server, create the projects the test needs, or use Assert.Inconclusive when there are too few.
| { | ||
| page = await this.client.GetProjectsAsync(parameters); | ||
| } | ||
| catch (JsonException exception) |
There was a problem hiding this comment.
This swallows a client read defect: a JsonException while reading GET /projects is logged and discovery stops, so a bad record past page 2 never fails a test. SerializationException and InvalidOperationException from the deserializer are not caught, so those abort the whole fixture instead. Suggest letting the exception fail OneTimeSetUp, or recording it and asserting on it in VerifyGetProjectsAsync. Discovery also reads branches, queries and tags of up to 40 projects unguarded (line 62), so one unreadable project takes down every test.
| }; | ||
| } | ||
|
|
||
| private static async Task AssertEmptyOrNotFoundAsync<T>(Func<Task<PagedResult<T>>> query) |
There was a problem hiding this comment.
The OpenAPI document declares a 404 for every operation this guards, so accepting an empty 200 means a server that diverges is never reported and the client's 404 path is not exercised. Suggest expecting the 404 strictly, and tolerating a server divergence by name as #423 does (KnownServerDivergences).
| } | ||
|
|
||
| [Test] | ||
| public async Task VerifyPostCommitByProjectAsync() |
There was a problem hiding this comment.
The commit is checked only for OwningProject and a moved branch head, so a request body that drops or mis-serializes the changed element would still pass. Suggest reading PartDefinitionId at the new commit (expect DeclaredName "Car") and at seedCommitId (still "Vehicle"), and checking that PreviousCommit holds the previous head. The elements of the seed commit are also only counted in the read fixture; here they can be compared with the three seed ids.
|
|
||
| var deletedBranch = await GetOrNullWhenNotFoundAsync(() => this.client.GetBranchesByProjectAndIdAsync(this.projectId, branch.Id)); | ||
|
|
||
| Assert.That(deletedBranch is null || deletedBranch.Deleted is not null, Is.True); |
There was a problem hiding this comment.
Assert.That(x is null || x.Deleted is not null, Is.True) gives no useful message on failure (TESTING.md §8). Suggest Assert.That(deletedBranch, Is.Null.Or.Property(nameof(Branch.Deleted)).Not.Null);. The same applies at line 211 for the tag.
| } | ||
|
|
||
| [Test] | ||
| public async Task VerifyMergeAsync() |
There was a problem hiding this comment.
A successful merge is checked only for OwningProject, and the conflict case only for the exception type, although the client parses MergeConflictException.Conflicts. Suggest checking that the target branch head equals the merge commit, that PartDefinitionId there reads "Bicycle", and that the conflicts hold PartDefinitionId. Use Assert.Inconclusive when a server lists no conflicts. Note that the pilot does not route merge or diff (nor does Flexo); record that as a known divergence.
| { | ||
| return (await this.client.PostCommitByProjectAsync(this.projectId, request, branchId)).Id; | ||
| } | ||
| catch (InvalidOperationException exception) |
There was a problem hiding this comment.
Falling back to the branch head on an InvalidOperationException hides a defect in reading the commit answer. Setup, VerifyDiffAsync and VerifyMergeAsync all go through here; only VerifyPostCommitByProjectAsync would still show it. Suggest dropping the fallback, so the defect fails where it happens.
| await Assert.ThatAsync(() => this.client.GetProjectByIdAsync(project.Id), | ||
| Throws.TypeOf<PsmApiException>().With.Property(nameof(PsmApiException.StatusCode)).EqualTo(HttpStatusCode.NotFound)); | ||
|
|
||
| await Assert.ThatAsync(() => this.client.DeleteProjectByIdAsync(UnknownId), Throws.TypeOf<PsmApiException>()); |
There was a problem hiding this comment.
This failure case accepts any status, so a client that mapped a 404 to the wrong status, or a server answering 500, would pass. The same pattern is at lines 122, 153, 182, 229, 290 and 300. Where the OpenAPI document declares a 404 for an unknown project, assert NotFound as the other tests do. DELETE /projects/{projectId} declares no 404, so this line may legitimately get a 204; that is worth deciding explicitly.
7199663 to
a637625
Compare
|
samatstariongroup
left a comment
There was a problem hiding this comment.
Thanks for the rework: most of the earlier review is addressed (strict 404s, exact checks on changes, relationships, commits and merge, discovery and the fallback removed, one fixture). The build is clean and the offline tests pass; the live suite was not run here.
Approving as is. The remaining points will be handled when this suite is harmonized with the Flexo suite of #423:
- Build the seed with
ModelBuilder/ToDto(includeDerivedProperties: true):DefaultSeedModelstill sends an emptymemberElement,sourceandtargetfor theOwningMembership, which the exact relationship asserts depend on. - Record diff, merge and
PUT /projects/{projectId}/queries/{queryId}as named known divergences per server; on the pilot they answer 404, whichSupportedAsyncdoes not treat as unsupported. - Local pilot container host, the
IntegrationTests//Helpers/layout, a per-server test category, andEndpointRecorder/ OpenAPI conformance checks. - Offline fixtures for
DefaultSeedModelandLiveServerTestHost, and the remaining one-sided tests (VerifyGetProjectUsageByProjectCommitElementAsync,VerifyPostProjectAsync,VerifyGetProjectsAsync,VerifyGetDatatypesAsync).



Prerequisites
Description
Fix #407 issue. The public API test is done. Some of API endpoints are working and some of them are not implemented yet. The results to be shared in wiki.