Skip to content

Client api test for external server. - #432

Merged
zeynepatsoprasteria merged 3 commits into
developmentfrom
407-integration-against-public-api
Nov 12, 2026
Merged

zeynepatsoprasteria merged 3 commits into
developmentfrom
407-integration-against-public-api

Conversation

@zeynepatsoprasteria

@zeynepatsoprasteria zeynepatsoprasteria commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Prerequisites

  • I have written a descriptive pull-request title
  • I have verified that there are no overlapping pull-requests open
  • I have verified that I am following the SysML2.NET code style guidelines
  • I have provided test coverage for my change (where applicable)

Description

Fix #407 issue. The public API test is done. Some of API endpoints are working and some of them are not implemented yet. The results to be shared in wiki.

@samatstariongroup samatstariongroup left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: run the live tests against a local server, and line up with the Flexo suite of #423

Thanks for this. It calls every one of the 35 operations of the client, which is a good base. The PR builds, and it merges cleanly with the Flexo branch of #423 (407-feature-verify-rest-api-client-against-compliant-server); offline tests pass on both. The 40 live tests were skipped in my run, because SYSML2_API_BASE_ADDRESS was not set.

The two suites are meant to live side by side, so most of the points below are about fitting in with #423. Inline comments carry the details.

1. Do not depend on a server hosted on the internet; start the pilot implementation in containers

The tests look written against the SST-hosted pilot implementation, Systems-Modeling/SysML-v2-API-Services. The datatype ids https://www.omg.org/spec/SysML/20250201/... are exactly what the pilot returns. A shared, remote server makes the tests depend on its uptime and its existing data (see the inline comment on VerifyGetProjectsAsync), and the write tests add data to it.

#423 already starts Flexo locally with Testcontainers (SySML2.NET.REST.Tests/Helpers/FlexoTestHost.cs). I checked that the pilot can be run the same way:

Check Result
Image No official image; built from source (release 2026-04, commit 0af711b1) in a multi-stage Dockerfile, using JDK 11 and sbt 1.2.8 and then sbt stage. The build takes about 5 minutes once and is cached by Docker after that.
Database PostgreSQL; the pilot creates its schema on start (hbm2ddl create-drop), so every run starts empty. Only the JDBC URL in conf/META-INF/persistence.xml needs patching to the database container's network alias.
Start-up Ready in about 90 s (GET /projects answers 200).
Authentication None.
Smoke test Creating a project, reading it, an unknown project (404), /meta/datatypes, a commit, and its roots all work. Roots come back with their derived properties.
Coverage 32 of the 35 operations of Resources/ptc-25-02-30.json are routed. Diff, merge, and PUT /projects/{projectId}/queries/{queryId} are missing; record them as known divergences, the way #423 does in KnownServerDivergences.
License EPL-2.0; we download and build it at test time and copy none of its code.

Proposal:

  • Add a PilotTestHost next to FlexoTestHost:
    • It builds the image from a Dockerfile under SySML2.NET.REST.Tests/Resources/Pilot/, with the source commit and the base images pinned.
    • It starts PostgreSQL with Testcontainers.PostgreSql, which the project already references.
    • It waits until GET /projects answers 200.
    • It calls Assert.Ignore only when Docker is unreachable, and fails with the container logs on anything else.
  • Keep SYSML2_API_BASE_ADDRESS only as an optional override for a remote server.

2. Line up with the layout and helpers of #423

  • Layout. SySML2.NET.REST.Tests/TEST_PROJECT_STRUCTURE.md in #423 puts integration fixtures in IntegrationTests/ and helper types in Helpers/, with model builders under Helpers/Models/, and gives every helper with behaviour an offline fixture in HelpersTests/. After both merge, a separate Integration/ folder breaks that.
    • Suggest: move the fixtures to IntegrationTests/, LiveServerTestHost to Helpers/, and DefaultSeedModel to Helpers/Models/.
    • Add a DefaultSeedModelTestFixture.
    • Document the suite and its run filter in TEST_PROJECT_STRUCTURE.md and INTEGRATION_TEST_PLAN.md.
  • Test category. Both suites use [Category("Integration")], so --filter TestCategory=Integration runs both, and a run against one server also starts the other.
    • Suggest: an extra category per server, or a documented FullyQualifiedName filter.
  • Request checking. #423 records every request through EndpointRecorder and checks requests and answers against the OpenAPI document (OpenApiConformance), reporting server divergences as warnings. Reusing that here would make the pilot run report OpenAPI conformance too, not only that a call succeeded.
  • Seed model. Build it with ModelBuilder (POCOs plus ToDto(includeDerivedProperties: true)); see the inline comment on DefaultSeedModel.

3. TESTING.md

The PR records no exception to TESTING.md §1–§3. The Flexo suite records its exception in plan §9. Deviations:

  • six methods have a [Test] in both the read and the write fixture:
    • VerifyGetRootsByProjectCommitAsync
    • VerifyDiffAsync
    • VerifyGetTagByProjectAndIdAsync
    • VerifyGetQueryByProjectAndIdAsync
    • VerifyGetQueryResultsByProjectIdQueryIdAsync
    • VerifyGetQueryResultsByProjectIdQueryAsync
  • PsmApiClient now has three fixtures;
  • a few tests cover only the success case or only the failure case.

Either merge them or record the exception.

4. Assertions that cannot fail

Several checks pass on an empty result. Against a local server with a known seed they can assert exact results instead; inline comments mark them.

Smaller points:

  • AssertEmptyOrNotFoundAsync is out of name order (DEVELOPMENT_STANDARDS §1.8).
  • The null guards added to the MockHttp matchers in SySML2.NET.REST.Tests/PsmApiClientTestFixture.cs are harmless but unrelated to this PR.

OwnedRelationship = [MembershipId]
};

var membership = new OwningMembership

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The DTO OwningMembership has ownedMemberElement { get; internal set; }, so it cannot be set from the test project. memberElement, source and target therefore go out as 00000000-0000-0000-0000-000000000000 in every seed commit. Membership::memberElement is [1..1]. A server that validates the commit rejects it and the whole write fixture fails in setup; a server that stores it holds a broken membership, and no test reads the membership back.

This is why #423 builds payloads as POCOs wired with AssignOwnership and converts them with ToDto(includeDerivedProperties: true) (SySML2.NET.REST.Tests/Helpers/ModelBuilder.cs); its EndpointRecorder flags exactly these empty references. Suggest building the seed with ModelBuilder once both are in. Related: #424.


try
{
using var response = await this.httpClient.GetAsync("projects?page%5Bsize%5D=1");

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The probe passes on any status. A server that answers 401 or 403 is treated as reachable, and then all 40 tests fail instead of being skipped. There is also no way to send a token, which a server such as Flexo needs (#423 adds BearerTokenHandler).

With a local pilot container (see the review summary), this class becomes a test host like FlexoTestHost: it ignores only when Docker is unreachable, fails with the container logs on anything else, and keeps SYSML2_API_BASE_ADDRESS as an optional override.

}

[Test]
public async Task VerifyGetProjectsAsync()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test requires at least PageSize + 1 projects on the server (a full first page, Next, then a non-empty second page). On a server with one or two projects it fails although the client is fine; it depends on data that someone else put on the shared server. TESTING.md §10 asks for self-calibrating assertions. Against a local server, create the projects the test needs, or use Assert.Inconclusive when there are too few.

{
page = await this.client.GetProjectsAsync(parameters);
}
catch (JsonException exception)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This swallows a client read defect: a JsonException while reading GET /projects is logged and discovery stops, so a bad record past page 2 never fails a test. SerializationException and InvalidOperationException from the deserializer are not caught, so those abort the whole fixture instead. Suggest letting the exception fail OneTimeSetUp, or recording it and asserting on it in VerifyGetProjectsAsync. Discovery also reads branches, queries and tags of up to 40 projects unguarded (line 62), so one unreadable project takes down every test.

};
}

private static async Task AssertEmptyOrNotFoundAsync<T>(Func<Task<PagedResult<T>>> query)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The OpenAPI document declares a 404 for every operation this guards, so accepting an empty 200 means a server that diverges is never reported and the client's 404 path is not exercised. Suggest expecting the 404 strictly, and tolerating a server divergence by name as #423 does (KnownServerDivergences).

}

[Test]
public async Task VerifyPostCommitByProjectAsync()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The commit is checked only for OwningProject and a moved branch head, so a request body that drops or mis-serializes the changed element would still pass. Suggest reading PartDefinitionId at the new commit (expect DeclaredName "Car") and at seedCommitId (still "Vehicle"), and checking that PreviousCommit holds the previous head. The elements of the seed commit are also only counted in the read fixture; here they can be compared with the three seed ids.


var deletedBranch = await GetOrNullWhenNotFoundAsync(() => this.client.GetBranchesByProjectAndIdAsync(this.projectId, branch.Id));

Assert.That(deletedBranch is null || deletedBranch.Deleted is not null, Is.True);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assert.That(x is null || x.Deleted is not null, Is.True) gives no useful message on failure (TESTING.md §8). Suggest Assert.That(deletedBranch, Is.Null.Or.Property(nameof(Branch.Deleted)).Not.Null);. The same applies at line 211 for the tag.

}

[Test]
public async Task VerifyMergeAsync()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A successful merge is checked only for OwningProject, and the conflict case only for the exception type, although the client parses MergeConflictException.Conflicts. Suggest checking that the target branch head equals the merge commit, that PartDefinitionId there reads "Bicycle", and that the conflicts hold PartDefinitionId. Use Assert.Inconclusive when a server lists no conflicts. Note that the pilot does not route merge or diff (nor does Flexo); record that as a known divergence.

{
return (await this.client.PostCommitByProjectAsync(this.projectId, request, branchId)).Id;
}
catch (InvalidOperationException exception)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Falling back to the branch head on an InvalidOperationException hides a defect in reading the commit answer. Setup, VerifyDiffAsync and VerifyMergeAsync all go through here; only VerifyPostCommitByProjectAsync would still show it. Suggest dropping the fallback, so the defect fails where it happens.

await Assert.ThatAsync(() => this.client.GetProjectByIdAsync(project.Id),
Throws.TypeOf<PsmApiException>().With.Property(nameof(PsmApiException.StatusCode)).EqualTo(HttpStatusCode.NotFound));

await Assert.ThatAsync(() => this.client.DeleteProjectByIdAsync(UnknownId), Throws.TypeOf<PsmApiException>());

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This failure case accepts any status, so a client that mapped a 404 to the wrong status, or a server answering 500, would pass. The same pattern is at lines 122, 153, 182, 229, 290 and 300. Where the OpenAPI document declares a 404 for an unknown project, assert NotFound as the other tests do. DELETE /projects/{projectId} declares no 404, so this line may legitimately get a 204; that is worth deciding explicitly.

@zeynepatsoprasteria
zeynepatsoprasteria force-pushed the 407-integration-against-public-api branch from 7199663 to a637625 Compare November 10, 2026 14:52
@sonarqubecloud

Copy link
Copy Markdown

@samatstariongroup samatstariongroup left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the rework: most of the earlier review is addressed (strict 404s, exact checks on changes, relationships, commits and merge, discovery and the fallback removed, one fixture). The build is clean and the offline tests pass; the live suite was not run here.

Approving as is. The remaining points will be handled when this suite is harmonized with the Flexo suite of #423:

  • Build the seed with ModelBuilder / ToDto(includeDerivedProperties: true): DefaultSeedModel still sends an empty memberElement, source and target for the OwningMembership, which the exact relationship asserts depend on.
  • Record diff, merge and PUT /projects/{projectId}/queries/{queryId} as named known divergences per server; on the pilot they answer 404, which SupportedAsync does not treat as unsupported.
  • Local pilot container host, the IntegrationTests/ / Helpers/ layout, a per-server test category, and EndpointRecorder / OpenAPI conformance checks.
  • Offline fixtures for DefaultSeedModel and LiveServerTestHost, and the remaining one-sided tests (VerifyGetProjectUsageByProjectCommitElementAsync, VerifyPostProjectAsync, VerifyGetProjectsAsync, VerifyGetDatatypesAsync).

@zeynepatsoprasteria
zeynepatsoprasteria merged commit 05d5361 into development Nov 12, 2026
8 checks passed
@zeynepatsoprasteria
zeynepatsoprasteria deleted the 407-integration-against-public-api branch November 12, 2026 08:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Verify REST Api client against compliant server

2 participants