add CRS Support for OGC API Feature pygeoapi Provider (geopython#1174)

* OGC API - Features Part 2 (groundwork+CRS-BBOX) from PR geopython#1155 - contributes to issue geopython#1128

* geopython#1128 provide conformance class for OAPIF Part 2 in /conformance page

* geopython#1128 bitten by flake8...

* geopython#1128 configurability CRS Feature Providers with syntax, defaults and tests

* geopython#1128 configurability CRS Feature Providers refine for default values

* geopython#1128 display supported CRSs in HTML Collection template

* geopython#1128 config, mmetadata and tests for storageCRS and storageCrsCoordinateEpoch

* geopython#1128 WIP for bbox-crs parameter support

* geopython#1128 utility function and tests for default/mandatory supprted CRS list

* geopython#1128 default supported CRS adaptation to OAPIF Part 2 standard

* geopython#1128 grr flake8 whitespace

* geopython#1128 start adding full API tests OGR for bbox-crs and crs parms

* geopython#1128 fix flake8

* geopython#1128 fix flake8 - install GDAL in workflow main for OGR tests

* geopython#1128 fix flake8 - install GDAL in workflow main for OGR tests - need pip package?

* geopython#1128 fix flake8 - install GDAL in workflow main for OGR tests - using libgdal-dev gdal-bin

* geopython#1128 fix SensorThings test for main.yml Workflow

* geopython#1128 fix SensorThings test for main.yml Workflow nr 2

* geopython#1128 make all OGR tests working again

* geopython#1128 make all OGR tests working again - flake8

* geopython#1128 make all OGR tests working again - GeoSolutions WFS bbox

* geopython#1128 geopython#1155 add documentation for OGC OAPIF Part 2 CRS CRS BBOX support

* geopython#1128 geopython#1155 refine documentation for OGC OAPIF Part 2 CRS CRS BBOX support

* geopython#1128 geopython#1155 refine documentation to align with geopython#1149

* geopython#1128 geopython#1155 rework from review OAS and pygeoapi config schema

* geopython#1128 geopython#1155 minor: compile Re for CRS URI only once as global var

* geopython#1128 merge in changes from PR geopython#1173 - fix missing import

* WIP Ogcapi features part 2 - Support for crs query parameter (geopython#1149)

* feat(ogcapi_features_crs): start implementing crs support from ogcapi features part2

* Pass input and output CRSs WKT instead of crs transformation object

* fix longs lines and blank lines

* fix typo

* fix import for type annotation not supported by python version

* fix variable visibility in local scope

* fix tabs/spaces indentations

* Add support for the crs parameter to OGRProvider

* make flake8 happy

* Make crs transformation mechanism more consistent between PostgreSQL and OGR providers

* test(util): add two test functions in util.py

New functions: test_get_crs_from_uri and test_get_transform_from_crs

* fix too long lines...

* Update get_crs_from_uri and corresponding test function

* fix(get_crs_from_uri): make the error more explicit in if wrong crs uri format

* flake8 again...

* Keep support for source_srs/target_srs in config for OGRProvider

* revert changes made to pygeoapi-config-0.x.yml, overlap with PR 1155

* test: add test data and update test config file

* Extract 'crs' and 'storage_crs' and provider level instead of collection level

* feat(crs): new decorator to support coordinates transformation of feature collections

* feat(crs): 'crs' query parameter for CSVProvider

* test(crs): add tests for 'crs' query parameter

* test: update number of collections in test_describe_collections

* test: update number of collections in test_filter_dict_by_key_value

* fix(crs_transform): change the crs transformation decorator

Change the logic of the decorator so that it works for both functions that
return FeatureCollections and for functions tha return single Features.

* test: add tests for get_collection_item end-point with 'crs' parameter

* fix(test_get_collection_item_crs): id as path parameter, not query parameter

* test: unpack coordinates to create point geometry

* feat(crs): add suuport for crs query parameter for all providers of type 'feature'

* docs(crs): add documentation to illustrate use of 'crs' query parameters

* docs(crs): more data access examples

* fix typo and add new line

* refactor: specify None as default value for crs_transform_out parameter in _sqlalchemy_to_feature method

* changes for PR 1149, test_api and style formatting

* CRS84 as default crs also for test_get_collection_items_crs

* test(crs): test coordinates transformation implementation of PostgreSQLProvider

* test(crs): move tests to test_postgresql_provider

* fix test function calls

* change test to ensure returned features are the same

* add json format to request object

* test(crs): test coordinates transformation implementation of OGRProvider

* refactor(crs): make more compact get_collection_item and get_collection_items

Define two new static methods in API class, to create crs_transform_wkt and
setting content-crs header. These methods can be re-used in both
get_collection_item and get_collection_items methods and removes code
duplication.

---------

Co-authored-by: Just van den Broecke <just@justobjects.nl>

* geopython#1178 fix flake8 error

* geopython#1178 use EPSG:28992 i.s.o. 32631 - fix unit test OGR Shapefile

* geopython#1174 use CRS-compliant Axis ordering for crs support

* geopython#1174 fix and honour CRS 4258disable native CRS Transform in OGR Provider - Axis ordering not honoured...

* geopython#1174 remove ADR tests rom test_util.py

* geopython#1174 enable native CRS transform again in OGR Provider

* geopython#1174 enable native CRS transform again in OGR Provider - fix config

* geopython#1174 remove support for source/target_srs in OGRProvider - enforce transforms always based on storageCRS

* geopython#1174 fix tests Postgresql Provider for Transforms

* geopython#1174 fix tests Postgresql Provider for Transforms

* geopython#1174 add tests for OGR Transformation and Axis Order

* geopython#1174 Suppress potential axis-swapping in OGR ExportToJSON

* geopython#1174 minor fix test - unassign spatialref before setgeom infeat

* geopython#1174 minor fix test - unassign spatialref before setgeom infeat - flake8

* geopython#1174 solve CI WFS test failures with GDAL HTTP config options

* geopython#1174 bbox and bbox-crs defs local in openapi.py for CITE validators

* geopython#1174 merge master - geopython#1152 geopython#1203 etc

* geopython#1174 small doc changes

* geopython#1174 move GeomObject typedef to beginning of util.py

* geopython#1174 added debug logging in transform Decorator func

---------

Co-authored-by: Mathieu Tachon <92298764+MTachon@users.noreply.github.com>

Author: justb4

.github/workflows/main.yml

@@ -80,6 +80,11 @@ jobs:
       with:
         packages: libsqlite3-mod-spatialite
         version: 4.3.0a-6build1
+    - name: Install GDAL with Python bindings
+      uses: awalsh128/cache-apt-pkgs-action@latest
+      with:
+        packages: gdal-bin libgdal-dev
+        version: 3.0.4
     - name: Install requirements 📦
       run: |
         pip3 install -r requirements.txt
@@ -89,6 +94,7 @@ jobs:
         python3 setup.py install
         pip3 install --upgrade numpy elasticsearch
         pip3 install --upgrade numpy "sqlalchemy<2"
+        pip3 install --global-option=build_ext --global-option="-I/usr/include/gdal" GDAL==`gdal-config --version`
         #pip3 install --upgrade rasterio==1.1.8
     - name: setup test data ⚙️
       run: |
@@ -102,6 +108,7 @@ jobs:
         POSTGRESQL_PASSWORD: ${{ secrets.DatabasePassword || 'postgres' }}
       run: |
         pytest tests/test_api.py
+        pytest tests/test_api_ogr_provider.py
         pytest tests/test_config.py
         pytest tests/test_csv__formatter.py
         pytest tests/test_csv__provider.py
@@ -110,12 +117,12 @@ jobs:
         pytest tests/test_filesystem_provider.py
         pytest tests/test_geojson_provider.py
         pytest tests/test_mongo_provider.py
-        #pytest tests/test_ogr_csv_provider.py
-        #pytest tests/test_ogr_esrijson_provider.py
-        #pytest tests/test_ogr_gpkg_provider.py
-        #pytest tests/test_ogr_shapefile_provider.py
-        #pytest tests/test_ogr_sqlite_provider.py
-        #pytest tests/test_ogr_wfs_provider.py
+        pytest tests/test_ogr_csv_provider.py
+        pytest tests/test_ogr_esrijson_provider.py
+        pytest tests/test_ogr_gpkg_provider.py
+        pytest tests/test_ogr_shapefile_provider.py
+        pytest tests/test_ogr_sqlite_provider.py
+        pytest tests/test_ogr_wfs_provider.py
         pytest tests/test_openapi.py
         pytest tests/test_postgresql_provider.py
         pytest tests/test_rasterio_provider.py

.gitignore

@@ -115,3 +115,9 @@ ENV/
 
 .DS_Store
 examples/django/sample_project/db.sqlite3
+
+# Backup files
+*~
+
+# Pre-commit hooks config file
+.pre-commit-config.yaml

aws-lambda/function/pygeoapi-config.yml

@@ -296,8 +296,8 @@ resources:
               data:
                   source_type: WFS
                   source: WFS:https://service.pdok.nl/kadaster/rdinfo/wfs/v1_0?
-                  source_srs: EPSG:28992
-                  target_srs: EPSG:4326
+#                  source_srs: EPSG:28992
+#                  target_srs: EPSG:4326
                   source_capabilities:
                       paging: True
 
@@ -312,6 +312,12 @@ resources:
                       # GDAL_PROXY_AUTH: (optional auth for remote WFS)
                       CPL_DEBUG: NO
 
+              crs:
+                - http://www.opengis.net/def/crs/OGC/1.3/CRS84
+                - http://www.opengis.net/def/crs/EPSG/0/4326
+                - http://www.opengis.net/def/crs/EPSG/0/4258
+                - http://www.opengis.net/def/crs/EPSG/0/28992
+              storage_crs: http://www.opengis.net/def/crs/EPSG/0/28992
               id_field: gml_id
               layer: rdinfo:stations
 

docker/default.config.yml

@@ -211,8 +211,8 @@ resources:
               data:
                   source_type: WFS
                   source: WFS:https://service.pdok.nl/kadaster/rdinfo/wfs/v1_0?
-                  source_srs: EPSG:28992
-                  target_srs: EPSG:4326
+#                  source_srs: EPSG:28992
+#                  target_srs: EPSG:4326
                   source_capabilities:
                       paging: True
 
@@ -227,6 +227,10 @@ resources:
                       # GDAL_PROXY_AUTH: (optional auth for remote WFS)
                       CPL_DEBUG: NO
 
+              crs:
+                - http://www.opengis.net/def/crs/EPSG/0/4258
+                - http://www.opengis.net/def/crs/EPSG/0/28992
+              storage_crs: http://www.opengis.net/def/crs/EPSG/0/28992
               id_field: gml_id
               layer: rdinfo:stations
 
@@ -258,8 +262,8 @@ resources:
               data:
                   source_type: WFS
                   source: WFS:http://demo.deegree.org/utah-workspace/services/wfs?TYPENAME=app:SGID93_LOCATION_UDOTMap_CityLocations
-                  source_srs: EPSG:26912
-                  target_srs: EPSG:4326
+#                  source_srs: EPSG:26912
+#                  target_srs: EPSG:4326
                   source_capabilities:
                       paging: True
 
@@ -274,6 +278,10 @@ resources:
                       # GDAL_PROXY_AUTH: (optional auth for remote WFS)
                       CPL_DEBUG: NO
 
+              crs:
+                - http://www.opengis.net/def/crs/EPSG/0/4258
+                - http://www.opengis.net/def/crs/EPSG/0/26912
+              storage_crs: http://www.opengis.net/def/crs/EPSG/0/26912
               id_field: NAME
               layer: app:SGID93_LOCATION_UDOTMap_CityLocations
 
@@ -304,8 +312,8 @@ resources:
               data:
                   source_type: WFS
                   source: WFS:https://gs-stable.geosolutionsgroup.com/geoserver/wfs
-                  source_srs: EPSG:32632
-                  target_srs: EPSG:4326
+#                  source_srs: EPSG:32632
+#                  target_srs: EPSG:4326
                   source_capabilities:
                       paging: True
 
@@ -320,6 +328,10 @@ resources:
                       # GDAL_PROXY_AUTH: (optional auth for remote WFS)
                       CPL_DEBUG: NO
 
+              crs:
+                - http://www.opengis.net/def/crs/EPSG/0/4258
+                - http://www.opengis.net/def/crs/EPSG/0/32632
+              storage_crs: http://www.opengis.net/def/crs/EPSG/0/32632
               id_field: gml_id
               layer: unesco:Unesco_point
 
@@ -355,8 +367,8 @@ resources:
               data:
                   source_type: GPKG
                   source: tests/data/poi_portugal.gpkg
-                  source_srs: EPSG:4326
-                  target_srs: EPSG:4326
+#                  source_srs: EPSG:4326
+#                  target_srs: EPSG:4326
                   source_capabilities:
                       paging: True
 
@@ -395,8 +407,8 @@ resources:
               data:
                   source_type: GeoJSON
                   source: tests/data/ne_110m_lakes.geojson
-                  source_srs: EPSG:4326
-                  target_srs: EPSG:4326
+#                  source_srs: EPSG:4326
+#                  target_srs: EPSG:4326
                   source_capabilities:
                       paging: True
 
@@ -438,8 +450,8 @@ resources:
                   source_type: SQLite
                   # source: tests/data/ne_110m_admin_0_countries.sqlite
                   source: tests/data/dutch_addresses_4326.sqlite
-                  source_srs: EPSG:4326
-                  target_srs: EPSG:4326
+#                  source_srs: EPSG:4326
+#                  target_srs: EPSG:4326
                   source_capabilities:
                       paging: True
 
@@ -480,8 +492,8 @@ resources:
               data:
                   source_type: GPKG
                   source: tests/data/dutch_addresses_4326.gpkg
-                  source_srs: EPSG:4326
-                  target_srs: EPSG:4326
+#                  source_srs: EPSG:4326
+#                  target_srs: EPSG:4326
                   source_capabilities:
                       paging: True
 

docs/source/configuration.rst

@@ -161,8 +161,8 @@ default.
               - observations
               - monitoring
           linked-data: # linked data configuration (see Linked Data section)
-              item_template: tests/data/base.jsonld 
-              context:  
+              item_template: tests/data/base.jsonld
+              context:
                   - datetime: https://schema.org/DateTime
                   - vocab: https://example.com/vocab#
                     stn_id: "vocab:stn_id"
@@ -195,6 +195,13 @@ default.
                 uri_field: uri # optional field corresponding to the Uniform Resource Identifier (see Linked Data section)
                 time_field: datetimestamp  # optional field corresponding to the temporal property of the dataset
                 title_field: foo # optional field of which property to display as title/label on HTML pages
+                crs: # optional: supported CRSs for parameters 'crs' and 'bbox-crs' (OGC OAPIF Part 2)
+                     # default: http://www.opengis.net/def/crs/OGC/1.3/CRS84
+                    - http://www.opengis.net/def/crs/EPSG/0/4326
+                    - http://www.opengis.net/def/crs/EPSG/0/3857
+                    - http://www.opengis.net/def/crs/EPSG/0/28992
+                storage_crs: http://www.opengis.net/def/crs/OGC/1.3/CRS84 # optional CRS in which data is stored, default: as 'crs' field
+                storage_crs_coordinate_epoch: : 2017.23 # optional, if storage_crs is a dynamic coordinate reference system
                 format:  # optional default format
                     name: GeoJSON  # required: format name
                     mimetype: application/json  # required: format mimetype
@@ -203,6 +210,14 @@ default.
                 properties:  # optional: only return the following properties, in order
                     - stn_id
                     - value
+                # coordinate reference systems (CRS) section is optional
+                # default CRSs are http://www.opengis.net/def/crs/OGC/1.3/CRS84 (coordinates without height)
+                # and http://www.opengis.net/def/crs/OGC/1.3/CRS84h (coordinates with ellipsoidal height)
+                storage_crs: http://www.opengis.net/def/crs/EPSG/0/28992 # CRS of the dataset to publish
+                crs: # supported coordinate reference systems (CRS) for 'crs' query parameter
+                    - http://www.opengis.net/def/crs/EPSG/0/28992
+                    - http://www.opengis.net/def/crs/OGC/1.3/CRS84
+                    - http://www.opengis.net/def/crs/EPSG/0/4326
 
       hello-world:  # name of process
           type: collection  # REQUIRED (collection, process, or stac-collection)
@@ -570,12 +585,12 @@ deployment flexibility, the path can be specified with string interpolation of e
 .. code-block:: yaml
 
     linked-data:
-      item_template: tests/data/base.jsonld 
+      item_template: tests/data/base.jsonld
       context:
         - datetime: https://schema.org/DateTime
 
 .. note::
-   The template ``tests/data/base.jsonld`` renders the unmodified JSON-LD. For more information on the capacities 
+   The template ``tests/data/base.jsonld`` renders the unmodified JSON-LD. For more information on the capacities
    of Jinja2 templates, see :ref:`html-templating`.
 
 Summary