Skip to content

Commit b74bc79

Browse files
noellehmchengrabii-chaarani
noellehmcheng
and
rabii-chaarani
authored
feat: Processing/processing tools sampler (#19)
* sampler * fix sorter * fix data type of spacing and decimator in sampler * rename unused loop idx in sampler * add dtm to input * add validation in sampler * spacing decimator test * refactor sampler tests for cicd compatibility * update tester.yml workflow for all branches * change image * update testing.txt * install map2loop in tester.yml * fix: update field types to use QVariant * fix optional dtm parameter in decimator --------- Co-authored-by: Rabii Chaarani <50892556+rabii-chaarani@users.noreply.github.com>
1 parent 671fed2 commit b74bc79

28 files changed

Lines changed: 334 additions & 107 deletions

‎.github/workflows/tester.yml‎

Lines changed: 59 additions & 56 deletions
Original file line numberDiff line numberDiff line change
@@ -2,16 +2,12 @@ name: "🎳 Tester"
22

33
on:
44
push:
5-
branches:
6-
- main
75
paths:
86
- '**.py'
97
- .github/workflows/tester.yml
108
- requirements/testing.txt
119

1210
pull_request:
13-
branches:
14-
- main
1511
paths:
1612
- '**.py'
1713
- .github/workflows/tester.yml
@@ -45,55 +41,62 @@ jobs:
4541
- name: Run Unit tests
4642
run: pytest -p no:qgis tests/unit/
4743

48-
# test-qgis:
49-
# runs-on: ubuntu-latest
50-
51-
# container:
52-
# image: qgis/qgis:3.4
53-
# env:
54-
# CI: true
55-
# DISPLAY: ":1"
56-
# MUTE_LOGS: true
57-
# NO_MODALS: 1
58-
# PYTHONPATH: "/usr/share/qgis/python/plugins:/usr/share/qgis/python:."
59-
# QT_QPA_PLATFORM: "offscreen"
60-
# WITH_PYTHON_PEP: false
61-
# # be careful, things have changed since QGIS 3.40. So if you are using this setup
62-
# # with a QGIS version older than 3.40, you may need to change the way you set up the container
63-
# volumes:
64-
# # Mount the X11 socket to allow GUI applications to run
65-
# - /tmp/.X11-unix:/tmp/.X11-unix
66-
# # Mount the workspace directory to the container
67-
# - ${{ github.workspace }}:/home/root/
68-
69-
# steps:
70-
# - name: Get source code
71-
# uses: actions/checkout@v4
72-
73-
# - name: Print QGIS version
74-
# run: qgis --version
75-
76-
# # Uncomment if you need to run a script to set up the plugin in QGIS docker image < 3.40
77-
# # - name: Setup plugin
78-
# # run: qgis_setup.sh ${{ env.PROJECT_FOLDER }}
79-
80-
# - name: Install Python requirements
81-
# run: |
82-
# apt update && apt install -y python3-pip python3-venv pipx
83-
# # Create a virtual environment
84-
# cd /home/root/
85-
# pipx run qgis-venv-creator --venv-name ".venv"
86-
# # Activate the virtual environment
87-
# . .venv/bin/activate
88-
# # Install the requirements
89-
# python3 -m pip install -U -r requirements/testing.txt
90-
91-
# - name: Run Unit tests
92-
# run: |
93-
# cd /home/root/
94-
# # Activate the virtual environment
95-
# . .venv/bin/activate
96-
# # Run the tests
97-
# # xvfb-run is used to run the tests in a virtual framebuffer
98-
# # This is necessary because QGIS requires a display to run
99-
# xvfb-run python3 -m pytest tests/qgis --junitxml=junit/test-results-qgis.xml --cov-report=xml:coverage-reports/coverage-qgis.xml
44+
test-qgis:
45+
runs-on: ubuntu-latest
46+
47+
container:
48+
image: qgis/qgis:latest
49+
env:
50+
CI: true
51+
DISPLAY: ":1"
52+
MUTE_LOGS: true
53+
NO_MODALS: 1
54+
PYTHONPATH: "/usr/share/qgis/python/plugins:/usr/share/qgis/python:."
55+
QT_QPA_PLATFORM: "offscreen"
56+
WITH_PYTHON_PEP: false
57+
# be careful, things have changed since QGIS 3.40. So if you are using this setup
58+
# with a QGIS version older than 3.40, you may need to change the way you set up the container
59+
volumes:
60+
# Mount the X11 socket to allow GUI applications to run
61+
- /tmp/.X11-unix:/tmp/.X11-unix
62+
# Mount the workspace directory to the container
63+
- ${{ github.workspace }}:/home/root/
64+
65+
steps:
66+
- name: Get source code
67+
uses: actions/checkout@v4
68+
69+
- name: Print QGIS version
70+
run: qgis --version
71+
72+
# Uncomment if you need to run a script to set up the plugin in QGIS docker image < 3.40
73+
# - name: Setup plugin
74+
# run: qgis_setup.sh ${{ env.PROJECT_FOLDER }}
75+
76+
- name: Install Python requirements
77+
run: |
78+
apt update && apt install -y python3-pip python3-venv pipx
79+
# Create a virtual environment
80+
cd /home/root/
81+
pipx run qgis-venv-creator --venv-name ".venv"
82+
# Activate the virtual environment
83+
. .venv/bin/activate
84+
# Install the requirements
85+
python3 -m pip install -U -r requirements/testing.txt
86+
python3 -m pip install git+https://github.com/Loop3D/map2loop.git@noelle/contact_extractor
87+
88+
- name: verify input data
89+
run: |
90+
cd /home/root/
91+
. .venv/bin/activate
92+
ls -la tests/qgis/input/ || echo "Input directory not found"
93+
94+
- name: Run Unit tests
95+
run: |
96+
cd /home/root/
97+
# Activate the virtual environment
98+
. .venv/bin/activate
99+
# Run the tests
100+
# xvfb-run is used to run the tests in a virtual framebuffer
101+
# This is necessary because QGIS requires a display to run
102+
xvfb-run python3 -m pytest tests/qgis --junitxml=junit/test-results-qgis.xml --cov-report=xml:coverage-reports/coverage-qgis.xml

‎m2l/processing/algorithms/sampler.py‎

Lines changed: 79 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,9 @@
1010
"""
1111
# Python imports
1212
from typing import Any, Optional
13-
from qgis.PyQt.QtCore import QMetaType
13+
from qgis.PyQt.QtCore import QVariant
14+
from osgeo import gdal
15+
import pandas as pd
1416

1517
# QGIS imports
1618
from qgis.core import (
@@ -22,13 +24,17 @@
2224
QgsProcessingFeedback,
2325
QgsProcessingParameterFeatureSink,
2426
QgsProcessingParameterFeatureSource,
25-
QgsProcessingParameterString,
27+
QgsProcessingParameterRasterLayer,
28+
QgsProcessingParameterEnum,
2629
QgsProcessingParameterNumber,
30+
QgsFields,
2731
QgsField,
2832
QgsFeature,
2933
QgsGeometry,
3034
QgsPointXY,
31-
QgsVectorLayer
35+
QgsVectorLayer,
36+
QgsWkbTypes,
37+
QgsCoordinateReferenceSystem
3238
)
3339
# Internal imports
3440
from ...main.vectorLayerWrapper import qgsLayerToGeoDataFrame
@@ -68,17 +74,20 @@ def initAlgorithm(self, config: Optional[dict[str, Any]] = None) -> None:
6874

6975

7076
self.addParameter(
71-
QgsProcessingParameterString(
77+
QgsProcessingParameterEnum(
7278
self.INPUT_SAMPLER_TYPE,
7379
"SAMPLER_TYPE",
80+
["Decimator", "Spacing"],
81+
defaultValue=0
7482
)
7583
)
7684

7785
self.addParameter(
78-
QgsProcessingParameterFeatureSource(
86+
QgsProcessingParameterRasterLayer(
7987
self.INPUT_DTM,
8088
"DTM",
8189
[QgsProcessing.TypeRaster],
90+
optional=True,
8291
)
8392
)
8493

@@ -104,6 +113,8 @@ def initAlgorithm(self, config: Optional[dict[str, Any]] = None) -> None:
104113
QgsProcessingParameterNumber(
105114
self.INPUT_DECIMATION,
106115
"DECIMATION",
116+
QgsProcessingParameterNumber.Integer,
117+
defaultValue=1,
107118
optional=True,
108119
)
109120
)
@@ -112,14 +123,16 @@ def initAlgorithm(self, config: Optional[dict[str, Any]] = None) -> None:
112123
QgsProcessingParameterNumber(
113124
self.INPUT_SPACING,
114125
"SPACING",
126+
QgsProcessingParameterNumber.Double,
127+
defaultValue=200.0,
115128
optional=True,
116129
)
117130
)
118131

119132
self.addParameter(
120133
QgsProcessingParameterFeatureSink(
121134
self.OUTPUT,
122-
"Sampled Contacts",
135+
"Sampled Points",
123136
)
124137
)
125138

@@ -130,60 +143,77 @@ def processAlgorithm(
130143
feedback: QgsProcessingFeedback,
131144
) -> dict[str, Any]:
132145

133-
dtm = self.parameterAsSource(parameters, self.INPUT_DTM, context)
134-
geology = self.parameterAsSource(parameters, self.INPUT_GEOLOGY, context)
135-
spatial_data = self.parameterAsSource(parameters, self.INPUT_SPATIAL_DATA, context)
136-
decimation = self.parameterAsSource(parameters, self.INPUT_DECIMATION, context)
137-
spacing = self.parameterAsSource(parameters, self.INPUT_SPACING, context)
138-
sampler_type = self.parameterAsString(parameters, self.INPUT_SAMPLER_TYPE, context)
146+
dtm = self.parameterAsRasterLayer(parameters, self.INPUT_DTM, context)
147+
geology = self.parameterAsVectorLayer(parameters, self.INPUT_GEOLOGY, context)
148+
spatial_data = self.parameterAsVectorLayer(parameters, self.INPUT_SPATIAL_DATA, context)
149+
decimation = self.parameterAsInt(parameters, self.INPUT_DECIMATION, context)
150+
spacing = self.parameterAsDouble(parameters, self.INPUT_SPACING, context)
151+
sampler_type_index = self.parameterAsEnum(parameters, self.INPUT_SAMPLER_TYPE, context)
152+
sampler_type = ["Decimator", "Spacing"][sampler_type_index]
153+
154+
if spatial_data is None:
155+
raise QgsProcessingException("Spatial data is required")
156+
157+
if sampler_type == "Decimator" and geology is None:
158+
raise QgsProcessingException("Geology is required")
139159

140160
# Convert geology layers to GeoDataFrames
141161
geology = qgsLayerToGeoDataFrame(geology)
142-
spatial_data = qgsLayerToGeoDataFrame(spatial_data)
162+
spatial_data_gdf = qgsLayerToGeoDataFrame(spatial_data)
163+
dtm_gdal = gdal.Open(dtm.source()) if dtm is not None and dtm.isValid() else None
143164

144-
if sampler_type == "decimator":
165+
if sampler_type == "Decimator":
145166
feedback.pushInfo("Sampling...")
146-
sampler = SamplerDecimator(decimation=decimation, dtm_data=dtm, geology_data=geology, feedback=feedback)
147-
samples = sampler.sample(spatial_data)
167+
sampler = SamplerDecimator(decimation=decimation, dtm_data=dtm_gdal, geology_data=geology)
168+
samples = sampler.sample(spatial_data_gdf)
148169

149-
if sampler_type == "spacing":
170+
if sampler_type == "Spacing":
150171
feedback.pushInfo("Sampling...")
151-
sampler = SamplerSpacing(spacing=spacing, dtm_data=dtm, geology_data=geology, feedback=feedback)
152-
samples = sampler.sample(spatial_data)
153-
154-
155-
# create layer
156-
vector_layer = QgsVectorLayer("Point", "sampled_points", "memory")
157-
provider = vector_layer.dataProvider()
158-
159-
# add fields
160-
provider.addAttributes([QgsField("ID", QMetaType.Type.QString),
161-
QgsField("X", QMetaType.Type.Float),
162-
QgsField("Y", QMetaType.Type.Float),
163-
QgsField("Z", QMetaType.Type.Float),
164-
QgsField("featureId", QMetaType.Type.QString)
165-
])
166-
vector_layer.updateFields() # tell the vector layer to fetch changes from the provider
167-
168-
# add a feature
169-
for i in range(len(samples)):
170-
feature = QgsFeature()
171-
feature.setGeometry(QgsGeometry.fromPointXY(QgsPointXY(samples.X[i], samples.Y[i], samples.Z[i])))
172-
feature.setAttributes([samples.ID[i], samples.X[i], samples.Y[i], samples.Z[i], samples.featureId[i]])
173-
provider.addFeatures([feature])
174-
175-
# update layer's extent when new features have been added
176-
# because change of extent in provider is not propagated to the layer
177-
vector_layer.updateExtents()
178-
# --- create sink
172+
sampler = SamplerSpacing(spacing=spacing, dtm_data=dtm_gdal, geology_data=geology)
173+
samples = sampler.sample(spatial_data_gdf)
174+
175+
fields = QgsFields()
176+
fields.append(QgsField("ID", QVariant.String))
177+
fields.append(QgsField("X", QVariant.Double))
178+
fields.append(QgsField("Y", QVariant.Double))
179+
fields.append(QgsField("Z", QVariant.Double))
180+
fields.append(QgsField("featureId", QVariant.String))
181+
182+
crs = None
183+
if spatial_data_gdf is not None and spatial_data_gdf.crs is not None:
184+
crs = QgsCoordinateReferenceSystem.fromWkt(spatial_data_gdf.crs.to_wkt())
185+
179186
sink, dest_id = self.parameterAsSink(
180187
parameters,
181188
self.OUTPUT,
182189
context,
183-
vector_layer.fields(),
184-
QgsGeometry.Type.Point,
185-
spatial_data.crs,
186-
)
190+
fields,
191+
QgsWkbTypes.PointZ if 'Z' in (samples.columns if samples is not None else []) else QgsWkbTypes.Point,
192+
crs
193+
)
194+
195+
if samples is not None and not samples.empty:
196+
for _index, row in samples.iterrows():
197+
feature = QgsFeature(fields)
198+
199+
# decimator has z values
200+
if 'Z' in samples.columns and pd.notna(row.get('Z')):
201+
wkt = f"POINT Z ({row['X']} {row['Y']} {row['Z']})"
202+
feature.setGeometry(QgsGeometry.fromWkt(wkt))
203+
else:
204+
#spacing has no z values
205+
feature.setGeometry(QgsGeometry.fromPointXY(QgsPointXY(row['X'], row['Y'])))
206+
207+
feature.setAttributes([
208+
str(row.get('ID', '')),
209+
float(row.get('X', 0)),
210+
float(row.get('Y', 0)),
211+
float(row.get('Z', 0)) if pd.notna(row.get('Z')) else 0.0,
212+
str(row.get('featureId', ''))
213+
])
214+
215+
sink.addFeature(feature)
216+
187217
return {self.OUTPUT: dest_id}
188218

189219
def createInstance(self) -> QgsProcessingAlgorithm:

‎m2l/processing/algorithms/sorter.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,7 @@ def initAlgorithm(self, config: Optional[dict[str, Any]] = None) -> None:
9292
self.addParameter(
9393
QgsProcessingParameterFeatureSink(
9494
self.OUTPUT,
95-
self.tr("Stratigraphic column"),
95+
"Stratigraphic column",
9696
)
9797
)
9898

@@ -177,7 +177,7 @@ def build_input_frames(layer: QgsVectorLayer, feedback) -> tuple:
177177
(units_df, relationships_df, contacts_df, map_data)
178178
"""
179179
import pandas as pd
180-
from m2l.map2loop.mapdata import MapData # adjust import path if needed
180+
from map2loop.map2loop.mapdata import MapData # adjust import path if needed
181181

182182
# Example: convert the geology layer to a very small units_df
183183
units_records = []

‎requirements/testing.txt‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,3 +3,5 @@
33

44
pytest-cov>=4
55
packaging>=23
6+
shapely
7+
geopandas

‎tests/qgis/input/dtm_rp.tif‎

158 KB
Binary file not shown.

‎tests/qgis/input/dtm_rp.tif.aux.xml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
<PAMDataset>
2+
<PAMRasterBand band="1">
3+
<Metadata>
4+
<MDI key="STATISTICS_MAXIMUM">1175</MDI>
5+
<MDI key="STATISTICS_MEAN">561.72162965205</MDI>
6+
<MDI key="STATISTICS_MINIMUM">297</MDI>
7+
<MDI key="STATISTICS_STDDEV">107.52060579962</MDI>
8+
<MDI key="STATISTICS_VALID_PERCENT">99.73</MDI>
9+
</Metadata>
10+
</PAMRasterBand>
11+
</PAMDataset>

‎tests/qgis/input/faults_clip.cpg‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
ISO-8859-1

‎tests/qgis/input/faults_clip.dbf‎

48.8 KB
Binary file not shown.

‎tests/qgis/input/faults_clip.prj‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
PROJCS["GDA94_MGA_zone_50",GEOGCS["GCS_GDA94",DATUM["D_Geocentric_Datum_of_Australia_1994",SPHEROID["GRS_1980",6378137.0,298.257222101]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Transverse_Mercator"],PARAMETER["False_Easting",500000.0],PARAMETER["False_Northing",10000000.0],PARAMETER["Central_Meridian",117.0],PARAMETER["Scale_Factor",0.9996],PARAMETER["Latitude_Of_Origin",0.0],UNIT["Meter",1.0]]

‎tests/qgis/input/faults_clip.shp‎

9.09 KB
Binary file not shown.

0 commit comments

Comments
 (0)