Skip to content

updated dbprocessing code for sqlalchemy 2.0 #137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ezd1000 wants to merge 31 commits into
base: main
Choose a base branch
from
Open

updated dbprocessing code for sqlalchemy 2.0 #137

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
7aae917
DButils: changed to 2.0 metadata binding syntax
Jan 10, 2024
38f12f2
DButils: changed to declarative mapping
Jan 10, 2024
6099814
dbp_testing.py: add bind to table drop
Jan 11, 2024
9903e22
DButils.getTraceback: get the sqlalchemy 2.0 join syntax right
Jan 17, 2024
8eb5697
DButils.py: fixed join methods
Jan 24, 2024
028dbf5
moved the bind from Metadata constructor to table creation
Jan 24, 2024
9c54844
DButils.py: fixed assertion error
Jan 25, 2024
4becd0d
DButils.py: fixed all warning messages
Jan 30, 2024
812baa8
undid change so CI will run 2.0
Feb 2, 2024
7a12790
added bind to drop_all and removed self from dbu.engine
Feb 6, 2024
83c4964
casted string with sqlalchemy.sql.text()
Feb 22, 2024
fbe6787
added my name to contributors
Feb 22, 2024
2e1cb80
added bind to create_all
May 20, 2024
bde52aa
removed bind from metadata
May 20, 2024
aca843c
scrubber: fix raw SQL syntax in version_number_check
May 20, 2024
8002ce0
clean_test_db: fix raw SQL syntax in vacuum
May 20, 2024
34a04ab
removed extra parentheses
May 21, 2024
edc8b6a
sqlalchemy.sql was used without it imported
Sep 18, 2024
80c5eab
move the bind from Metadata constructor to table creation (to reflect…
Jan 9, 2025
ac70274
execute no longer taking direct SQL string
Jan 17, 2025
e05685a
adding self to bind argument
Jan 20, 2025
ba5c4bf
Creates error directory if it doesn't exist
Feb 7, 2025
403d1c7
Updating python print syntax to a function call
Feb 7, 2025
587b029
Updated imp to importlib for load_source past Python 3.4
Feb 10, 2025
f92c10c
Updated error message checking to comply with Python 3.12 syntax
Mar 11, 2025
c862aed
Updating concatanation
Apr 1, 2025
62ed2ee
Creating function to handle the update of imp to importlib
Apr 1, 2025
d60fac1
Test for load_source function
Apr 1, 2025
84feb78
General cleanup from sqlalchemy 2.0
jtniehof Jan 27, 2026
f223c4f
CI: switch to next-gen images
jtniehof Jan 27, 2026
1b7d944
CI: split python 2 and 3 completely
jtniehof Jan 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 42 additions & 36 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,23 @@ version: 2.1
jobs:
deps:
docker:
- image: circleci/python:2.7
- image: cimg/python:2.7
auth:
username: $DOCKERHUB_USERNAME
password: $DOCKERHUB_PASSWORD
- image: circleci/python:3.7
steps:
- checkout
- run:
command: |
pip install --user sqlalchemy
pip install --user python-dateutil
- persist_to_workspace:
root: ~/.local
paths:
- lib # Save installed Python libraries
deps3:
docker:
- image: cimg/python:3.7
auth:
username: $DOCKERHUB_USERNAME
password: $DOCKERHUB_PASSWORD
Expand All @@ -18,17 +30,13 @@ jobs:
command: |
pip install --user sqlalchemy
pip install --user python-dateutil
sudo apt update
sudo apt install python3-pip
pip3 install --user "sqlalchemy<2.0"
pip3 install --user python-dateutil
- persist_to_workspace:
root: ~/.local
paths:
- lib # Save installed Python libraries
unittest:
docker:
- image: circleci/python:2.7
- image: cimg/python:2.7
auth:
username: $DOCKERHUB_USERNAME
password: $DOCKERHUB_PASSWORD
Expand All @@ -42,7 +50,7 @@ jobs:
python unit_tests/test_all.py -v
unittest3:
docker:
- image: circleci/python:3.7
- image: cimg/python:3.7
auth:
username: $DOCKERHUB_USERNAME
password: $DOCKERHUB_PASSWORD
Expand All @@ -52,20 +60,16 @@ jobs:
at: ~/.local
- run:
command: |
python3 setup.py build
python3 unit_tests/test_all.py -v
python3.7 setup.py build
python3.7 unit_tests/test_all.py -v
testpostgres:
docker:
- image: circleci/python:2.7
- image: cimg/python:2.7
auth:
username: $DOCKERHUB_USERNAME
password: $DOCKERHUB_PASSWORD
environment:
DATABASE_URL: postgresql://root@localhost/circle_test?sslmode=disable
- image: circleci/postgres:9.6.5-alpine-ram
auth:
username: $DOCKERHUB_USERNAME
password: $DOCKERHUB_PASSWORD
environment:
PGPORT: 5432
PGUSER: root
Expand All @@ -79,23 +83,20 @@ jobs:
- run:
command: |
sudo apt update
sudo apt install -y postgresql-client
sudo apt install -y postgresql postgresql-client
sudo -u postgres /usr/lib/postgresql/10/bin/pg_ctl -D /var/lib/postgresql/10/main -l logfile start
createdb -h localhost dbp_unittest -O root
pip install --user psycopg2
python setup.py build
python unit_tests/test_all.py -v
testpostgres3:
docker:
- image: circleci/python:3.7
- image: cimg/python:3.7
auth:
username: $DOCKERHUB_USERNAME
password: $DOCKERHUB_PASSWORD
environment:
DATABASE_URL: postgresql://root@localhost/circle_test?sslmode=disable
- image: circleci/postgres:9.6.5-alpine-ram
auth:
username: $DOCKERHUB_USERNAME
password: $DOCKERHUB_PASSWORD
environment:
PGPORT: 5432
PGUSER: root
Expand All @@ -109,14 +110,15 @@ jobs:
- run:
command: |
sudo apt update
sudo apt install -y postgresql-client python3-pip
sudo apt install -y postgresql postgresql-client
sudo -u postgres /usr/lib/postgresql/10/bin/pg_ctl -D /var/lib/postgresql/10/main -l logfile start
createdb -h localhost dbp_unittest -O root
pip3 install --user psycopg2
python3 setup.py build
python3 unit_tests/test_all.py -v
pip install --user psycopg2
python setup.py build
python unit_tests/test_all.py -v
docs:
docker:
- image: circleci/python:2.7
- image: cimg/python:2.7
auth:
username: $DOCKERHUB_USERNAME
password: $DOCKERHUB_PASSWORD
Expand Down Expand Up @@ -147,15 +149,18 @@ workflows:
- testpostgres:
requires:
- deps
- unittest3:
- docs:
requires:
- deps
- testpostgres3:
main3:
jobs:
- deps3
- unittest3:
requires:
- deps
- docs:
- deps3
- testpostgres3:
requires:
- deps
- deps3

# Schedule disables other triggers; nightly identical to main except
# triggers and branch restriction
Expand All @@ -176,12 +181,13 @@ workflows:
- testpostgres:
requires:
- deps
- unittest3:
- docs:
requires:
- deps
- testpostgres3:
- deps3
- unittest3:
requires:
- deps
- docs:
- deps3
- testpostgres3:
requires:
- deps
- deps3
36 changes: 18 additions & 18 deletions dbprocessing/DBstrings.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,29 +24,29 @@ class DBformatter(string.Formatter):
"""

SPECIAL_FIELDS = {
'Y': ('{Y:04d}', '(19|2\d)\d\d'),
'm': ('{m:02d}', '(0\d|1[0-2])'),
'Y': ('{Y:04d}', r'(19|2\d)\d\d'),
'm': ('{m:02d}', r'(0\d|1[0-2])'),
'b': ('{b}', 'Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec'),
'd': ('{d:02d}', '[0-3]\d'),
'y': ('{y:02d}', '\d\d'),
'j': ('{j:03d}', '[0-3]\d\d'),
'H': ('{H:02d}', '[0-2]\d'),
'M': ('{M:02d}', '[0-6]\d'),
'S': ('{S:02d}', '[0-6]\d'),
'MILLI': ('{MILLI:03d}', '\d{3}'),
'MICRO': ('{MICRO:03d}', '\d{3}'),
'd': ('{d:02d}', r'[0-3]\d'),
'y': ('{y:02d}', r'\d\d'),
'j': ('{j:03d}', r'[0-3]\d\d'),
'H': ('{H:02d}', r'[0-2]\d'),
'M': ('{M:02d}', r'[0-6]\d'),
'S': ('{S:02d}', r'[0-6]\d'),
'MILLI': ('{MILLI:03d}', r'\d{3}'),
'MICRO': ('{MICRO:03d}', r'\d{3}'),
'QACODE': ('{QACODE}', 'ok|ignore|problem'),
'VERSION': ('{VERSION}', '\d+\.\d+\.\d+'),
'DATE': ('{DATE}', '(19|2\d)\d\d(0\d|1[0-2])[0-3]\d'),
'datetime': ('{datetime}', '(19|2\d)\d\d(0\d|1[0-2])[0-3]\d'),
'mday': ('{mday:d}', '-?\d+'),
'APID': ('{APID:x}', '[\da-fA-F]+'),
'VERSION': ('{VERSION}', r'\d+\.\d+\.\d+'),
'DATE': ('{DATE}', r'(19|2\d)\d\d(0\d|1[0-2])[0-3]\d'),
'datetime': ('{datetime}', r'(19|2\d)\d\d(0\d|1[0-2])[0-3]\d'),
'mday': ('{mday:d}', r'-?\d+'),
'APID': ('{APID:x}', r'[\da-fA-F]+'),
'??': ('{??}', '..'),
'???': ('{???}', '...'),
'????': ('{????}', '....'),
'nn': ('{nn}', '\d\d'),
'nnn': ('{nnn}', '\d\d\d'),
'nnnn': ('{nnnn}', '\d\d\d\d'),
'nn': ('{nn}', r'\d\d'),
'nnn': ('{nnn}', r'\d\d\d'),
'nnnn': ('{nnnn}', r'\d\d\d\d'),
}
"""indexed by field name; each element contains a fully-formatted
representation of the field and a regular expression that should match
Expand Down
Loading