feat: deferred app calls, modular test context, refined access to value generators, numerous fixes/improvements (#4)

* feat: work in progress on asset, application related state ops

* test: adding extra tests

* feat: extra tests and implementation wrappers around AppLocal

* chore: wip

* chore: update src/algopy_testing/op.py

Co-authored-by: Daniel McGregor <daniel.mcgregor@makerx.com.au>

* feat: adding acctparamsget; extra tests; pr comments

* refactor: adding final bits around AcctParamsGet; unit tests and fixes

* refactor: adding lookup by index to acct/app/asset get ops; tweaking ci

* refactor: addressing pr comments

* chore: fixing failing test

* refactor: simplifying test_context validation

* use specific enum types in box example with latest puya version

* include box types in algopy_testing

* fix inconsistent usage of field names on application fields
use state total overrides when determining state totals
reduce usage of `import algopy` in implementations

* expose fields property on application to aid debugging

* added section to CONTRIBUTING.md describing relationship between `algopy` and `algopy_testing`

* remove lazy algopy imports from utils
remove some unnecessary ignores
add TODO

* simplify abimethod and add TODO's

* add TODO for state totals

* add some tests (including currently failing ones) for app transactions

* feat: add arc4factory

* refactor: ensuring underlying _key is properly reflected on local/global states

* refactor: change guards for setting keys to explicitly check for None

* refactor: use implementation types in internal mappings

* refactor: remove usages of `import algopy` from op.py, remove explicit imports from typing module
add TODO's

* test: use non-abstract contract base

* allow empty box prefix

* refactor

* use immutable param defaults

* fix: handle populating foreign arrays correctly for abi method calls

* refactor: remove lazy import algopy

* remove irrelevant comment

* initialize accounts correctly

* build: adding post install command into examples venv in hatch settings

* refactor: refine arc4 factory; add corresponding tests

* chore: adding the missing clear methods

* chore: merging everything from docs branch except docs changes

* chore: merge conflicts

* refactor: simplify txn implementations
provide default values for unspecified txn fields

* docs: adding pep257 formatter; using reST docstrings style for context.py

* test: adding tests for scratch slots

* refactor: renaming set_txn_fields -> scoped_txn_fields

* chore: adding `amount` field and open question under TODO;

also adding adding get_box_map  that reuses get_box but appends the bytes box_map prefix

* chore: bumping ruff

* refactor: adding context manager for lsig args setup (similar to algopy.Txn)

also running latest ruff - some rules are updated

* refactor: move helper classes into their own file

* refactor: simplify itxn loader

* refactor: isolate get_test_context to reduce circular imports

* chore: using multiprocessing in refresh test artifacts script

* refactor: adding tests for ITxn, ITxnCreate and GITxn, fixing related bugs

* refactor: default_creator -> default_sender; setting creator as default_sender

* chore: parsing name to op name in ITxn

* chore: updating default extension for mypy to use ms-python

* test: remove incorrect test and replace with TODO

* chore: add TODO about subroutine support

* add stricter type checks for primitives

* track when contracts are in a "creating" state or not

* todos

* refactor: moving GITxn class to itxn.py

* refactor: generate arc4 signatures from types
added more robust system for tracking arc4 types
removed unneeded functions on StaticArray

* only support native tuples when handling generic aliases in arc4 tuples

* refactor: 1/2 adding paged access to clear state program in txn fields

* refactor: consolidating txn and itxn related context attributes/methods

* minor refactors

* support arc4 structs

* refactor: simplify logic sig implementation, and remove mapping

* refactor: fix itxn op behaviour with program pages, and other array like fields

* refactor: simplify account properties

* refactor: move crypto ops into their own module

* refactor: move pure ops into their own module

* refactor: move other misc ops

* refactor: consolidating value generators; ledger and txn contexts;

* refactor: add active group/txn properties
change local/global state storage to store values against the app, not the contract instance
add UInt64Backed type to simplify serialization to/from int/bytes

* refactor: remove nested private modules, replace usages of get_test_context with lazy_context

* refactor: move inner transactions onto transaction group

* refactor: remove scoped_lsig_args

* refactor: remove maybe_active_app_id

* refactor: include bool in test for uint64

* refactor: ensure arc4 values always have fully parametrized types

* refactor: use _paramatize_type

* refactor: addressing TODOs

refactor: removing txn from method names inside txn context manager prop

chore: restoring initial pre-commit

refactor: expanding scoped_execution

chore: remove redundant fields

chore: addressing minor todos and removing the ones already addressed

* refactor: adding unit tests for global/local state with implicit keys

* refactor: improving handling of initial value for implicit global/local state keys

* test: extra test cases for accessing implicit/explicit keyed local/global state

* refactor: wip adding txn_group_for method

* chore: fix linting errors

* feat: continue with txn_group_for and add a test

* chore: remove scoped_txn_fields methods

* add some additional TODO's for scoped_execution

* remove TODO

* expand gaid TODO

* tweak op.exit implementation and add TODO

* remove arc4 property from AlgopyTestContext

* add more TODOs

* refactor: addressing TODOs; adding marketplace contract example (devrel bootcamps)

* test: fixing failing tests

---------

Co-authored-by: Daniel McGregor <daniel.mcgregor@makerx.com.au>

Author: aorumbayev

.github/workflows/cd.yaml

@@ -57,7 +57,7 @@ jobs:
           cache: "pip"
 
       - name: Start LocalNet
-        run: hatch run cicd:localnet_start
+        run: pipx install algokit && algokit localnet start
 
       - name: Check pre-commits
         run: hatch run check

.github/workflows/ci.yaml

@@ -3,6 +3,8 @@ name: Check Python Code (algopy_testing)
 on:
   workflow_call:
   pull_request:
+  schedule:
+    - cron: "0 8 * * 1" # Each monday 8 AM UTC
 
 jobs:
   check-python:
@@ -21,14 +23,17 @@ jobs:
           cache: "pip"
 
       - name: Start LocalNet
-        run: hatch run cicd:localnet_start
+        run: pipx install algokit && algokit localnet start
 
       - name: Check pre-commits
         run: hatch run check
 
       - name: Check pre-commits (examples)
         run: hatch run examples:check
 
+      - name: Validate examples folder
+        run: hatch run validate_examples
+
       - name: Check wheels can be built
         run: hatch build
 

.pre-commit-config.yaml

@@ -1,18 +1,10 @@
 repos:
   - repo: local
     hooks:
-      - id: pre-commit
+      - id: pre-commit (src & examples)
         name: pre-commit
         description: "Run pre-commit task via hatch"
         entry: hatch run pre_commit
         language: system
         additional_dependencies: []
         minimum_pre_commit_version: "0"
-
-      - id: examples-pre-commit
-        name: examples-pre-commit
-        description: "Run examples venv pre-commit task via hatch"
-        entry: hatch run examples:pre_commit
-        language: system
-        additional_dependencies: []
-        minimum_pre_commit_version: "0"

.vscode/extensions.json

@@ -0,0 +1,10 @@
+{
+    "recommendations": [
+        "ms-python.python",
+        "charliermarsh.ruff",
+        "ms-python.mypy-type-checker",
+        "ms-python.black-formatter",
+        "tamasfe.even-better-toml",
+        "editorconfig.editorconfig"
+    ]
+}

.vscode/settings.json

@@ -28,10 +28,11 @@
     "ruff.codeAction.fixViolation": {
         "enable": true
     },
-
-    "mypy.configFile": "pyproject.toml",
-    // set to empty array to use config from project
-    "mypy.targets": ["--cache-dir=/dev/null"],
-    "mypy.runUsingActiveInterpreter": true,
-    "cSpell.words": ["addw", "mulw"]
+    "mypy-type-checker.importStrategy": "fromEnvironment",
+    "mypy-type-checker.args": [
+        "--enable-incomplete-feature=Unpack",
+        "--config-file=pyproject.toml",
+        "--strict",
+        "--show-column-numbers"
+    ]
 }

CONTRIBUTING.md

@@ -40,3 +40,59 @@ Examples folder uses a dedicated 'venv.examples' virtual environment managed by
 Project relies on [python-semantic-release](https://python-semantic-release.readthedocs.io/en/latest/) for release automation.
 
 Releases are triggered by a `workflow_dispatch` event on the `main` branch with `prerelease` set to accordingly using the `.github/workflows/cd.yaml` file.
+
+# Implementation vs Stubs
+
+The canonical definition of the algopy API comes from the [`alogrand-python`](https://pypi.org/project/algorand-python/) package which consists only of typing information defined in the `algopy-stubs` module.
+In this library the testing implementation should follow the "shape" of the API defined in these stubs. In particular, types, methods and attributes should reflect what is described in the stubs.
+
+However, implementation specific details can diverge from what is described in the stubs. For example methods marked as `@staticmethod` in the stubs do not need to be static methods in the implementation.
+Implementation types may provide more methods or properties than described in the stubs, the stubs are just the minimal set of what should be defined.
+
+The `algopy_testing` module contains the implementation of the stubs and additional parts of the testing framework.
+The `algopy` module is used to alias the relevant `algopy_testing` implementations into the correct namespaces defined in the stubs.
+
+Implementations within `algopy_testing` should use `algopy_testing` types as much as possible, except for typing annotations which should use the `algopy` equivalents instead. A rule of thumb to follow is 
+withing the `algopy_testing` module, `import algopy` should only be used for type definitions e.g.
+
+```python
+import typing
+
+if typing.TYPE_CHECKING:
+    import algopy
+
+def do_something_with_bytes(a: algopy.Bytes) -> algopy.Bytes:
+    ...
+```
+
+Instance checks should be done via the `algopy_testing` namespace, this will ensure any additional attributes on the implementation will be type checked correctly e.g.
+
+```python
+import algopy_testing
+
+
+def do_something(value: object) -> None:
+    if isinstance(value, algopy_testing.Bytes):
+        raw_bytes: bytes = value.value
+        ...
+```
+
+Occasionally this will require silencing Mypy warnings, within the scope of the implementation code this is considered acceptable.
+
+```python
+import typing
+
+import algopy_testing
+
+if typing.TYPE_CHECKING:
+    import algopy
+
+def get_some_bytes() -> algopy.Bytes:
+    value = algopy_testing.Bytes(b"42")
+    # in the following statement, the return type warning is silenced as 
+    # algopy.Bytes is an alias of algopy_testing.Bytes
+    return value # type: ignore[return-value]
+```
+ 
+
+

docs/coverage.md

@@ -15,11 +15,10 @@ See which `algorand-python` stubs are implemented by the `algorand-python-testin
 | Primitives (UInt64, BigUInt, Bytes, String)                  | Implemented           |
 | urange                                                       | Implemented           |
 | All crypto ops in op.\* namespace (to be expanded in detail) | Implemented           |
-| Txn, GTxn, ITxn                                              | Implemented           |
 | arc4.\* namespace (to be expanded in detail)                 | Implemented           |
 | uenumerate                                                   | Implemented           |
-| op.ITxnCreate                                                | Implemented           |
 | StateTotals                                                  | Implemented           |
+| Txn, GTxn, ITxn                                              | Emulated              |
 | Asset                                                        | Emulated              |
 | Account                                                      | Emulated              |
 | Application                                                  | Emulated              |
@@ -34,10 +33,11 @@ See which `algorand-python` stubs are implemented by the `algorand-python-testin
 | log                                                          | Emulated              |
 | itxn.\* namespace (inner transactions)                       | Emulated              |
 | gtxn.\* namespace (group transactions)                       | Emulated              |
+| op.ITxnCreate                                                | Emulated              |
 | ensure_budget                                                | Mockable              |
 | op.EllipticCurve                                             | Mockable              |
-| op.AssetParamsGet                                            | Mockable              |
-| op.AppParamsGet                                              | Mockable              |
-| op.AppLocal                                                  | Mockable              |
-| op.AppGlobal                                                 | Mockable              |
-| op.AcctParamsGet                                             | Mockable              |
+| op.AssetParamsGet                                            | Emulated              |
+| op.AppParamsGet                                              | Emulated              |
+| op.AppLocal                                                  | Emulated              |
+| op.AppGlobal                                                 | Emulated              |
+| op.AcctParamsGet                                             | Emulated              |

examples/README.md

@@ -0,0 +1,56 @@
+# Algorand Python Testing examples
+
+This directory contains examples demonstrating how to use algorand-python-testing to test smart contracts written in Algorand Python.
+
+## Overview
+
+`algorand-python-testing` provides a powerful framework for unit testing Algorand smart contracts written in Algorand Python. It allows developers to emulate AVM behavior in a Python interpreter, making it easier to write and run tests for your Algorand applications.
+
+## Key Features
+
+1. **Test Environment Setup**: Use the `context` fixture to set up a test environment that emulates AVM behavior.
+2. **Arrange, Act, Assert Pattern**: Examples follow the 'arrange, act, assert' pattern for clear and structured tests.
+3. **Asset and Account Creation**: Easily create test assets and accounts using `context.any.asset` and `context.any.account`.
+4. **State Manipulation**: Test global and local state changes in your smart contracts.
+5. **ABI Method Testing**: Examples of how to test ABI methods in your smart contracts.
+
+## Quickstart
+
+To run the examples in a self-contained manner:
+
+1. Install [hatch](https://hatch.pypa.io/latest/)
+2. From the root of the repo, run:
+
+```bash
+hatch run examples:test examples/<example_directory>
+```
+
+Replace `<example_directory>` with the specific example you want to run (e.g., `auction`, `abi`, etc.).
+
+## Writing Tests
+
+When writing tests for your Algorand smart contracts:
+
+1. Use the `context` fixture to set up your test environment.
+2. Create test assets and accounts as needed using `context.any.asset` and `context.any.account`.
+3. Interact with your smart contract methods.
+4. Assert the expected outcomes, including state changes and transaction results.
+
+For detailed examples, refer to the individual test files in each example directory.
+
+## Contributing
+
+If you have additional examples or improvements, feel free to contribute by submitting a pull request.
+
+1. Follow the [contribution guidelines](https://github.com/algorandfoundation/algorand-python-testing/blob/main/CONTRIBUTING.md).
+2. Create a new folder under `examples/` with your contract/signature and test files.
+3. If you are contributing a smart contract example make sure to name the file `contract.py`. For logic signatures, name it `signature.py`.
+4. For test files use `test_contract.py` or `test_signature.py` respectively.
+5. Use the default PR template when opening a PR, and describe what sort of example are you adding as well as which feature of `algorand-python-testing` it demonstrates.
+
+## Resources
+
+-   [Algorand Developer Documentation](https://developer.algorand.org/)
+-   [algorand-python](https://algorandfoundation.github.io/puya/)
+-   [algorand-python-testing](https://algorandfoundation.github.io/algorand-python-testing/)
+-   ['arrange, act, assert' patern](https://automationpanda.com/2020/07/07/arrange-act-assert-a-pattern-for-writing-good-tests/)