Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/contribute/code_guide.rst
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,8 @@ Python Code Styles
Writing Python Tests
--------------------
We use `pytest <https://docs.pytest.org/en/stable/>`_ for all python testing. ``tests/python`` contains all the tests.
See :doc:`testing` for details on running tests, target parametrization,
and the target-specific marks used by CI.

If you want your test to run over a variety of targets, use the :py:func:`tvm.testing.parametrize_targets` decorator. For example:

Expand Down
1 change: 1 addition & 0 deletions docs/contribute/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ Here are guidelines for contributing to various aspect of the project:
committer_guide
document
code_guide
testing
git_howto
ci
release_process
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,17 @@
specific language governing permissions and limitations
under the License.

Testing TVM
===========

This page describes how to write and run Python tests for TVM,
including the target parametrization utilities used by CI.

Python Target Parametrization
=============================
-----------------------------

Summary
-------
~~~~~~~

For any supported runtime, TVM should produce numerically
correct results. Therefore, when writing unit tests that validate
Expand All @@ -28,7 +34,7 @@ runtimes. Since this is a very common use case, TVM has helper
functions to parametrize unit tests such that they will run on all
targets that are enabled and have a compatible device.

A single python function in the test suite can expand to several
A single Python function in the test suite can expand to several
parameterized unit tests, each of which tests a single target device.
In order for a test to be run, all of the following must be true.

Expand All @@ -47,7 +53,7 @@ In order for a test to be run, all of the following must be true.
runtime.

Unit-Test File Contents
-----------------------
~~~~~~~~~~~~~~~~~~~~~~~

.. _pytest-marks: https://docs.pytest.org/en/stable/how-to/mark.html

Expand Down Expand Up @@ -160,27 +166,28 @@ listed as skipped.
There also exists a ``tvm.testing.enabled_targets()`` that returns
all targets that are enabled and runnable on the current machine,
based on the environment variable ``TVM_TEST_TARGETS``, the build
configuration, and the physical hardware present. Most current tests
configuration, and the physical hardware present. Some legacy tests
explicitly loop over the targets returned from ``enabled_targets()``,
but it should not be used for new tests. The pytest output for this
style silently skips runtimes that are disabled in ``config.cmake``,
or do not have a device on which they can run. In addition, the test
halts on the first target to fail, which is ambiguous as to whether
the error occurs on a particular target, or on every target.
but this style should not be used for new tests. The pytest output
for this style silently skips runtimes that are disabled in
``config.cmake``, or do not have a device on which they can run. In
addition, the test halts on the first target to fail, which is
ambiguous as to whether the error occurs on a particular target, or on
every target.

.. code-block:: python

# Old style, do not use.
def test_function():
for target,dev in tvm.testing.enabled_targets():
for target, dev in tvm.testing.enabled_targets():
# Test code goes here



Running locally
---------------
Running Locally
~~~~~~~~~~~~~~~

To run the python unit-tests locally, use the command ``pytest`` in
To run the Python unit tests locally, use the command ``pytest`` in
the ``${TVM_HOME}`` directory.

- Environment variables
Expand All @@ -206,19 +213,19 @@ the ``${TVM_HOME}`` directory.
system without a specific backend installed.

- The ``-m`` argument only runs unit tests that are tagged with a
specific pytest marker. The most frequent usage is to use ``m
gpu`` to run only tests that are marked with
specific pytest marker. The most frequent usage is to use
``-m gpu`` to run only tests that are marked with
``@pytest.mark.gpu`` and use a GPU to run. It can also be used
to run only tests that do not use a GPU, by passing ``m 'not
gpu'``.
to run only tests that do not use a GPU, by passing ``not gpu``
as the marker expression to ``-m``.

Note: This filtering takes place after the selection of targets
based on the ``TVM_TEST_TARGETS`` environment variable. Even if
``-m gpu`` is specified, if ``TVM_TEST_TARGETS`` does not
contain GPU targets, no GPU tests will be run.

Running in local docker container
---------------------------------
Running in a Local Docker Container
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. _tlcpack: https://hub.docker.com/u/tlcpack

Expand All @@ -243,7 +250,7 @@ and make a symlink from ``build`` to the appropriate folder when
entering/exiting docker.

Running in CI
-------------
~~~~~~~~~~~~~

Everything in the CI starts from the task definitions present in the
Jenkinsfile. This includes defining which docker image gets used,
Expand Down
108 changes: 55 additions & 53 deletions docs/errors.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,56 +16,58 @@
under the License.


Handle TVM Errors
=================

When running TVM, you may encounter an error message like:

.. code::

---------------------------------------------------------------
An error occurred during the execution of TVM.
For more information, please see: https://tvm.apache.org/docs/errors.html
---------------------------------------------------------------

Congratulations! You found this page. Below are some hints on how to interpret
these error messages and what you can do when they occur.

Where do these errors come from?
--------------------------------

This error is caused by an internal invariant being violated during TVM's
execution. On a technical level, the message is generated by the
``TVM_FFI_ICHECK`` macro, found in ``include/tvm/runtime/logging.h``.
The ``TVM_FFI_ICHECK`` macro is used in many places in the TVM code to assert
some condition is true during execution; any time the assertion fails, TVM
will exit with the error message shown above.

For more details about how errors are handled and generated by TVM, please
see :ref:`error-handling-guide`.

What should I do when I encounter such an error?
------------------------------------------------

First of all, *don't panic*. Well, you can panic, but it won't help.

The best course of action is to search the
`Apache TVM Discuss Forum <https://discuss.tvm.apache.org/>`_
for the error you are encountering, to see if this has been a problem
that others have encountered, and what the solution might be.
If this error is the result of a bug that has been fixed in a more
recent version of TVM, you may need to update to a newer version.

If you do not find an existing Discuss Forum thread about your
issue, you are welcome to start a new thread on the forum with details
on the problem. *Please* include in your posting the following key
pieces of information:

* The version of TVM you are using (e.g., the git commit hash of your source tree).
* Which hardware and operating system version you are running TVM on.
* Which hardware device and OS you are targeting for your TVM compilation.
* Details on the model, inputs, or other information about the workload, which can
be used to reproduce your problem.

Without these details it is very difficult for the TVM developers to do very
much to help you.
TVM Errors
==========

TVM may raise errors from Python code, from C++ code reached through the
FFI, or from generated runtime modules. Error messages usually include
a Python stack trace, and may also include a C++ stack trace when the
error crosses the TVM FFI boundary.

Some errors report invalid user input, unsupported operators, missing
runtime features, or unavailable hardware. Others report a failed
internal check, usually raised by ``TVM_FFI_ICHECK`` or
``TVM_FFI_THROW`` in C++ code. Internal check failures often indicate
that TVM reached a state that the implementation did not expect.

What to Check First
-------------------

- Make sure the TVM Python package and native libraries come from the
same build. A common symptom of a mismatched environment is importing
Python files from one checkout while loading ``libtvm`` from another.
- Check that the required runtime is enabled in ``config.cmake``. For
example, CUDA tests and CUDA compilation require a TVM build with
CUDA support enabled.
- Check that the target hardware is available to the process. GPU
tests may be skipped or fail if the device is not visible inside the
current container or environment.
- If the error occurs while importing or converting a model, reduce the
input to the smallest model, operator, or shape that reproduces the
issue.

Reporting an Issue
------------------

Search the `Apache TVM Discuss Forum <https://discuss.tvm.apache.org/>`_
and the `TVM issue tracker <https://github.com/apache/tvm/issues>`_
for the exact error message first. If you do not find an existing
report, include the following details when starting a new discussion or
filing an issue:

- The TVM version or git commit hash.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a quick comment that we may have to provide a script for collecting environment so users don't have to manually collect them.
I've prototyped it based on pytorch's collect_env.py.
https://github.com/mshr-h/tvm-collect-env

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, that makes sense. Would update the doc once you finish

- The Python version, operating system, and hardware.
- The target and runtime being used, such as LLVM, CUDA, Vulkan, or RPC.
- The relevant build configuration from ``config.cmake``.
- A minimal script, model, input shape, or IR module that reproduces the
failure.
- The full error message, including both Python and C++ stack traces
when present.

Developer Notes
---------------

For guidance on raising typed errors from TVM code, see
:ref:`error-handling-guide`. That guide covers when to use specific
error types, how C++ error prefixes map to Python exceptions, and how
``TVM_FFI_ICHECK`` interacts with TVM's error handling.
28 changes: 0 additions & 28 deletions docs/how_to/dev/index.rst

This file was deleted.

Loading
Loading