PHP Parser Integration Problems – FileNotFoundError, Language Version Incompatibility, and Tree-sitter API Issues

[ghost-in-the-system]

Overview

When attempting to use code_ast==0.1.1 to parse PHP files, I encountered a series of critical issues related to Tree-sitter language integration. These include missing parser source files, language version incompatibility, and API mismatches after updating dependencies. While I managed to implement a local workaround, these problems indicate that PHP support in code_ast is currently broken and needs a proper fix in the package.

Steps to Reproduce

1. Install code_ast==0.1.1 in a fresh Python 3.12 environment.
2. Attempt to parse a PHP file using:

import code_ast
ast_obj = code_ast.ast(source_code, lang="php", syntax_error="warn")

Problems Observed

1. FileNotFoundError: Missing PHP Parser Source Directory

FileNotFoundError: [Errno 2] No such file or directory: '/.../site-packages/build/tree-sitter-php/src/parser.c'

• The code_ast package tries to compile the PHP Tree-sitter parser from a path that does not include the necessary "php" subdirectory.
• This results in a missing file error.

2. ValueError: Incompatible Language Version

After manually patching the path in load_language() to include "php", the following error occurs:

ValueError: Incompatible Language version 15. Must be between 13 and 14

• Indicates a mismatch between the Tree-sitter runtime version and the PHP grammar version.

3. TypeError: Tree-sitter API Incompatibility

After updating both tree-sitter and tree-sitter-php to version 0.23.2 (to resolve the version mismatch), the following error appears:

TypeError: __init__() takes exactly 1 argument (2 given)

• This suggests that the code_ast package is not compatible with the latest Tree-sitter Python API.

Workaround Applied

To make PHP parsing work locally:

• Moved the PHP path patching logic into the ASTParser constructor.
• Imported tree_sitter_php directly and instantiated the language and parser objects using the updated API.

Patch Example:

if lang == "php":
    import tree_sitter_php as ts_php
    self.lang = Language(ts_php.language_php())
    self.parser = Parser(self.lang)
else:
    self.lang = load_language(lang)
    # existing parser initialization

• This workaround bypasses the broken path and version issues, but it is not a long-term solution.

Additional Information

• The workaround above is a temporary fix and not suitable for production.
• These issues prevent out-of-the-box PHP parsing and may affect other languages if their grammars or APIs change.
• See also: tree-sitter/tree-sitter-rust#273 for similar versioning issues.

Environment

• OS: macOS
• Python: 3.12
• code_ast: 0.1.1
• tree-sitter: 0.23.2 (after update)
• tree-sitter-php: 0.23.2 (after update)

[cedricrupb]

Hey! Thank you for noticing. There are actually two independent problem:

1. We are currently stuck at tree-sitter==0.21.1 because in 0.22.0 a new way to install languages is introduced. Eventually, we might have to support newer versions, but I am not completely sure about all the requiring users to install further libraries business.
2. We are currently not parsing the tree-sitter.json file. This is typically no problem (e.g. for c, python, java, ...). However, PHP defines two grammars (PHP with HTML? and PHP_only) and hence defines different subpaths were the parser implementation can be found. We probably have to parse this file to identify the proper parser implementation.

I have to handle both cases sometimes in the future. I am not sure when I get to it.