Skip to content

Fix the extended Arabic ranges in HARAKAT_RE - #107

Open
anqorithm wants to merge 2 commits into
mpcabd:masterfrom
anqorithm:fix/harakat-ranges
Open

anqorithm wants to merge 2 commits into
mpcabd:masterfrom
anqorithm:fix/harakat-ranges

Conversation

@anqorithm

@anqorithm anqorithm commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #63

The issue is right about both lines. Checked against Unicode 16:

  • ࣔ-࣡ and ࣔ-࣭ overlap, and together they match U+08E2 ARABIC DISPUTED END OF AYAH. That's a format character (Cf), not a mark, so it was deleted along with the harakat.
  • 19 combining marks weren't matched: U+0897–089F and U+08CA–08D3, which were added in Unicode 9 to 15. When one of them sits between two letters, the letters stop joining. For example, ب + U+08D0 + ب gave two isolated behs instead of initial and final.

The extended ranges are now ࢗ-࢟, ࣊-࣡ and ࣣ-ࣿ. Every Arabic Mn character in U+0600–08FF is now matched.

Behaviour change to check: U+08E2 is no longer treated as a haraka, so delete_harakat keeps it. That seems right for a Quranic sign, but it's your call. I left U+06E5 and U+06E6 (small waw and small yeh, Lm) as they were.

Testing: added test_006_harakat_ranges.py, which checks joining across the new marks and that U+08E2 is kept. Both tests fail without this change, and the whole suite passes. I tested on Python 3.11, which ships Unicode 14, so U+0897 (added in 15) is covered by the regex but wasn't checked against a Unicode 15 database.

CI follow-up: restrict the credentialed Claude review job to branches in this repository. Fork PRs cannot receive its OAuth secret or write OIDC token; the ordinary test workflow remains enabled for forks. Local validation: 20 passed, 3 optional-dependency skips, plus actionlint. Upstream CI now passes on Python 3.10–3.15; the credentialed review job is skipped for this fork.

Remove the overlapping 08D4 range, which matched U+08E2, and add the combining marks at U+0897-089F and U+08CA-08D3.

Signed-off-by: anqorithm <61029571+anqorithm@users.noreply.github.com>
@anqorithm

Copy link
Copy Markdown
Contributor Author

Fixed the fork-only review failure in 55cc668. The credentialed Claude review job now runs only for same-repository branches; ordinary CI continues to test fork PRs. Upstream CI passes on Python 3.10–3.15, and the Claude job correctly skips for this fork. Locally: 20 tests passed, 3 optional-dependency skips, and actionlint passed.

Validation report

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

About HARAKAT_RE

1 participant