Skip to content

refactor(parse): unify document parsing with AnyDoc - #3957

Draft
qin-ctx wants to merge 3 commits into
mainfrom
refactor/anydoc-document-parsers
Draft

refactor(parse): unify document parsing with AnyDoc#3957
qin-ctx wants to merge 3 commits into
mainfrom
refactor/anydoc-document-parsers

Conversation

@qin-ctx

@qin-ctx qin-ctx commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Description

Unify the existing Office and EPUB parsing paths on AnyDoc, while keeping image extraction under OpenViking's ingestion policy. Replace the previous PDF/MinerU path with pdf-inspector-owned text and layout extraction plus pdfplumber-owned image extraction.

Human Involvement

  • A human participated in the implementation or review loop
  • This PR was generated entirely by AI agents without human participation in the loop

Related Issue

N/A

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test update

Changes Made

  • Route .doc, .docx, .pptx, .xls, .xlsx, .xlsm, and .epub through AnyDoc's public document model, preserving supported embedded-image positions and rendering PowerPoint speaker notes as ordered Speaker Notes sections in the same resource.
  • Make pdf-inspector authoritative for PDF page text, headings, tables, reading order, and OCR diagnostics; retain pdfplumber only for per-page image extraction after page text.
  • Remove the legacy format-specific parsers, MinerU integration, ExcelConfig, obsolete production dependencies, and outdated tests; share one OpenViking image validation policy across Markdown, AnyDoc, and PDF ingestion.

Testing

  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have tested this on the following platforms:
    • Linux
    • macOS
    • Windows

Validated with:

  • pytest -q -o addopts='' tests/parse tests/test_config_loader.py (494 passed, 31 skipped)
  • Ruff, targeted mypy, compileall, uv lock --check, and git diff --check
  • Existing-test audit confirming no new test files, test functions, or parameterized cases

Checklist

  • My code follows the project's coding style
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

Screenshots (if applicable)

N/A

Additional Notes

  • parsers.excel is removed and now rejected instead of being silently ignored.
  • AnyDoc 0.1.8 does not expose embedded XLSX images through its document assets, so Excel currently guarantees worksheet Markdown only. No parallel openpyxl image extraction path is introduced.
  • PDF pages requiring OCR produce explicit warnings; this change intentionally adds no OCR fallback.

Replace unmaintained format-specific parsers and MinerU integration with explicit document parsing ownership while preserving embedded images and presentation notes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

1 participant