Feat/page limit in pdf

[mackurzawa]

Description

Handled PageCountExceededError from the unstructured open-source library. Added the UNSTRUCTURED_MAX_PDF_PAGES environment variable, which indicates the maximum number of pages in a PDF file when the hi_res strategy is chosen (either directly by the consumer of the API or transformed from the auto strategy).

Testing

# works, number of pages is not exceeding UNSTRUCTURED_MAX_PDF_PAGES (300 by default), auto strategy transforms to hi_res
curl -X 'POST'   'http://localhost:8000/general/v0/general'   -H 'accept: application/json'   -H 'Content-Type: multipart/form-data'   -F 'files=@sample-docs/DA-1p-with-duplicate-pages.pdf'   -F 'output_format="text/csv"'

# on server
export UNSTRUCTURED_MAX_PDF_PAGES=2
# throws HTTPException - number of pages exceeds UNSTRUCTURED_MAX_PDF_PAGES while auto strategy transforms to hi_res
curl -X 'POST'   'http://localhost:8000/general/v0/general'   -H 'accept: application/json'   -H 'Content-Type: multipart/form-data'   -F 'files=@sample-docs/DA-1p-with-duplicate-pages.pdf'   -F 'output_format="text/csv"'

# on server
export UNSTRUCTURED_MAX_PDF_PAGES=2
# works, number of pages exceeds UNSTRUCTURED_MAX_PDF_PAGES, but fast strategy is chosen
curl -X 'POST'   'http://localhost:8000/general/v0/general'   -H 'accept: application/json'   -H 'Content-Type: multipart/form-data'   -F 'files=@sample-docs/DA-1p-with-duplicate-pages.pdf'   -F 'output_format="text/csv"' -F 'strategy=fast'

[pawel-kmiecik]

Let's name it more precisely - UNSTRUCTURED_PDF_HI_RES_MAX_PAGES