Implement iterator to lazily go through the paged response #3365

Activity

1. harishkesavarao commented on May 17, 2026

   

   harishkesavarao

   on May 17, 2026

   More actions

   Is this open for me to work on? I would like to work on it if so.
2. ebyhr commented on May 17, 2026

   

   ebyhr

   on May 17, 2026

   Member

   More actions

   I suggest discussing and reaching a consensus before beginning your work. This change may become unnecessary once we implement the page-size option in #3377.
3. kevinjqliu commented on May 24, 2026

   

   kevinjqliu

   on May 24, 2026

   Contributor

   More actions

   #3377 is merged, thanks! An iterator might still be useful here to lazily fetch pages on the client side. page-size only controls the size of each individual response, not when or whether subsequent pages are fetched. Without an iterator, the client eagerly requests all pages upfront, even if the caller only needs the first few results.
4. Fokko commented on May 25, 2026

   

   Fokko

   on May 25, 2026

   ContributorAuthor

   More actions

   I think we need to revive #2172. This is a breaking change when it comes to the returned types, but I think that's it. We might want to double check for len if that works the same way.
5. rambleraptor commented on May 26, 2026

   

   rambleraptor

   on May 26, 2026

   Last edited by rambleraptor

   Collaborator

   More actions

   Iterators don't have a len method, so it would throw an error.

   What would we think about creating our own list? That way, we can simply add in pagination ourselves while keeping the list syntax. Here's some quick example code (I think this would work)

   class PaginationList(list):
        # The list stores a token + callback to fetch new results.
        self._pagination_token = ""
        self._pagination_callback: Callable = None
   
        # Calls the next page
        # If pagination has no results, list will be set to empty.
        def next_page() -> None:
   
   
   # Example
   tables: PaginationList = catalog.list_tables()
   while len(tables) > 0:
        for table in tables:
            print(table)
        tables.next_page()
             
   
6. GayathriSrividya

   added a commit that references this issue on May 27, 2026

   Fix: implement lazy pagination iterator for catalog list methods (apa…

   ...

   7e1538e
7. Fokko commented on Jun 6, 2026

   

   Fokko

   on Jun 6, 2026

   ContributorAuthor

   More actions

   Yes, I've also stumbled upon it in #2172 (comment)

   I think #3454 is the right approach, this way we don't have to introduce any breaking changes.