Skip to content

Update to use iterative polling (while loop) instead of recursion cursor - #154

Open
TriDuong-TD wants to merge 1 commit into
masterfrom
INTEG-6317
Open

TriDuong-TD wants to merge 1 commit into
masterfrom
INTEG-6317

Conversation

@TriDuong-TD

Copy link
Copy Markdown
Contributor

Summary

Cursor._do_execute() polled for job status by calling itself after each time.sleep(wait_interval). Every poll cycle added a stack frame, so a long-running job eventually exceeded Python's default recursion limit (1000) and raised RecursionError.

This PR replaces the recursion with a while self._rows is None: loop. Behavior is otherwise unchanged.

Background

A customer's production workflow failed on a long-running query.

  • Failing task: mapp_custom_attributes+group_api^sub (py> operator)
  • Call path: pytd client.query() → tdclient.Cursor._do_execute()
  • Failure: RecursionError: maximum recursion depth exceeded after 967 recursive calls, about 4h43m into the job
  • Affected versions: the recursive implementation is also in the latest release, v1.7.0

A temporary workaround (sys.setrecursionlimit(10000)) has been shared with the customer. It only raises the ceiling, so jobs that run long enough would still fail.

Purpose

  • Fix _do_execute() in tdclient/cursor.py to use iterative polling (while loop) instead of recursion.

@kietdo360 kietdo360 changed the title INTEG-6317: Update to use iterative polling (while loop) instead of recursion cursor Update to use iterative polling (while loop) instead of recursion cursor Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant