Skip to content

fix!: preserve streamed log severity and drain async logs on stop - #1229

Open
pcbeingused333 wants to merge 4 commits into
apify:masterfrom
pcbeingused333:fix/c3526-streamed-log
Open

pcbeingused333 wants to merge 4 commits into
apify:masterfrom
pcbeingused333:fix/c3526-streamed-log

Conversation

@pcbeingused333

Copy link
Copy Markdown

Fixes #1180

Streamed logs currently infer severity from substrings anywhere in a message, so INFO Saved ERROR_PAGE screenshot becomes ERROR. Read only the level token after the platform timestamp and optional bracketed logger name, preserving the standard logging aliases and falling back to INFO when the token is absent or unknown.

Give asynchronous log streaming a one-second grace period on stop so in-flight final chunks can arrive. Cancel streams that remain open after that period. Caller cancellation cancels and awaits the streaming task, clears its handle, and propagates without fetching the whole log. Normal stopping retains buffered-tail flushing and the whole-log fallback when no bytes arrived.

BREAKING CHANGE: Redirected log severity now comes from the level token rather than words in the message body. Asynchronous stop may wait up to one second before cancelling an open stream.

Testing and validation

Unit tests cover misleading message text, bracketed logger names, unknown or absent levels, delivery on the next event-loop turn, and cancellation of a silent stream. Explicit caller cancellation leaves the stopping task cancelled, and an enclosing asyncio.timeout raises TimeoutError; neither path fetches the whole log. The focused suite passes all 16 tests (python -m pytest tests/unit/test_streamed_log.py -q). The two caller-cancellation regressions fail against the previous revision (14 passed, 2 failed). Ruff lint and format checks pass.

AI assistance: This change and its tests were prepared with OpenAI Codex.

Read the level token after the platform timestamp and optional logger prefix. Allow in-flight asynchronous log chunks to finish for one second before cancellation.

BREAKING CHANGE: Redirected severity follows the level token rather than message substrings; async stop may wait up to one second before cancelling.
@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 15, 2026
@vdusek
vdusek requested a review from Pijukatel November 16, 2026 11:00
Comment thread src/apify_client/_streamed_log.py Outdated
"""Guess the log level from the message."""
# Using only levels explicitly mentioned in the logging module
"""Read the level token after the platform timestamp and optional bracketed logger name."""
match = re.match(r'^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z[ \t]+(?:\[[^\]\n]*\][ \t]+)?(\S+)', message)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Usually the platform logs have color coding around them. This regexp does not handle that majority use-case. Also nested level redirects will probably not match.

This will always be a guessing. No way to make this 100% correct and not worth putting much effort into it as user can define their own logger.

So I would be in favor of keeping this as simple as possible.

Maybe small improvement and still simple enough would be printing the first matching level.

Something like

ANSI_CODE = re.compile(r'\x1b\[[0-9;]*m')
FIRST_WORD_PATTERN = re.compile(r'\b(CRITICAL|FATAL|ERROR|WARNING|WARN|INFO|DEBUG)\b')
LEVELS = logging.getLevelNamesMapping()

def guess_first_word(message: str) -> int:
    """First whole-word level name anywhere in the line, with ANSI color codes removed first."""
    match = FIRST_WORD_PATTERN.search(ANSI_CODE.sub('', message))
    return LEVELS[match[1]] if match else logging.INFO

Comment thread src/apify_client/_streamed_log.py Outdated
Comment on lines +300 to +312
try:
await asyncio.wait_for(asyncio.shield(self._streaming_task), timeout=self._stop_timeout_s)
except TimeoutError:
self._streaming_task.cancel()
except asyncio.CancelledError:
self._streaming_task.cancel()
try:
await self._streaming_task
except asyncio.CancelledError:
pass
finally:
self._streaming_task = None
raise

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are on Python 3.11+

Can t we do just something as simple as

try:
   await asyncio.wait_for(self._streaming_task, timeout=self._stop_timeout_s)
except (TimeoutError, asyncio.CancelledError):
   pass

Use the first whole-word level name with ANSI colors removed, and wait for the streaming task with a plain wait_for, as suggested in review.
@pcbeingused333

Copy link
Copy Markdown
Author

Thanks @Pijukatel. I pushed e71c4f6 with the review applied: the level guess now strips ANSI codes and only looks at the first level, and the async stop wait is simplified to a plain wait_for. One addition: cancellation from the caller propagates through cancelling(), and a unit test covers it.

This comment was drafted with Claude Code.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Streamed log redirection guesses levels from any substring and drops in-flight lines on async stop

2 participants