gh-144415: Android testbed fixes

[mhsmith]

Fixes for several issues found while building NumPy and related packages:

• Gradle automatically decompresses assets whose filenames end with ".gz", which broke some packages' tests.

  • Worked around by escaping these filenames.

• The stdout/stderr logging couldn't distinguish between log messages which were triggered by a newline, and those triggered by a manual call to flush. This particularly affected pytest's "one dot per test" progress indicator, which was printing every dot on a separate line.

  • I didn't end up needing the "trailing space" idea I mentioned the other day, because it turns out the newline information was already in the log, it just wasn't visible via the logcat tool in --format tag mode. Fixed by switching to --binary mode.
  • The logcat tool on API level 33 buffers its output incorrectly in --binary mode, causing lines to be delayed or lost. Fixed by reducing the tesbed to API level 32.

• Closes Android testbed logs have excessive newlines #144415

[mhsmith]

!buildbot android

[mhsmith]

!buildbot aarch64 android

[mhsmith]

!buildbot android

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @mhsmith for commit 29ec458 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F142912%2Fmerge

The command will test the builders whose names match following regular expression: android

The builders matched are:

• aarch64 Android PR
• AMD64 Android PR

[mhsmith]

@freakboy3742: I didn't manage to complete the emulator RAM size fix today, so let's merge this PR without it. So far RAM size has only been a blocker for Pandas, and that's unlikely to be ready to release within the next two months anyway, given how many other things have to be released first.

GitHub Actions is broken at the moment; please rerun CI once that's resolved.

[freakboy3742]

The buildbot and CI runs suggest this change works; but I want to make sure I follow how these changes avoid the "pytest progress" issue.

I can see this moves to a "raw binary" mode of reading the ADB log, which means we get the raw messages being posted rather than the logged output.

That then means that we're reading "raw" log output at the level of individual messages, rather than log output that is buffered by newline/flush elements; so when we're streaming log output, we have visibility on the newlines that have been printed to stdout/err, rather than that content being part of the buffering process.

The "leading space" trick is needed to preserve newlines that would otherwise been trimmed from the log by an internal logcat mechanism. AFAICT, this does mean that successive writes of hello and \nworld would result in a space at the end of the "hello" as printed to the screen - but that's visually inert unless you're really unlucky and trigger size of screen line wrapping, so it's worth living with.

It also means that anyone monitoring the "text mode" ADB stream will still get "one dot per line" output for pytest suites etc (e.g., Briefcase's Android runner logs); but the same "binary mode" logger could be used there as well.

Have I understood what is going on here?

Assuming I have - on a meta level, I think this is a big enough change that it needs a ticket and NEWS entry - if only to provide tracking for the backports.

[mhsmith]

That's mostly correct, except for a couple of details:

That then means that we're reading "raw" log output at the level of individual messages, rather than log output that is buffered by newline/flush elements

We were always getting individual messages: the Logcat system doesn't do any buffering itself. The issue was that when you use adb logcat in text mode, it prints a newline after every log message regardless of whether the original message had a newline or not. This makes it difficult to reconstruct non-line-based output like pytest's progress bar.

AFAICT, this does mean that successive writes of hello and \nworld would result in a space at the end of the "hello" as printed to the screen - but that's visually inert unless you're really unlucky and trigger size of screen line wrapping, so it's worth living with.

If you called flush after each of these writes, then the behavior would be as you say. However, if you didn't manually flush, then Python itself would buffer until it hit the newline, so the newline would be logged in the same message as the "hello", it would no longer be at the start of a message, and no extra space would be added. This is probably the more common case.

I think this is a big enough change that it needs a ticket and NEWS entry

Done.

[freakboy3742]

Thanks for those explanations 👍

[miss-islington-app]

Thanks @mhsmith for the PR, and @freakboy3742 for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14.
🐍🍒⛏🤖

[bedevere-app]

GH-144416 is a backport of this pull request to the 3.14 branch.

[bedevere-app]

GH-144417 is a backport of this pull request to the 3.13 branch.