rust/library
bors cc9bb1522e Auto merge of #83342 - Count-Count:win-console-incomplete-utf8, r=m-ou-se
Allow writing of incomplete UTF-8 sequences to the Windows console via stdout/stderr

# Problem
Writes of just an incomplete UTF-8 byte sequence (e.g. `b"\xC3"` or `b"\xF0\x9F"`)  to stdout/stderr with a Windows console attached error with `io::ErrorKind::InvalidData, "Windows stdio in console mode does not support writing non-UTF-8 byte sequences"` even though further writes could complete the codepoint. This is currently a rare occurence since the [linewritershim](2c56ea38b0/library/std/src/io/buffered/linewritershim.rs) implementation flushes complete lines immediately and buffers up to 1024 bytes for incomplete lines. It can still happen as described in #83258.

The problem will become more pronounced once the developer can switch stdout/stderr from line-buffered to block-buffered or immediate when the changes in the "Switchable buffering for Stdout" pull request (#78515) get merged.

# Patch description
If there is at least one valid UTF-8 codepoint all valid UTF-8 is passed through to the extracted `write_valid_utf8_to_console()` fn. The new code only comes into play if `write()` is being passed a short byte slice comprising an incomplete UTF-8 codepoint. In this case up to three bytes are buffered in the `IncompleteUtf8` struct associated with `Stdout` / `Stderr`. The bytes are accepted one at a time. As soon as an error can be detected `io::ErrorKind::InvalidData, "Windows stdio in console mode does not support writing non-UTF-8 byte sequences"` is returned. Once a complete UTF-8 codepoint is received it is passed to the `write_valid_utf8_to_console()` and the buffer length is set to zero.

Calling `flush()` will neither error nor write anything if an incomplete codepoint is present in the buffer.

# Tests
Currently there are no Windows-specific tests for console writing code at all. Writing (regression) tests for this problem is a bit challenging since unit tests and UI tests don't run in a console and suddenly popping up another console window might be surprising to developers running the testsuite and it might not work at all in CI builds. To just test the new functionality in unit tests the code would need to be refactored. Some guidance on how to proceed would be appreciated.

# Public API changes
* `std::str::verifications::utf8_char_width()` would be exposed as `std::str::utf8_char_width()` behind the "str_internals" feature gate.

# Related issues
* Fixes #83258.
* PR #78515 will exacerbate the problem.

# Open questions
* Add tests?
* Squash into one commit with better commit message?
2021-09-02 03:31:17 +00:00
..
alloc Rollup merge of #88040 - nbdd0121:btreemap, r=m-ou-se 2021-09-01 09:23:23 +02:00
backtrace@4f925f8d81 Update the backtrace crate in libstd 2021-08-19 07:31:49 -07:00
core Auto merge of #83342 - Count-Count:win-console-incomplete-utf8, r=m-ou-se 2021-09-02 03:31:17 +00:00
panic_abort Use C-unwind ABI for __rust_start_panic in panic_abort 2021-08-05 18:01:17 +08:00
panic_unwind STD support for the ESP-IDF framework 2021-08-10 12:09:00 +03:00
proc_macro Fix typos “a”→“an” 2021-08-22 15:35:11 +02:00
profiler_builtins rfc3052: Remove authors field from Cargo manifests 2021-07-29 14:56:05 -07:00
rtstartup
rustc-std-workspace-alloc rfc3052: Remove authors field from Cargo manifests 2021-07-29 14:56:05 -07:00
rustc-std-workspace-core rfc3052: Remove authors field from Cargo manifests 2021-07-29 14:56:05 -07:00
rustc-std-workspace-std rfc3052: Remove authors field from Cargo manifests 2021-07-29 14:56:05 -07:00
std Auto merge of #83342 - Count-Count:win-console-incomplete-utf8, r=m-ou-se 2021-09-02 03:31:17 +00:00
stdarch@89b0e355bc Update the stdarch submodule 2021-08-24 09:02:44 -07:00
test rfc3052: Remove authors field from Cargo manifests 2021-07-29 14:56:05 -07:00
unwind build llvm libunwind.a in rustbuild 2021-08-28 14:14:22 +08:00