diff --git a/library/test/src/cli.rs b/library/test/src/cli.rs index c0b5197e997..b7791b1b24d 100644 --- a/library/test/src/cli.rs +++ b/library/test/src/cli.rs @@ -51,14 +51,8 @@ fn optgroups() -> getopts::Options { .optflag("", "test", "Run tests and not benchmarks") .optflag("", "bench", "Run benchmarks instead of tests") .optflag("", "list", "List all tests and benchmarks") - .optflag("h", "help", "Display this message (longer with --help)") - .optopt( - "", - "logfile", - "Write logs to the specified file instead \ - of stdout", - "PATH", - ) + .optflag("h", "help", "Display this message") + .optopt("", "logfile", "Write logs to the specified file", "PATH") .optflag( "", "nocapture", diff --git a/src/doc/rustc/src/SUMMARY.md b/src/doc/rustc/src/SUMMARY.md index dd198615736..af5834525fa 100644 --- a/src/doc/rustc/src/SUMMARY.md +++ b/src/doc/rustc/src/SUMMARY.md @@ -11,6 +11,7 @@ - [Deny-by-default lints](lints/listing/deny-by-default.md) - [Codegen options](codegen-options/index.md) - [JSON Output](json.md) +- [Tests](tests/index.md) - [Platform Support](platform-support.md) - [Targets](targets/index.md) - [Built-in Targets](targets/built-in.md) diff --git a/src/doc/rustc/src/command-line-arguments.md b/src/doc/rustc/src/command-line-arguments.md index 3d6579250a0..7d3cad7988c 100644 --- a/src/doc/rustc/src/command-line-arguments.md +++ b/src/doc/rustc/src/command-line-arguments.md @@ -204,7 +204,8 @@ out a longer explanation of a given error. ## `--test`: build a test harness When compiling this crate, `rustc` will ignore your `main` function -and instead produce a test harness. +and instead produce a test harness. See the [Tests chapter](tests/index.md) +for more information about tests. ## `--target`: select a target triple to build diff --git a/src/doc/rustc/src/tests/index.md b/src/doc/rustc/src/tests/index.md new file mode 100644 index 00000000000..ec23d4fe0db --- /dev/null +++ b/src/doc/rustc/src/tests/index.md @@ -0,0 +1,284 @@ +# Tests + +`rustc` has a built-in facility for building and running tests for a crate. +More information about writing and running tests may be found in the [Testing +Chapter] of the Rust Programming Language book. + +Tests are written as free functions with the [`#[test]` +attribute][attribute-test]. For example: + +```rust +#[test] +fn it_works() { + assert_eq!(2 + 2, 4); +} +``` + +Tests "pass" if they return without an error. They "fail" if they [panic], or +return a [`Result`] with an error. + +By passing the [`--test` option] to `rustc`, the compiler will build the crate +in a special mode to construct an executable that will run the tests in the +crate. The `--test` flag will make the following changes: + +* The crate will be built as a `bin` [crate type], forcing it to be an + executable. +* Links the executable with [`libtest`], the test harness that is part of the + standard library, which handles running the tests. +* Synthesizes a [`main` function] which will process command-line arguments + and run the tests. This new `main` function will replace any existing `main` + function as the entry point of the executable, though the existing `main` + will still be compiled. +* Enables the [`test` cfg option], which allows your code to use conditional + compilation to detect if it is being built as a test. +* Enables building of functions annotated with the [`test`][attribute-test] + and [`bench`](#benchmarks) attributes, which will be run by the test + harness. + +After the executable is created, you can run it to execute the tests and +receive a report on what passes and fails. If you are using [Cargo] to manage +your project, it has a built-in [`cargo test`] command which handles all of +this automatically. An example of the output looks like this: + +```text +running 4 tests +test it_works ... ok +test check_valid_args ... ok +test invalid_characters ... ok +test walks_the_dog ... ok + +test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s +``` + +> **Note**: Tests must be built with the [`unwind` panic +> strategy][panic-strategy]. This is because all tests run in the same +> process, and they are intended to catch panics, which is not possible with +> the `abort` strategy. See the unstable [`-Z panic-abort-tests`] option for +> experimental support of the `abort` strategy by spawning tests in separate +> processes. + +## Test attributes + +Tests are indicated using attributes on free functions. The following +attributes are used for testing, see the linked documentation for more +details: + +* [`#[test]`][attribute-test] — Indicates a function is a test to be run. +* `#[bench]` — Indicates a function is a benchmark to be + run. Benchmarks are currently unstable and only available in the nightly + channel, see the [unstable docs][bench-docs] for more details. +* [`#[should_panic]`][attribute-should_panic] — Indicates that the test + function will only pass if the function [panics][panic]. +* [`#[ignore]`][attribute-ignore] — Indicates that the test function will be + compiled, but not run by default. See the [`--ignored`](#--ignored) and + [`--include-ignored`](#--include-ignored) options to run these tests. + +## CLI arguments + +The libtest harness has several command-line arguments to control its +behavior. + +> Note: When running with [`cargo test`], the libtest CLI arguments must be +> passed after the `--` argument to differentiate between flags for Cargo and +> those for the harness. For example: `cargo test -- --nocapture` + +### Filters + +Positional arguments (those without a `-` prefix) are treated as filters which +will only run tests whose name matches one of those strings. The filter will +match any substring found in the full path of the test function. For example, +if the test function `it_works` is located in the module +`utils::paths::tests`, then any of the filters `works`, `path`, `utils::`, or +`utils::paths::tests::it_works` will match that test. + +See [Selection options](#selection-options) for more options to control which +tests are run. + +### Action options + +The following options perform different actions other than running tests. + +#### `--list` + +Prints a list of all tests and benchmarks. Does not run any of the tests. +[Filters](#filters) can be used to list only matching tests. + +#### `-h`, `--help` + +Displays usage information and command-line options. + +### Selection options + +The following options change how tests are selected. + +#### `--test` + +This is the default mode where all tests will be run as well as running all +benchmarks with only a single iteration (to ensure the benchmark works, +without taking the time to actually perform benchmarking). This can be +combined with the `--bench` flag to run both tests and perform full +benchmarking. + +#### `--bench` + +This runs in a mode where tests are ignored, and only runs benchmarks. This +can be combined with `--test` to run both benchmarks and tests. + +#### `--exact` + +This forces [filters](#filters) to match the full path of the test exactly. +For example, if the test `it_works` is in the module `utils::paths::tests`, +then only the string `utils::paths::tests::it_works` will match that test. + +#### `--skip` _FILTER_ + +Skips any tests whose name contains the given _FILTER_ string. This flag may +be passed multiple times. + +#### `--ignored` + +Runs only tests that are marked with the [`ignore` +attribute][attribute-ignore]. + +#### `--include-ignored` + +Runs both [ignored](#--ignored) and non-ignored tests. + +#### `--exclude-should-panic` + +Excludes tests marked with the [`should_panic` +attribute][attribute-should_panic]. + +⚠️ 🚧 This option is [unstable](#unstable-options), and requires the `-Z +unstable-options` flag. See [tracking issue +#82348](https://github.com/rust-lang/rust/issues/82348) for more information. + +### Execution options + +The following options affect how tests are executed. + +#### `--test-threads` _NUM_THREADS_ + +Sets the number of threads to use for running tests in parallel. By default, +uses the amount of concurrency available on the hardware as indicated by +[`available_concurrency`]. + +This can also be specified with the `RUST_TEST_THREADS` environment variable. + +#### `--force-run-in-process` + +Forces the tests to run in a single process when using the [`abort` panic +strategy][panic-strategy]. + +⚠️ 🚧 This only works with the unstable [`-Z panic-abort-tests`] option, and +requires the `-Z unstable-options` flag. See [tracking issue +#67650](https://github.com/rust-lang/rust/issues/67650) for more information. + +#### `--ensure-time` + +⚠️ 🚧 This option is [unstable](#unstable-options), and requires the `-Z +unstable-options` flag. See [tracking issue +#64888](https://github.com/rust-lang/rust/issues/64888) and the [unstable +docs](../../unstable-book/compiler-flags/report-time.html) for more information. + +### Output options + +The following options affect the output behavior. + +#### `-q`, `--quiet` + +Displays one character per test instead of one line per test. This is an alias +for [`--format=terse`](#--format-format). + +#### `--nocapture` + +Does not capture the stdout and stderr of the test, and allows tests to print +to the console. Usually the output is captured, and only displayed if the test +fails. + +This may also be specified by setting the `RUST_TEST_NOCAPTURE` environment +variable set to anything but `0`. + +#### `--show-output` + +Displays the stdout and stderr of successful tests after all tests have run. + +Contrast this with [`--nocapture`](#--nocapture) which allows tests to print +*while they are running*, which can cause interleaved output if there are +multiple tests running in parallel, `--show-output` ensures the output is +contiguous, but requires waiting for all tests to finish. + +#### `--color` _COLOR_ + +Control when colored terminal output is used. Valid options: + +* `auto`: Colorize if stdout is a tty and [`--nocapture`](#--nocapture) is not + used. This is the default. +* `always`: Always colorize the output. +* `never`: Never colorize the output. + +#### `--format` _FORMAT_ + +Controls the format of the output. Valid options: + +* `pretty`: This is the default format, with one line per test. +* `terse`: Displays only a single character per test. [`--quiet`](#-q---quiet) + is an alias for this option. +* `json`: Emits JSON objects, one per line. ⚠️ 🚧 This option is + [unstable](#unstable-options), and requires the `-Z unstable-options` flag. + See [tracking issue #49359](https://github.com/rust-lang/rust/issues/49359) + for more information. + +#### `--logfile` _PATH_ + +Writes the results of the tests to the given file. + +#### `--report-time` _FORMAT_ + +⚠️ 🚧 This option is [unstable](#unstable-options), and requires the `-Z +unstable-options` flag. See [tracking issue +#64888](https://github.com/rust-lang/rust/issues/64888) and the [unstable +docs](../../unstable-book/compiler-flags/report-time.html) for more information. + +### Unstable options + +Some CLI options are added in an "unstable" state, where they are intended for +experimentation and testing to determine if the option works correctly, has +the right design, and is useful. The option may not work correctly, break, or +change at any time. To signal that you acknowledge that you are using an +unstable option, they require passing the `-Z unstable-options` command-line +flag. + +## Benchmarks + +The libtest harness supports running benchmarks for functions annotated with +the `#[bench]` attribute. Benchmarks are currently unstable, and only +available on the [nightly channel]. More information may be found in the +[unstable book][bench-docs]. + +## Custom test frameworks + +Experimental support for using custom test harnesses is available on the +[nightly channel]. See [tracking issue +#50297](https://github.com/rust-lang/rust/issues/50297) and the +[custom_test_frameworks documentation] for more information. + +[`--test` option]: ../command-line-arguments.md#option-test +[`-Z panic-abort-tests`]: https://github.com/rust-lang/rust/issues/67650 +[`available_concurrency`]: ../../std/thread/fn.available_concurrency.html +[`cargo test`]: ../../cargo/commands/cargo-test.html +[`libtest`]: ../../test/index.html +[`main` function]: ../../reference/crates-and-source-files.html#main-functions +[`Result`]: ../../std/result/index.html +[`test` cfg option]: ../../reference/conditional-compilation.html#test +[attribute-ignore]: ../../reference/attributes/testing.html#the-ignore-attribute +[attribute-should_panic]: ../../reference/attributes/testing.html#the-should_panic-attribute +[attribute-test]: ../../reference/attributes/testing.html#the-test-attribute +[bench-docs]: ../../unstable-book/library-features/test.html +[Cargo]: ../../cargo/index.html +[crate type]: ../../reference/linkage.html +[custom_test_frameworks documentation]: ../../unstable-book/language-features/custom-test-frameworks.html +[nightly channel]: ../../book/appendix-07-nightly-rust.html +[panic-strategy]: ../../book/ch09-01-unrecoverable-errors-with-panic.html +[panic]: ../../book/ch09-01-unrecoverable-errors-with-panic.html +[Testing Chapter]: ../../book/ch11-00-testing.html