This fixes multiple network issues we had when downloading awscli from
PyPI on Azure Pipelines by vendoring awscli itself and its dependencies
in our S3 bucket. Instructions on how to update the cache are present at
the top of src/ci/install-awscli.sh
ci: debug network failures while downloading awscli from PyPI
This adds some random debug code to our CI script while downloading awscli, to *hopefully* pinpoint what's causing the network failures.
r? @Mark-Simulacrum
cc #62967
ci: Install clang on Windows through tarballs
Previously we used the executables built the LLVM project but these
executables are difficult to run in a CI environment, they can
accidentally pollute global state, etc. In testing some of the possible
4-core machine environments for Azure this step would frequently cause
issues.
To assuage these future issues and hopefully make builds slightly more
self-contained, this commit changes to install from a tarball instead.
The tarball isn't provided by LLVM itself, but we use the offical LLVM
installer to extract itself and then we pack up the LLVM installation
directory into the tarball.
docutils 0.15, a dependency of awscli, broke our CI since it's not
compatible with Python 2 due to a bug. This pins all the dependencies of
awscli with docutils 0.14, to make sure this kind of regressions doesn't
happen again.
This commit updates some of our assorted Azure/CI configuration to
prepare for some 4-core machines coming online. We're still in the
process of performance testing them to get final numbers, but some
changes are worth landing ahead of this. The updates here are:
* Use `C:/` instead of `D:/` for submodule checkout since it should have
plenty of space and the 4-core machines won't have `D:/`
* Update `lzma-sys` to 0.1.14 which has support for VS2019, where 0.1.10
doesn't.
* Update `src/ci/docker/run.sh` to work when it itself is running inside
of a docker container (see the comment in the file for more info)
* Print step timings on the `try` branch in addition to the `auto`
branch in. The logs there should be seen by similarly many humans (not
many) and can be useful for performance analysis after a `try` build
runs.
* Install the WIX and InnoSetup tools manually on Windows instead of
relying on pre-installed copies on the VM. This gives us more control
over what's being used on the Azure cloud right now (we control the
version) and in the 4-core machines these won't be pre-installed. Note
that on AppVeyor we actually already were installing InnoSetup, we
just didn't carry that over on Azure!
Previously we used the executables built the LLVM project but these
executables are difficult to run in a CI environment, they can
accidentally pollute global state, etc. In testing some of the possible
4-core machine environments for Azure this step would frequently cause
issues.
To assuage these future issues and hopefully make builds slightly more
self-contained, this commit changes to install from a tarball instead.
The tarball isn't provided by LLVM itself, but we use the offical LLVM
installer to extract itself and then we pack up the LLVM installation
directory into the tarball.
ci: add a pr builder to test tools when submodules are updated
This PR adds the x86_64-gnu-tools builders to PRs where submodules are updated.
Since it's not possible to *start* the builder only when submodule changes are detected, I opted into adding a "decider" task at the start of the job which sets the `SKIP_JOB` environment variable when submodules are not updated, and I gated the most time-consuming tasks (the actual build and artifacts upload) on the variable not being there. All of this is conditionally included in the `steps/run.yml` only when a template parameter is present, so it should only affect that builder on PRs.
The cost for this should be a dummy builder running for 2/3 minutes for each PR, and we should be able to handle it.
Fixes https://github.com/rust-lang/rust/issues/61837
r? @alexcrichton
During installation of mingw, at least, the git directories change, so
we need to reset the core.autocrlf config to false.
Once we finish checking out submodules, check that the line endings are
\n and not \r\n.
The Azure image enables CRLF conversion on Windows builders, but that
caused regressions both in our test suite (the miri test suite broke)
and in the ecosystem, since we started shipping install scripts with
CRLF endings instead of the old LF. The Godbolt Compiler Explorer is one
such case of breakage.
This adds a step to the build explicitly disabling the conversion before
the repository is checked out.
Currently just run it through its paces but don't actually push to
official locations. Instead let's just push to a separate fork (mine) as
well as open issues in a separate fork (mine). Make sure that people
aren't pinged for these issues as well!
This should hopefully ensure that everything is working on Azure and
give us a chance to work through any issues that come up.
ci: Collect CPU usage statistics on Azure
This commit adds a script which we'll execute on Azure Pipelines which
is intended to run in the background and passively collect CPU usage
statistics for our builders. The intention here is that we can use this
information over time to diagnose issues with builders, see where we can
optimize our build, fix parallelism issues, etc. This might not end up
being too useful in the long run but it's data we've wanted to collect
for quite some time now, so here's a stab at it!
Comments about how this is intended to work can be found in the python
script used here to collect CPU usage statistics.
Closes#48828
This commit adds a script which we'll execute on Azure Pipelines which
is intended to run in the background and passively collect CPU usage
statistics for our builders. The intention here is that we can use this
information over time to diagnose issues with builders, see where we can
optimize our build, fix parallelism issues, etc. This might not end up
being too useful in the long run but it's data we've wanted to collect
for quite some time now, so here's a stab at it!
Comments about how this is intended to work can be found in the python
script used here to collect CPU usage statistics.
Closes#48828
Looks like Azure has updated images recently to install Rust by default,
but that can interfere with our own compiler (for example Cargo's test
suite we think) so be sure to uninstall it before proceeding.
Speed up Azure CI installing Windows dependencies
There is known issue where PowerShell is unreasonably slow downloading
files due to an issue with rendering the progress bar, see this [issue](https://github.com/PowerShell/PowerShell/issues/2138)
That issue is fixed in PowerShell Core (available in Azure Pipelines as
pwsh.exe) but it can also be worked around by setting:
$ProgressPreference = 'SilentlyContinue'
I measured downloading LLVM and it took about 220s before, 5s after, so the improvement is significant.
ci: Favor SCRIPT instead of RUST_CHECK_TARGET
Since #61212 we've been timing out on OSX, and this looks to be because
we're building tools like Cargo and the RLS twice instead of once. This
turns out to be a slight bug in our configuration. CI builders using the
`RUST_CHECK_TARGET` directive actually execute `make all` just before
their acual target. In `make all` we're building a stage2 cargo, and
then in `make dist` we're building a stage1 cargo.
Other builders use `SCRIPT` which provides explicit control over what
`x.py` script, for example, is used to execute the build. This moves
almost all targets to using `SCRIPT` to ensure that we're explicitly
specifying what's being built where. Additionally this updates the logic
of `RUST_CHECK_TARGET` to remove the pre-flight tidy as well as the
pre-flight `make all`. The system LLVM builder (run on PRs) now
explicitly runs tidy first and then runs the rest of the test suite.
Since #61212 we've been timing out on OSX, and this looks to be because
we're building tools like Cargo and the RLS twice instead of once. This
turns out to be a slight bug in our configuration. CI builders using the
`RUST_CHECK_TARGET` directive actually execute `make all` just before
their acual target. In `make all` we're building a stage2 cargo, and
then in `make dist` we're building a stage1 cargo.
Other builders use `SCRIPT` which provides explicit control over what
`x.py` script, for example, is used to execute the build. This moves
almost all targets to using `SCRIPT` to ensure that we're explicitly
specifying what's being built where. Additionally this updates the logic
of `RUST_CHECK_TARGET` to remove the pre-flight tidy as well as the
pre-flight `make all`. The system LLVM builder (run on PRs) now
explicitly runs tidy first and then runs the rest of the test suite.
There is known issue where PowerShell is unreasonably slow downloading
files due to an issue with rendering the progress bar, see:
https://github.com/PowerShell/PowerShell/issues/2138
That issue is fixed in PowerShell Core (available in Azure Pipelines as
pwsh.exe) but it can also be worked around by setting:
$ProgressPreference = 'SilentlyContinue'
* Clean up the `install-windows-build-deps.yml` file and add some more
comments where appropriate.
* Add some comments to `run.yml`
* Don't fast path the `rustfmt` submodule, but we'll take care of that
later if necessary.
This commit is intended to go through and review/refactor the azure
pipelines configuration we have. The major changes are:
* The separate `{windows,macos,linux}.yml` files are now all merged into
one `run.yml`. This allows a shared "master flow" for all platforms
with divergence only where necessary.
* Some install steps have been separated as `install-*.yml` scripts,
where each script internally matches on the appropriate OS and then
delegates accordingly.
* Some various bits and pieces of cruft have been removed which were
artifacts of Travis's setup or similar.
