rustup install behind a corporate proxy fails with TLS handshake eof
#4305
Comments
|
cc @djc since the above log looks related to the download/TLS backend. |
Merged
|
Sure, some instructions would be appreciated, a |
@muja Can you see the download links in https://github.com/rust-lang/rustup/actions/runs/14660975811?pr=4307? |
|
Works for me: You'll want to use |
|
|
Yeah that works. The logs with the new build: FWIW the reason |
For now you'll need to build OpenSSL to build rustup in the default configuration... that's the thing we're trying to fix here. |
|
Okay, and how do I do that? I couldn't find a |
|
It might be easier to drop that stuff for now, try |
|
That works, the output (if at all different from my previous comment): |
|
Ugh, that error is useless. |
|
Alright, this might help a little bit: #4309. |
|
At this point I'm just checking out the specific branch and building/running, trusting that you don't put a backdoor in there somewhere |
|
I see it was merged to master? |
|
Okay, so it looks like the proxy is just closing the TCP connection once rustls sends the TLS ClientHello. I suggest you contact support -- maybe using the forum on the Fortinet website? |
As another data point, could you apply this patch and try again? diff --git a/Cargo.toml b/Cargo.toml
index 9f19b5a9..0a55e6b9 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -70,7 +70,7 @@ remove_dir_all = { version = "1.0.0", features = ["parallel"] }
reqwest = { version = "0.12", default-features = false, features = ["blocking", "gzip", "socks", "stream"], optional = true }
retry = { version = "2", default-features = false, features = ["random"] }
rs_tracing = { version = "1.1", features = ["rs_tracing"] }
-rustls = { version = "0.23", optional = true, default-features = false, features = ["logging", "aws_lc_rs", "tls12"] }
+rustls = { version = "0.23", optional = true, default-features = false, features = ["logging", "aws_lc_rs", "tls12", "prefer-post-quantum"] }
rustls-platform-verifier = { version = "0.5", optional = true }
same-file = "1"
semver = "1.0"
Thanks! |
|
This is not acceptable. Using curl or anything else works flawlessly, here is the verbose output from As such it is definitely an issue with rustup. I even tested it out with my own binary, and it connects successfully. No idea what rustup is doing differently? #[tokio::main]
async fn main() {
let client = reqwest::ClientBuilder::new().use_rustls_tls().build().unwrap();
let response = client.get("https://static.rust-lang.org/dist/channel-rust-beta.toml.sha256").send().await.unwrap().text().await;
println!("{response:?}");
}prints: Cargo.toml: [package]
name = "foo"
version = "0.1.0"
edition = "2024"
[dependencies]
reqwest = { version = "0.12.15", default-features = false, features = ["rustls-tls", "rustls-tls-native-roots-no-provider"] }
tokio = { version = "1.44.2", features = ["full"] } |
Ahh, thanks for testing that, that is very interesting! |
|
Can you try to replace the reqwest setup with this? Client::builder()
// HACK: set `pool_max_idle_per_host` to `0` to avoid an issue in the underlying
// `hyper` library that causes the `reqwest` client to hang in some cases.
// See <https://github.com/hyperium/hyper/issues/2312> for more details.
.pool_max_idle_per_host(0)
.gzip(false)
.proxy(Proxy::custom(env_proxy))
.read_timeout(Duration::from_secs(30))
.use_preconfigured_tls(
rustls::ClientConfig::builder_with_provider(Arc::new(
aws_lc_rs::default_provider(),
))
.with_safe_default_protocol_versions()
.unwrap()
.with_platform_verifier()
.with_no_client_auth(),
)
.user_agent(super::REQWEST_RUSTLS_TLS_USER_AGENT)
.build()Presumably one of these calls is causing the difference. |
No difference |
Thanks! |
|
okay after adding all the dependencies, i was able to run it with this main: use rustls_platform_verifier::*;
#[tokio::main]
async fn main() {
let client = reqwest::Client::builder()
// HACK: set `pool_max_idle_per_host` to `0` to avoid an issue in the underlying
// `hyper` library that causes the `reqwest` client to hang in some cases.
// See <https://github.com/hyperium/hyper/issues/2312> for more details.
.pool_max_idle_per_host(0)
.gzip(false)
.read_timeout(std::time::Duration::from_secs(30))
.use_preconfigured_tls(
rustls::ClientConfig::builder_with_provider(std::sync::Arc::new(
rustls::crypto::aws_lc_rs::default_provider(),
))
.with_safe_default_protocol_versions()
.unwrap()
.with_platform_verifier()
.with_no_client_auth(),
)
.user_agent("whatever")
.build()
.unwrap();
let response = client
.get("https://static.rust-lang.org/dist/channel-rust-beta.toml.sha256")
.send()
.await
.unwrap()
.text()
.await;
println!("{response:?}");
}output: Cargo.toml [package]
name = "foo"
version = "0.1.0"
edition = "2024"
[dependencies]
reqwest = { version = "0.12.15", default-features = false, features = ["gzip", "rustls-tls", "rustls-tls-native-roots-no-provider"] }
rustls = { version = "0.23", default-features = false, features = ["logging", "aws_lc_rs", "tls12"] }
tokio = { version = "1.44.2", features = ["full"] }
rustls-platform-verifier = { version = "0.5" } |
|
@muja so that's the ClientHello from your test program? Can you grab one from rustup as well, and diff them? I don't think the server certificate verifier (either rustls-platform-verifier or rustls-native-certs) should influence the ClientHello... |
Very good spot. So that means |
|
From my program: from rustup: diff: ❯ diff target/rustup target/my
1c1
< 2025-04-25T11:59:52.683530Z TRACE run_rustup:run_rustup_inner:main:install:install:update_from_dist: rustls::client::hs: Sending ClientHello Message {
---
> [2025-04-25T14:39:52Z TRACE rustls::client::hs] Sending ClientHello Message {
9,10c9,10
< random: d16c99467414251dd9198ba1356af7630e3e0e267a64469fbcb982301a2ccf25,
< session_id: 9bea9d8d79d3702947ed12577f39e96bedc01361d863ec1d2a492f4b52a300f3,
---
> random: eff252c8d0dcf927248e2ca2597db5107aa5c22b52bbdcd2637932d6f4c56354,
> session_id: c89d4319117547a38ad9563acdba70131d85b577b2438d712bf54e91a70d2d27,
29a30,37
> KeyShare(
> [
> KeyShareEntry {
> group: X25519,
> payload: 55702288539fca5f20219a360bb21bec8a4727c8ae2aea47108f42ecc758f15b,
> },
> ],
> ),
36,37c44
< ExtendedMasterSecretRequest,
< NamedGroups(
---
> SignatureAlgorithms(
39,42c46,55
< X25519,
< secp256r1,
< secp384r1,
< X25519MLKEM768,
---
> ECDSA_NISTP384_SHA384,
> ECDSA_NISTP256_SHA256,
> ECDSA_NISTP521_SHA512,
> ED25519,
> RSA_PSS_SHA512,
> RSA_PSS_SHA384,
> RSA_PSS_SHA256,
> RSA_PKCS1_SHA512,
> RSA_PKCS1_SHA384,
> RSA_PKCS1_SHA256,
56a70
> ExtendedMasterSecretRequest,
62,65c76
< SignatureAlgorithms(
< [],
< ),
< PresharedKeyModes(
---
> NamedGroups(
67c78,81
< PSK_DHE_KE,
---
> X25519,
> secp256r1,
> secp384r1,
> X25519MLKEM768,
78c92
< KeyShare(
---
> PresharedKeyModes(
80,83c94
< KeyShareEntry {
< group: X25519,
< payload: 7fc9067623fa270beb8df1c1bc57686e3c317ca63e3d58b59ab29be6ec3dff7a,
< },
---
> PSK_DHE_KE,
90c101
< encoded: 010000de0303d16c99467414251dd9198ba1356af7630e3e0e267a64469fbcb982301a2ccf25209bea9d8d79d3702947ed12577f39e96bedc01361d863ec1d2a492f4b52a300f30014130213011303c02cc02bcca9c030c02fcca800ff0100008100230000002b0005040304030300170000000a000a0008001d0017001811ec0000001900170000147374617469632e727573742d6c616e672e6f7267000b00020100000d00020000002d00020101000500050100000000003300260024001d00207fc9067623fa270beb8df1c1bc57686e3c317ca63e3d58b59ab29be6ec3dff7a,
---
> encoded: 010000f20303eff252c8d0dcf927248e2ca2597db5107aa5c22b52bbdcd2637932d6f4c5635420c89d4319117547a38ad9563acdba70131d85b577b2438d712bf54e91a70d2d270014130213011303c02cc02bcca9c030c02fcca800ff0100009500230000003300260024001d002055702288539fca5f20219a360bb21bec8a4727c8ae2aea47108f42ecc758f15b002b00050403040303000d0016001405030403060308070806080508040601050104010000001900170000147374617469632e727573742d6c616e672e6f726700170000000b00020100000a000a0008001d0017001811ec000500050100000000002d00020101,
92c103
< } current_dir="/tmp/tmp.pG4aiqSug8/rustup" args="[\"./rustup\", \"install\", \"beta\"]" profile="Default" prefix="/home/user/.rustup/toolchains/beta-x86_64-unknown-linux-gnu"
---
> } |
|
@muja can you look at the results of |
|
Here is the sorted diff: ❯ diff target/rustup target/my
1c1
< 2025-04-25T11:59:52.683530Z TRACE run_rustup:run_rustup_inner:main:install:install:update_from_dist: rustls::client::hs: Sending ClientHello Message {
---
> [2025-04-25T14:39:52Z TRACE rustls::client::hs] Sending ClientHello Message {
9,10c9,10
< random: d16c99467414251dd9198ba1356af7630e3e0e267a64469fbcb982301a2ccf25,
< session_id: 9bea9d8d79d3702947ed12577f39e96bedc01361d863ec1d2a492f4b52a300f3,
---
> random: eff252c8d0dcf927248e2ca2597db5107aa5c22b52bbdcd2637932d6f4c56354,
> session_id: c89d4319117547a38ad9563acdba70131d85b577b2438d712bf54e91a70d2d27,
63c63,74
< [],
---
> [
> ECDSA_NISTP384_SHA384,
> ECDSA_NISTP256_SHA256,
> ECDSA_NISTP521_SHA512,
> ED25519,
> RSA_PSS_SHA512,
> RSA_PSS_SHA384,
> RSA_PSS_SHA256,
> RSA_PKCS1_SHA512,
> RSA_PKCS1_SHA384,
> RSA_PKCS1_SHA256,
> ],
82c93
< payload: 7fc9067623fa270beb8df1c1bc57686e3c317ca63e3d58b59ab29be6ec3dff7a,
---
> payload: 55702288539fca5f20219a360bb21bec8a4727c8ae2aea47108f42ecc758f15b,
85,86c96
< ),
< ],
---
> ), ],
90c100
< encoded: 010000de0303d16c99467414251dd9198ba1356af7630e3e0e267a64469fbcb982301a2ccf25209bea9d8d79d3702947ed12577f39e96bedc01361d863ec1d2a492f4b52a300f30014130213011303c02cc02bcca9c030c02fcca800ff0100008100230000002b0005040304030300170000000a000a0008001d0017001811ec0000001900170000147374617469632e727573742d6c616e672e6f7267000b00020100000d00020000002d00020101000500050100000000003300260024001d00207fc9067623fa270beb8df1c1bc57686e3c317ca63e3d58b59ab29be6ec3dff7a,
---
> encoded: 010000f20303eff252c8d0dcf927248e2ca2597db5107aa5c22b52bbdcd2637932d6f4c5635420c89d4319117547a38ad9563acdba70131d85b577b2438d712bf54e91a70d2d270014130213011303c02cc02bcca9c030c02fcca800ff0100009500230000003300260024001d002055702288539fca5f20219a360bb21bec8a4727c8ae2aea47108f42ecc758f15b002b00050403040303000d0016001405030403060308070806080508040601050104010000001900170000147374617469632e727573742d6c616e672e6f726700170000000b00020100000a000a0008001d0017001811ec000500050100000000002d00020101,
92c102
< } current_dir="/tmp/tmp.pG4aiqSug8/rustup" args="[\"./rustup\", \"install\", \"beta\"]" profile="Default" prefix="/home/user/.rustup/toolchains/beta-x86_64-unknown-linux-gnu"
---
> } |
|
my program prints a LOT, rustup: |
|
So there's a difference between the rustup environment and your program's environment? Is there some Fedora seccomp thing or |
|
I have no idea what this would be, but I've gotten a bit closer to whatever the problem is... When I do When I do |
|
As mentioned previously, try looking at the output of |
|
From the earlier curl logs:
Seems odd -- do you have
This sounds like rust-lang/cargo#3676 |
|
|
Confirmed that works locally for me. Before fixAfter fix |
|
Sorry I had to leave for a bit, so yes rust-lang/cargo#3676 was hitting me in this case, I do not have a
|
Yes, I have this set (but no And in the EDIT: and just to verify, there is no other environment variable pointing to a ca-bundle apart from these: |
For this, cargo uses the host openssl installation, which will have been customised by your distribution to know the most accurate location. That is good. But that cargo sets the environment variables at all is a mistake.
No. The
Why is that? Does it point to a directory full of
Could you run this? And then inspect the file and directory it prints? FWIW on ubuntu for me this gives: The file $ file /usr/lib/ssl/cert.pem
/usr/lib/ssl/cert.pem: symbolic link to /etc/ssl/certs/ca-certificates.crt
$ file /etc/ssl/certs/ca-certificates.crt
/etc/ssl/certs/ca-certificates.crt: PEM certificateThe directory $ ls -lha /usr/lib/ssl/certs | head
Permissions Size User Date Modified Name
lrwxrwxrwx - root 24 Apr 2024 0a775a30.0 -> GTS_Root_R3.pem
lrwxrwxrwx - root 24 Apr 2024 0b1b94ef.0 -> CFCA_EV_ROOT.pem
lrwxrwxrwx - root 24 Apr 2024 0b9bc432.0 -> ISRG_Root_X2.pem
lrwxrwxrwx - root 24 Apr 2024 0bf05006.0 -> SSL.com_Root_Certification_Authority_ECC.pem
lrwxrwxrwx - root 24 Apr 2024 0c31d5ce.0 -> ssl-cert-snakeoil.pem
lrwxrwxrwx - root 24 Apr 2024 0f5dc4f3.0 -> UCA_Extended_Validation_Root.pem
lrwxrwxrwx - root 24 Apr 2024 0f6fa695.0 -> GDCA_TrustAUTH_R5_ROOT.pem
lrwxrwxrwx - root 24 Apr 2024 002c0b4f.0 -> GlobalSign_Root_R46.pem
lrwxrwxrwx - root 24 Apr 2024 0179095f.0 -> BJCA_Global_Root_CA1.pemReally appreciate your patience as we get to the bottom of this! |
I believe this SSL_CERT_DIR is pointing to a directory that Kubernetes is mounting volumes from a ConfigMap: EDIT: so unsetting |
UGH. I realised now I just walked you into a trap. Sorry. Even Could you add the
Interesting. My understanding of that directory is that openssl will only read entries lazily, using file names that are hashes (like |
|
No difference: |
|
@djc I think this should not be a blocker of the upcoming v1.28.2 release (at least the old backends are still usable), so I'm pushing this back to v1.28.3 (or whatever comes next). Is that okay? |
|
Seems fair. It looks like this is a pretty rare platform-specific issue anyway, and we have some mitigations/improvements already in place. |
|
Really greatest and warmest thanks to you all for getting so thoroughly involved. Was some crazy investigating but now that we've gotten to the bottom of this issue, it's easy to fix on my side so the issue can be addressed later, whatever the fix may be. Diagnostics were already but could definitely be further improved in the long-term, but that is indeed a never ending battle. Thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Verification
Problem
When running rustup behind a corporate proxy (without
RUSTUP_USE_CURL=1) this error is produced:Dropping the protocol from the environment variable makes no difference. Adding
RUSTUP_USE_CURL=1fixes it, but apparently that's deprecated.One thing that may be relevant is that the corporate proxy is some kind of Fortinet Proxy and does not support the
zstdencoding, although it doesn't seem that the site tries to send it, even if advertised inAccept-Encodingheader:Steps
Output with
RUSTUP_LOG=TRACE:Possible Solution(s)
No response
Notes
No response
Rustup version
Installed toolchains
OS version
The text was updated successfully, but these errors were encountered: