[Bug 68387] New: beta labs no longer listens for HTTPS

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=68387

Bug ID: 68387
Summary: beta labs no longer listens for HTTPS
Product: Wikimedia Labs
Version: unspecified
Hardware: All
OS: All
Status: NEW
Severity: normal
Priority: Unprioritized
Component: deployment-prep (beta)
Assignee: wikibugs-***@lists.wikimedia.org
Reporter: ***@wikimedia.org
CC: ***@gmail.com, ***@wikimedia.org,
***@wikimedia.org, ***@free.fr,
***@wikimedia.org, ***@gmail.com,
***@tim-landscheidt.de
Web browser: ---
Mobile Platform: ---

https://en.wikipedia.beta.wmflabs.org/ no longer responds at all.

While we have never had a valid cert for beta, we did in the past answer HTTPS
URLS, forcing the user to proceed manually over a security warning. As of
sometime fairly recently, we no longer listen on HTTPS at all.

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=68387

--- Comment #1 from Antoine "hashar" Musso <***@free.fr> ---
HTTPS is handled using nginx on the varnish server by applying
role::protoproxy::ssl::beta

Looking at the puppet run of deployment-cache-text02.eqiad.wmflabs (the text
cache) I find:

Debug: Executing '/etc/init.d/nginx status'

So puppet knows about nginx but for some reason does not start it :-(

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=68387

Greg Grossmeier <***@wikimedia.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
Priority|Unprioritized |Normal
CC| |***@wikimedia.org,
| |***@wikimedia.org

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=68387

--- Comment #2 from Antoine "hashar" Musso <***@free.fr> ---
I attempted to start it manually:

# service nginx start
Starting nginx: nginx: [emerg]
SSL_CTX_use_PrivateKey_file("/etc/ssl/private/star.wmflabs.org.key") failed
(SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key
values mismatch)
nginx: configuration file /etc/nginx/nginx.conf test failed



I can't remember how we got the SSL keys deployed for beta :-/ Some ops with
better knowledge about SSL than me would probably know.

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=68387

--- Comment #3 from Antoine "hashar" Musso <***@free.fr> ---
Apparently broken since April 11 :/

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=68387

--- Comment #4 from Bryan Davis <***@wikimedia.org> ---
This has been broken as long as we have been in eqiad as far as I know.
role::protoproxy::ssl::beta is used to setup the nginx ssl terminators in front
of *.beta.wmflabs.org. That in turn applies role::protoproxy::ssl::beta::common
which includes `install_certificate{'star.wmflabs.org': privatekey => false}`.
The "privatekey => false" bit there tells puppet not to try and manage the ssl
private key install. This is done because labs/private.git does not contain the
x509 private key for the real *.wmflabs.org cert (for good reason).

To fix it we need to either:
a) Have an Opsen populate /etc/ssl/private/star.wmflabs.org.key on all of the
frontend boxes for beta [0]. This private key must match the public key in
operations/puppet [1].
b) Create a self-signed cert for beta and change puppet
** Put the private key in labs/private/ssl on deployment-salt
** Put the public key in operations/puppet/files/ssl on deployment-salt (or
operations/puppet)
** Change role::protoproxy::ssl::beta::common to install the new self-signed
cert


[0]:
https://wikitech.wikimedia.org/w/index.php?title=Special:Ask&q=%5B%5BResource+Type%3A%3Ainstance%5D%5D%5B%5BPuppet+Class%3A%3Arole%3A%3Aprotoproxy%3A%3Assl%3A%3Abeta%5D%5D&p=format%3Dbroadtable%2Flink%3Dall%2Fheaders%3Dshow%2Fsearchlabel%3D%E2%80%A6-20further-20results%2Fclass%3Dsortable-20wikitable-20smwtable&po=%3FInstance+Name%0A%3FPuppet+Class%0A%3FPuppet+Var%0A&sort=Modification+date&order=DESC&limit=50&eq=no
[1]:
https://github.com/wikimedia/operations-puppet/blob/production/files/ssl/star.wmflabs.org.pem

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=68387

***@wikimedia.org changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@wikimedia.org

--- Comment #5 from ***@wikimedia.org ---
(In reply to Bryan Davis from comment #4)

Post by b***@wikimedia.org
This has been broken as long as we have been in eqiad as far as I know.

FWIW I'm about 90% sure that https to beta labs worked in eqiad. My browser
autocompletion URLs for Flow pages on beta were all https and I had a
forceHTTPS cookie for beta labs, and as I recall it worked fine until 2-3 weeks
ago. I had to manually remove the cookie in order to login and now I'm OK.

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=68387

--- Comment #6 from Mark Holmquist <***@member.fsf.org> ---
Especially given that Fabrice reports it only broke for him yesterday, I'm
pretty sure this had been working until pretty recently.

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=68387

Bryan Davis <***@wikimedia.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
See Also| |https://bugzilla.wikimedia.
| |org/show_bug.cgi?id=48501

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=68387

se4598 <***@se4598.eu> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@se4598.eu
See Also| |https://bugzilla.wikimedia.
| |org/show_bug.cgi?id=63538

--- Comment #8 from se4598 <***@se4598.eu> ---
I'm pretty sure it has not/never worked the last month, b/c occasionally I
still hit a old https-beta link from my history, which was never working after
migration.

This bug would be a duplicate of bug 63538, if this wouldn't have been marked
as "resolved fixed" because "there is no need to have two bugs to track the
issue"...

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=68387

Bryan Davis <***@wikimedia.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
Keywords| |ops

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=68387

Antoine "hashar" Musso (WMF) <***@free.fr> changed:

What |Removed |Added
----------------------------------------------------------------------------
Blocks| |65421

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=68387

Bryan Davis <***@wikimedia.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@gmail.com

--- Comment #9 from Bryan Davis <***@wikimedia.org> ---
*** Bug 73680 has been marked as a duplicate of this bug. ***