[Bug 72894] New: RegEx-style Cirrus searches are ignored on en.wikipedia

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=72894

Bug ID: 72894
Summary: RegEx-style Cirrus searches are ignored on
en.wikipedia
Product: MediaWiki extensions
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: Unprioritized
Component: CirrusSearch
Assignee: wikibugs-***@lists.wikimedia.org
Reporter: ***@gmail.com
CC: ***@wikimedia.org, ***@gmail.com,
neverett+***@wikimedia.org
Web browser: ---
Mobile Platform: ---

A search for "wyoming"insource:/wyoming/ should find only lower-case examples,
but over 45,000 articles, mostly capitalized, are returned. This worked 2 days
ago.

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=72894

--- Comment #1 from Chad H. <***@gmail.com> ---
Yes, they're disabled for the time being as they were causing downtime.

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=72894

--- Comment #2 from ***@gmail.com ---
And how were the users of WP notified that the feature was turned off? How long
is "the time being"?

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=72894

--- Comment #3 from Andre Klapper <***@wikimedia.org> ---
https://en.wikipedia.org/w/index.php?title=Wikipedia:Village_pump_%28technical%29&oldid=632239801#Search_for_special_chars
mentions it.

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=72894

Nik Everett <neverett+***@wikimedia.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|UNCONFIRMED |ASSIGNED
Assignee|wikibugs-***@lists.wikimedia. |neverett+***@wikimedia
|org |.org
Ever confirmed|0 |1

--- Comment #4 from Nik Everett <neverett+***@wikimedia.org> ---
Hi Grumbler!

I sent an email to wikitech-***@lists.wikimedia.org on October 31st at
11:30 US Eastern time. Its how I've done the bulk of the Cirrus communication
at this point and I thought it was the right way to announce it.

I should have updated a recent conversation on en:wp's Village Pump and did
when someone pinged me. I should have remembered to do it earlier but I was
trying to fix the problem.

As far as how long the time being is - I'm working on it now. I have a hacked
together solution the demonstrably works but that is only the first step. The
remaining steps are mostly these:
1. Work with upstream (Lucene) to get a version of the hack that'd be
acceptable for them to merge.
2. Backport those changes to Lucene to the plugins we use for regex search and
highlighting.
3. Release new versions of those.
4. Deploy them to our search cluster.
5. Reenable the feature.


You can sort of fudge on 1 and just backport something more hacky but that is
somewhat more dangerous from a stability standpoint. But that is life and
we'll do it if upstream drags their feet. So far they've been reasonably
responsive though.

Estimating a time from that is tricky. A week from today?

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=72894

--- Comment #5 from Nik Everett <neverett+***@wikimedia.org> ---
Quick update:
Step 1 is moving along pretty quickly. I'm communicating closely with a Lucene
committer on getting a patch merged for this. It feels like we'll get
something merged today which means its worth waiting for that before moving on
to step 2.

Also there is a step 2.5: Update Cirrus to catch the new error message from
step #2 and produce a useful message for the user.

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=72894

--- Comment #6 from ***@gmail.com ---
CirrusSearch is set to be rolled out to en.wikipedia on November 19. Please
tell me that when that date was set they knew about this outage and were
confident that this will be fixed in time for the The Grand Opening.

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=72894

--- Comment #7 from Chad H. <***@gmail.com> ---
(In reply to SpontaneousGrumbler from comment #6)

Post by b***@wikimedia.org
CirrusSearch is set to be rolled out to en.wikipedia on November 19. Please
tell me that when that date was set they knew about this outage and were
confident that this will be fixed in time for the The Grand Opening.

I would hope that Nik and I knew about the outage when we picked that date.

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=72894

--- Comment #8 from Nik Everett <neverett+***@wikimedia.org> ---
(In reply to SpontaneousGrumbler from comment #6)

Post by b***@wikimedia.org
CirrusSearch is set to be rolled out to en.wikipedia on November 19. Please
tell me that when that date was set they knew about this outage and were
confident that this will be fixed in time for the The Grand Opening.

Yup.

Here is the status:
The fix for the cause of the outage is live in beta. When I tried it on
Saturday I found an error where sometimes the right error message isn't shown
when the regex is too complex to use. I've *just* finished the fix for that.

The plan right now is get that to beta today and validate it.

We'll deploy the fix to production on Wednesday. Today or Tuesday would have
been better but Tuesday is a US holiday and we'll have less people on hand in
the unlikely event that something goes wrong.

That puts us reenabling regex search on Thursday. Its long than we'd
thought/hoped.

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=72894

--- Comment #9 from ***@gmail.com ---
Thanks for the update.

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=72894

Nik Everett <neverett+***@wikimedia.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
Priority|Unprioritized |High

--- Comment #10 from Nik Everett <neverett+***@wikimedia.org> ---
Plugins deployed. We'll be pushing code to reenable the searches in our
general window which starts in an hour and a half.

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=72894

--- Comment #11 from ***@gmail.com ---
Thanks. It appears to be working now.

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=72894

--- Comment #12 from Nik Everett <neverett+***@wikimedia.org> ---
Hey, glad its working for you. I keep getting:
An error has occurred while searching: Too many regular expression searches
currently running. Please try again later.
which is a pain. I think something is up with the counter because I totally
don't see that many regex searches.

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=72894

--- Comment #13 from ***@gmail.com ---
Yes, well, it worked for one search. Ever since then, I keep getting the same
error message, "Too many regular expression searches". Yes, something is wrong;
failing to decrement the counter seems a likely cause.

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=72894

--- Comment #14 from Nik Everett <neverett+***@wikimedia.org> ---
Yeah. Something. I'll be able to spend some time with it in the morning.
I've he a suspicion that that counter is lying for a while now. Its actually
the same counter that we for all kinds of stuff and its pretty hard *not* to
decrement it.

I'm at least glad it doesn't just hate me.

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=72894

--- Comment #15 from Nik Everett <neverett+***@wikimedia.org> ---
OK! I found the problem. Our pool counter work differently then how I thought
it did. I've prepared a patch to deploy and we'll sync it out during the
Monday morning deploy.

https://gerrit.wikimedia.org/r/#/q/I7586162cfb32ddbe460a25c956c845f2f4a49b0f,n,z

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=72894

--- Comment #16 from ***@gmail.com ---
Good news!

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=72894

--- Comment #17 from ***@gmail.com ---
I would say it has been much better for the last few days. If you want to mark
this bug as "fixed", I won't disagree.

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=72894

Chad H. <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|ASSIGNED |RESOLVED
Resolution|--- |FIXED

--- Comment #18 from Chad H. <***@gmail.com> ---
Between Nik's fixes and the fact that we segmented enwiki's PoolCounter traffic
to its own key I think we're in a way better spot than before.

Resolving FIXED. Please reopen if this becomes a problem again.