[Bug 48002] New: Jenkins: Check for server-side php errors during web request

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=48002

Web browser: ---
Bug ID: 48002
Summary: Jenkins: Check for server-side php errors during web
request
Product: Wikimedia
Version: wmf-deployment
Hardware: All
OS: All
Status: NEW
Severity: normal
Priority: Unprioritized
Component: Continuous integration
Assignee: wikibugs-***@lists.wikimedia.org
Reporter: ***@gmail.com
CC: ***@free.fr, ***@gmail.com,
***@gmail.com
Classification: Unclassified
Mobile Platform: ---

We currently already set up a web-facing instance of MediaWiki (during the
qunit build).

Let's extend our test suite by checking the error log before and after and
consider any new entries a fatal build failure.

php notices, warnings, errors etc.

The primary motivation here is to catch php errors triggered during load.php
requests because those are almost invisible in general, and even more so when
all we see is the qunit callback through phantomjs.

There've been a few commits lately in master that messed something up in the
registration of javascript modules but nothing caught it. They are only
reliably detectable at run-time.

And of course it helps catching lots of other errors.

Since we already have a policy of not tolerating php errors, this should be
trivial to enable. Though we do have a few known php errors, none of them
trigger on regular usage (open wiki page, request load.php, run qunit tests
etc.).

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=48002

--- Comment #1 from Krinkle <***@gmail.com> ---
Recommended approach:

* Rename mediawiki-core-qunit to mediawiki-core-browsertest
* Build steps:
- Install MediaWiki
- curl /wiki/Special:BlankPage (anything not HTTP 200 is error)
- grunt qunit /wiki/Special:JavaScriptTest/qunit
- assert apache/php error log is empty or unchanged

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=48002

--- Comment #2 from Antoine "hashar" Musso <***@free.fr> ---
Another way would be to override the PHP handler to log all the exceptions,
format them in a nice format (such as HTML?) and then use a post build action
in Jenkins to publish the report.

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=48002

--- Comment #3 from Krinkle <***@gmail.com> ---
HTML is difficult to work with and format from plain bash (or most programming
languages for that matter). Plus we're striving towards getting rid of Jenkins
formatters to only use plain text reporters (with regards to build reports
static caching).

I'd say plain text is more than clear enough. We're all familiar with PHP's
error_log format.

We just need:
* php error log to go to a separate file for the qunit.localhost requests
* Jenkins needs to have permission to truncate this file
* Apache and/or PHP needs to not keep the handle open on this file, so that
when each job truncates the file, it really does truncates it.

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=48002

--- Comment #4 from Krinkle <***@gmail.com> ---
(In reply to comment #3)

Post by b***@wikimedia.org
HTML is difficult to work with and format from plain bash (or most
programming
languages for that matter). Plus we're striving towards getting rid of
Jenkins
formatters to only use plain text reporters (with regards to build reports
static caching).
I'd say plain text is more than clear enough. We're all familiar with PHP's
error_log format.
* php error log to go to a separate file for the qunit.localhost requests
* Jenkins needs to have permission to truncate this file
* Apache and/or PHP needs to not keep the handle open on this file, so that
when each job truncates the file, it really does truncates it.

Also, even if we'd make a custom HTML reporter or something, we still have
these same problems. So let's deal with that first.

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=48002

--- Comment #5 from Antoine "hashar" Musso <***@free.fr> ---
The important part in my reply was using a custom PHP error/exception handler.
It seems to be easier to tweak and more robust.

How would you handle several jobs running QUnit tests in parallel ?

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=48002

Antoine "hashar" Musso <***@free.fr> changed:

What |Removed |Added
----------------------------------------------------------------------------
Severity|normal |enhancement

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=48002

--- Comment #6 from Antoine "hashar" Musso <***@free.fr> ---
Parsing the apache log is a smart idea to catch all issues that happens during
run time and we can't catch using static analysis. But since all QUnit jobs are
hitting the same virtualhost, you will end up with erros from different
extensions in the apache log.

Instead we could use php auto_prepend_file that would let you inject whatever
PHP at the top of any PHP file being included (such as index.php / api.php ...
)

http://www.php.net/manual/fr/ini.core.php#ini.auto-prepend-file

That let you write a small PHP File that fetch the Jenkins env variable
override the error_log and have something a bit more robust and easy to tweak.

Will have to make sure the injected code is only executed when PHP require a
MediaWiki entry point.

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=48002

--- Comment #7 from Krinkle <***@gmail.com> ---
To clarify, we'd change the php init error_log setting to point to a file
inside the env.WORKSPACE directory from Jenkins so that we have the errors on a
per job base.

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=48002

--- Comment #8 from Krinkle <***@gmail.com> ---
PS2: Depending on when we get to implementation of this, if we have jobs
running in Vagrant we don't need any of that hackery since the VM would be
dedicated to that one job only.

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=48002

Krinkle <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Summary|Jenkins: Check for |Jenkins: Check for
|server-side php errors |server-side php errors
|during web request |

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=48002

Krinkle <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Summary|Jenkins: Check for |Jenkins: Assert server-side
|server-side php errors |error logs for
| |Apache/PHP/MediaWiki are
| |empty

--- Comment #9 from Krinkle <***@gmail.com> ---
So we need to:

1) Add configuration to the MediaWiki install to enable logging of errors
extensively to the log/ directory in the workspace.

2) Find a way to either have Apache send its error log (which includes PHPs
stderr) to a local directory. Assuming that's too complicated as long as we're
using the same Apache and vhost for all jobs, lets forget about Apache issues
and instead find a way to hijack php's error log.

Antoine has started on #1 by setting up logging. We're not yet asserting them,
but at least we have them available now from the build artifacts.

Antoine suggested we implement #2 by prepending php ini_set (or something like
that) to all web entry points. For maintenance scripts we don't need to since
we already have stderr there, and for any mediawiki internal things, those are
logged to log/ which applies to both web and cli runs.

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=48002

--- Comment #10 from Antoine "hashar" Musso <***@free.fr> ---
See also PHP ini setting auto_prepend_file I mentioned in comment #6. Could be
pointed to some custom error handler under /srv/slave-scripts/mediawiki (which
is integration/jenkins.git).

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=48002

Antoine "hashar" Musso <***@free.fr> changed:

What |Removed |Added
----------------------------------------------------------------------------
Priority|Unprioritized |Low

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=48002

Krinkle <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Summary|Jenkins: Assert server-side |Jenkins: Assert no PHP
|error logs for |errors (notices, warnings)
|Apache/PHP/MediaWiki are |were raised or exceptions
|empty |were thrown

--- Comment #11 from Krinkle <***@gmail.com> ---
Since I added a MediaWiki log group for php errors (notices and warnings) we no
longer need to get them via the apache2 log or from php sterr. We can capture
them directly, and assert them to be empty.

Let's set up logGroup[error] and logGroup[exception] for -qunit (www, frontend)
and -phpunit (cli, installer, backend) jobs and assert them to be empty.

The only thing this still won't catch is notices and warnings very early on in
MediaWiki run time. Those would still be seen (for CLI scripts, these would
show up in the Jenkins job console output, and for index.php/load.php the qunit
job output would show these), but not detected automatically.

But that's an edge case we can try and catch later. Right now we're not
catching any of them.

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=48002

Krinkle <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Priority|Low |Normal
Assignee|wikibugs-***@lists.wikimedia. |***@gmail.com
|org |

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=48002

Krinkle <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |ASSIGNED

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=48002

--- Comment #12 from Gerrit Notification Bot <***@wikimedia.org> ---
Change 174810 had a related patch set uploaded by Krinkle:
mediawiki: Configure 'error' and 'exception' log groups

https://gerrit.wikimedia.org/r/174810

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=48002

Gerrit Notification Bot <***@wikimedia.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|ASSIGNED |PATCH_TO_REVIEW