[Bug 49203] New: ask query offset has a hardcoded limit

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=49203

Web browser: ---
Bug ID: 49203
Summary: ask query offset has a hardcoded limit
Product: MediaWiki extensions
Version: master
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: Unprioritized
Component: Semantic MediaWiki
Assignee: wikibugs-***@lists.wikimedia.org
Reporter: ***@gmail.com
CC: ***@gmail.com, ***@yahoo.com,
***@semantic-mediawiki.org, ***@gmail.com
Classification: Unclassified
Mobile Platform: ---

Created attachment 12471
--> https://bugzilla.wikimedia.org/attachment.cgi?id=12471&action=edit
SMWQueryProcessor without $params['offset']['upperbound'].

Running an ask query with an offset greater than 5000 returns the results from
the query with offset=0 instead. Because of this, the only way to reliably
fetch all results for large queries is to set offset=0,
limit=<very_large_number> and fetch all results at once. This is less
desirable for the user as well as the site host than "paging through" large
queries by specifying a limit and incrementing the offset by the limit size for
every page queried. But limiting the offset in any way makes this preferred
scenario impossible.

The code responsible is the 'upperbound' setting for $params['offset'] in
SemanticMediaWiki/includes/SMW_QueryProcessor.php at line 601:

$params['offset'] = array(
'type' => 'integer',
'default' => 0,
'negatives' => false,
'upperbound' => 5000 // TODO: make setting
);

My guess is that the upperbound was intended for $params['limit'], but was
added to $params['offset'] by mistake.

The solution is not to make 'upperbound' configurable with a global variable as
the comment implies, but simply to remove this line altogether (see
attachment).

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=49203

Jamie Thingelstad <***@thingelstad.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@thingelstad.com
Summary|ask query offset has a |SMW Ask query offset has a
|hardcoded limit |hardcoded limit

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=49203

--- Comment #1 from Jamie Thingelstad <***@thingelstad.com> ---
I've now hit this issue a few times, but just now did the debugging to figure
out what was going on. I've been recently trying Yaron's new Miga [1] project
and have been trying to use it with the Wikinosh [2] dataset. Wikinosh has
21,000+ food items and I hit this bug trying to page through results and export
everything to Miga [3].

[1] http://migadv.com
[2] http://wikinosh.com/wiki/Main_Page
[3] https://github.com/yaronkoren/miga/issues/4

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=49203

Nemo <***@tiscali.it> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|UNCONFIRMED |NEW
CC| |***@tiscali.it
Ever confirmed|0 |1

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=49203

[[kgh]] <***@kghoffmeyer.de> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@kghoffmeyer.de

--- Comment #2 from [[kgh]] <***@kghoffmeyer.de> ---
This bug is now recorded on GitHub as issue 242 [1]


[1] https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/242

[b***@wikimedia.org]

https://bugzilla.wikimedia.org/show_bug.cgi?id=49203

[[kgh]] <***@kghoffmeyer.de> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |FIXED

--- Comment #3 from [[kgh]] <***@kghoffmeyer.de> ---
This bug was fixed with pull request 637 [1]. Thus closing as RESOLVED FIXED.

[1] https://github.com/SemanticMediaWiki/SemanticMediaWiki/pull/637