Bug#809211: tracker.debian.org: sqlite backend not usable
Hello Raphael,
Thanks for a great analysis. I can confirm this is a bug in Django.
I didn’t realize SQLITE_MAX_COLUMN would be the limit here.
Would you mind opening a ticket on https://code.djangoproject.com/?
(Alternatively I can do it.)
Best regards,
--
Aymeric.
> On 8 janv. 2016, at 10:55, Raphael Hertzog <hertzog@debian.org> wrote:
>
> [ CCing upstream Django developer to have its opinion ]
>
> Hello,
>
> On Fri, 08 Jan 2016, Christophe Siraut wrote:
>> Sorry for the confusion, I have the same libsqlite3 version as you, and
>> it is that one I recompiled with SQLITE_MAX_COLUMN 32767.
>
> I can't reproduce the issue with Django 1.8. It looks like a regression of
> Django 1.9 that uses an SQL request to generate quoted values for the
> parameters fed into requests of the from "WHERE foo IN (...value
> list...)".
>
> Also does this also happen if you drop debug_toolbar from INSTALLED_APPS ?
> I have the feeling that this code path is only used when you actually want
> to print the SQL query that has been executed and this is usally only
> needed during development.
>
> The regression has been introduced by this commit:
>
> https://github.com/django/django/commit/4f6a7663bcddffb114f2647f9928cbf1fdd8e4b5
>
> commit 4f6a7663bcddffb114f2647f9928cbf1fdd8e4b5
> Author: Aymeric Augustin <aymeric.augustin@m4x.org>
> Date: Sun Sep 13 09:30:35 2015 +0200
>
> Refs #14091 -- Fixed connection.queries on SQLite.
>
> It's supposed to fix https://code.djangoproject.com/ticket/14091 for
> SQLite. Christophe, can you verify that the issue is gone if you revert
> that change in your Django ?
>
> Aymeric, the above change is problematic because for a request that
> looks like MyModel.objects.filter(foo__in=my_array) it will break as soon
> as my_array goes above 2000 entries in length since that's the default
> limit for SQLITE_MAX_COLUMN and you will trigger it with your huge
> "SELECT QUOTE(?),...." query.
>
> I know the query can also trigger the limit set with
> SQLITE_MAX_VARIABLE_NUMBER which defaults to 999 but at least on Debian
> we increased that limit significantly (to 250000) so that limit
> is not a practical problem. The above limit is more problematic
> as we can't increase it to the same value (the max is 32767) and
> while the limit on the variable number was unreasonably low, I can
> understand the limit on the number of columns much better.
>
> See https://sqlite.org/limits.html
>
> Aymeric, what do you think ? Shall we open a ticket for this regression ?
>
> Can you update the code to process parameters by batch of less than 2000
> entries ?
>
> You can look at https://bugs.debian.org/809211 for the former discussion
> in this bug report.
>
> Cheers,
> --
> Raphaël Hertzog ◈ Debian Developer
>
> Support Debian LTS: http://www.freexian.com/services/debian-lts.html
> Learn to master Debian: http://debian-handbook.info/get/
Reply to: