Bulk validation and creation

[Tom Gowland]

Hello all,

I hope this hasn't been asked before here (I've had a search and couldn't
find anything!).

I am running a reasonably large POST operation on my API (around 32,000
JSON objects).
Unfortunately, processing this POST takes more than a minute to complete
with the longest steps being the validation and creation of the entries.

When I use django-extensions with the --print-sql flag, I see that each
validation step is done as a separate database query. The same is true for
each object creation.
I'm guessing that if I did each of these operations in one query then
things would be much faster but I can't find any information on how to do
it!.

Please have a look at my SO post for the source
code: https://stackoverflow.com/questions/53364738/django-rest-framework-serialization-post-is-slow/53369867#53369867

Many thanks,
Tom

[Xavier Ordoquy]

Hi,

This is an issue I faced 2 or 3 years ago.
I ended up rewriting most of the serializer with custom code.
I checked a few things by myself first and used bulk inserts.
If everything went fine, good, I could send back how happy the server was within a few seconds.
If there was errors, I did fallback on a more complex and time consuming check to see where the errors could be and return a meaningful message.
Unfortunately this is private code and I’m not allowed to share it.

Regards,
Xavier O.

Post by Tom Gowland
Hello all,
I hope this hasn't been asked before here (I've had a search and couldn't find anything!).
I am running a reasonably large POST operation on my API (around 32,000 JSON objects).
Unfortunately, processing this POST takes more than a minute to complete with the longest steps being the validation and creation of the entries.
When I use django-extensions with the --print-sql flag, I see that each validation step is done as a separate database query. The same is true for each object creation.
I'm guessing that if I did each of these operations in one query then things would be much faster but I can't find any information on how to do it!.
Please have a look at my SO post for the source code: https://stackoverflow.com/questions/53364738/django-rest-framework-serialization-post-is-slow/53369867#53369867
Many thanks,
Tom
--
You received this message because you are subscribed to the Google Groups "Django REST framework" group.
For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.