TIL: Django has a field limit

• Well, technically, this happened to me last week, but I’m still going to title this as TIL (Today I Learned) instead of LWIL (Last Week I Learned).
• So, there I was, knee-deep in a schema change for performance reasons, wanting to see if my tweaks made a difference in terms of speed and efficiency. I obviously wanted to try the changes locally first to check if they were actually more efficient as I had planned.
• Thing is, my database was already chock-full of data. I needed to clear them out before I could start testing. There were more than 1,000 (important number we’ll come back to) instances in the model I was modifying.
• So I logged into the Django admin page, selected all the instances, chose the delete action, and hit Go. That’s when the TooManyFields error you see below popped up.

The number of GET/POST parameters exceeded settings.DATA_UPLOAD_MAX_NUMBER_FIELDS.
Bad Request: /mappings/
[31/Jan/2024 5:55:33] "POST /mappings/ HTTP/1.1" 400 121729

• This was the first time I’ve seen this kind of error. Usually when I performed CRUD operations on instances, it’s always been in chunks of 100s and not all at once. And they also didn’t have as many fields as the one I was currently working with.
• So I looked a bit into it and found this. Django has a maximum number of GET/POST parameters that will be read before a SuspiciousOperation is raised.
• In this case, the SuspiciousOperation was TooManyFieldsSent. As you can see, by default, Django limits the number of fields sent in a request to 1,000.

# Maximum number of GET/POST parameters that will be read before a
# SuspiciousOperation (TooManyFieldsSent) is raised.
DATA_UPLOAD_MAX_NUMBER_FIELDS = 1000

• So to get over this, all you have to do is increase the value in your very own settings file or set it to None.

DATA_UPLOAD_MAX_NUMBER_FIELDS = 100_000 # you can use None as an alternative if you don't know what number to set

• Here’s the doc talking more about it.

Note:

• The setting is there for a reason — to protect you from certain types of denial-of-service (DoS) attacks. Unless your application legitimately requires accepting requests with a large number of fields, don’t change it. Large requests could be used as a denial-of-service attack vector if left unchecked.
• If there are a large number of fields in your request, consider chunking the logic so that each request sends a reasonable number of fields at once. (reasonable number of fields is dependent on your application)
• Since DRF (Django REST Framework) builds upon Django’s core functionality, including its handling of request data, this setting will also affect incoming processes if the data is sent as form data.
• So, next time you find yourself facing this error, remember it’s not just a roadblock; it’s Django looking out for you, keeping potentially DoS attacks at bay. And, if you ever need to lift that limit, now you know how to do it safely—just don’t forget why it’s there in the first place. 🦄️