The 180 MB scan that killed the pod: user uploads that never touch your app server

The first bug report said the upload button was broken. It wasn't: the ingress in front of a client's document portal was returning 413 Request Entity Too Large to anyone attaching more than a megabyte, because that is what nginx.ingress.kubernetes.io/proxy-body-size does when nobody sets it. We set it to 25m, shipped it in nine minutes, and felt good about the day.

Three weeks later the same button stopped working again, and this time it didn't return anything at all. An architect had attached a 180 MB scan of a building permit — a real document, the kind that portal exists for — and the pod handling the request was OOMKilled before it could answer. The retry hit a different pod and killed that one too. For about four minutes, a single upload took down a portal that four hundred other people were using for something else entirely.

What made that incident worth writing down isn't the crash. It's that both fixes we reached for — raise the proxy limit, raise the memory limit — were the same fix, and it was the wrong one twice. The number was never the problem. The problem was that the bytes were going through the application at all.

Your app server should handle permission, not payload

A file upload looks like a request with a big body, so it ends up wherever requests go. Then it inherits every constraint that was designed for requests carrying a form with six fields.

It occupies memory proportional to the file. await request.formData() in a route handler is a buffering call — the whole body materialises before your first line of logic runs. Right-sizing a pod means giving it the memory its steady state needs plus headroom, and "plus headroom" cannot reasonably mean "plus whatever the largest file anyone will ever attach, times the number of people attaching one right now."

It occupies a worker for the duration of the client's uplink, not yours. A Belgian office on a 10 Mbps upstream needs about two and a half minutes to push 180 MB. That's two and a half minutes of a connection, a request slot and a memory allocation held hostage to somebody else's ADSL, and a rolling deploy in the middle of it kills the upload dead.

It multiplies by replicas and proxies. The bytes cross the ingress, then the pod, then go out again to wherever they're actually stored. Three copies in flight for one file, and the only one that had to exist is the last one.

It makes retries expensive. A connection that drops at 95% starts again at 0%, because an HTTP request body is not resumable. Users experience this as "the site ate my file," and they're not wrong.

None of that is fixed by a bigger number. It's fixed by noticing that your application has exactly one thing to contribute to an upload — deciding whether this person is allowed to put a file there — and that this decision is a few hundred bytes of JSON. The file itself has no business in the same process.

So the shape we use now, on every portal we build: the browser asks the application for permission, the application answers with a short-lived credential, and the browser sends the bytes straight to object storage. The app server never sees a single byte of the file.

The handshake

Three calls, and the middle one doesn't involve you.

import { createPresignedPost } from "@aws-sdk/s3-presigned-post"
 
const MAX_BYTES = 250 * 1024 * 1024
 
export async function POST(request: Request) {
  const session = await requireSession()          // authorisation lives here
  const { filename, contentType } = intentSchema.parse(await request.json())
 
  // The client's filename is a label, never a path. The key is ours.
  const key = `dossiers/${session.dossierId}/${crypto.randomUUID()}`
 
  const { url, fields } = await createPresignedPost(s3, {
    Bucket: "uploads-quarantine",
    Key: key,
    Expires: 300,
    Conditions: [
      ["content-length-range", 1, MAX_BYTES],
      ["eq", "$Content-Type", contentType],
    ],
  })
 
  await db.upload.create({
    id: key,
    dossierId: session.dossierId,
    originalName: filename,           // stored as data, never used as a path
    status: "pending",
    createdBy: session.userId,
  })
 
  return Response.json({ url, fields, key })
}

Four details in there are load-bearing.

Use a presigned POST policy, not a presigned PUT, when size matters. This trips up almost everyone the first time. A presigned PUT URL is a signature over a method, a bucket and a key — it cannot constrain how many bytes arrive, so a five-minute URL for a 250 MB cap is equally valid for a 40 GB object. The POST policy form takes a content-length-range condition that the storage server itself enforces and rejects. If you're on getSignedUrl from @aws-sdk/s3-request-presigner because it looked simpler, the cap you think you have is a number in your frontend.

Expire in minutes. The credential authorises one upload starting now, not a permanent write capability that leaks into someone's browser history. Five minutes is enough to start an upload; it doesn't need to cover the whole transfer.

The key is server-generated. A UUID under a prefix you control. The user's filename becomes a column, and the difference matters more than it looks: filenames are attacker-controlled strings that contain ../, null bytes, 4 000 characters of Unicode and occasionally a different file extension than the file. Keep them as data and none of that is your problem.

The pending row exists before the upload does. You now have a record of every upload that was ever authorised, which is what makes the next section possible.

The browser then posts the file directly to the storage endpoint, with a progress bar that is finally telling the truth, and the bucket needs a CORS rule allowing POST from your origin — the single most common reason a correct implementation fails with an error message that mentions nothing about CORS.

Never believe the browser when it says "done"

The tempting fourth step is a POST /uploads/:key/complete from the client after the upload succeeds. Every portal we've inherited has one, and it is the source of most of their inconsistent data.

It fails in both directions. The user closes the tab between the upload finishing and the callback firing, so a perfectly good object sits in the bucket while the application believes nothing was uploaded. Or the callback is just an HTTP request to your API, so anybody who reads your network tab can mark an upload complete without uploading anything — and if your code then trusts status: "ready" as proof a file exists, every downstream consumer inherits that lie.

Confirm against storage, not against the browser:

export async function POST(request: Request, { params }) {
  const { key } = await params
  const record = await db.upload.findOwned(key, await requireSession())
 
  const head = await s3.send(
    new HeadObjectCommand({ Bucket: "uploads-quarantine", Key: key })
  )  // throws if the object isn't there — that's the check
 
  await db.upload.update(key, {
    status: "uploaded",
    bytes: head.ContentLength,
    etag: head.ETag,
  })
 
  await scanQueue.add({ key })   // the bytes are not trusted yet
  return Response.json({ ok: true })
}

The client call is now a hint that it's worth looking, which is a much weaker thing to depend on — and because it's weak, you're forced to handle the case where it never comes. Two mechanisms cover that. Bucket notifications (MinIO to a webhook, S3 to SQS or EventBridge) tell you about objects nobody announced, which catches the closed-tab case within seconds. And a sweep catches everything else:

DELETE FROM uploads
WHERE status = 'pending'
  AND created_at < now() - interval '1 hour';

An hour, because the presigned credential expired long before that and a pending row older than the credential can never become anything. Run it as a cron next to the application — this is exactly the class of small recurring job that doesn't justify infrastructure, and a table with SKIP LOCKED is already enough if you'd rather have it queued than scheduled.

The bytes still have to be checked — just not by the web request

Moving the transfer out of the application removes the resource problem, not the trust problem. The file is still attacker-supplied; it's now attacker-supplied and sitting in your infrastructure. This is why the bucket in the code above is called uploads-quarantine and why nothing reads from it.

The declared content type is a hint. The browser sends what the OS guessed from the extension, and a caller using curl sends whatever they like. Read the first bytes instead. We do this on our own job application form for exactly this reason, and the check is genuinely four lines:

const SIGNATURES = [
  [0x25, 0x50, 0x44, 0x46],              // PDF
  [0x50, 0x4b, 0x03, 0x04],              // ZIP — so also DOCX, XLSX, PPTX
]
 
const head = Buffer.from(await firstBytes(key, 8))
const known = SIGNATURES.some((sig) => sig.every((b, i) => head[i] === b))

Scan asynchronously, in a job with its own limits. ClamAV over clamd's INSTREAM is the boring choice and the right one. It belongs in a worker with a CPU limit and a memory limit, not in a request — the same reason a backfill is a job beside the deploy and not a step inside it. Same for thumbnailing: a 4 KB PNG can decompress to 40 GB of pixels, and an image library with no pixel-count cap will cheerfully try, in whichever process asked it to.

Two buckets, one promotion. Quarantine takes the upload; a scanned, sniffed, accepted object is copied to the bucket the application reads from. A file that fails moves to a rejected prefix with the reason attached rather than being deleted, because "the portal ate my file" is a support ticket you want to be able to answer.

Serving is the other half, and it's where stored XSS lives. An SVG or an HTML file served from your own origin runs script in your users' session. Serve user content from a separate hostname, always with Content-Disposition: attachment, and never with a Content-Type you took from the upload. Downloads go through short-lived presigned GET URLs generated after an authorisation check — the same pattern as the upload, in reverse. Note the distinction from the CDN in front of our own media: that's public, versioned, build-time content and it should be cached at the edge forever. User uploads are private, per-request and must never be in a shared cache.

Rate-limit the intent endpoint. It's cheap for you and cheap to call, which makes it an attractive way to mint write credentials in bulk. Per-session and per-IP counters are enough, and the same quiet rejection we use on public forms applies: an abuser should learn nothing from the response.

When the file belongs in the ERP

This is where teams building on Odoo undo all of it, usually in one line. The file is uploaded correctly to object storage, and then the integration does the obvious thing and pushes it into the ERP as an attachment:

env["ir.attachment"].create({
    "name": filename,
    "datas": base64.b64encode(file_bytes),
    "res_model": "sale.order",
    "res_id": order.id,
})

datas is base64, which is a third larger than the file. Sent over JSON-RPC it is parsed as one enormous JSON string, decoded to bytes, and written to the filestore — and at no point does any of that stream. A 180 MB scan is roughly 240 MB of base64 arriving as a single field, several copies of which are live in the worker at once. Odoo's limit_memory_hard does what it's there for and kills the worker mid-request, which means you have reproduced the original incident precisely, one service to the left. Putting an API layer in front of the ERP doesn't change the arithmetic either — it just moves which process dies.

The rule we apply to every ERP integration is short: one copy of the bytes exists, and everything else holds a pointer to it. Object storage owns the file. The ERP record gets a URL and enough metadata to be useful — filename, size, checksum, who uploaded it, when. An ir.attachment with type="url" does this natively and renders as a normal attachment in the chatter, so nobody in the business notices the difference:

env["ir.attachment"].create({
    "name": filename,
    "type": "url",
    "url": f"https://files.example.be/d/{key}",   # authorised, short-lived
    "res_model": "sale.order",
    "res_id": order.id,
})

Then make the URL resolve through your application rather than pointing at a public object, so the ERP's own access rules still decide who gets the file. A permanent public link inside a record that only three people can open is an access-control bug with a very long half-life.

There is an honest exception, and we live in it. This site's own job application form takes the CV, validates the signature, and sends the bytes to Odoo as base64 — the thing this section just argued against. It's the right call there because the cap is 10 MB, the volume is a handful a week, and a candidate's CV belongs to the recruitment record and nowhere else; building a quarantine bucket and a scanning worker for that would be infrastructure nobody needs. Under about 10 MB and at low volume, through the app server is fine. Above it, or at any volume where two uploads can overlap, it stops being fine quickly — and the cost of finding out in production is a pod, not an error message.

The bucket is production data now

The last thing that changes is what you're responsible for. Before, "the database is backed up" covered everything a user could lose. Now the application's state is split across two systems that fail independently, and neither is complete without the other.

Both go in the same recovery story. Restoring yesterday's database next to today's bucket leaves rows pointing at objects that exist but aren't referenced, which is survivable. The reverse — today's database, yesterday's bucket — is rows pointing at nothing, which your users experience as deleted files. The restore drill has to include both, and it has to include opening one of the restored files — point-in-time recovery on the database only ever gets you one half of the portal back. None of this is novel, incidentally: Odoo has kept its attachments in a filestore beside the database, not inside it, for years, and its backups have had exactly this shape the whole time.

Turn on versioning and an object lock. A bug that deletes the wrong key is recoverable in about a second with versioning and not at all without it. Cheap insurance, and unlike a snapshot of a Longhorn or Ceph volume it's per-object.

Sweep for orphans in both directions, and alert on one of them. Objects with no row are wasted spend. Rows with no object are broken downloads, and that one is a bug — it means something deleted an object without going through the application, and you want to know on the day it starts, not when a customer clicks a three-month-old link.

Lifecycle rules are policy, not cleanup. "Quarantine objects expire after 7 days" and "rejected uploads expire after 30" are two lines of bucket config that replace a cron job you'd otherwise maintain. If the client's retention obligation is seven years, that's also a lifecycle rule — and a conversation to have before the first upload, not after the first audit.

The short version

An upload is two unrelated operations wearing one request: an authorisation decision that is small, fast and yours, and a byte transfer that is large, slow and has no reason to pass through your code. Split them, and the limits that used to hurt stop existing — your app server's memory no longer depends on how big anyone's PDF is, and the transfer scales with the storage layer instead of with your replica count.

What you take on in exchange is bookkeeping, and it's worth being clear that it isn't free. A pending row before the upload, a server-side confirmation that doesn't trust the client, a sweep for the ones that never arrived, a quarantine bucket with a scanner behind it, and a backup story covering two systems instead of one. Every one of those is straightforward; forgetting any one of them produces a class of bug that only appears with real users.

And keep the exception in view, because the pattern is easy to over-apply. Small files, low volume, one destination — send them through the application and move on. The moment a file is big enough that you're tempted to raise a limit to accommodate it, the limit isn't what's wrong.

It's the shape we use for the portals and internal tools we build, on the clusters we run, against the ERP behind them. If your document portal's incident history contains the phrase "we increased the body size limit," it's worth checking what the next file is going to be.