‘Deleted 500’ Every Night for a Year: The Clean-Up That Never Left Page 1

Every night at 1 AM, for about a year, a scheduled job logged the same line: Deleted 500 export files from US Cloudinary tenant. Five times in a row, then it went back to sleep.

Over that same year, about 10,000 export files, 26 GB, piled up in the folder it was supposed to be emptying. Nobody complained, because nothing was slow.

The job wasn’t broken the way a crash is broken. It ran on schedule, called the vendor and logged a number every time. The number was counting the wrong thing, and nobody found that through the log line. I found it listing every file on both Cloudinary accounts to see what the storage line was made of, and 26 GB of it turned out to be a folder a job had been “cleaning” every night. Pulling on that thread turned up a backup setting nobody meant to switch on, a second delete bug in a path that had nothing to do with exports, and roughly 612 GB of backups sitting on the bill.

Two posts. This one is the leaks and what we did about them. The next one is cleaning up what was already dead by the time I noticed.

A Spring Boot monolith that keeps user files on Cloudinary, one account per region (US and EU). Client details are anonymised; the log lines and figures are real, measured over the second week of October 2026.

1. The job, and what its number was counting

When a user exports a report, a worker builds the spreadsheet, uploads it to Cloudinary under an exports/ prefix as a raw asset with the authenticated delivery type, and hands back a download link. Those files never get a database row. They’re meant to be temporary, and a nightly job exists to clear the folder.

The job ran at 1 AM, once per region, and the whole of it fit in one controller method:

// nightly job, per region (US, EU)
int maxAttempts = 5;
do {
  // no cursor: this is always page 1 of the "exports" folder
  exports = cloudinary.listFolder(location, "exports", PRIVATE, 500);
  int nbDeleted = cloudinary.deleteAssetsByIds(location, assetIds(exports));
  log.info("Deleted {} export files from {} Cloudinary tenant", nbDeleted, location);
} while (!exports.isEmpty() && --maxAttempts > 0);

// inside deleteAssetsByIds, per chunk of 100 ids
api.deleteResourcesByAssetIds(chunk, options);
deletedCount.addAndGet(chunk.size());   // counts the request, never reads the answer

List the folder, send every asset id on the page to the bulk delete endpoint, log the count, repeat while the page comes back non-empty, give up after five rounds. The code sends the ids to the bulk delete in chunks of 100, so a full page is five calls behind one log line.

It also ran on every environment that had the scheduler, and three of those point at the same US account. The lock that stops two nodes of one environment running the job together knows nothing about the other environments, so three copies of this loop hit one account in the same minute. That collision had already surfaced once, as a burst of errors in Sentry in July.

The count is the part to stare at. deletedCount.addAndGet(chunk.size()) runs right after the API call returns, and the response goes nowhere. “Deleted 500” means 500 ids were sent. Whether Cloudinary deleted 500 of them, or 4, or none, the line reads the same.

Which is how it managed to read the same for a year.

2. A folder with 11,352 entries, most of them empty

The listing was for a cost exercise: both accounts, file by file, to see where the storage went. The US side came back with 11,352 entries under exports/, 26.1 GB, piling up since 2025 in a folder the job had been “cleaning” five hundred files at a time, every night.

The first page explained the log. Of its 500 entries, 496 had placeholder: true and bytes: 0. Not small files or stale files: entries with no bytes behind them at all. In our listings, that’s what a deleted asset looks like when something is still holding on to it. Trimmed to the fields that matter:

{ "public_id": "exports/report_3f9c.xlsx", "resource_type": "raw",
  "bytes": 0, "placeholder": true }

The Admin API’s resource listing returns up to 500 entries per page and a next_cursor for the next one. The job never passed a cursor, so each of its five attempts fetched page 1, and page 1 was 496 placeholders plus a handful of real files. Send 500 ids, log 500, list again, same page, five times, done. The actual backlog, 9,957 files, sat on pages 2 to 23.

exports/ listing, US account, 500 entries per page

page 1    496 placeholders + a few real files   <- read five times a night
page 2    real files
page 3    real files
 ...
page 23   real files                             <- never requested

23 pages, 11,352 entries, 9,957 real files, 26.1 GB

The EU account runs the same code on the same cron and had no pile. One job, two accounts, one of them clean, which pointed at the accounts rather than the code.

3. The setting underneath: automatic backup was on

The US account had automatic backup enabled. The EU account didn’t. It had been switched on by mistake, which product confirmed when we asked, and nothing in the app relied on it.

What the setting changes is spelled out in the destroy reference:

If you have backups enabled, the deleted asset remains in your backed up storage. Otherwise, the asset is permanently deleted, and can only be recovered via a special request to customer support.

So with backup on, destroy and the bulk delete remove the asset from the product environment and a copy stays in backup storage, restorable from the Media Library or through the Admin API’s restore method. The backup system reserves the public ID so the asset can come back with its history, and that reserved entry is what our listings were showing as placeholder: true, bytes: 0. The Search API leaves them out unless you ask for status=deleted. The resource listing the job used returned them.

destroy("exports/report.xlsx")

BACKUP OFF (EU)                   BACKUP ON (US)

asset removed                     asset removed from the environment
gone from the listing             copy kept in backup storage, billed
storage goes down                 public id reserved for restore
                                  listing: placeholder: true, bytes: 0

The part that turns a confusing listing into a bill is in the backup documentation:

These backups count toward your managed storage quota and can significantly increase usage.

Deleting a file on the US account doesn’t reduce storage. The console’s storage tooltip on 9 Oct, for the whole account:

Only originals get backed up, since Cloudinary can regenerate derived versions, so the 611.7 GB is originals only. It can be larger than the live originals because it also keeps what’s been deleted or replaced since the setting went on, and nearly all of it sits on the US side.

On EU, with the setting off, a deleted export leaves the listing, page 1 refills with real files, and five rounds of 500 keep up. Same code, right outcome, by accident.

4. The exports fix: walk every page, count what comes back

Four changes to the loop and two guards around it.

Walk the folder by cursor, and skip the placeholders. Each page’s next_cursor feeds the next call, and an entry flagged as a placeholder is counted and left alone, because there’s nothing there to delete.

String cursor = null;
while (true) {
  var page = cloudinary.listFolder(location, "exports/", PRIVATE, 500, cursor);
  pages++;
  for (var asset : page.assets()) {
    if (Boolean.TRUE.equals(asset.getPlaceholder())) {
      placeholders++;                       // backup leftover, nothing to delete
    } else if (assetIds.size() < MAX_DELETIONS_PER_RUN) {
      assetIds.add(asset.getAssetId());
    }
  }
  cursor = page.nextPageToken();
  if (cursor == null)                           return stop(END_OF_FOLDER);
  if (assetIds.size() >= MAX_DELETIONS_PER_RUN) return stop(RUN_LIMIT);   // 2,500
  if (pages >= MAX_PAGES)                       return stop(PAGE_LIMIT);  // 100
}

Count what Cloudinary confirms, not what we sent. The bulk delete answers with a map of asset id to status, and only the entries that say deleted count.

// Cloudinary answers { "deleted": { "<asset_id>": "<status>", ... } }; only "deleted" counts
private static int countDeleted(ApiResponse response) {
  Object deleted = response.get("deleted");
  Map<String, String> statuses =
      deleted instanceof Map ? (Map<String, String>) deleted : Map.of();
  return (int) statuses.values().stream().filter("deleted"::equals).count();
}

Treat zero confirmed deletions as an error. A batch of ids that comes back with nothing deleted is the old bug in a new coat, so it throws. The per-region catch logs that at ERROR, which the logging appender forwards to Sentry, and the other region still runs.

for (List<String> batch : windows(assetIds, 500)) {
  int confirmed =
      cloudinary.deleteAssetsByIds(location, batch, Map.of("type", "authenticated"));
  if (confirmed == 0) {
    throw new CloudinaryException(
        "Cloudinary confirmed none of the %d export files sent in %s, %d deleted before"
            .formatted(batch.size(), location, deleted));
  }
  deleted += confirmed;
}
log.info("Deleted {} export files from {} Cloudinary tenant, sent {}, placeholders {}, pages {}",
    deleted, location, assetIds.size(), placeholders, pages);

Cap the night. At most 2,500 deletions and 100 listing pages per account per run. Those two numbers are ours, not Cloudinary’s; the 500 entries per listing page is theirs, and the code keeps sending ids to the delete call in chunks of 100. Hitting the page cap deletes what was collected and writes the log line first, then raises.

Run only where the account is owned. The job checks the active Spring profile and skips itself, with an INFO line, on the three environments that share production’s accounts. Production’s nightly run cleans their exports too, since it’s the same folder.

The new log line carries the numbers that would have told us a year earlier:

Deleted 2500 export files from US Cloudinary tenant, sent 2500, placeholders 496, pages 6

That’s one of the five nights after the deploy, and the other four look the same. deleted equals sent, the placeholders are counted instead of hidden, and six pages means the cursor is doing its job. After the fifth night the backlog of 9,957 files was gone.

5. The second bug: a delete that never said what it was deleting

The same listing had a bigger number in it: 131,883 files across the two accounts with no database row pointing at them, 64.3 GB, of which exports were 11,352. The rest had to come from somewhere, and the delete path in the same Cloudinary service was the next place to look.

Every tracked file has a row with the resource type Cloudinary stored it under (IMAGE, VIDEO, AUDIO or RAW) and its delivery type (PUBLIC or PRIVATE). On production, 344k rows say RAW, and 322k of those are PDFs. When a user deleted one of them, this ran:

public void delete(UUID tenantId, String publicId, String type) {
  var options = ObjectUtils.asMap("type", type);          // "authenticated" | "upload"
  cloudinary(tenantId).uploader().destroy(publicId, options); // resource_type absent -> image
  // response ignored: { "result": "not found" } for a raw PDF looked the same as success
}

// caller
cloudinaryService.delete(tenantId, publicId, file.getUploadType().toCloudinary());
mediaFileRepository.delete(file);

The delivery type is there. The resource type isn’t, and the destroy reference says what happens then: resource_type takes image, raw or video, and defaults to image. The delete guide adds that an asset that isn’t an image, or isn’t delivery type upload, needs resource_type and/or type passed explicitly. We were passing half.

Cloudinary addresses an asset by three parts, not one

              resource_type    type             public_id

stored as     raw              authenticated    docs/contract.pdf
delete sent   (absent)         authenticated    docs/contract.pdf
looked up as  image            authenticated    docs/contract.pdf
              ^^^^^
              the SDK default, so the wrong shelf

For a raw PDF the call went looking for an image named docs/contract.pdf, found none, and came back with {"result": "not found"}. The method returned void, nobody read the answer, and the next line deleted the row. From the user’s side the file was gone. From Cloudinary’s side nothing had happened.

Image deletes worked the whole time, since image is what the default fills in.

A method that returns void has no way to tell you it did nothing.

6. The delete fix: send the type, read the answer, delete the row anyway

public boolean delete(UUID tenantId, String publicId,
                      MediaFileResourceType resourceType, MediaFileUploadType uploadType) {
  Cloudinary cloudinary = cloudinary(tenantId);
  var options = ObjectUtils.asMap(
      "resource_type", resourceType.toCloudinary(),   // image | video | audio | raw
      "type", uploadType.toCloudinary());             // upload | authenticated
  try {
    var response = cloudinary.uploader().destroy(publicId, options);
    return "ok".equals(response.get("result"));
  } catch (Exception e) {
    throw new CloudinaryException("Error deleting file from cloudinary", e);
  }
}

The resource type and delivery type come from the row, which is to say from what the asset was stored under, not from the file extension. That distinction matters: Cloudinary stores PDFs as image by default, and a PDF only lands under raw when the upload said so. Guessing from .pdf would get it wrong in both directions.

The caller reads the boolean and does one thing with it:

var resourceType = Optional.ofNullable(file.getType()).orElse(MediaFileResourceType.IMAGE);
var uploadType = file.getUploadType();
if (!cloudinaryService.delete(tenantId, publicId, resourceType, uploadType)) {
  log.atWarn()
      .addKeyValue("filePublicId", publicId)
      .addKeyValue("fileResourceType", resourceType)
      .addKeyValue("fileUploadType", uploadType)
      .log("No Cloudinary file deleted for media file #{}", publicId);
}
mediaFileRepository.delete(file);   // the user's delete is never blocked

"ok" becomes true. Anything else becomes a WARN with the public id, the resource type and the upload type attached, and the row goes anyway. That last part is deliberate: a file that’s already missing on Cloudinary shouldn’t block a user’s delete, and the WARN is for us, not for them.

A row with no type at all falls back to image, which is what the old request was sending anyway; production has none. Image deletes are unchanged, since image sent explicitly is the same request as image by default. There’s no retry and no back-fill: the files that leaked before the fix are still on Cloudinary, and they belong to the clean-up in the next post.

7. 612 GB of backups, and where to put them

The reflex after section 3 is to switch the toggle off. We didn’t, for two reasons.

A restore path for deleted user documents is a feature, and one we’d rather keep. And the toggle isn’t a clean-up: the backup docs route the removal of existing backups through a support request for Cloudinary’s default storage (cloud name plus the last four characters of the API secret), or through the Admin API’s delete-backed-up-versions call, per asset and irreversible. Switching the setting off is a separate step from getting rid of the 611.7 GB.

What it costs today comes from our plan’s usage API, read on 13 Oct. The US environment sat at 2.29 of its 3.0 monthly credits (76%), with storage at 0.86 of those and transformations the biggest line at 1.35. Storage was 947 GB on the same reading, so on our plan a credit buys roughly a terabyte of storage for a month. That’s our numbers, not a pricing rule. Prorated, the backups are about 0.6 credit a month, a fifth of the monthly allowance.

So the backups move to a bucket we own. Cloudinary backs up to a customer’s own S3, Google Cloud Storage or Azure Blob container on paid plans; the bucket has to be allowlisted by Cloudinary and left alone once in use, and the docs are direct about what it does to the meter:

These backups do not count against your Cloudinary storage usage.

On the AWS price list for us-east-1 (effective 1 Oct 2026), the same 611.7 GB costs:

Storage only; requests and retrievals aren’t in the table. And since Cloudinary wants the bucket without a lifecycle policy, the Standard line is the one to budget; the cheaper tiers are what the bytes would cost, not what you get to configure. On the 13 Oct reading, taking the backups out of Cloudinary’s storage would move the US environment from 76% of its monthly credits to about 56%, which is arithmetic on the usage report, not a measurement.

Fourteen dollars a month, at the vendor whose job that is.

8. What the two fixes leave behind

Both changes stop new leaks. Neither touches what leaked already.

The file-by-file listing put the dead total at 88.6 GB: 90,178 files that have a row but nothing referencing it, 24.3 GB (an upper bound, since references from JSON columns haven’t been measured yet), plus the 131,883 files with no row at all, 64.3 GB. Together that’s about 222k files and a quarter of the 348 GB of original files on the two accounts (the console’s own originals figure is a few GB lower; different measurement, same week).

A quarter of the originals, not a quarter of the bill. Against everything Cloudinary stores, backups and derived versions included, it’s nearer 8%. Still 88.6 GB nobody is using, on an account where a delete keeps a copy.

9. What carries over to any vendor API

A vendor default is a decision you didn’t make, and it’s still yours. resource_type defaults to image, which is the right default for an image service and quietly wrong for 344k raw files. The backup toggle was one click on one screen, and it changed what “delete” means for an entire account. Neither default is unreasonable. The code that depended on them just didn’t know they existed, and tbh neither did I until the listing.

A log that counts what you sent is a diary, not a measurement. “Deleted 500” was a true statement about the request. Cloudinary had an opinion about every one of those ids, in a map keyed by id, and we threw the map away. Read the response, count the statuses, and make zero confirmed deletions an error, not a quiet night.

Three environments on one account is three schedulers with one target. The lock that keeps a job from running twice lives in each environment’s own database, so it can’t see the other two. The fix was embarrassingly small: a profile check, so the job runs where the account is owned and nowhere else.

Two bugs, one year, no incident. Nothing was down, nothing was slow, every user-facing delete succeeded. The only signal was a storage number, and storage numbers get looked at when someone asks what the bill is for. Same blind spot as the feature flag quota that two sensible fixes couldn’t touch: every dashboard called it healthy.

When a log line hasn’t changed in a year, go and find out who’s doing the counting.

That’s the leaks plugged. The 88.6 GB that got out before the plugs is the next post: a nightly job that archives before it purges, and a reference table kept at write time so “is anything still using this file?” is a lookup and not a full listing. For another bug that lived in production for the best part of a year before a side signal gave it away, there’s the duplicate invoice story behind a Slack alert.