Shared media deduplication

Status: Phase 1 + Phase 2 in source (refcount-safe delete, unique-byte quotas, global index default). Deploy Functions + ship client to mark fully live — see docs/SHARED_MEDIA_DEDUP_VERIFY.md.
Surfaces: Communities · Broadcast channels · Open-history groups (OHG)
Out of scope: Stories · 1-to-1 · Whisper private chat · E2E groups

This page explains why Buzzio stores media files as CDN bytes with content-hash reuse on those shared surfaces. Text/captions stay server-held DEK ciphertext.

Short answer

Buzzio still does not sell content. Shared rooms were never operator-blind E2EE — see Sealed vs shared.

Why we do this

1. Shared mode already allows the operator to operate the room

Communities, Broadcast, and open-history groups use server-held or server-managed keys so late joiners, moderation, feeds, and multi-device history work. Encrypting media under that same service-held DEK does not make those blobs operator-blind. It mainly adds upload CPU, download decrypt cost, and unique ciphertext per send.

Dropping media ciphertext on those surfaces is therefore an honest product trade, not a silent downgrade of a sealed promise.

2. Content-addressable reuse (Telegram-style storage savings)

Large messengers avoid storing the same popular video a thousand times:

1. Client computes SHA-256 of the media bytes (before upload).
2. Client asks the server: “Do we already have hash H in this scope?”
3. If yes — reuse the existing Bunny path; skip uploading the file again.
4. If no — upload once; index H → storage_path.
5. Downloaders still get a short-lived signed CDN URL after membership checks.

That only works cleanly when the bytes stored on Bunny match what was hashed. Per-upload AES-GCM (random nonces) makes ciphertext unique even for the same video under the same DEK — so classic dedup fails unless media is stored as the hashed bytes (or a carefully designed shared-ciphertext scheme we are not using here).

3. Text stays encrypted on purpose

Message bodies and captions remain server-held DEK ciphertext in history / Firestore / R2 paths as today:

• Not stored as readable plaintext in product databases
• Still decryptable by the service for moderation, search ops, and late-joiner history (shared-mode design)
• Not end-to-end / operator-blind

Media and text are treated differently because media is large, often repeated, and already effectively readable under service keys — while keeping text ciphertext-at-rest remains good hygiene and matches existing shared-mode docs.

4. What we deliberately leave alone

Access control after the change

Media is not “public forever on the open internet.”

Threat framing: Threat model · Subprocessors.

Dedup vs media pool (two layers)

Shared rooms use two separate systems. Mixing them causes the “can’t delete yet / free re-attach” confusion.

Strong rule (implemented): CDN dedup never gives a free pool. Every media post charges this room’s pool by file size, even on a hash hit. After expiry, re-using the same video (if another room still holds the CDN object) skips upload but still fills this room’s new pool.

Delete rule: “Gone for this room” ≠ “gone from Bunny.” Other rooms can keep the file until they expire or delete their last reference.

flowchart TD
  upload[User posts video]
  hash[SHA-256 lookup]
  put{CDN hash ready?}
  bunny[Bunny PUT once]
  reuse[Reuse path skip PUT]
  pool[Charge THIS room media pool]
  msg[Post message]
  expire[Room pool expires]
  detach[Detach room ACL]
  orphan{Any room left?}
  del[Delete Bunny + hash]
  keep[Keep bytes for other rooms]

  upload --> hash --> put
  put -->|no| bunny --> pool
  put -->|yes| reuse --> pool
  pool --> msg
  expire --> detach --> orphan
  orphan -->|no| del
  orphan -->|yes| keep

What users should expect

• Sending the same video again may skip upload (hash hit) while still posting a normal message and still using media-pool quota.
• When this room’s media pool expires, that room loses access; another room that still has the video keeps it until its pool expires.
• After expiry, posting the same video again can skip CDN upload if another room still holds it, but it counts against the new pool.
• Opening sealed chat or Stories behaves as before.
• Docs and store Data Safety wording must say: shared-room media is CDN-stored with access control; shared-room text remains service-key encrypted at rest — neither is sealed E2EE.

Implementation plan

Status legend: Docs (this change) · Code (not started until engineering picks up the plan).

Phase 0 — Documentation

• Public explanation page (this file)
• Update Communities, Broadcast, Groups, crypto, sealed-vs-shared, architecture, FAQ, changelog, index
• Mark shipped in changelog only after Functions deploy + client release

Phase 1 — Media plaintext path + within-room dedup

Status: implemented in app + Functions source (deploy required).

1. Client (Flutter) — done for Communities / Broadcast / OHG

   • Stop encrypting media blobs before Bunny PUT.
   • Compute SHA-256; lookupOrAllocateSharedMedia before PUT.
   • On hash hit: reuse bunny_path; skip bytes upload.
   • Download: dual-read — no media DEK decrypt when encryption: none; legacy .enc still decrypts.
   • Text/caption encrypt under server-held DEK unchanged.

2. Cloud Functions — done in source

   • lookupOrAllocateSharedMedia / confirmSharedMediaHash
   • Firestore shared_media_hashes/{scopeType_scopeId_sha256}
   • Upload/download path asserts allow .enc or /by_hash/

3. Bunny paths — …/media/by_hash/{sha256}/… (scoped per room)

4. Compat — old .enc objects remain downloadable until retention deletes them.

Phase 2 — Hardening + global index

Status: implemented in Functions source.

• Reference counting / safe delete (releaseSharedMediaPaths) — Bunny object removed only when no room ACL remains
• Global hash index + shared_media/by_hash/ paths (default for new allocations)
• Pool quota always charges this room’s file size (CDN dedup ≠ free pool)
• Firestore rules deny client access to hash collections
• Metrics: anonymous counters in ops_cost_metrics (see docs/OPS_COST_TELEMETRY.md)

Phase 3 — Docs “shipped” + store forms

• Flip changelog to Shipped after production deploy + client release (checklist: docs/SHARED_MEDIA_DEDUP_VERIFY.md)
• Align Privacy Policy / store Data Safety media wording with CDN-stored shared media
• Confirm Stories / sealed media still described as encrypted

Suggested engineering order

Internal code pointers (not public API): lib/features/communities/services/community_device_media_transfer.dart, broadcast_device_media_transfer.dart, open_history_group_device_media_transfer.dart, functions/*/media.js.

Related pages

• Sealed vs shared
• Communities
• Broadcast channels
• Groups
• Groups cryptography
• System architecture
• Stories (unchanged sealed-leaning media)
• Changelog