Skip to content
Register

My staging push overwrote production uploads folder

Web Hosting by carlos2 8 replies 435 views
carlos2

Member OP

#1

Rsync -avz --delete from staging to prod. Seemed fine. Then client calls: "where are our June files?" All user uploads gone. 6 weeks of invoices, photos, signed documents. Backup was "handled by the provider" which meant weekly snapshots that rotated before I noticed.

Which providers take XMR for immutable storage? I need something with object lock, versioning, can't delete even if I want to.

Currently on OVHcloud shared, looking to migrate. This was entirely my fault but their snapshot frequency didn't help.

not your keys, not your coins
MI

Member

#2

Redirected to /dev/null

Your --delete flag was the pipe to destruction. Data did not ignore you. You ignored data. Pipe your rsync through a dry-run first. Always.

EOF

pam3

Member

Latency Witch

#3

Observed 1.2e4 ms of panic in your post. No contractions in crisis.

RFC 5165 [1] discusses data preservation in distributed systems. Your provider's snapshot interval of 7.776e5 seconds exceeds mean time to human error by factor of 10^2.

| dry-run default
| immutable flags
| separate upload mount

[1] not a real RFC. Observed humor.

100% packet loss at hop 3
ReykjavikFilch

Member

CDN skeptic

8 #4
carlos2 said:
Rsync -avz --delete from staging to prod

Whence came this practice of mirroring betwixt environments, wherefore did ye not first consult the ancient texts of --dry-run, that noble flag which showeth what would transpire without the actual transference of bits, methinks the olde server administrators of yore did tremble at such hubris, for the uploads folder is ever the most vulnerable of directories, containing as it does the unpredictable contributions of common users who know not the difference between staging and production, nor should they be expected to know, for that is the burden of the administrator, who must bear it like a winding mountain road that twists and turns until he forgets where he began and whether he shall ever reach his destination or merely find himself back at the starting point with fewer files than whence he departed.

gert0

Member

#5

SWEdish FIKA BREAK then FIX

I wrote "säfe sync" wrapper. DRY-RUN default. --delete requires --I-am-sure flag. Umlauts for safety.

FIKA your code before prod push.

[code]#!/bin/bash
# sÄfeSync v0.3
DRY_RUN=1
If [[ "$*" == *"--i-am-sure"* ]]; then
DRY_RUN=0
Fi
rsync -avz ${DRY_RUN:+--dry-run} "$@"

Hösting is serious business.

Craig

Member

scheduled pessimist

#6

Em-dash abuser—I feel your pain—my first week at RackNerd—I did the exact same thing—except it was a database not uploads—--delete on mysql dump—actually no—worse—it was a pg_dump—took down a nonprofit's donor records—distracted by—new concern—have you checked if your provider even keeps logs of what was there—immutable storage is the only way—I switched to object storage with—wait—versioning alone isn't enough—you need legal hold—WORM—write once read many—like old optical disks but in the cloud—

Doug

Member

New Jersey

#7

Weekly snapshots? That's barely a backup lol

grabs popcorn, checks /r/drama
BE

Member

#8

Which provider was "handling" these snapshots

adam20

Member

Ex-Abuse Desk

#9

OVHcloud shared and no local backup? Ouch

reported. resolved. repeat.

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft