So I run a video production company. We have 300TB of archived projects (and growing daily).

Many years ago, our old solution for archiving was simply to dump old projects off onto an external drive, duplicate that, and have one drive at the office, one offsite elsewhere. This was ok, but not ideal. Relatively expensive per TB, and just a shit ton of physical drives.

A few years ago, we had an unlimited Google Drive and 1000/1000 fibre internet. So we moved to a system where we would drop a project onto an external drive, keep that offsite, and have a duplicate of it uploaded to Google Drive. This worked ok until we reached a hidden file number limit on Google Drive. Then they removed the unlimited sizing of Google Drive accounts completely. So that was a dead end.

So then we moved that system to Dropbox a couple of years ago, as they were offering an unlimited account. This was the perfect situation. Dropbox was feature rich, fast, integrated beautifully into finder/explorer and just a great solution all round. It meant it was easy to give clients access to old data directly if they needed, etc. Anyway, as you all know, that gravy train has come to an end recently, and we now have 12 months grace with out storage on there before we have to have this sorted back to another sytem.

Our options seem to be:

  • Go back to our old system of duplicated external drives, with one living offsite. We’d need ~$7500AUD worth of new drives to duplicate what we currently have.
  • Buy a couple of LTO-9 tape drives (2 offices in different cities) and keep one copy on an external drive and one copy on a tape archive. This would be ~$20000AUD of hardware upfront + media costs of ~$2000AUD (assuming we’d get maybe 30TB per tape on the 18TB raw LTO 9 tapes). So more expensive upfront but would maybe pay off eventually?
  • Build a linustechtips style beast of a NAS. Raw drive cost would be similar to the external drives, but would have the advantage of being accessible remotely. Would then need to spend $5000-10000AUD on the actual hardware on top of the drives. Also have the problem of ever growing storage needs. This solution we could potentially not duplicate the data to external drives though and live with RAID as only form of redundancy…
  • Another clour storage service? Anything fast and decent enough that comes at a reasonable cost?

Any advice here would be appreciated!

  • MrB2891@alien.topB
    link
    fedilink
    English
    arrow-up
    3
    ·
    11 months ago

    NAS.

    Over the last 24 months I’ve built 300TB (a mix of 10 and 14TB disks) for $2500 in disks. I could do that right now for $2100. A 18TB LTO9 tape is more expensive than what I’m paying per TB for 14TB disks.

    $700 in hardware to build the NAS with 25 bays.

    Glacier would cost you $1080/mo in storage fees alone (300,000GB @ $0.0036) not including the $0.09/GB to get any data back out. Deep Glacier is less (by half, for storage), but comes with strings attached.

    Don’t forget to factor in labor hours of what it’s going to cost you to maintain a tape library or a local server in general.

    Are you charging clients for long term storage after a project is complete? If not, you should be.

  • Simple-Purpose-899@alien.topB
    link
    fedilink
    English
    arrow-up
    3
    ·
    11 months ago

    AWS Glacier Deep Freeze is designed for this. Something you access a couple of times per year if that, and it’s $.99/TB/mo. Price that out compared to a $10k NAS or tape backup that will still need consumables like drives and tapes, and it might be your best option. There are costs on retrieval, but since as you’ve said this is archive footage that customers might request you could pass that cost down to them.

  • Ok_Crow_2386@alien.topB
    link
    fedilink
    English
    arrow-up
    2
    ·
    11 months ago

    Have you considered Amazon S3? It’s made for enterprises with unlimited storage, a lot of pricing options and could save you a lot of headaches long term.

    • chili_oil@alien.topB
      link
      fedilink
      English
      arrow-up
      1
      ·
      11 months ago

      s3 is designed with high availability and high throughput in mind, op needs a cold storage solution like aws glacier or azure cold storage. but even that is not cheap

  • subven1@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    11 months ago

    Really depends on how often you need to touch your data. Tape has high upfront cost (4-5k $ for a LTO-9 tape drive + ~3,5 $/TB in tapes) but you don’t have to worry about archive space anymore. Otherwise, NAS space (if you selfhost) is ~15 $/TB + a server which would also be slightly above 5k right now to store your 300 TB.

  • senzapatria@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    11 months ago

    Try AWS Amazon S3 glacier service. If you’re talking about work data that generate income, you probably should go for a professional storage solution.

    • jkirkcaldy@alien.topB
      link
      fedilink
      English
      arrow-up
      1
      ·
      11 months ago

      Until you need to actually restore a project consisting of multiple TB and it would have been cheaper to get a local backup server.

    • campster123@alien.topOPB
      link
      fedilink
      English
      arrow-up
      1
      ·
      11 months ago

      Agreed. But we’re a relatively small business, so need a balanced cost. AWS is ~$10,000AUD/month for what we’re after from memory.

      • fivenines-@alien.topB
        link
        fedilink
        English
        arrow-up
        2
        ·
        11 months ago

        AWS as well as Azure provide cold storage on the order of $1/TB/mo. There are caveats, such as retrieval costs and such, but depending on your situation that might be OK.

      • r0ck0@alien.topB
        link
        fedilink
        English
        arrow-up
        1
        ·
        11 months ago

        AWS is ~$10,000AUD/month

        Was that for Glacier?

        Or just regular S3 / something else?

      • hwarner1211@alien.topB
        link
        fedilink
        English
        arrow-up
        1
        ·
        11 months ago

        I use this for a client - they have an on site server, which backs up to Azure Archive Tier storage. They have around 60TB up there and pay just under $100 per month.

        Message me if you like and I’ll go into exactly what we did, but it works well for them!

    • campster123@alien.topOPB
      link
      fedilink
      English
      arrow-up
      1
      ·
      11 months ago

      Yeah sweet. I haven’t checked in on the Slow Mo Guys storage setups in a while. I’ll have a watch.

  • Molecule_Man@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    11 months ago

    Have you looked at any MAMs?

    Most will combine on-premise and multiple cloud storages and then proxy a low res for previewing, with custom metadata modeling to find and retrieve everything.

    Sounds like what you need.

  • Spare-Appeal4422@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    11 months ago

    IMO it depends on how organized you are and how often you need to access archived video.

    LTO-9 is cheaper per TB (haven’t run the numbers, but on the order of 100s of TB it’s almost definitely true) but relies on someone physically finding the right tape and putting it into the system (unless you shell out for a very expensive automated system). Not good for fast access, but cheaper for expanding.

    If you need fast, automated access I’d recommend the NAS option, but keep in mind that it would be in one physical location. A fire or flood and you’re fucked.

    Plus, since the cost per TB of tape is so much cheaper than HDD, expanding your archive is probably much cheaper with tape (keeping in mind the organization/automation aspect)

  • BryceJDearden@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    11 months ago

    RAID is not a backup! A single raid array in a single server is still only one copy and one very big single point of failure.

  • FragileRasputin@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    11 months ago

    Back blaze B2, or Wasabi seem to be a lot cheaper than going AWS S3 I’ve checked with wasabi a while ago and there are no fees for downloading/uploading. $6/TB/month

  • vinsan98@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    11 months ago

    I’d recommend the hybrid approach with NAS and Tape Drive

    Build a robust NAS system for remote accessibility, but consider setting up a hierarchical storage management (HSM) system. Frequently accessed or recent projects can reside on the NAS, while older and less accessed ones can be automatically moved to more cost-effective storage.

    Invest in LTO-9 tape drives for archival purposes. While the upfront cost is more but tapes provide long-term, cost-effective storage. This is particularly useful for archival data that doesn’t require frequent access. It adds an extra layer of redundancy and security.

  • sandbagfun1@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    11 months ago

    Raid/NAS, as many others have said, isn’t a backup.

    However, you could have a single NAS and backup to AWS Glacier where storage costs for larger files is cheap going in and getting out in DR scenario is expensive, but maybe covered by your insurance depending on the DR event.