Any snapraid users?


Author
Message
klepp0906
klepp0906
Junior Member
Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)
Group: Forum Members
Posts: 38, Visits: 149
Is it possible to script a snapraid sync but only after a purge happens?  I use a GFS backup and when it hits its thresholds it purges the oldest sets of data.  

this is danger-zone with snapraid because a full backup set is a huge amount of data. 

if i were to have a disk fail after macrium purges like 1TB but before I sync again at the end of that day, i'd stand to lose 1TB of who knows what (depends on which disk failed).

ideally id have snapraid sync but only after it sees macrium run and purge.   as a non programmer I have no idea how to implement this or if theres even a way to trigger based on purge etc.
Beardy
Beardy
Expert
Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)
Group: Forum Members
Posts: 640, Visits: 2.4K
I'm not a snapraid user, that said:

Making a command run after every backup is as simple as having Macrium generate a script, placing the command after the backup in that script, testing it, & then scheduling it.
So sync every time should be simple.

Making it run contingent on old backups being purged is harder & depends on your ability to script detecting that old backups were purged.

Now the free advice:  If you're going to do RAID at all do it properly with a well known & widely used solution.
Ideally a proper hardware RAID card
Motherboard RAID if you must.
If you're doing software raid, again something widely used.
Storage Spaces
Dynamic Disks

At least then if things go pear shaped any recovery lab or company you're forced to approach has encountered it before & is familiar with dealing with it.

Also, avoid RAID 5 like the plague, with modern big disks it's unequivocally obsolete & provides at best a false sense of security.
Have a look here and scare yourself about the chances of a successful rebuild.
RAID 10,6,1 or don't bother.
klepp0906
klepp0906
Junior Member
Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)
Group: Forum Members
Posts: 38, Visits: 149
Beardy - 8 March 2022 8:16 PM
I'm not a snapraid user, that said:

Making a command run after every backup is as simple as having Macrium generate a script, placing the command after the backup in that script, testing it, & then scheduling it.
So sync every time should be simple.

Making it run contingent on old backups being purged is harder & depends on your ability to script detecting that old backups were purged.

Now the free advice:  If you're going to do RAID at all do it properly with a well known & widely used solution.
Ideally a proper hardware RAID card
Motherboard RAID if you must.
If you're doing software raid, again something widely used.
Storage Spaces
Dynamic Disks

At least then if things go pear shaped any recovery lab or company you're forced to approach has encountered it before & is familiar with dealing with it.

Also, avoid RAID 5 like the plague, with modern big disks it's unequivocally obsolete & provides at best a false sense of security.
Have a look here and scare yourself about the chances of a successful rebuild.
RAID 10,6,1 or don't bother.

yea the running post macrium was the easy part, contingent upon purges was where it all fell apart for me lol.   and yes snapraid is software raid, and it is widely used as its open source and free but i digress.

thank you for the repl
Beardy
Beardy
Expert
Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)
Group: Forum Members
Posts: 640, Visits: 2.4K
Do you use robocopy to mirror the backup elsewhere?
If so you might be able to use the exit status of robocopy to detect that a purge happened.
It'll be 3 if it had to remove things at the other end because of a purge.
This will only work as a detection method if no consolidation of incremental backups took place though, because it'll react on those also.

Generally for anything data critical in Windows I'll take "Provoided & supported by Microsoft themselves", or "Provided by a commercial entity with solid a support system" over "Open source & Free" unless the Open source & free thing has a HUGE support setup like the major Linux distributions.
Edited 8 March 2022 9:38 PM by Beardy
klepp0906
klepp0906
Junior Member
Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)
Group: Forum Members
Posts: 38, Visits: 149
Beardy - 8 March 2022 9:28 PM
Do you use robocopy to mirror the backup elsewhere?
If so you might be able to use the exit status of robocopy to detect that a purge happened.
It'll be 3 if it had to remove things at the other end because of a purge.
This will only work as a detection method if no consolidation of incremental backups took place though, because it'll react on those also.

Generally for anything data critical in Windows I'll take "Provoided & supported by Microsoft themselves", or "Provided by a commercial entity with solid a support system" over "Open source & Free" unless the Open source & free thing has a HUGE support setup like the major Linux distributions.

yea thats the rub.  support is very few and far between as most users are well traveled linux enthusiast types and IT professionals.  I'm just a user.  sure i have a fair amount of experience but in the grand scheme i know where i stand.

as for the consolidations, thats the other issue.  i'm working on preventing them from happening by spacing out my full/differential/incrementals evenly so they just purge instead of trying to roll up but it remains to be seen if it works or not.

i believe since snapraid sync's are done manually and in my case nightly, it will be seen the same as a deletion and annihilate parity data.  I woulda loved a real time parity solution but i wasnt quite ready or willing to change my entire filesystem and such.  More importantly it would have required a format which I could not and would not do.  That leaves me with drivepool in my case. 

To be entirely fair i had no protection at all in the past so even if it comes with caveats its better than i was before.   With the amount of data i now have, trying to revolve disks in under 3 years to decrease likilihood of a loss isnt financially responsible.




Beardy
Beardy
Expert
Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)
Group: Forum Members
Posts: 640, Visits: 2.4K
as for the consolidations, thats the other issue.  i'm working on preventing them from happening by spacing out my full/differential/incrementals evenly so they just purge instead of trying to roll up but it remains to be seen if it works or not.

Well you can turn off incremental retention rules altogether, that has other consequences but stops consolidation.
With that unset, If you always run a diff as your next backup after a full, the diff rules should look after the incrementals.. you'll probably be stuck with many more than you want with that strategy, but it'll kind-of work.
So for example full every first Monday of Month, Diff every Tuesday, & Inc every day.

It's not ideal, but it's about the easiest thing to arrange currently on offer.

believe since snapraid sync's are done manually and in my case nightly, it will be seen the same as a deletion and annihilate parity data.  I woulda loved a real time parity solution but i wasnt quite ready or willing to change my entire filesystem and such.

Think you can set up storage spaces to do parity without a reformat, you can for sure set up mirroring.
Those give you realtime parity.

I've never done parity with dynamic disks, but I know you can set up mirroring without a reformat even there.
Edited 8 March 2022 9:58 PM by Beardy
Beardy
Beardy
Expert
Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)
Group: Forum Members
Posts: 640, Visits: 2.4K
The other question to ask yourself is:
If I had to recreate my array & restore a file & folder backup to it, could I do the job from the rescue environment if I needed to?
How would you manage with a third party solution?

Set up using dynamic disks you can recreate the array using diskpart in the standard rescue environment, even though they're now mostly deprecated.
For Storage Spaces, I think you'd need to modify rescue to include Powershell & the appropriate WMI add-on.. but that's doable, they're both available (from Microsoft) PE/RE add-ons.
I've not had chance to double-check if powershell is available in rescue or not yet.

What you'd do for a third party raid setup might be pretty involved.
Edited 8 March 2022 10:29 PM by Beardy
klepp0906
klepp0906
Junior Member
Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)
Group: Forum Members
Posts: 38, Visits: 149
hmm.  ill have to consider that whole full/differential/lotta incremental chain idea.  like you said, not ideal but concessions are gonna have to be made somewhere lol.  with limited time using valuable brain power just thinking about it is disappointing lol.

as for storage spaces.  requires clean drives last i checked, its one reason i went with drivepool over it.  and snapraid isnt really related to raid.   nor is drivepool.  its just called that.  i was rushing earlier when i said "software raid" i aologize, its all software controlled but not raid at all.   drivepool is just a JBOD with a virtual disk that makes them all appear as one and puts things how you want them/balances them based on rules that you define.   Snapraid you can assign to as many disks as you want, part of an array, single disks, both.   it just keeps parity on them all so if one fails (and you didnt muck with it too much) you can simply restore it.  Great for 25 disks and 150TB of info.  not great when you cant control to a relatively fine degre what is being deleted and when.

additions are fine with parity backups, even renames.  but modifications that change hash, or especially mass deletions between sync's can be super problematic at best.

thats why the recommended action is to temp-delete things.   move them to an excluded folder/recycling bin until after you sync, then perma delete them.  because if something fails and you dont have them to put back, you lose that much in parity info from the disk your trying to restore.  unfortunately macrium makes them disappear into the nether, no option to send them to a folder of your choosing instead or something.  its all or nothing, hence my conundrum. 
Edited 8 March 2022 11:45 PM by klepp0906
Beardy
Beardy
Expert
Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)Expert (817 reputation)
Group: Forum Members
Posts: 640, Visits: 2.4K

Overall I'm struggling to think of a good test, that said, once you hit equilibrium, you'll purge a diff & any associated incs every time a diff runs, & an old full (plus child backups if they're not already purged) anytime a full runs..
So maybe sync anytime the backup type isn't an incremental..


klepp0906
klepp0906
Junior Member
Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)Junior Member (57 reputation)
Group: Forum Members
Posts: 38, Visits: 149
Beardy - 9 March 2022 12:19 AM

Overall I'm struggling to think of a good test, that said, once you hit equilibrium, you'll purge a diff & any associated incs every time a diff runs, & an old full (plus child backups if they're not already purged) anytime a full runs..
So maybe sync anytime the backup type isn't an incremental..


yeaaaa.  now that i think of the complications involved i'm probably just going to exclude the backup folder.   I run a full every 4 weeks and keep 6.  a differential every week and keep 9.  an incremental daily and keep 24.    That would mean a whole lot of manual intervention.  I dont like manual intervention Tongue

honestly the only way having those included in parity info is going to do me any favors is if the specific disk theyre on fails.  25:1 odds its not going to be that disk and will be others instead.   In that scenario macrium purging 1TB of data potentially sets me up for a world of hurt.   worst case 1TB of random films or tv shows or games missing.  and thats hundreds and hundreds.

The loss would be a reference of files and folders and configs and a means of restoring from said point, but id just be starting from day 0 with macrium backups again.  and i fling fulls here or there at usually monthly intervals so...

still trying to determine the best way to automate them into dropbox on some sort of schedule instead of doing that manually too Tongue  dropbox folder is on my ssd and i dont like the idea of all that writing to it in perpetuity.  also dont want to move it as it holds a lot of portable software thats path'd in from that user folder etc. 

looking into a cloud drive of some sort i can stick on a spinner maybe, but im digressing.   #1stworldproblems as they say. 

GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Reading This Topic

Login

Explore
Messages
Mentions
Search