By lakises - 15 September 2023 9:03 PM
Hello,
I have a few questions about different backup strategies. Any help would be appreciated.
1) I'm planning to secure my data using files and folders backup to a "primary backup disk". I want, additionally, to make an off-site copy of the primary backup disk to a secondary backup disk every now and then, copying the backup set(s) to an external drive kept elsewhere. Is it a good idea to do this by setting Macrium up to make an image of the primary backup disk? Or would it be better to use again files and folders and why? This time the files that will be backed up for the secondary disk consist of the original backup files only (the ones on the primary disk). Or would xcopy or robocopy do the second phase (off-site copying) somehow better?
2) If I use Macrium for file and folder backup often and then again Macrium for creating image of the primary backup disk more rarely, can I expect to restore everything from the off-site disk without problems should the primary backup disk fail? I do understand that the secondary backup disk will not contain the most recent changes made after the most recent off-site backup, but are there other problems in this approach?
3) More generally, I understand the basic difference between backing up an image vs files and folders. But I am not sure what are the implications. I mean which approach to use for what and when? I assume that it is best to use files and folders method when backing up data whereas backing up disks containing software (like folder Program Files) is better to do as an image. Am I right and are there other easy rules of thumb?
|
By DanDanz - 15 September 2023 9:20 PM
There are other things to consider. What would you do if the system disk failed? You would need an image backup of all system partitions on that disk in order to easily recover to a replacement disk. 2nd question? Is your primary FOLDER backup repository on that same disk? In a different partition? Will the frequency of full/diff/incr backups be suitable if you included the repository data in the image backup. If repository is on another disk, how will you recover ALL the data on that disk in the event that second disk failst?
|
By lakises - 15 September 2023 10:17 PM
+xThere are other things to consider. What would you do if the system disk failed? You would need an image backup of all system partitions on that disk in order to easily recover to a replacement disk. 2nd question? Is your primary FOLDER backup repository on that same disk? In a different partition? Will the frequency of full/diff/incr backups be suitable if you included the repository data in the image backup. If repository is on another disk, how will you recover ALL the data on that disk in the event that second disk failst?
Thanks! I understand that the system recovery needs to be addressed as well. Just wanted to keep it simpler and limit the question to data backup.
I have data both on the system disk and two other disks. The backup repository (I guess that means the disk where the backup is stored, i.e. what I called primary backup disk) is a separate disk. Does this change the feasibility of my scenario?
I'm sorry I don't understand the question about frequency of backups, could you elaborate?
Also, the last question I don't understand. The primary backup would be used as the default backup if my data disk(s) failed. The secondary backup would be used for the purpose of the unlikely event that both the data disks and the primary backup disk failed at the same time.
|
By JK - 15 September 2023 10:39 PM
One alternative option you can consider would be instead of designating a primary and secondary disk, just swap the two disks out periodically, so that the disk that is the destination of your backup job (what you're calling your "primary backup disk") alternates. Thus, if you swap the disks every month, one disk would have backups from January, March, May, July, etc., while the second disk would have backups from February, April, June, August, etc. One benefit of doing it this way is that the two backup disks would only be in the same location very briefly, and you would never have the source computer and both backup disks in the same location all at once. Consider the consequences of something bad happening (ransomware attack, electrical surge, house fire, etc.) while you have the two disks and your computer in the same location, when you are transferring data from the "primary" to the "secondary" disk.
Some users do make a "secondary" copy of a "primary" Reflect backup, but my impression is that those users do not use Reflect for this purpose (instead using robocopy, syncthing, etc.).
|
By DanDanz - 15 September 2023 11:14 PM
+xOne alternative option you can consider would be instead of designating a primary and secondary disk, just swap the two disks out periodically, so that the disk that is the destination of your backup job (what you're calling your "primary backup disk") alternates. Thus, if you swap the disks every month, one disk would have backups from January, March, May, July, etc., while the second disk would have backups from February, April, June, August, etc. One benefit of doing it this way is that the two backup disks would only be in the same location very briefly, and you would never have the source computer and both backup disks in the same location all at once. Consider the consequences of something bad happening (ransomware attack, electrical surge, house fire, etc.) while you have the two disks and your computer in the same location, when you are transferring data from the "primary" to the "secondary" disk. Some users do make a "secondary" copy of a "primary" Reflect backup, but my impression is that those users do not use Reflect for this purpose (instead using robocopy, syncthing, etc.).
I clone the repository disk monthly, but because the 4 disks {1 live, 3 in rotation} are BaraCuda SMR disks, the clone takes 6+ hours. I like your scheme. It could use one BDF with alternate repository volumes scheduled to run every backup period, rather than 3 different BDFs each with a different target, scheduled to repeat run every 3 months.
|
By jphughan - 15 September 2023 11:32 PM
If you want to back up the entire contents of a disk or partition, make an image backup. You can still extract individual files and folders from it. The only exception would be if you had complex custom NTFS permissions in your data set, in which case a File & Folder backup can be easier to restore those. If you don't understand what I'm talking about here, then it won't apply to you. But if you only want to back up a subset of the data on a partition, then an F&F backup might be preferable. For a given amount of data, an image backup is typically much faster than File & Folder, fyi.
Reflect has a "directory synchronization" option if you choose to generate a PowerShell script from a definition file. That can be used to have Reflect replicate the main destination folder to a secondary location after the backup completes. Otherwise, there are certainly other ways to sync files across locations. And as noted above, you could alternatively have a destination rotation strategy.
|
By DanDanz - 15 September 2023 11:33 PM
+x+xThere are other things to consider. What would you do if the system disk failed? You would need an image backup of all system partitions on that disk in order to easily recover to a replacement disk. 2nd question? Is your primary FOLDER backup repository on that same disk? In a different partition? Will the frequency of full/diff/incr backups be suitable if you included the repository data in the image backup. If repository is on another disk, how will you recover ALL the data on that disk in the event that second disk failst? Thanks! I understand that the system recovery needs to be addressed as well. Just wanted to keep it simpler and limit the question to data backup. I have data both on the system disk and two other disks. The backup repository (I guess that means the disk where the backup is stored, i.e. what I called primary backup disk) is a separate disk. Does this change the feasibility of my scenario? I'm sorry I don't understand the question about frequency of backups, could you elaborate? Also, the last question I don't understand. The primary backup would be used as the default backup if my data disk(s) failed. The secondary backup would be used for the purpose of the unlikely event that both the data disks and the primary backup disk failed at the same time. Yes the repository is where you store backups.
Some users backup the critical system partitions on a less frequently scheduled image backup than a separate, more frequent backup of separate data partitions. If you backed up all of the disk in the same backup, you'd have to do it at the needed frequency for the data which increases repository size.
You have answered the last question as long as you understand that to replace a failed data disk your scheme requires you to copy data from the surviving data disk to the replacement disk.
I think in terms of IMAGE backups instead of F&F backups.
The questions were rhetorical, to make sure you considered other factors.
|
By lakises - 16 September 2023 8:33 AM
+xIf you want to back up the entire contents of a disk or partition, make an image backup. You can still extract individual files and folders from it. The only exception would be if you had complex custom NTFS permissions in your data set, in which case a File & Folder backup can be easier to restore those. If you don't understand what I'm talking about here, then it won't apply to you. But if you only want to back up a subset of the data on a partition, then an F&F backup might be preferable. For a given amount of data, an image backup is typically much faster than File & Folder, fyi. Reflect has a "directory synchronization" option if you choose to generate a PowerShell script from a definition file. That can be used to have Reflect replicate the main destination folder to a secondary location after the backup completes. Otherwise, there are certainly other ways to sync files across locations. And as noted above, you could alternatively have a destination rotation strategy.
Thanks! Would these "complex custom NTFS" permissions include my setting when I have multiple users on the computer (family members) with their dedicated default data folders (created automatically by Win 10 Pro) as well as further data folders on disks other than the system disk, where I have manually set restrictions as to who can access the folder. E.g. e:\dataJohn is a folder that only John and admins and system can use.
|
By lakises - 16 September 2023 9:29 AM
+x+x+xThere are other things to consider. What would you do if the system disk failed? You would need an image backup of all system partitions on that disk in order to easily recover to a replacement disk. 2nd question? Is your primary FOLDER backup repository on that same disk? In a different partition? Will the frequency of full/diff/incr backups be suitable if you included the repository data in the image backup. If repository is on another disk, how will you recover ALL the data on that disk in the event that second disk failst? Thanks! I understand that the system recovery needs to be addressed as well. Just wanted to keep it simpler and limit the question to data backup. I have data both on the system disk and two other disks. The backup repository (I guess that means the disk where the backup is stored, i.e. what I called primary backup disk) is a separate disk. Does this change the feasibility of my scenario? I'm sorry I don't understand the question about frequency of backups, could you elaborate? Also, the last question I don't understand. The primary backup would be used as the default backup if my data disk(s) failed. The secondary backup would be used for the purpose of the unlikely event that both the data disks and the primary backup disk failed at the same time. Yes the repository is where you store backups. Some users backup the critical system partitions on a less frequently scheduled image backup than a separate, more frequent backup of separate data partitions. If you backed up all of the disk in the same backup, you'd have to do it at the needed frequency for the data which increases repository size. You have answered the last question as long as you understand that to replace a failed data disk your scheme requires you to copy data from the surviving data disk to the replacement disk. I think in terms of IMAGE backups instead of F&F backups. The questions were rhetorical, to make sure you considered other factors.
Thanks again! When you wrote: "... as long as you understand that to replace a failed data disk your scheme requires you to copy data from the surviving data disk to the replacement disk." Isn't this the case always, I mean to copy data from the backup disk to the new disk on the computer (I assume that's what you mean with "replacement disk"). Or is there something more complicated in the method I suggested as compared to some other strategy?
|
By DanDanz - 16 September 2023 1:30 PM
+x+x+x+xThere are other things to consider. What would you do if the system disk failed? You would need an image backup of all system partitions on that disk in order to easily recover to a replacement disk. 2nd question? Is your primary FOLDER backup repository on that same disk? In a different partition? Will the frequency of full/diff/incr backups be suitable if you included the repository data in the image backup. If repository is on another disk, how will you recover ALL the data on that disk in the event that second disk failst? Thanks! I understand that the system recovery needs to be addressed as well. Just wanted to keep it simpler and limit the question to data backup. I have data both on the system disk and two other disks. The backup repository (I guess that means the disk where the backup is stored, i.e. what I called primary backup disk) is a separate disk. Does this change the feasibility of my scenario? I'm sorry I don't understand the question about frequency of backups, could you elaborate? Also, the last question I don't understand. The primary backup would be used as the default backup if my data disk(s) failed. The secondary backup would be used for the purpose of the unlikely event that both the data disks and the primary backup disk failed at the same time. Yes the repository is where you store backups. Some users backup the critical system partitions on a less frequently scheduled image backup than a separate, more frequent backup of separate data partitions. If you backed up all of the disk in the same backup, you'd have to do it at the needed frequency for the data which increases repository size. You have answered the last question as long as you understand that to replace a failed data disk your scheme requires you to copy data from the surviving data disk to the replacement disk. I think in terms of IMAGE backups instead of F&F backups. The questions were rhetorical, to make sure you considered other factors. Thanks again! When you wrote: "... as long as you understand that to replace a failed data disk your scheme requires you to copy data from the surviving data disk to the replacement disk." Isn't this the case always, I mean to copy data from the backup disk to the new disk on the computer (I assume that's what you mean with "replacement disk"). Or is there something more complicated in the method I suggested as compared to some other strategy?
I said "copy" as opposed to "restore from backup". As I understand it, you are creating mirrors of your repositories (maybe that's the disconnect). I'm not clear whether the repository disks themselves are to be backed up by Reflect (which might be overkill). If not, then when one of them fails, you won't be able to restore from a backup image of it, you will be forced to install a new disk and copy or clone the contents of the surviving mirror of that disk to the new disk. On the other hand, if a DATA or SYSTEM disk fails, you can restore its latest image from a chosen backup image on one of the repository mirrors. Also, you're thinking of File-and-Folder backups for some things. The will let you restore only those folders if a DATA or SYSTEM disk fails. You'll have to have a backup image if the original disk, and if you don't have it, then you'll have to build the new disk from scratch and then you can restore the F&F backup. A previous update to this post by @jphughan gave you excellent reasons why image backups may work better for your circumstances. Personally, I don't use F&F backups, only IMAGE.
Your interpretation of "replacement disk" is correct -- but you're not thinking of a REPOSITORY disk failure when you say "I mean to copy data from the backup disk to the new disk", I think you mean if a DATA or SYSTEM disk fails, you will RESTORE data to the new DATA or SYSTEM disk from its backup image files on one of the mirrored REPOSITORY disks. That's a given and excellent use of Reflect, but RESTORE is not available with your present scheme if a REPOSITORY disk fails. In your mind, keep REPOSITORY disks separate from DATA or SYSTEM disks, and don't say BACKUP DISKS because it's not specific enough. or is it me that's confused? 
|
By jphughan - 16 September 2023 1:38 PM
Lakises,
The permissions that Windows sets on user profiles aren’t custom in the sense that you are altering them, even though they don’t have the same permissions as their parent folders. So that wouldn’t count.
As for the other locations, if those locations are being included in your backups AND the custom permissions you’ve defined would be arduous to set up again and/or difficult to remember, then there may be a case for using an F&F backup. But if the tweaks were minor, that shouldn’t dissuade you from using an image of an image would otherwise be a better fit. The scenario I was describing was more along the lines of a file server that hosted departmental shares and might even involve several cases of folders spread throughout the hierarchy that have non-inheriting permissions, meaning that they’re more restrictive than their parent folder. If you have a lot of that, reimplementing those customizations and even remembering them all can be a chore.
|
By JK - 16 September 2023 1:52 PM
The scenario I was describing was more along the lines of a file server that hosted departmental shares and might even involve several cases of folders spread throughout the hierarchy that have non-inheriting permissions, meaning that they’re more restrictive than their parent folder.
Wouldn't all of those file/folder permissions permissions be restored properly if one restores the full image? I was under the impression that the loss of NTFS permissions only happens if one partially restores backed up data by mounting the image and copying files/folders from the mounted image back to the original location. And for such scenarios, wouldn't it be possible to create a F&F backup of the folders of interest using the mounted image as a source, then subsequently restoring to the original hard drive location (while preserving NTFS permissions)?
|
By jphughan - 16 September 2023 3:06 PM
That’s correct, but on a file server you’re much more likely to need to restore specific content than roll back an entire partition’s worth of data. So if you’re picking a backup strategy based on your most likely restore scenarios, there’s a case to be made for F&F. And unlike for example a Windows partition, even if the data partition of a file server were lost, there are fewer cases where there’s a significant advantage to being able to restore from an image backup as opposed to just creating a new partition and performing a file-level restore (apart from the image restore being faster).
It’s also technically possible to mount an image backup and then restore specific content such that the original NTFS permissions are restored as well. But that’s not how Windows Explorer does it, and not everyone is comfortable working with tools that achieve the outcome I just described.
|
By JK - 16 September 2023 4:56 PM
JP, thanks for the clarification. I've taken a greater interest in these topics since I recently switched from backing up my data partition using F&F to using image backups (with a significant speed improvement).
not everyone is comfortable working with tools that achieve the outcome I just described.
I am probably someone who would be comfortable with, so if you don't mind sharing what tool you are referring to (and perhaps a hint about what flags/settings are required to copy with NTFS permissions preserved), that would be appreciated. Would xcopy /k/x work?
|
By jphughan - 16 September 2023 5:31 PM
I haven’t had to use command line tools to copy files while preserving custom NTFS permissions in a while, but I think some combination of Xcopy, Robocopy, and/or icacls can get it done.
|
By DanDanz - 16 September 2023 8:45 PM
+xJP, thanks for the clarification. I've taken a greater interest in these topics since I recently switched from backing up my data partition using F&F to using image backups (with a significant speed improvement). not everyone is comfortable working with tools that achieve the outcome I just described. I am probably someone who would be comfortable with, so if you don't mind sharing what tool you are referring to (and perhaps a hint about what flags/settings are required to copy with NTFS permissions preserved), that would be appreciated. Would xcopy /k/x work?
I have never used xcopy, preferring robocopy instead. Perusing it's options, obtained with Robocopy /? these look interesting for what you'd be trying to do. The whole list of options might be overwhelming to some, so I tried to highlight those that might do what you need.
My Legend for this post: Bold - some variants to definitely understand, because they are largely included in other groupings
Gray Background - maybe useful overall, but not germane to security,
Red - recommended
Blue - More explanation needed (i.e. I'm not sure of what this does).
:::
:: Some RoboCopy options :
::
/S :: copy Subdirectories, but not empty ones.
/E :: copy subdirectories, including Empty ones.
/LEV:n :: only copy the top n LEVels of the source directory tree.
/Z :: copy files in restartable mode.
/B :: copy files in Backup mode.
/ZB :: use restartable mode; if access denied use Backup mode.
/J :: copy using unbuffered I/O (recommended for large files).
/EFSRAW :: copy all encrypted files in EFS RAW mode.
/COPY:copyflag[s] :: what to COPY for files (default is /COPY:ADT).
(copyflags : D=Data, A=Attributes, T=Timestamps, X=Skip alt data streams).
(S=Security=NTFS ACLs, O=Owner info, U=aUditing info).
/SEC :: copy files with SECurity (equivalent to /COPY:ADTS).
/COPYALL :: COPY ALL file info (equivalent to /COPY:ADTSOU).
/NOCOPY :: COPY NO file info (useful with /PURGE).
/SECFIX :: FIX file SECurity on all files, even skipped files.
/TIMFIX :: FIX file TIMes on all files, even skipped files.
|
By lakises - 18 September 2023 7:49 AM
+x+x+x+x+xThere are other things to consider. What would you do if the system disk failed? You would need an image backup of all system partitions on that disk in order to easily recover to a replacement disk. 2nd question? Is your primary FOLDER backup repository on that same disk? In a different partition? Will the frequency of full/diff/incr backups be suitable if you included the repository data in the image backup. If repository is on another disk, how will you recover ALL the data on that disk in the event that second disk failst? Thanks! I understand that the system recovery needs to be addressed as well. Just wanted to keep it simpler and limit the question to data backup. I have data both on the system disk and two other disks. The backup repository (I guess that means the disk where the backup is stored, i.e. what I called primary backup disk) is a separate disk. Does this change the feasibility of my scenario? I'm sorry I don't understand the question about frequency of backups, could you elaborate? Also, the last question I don't understand. The primary backup would be used as the default backup if my data disk(s) failed. The secondary backup would be used for the purpose of the unlikely event that both the data disks and the primary backup disk failed at the same time. Yes the repository is where you store backups. Some users backup the critical system partitions on a less frequently scheduled image backup than a separate, more frequent backup of separate data partitions. If you backed up all of the disk in the same backup, you'd have to do it at the needed frequency for the data which increases repository size. You have answered the last question as long as you understand that to replace a failed data disk your scheme requires you to copy data from the surviving data disk to the replacement disk. I think in terms of IMAGE backups instead of F&F backups. The questions were rhetorical, to make sure you considered other factors. Thanks again! When you wrote: "... as long as you understand that to replace a failed data disk your scheme requires you to copy data from the surviving data disk to the replacement disk." Isn't this the case always, I mean to copy data from the backup disk to the new disk on the computer (I assume that's what you mean with "replacement disk"). Or is there something more complicated in the method I suggested as compared to some other strategy? I said " copy" as opposed to " restore from backup". As I understand it, you are creating mirrors of your repositories (maybe that's the disconnect). I'm not clear whether the repository disks themselves are to be backed up by Reflect (which might be overkill). If not, then when one of them fails, you won't be able to restore from a backup image of it, you will be forced to install a new disk and copy or clone the contents of the surviving mirror of that disk to the new disk. On the other hand, if a DATA or SYSTEM disk fails, you can restore its latest image from a chosen backup image on one of the repository mirrors. Also, you're thinking of File-and-Folder backups for some things. The will let you restore only those folders if a DATA or SYSTEM disk fails. You'll have to have a backup image if the original disk, and if you don't have it, then you'll have to build the new disk from scratch and then you can restore the F&F backup. A previous update to this post by @jphughan gave you excellent reasons why image backups may work better for your circumstances. Personally, I don't use F&F backups, only IMAGE. Your interpretation of "replacement disk" is correct -- but you're not thinking of a REPOSITORY disk failure when you say "I mean to copy data from the backup disk to the new disk", I think you mean if a DATA or SYSTEM disk fails, you will RESTORE data to the new DATA or SYSTEM disk from its backup image files on one of the mirrored REPOSITORY disks. That's a given and excellent use of Reflect, but RESTORE is not available with your present scheme if a REPOSITORY disk fails. In your mind, keep REPOSITORY disks separate from DATA or SYSTEM disks, and don't say BACKUP DISKS because it's not specific enough. or is it me that's confused?
Sorry for the delayed reply! And thank you, now I got it from your explanation. I have now a much clearer picture of what various strategies entail. Need to think. :
|
|