Are Incremental Backups Safe


Author
Message
jbconnell
jbconnell
New Member
New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)
Group: Forum Members
Posts: 9, Visits: 17
I have a daily incremental file backup with a retention rule of 312 weeks. I am backing up to an external hard drive.

My concern is the integrity of the incremental backup. If the oldest incremental backup becomes corrupt, then all subsequent backups are invalid as well, correct? How likely is that to happen? Should I add an  occasional full backup? Or would the occasional differential be sufficient? I need 6 years of daily "snapshots" so the retention length cannot be reduced. 


Drac144
Drac144
Expert
Expert (632 reputation)Expert (632 reputation)Expert (632 reputation)Expert (632 reputation)Expert (632 reputation)Expert (632 reputation)Expert (632 reputation)Expert (632 reputation)Expert (632 reputation)
Group: Forum Members
Posts: 401, Visits: 1.4K
In a perfect world your plan would be just fine.  In our real world, "stuff" happens.  Your 312 backup limit will be just fine as long as the disk containing those backups is working perfectly.  When that is no longer the case, then, as you say, all backups after the one (or ones) that are corrupt are useless. 

The issue is storage space.  You could do more full backups with incremental backups between them (for example two full backups a month).  But that would use more storage space.  You could use differential backups, which do not depend on other differential backups - just the full backup upon which they are based.  But that would also take a lot more storage and the size of each differential would increase as time passed without another full backup.

Most users find that full backups should be done at least once a month with incremental and/or differential backups daily.  If you have enough external storage, saving a years worth of these is not beyond reality.  Of course if your backups are very large, that could become an issue. 

Bottom line, there is NO magic solution.  If you want reliability AND long term daily recoverability you need to spend the money to get sufficient offline storage to keep enough backup data.  And you need to check the integrity of that offline storage frequently.


Froggie
Froggie
Master
Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)
Group: Forum Members
Posts: 874, Visits: 7.4K
Without duplication, 6-yrs of daily snapshots, that CANNOT FAIL, IMHO, is a crapshoot.  You're asking an awful lot from a storage device not to fail in 6-yrs, especially HDDs.  Their failure modes are much different than SSDs, they can get "flakey" along the way due to many internal mechanical issues, where SSDs, most likely, will just fail.

A successful CANNOT FAIL daily snapshot system definitely needs replication... and probably more than 2.  Maybe a Cloud Service would be a better choice... Google (Gmail) still uses Magnetic Tapes and constantly re-verifies those tapes on a regular basis.
Edited 14 November 2016 9:35 PM by Froggie
jbconnell
jbconnell
New Member
New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)
Group: Forum Members
Posts: 9, Visits: 17
Thanks for the feedback. So if I was to do the "Grandfather, Father, Son" scheme and set the retention rules for each daily, differential, and incremental to 312 weeks then that should preserve 6 years with effectively a daily "snapshot" of each, correct?
Froggie
Froggie
Master
Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)
Group: Forum Members
Posts: 874, Visits: 7.4K
JB, it's difficult to determine exactly what you want to do.  It sounded like you needed a failsafe 6-yr option.  Using INCs for more than 1-day makes you vulnerable to losing multiple days due to INC file failure, that doesn't sound like failsafe to me.

If you can lose some days, give us an idea of what's acceptable and we can probably suggest a GFS (or other) method to accomplish that.  I guess I just don't completely understand your requirements.
jbconnell
jbconnell
New Member
New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)
Group: Forum Members
Posts: 9, Visits: 17

It is important to not lose any data. However, it seems very unlikely that I would experience a (near) simultaneous failure of both the primary data and the backup. The default GFS scheme (monthly, weekly, daily) seems to be a good compromise between maintaining the integrity of the backup and not exceeding storage capacity limits. I've estimated the backup size shouldn't exceed 600 GB using the GFS scheme.

My only uncertainty was the retention rules. If I set the full, differential, and incremental all to 312 weeks then I should effectively have a daily snapshot of 6 years of data, correct (assuming no corruption of the backup files)?


Froggie
Froggie
Master
Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)
Group: Forum Members
Posts: 874, Visits: 7.4K
jbconnell - 16 November 2016 12:31 PM

It is important to not lose any data. However, it seems very unlikely that I would experience a (near) simultaneous failure of both the primary data and the backup.

If the PRIMARY DATA you mention above is the LIVE System, your statement doesn't make any sense.  Your LIVE System doesn't have 312-weeks of different DATA on it, only the most recent day.  If PRIMARY refers to target storage devices containing image DATA, that tells me that replication is in use and makes me feel better about your requirements Smile

If you indeed are using replication (at least 2-copies on 2-different storage devices), you can successfully use your original scheme of a FULL + 2184 INCs (312-weeks of INC retention).  Unless you're using SYNTHETIC FULL, your original FULL will age after 312-weeks and the merging forward of your 2184 INCs will start to get arduous in terms of time (the original INC will get bigger and bigger).  If you need to do a restore with this type of imaging backup, it will also be extremely time consuming due to the fact that the original FULL must be restored followed by the processing of all of tghe 2184-INCs.  Even if you use SYNTHETIC FULL, the forward merging and restoration process will become very time consuming.

The easiest way for you to manage a retention level of 312-weeks and be able to restore in a reasonable period of time would, indeed, be to use a GFS scheme.  It requires more DATA storage but allows for a fairly quick restore.  If you set up all your schedules for the same day and time (Reflect handles the conflicts automatically), schedule a daily INC, a weekly DIFF and a FULL once every 4-weeks with a FULL retention of 79 BACKUPs... you will always have 78-FULLs with all their associated DIFFs and INCs.  After the 80th FULL is taken, the 1st FULL and all its associated DIFFs and INCs will be automatically deleted based on the FULL retention previously set (79 BACKUPs).

It's gonna take some storage, especially if you're using duplication of the images, but should be effective.


Edited 16 November 2016 1:46 PM by Froggie
jbconnell
jbconnell
New Member
New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)
Group: Forum Members
Posts: 9, Visits: 17
Thank-you for the feedback and recommendations. To clarify, you're recommending setting the retention for full based on the number of backups and not on the number of weeks. Is there a reason for this? What's the effective difference?
Froggie
Froggie
Master
Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)Master (1.5K reputation)
Group: Forum Members
Posts: 874, Visits: 7.4K
JB, you can easily use the # of weeks (312) as well... that will allow you to take unscheduled images within your time frame without losing any of the important ones.  I was thinking pure scheduling (automatic images) when I made my suggestion.

The effect will be the same.
jbconnell
jbconnell
New Member
New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)New Member (10 reputation)
Group: Forum Members
Posts: 9, Visits: 17
OK, great. Thanks for all the help!
GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Similar Topics

Reading This Topic

Login

Explore
Messages
Mentions
Search