Feature request - deduplication


Author
Message
gbevan
gbevan
New Member
New Member (7 reputation)New Member (7 reputation)New Member (7 reputation)New Member (7 reputation)New Member (7 reputation)New Member (7 reputation)New Member (7 reputation)New Member (7 reputation)New Member (7 reputation)New Member (7 reputation)
Group: Forum Members
Posts: 5, Visits: 14
Hi. I'm using MSM to backup approx 50 endpoints. They're all a similar spec (hardware, OS, applications etc.). In theory the differences between the images will be minimal. I'd imagine the space required in the repository would decrease drastically if MSM could perform deduplication on the backed up images. Regards.
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)
Group: Forum Members
Posts: 12K, Visits: 72K
This would be a killer feature, and it’s a main reason why one of my clients uses Windows Server Essentials for their workstation backups, which works this way, but I imagine it would involve a significant engineering effort that would cause Reflect Agent backups to run very differently from regular Reflect installations. And in fairness, it does have its drawbacks — none of them necessarily dealbreakers, but worth considering.

First, it would require a centralized data store rather than independent backups, which would in turn mean that a server application would have to be running whenever a client wanted to run a backup in order to “expose” that specific client’s most recent backup so the new data to be backed up could be determined, and to accept the client’s new data into the centralized store. Clients wouldn’t be able to run backups autonomously anymore. Restores would also probably require a running server for the same reason unless Reflect clients were given the code to parse an entire central store.

The server application would also have to handle all purging. In this scenario, the Reflect agents’ retention policy would probably be reduced to just marking certain backups as no longer required, and then the server application would have to verify that no OTHER clients still needed any data blocks from those unneeded backups before actually deleting anything.

I think a centralized store would also require abandoning traditional GFS backup strategies and its concept of Full/Diff/Inc backups. Instead, you would just specify how often backups would run and how long to retain them, perhaps with separate options for how many daily/weekly/monthly backups to retain as WSE offers.

And other unavoidable issue with deduplication is the increased impact of corruption. There are threads about Windows Server Essentials (and previously Windows Home Server) where central store issues prevented any backups from being restored at all. To add insult to injury, Microsoft doesn’t even provide a tool to “export” specific backups out of the store when desired, although there is at least a third-party utility that sort of works, and Macrium could certainly provide this if they implemented a design like this.

Again, I’m not saying any of the above makes this a bad idea. My client loves that their workstation backups occupy so little space and that newly imaged workstations can run their first backup so quickly because so much of the data they have is already in the central store. But I can also see this being a massive engineering effort that would require major changes to the Reflect Agent interface. You’d probably need to allow it to show EITHER the current interface or choose “Connect to central store”, in which case your options for backup strategies and such would be completely different. The Restore tab would also have to populate based on connecting to the store rather than looking at a target folder. At that point it might even be easier to develop an entirely separate agent or product for this strategy.
gbevan
gbevan
New Member
New Member (7 reputation)New Member (7 reputation)New Member (7 reputation)New Member (7 reputation)New Member (7 reputation)New Member (7 reputation)New Member (7 reputation)New Member (7 reputation)New Member (7 reputation)New Member (7 reputation)
Group: Forum Members
Posts: 5, Visits: 14
Hi jphughan. Very interesting insights. I fully agree with everything you've said there. I appreciate it's probably a big ask but as you say, it'd be a killer feature. Regards.
Stewart Ives
Stewart Ives
New Member
New Member (45 reputation)New Member (45 reputation)New Member (45 reputation)New Member (45 reputation)New Member (45 reputation)New Member (45 reputation)New Member (45 reputation)New Member (45 reputation)New Member (45 reputation)New Member (45 reputation)
Group: Forum Members
Posts: 25, Visits: 128
gbevan - 25 May 2018 12:02 PM
Hi jphughan. Very interesting insights. I fully agree with everything you've said there. I appreciate it's probably a big ask but as you say, it'd be a killer feature. Regards.

jphughan, Interesting points but not my cup of tea. When it comes to backups I want to keep it simple and the ability to go to a directory and see a file sitting there just gives me more assurance than a significantly more complicated system that has to put together a restore from multiple threads. I would see one thread not function correctly and you may be SOL. I know the same is with backups that are file based (cylinders/sectors) BUT I have the ability to make multiple copies of that file backup and store it in multiple places, on server, on site, off site.

The main reason I moved to Macrium is its simplicity and the ability for me to move the backup files around like any other file. Move it from a NAS to a USB key, move it from a server down to the workstation, etc.

Hey, keep up the good work.

-stew

jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)
Group: Forum Members
Posts: 12K, Visits: 72K
^ And that’s the exact the flip side that I think many other existing users will share, which is why I said at the end of my post that a central store-based backup solution might be a sufficiently large change to be considered a separate product. I too appreciate Reflect’s “elegance in simplicity”, so to speak, and it’s partly why I moved the client I mentioned above onto it from Backup Exec for their SERVER backups. My favorite example is the fact that it’s so easy to run a disk rotation with Reflect because it makes each disk self-contained with its own backup set, compared to Backup Exec that would write Incrementals of the same set across multiple destinations, which created multi-disk dependencies for restores. Reflect’s design is just so easy to think about and much more reliable. But this client would need a lot more storage to use Reflect for their workstations, whereas WSE minimizes that because it uses deduplication (plus it was included with their server license).

But even if Macrium does offer something like this and decide to build it into Reflect rather than creating something else, I can’t imagine they would remove the ability to stick with the existing design of each client performing completely independent backups. Smile
Edited 25 May 2018 2:24 PM by jphughan
Alex
Alex
Macrium Representative
Macrium Representative (515 reputation)Macrium Representative (515 reputation)Macrium Representative (515 reputation)Macrium Representative (515 reputation)Macrium Representative (515 reputation)Macrium Representative (515 reputation)Macrium Representative (515 reputation)Macrium Representative (515 reputation)Macrium Representative (515 reputation)Macrium Representative (515 reputation)
Group: Moderators
Posts: 275, Visits: 941
Hi all,

While we're not planning anything on the deduplication front in the short term, it is a hot topic of conversation in our dev team and it's very much on our radar. There are some good points here around simplicity of backup vs. functionality, after all, in a DR scenario a complex deduplication system is just one more thing that can go wrong. When we do address deduplication, it will always be an opt-in feature rather than a required part of the product, we absolutely don't want to take away from Reflect's position as a simple, direct solution for taking backups.

Regards,
Alex Stevenson
Macrium Software


Kind Regards,

Alex

Macrium Development

Next Webinar

See our reviews on

Trustpilot Logo
Trustpilot Stars


rrnworks
rrnworks
New Member
New Member (1 reputation)New Member (1 reputation)New Member (1 reputation)New Member (1 reputation)New Member (1 reputation)New Member (1 reputation)New Member (1 reputation)New Member (1 reputation)New Member (1 reputation)New Member (1 reputation)
Group: Forum Members
Posts: 1, Visits: 5
We would be interested in this as well.  Or effective workarounds?  Perhaps an underlying storage system that supports deduplication such as:
https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/understand
and
https://blog.paessler.com/use-windows-refs-with-veeam-and-prtg-for-backups
I believe Macrium's incremental forever works the same way as veeam.



Edited 15 March 2020 6:59 PM by rrnworks
GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Reading This Topic

Login

Explore
Messages
Mentions
Search