Suggestion for Macrium Reflect.


Author
Message
Klaatu.barada.nikto
Klaatu.barada.nikto
New Member
New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)
Group: Forum Members
Posts: 23, Visits: 97
One thing that I came across while using Reflect was NTFS Compressed files are backed up as uncompressed files i.e. as per normally stored files in NTFS.

I had a drive that had 250GB free because the files had been stored compressed on the volume.
When the files were restored the files filled the volume and I had virtually no space left.
As Macrium Reflect could know the files are compressed from their attributes it would be useful to as least flag up the files 'had been compressed before backup' when you try to restore.
(To give you a chance to set compression on the NTFS volume, at least.)
More useful would be to set the compression attribute on the files when they are restored.

Don't know if this is a problem to do ..... Just an idea that might be useful.

P.S.
I know Disks are cheap but why not have both Space and Compression, if performance is not a problem.
Personally, NTFS Compression has worked for many years when needing more space without immediate access to more physical disks.
Even with s/w that supposedly will not work !!!
You need to ensure you are not applying NTFS Compression after the fact when some s/w will find 'critical' files in a format it cannot handle.
Often performing the installation/configuration on a NTFS Compressed volume works as the s/w can set the correct attributes on the files it needs to keep uncompressed etc.

Even if not very popular, if NTFS still supports it, should it not be handled better for the 'few' that do use it. !!??

jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)
Group: Forum Members
Posts: 14K, Visits: 84K
Since you ended up making your own thread, I figured I'd move my reply here as well.  I assume this pertains to using the Restore wizard for File & Folder backups?  When restoring an image backup, compression would be preserved, and when browsing either type of image and copying data from the mounted virtual disk, Reflect wouldn't have control over how this works because you'd be using Windows Explorer to copy the files. In that case, the standard Windows behavior would apply, which is that files being copied into a folder inherit the compression status of the parent folder, regardless of the compression status of the source copy of the files.

But even when dealing with using the Restore wizard for File & Folder backups, this might not be as easy to implement as it sounds. Below are some things Reflect would probably have to implement in order to do this properly and not create undesired results for users who have other use cases, and these are just off the top of my head:

- For every backup, record the compression attribute for each file/folder
- For every restore, recurse through all files/folders selected for restoration to check their original compression status in order to determine whether any special action is appropriate.
- If any files being restored had compression enabled and the destination doesn't, ask the user what to do: maintain source compression status, compress nothing, compress all files being restored
- If NO files being restored had compression enabled and the destination DOES, ask the user what to do: compress all files being restored, do not compress files being restored
- Add an Edit Defaults option to specify what to do in the above cases so the user can avoid being prompted each time these situations occur.
- If any files being restored had compression enabled and are being restored to a non-NTFS partition, warn the user.

That seems like a lot of work just to avoid the far simpler solution of the user enabling NTFS compression on the target disk/folder before starting the restore if they want the restored files to be compressed.  After all, if they were copying/restoring files just using regular Windows Explorer rather than Reflect, they would have to do that anyway, so I'm not sure Reflect needs to behave differently from what Windows would do itself in this particular case.  That solution just as flexible with a LOT less effort compared to this proposed Reflect feature, so I personally don't think it makes sense for Macrium to put a bunch of engineering effort into this just to avoid users having to do something that's already easy.

Lastly, consider what the worst case scenario is based on how things work today.  If the user starts a restore and it ends up failing because the destination ran out of space, then they just delete the restored data, enable compression at the destination, and start it again.  That's not such a big deal.  Or if the restore completes but they have less free space than expected, they can either do what I just said or simply enable compression in-place on the restored data.  Again, not such a big deal.  It just takes a bit more time.  So once again, this seems like a lot of effort to avoid pretty minimal downsides of the current process, which again users would have to go through anyway if they were using regular Windows for these file copy operations.

Edited 9 September 2017 11:23 PM by jphughan
Klaatu.barada.nikto
Klaatu.barada.nikto
New Member
New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)
Group: Forum Members
Posts: 23, Visits: 97
[quote]
jphughan - 9 September 2017 11:11 PM
- Recurse through all files/folders selected for restoration to check their original compression status in order to determine whether any special action is appropriate.
- If any files being restored had compression enabled and the destination doesn't, ask the user what to do: maintain source compression status, compress nothing, compress all files being restored
- If NO files being restored had compression enabled and the destination DOES, ask the user what to do: compress all files being restored, do not compress files being restored
- Add an Edit Defaults option to specify what to do in the above cases so the user can avoid being prompted each time these situations occur.
- If any files being restored had compression enabled and are being restored to a non-NTFS partition, warn the user.

You are making it too complicated.
1. If the destination has compression enabled or not is not Macrium Reflects problem. Macrium did not care that the files were compressed when they were backed up.
   It is a courtesy to advise the user that they are doing something that they may not want to do.
2. When the files are backed up simply set a flag to state to Macrium that this backup/backup set contains NTFS compressed files. (The attributes are already on the files in NTFS)
3. When you start the restore process the Directory Structure is read before displaying the following dialog box (or whatever it is called):
    https://forum.macrium.com/uploads/images/39d026b1-7fd6-497c-a0be-e10f.png
    This is the point where Macrium Reflect can display whether the file/folder was compressed as you navigate through the structure selecting your Folders/Files.
    There is room on the righthand side to display the attributes to advise the Folder/File is NTFS Compressed.
    If you select something that was compressed you can display a warning that the Folder/file will 'only' be compressed on a NTFS volume.
    When the 'next' button is pressed you select your destination:
    https://forum.macrium.com/uploads/images/eb426a10-3959-4c61-998e-b0a6.png
    If the previous screen has selected a Compressed Folder/File you check if the selected destination is a NTFS volume.
    If not you flag up that Compressed files cannot be restored in a compressed format to this volume and to cancel if this is NOT what you want.
    For completeness you could add a tickbox to allow NTFS Compression to be set on/off which would be grayed out if the selected destination is not a NTFS volume.
 
Does this not do what I was asking for with less complication ?


 
 
 
 


jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)
Group: Forum Members
Posts: 14K, Visits: 84K
Are you suggesting that Reflect always retain the source file's compression status, i.e. restore it compressed if it was originally compressed and restore it uncompressed if it was originally uncompressed, and not offer any way to override that?  If so, yes your suggested feature does what YOU seem to want, and it's less complicated, but that's only because your reduced complexity is coming at the cost of flexibility and introducing a new risk of behavior that could just as easily be UNdesirable for others AND not offer them any recourse. Consider:

- Suppose a user had NTFS compression enabled at the original source because they were low on space, but they're now restoring to a larger partition, so they specifically do NOT want NTFS compression to be reinstated. Today, they could simply make sure NTFS compression isn't enabled on their destination and restore files as normal, and they get what they want.  Under your proposed implementation, Reflect would seemingly force compression to be enabled, requiring the user to manually disable it after the restore. This is inconvenient and it would have Reflect overriding standard Windows behavior that the user would have rightfully expected to have applied in this scenario.
- Suppose a user did NOT have NTFS compression enabled originally, but they're now trying to restore their files onto a SMALLER partition, so they specifically DO want compression enabled.  Today, they'd just need to enable NTFS compression upfront on the target drive/folder and then proceed with the restore.  Under your proposed implementation, Reflect would ignore that setting on the destination and restore the files uncompressed. In this case, the best case scenario is that the user would have to manually enable compression after the restore. Worst case, the restore wouldn't even be able to complete due to insufficient capacity, so the user would have to break the restore into multiple parts, enabling compression on the newly restored data after each part.

This is why I said that Reflect would need to offer options for dealing with all of this.  That's not to say it couldn't be done, but again, enabling/disabling NTFS compression at the target beforehand is very simple and is something that users would have to do anyway if they were working with files in Windows Explorer -- and even if they DON'T do that beforehand, the worst case scenario is they have to either wait for files to compress/decompress after they're restored, or maybe delete the files and run the restore again.  Those both seem like fairly insignificant amounts of effort compared to implementing this feature with an appropriate level of flexibility, especially after taking into account the code required to capture compression status, recurse through it, check restore conditions to determine when warnings are appropriate, etc.

When you say, "If the destination has compression enabled or not is not Macrium Reflects problem", that's technically true, but Reflect should absolutely respect the destination's compression setting, just as any other Windows application would. Your implementation would not necessarily do that.

You also can't just use a flag for the entire backup that says, "This backup contains compressed files".  NTFS compression can be enabled on a per-file basis, and maybe a user only deliberately only compressed certain data while leaving other data uncompressed to avoid the performance penalty on that data -- so it's entirely possible that a backup would include both compressed and uncompressed data in its scope. In order to handle that appropriately, Reflect would need to record the compression status of each individual file when performing a backup and check the status of each file selected for a restore.

Edited 10 September 2017 1:18 AM by jphughan
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)
Group: Forum Members
Posts: 14K, Visits: 84K
Here is perhaps a clearer, more concise summary of my perspective on this feature:

- Today's way to avoid this entire problem is for the user to set their desired compression state on the destination before copying data there. That is flexible (users won't necessarily want to maintain the compression state they originally chose), broadly applicable (this design applies to file operations outside of Reflect, which means users are more likely to be familiar with it), very simple, and already exists today without any effort from Macrium. Today's solution has a compelling list of benefits.

- If the user forgets to set their desired compression state beforehand, they can either delete the restored data and run the restore again, or change the compression state in-place. Both take extra time, but that's all.  So the best case scenario for this proposed feature is eliminating the need for a fix that is only a trivial amount of inconvenience and effort.

- Even the simplest implementation of this proposed feature, i.e. with minimal flexibility, would involve some engineering effort from Macrium in terms of changing the data recorded during the backup, checking files selected for restore, checking the target file system, determining whether to warn the user based on all of that information, adjusting the UI, and writing various warning messages based on various circumstances. And even after all of that, this simple version would be less flexible than today's solution. Yes, it may give users a "courtesy" notification, but the decreased flexibility and deviation from standard Windows behavior could also create confusion and larger problems for other users. Engineering effort to build a solution that could create more frequent and/or larger problems than it solves is not a good cost/benefit proposition.

- Adding more flexibility for this proposed feature to display the "courtesy" notification without creating problems for other people's scenarios would require even more engineering effort. But as I said above, the best case scenario here is avoiding the need to perform a fix that is already trivial. That is not a good cost/benefit proposition either.

- Only a tiny portion of users even enable NTFS compression to begin with, and this proposed feature would only apply to the subset of those users who forgot to set their desired compression state prior to performing a restore. This means that even in an ideal implementation of this feature, the scope of the benefit will be very limited, which (yet again) is not a good cost/benefit proposition.

Edited 10 September 2017 1:41 AM by jphughan
Klaatu.barada.nikto
Klaatu.barada.nikto
New Member
New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)
Group: Forum Members
Posts: 23, Visits: 97
Sorry for the delay, checking my facts as I have been out of the Programming game for a long long time. My memory may be faulty after all this time. Smile
If you re-read my suggestion it does allow a choice of whether NTFS Folders/Files are compressed when restored.
It is why I said ' For completeness you could add a tickbox to allow NTFS Compression to be set on/off which would be grayed out if the selected destination is not a NTFS volume.'
By simply setting the tickbox to 'off' all files are restored without NTFS Compression being set which should even allow you to override NTFS Compression set on a volume. (If I remember correctly the setting are Defaults unless 'overridden')

jphughan - 10 September 2017 12:13 AM
And you can't just set a flag for the entire backup saying "This backup contains compressed files".  That would only be appropriate if compression were enabled globally, but compression can be enabled down to a per-file basis, and maybe a user only wanted to compress certain data while leaving other data deliberately uncompressed to avoid the performance penalty on that data -- so it's entirely possible that a backup would have included both compressed and uncompressed data in its scope. In order to handle that appropriately, Reflect would need to record the compression status of each individual file.

The flag is simply to allow Reflect to display a message to say there are NTFS Compressed files in this Backup/Backup set and to enable/switch-on the extra functionality in the Dialog box when the Backup is opened for restoration.
The file format does not need to change and surely there is some 'slack' built into the Header Information to allow change/future expansion of the functionality of Reflect, for a flag of only 1 bit !!!???.
Each Volume/Folder/File already has the extended attributes in NTFS to say whether it is NTFS compressed or not.
If all original attributes/extended attributes are saved then this is already there as of when you read in the Directory Structure in the current s/w.

[Worse case additional 'code']
You perform 2 different function calls to get the Compressed & Uncompressed File size (on the original Folders/Files when reading in the data for the Backup), if not equal the Folder/File is compressed.
(The call to get the Compressed size will return the uncompressed/actual size if not 'NTFS compressed' so this will work!!!)]

See attached sample/example .pdf's from MSDN .... I needed to confirm memory. Smile

All the information is in the NTFS Filesystem to be queried, if Reflect saves all the information it is there already in the present saved backup files as part of the Directory information.

I am detecting a real dislike for this idea Smile , fine if the implementation is problematic/costly or it breaks something else ..... but the information is there at little extra effort/cost as far as I can see. (Dependant on the 'if' above.)

At this point dismiss it if you want, as is your right, I am fine with that.
I am yet to be convinced it is impossible of its self but I am not going to push the case any more if you are so convinced it cannot be done.


IMHO:
The functionality does not impede or break anything currently done in the software as far as I can see.
No extra scanning of Folders/Files is needed in its own right as the Folders/Files are scanned when the Backup is performed and it requires taking notice of some of the information/data that is already there when reading the Folders/Files Data to perform the backup.

It is now late so I will pack up for now.
It was an idea, thats all.
As I am not rewarded on it happening or not, it is of little note at the end of the day, a little entertainment engaging the brain for a bit that's all, when I cannot sleep. [Insomniac for my sins.]. Smile

Thanks for your time and attention.

Attachments
Paddy
Paddy
New Member
New Member (45 reputation)New Member (45 reputation)New Member (45 reputation)New Member (45 reputation)New Member (45 reputation)New Member (45 reputation)New Member (45 reputation)New Member (45 reputation)New Member (45 reputation)New Member (45 reputation)
Group: Forum Members
Posts: 28, Visits: 88
Please add the ability to export virtual machines to Hyper-V and VMware formats from within viBoot.

Edited 19 November 2021 4:22 AM by Paddy
GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Reading This Topic

Login

Explore
Messages
Mentions
Search