Compression Algorithms Used in Macrium Reflect


Author
Message
jamespedersen
jamespedersen
New Member
New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)
Group: Forum Members
Posts: 7, Visits: 31
Hi,

What specific compression algorithms are used when creating medium compression and high compression disk images?

Thank you,
James Pedersen

Dan Danz
Dan Danz
Macrium Hero
Macrium Hero (2.7K reputation)Macrium Hero (2.7K reputation)Macrium Hero (2.7K reputation)Macrium Hero (2.7K reputation)Macrium Hero (2.7K reputation)Macrium Hero (2.7K reputation)Macrium Hero (2.7K reputation)Macrium Hero (2.7K reputation)Macrium Hero (2.7K reputation)Macrium Hero (2.7K reputation)
Group: Forum Members
Posts: 1.2K, Visits: 9.5K
The Reflect Knowledge Base has this to say about compression in a discussion of advanced backup and default settings:
https://knowledgebase.macrium.com/display/KNOW80/Backup+Defaults.
Corrected link or direct to article about Compression
Backup files can be reduced in size without compromising data integrity. Compression results may vary depending on compressibility of the source data, e.g., a standard Windows install OS partition will compress to around 60-70% of its original size. The compression effectiveness for user data partitions and files will depend on the types of files being backed up. Files that won't compress further include most audio and video files, as well as existing compressed file such as .zip, .tar, .7z etc.

  • None  Backup without compressing.
  • Medium (Recommended)  Medium compression generally provides the best compromise for performance and file size.
  • High  High compression may make backups take considerably longer to complete but the difference in file size may be marginal.
Macrium Reflect uses a very fast, real-time, *streaming block compression algorithm. This will not provide the same overall compression ratio as common compression utilities such a 7-Zip which use, much slower, whole file data compression techniques.

*All 'mountable' backup files, such as those created by Macrium Reflect, require discrete blocks of data to be compressed and decompressed 'on the fly'. This enables images and backup files to be incremented and mounted as drives in Windows Explorer,

L.W. (Dan) Danz, Overland Park KS
Reflect v8.1.7847+ on Windows 11 Home 23H2 22631.3085+ | Reflect v8.1.7853+ on Windows 10 Pro 22H2 19045.3996+
Reflect v8.1.7784+ on 2 systems Windows 10 Home 22H2 19045.3803+

Edited 24 August 2023 7:17 PM by Dan Danz
Danskeman
Danskeman
Expert
Expert (750 reputation)Expert (750 reputation)Expert (750 reputation)Expert (750 reputation)Expert (750 reputation)Expert (750 reputation)Expert (750 reputation)Expert (750 reputation)Expert (750 reputation)Expert (750 reputation)
Group: Forum Members
Posts: 526, Visits: 4.5K
Dan Danz - 10 August 2023 12:35 PM
The Reflect Knowledge Base has this to say about compression in a discussion of advanced backup and default settings:
https://knowledgebase.macrium.com/display/KNOW80/Backup+Defaults.

Backup files can be reduced in size without compromising data integrity. Compression results may vary depending on compressibility of the source data, e.g., a standard Windows install OS partition will compress to around 60-70% of its original size. The compression effectiveness for user data partitions and files will depend on the types of files being backed up. Files that won't compress further include most audio and video files, as well as existing compressed file such as .zip, .tar, .7z etc.

  • None  Backup without compressing.
  • Medium (Recommended)  Medium compression generally provides the best compromise for performance and file size.
  • High  High compression may make backups take considerably longer to complete but the difference in file size may be marginal.
Macrium Reflect uses a very fast, real-time, *streaming block compression algorithm. This will not provide the same overall compression ratio as common compression utilities such a 7-Zip which use, much slower, whole file data compression techniques.

*All 'mountable' backup files, such as those created by Macrium Reflect, require discrete blocks of data to be compressed and decompressed 'on the fly'. This enables images and backup files to be incremented and mounted as drives in Windows Explorer,

I have done tests on high versus medium compression a number of times, and I always find high compression barely has any impact over medium and it takes a lot longer.

Contrast that with no compression which goes like muck off a shovel for my nvme to nvme backup, but image is nearly twice the size. 

In the end, the time saving is easily offset by extra storage.

I totally agree with the recommendation that medium provides a good balance between speed of backup versus image size.

Of course, peoples mileage will vary depending on cpu, drive type, Ram etc.  Having said that, I have tried this on various pc configurations and results are always the same.
jamespedersen
jamespedersen
New Member
New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)
Group: Forum Members
Posts: 7, Visits: 31
@DanDanz Thank you for your reply, although I will note that the URL that you posted is a broken link.

Dan Danz
Dan Danz
Macrium Hero
Macrium Hero (2.7K reputation)Macrium Hero (2.7K reputation)Macrium Hero (2.7K reputation)Macrium Hero (2.7K reputation)Macrium Hero (2.7K reputation)Macrium Hero (2.7K reputation)Macrium Hero (2.7K reputation)Macrium Hero (2.7K reputation)Macrium Hero (2.7K reputation)Macrium Hero (2.7K reputation)
Group: Forum Members
Posts: 1.2K, Visits: 9.5K
jamespedersen - 24 August 2023 6:40 PM
@DanDanz Thank you for your reply, although I will note that the URL that you posted is a broken link.

Sorry for the bad link  -- I fixed the original post.    If you still get a broken link, go to the Macrium Knowledge Base and User Guide and search for Backup Defaults, and then the Heading "Compression".


L.W. (Dan) Danz, Overland Park KS
Reflect v8.1.7847+ on Windows 11 Home 23H2 22631.3085+ | Reflect v8.1.7853+ on Windows 10 Pro 22H2 19045.3996+
Reflect v8.1.7784+ on 2 systems Windows 10 Home 22H2 19045.3803+

Dan Danz
Dan Danz
Macrium Hero
Macrium Hero (2.7K reputation)Macrium Hero (2.7K reputation)Macrium Hero (2.7K reputation)Macrium Hero (2.7K reputation)Macrium Hero (2.7K reputation)Macrium Hero (2.7K reputation)Macrium Hero (2.7K reputation)Macrium Hero (2.7K reputation)Macrium Hero (2.7K reputation)Macrium Hero (2.7K reputation)
Group: Forum Members
Posts: 1.2K, Visits: 9.5K
Danskeman - 24 August 2023 6:17 PM
Dan Danz - 10 August 2023 12:35 PM
The Reflect Knowledge Base has this to say about compression in a discussion of advanced backup and default settings:
https://knowledgebase.macrium.com/display/KNOW80/Backup+Defaults.

Backup files can be reduced in size without compromising data integrity. Compression results may vary depending on compressibility of the source data, e.g., a standard Windows install OS partition will compress to around 60-70% of its original size. The compression effectiveness for user data partitions and files will depend on the types of files being backed up. Files that won't compress further include most audio and video files, as well as existing compressed file such as .zip, .tar, .7z etc.

  • None  Backup without compressing.
  • Medium (Recommended)  Medium compression generally provides the best compromise for performance and file size.
  • High  High compression may make backups take considerably longer to complete but the difference in file size may be marginal.
Macrium Reflect uses a very fast, real-time, *streaming block compression algorithm. This will not provide the same overall compression ratio as common compression utilities such a 7-Zip which use, much slower, whole file data compression techniques.

*All 'mountable' backup files, such as those created by Macrium Reflect, require discrete blocks of data to be compressed and decompressed 'on the fly'. This enables images and backup files to be incremented and mounted as drives in Windows Explorer,

I have done tests on high versus medium compression a number of times, and I always find high compression barely has any impact over medium and it takes a lot longer.

Contrast that with no compression which goes like muck off a shovel for my nvme to nvme backup, but image is nearly twice the size. 

In the end, the time saving is easily offset by extra storage.

I totally agree with the recommendation that medium provides a good balance between speed of backup versus image size.

Of course, peoples mileage will vary depending on cpu, drive type, Ram etc.  Having said that, I have tried this on various pc configurations and results are always the same.
Thanks.  Earlier this year, I had noticed that my repository disk was beginning to fill up so I switched to high compression.   But after delving into the subject again (partly because of this topic on the forum), I decided to revert all 4 of my systems to use medium compression.  I reached the same conclusion:  the savings in space for high compression versus medium compression is not significantly different,  and is offset by increased backup time. 

I did discover something that, at first, was puzzling.  I switched the compression setting from high to medium in the BDF for an existing backup set, and ran a new differential right after the exisitng (high) differential.  The new one ran another HIGH compression differential according to the log.  Even the {COMPRESSION} replacement in the email showed High.    After a while, I realized that Reflect cannot change Compression in the middle of a backup set.  So, forcing a new full at Medium Compression, followed by a medium compression differential showed both had Medium Compression.   @JoeA A suggestion: add this information to the Knowledge Base article about Backup Defaults-Compression.

L.W. (Dan) Danz, Overland Park KS
Reflect v8.1.7847+ on Windows 11 Home 23H2 22631.3085+ | Reflect v8.1.7853+ on Windows 10 Pro 22H2 19045.3996+
Reflect v8.1.7784+ on 2 systems Windows 10 Home 22H2 19045.3803+

JK
JK
Macrium Hero
Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)
Group: Forum Members
Posts: 1.5K, Visits: 6K
@DanDanz I had my own experience getting confused by the behavior of compression settings in an old thread, in which I also offered some suggestions for making things less confusing.

                                
jamespedersen
jamespedersen
New Member
New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)
Group: Forum Members
Posts: 7, Visits: 31
Perhaps someone from Macrium support would be so kind as to share the exact compression algorithms used to compress the blocks of data for medium-compression and high-compression images? So that everyone can understand exactly how the data in their backups is being compressed?

Thanks in advance,
James Pedersen

Edited 30 August 2023 6:00 AM by jamespedersen
JamieW
JamieW
Macrium Representative
Macrium Representative (600 reputation)Macrium Representative (600 reputation)Macrium Representative (600 reputation)Macrium Representative (600 reputation)Macrium Representative (600 reputation)Macrium Representative (600 reputation)Macrium Representative (600 reputation)Macrium Representative (600 reputation)Macrium Representative (600 reputation)Macrium Representative (600 reputation)
Group: Macrium Moderators
Posts: 367, Visits: 7.2K
jamespedersen - 30 August 2023 5:59 AM
Perhaps someone from Macrium support would be so kind as to share the exact compression algorithms used to compress the blocks of data for medium-compression and high-compression images? So that everyone can understand exactly how the data in their backups is being compressed?

Thanks in advance,
James Pedersen

Hi @jamespedersen,

We use a real-time, streaming-block, data-compression algorithm that is loosely based on the Lempel-Ziv (LZ) algorithm.

More in-depth, technical information on the LZ algorithm can be found here: https://en.wikipedia.org/wiki/LZ77_and_LZ78

Kind Regards

Jamie

Macrium Development

Next Webinar

See our reviews on

Trustpilot Logo
Trustpilot Stars


dbminter
dbminter
Macrium Evangelist
Macrium Evangelist (7.7K reputation)Macrium Evangelist (7.7K reputation)Macrium Evangelist (7.7K reputation)Macrium Evangelist (7.7K reputation)Macrium Evangelist (7.7K reputation)Macrium Evangelist (7.7K reputation)Macrium Evangelist (7.7K reputation)Macrium Evangelist (7.7K reputation)Macrium Evangelist (7.7K reputation)Macrium Evangelist (7.7K reputation)
Group: Forum Members
Posts: 4.8K, Visits: 51K
@JamieW


So, Reflect uses a proprietary compression algorithm?  It's based on an existing one, but it is still proprietary?  Which, IMO, is a good thing.  If it's not open source, it is more "secure, as far as I'm concerned.


Thanks!
GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Reading This Topic

Login

Explore
Messages
Mentions
Search