Script to daily FTP a backup to a server - possible?


Author
Message
Alan UK
Alan UK
New Member
New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)
Group: Forum Members
Posts: 20, Visits: 68
MR v8 Win10Prof
I have a MR definition that runs daily with a full image backup on Mondays and incremental on other days. Backups are kept for 32 days and stored on a local drive. I would like to automatically FTP these files to a Linux server that has proftpd installed already.

I see I can get MR to create a MSDos script that I can schedule. I could add commands to call MSDOS ftp command. Example here:
https://stackoverflow.com/questions/4871018/batch-file-to-copy-text-file-to-ftp-site
First issue is that I can not see whether MR passes the backup file name(s) to the script?

I have found a complex script that will FTP upload all files from the source (backup) directory that do not exist in the target (server) directory:
https://www.dostips.com/DtTipsFtpBatchScript.php  and section "FTP - Upload Only New Files - .........."
Issue here is that once there are more than the 32 files, then how to delete the older ones (7 files).

What worries me with the above is that any failure might go unnoticed.

Has anyone found a simpler solution?
PS It would be nice if MR could FTP and, as now, send any error notifications by email.

Alan, UK



jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)
Group: Forum Members
Posts: 14K, Visits: 85K
Don't use MS-DOS scripts.  I'd recommend PowerShell unless you have a specific need for VBScript, but either of those two will surface additional options in the script generation wizard, including the ability to specify an application that should run after the backup.  So if you can distill your FTP command down to a single command line with arguments, you can just put that into the wizard.  If not, that would probably still be a better foundation from which to build something custom.  For one thing, PowerShell allows you to make the execution of post-backup applications conditional on whether the backup actually succeeded.  And anything that can be called in Command Prompt can be called in PowerShell, possibly with some syntax tweaks.

Alternatively, if you can enable SMB on the Linux server (or Samba, as they call it), then you could potentially just use the Directory Synchronization option baked right into the script generation wizard.  The easiest option of all would be to configure the backups to run under a Windows account with credentials that both have admin privileges on the local system AND have access to this Samba share you would create, in which case you wouldn't even need to hard code any authentication info into the script.  The Directory Synchronization always causes the regular destination folder's Reflect backups to be mirrored to the synchronization target, which would include deleting files at the synchronization location that no longer exist at the regular backup destination.  I would imagine something similar could be achieved via FTP though if necessary.

But no, Reflect does not return the name of the newly generated backup when it exits, so you wouldn't be able to use that as an input to something else later in the script.  That's partially why folder mirroring as I described above can be useful, but if that isn't feasible, you could do something like checking the destination folder for an MRIMG file with a Creation Date within the last minute or something in order lock onto the correct file.

Edited 6 November 2022 5:06 PM by jphughan
Alan UK
Alan UK
New Member
New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)
Group: Forum Members
Posts: 20, Visits: 68
Many thanks for a very useful and prompt reply. I've never used PowerShell but who says I can't learn it!  I already have a samba share directory for development but these tend to be small files. I've found that Windows is very slow on transferring big files over the network and I've read that FTP is much faster. But I appreciate Samba/shared folders might be the easiest to implement. The full backups are 6GB.

I did many years ago buy a NAS but it was so slow I moved the disks to external USB enclosures and manually copy my images for secure backups, but there does tend to be a lag in doing the copies, hence trying to automate using an existing Pi 4 with SSD. I will try it and time it.

jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)
Group: Forum Members
Posts: 14K, Visits: 85K
I wouldn't consider 6 GB to be a large file by today's standards, but I haven't had trouble transferring large files over SMB -- though I don't typically perform such transfers to Linux systems.  Newer versions of Windows can even do things like SMB multichannel, where if there are multiple network interfaces that can reach the target (such as Ethernet and Wi-Fi), it will use both interfaces -- but that requires SMB 3.0 and may not be available on Linux Samba implementations.  I would at least make sure that your Linux box is configured to support SMBv2, though, since that added some fairly drastic improvements over SMBv1.  Somehow, despite SMBv2 itself being about 15 years old and SMBv1 being horribly slow and insecure by today's standards, SMBv1 still persists in some Linux distros and therefore NAS devices as the only SMB protocol version that's enabled by default.

Edited 6 November 2022 7:30 PM by jphughan
Alan UK
Alan UK
New Member
New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)
Group: Forum Members
Posts: 20, Visits: 68
It's Samba 4.9.5 Debian and according to Wikipedia v4.9 dates from September 13, 2018
I just done a test and 6GB took 1 minute which is fine and there were no permission problems. My old NAS was painfully slow but then the Pi 4 does very well given it's small size (they are hard to get now that they are so popular). The LAN goes from PC -> dumb switch -> router -> dumb switch -> Pi 4 (about 60ft/18m) but there is little other traffic.

Thanks again for your help.

PS Add: the 1 possible problem is that Windows occasionally loses its shares.

Edited 6 November 2022 8:24 PM by Alan UK
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)
Group: Forum Members
Posts: 14K, Visits: 85K
I don't know what SMB protocol versions are supported by default using that Samba release and Linux distro, but the version of the Samba package is not directly tied to the SMB versions that would be supported and/or enabled.  That's certainly new enough for SMBv2 support, but again even some relatively recent NAS boxes seem to only have SMBv1 enabled by default.  But if you want to pursue this SMB option further, I'd be happy to try to help get the Directory Synchronization option working either by creating a suitable user on your Linux environment to allow implicit credentials to be used or by customizing a PowerShell script to include and provide hardcoded credentials for the Linux target.  I'm not sure what you mean in terms of Windows losing its shares, but it sounds like in this scenario the share would exist on the Linux box.  Windows would only be connecting to it.  But having worked in various Windows IT roles over the last 15 years, mostly in the server space, I can't say I've found any systemic problems of Windows losing shares either in terms of shares it's hosting or shares it's connecting to.

Patrick O'Keefe
Patrick O'Keefe
Expert
Expert (879 reputation)Expert (879 reputation)Expert (879 reputation)Expert (879 reputation)Expert (879 reputation)Expert (879 reputation)Expert (879 reputation)Expert (879 reputation)Expert (879 reputation)Expert (879 reputation)
Group: Forum Members
Posts: 604, Visits: 4.1K
Alan UK - 6 November 2022 4:20 PM
Has anyone found a simpler solution?
PS It would be nice if MR could FTP and, as now, send any error notifications by email.
I found a solution to a similar need.  (In my case the SMB solution would not work for some odd reason. On 1 out of 4 computers the SMB connections would intermittently fail with credential errors.  Never figured out the problem.)  The solution was to take the backups to a local drive and have a Macrium PS script kick off SyncBackPro FTP "mirror" copy of the whole folder Reflect used as it's target.  A SyncBackPro 'mirror" will copy only newly added or changed files to the NAS and will delete newly deleted files.  In addition, you can use FTP, SFTP, or FTPS.  SyncBackPro casts, but it's not exorbitant.  (It's possible that SyncBackFree will do the job.)  

There ware some strange problems getting the PS scripts to provoke the SyncBackPro backup when the script ran under SYSTEM (as it would with a scheduled MR backup), but I managed to get it to work.  I could dig up the details if you want this solution.


Alan UK
Alan UK
New Member
New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)New Member (43 reputation)
Group: Forum Members
Posts: 20, Visits: 68
Thank you
@jphughan and @Patrick O'Keefe for offering practical help. Today my new MR Definition ran OK and did the backup so I will run with this. Hopefully your comments will help others as I saw some old posts asking for a backup using FTP function.


Alan

GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Reading This Topic

Login

Explore
Messages
Mentions
Search