Rsync Folders

[TomBenz]

@Acidham is it possible to provide more information at the start or end of sync activities. Examples Number of new Files or data copied please.

Edited January 23, 2023 by TomBenz

[Acidham]

@TomBenzI released a new version with an option to show rsync log in a text file after completion. let me know if this fits your request?

 

https://github.com/Acidham/rsync-folders/releases/tag/v1.6.0

[TomBenz]

@Acidham thanks. I tried this. The version v1.6 has not appeared in Alfred Gallery so I did manual update. The files got copied, but I don't see rsync_log.txt. Following is the message in the debug log:

 

ERROR: RSync Folders[Run Script] Traceback (most recent call last):
  File "/Users/xyz/Downloads/Alfred.alfredpreferences/workflows/user.workflow.73A1DA96-BA16-4EBE-B619-FD5C60A972A3/exec.py", line 58, in <module>
    mg = int(match.group(1))
AttributeError: 'NoneType' object has no attribute 'group'

 

Few questions:

1. Will the log file be shown in debug log or in the notification pane? 

2. What is the maximum size that you have used Rsync? Can it be used for incremental backup of 50-60 GB (with 1-2 GB of new files every day) from one drive to iCloud? Is  there a better option for such kind of use case?

3. Can Source folder name be used to create sub-folder in target location? I had expected this but lost some other files in target location. Now I create sub-folder at target first but it will be easier if it can create sub-folder or ask user about it.

 

Thanks for this great workflow and your time on it.

[Acidham]

Alfred Gallery will update with some delay. 

I found this bug and just fixed it. Please download it again.

 

re questions:

1. will be opened as txt file in default txt editor

2. I don't know ... test it 

3. I don't understand the use case. Sub-Folders and files below will be in sync as well. In general, the WF just keep both folder in sync including deletions, that's it. 

[TomBenz]

7 hours ago, Acidham said:

Alfred Gallery will update with some delay. 

I found this bug and just fixed it. Please download it again.

 

re questions:

1. will be opened as txt file in default txt editor

2. I don't know ... test it 

3. I don't understand the use case. Sub-Folders and files below will be in sync as well. In general, the WF just keep both folder in sync including deletions, that's it. 

 

1. Alfred gallery still has v1.5.0. Re-tested and it words great. Got text file with this very useful info. 

 

Number of files: 9 (reg: 8, dir: 1)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 191,807 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 504
Total bytes received: 18

sent 504 bytes  received 18 bytes  1,044.00 bytes/sec
total size is 191,807  speedup is 367.45

 

 

3. Yes, it create sub-folders. Request for creating "same folder" as source folder at target location first and then copy all files within it. I had given a generic target folder such as download initially and it removed everything else at this location. I am now creating Target folder first properly and then using it. 

[vitor]

1 hour ago, TomBenz said:

Alfred gallery still has v1.5.0.

 

As @Acidham mentioned, it’s not updated there immediately. This is by design. Among other things, it allows for this short feedback loop where a workflow creator can add a requested feature, it’s tested and kinks are ironed out, then it’s released. This way we avoid more people getting the intermediary version and can serve the known good one. It should be live by tomorrow. You don’t have to worry about it, the Gallery understands it’s the same workflow and will offer updates even if you downloaded it from GitHub.

[Acidham]

@vitor 

 

Among other things, it allows for this short feedback loop where a workflow creator can add a requested feature, it’s tested and kinks are ironed out, then it’s released.

+100 …the delay makes total sense!

 

@TomBenz 

 

 Request for creating “same folder” as source folder at target location first and then copy all files within it. I had given a generic target folder such as download initially and it removed everything else at this location. I am now creating Target folder first properly and then using it. 

 

I got it now and makes sense. I will add this to the backlog!

[TomBenz]

1 hour ago, Acidham said:

@vitor 

 

 

 

+100 …the delay makes total sense!

 

@TomBenz 

 

 

 

 

I got it now and makes sense. I will add this to the backlog!

Thanks @Acidham and @vitor

 

One more request:

Is it possible to add "Update" option where source path can be reviewed and updated? If the source folder is renamed, one has to delete and add new configuration to set it up again. Is the source and target getting saved in some xml or plist file where it can be edited? Can these details appear in environment variables to easily update (as done in I Sheet You Not workflow by Dean)?

[Acidham]

I added the ability to let the WF create src, target folders when not available. 

https://github.com/Acidham/rsync-folders/releases/tag/v1.6.5

 

Is it possible to add “Update” option where source path can be reviewed and updated? If the source folder is renamed, one has to delete and add new configuration to set it up again. Is the source and target getting saved in some xml or plist file where it can be edited? Can these details appear in environment variables to easily update (as done in I Sheet You Not workflow by Dean)?

 

I am not planning to implement that..too much work given the value getting out of it. Re-Creating the rsync set is not a big deal in case folders were deleted. 

[TomBenz]

19 minutes ago, Acidham said:

I added the ability to let the WF create src, target folders when not available. 

https://github.com/Acidham/rsync-folders/releases/tag/v1.6.5

 

 

 

 

I am not planning to implement that..too much work given the value getting out of it. Re-Creating the rsync set is not a big deal in case folders were deleted. 

 For future backlogs if any  thanks. If you can explain where these are stored, I can edit and address these

[TomBenz]

33 minutes ago, Acidham said:

I added the ability to let the WF create src, target folders when not available. 

https://github.com/Acidham/rsync-folders/releases/tag/v1.6.5

 

 

 

 

I am not planning to implement that..too much work given the value getting out of it. Re-Creating the rsync set is not a big deal in case folders were deleted. 

 

It is done differently and I will have to figure out how to use it.

 

1. If src folder is \ABC

and target folder is defined as \rsync backup via search (the current configurations requires user to define full path for target?) then it creates \rsync backup\ABC folder and then copies files

 

2. If src folder is \ABC and target folder is defined as \rsync backup\ABC i.e folder name matches then it copies the files directly

 

Anyway, I can live with workaround if this is getting complex and specific requirement.

 

The log file is a great addition. thanks a lot for it.

 

regards,

 

 

 

[Acidham]

It is only available in “Add” not Search. Search require existing folders by nature

 

You can add ~/ABC or absolute path.

 

The config.json is stored in Workflow Data directory under Application Support/Alfred 

[TomBenz]

On 1/25/2023 at 6:51 PM, Acidham said:

It is only available in “Add” not Search. Search require existing folders by nature

 

You can add ~/ABC or absolute path.

 

The config.json is stored in Workflow Data directory under Application Support/Alfred 

@Acidham I have used this workflow to backup my Onedrive content on iCloud, and it worked great. Many thanks.

 

iCloud doesn't allow folder named "OneDrive" so that I had to change it to "OD" for uploading. Now I want to change destination in rsync for incremental backup but don't want to add again and do full backup. How can I change destination folder from "OneDrive" to "OD" safely in rsync config or Json file?

[Acidham]

> Now I want to change destination in rsync for incremental backup but don't want to add again and do full backup

 

you can change in config json but you can also delete and add again. Rsync does not perform a full backup it just sync, means it starts where you ended before you deleted the entry.

[TomBenz]

13 hours ago, Acidham said:

> Now I want to change destination in rsync for incremental backup but don't want to add again and do full backup

 

you can change in config json but you can also delete and add again. Rsync does not perform a full backup it just sync, means it starts where you ended before you deleted the entry.

Thanks @Acidham as it is significant data that I am backing, I would like to re-confirm that it is doing one-way mirroring from source to destination and not doing any two-way sync. I will delete "OneDrive" config and create a new one while you using the earlier created destination folder "OD".

 

[Acidham]

Here's a rundown of what each option in the rsync command does:

-r: This stands for "recursive," which means that rsync will copy all subdirectories and their contents, not just the files in the top-level directory.

-t: This option preserves the modification times of files.

-v: "Verbose." This makes rsync provide more detailed output about what it's doing.

-u: This tells rsync to only copy files that have a more recent modification time on the source than on the target. In other words, if the target file is newer or the same age as the source file, it won't be overwritten.

-c: Instead of just looking at file modification times, this option makes rsync check the contents of files to determine whether they need to be copied. It uses a checksum for this.

--delete-after: This option means that after rsync has copied all the necessary files, it will delete any files in the target directory that aren't in the source directory. It ensures that the target becomes a mirror image of the source.

--ignore-errors: This tells rsync to continue transferring the rest of the files even if some files give errors.

--stats: This option makes rsync provide a summary of how much data was transferred and other related stats.

Edited October 20, 2023 by Acidham
explain rsync for refernce

[TomBenz]

@Acidham I love this workflow and use it for backup. Request for some info on it:

 

1. rsync_log lists files getting added or deleted during the sync process. How are these identified? I sometimes get files that have stayed the same during the incremental backups. It will be good to understand this better

2. Is rsync_log storing the last sync time somewhere or does it create a complete rsync list by comparing source and destination fully? How much time should this step typically take for 50/100 GB of data?

3. I am using rsync to do a backup of OneDrive to iCloud. OneDrive files are not always on disk (even after marking it to download all) so build the list takes a long time. Is it the reason for the long time to generate file list? Is anybody using using OneDrive and facing challenges in keeping all files local?

 

 

[TomBenz]

On 9/18/2022 at 10:25 PM, Acidham said:

Should be available in v1.4.

 

Select Search&Add

 

 

@Acidham is it possible to add a destination folder by navigating to it? I have excluded this location from indexing so search and add don't work. I would like to use universal action to select the source and then navigate to the destination on a non-indexed location on iCloud to add as the destination.