Re: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsyncq
On Tue, 2023-09-05 at 20:01 -0400, Default User wrote:
> On Fri, 2023-09-01 at 22:44 +0200, Michel Verdier wrote:
> > > > > On 2023-09-01, Default User wrote:
> > > > >
> > > > > > > > > Yes, it does require considerable space (no data
> > > > > > > > > de-duplication),
> > > > > > > > > and
> > > > > > > > > the rsync of the backup drives does take considerable
> > > > > > > > > time.
> > > > > > > > > But to
> > > > > > > > > me,
> > > > > > > > > it is worth it, to avoid the methodological
> > > > > > > > > equivalent
> > > > > > > > > of
> > > > > > > > > "vendor
> > > > > > > > > lock-
> > > > > > > > > in".
> > > > >
> > > > > You must have a bad configuration : rsnaphot de-duplicate
> > > > > using
> > > > > hard
> > > > > links so you never have duplicated files. Keeping 52 weekly
> > > > > and
> > > > > 7
> > > > > daily
> > > > > and 24 hourly I need only 130% of original space. And it
> > > > > takes
> > > > > minimal
> > > > > time as it transfers only changes and can use ssh
> > > > > compression.
> > > > >
>
>
> Okay, first:
>
> I said that my "system" files (everything except /home/[user] was
> backed up using Timeshift. That is correct. It is done by Timeshift
> automatically, once a day, as well as weekly, monthly, and yearly.
>
> But I was wrong about rsnapshot. I said that it was set up to back up
> /home/[user] only. That is not correct. I now realize that I have it
> set up to back up all of /, except for:
> exclude /dev/*
> exclude /proc/*
> exclude /sys/*
> exclude /tmp/*
> exclude /run/*
> exclude /mnt/*
> exclude /media/*
> exclude /lost+found
> exclude /home/lost+found
> exclude /var/lost+found
>
> Now sudo du -sh / says that / seems to be using about 30 Gb. But sudo
> du -sh /media/user/rsnapshot_backups_of_host, says that the backup
> directory, /media/user/rsnapshot_backups_of_host on backup drive A,
> is
> using a whopping 88 Gb for 24 hourly, 7 daily, and 3 weekly!
>
> I am thinking, that CAN'T be right.
> Maybe each hard link is being counted as a full, actual file, when
> adding up the space allegedly used.
>
> So, how can I determine how much space is really being used for the
> backups?
>
>
>
>
> ---------------------------------------------------------------------
> --
>
>
> FWIW, here is my /etc/rsnapshot.conf file:
> #################################################
> # rsnapshot.conf - rsnapshot configuration file #
> #################################################
> # #
> # PLEASE BE AWARE OF THE FOLLOWING RULE: #
> # #
> # This file requires tabs between elements #
> # #
> #################################################
>
> #######################
> # CONFIG FILE VERSION #
> #######################
>
> config_version 1.2
>
> ###########################
> # SNAPSHOT ROOT DIRECTORY #
> ###########################
>
> # All snapshots will be stored under this root directory.
> #
> snapshot_root /media/user/MSD00001/rsnapshot_backups_of_host/
>
> # If no_create_root is enabled, rsnapshot will not automatically
> create
> the
> # snapshot_root directory. This is particularly useful if you are
> backing
> # up to removable media, such as a FireWire or USB drive.
> #
> no_create_root 1
>
> #################################
> # EXTERNAL PROGRAM DEPENDENCIES #
> #################################
>
> # LINUX USERS: Be sure to uncomment "cmd_cp". This gives you extra
> features.
> # EVERYONE ELSE: Leave "cmd_cp" commented out for compatibility.
> #
> # See the README file or the man page for more details.
> #
> cmd_cp /bin/cp
>
> # uncomment this to use the rm program instead of the built-in perl
> routine.
> #
> cmd_rm /bin/rm
>
> # rsync must be enabled for anything to work. This is the only
> command
> that
> # must be enabled.
> #
> cmd_rsync /usr/bin/rsync
>
> # Uncomment this to enable remote ssh backups over rsync.
> #
> #cmd_ssh /usr/bin/ssh
>
> # Comment this out to disable syslog support.
> #
> cmd_logger /usr/bin/logger
>
> # Uncomment this to specify the path to "du" for disk usage checks.
> # If you have an older version of "du", you may also want to check
> the
> # "du_args" parameter below.
> #
> cmd_du /usr/bin/du
>
> # Uncomment this to specify the path to rsnapshot-diff.
> #
> cmd_rsnapshot_diff /usr/bin/rsnapshot-diff
>
> # Specify the path to a script (and any optional arguments) to run
> right
> # before rsnapshot syncs files
> #
> #cmd_preexec /path/to/preexec/script
>
> # Specify the path to a script (and any optional arguments) to run
> right
> # after rsnapshot syncs files
> #
> #cmd_postexec /path/to/postexec/script
>
> # Paths to lvcreate, lvremove, mount and umount commands, for use
> with
> # Linux LVMs.
> #
> #linux_lvm_cmd_lvcreate /sbin/lvcreate
> #linux_lvm_cmd_lvremove /sbin/lvremove
> #linux_lvm_cmd_mount /bin/mount
> #linux_lvm_cmd_umount /bin/umount
>
> #########################################
> # BACKUP LEVELS / INTERVALS #
> # Must be unique and in ascending order #
> # e.g. alpha, beta, gamma, etc. #
> #########################################
>
> # retain hourly 24
> retain daily 7
> retain weekly 4
> retain monthly 12
> retain yearly 100
>
> ############################################
> # GLOBAL OPTIONS #
> # All are optional, with sensible defaults #
> ############################################
>
> # Verbose level, 1 through 5.
> # 1 Quiet Print fatal errors only
> # 2 Default Print errors and warnings only
> # 3 Verbose Show equivalent shell commands being executed
> # 4 Extra Verbose Show extra verbose information
> # 5 Debug mode Everything
> #
> verbose 5
>
> # Same as "verbose" above, but controls the amount of data sent to
> the
> # logfile, if one is being used. The default is 3.
> # If you want the rsync output, you have to set it to 4
> #
> loglevel 3
>
> # If you enable this, data will be written to the file you specify.
> The
> # amount of data written is controlled by the "loglevel" parameter.
> #
> logfile /var/log/rsnapshot.log
>
> # If enabled, rsnapshot will write a lockfile to prevent two
> instances
> # from running simultaneously (and messing up the snapshot_root).
> # If you enable this, make sure the lockfile directory is not world
> # writable. Otherwise anyone can prevent the program from running.
> #
> lockfile /var/run/rsnapshot.pid
>
> # By default, rsnapshot check lockfile, check if PID is running
> # and if not, consider lockfile as stale, then start
> # Enabling this stop rsnapshot if PID in lockfile is not running
> #
> #stop_on_stale_lockfile 0
>
> # Default rsync args. All rsync commands have at least these options
> set.
> #
> #rsync_short_args -a
> #rsync_long_args --delete --numeric-ids --relative --delete-excluded
>
> # ssh has no args passed by default, but you can specify some here.
> #
> #ssh_args -p 22
>
> # Default arguments for the "du" program (for disk space reporting).
> # The GNU version of "du" is preferred. See the man page for more
> details.
> # If your version of "du" doesn't support the -h flag, try -k flag
> instead.
> #
> #du_args -csh
>
> # If this is enabled, rsync won't span filesystem partitions within a
> # backup point. This essentially passes the -x option to rsync.
> # The default is 0 (off).
> #
> #one_fs 0
>
> # The include and exclude parameters, if enabled, simply get passed
> directly
> # to rsync. If you have multiple include/exclude patterns, put each
> one
> on a
> # separate line. Please look up the --include and --exclude options
> in
> the
> # rsync man page for more details on how to specify file name
> patterns.
> #
> #include ???
> #include ???
> #exclude ???
> #exclude ???
> exclude /dev/*
> exclude /proc/*
> exclude /sys/*
> exclude /tmp/*
> exclude /run/*
> exclude /mnt/*
> exclude /media/*
> exclude /lost+found
> exclude /home/lost+found
> exclude /var/lost+found
> #
> # The include_file and exclude_file parameters, if enabled, simply
> get
> # passed directly to rsync. Please look up the --include-from and
> # --exclude-from options in the rsync man page for more details.
> #
> #include_file /path/to/include/file
> #exclude_file /path/to/exclude/file
>
> # If your version of rsync supports --link-dest, consider enabling
> this.
> # This is the best way to support special files (FIFOs, etc) cross-
> platform.
> # The default is 0 (off).
> #
> link_dest 1
>
> # When sync_first is enabled, it changes the default behaviour of
> rsnapshot.
> # Normally, when rsnapshot is called with its lowest interval
> # (i.e.: "rsnapshot alpha"), it will sync files AND rotate the lowest
> # intervals. With sync_first enabled, "rsnapshot sync" handles the
> file
> sync,
> # and all interval calls simply rotate files. See the man page for
> more
> # details. The default is 0 (off).
> #
> #sync_first 0
>
> # If enabled, rsnapshot will move the oldest directory for each
> interval
> # to [interval_name].delete, then it will remove the lockfile and
> delete
> # that directory just before it exits. The default is 0 (off).
> #
> #use_lazy_deletes 0
>
> # Number of rsync re-tries. If you experience any network problems or
> # network card issues that tend to cause ssh to fail with errors like
> # "Corrupted MAC on input", for example, set this to a non-zero value
> # to have the rsync operation re-tried.
> #
> #rsync_numtries 0
>
> # LVM parameters. Used to backup with creating lvm snapshot before
> backup
> # and removing it after. This should ensure consistency of data in
> some
> special
> # cases
> #
> # LVM snapshot(s) size (lvcreate --size option).
> #
> #linux_lvm_snapshotsize 100M
>
> # Name to be used when creating the LVM logical volume snapshot(s).
> #
> #linux_lvm_snapshotname rsnapshot
>
> # Path to the LVM Volume Groups.
> #
> #linux_lvm_vgpath /dev
>
> # Mount point to use to temporarily mount the snapshot(s).
> #
> #linux_lvm_mountpath /path/to/mount/lvm/snapshot/during/backup
>
> ###############################
> ### BACKUP POINTS / SCRIPTS ###
> ###############################
>
> # LOCALHOST
> backup / localhost/
> #backup /home/ localhost/
> #backup /etc/ localhost/
> #backup /usr/ localhost/
> #backup /usr/local/ localhost/
> #backup /var/ localhost/
> #backup /var/log/rsnapshot localhost/
> #backup /etc/passwd localhost/
> #backup /home/foo/My Documents/ localhost/
> #backup /foo/bar/ localhost/ one_fs=1,rsync_short_args=-urltvpog
> #backup_script /usr/local/bin/backup_pgsql.sh localhost/postgres/
> # You must set linux_lvm_* parameters below before using lvm
> snapshots
> #backup lvm://vg0/xen-home/ lvm-vg0/xen-home/
>
> # EXAMPLE.COM
> #backup_exec /bin/date "+ backup of example.com started at %c"
> #backup root@example.com:/home/ example.com/ +rsync_long_args=--
> bwlimit=16,exclude=core
> #backup root@example.com:/etc/ example.com/ exclude=mtab,exclude=core
> #backup_exec ssh root@example.com "mysqldump -A >
> /var/db/dump/mysql.sql"
> #backup root@example.com:/var/db/dump/ example.com/
> #backup_exec /bin/date "+ backup of example.com ended at %c"
>
> # CVS.SOURCEFORGE.NET
> #backup_script /usr/local/bin/backup_rsnapshot_cvsroot.sh
> rsnapshot.cvs.sourceforge.net/
>
> # RSYNC.SAMBA.ORG
> #backup rsync://rsync.samba.org/rsyncftp/ rsync.samba.org/rsyncftp/
>
>
> ---------------------------------------------------------------------
> -
>
>
> And here is the default /etc/rsnapshot.conf that comes with
> rsnapshot:
> #################################################
> # rsnapshot.conf - rsnapshot configuration file #
> #################################################
> # #
> # PLEASE BE AWARE OF THE FOLLOWING RULE: #
> # #
> # This file requires tabs between elements #
> # #
> #################################################
>
> #######################
> # CONFIG FILE VERSION #
> #######################
>
> config_version 1.2
>
> ###########################
> # SNAPSHOT ROOT DIRECTORY #
> ###########################
>
> # All snapshots will be stored under this root directory.
> #
> snapshot_root /var/cache/rsnapshot/
>
> # If no_create_root is enabled, rsnapshot will not automatically
> create
> the
> # snapshot_root directory. This is particularly useful if you are
> backing
> # up to removable media, such as a FireWire or USB drive.
> #
> #no_create_root 1
>
> #################################
> # EXTERNAL PROGRAM DEPENDENCIES #
> #################################
>
> # LINUX USERS: Be sure to uncomment "cmd_cp". This gives you extra
> features.
> # EVERYONE ELSE: Leave "cmd_cp" commented out for compatibility.
> #
> # See the README file or the man page for more details.
> #
> cmd_cp /bin/cp
>
> # uncomment this to use the rm program instead of the built-in perl
> routine.
> #
> cmd_rm /bin/rm
>
> # rsync must be enabled for anything to work. This is the only
> command
> that
> # must be enabled.
> #
> cmd_rsync /usr/bin/rsync
>
> # Uncomment this to enable remote ssh backups over rsync.
> #
> #cmd_ssh /usr/bin/ssh
>
> # Comment this out to disable syslog support.
> #
> cmd_logger /usr/bin/logger
>
> # Uncomment this to specify the path to "du" for disk usage checks.
> # If you have an older version of "du", you may also want to check
> the
> # "du_args" parameter below.
> #
> #cmd_du /usr/bin/du
>
> # Uncomment this to specify the path to rsnapshot-diff.
> #
> #cmd_rsnapshot_diff /usr/bin/rsnapshot-diff
>
> # Specify the path to a script (and any optional arguments) to run
> right
> # before rsnapshot syncs files
> #
> #cmd_preexec /path/to/preexec/script
>
> # Specify the path to a script (and any optional arguments) to run
> right
> # after rsnapshot syncs files
> #
> #cmd_postexec /path/to/postexec/script
>
> # Paths to lvcreate, lvremove, mount and umount commands, for use
> with
> # Linux LVMs.
> #
> #linux_lvm_cmd_lvcreate /sbin/lvcreate
> #linux_lvm_cmd_lvremove /sbin/lvremove
> #linux_lvm_cmd_mount /bin/mount
> #linux_lvm_cmd_umount /bin/umount
>
> #########################################
> # BACKUP LEVELS / INTERVALS #
> # Must be unique and in ascending order #
> # e.g. alpha, beta, gamma, etc. #
> #########################################
>
> retain alpha 6
> retain beta 7
> retain gamma 4
> #retain delta 3
>
> ############################################
> # GLOBAL OPTIONS #
> # All are optional, with sensible defaults #
> ############################################
>
> # Verbose level, 1 through 5.
> # 1 Quiet Print fatal errors only
> # 2 Default Print errors and warnings only
> # 3 Verbose Show equivalent shell commands being executed
> # 4 Extra Verbose Show extra verbose information
> # 5 Debug mode Everything
> #
> verbose 2
>
> # Same as "verbose" above, but controls the amount of data sent to
> the
> # logfile, if one is being used. The default is 3.
> # If you want the rsync output, you have to set it to 4
> #
> loglevel 3
>
> # If you enable this, data will be written to the file you specify.
> The
> # amount of data written is controlled by the "loglevel" parameter.
> #
> #logfile /var/log/rsnapshot.log
>
> # If enabled, rsnapshot will write a lockfile to prevent two
> instances
> # from running simultaneously (and messing up the snapshot_root).
> # If you enable this, make sure the lockfile directory is not world
> # writable. Otherwise anyone can prevent the program from running.
> #
> lockfile /var/run/rsnapshot.pid
>
> # By default, rsnapshot check lockfile, check if PID is running
> # and if not, consider lockfile as stale, then start
> # Enabling this stop rsnapshot if PID in lockfile is not running
> #
> #stop_on_stale_lockfile 0
>
> # Default rsync args. All rsync commands have at least these options
> set.
> #
> #rsync_short_args -a
> #rsync_long_args --delete --numeric-ids --relative --delete-excluded
>
> # ssh has no args passed by default, but you can specify some here.
> #
> #ssh_args -p 22
>
> # Default arguments for the "du" program (for disk space reporting).
> # The GNU version of "du" is preferred. See the man page for more
> details.
> # If your version of "du" doesn't support the -h flag, try -k flag
> instead.
> #
> #du_args -csh
>
> # If this is enabled, rsync won't span filesystem partitions within a
> # backup point. This essentially passes the -x option to rsync.
> # The default is 0 (off).
> #
> #one_fs 0
>
> # The include and exclude parameters, if enabled, simply get passed
> directly
> # to rsync. If you have multiple include/exclude patterns, put each
> one
> on a
> # separate line. Please look up the --include and --exclude options
> in
> the
> # rsync man page for more details on how to specify file name
> patterns.
> #
> #include ???
> #include ???
> #exclude ???
> #exclude ???
>
> # The include_file and exclude_file parameters, if enabled, simply
> get
> # passed directly to rsync. Please look up the --include-from and
> # --exclude-from options in the rsync man page for more details.
> #
> #include_file /path/to/include/file
> #exclude_file /path/to/exclude/file
>
> # If your version of rsync supports --link-dest, consider enabling
> this.
> # This is the best way to support special files (FIFOs, etc) cross-
> platform.
> # The default is 0 (off).
> #
> #link_dest 0
>
> # When sync_first is enabled, it changes the default behaviour of
> rsnapshot.
> # Normally, when rsnapshot is called with its lowest interval
> # (i.e.: "rsnapshot alpha"), it will sync files AND rotate the lowest
> # intervals. With sync_first enabled, "rsnapshot sync" handles the
> file
> sync,
> # and all interval calls simply rotate files. See the man page for
> more
> # details. The default is 0 (off).
> #
> #sync_first 0
>
> # If enabled, rsnapshot will move the oldest directory for each
> interval
> # to [interval_name].delete, then it will remove the lockfile and
> delete
> # that directory just before it exits. The default is 0 (off).
> #
> #use_lazy_deletes 0
>
> # Number of rsync re-tries. If you experience any network problems or
> # network card issues that tend to cause ssh to fail with errors like
> # "Corrupted MAC on input", for example, set this to a non-zero value
> # to have the rsync operation re-tried.
> #
> #rsync_numtries 0
>
> # LVM parameters. Used to backup with creating lvm snapshot before
> backup
> # and removing it after. This should ensure consistency of data in
> some
> special
> # cases
> #
> # LVM snapshot(s) size (lvcreate --size option).
> #
> #linux_lvm_snapshotsize 100M
>
> # Name to be used when creating the LVM logical volume snapshot(s).
> #
> #linux_lvm_snapshotname rsnapshot
>
> # Path to the LVM Volume Groups.
> #
> #linux_lvm_vgpath /dev
>
> # Mount point to use to temporarily mount the snapshot(s).
> #
> #linux_lvm_mountpath /path/to/mount/lvm/snapshot/during/backup
>
> ###############################
> ### BACKUP POINTS / SCRIPTS ###
> ###############################
>
> # LOCALHOST
> backup /home/ localhost/
> backup /etc/ localhost/
> backup /usr/local/ localhost/
> #backup /var/log/rsnapshot localhost/
> #backup /etc/passwd localhost/
> #backup /home/foo/My Documents/ localhost/
> #backup /foo/bar/ localhost/ one_fs=1,rsync_short_args=-urltvpog
> #backup_script /usr/local/bin/backup_pgsql.sh localhost/postgres/
> # You must set linux_lvm_* parameters below before using lvm
> snapshots
> #backup lvm://vg0/xen-home/ lvm-vg0/xen-home/
>
> # EXAMPLE.COM
> #backup_exec /bin/date "+ backup of example.com started at %c"
> #backup root@example.com:/home/ example.com/ +rsync_long_args=--
> bwlimit=16,exclude=core
> #backup root@example.com:/etc/ example.com/ exclude=mtab,exclude=core
> #backup_exec ssh root@example.com "mysqldump -A >
> /var/db/dump/mysql.sql"
> #backup root@example.com:/var/db/dump/ example.com/
> #backup_exec /bin/date "+ backup of example.com ended at %c"
>
> # CVS.SOURCEFORGE.NET
> #backup_script /usr/local/bin/backup_rsnapshot_cvsroot.sh
> rsnapshot.cvs.sourceforge.net/
>
> # RSYNC.SAMBA.ORG
> #backup rsync://rsync.samba.org/rsyncftp/ rsync.samba.org/rsyncftp/
>
>
> -------------------------------------------------------------------
>
>
> [BTW, the rsnapshot backups don't seem to take too much time, but
> doing
> rsync of external usb backup drive A to external usb backup drive B
> does take over 90 minutes each time. And that's once a day, every
> day!
> Most of that time is apparently not for data transfer, but for rsync
> building the indexes it needs each time.]
>
> Here is the command I use to rsync backup drive A
> (/media/default/MSD00001) to backup drive B
> (/media/default/MSD00002):
>
> time sudo rsync -aAXHxvv --delete-after --numeric-ids --
> info=progress2,stats2,name2 --
> exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/med
> ia
> /*","/lost+found"} /media/default/MSD00001/ /media/default/MSD00002/
>
>
***********************************************************************
I just wanted to clarify:
Each time backup drive A is rsync'd to backup drive B, much more than
/media/user/MSD00001/rsnapshot_backups_of_host is being rsync'd. All
of /media/user/MSD00001 is being rsync'd, which is somewhere around
900Gb. Maybe that is why each rsync takes over 90 minutes!
Reply to: