[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

[Git][snapshot-team/snapshot][master] 2 commits: sync-farm-incremental: run 4 syncers in parallel

Title: GitLab

Peter Palfrader pushed to branch master at snapshot / snapshot

Commits:

  • e7780d89
    by Peter Palfrader at 2025-04-12T12:42:12+02:00 
    sync-farm-incremental: run 4 syncers in parallel
  • e080b7b3
    by Peter Palfrader at 2025-04-12T13:03:22+02:00 
    Add sync-farm-incremental ssh trigger that uses a systemd service (so we have sane logging and error reporting)

3 changed files:

Changes:

  • mirror/snapshot-sync-farm-incremental.service.sample
    1 
    +[Unit]
    2 
    +Description=snapshot incremental farm sync
    3 
    +
    4 
    +[Service]
    5 
    +ExecStart=/home/snapshot/code/mirror/sync-farm-incremental

  • mirror/sync-farm-incremental
    ... ... @@ -3,20 +3,43 @@
    3 3 
     set -e
    4 4 
     set -o pipefail
    5 5
    6 
    -farm="$(readlink -f farm)"
    7 
    -cd "$farm"
    8 
    -for i in *; do
    9 
    -  exec 200< "$i"
    6 
    +if [ "$#" = 2 ]; then
    7 
    +  farm="$1"; shift
    8 
    +  item="$1"; shift
    9 
    +  echo >&2 "Doing ${item}"
    10 
    +
    11 
    +  cd "$farm"
    12 
    +
    13 
    +  exec 200< "$item"
    10 14 
       flock -e 200
    11 15
    12 
    -  tmpdir="$(mktemp -d --tmpdir="$i" ".incremental-update-XXXXXX")"
    16 
    +  tmpdir="$(mktemp -d --tmpdir="$item" ".incremental-update-XXXXXX")"
    13 17 
       tmpdir="$(readlink -f "$tmpdir")"
    14 18 
       trap "rm -rf '$tmpdir'" EXIT
    15 19
    16 
    -  ~/code/mirror/farm-list-entries --farmpath "$farm" "$i" |
    17 
    -    ssh -4 -C snapshot-mlm-01.debian.org farm-make-update-tarball "$i" |
    20 
    +  ~/code/mirror/farm-list-entries --farmpath "$farm" "$item" |
    21 
    +    ssh -4 -C snapshot-mlm-01.debian.org farm-make-update-tarball "$item" |
    18 22 
         (cd $tmpdir && tar xaf -)
    19 23
    20 24 
       ~/code/mirror/process-farm-update --farmpath "$farm" "$tmpdir"
    21 25 
       rm -rf "$tmpdir"
    22 
    -done
    26 
    +elif [ "$#" = 0 ]; then
    27 
    +  self="${BASH_SOURCE[0]}"
    28 
    +
    29 
    +  farm="$(readlink -f farm)"
    30 
    +  while :; do
    31 
    +    rm -vf "${farm}/.incremental-sync-needed"
    32 
    +
    33 
    +    find "$farm" -mindepth 1 -maxdepth 1 -type d -printf '%f\n' | \
    34 
    +      grep '^[0-9a-f][0-9a-f]$' | \
    35 
    +      sort -R | \
    36 
    +      xargs -n 1 -P 4 "$self" "$farm"
    37 
    +
    38 
    +    [ -e "${farm}/.incremental-sync-needed" ] || break
    39 
    +    echo "Re-running because ${farm}/.incremental-sync-needed exists again."
    40 
    +  done
    41 
    +else
    42 
    +  echo >&2 "Usage: $0"
    43 
    +  echo >&2 "Usage: $0 <farmpath> <dir-to-process [0-9a-f][0-9a-f]>"
    44 
    +  exit 1
    45 
    +fi

  • mirror/sync-farm-incremental-ssh-trigger
    1 
    +#!/bin/bash
    2 
    +
    3 
    +set -e
    4 
    +set -o pipefail
    5 
    +farm="$(readlink -f farm)"
    6 
    +
    7 
    +touch "${farm}/.incremental-sync-needed"
    8 
    +systemctl --user start snapshot-sync-farm-incremental.service

    • —
    View it on GitLab.
    You're receiving this email because of your account on salsa.debian.org. Manage all notifications · Help

    Reply to: