[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

[Git][snapshot-team/snapshot][master] 2 commits: Move the indexing lock to the database

Title: GitLab

Julien Cristau pushed to branch master at snapshot / snapshot

Commits:

  • 28223016
    by Julien Cristau at 2024-04-06T11:41:10+02:00 
    Move the indexing lock to the database

We're currently running imports from 2 different hosts so we can't rely
on filesystem-based locking.
  • 2aa2c9f4
    by Julien Cristau at 2024-04-06T09:45:40+00:00 
    Merge branch 'index-lock-db' into 'master'

Move the indexing lock to the database

See merge request snapshot-team/snapshot!14

2 changed files:

Changes:

  • master/import-run
    ... ... @@ -53,14 +53,7 @@ case "$archive" in
    53 53
    54 54 
     		#"$BASE"/code/misc/dump-tools/add-new-dumps-to-git -c "$CONFIG" -b "$BASE"/dumps-git
    55 55
    56 
    -		indexlock="$BASE/.snapshot-index.lock"
    57 
    -		touch "$indexlock"
    58 
    -		exec 204< "$indexlock"
    59 
    -		if ! flock --exclusive 204; then
    60 
    -			echo >&2 "Warning: Not running index as we don't have the lock."
    61 
    -		else
    62 
    -			snapshot index --config "$CONFIG" --quick $verbose
    63 
    -		fi
    56 
    +		snapshot index --config "$CONFIG" --quick $verbose
    64 57
    65 58 
     		ls -l "$BASE"/fuse/.ctrl-reload-database > /dev/null 2>&1 || true
    66 59

  • snapshot
    ... ... @@ -1243,6 +1243,7 @@ class PackageIndexer
    1243 1243 
     	def index
    1244 1244 
     		if @only_this_mirrorrun
    1245 1245 
     				@db.begin
    1246 
    +				@db.execute("LOCK indexed_mirrorrun IN SHARE ROW EXCLUSIVE MODE")
    1246 1247 
     				row = @db.query_row("SELECT run, (SELECT name FROM archive WHERE archive.archive_id = mirrorrun.archive_id) AS archive
    1247 1248 
     									 FROM mirrorrun WHERE mirrorrun_id = $1", @only_this_mirrorrun)
    1248 1249 
     				if row.nil?
    ... ... @@ -1258,9 +1259,13 @@ class PackageIndexer
    1258 1259 
     			@db.query("SELECT mirrorrun_id, run, (SELECT name FROM archive WHERE archive.archive_id = mirrorrun.archive_id) AS archive
    1259 1260 
     					   FROM mirrorrun WHERE NOT mirrorrun_id IN (SELECT mirrorrun_id FROM indexed_mirrorrun) ORDER BY run") do |row|
    1260 1261 
     				@db.begin
    1261 
    -				$logger.info("Indexing mirrorrun ##{row['mirrorrun_id']} of #{row['archive']} from #{row['run']}")
    1262 
    -				source = index_mirrorrun(row['mirrorrun_id'])
    1263 
    -				@db.insert('indexed_mirrorrun', {'mirrorrun_id' => row['mirrorrun_id'], 'source' => source })
    1262 
    +				@db.execute("LOCK indexed_mirrorrun IN SHARE ROW EXCLUSIVE MODE")
    1263 
    +				check = @db.query_row("SELECT 1 FROM indexed_mirrorrun WHERE mirrorrun_id = $1", row['mirrorrun_id'])
    1264 
    +				if check.nil?
    1265 
    +					$logger.info("Indexing mirrorrun ##{row['mirrorrun_id']} of #{row['archive']} from #{row['run']}")
    1266 
    +					source = index_mirrorrun(row['mirrorrun_id'])
    1267 
    +					@db.insert('indexed_mirrorrun', {'mirrorrun_id' => row['mirrorrun_id'], 'source' => source })
    1268 
    +				end
    1264 1269 
     				@db.commit
    1265 1270 
     			end
    1266 1271 
     		end

    • —
    View it on GitLab.
    You're receiving this email because of your account on salsa.debian.org. Manage all notifications · Help

    Reply to: