Re: pore-c is interesting - git push lfs issue
Hi Steffen,
On Mon, Jun 08, 2020 at 03:17:39AM +0200, Steffen Möller wrote:
> Hello,
>
> pore-c has a series of dependencies that are of interest for Debian as a
> whole and still missing
>
> python3-pyarrow
> python3-streamz,
> python3-pyranges,
> python3-pairtools,
> python3-ncls,
> python3-intake
OK, so we can spent the time to package this until we have sorted out
the LFS issue. Please urgently remember: If the Python tool is somehow
biology or since related it makes perfectly sense to keep it inside
Debian Med team. We are way better to do QA once it is in our scope
and error reports go to some mailing list that is watched by our team
closely. The pure fact that a module is written in Python is no good
reason to to the packaging in Python team and as far as I understood
members of the Python team they want only those packages there that are
*closely watched* by members of the DPMT - which is not the case for our
Debian Med packages. So please stay here if it makes any sense and the
package is not extremely generic.
> The package builds but does not run without these. In an attempt to push
> what I have, I ran into
>
> remote: GitLab: LFS objects are missing. Ensure LFS is properly set up
> or try a manual "git lfs push --all".
> To salsa.debian.org:med-team/pore-c.git
> ! [remote rejected]master -> master (pre-receive hook declined)
> ! [remote rejected]pristine-tar -> pristine-tar (pre-receive hook declined)
> ! [remote rejected]upstream -> upstream (pre-receive hook declined)
> error: failed to push some refs to 'salsa.debian.org:med-team/pore-c.git'
> $ git lfs push --all
> Specify a remote and a remote branch name (`git lfs push origin master`)
> $ git lfs push origin master
> Locking support detected on remote "origin". Consider enabling it with:
> $ git config
> lfs.https://salsa.debian.org/med-team/pore-c.git/info/lfs.locksverify true
> LFS upload failed:cts: 0% (0/23), 0 B | 0 B/s
> (missing) tests/data/NlaIII_run01_GRCh38.haplotagged.bam
> (1214675dd07b5a072ecbe9bf22ae7513c6f2c6b1435a86c0b8b93e6e85da6293)
> (missing) tests/data/NlaIII_GRCh38.contacts.parquet/part.0.parquet
> (94b3ebd6d94f10a2de3093d46473a51c003cd41713c9b3712d56ab30b676e193)
> (missing) tests/data/NlaIII_GRCh38.concatemers.parquet/_metadata
> (91d16d1ec1483db8b8080179ba9b485dce5bb625110768c23c348df6bef19e60)
> (missing) tests/data/GM12878_NlaIII_reads.tsv
> (7ce4529be82183332f9f3e37fa85f19a0a2aa5afa887f742145016bb2edcca53)
> (missing) tests/data/GRCh38.chromsizes
> (50e35247e1305e2a0a1fd1b7c07ff3576cec9389382d611c19eb0055633fc82b)
> (missing) tests/data/GM12878.phased.conf.vcf.gz
> (a7efa2fe2641ac2c76c0596ad628f90d9a8531856115f8b840379e95ee0709c8)
> (missing) tests/data/GM12878_NlaIII_reads.fq.gz
> (9d722847361b9c2d037d50d7e9cd126c958f5a0f689694fe5dfef37ae3ac1c4a)
> (missing) tests/data/GRCh38.fasta.gz.gzi
> (be5698ffc76f644f9483d04b3d58c8c2c7f0ba6a1fd37e78ac5eb99f5901e4f3)
> (missing) tests/data/NlaIII_GRCh38.contacts.parquet/_common_metadata
> (950401c687f40c995d415799583fa38ed1e063ceddd500d77c182388dda35d00)
> (missing)
> tests/data/NlaIII_run01_GRCh38.align_table.parquet/_common_metadata
> (82b94a7abce2fe49c8321de6ce5cb127c832e6de3dd74aa1c0e756098ab26e86)
> (missing) tests/data/NlaIII_GRCh38.vd.fragments.parquet/_metadata
> (b97516f9b93688053d27efe7f604fde9573e63444354e42d31507922df8ca3b1)
> (missing) tests/data/GRCh38.fasta.gz.fai
> (cbd68404c124a5e3ee4e4335bf907cbd0db00b1a077d695042495260716dfd9b)
> (missing) tests/data/GRCh38.fasta.gz
> (79582e1bbeec718000027048849a6413ee576f17c1d9aa877211db1f8d6a7b1b)
> (missing) tests/data/NlaIII_run01_GRCh38.read_sort.bam
> (b451a851e775d7c364eb844feafa1a0054939be6035b0e6cfb808d2e13db2f82)
> (missing) tests/data/test_ns.sam
> (d9a0a31551e5cec6444e9fbfb6b87f0b644ef5a4ab9bad09329a856eb39160de)
> (missing) tests/data/GM12878.phased.conf.vcf.gz.tbi
> (0db1dcb337cfe2d23dd6d2369b49ec1414961ee2305e635ca6af7289aacf16ab)
> (missing) tests/data/NlaIII_GRCh38.contacts.parquet/_metadata
> (83aaf5e4f09b9cafa30446bd3f8016164c450d23ba840cbd62bcef55049e3356)
> Uploading LFS objects: 0% (0/23), 0 B | 0 B/s, done.
> (missing) tests/data/NlaIII_GRCh38.concatemers.parquet/part.0.parquet
> (24ce696c45a21106d36ca97278f49ac707c1d77b97f86c77408ae0ee42c9905f)
> (missing) tests/data/NlaIII_run01_GRCh38.align_table.parquet/_metadata
> (42660830367bfa83ef9b4a5a501bdf0e19fd6c3a826210e8098e297443aad04e)
> (missing) tests/data/NlaIII_GRCh38.vd.fragments.parquet/part.0.parquet
> (6684beefe4fa555e3ef5ba2dca13769cda043424c08ac8f2b647471d7b587c80)
> (missing)
> tests/data/NlaIII_run01_GRCh38.align_table.parquet/part.0.parquet
> (8fecdb8434662f0c0892dcdd103078ff2c48c96f40cc3546a9e76638b9c42740)
> (missing)
> tests/data/NlaIII_GRCh38.concatemers.parquet/_common_metadata
> (ec1a8f121551b7d0bccc8f2eb4097742029b30068313dc15a932b701e4d91b0c)
> (missing)
> tests/data/NlaIII_GRCh38.vd.fragments.parquet/_common_metadata
> (2902630654ed49ea1ea6648dc21bba0c20b499fa6182a6b2ec6b531b7564cb6e)
> hint: Your push was rejected due to missing or corrupt local objects.
> hint: You can disable this check with: 'git config
> lfs.allowincompletepush true'
>
> Well, kind of curious I did enable it, but could not push --all. After
> also pushing upstream, I went for pristine-tar without lsf
>
>
> $ git push --set-upstream origin pristine-tar
> Enumerating objects: 4, done.
> Counting objects: 100% (4/4), done.
> Delta compression using up to 2 threads
> Compressing objects: 100% (3/3), done.
> Writing objects: 100% (4/4), 3.11 KiB | 3.11 MiB/s, done.
> Total 4 (delta 0), reused 0 (delta 0), pack-reused 0
> To salsa.debian.org:med-team/pore-c.git
> * [new branch] pristine-tar -> pristine-tar
> Branch 'pristine-tar' set up to track remote branch 'pristine-tar' from
> 'origin'.
> moeller@steffen-laptop-debian:~/git/med-team/pore-c/pore-c$ git checkout
> master
> Downloading tests/data/GM12878.phased.conf.vcf.gz (516 KB)
> Error downloading object: tests/data/GM12878.phased.conf.vcf.gz
> (a7efa2f): Smudge error: Error downloading
> tests/data/GM12878.phased.conf.vcf.gz (a7efa2fe2641ac2c76c0596ad628f90d9a853
> 1856115f8b840379e95ee0709c8):
> [a7efa2fe2641ac2c76c0596ad628f90d9a8531856115f8b840379e95ee0709c8]
> Object does not exist on the server or you don't have permissions to
> access it: [404]
> Object does not exist on the server or you don't have permissions to
> access it
>
> Errors logged to
> /home/moeller/git/med-team/pore-c/pore-c/.git/lfs/logs/20200608T030628.335563029.log
> Use `git lfs logs last` to view the log.
> error: external filter 'git-lfs filter-process' failed
> fatal: tests/data/GM12878.phased.conf.vcf.gz: smudge filter lfs failed
> moeller@steffen-laptop-debian:~/git/med-team/pore-c/pore-c$ git branch
> master
> * pristine-tar
> upstream
>
> Hm. Ok. This is all new to me.
I admit I had to deal with lfs the first time in my first attempt with
gatk which was stalled. There the download tarball contained a really
huge file that would require git lfs.
However, I had to do again with git lfs in flappie and here the case was
different - and may be is the same as in your case. The flappie
download tarball did *not* contain large files but just broken git lfs
references. Pretty short files referencing some Git location. That's
pretty useless in a tarball - no idea how to prevent this at tarball
creation time. I simply excluded these files in
https://salsa.debian.org/med-team/flappie/-/blob/master/debian/copyright
Since all are test data we can find some other means if it is about
testing later. If I'm not missing anything in your log its also only
about test data. If you ask me you could try the same, remove the files
- which in case of non-working references should be cheap anyway - and
recreate the git repository.
> I know about the large file support, but
> why is this enforced on me when working with a regular tarball that is
> not even big (most likely because of those special files pointing to
> external data).
I noticed that on stable `gbp import-orig` is including the tarball -
just tested it since I suspected some bug in gbp from unstable.
However, its not a bug but a feature: We do not want non-working
references in our repository. Thus I removed this and our repository is
at least consistent while lacking **references** to test data. The test
data itself are missing in the tarball anyway. Your description smells
pretty much like the same I have observed in flappie.
> My immediate response would be to remove the test data
> and find other ways to perform the testing.
So we perfectly agree here. :-)
Kind regards
Andreas.
--
http://fam-tille.de
Reply to: