Bug#595720: future unblock: libdbm-deep-perl/2.0002-1
Package: release.debian.org
Severity: normal
User: release.debian.org@packages.debian.org
Usertags: unblock
libdbm-deep-perl can corrupt the database if too many transactions are
started[1]. This was fixed upstream in version 2.0002. There is
another minor change in behavior in 2.0001, but this should not affect
Debian (the only rdep, libipc-pubsub-perl, seems not to use this
feature).
The debdiff from the version currently in testing to the proposed upload
is attached. Should we go ahead with the upload? (Note that 2.0001 is
already in unstable.)
Regards,
Ansgar
PS: Please Cc: debian-perl@lists.debian.org in replies.
[1] <https://rt.cpan.org/Public/Bug/Display.html?id=60903>
Index: debian/control
===================================================================
--- debian/control (.../tags/libdbm-deep-perl/2.0000-1) (revision 62350)
+++ debian/control (.../trunk/libdbm-deep-perl) (revision 62350)
@@ -1,7 +1,7 @@
Source: libdbm-deep-perl
Section: perl
Priority: optional
-Build-Depends: debhelper (>= 7.3.7~)
+Build-Depends: debhelper (>= 8)
Build-Depends-Indep: perl, libtest-deep-perl (>= 0.095),
libtest-exception-perl (>= 0.21), libtest-warn-perl,
libio-stringy-perl, libfilehandle-fmode-perl, libtest-pod-coverage-perl,
@@ -11,7 +11,7 @@
Uploaders: AGOSTINI Yves <agostini@univ-metz.fr>,
gregor herrmann <gregoa@debian.org>,
Ansgar Burchardt <ansgar@43-1.org>
-Standards-Version: 3.9.0
+Standards-Version: 3.9.1
Homepage: http://search.cpan.org/dist/DBM-Deep/
Vcs-Svn: svn://svn.debian.org/pkg-perl/trunk/libdbm-deep-perl/
Vcs-Browser: http://svn.debian.org/viewsvn/pkg-perl/trunk/libdbm-deep-perl/
Index: debian/compat
===================================================================
--- debian/compat (.../tags/libdbm-deep-perl/2.0000-1) (revision 62350)
+++ debian/compat (.../trunk/libdbm-deep-perl) (revision 62350)
@@ -1 +1 @@
-7
+8
Index: debian/changelog
===================================================================
--- debian/changelog (.../tags/libdbm-deep-perl/2.0000-1) (revision 62350)
+++ debian/changelog (.../trunk/libdbm-deep-perl) (revision 62350)
@@ -1,3 +1,25 @@
+libdbm-deep-perl (2.0002-1) unstable; urgency=medium
+
+ * New upstream release.
+ + Errors out instead of corrupting the database when starting too many
+ transactions.
+ * debian/copyright: Refer to "Debian systems" instead of "Debian GNU/Linux
+ systems".
+
+ -- Ansgar Burchardt <ansgar@43-1.org> Mon, 06 Sep 2010 12:05:09 +0900
+
+libdbm-deep-perl (2.0001-1) unstable; urgency=low
+
+ * New upstream release.
+ * Use debhelper compat level 8; no longer pass --buildsystem=perl_build in
+ debian/rules as this is now selected by default.
+ * debian/rules: Remove DBM::Deep::ConfigData module that gets generated by
+ Module::Build.
+ * debian/copyright: Refer to /usr/share/common-licenses/GPL-1.
+ * Bump Standards-Version to 3.9.1.
+
+ -- Ansgar Burchardt <ansgar@43-1.org> Wed, 25 Aug 2010 22:12:50 +0900
+
libdbm-deep-perl (2.0000-1) unstable; urgency=low
[ Ansgar Burchardt ]
Index: debian/copyright
===================================================================
--- debian/copyright (.../tags/libdbm-deep-perl/2.0000-1) (revision 62350)
+++ debian/copyright (.../trunk/libdbm-deep-perl) (revision 62350)
@@ -18,8 +18,8 @@
This program is free software; you can redistribute it and/or modify
it under the terms of the Artistic License, which comes with Perl.
.
- On Debian GNU/Linux systems, the complete text of the Artistic License
- can be found in `/usr/share/common-licenses/Artistic'
+ On Debian systems, the complete text of the Artistic License can be
+ found in `/usr/share/common-licenses/Artistic'.
License: GPL-1+
This program is free software; you can redistribute it and/or modify
@@ -27,5 +27,5 @@
the Free Software Foundation; either version 1, or (at your option)
any later version.
.
- On Debian GNU/Linux systems, the complete text of the GNU General
- Public License can be found in `/usr/share/common-licenses/GPL'
+ On Debian systems, the complete text of version 1 of the GNU General
+ Public License can be found in `/usr/share/common-licenses/GPL-1'.
Index: debian/rules
===================================================================
--- debian/rules (.../tags/libdbm-deep-perl/2.0000-1) (revision 62350)
+++ debian/rules (.../trunk/libdbm-deep-perl) (revision 62350)
@@ -1,7 +1,16 @@
#!/usr/bin/make -f
+PKG := $(shell dh_listpackages)
+TMP := $(CURDIR)/debian/$(PKG)
+
%:
- dh $@ --buildsystem=perl_build
+ dh $@
+override_dh_auto_install:
+ dh_auto_install
+ # Generated by Module::Build, but not used.
+ rm $(TMP)/usr/share/perl5/DBM/Deep/ConfigData.pm \
+ $(TMP)/usr/share/man/man3/DBM::Deep::ConfigData.3pm
+
override_dh_auto_test:
- LONG_TESTS=1 TEST_SQLITE=1 dh_auto_test --buildsystem=perl_build
+ LONG_TESTS=1 TEST_SQLITE=1 dh_auto_test
Index: t/04_array.t
===================================================================
--- t/04_array.t (.../tags/libdbm-deep-perl/2.0000-1) (revision 62350)
+++ t/04_array.t (.../trunk/libdbm-deep-perl) (revision 62350)
@@ -112,7 +112,7 @@
# exists
##
ok( $db->exists(1), "The 1st value exists" );
- ok( $db->exists(0), "The 0th value doesn't exist" );
+ ok( !$db->exists(0), "The 0th value doesn't exist" );
ok( !$db->exists(22), "The 22nd value doesn't exists" );
ok( $db->exists(-1), "The -1st value does exists" );
ok( !$db->exists(-22), "The -22nd value doesn't exists" );
Index: t/43_transaction_maximum.t
===================================================================
--- t/43_transaction_maximum.t (.../tags/libdbm-deep-perl/2.0000-1) (revision 62350)
+++ t/43_transaction_maximum.t (.../trunk/libdbm-deep-perl) (revision 62350)
@@ -30,12 +30,15 @@
),
()
)
- } 2 .. $max_txns;
- if($reached_max) {
+ } 2 .. $max_txns-1; # -1 because the head is included in the number
+ if($reached_max) { # of transactions
diag "This OS apparently can open only $max_txns files.";
}
- cmp_ok( scalar(@dbs), '==', $max_txns, "We could open enough DB handles" );
+ cmp_ok(
+ scalar(@dbs), '==', $max_txns-1,
+ "We could open enough DB handles"
+ );
my %trans_ids;
for my $n (0 .. $#dbs) {
Index: t/06_error.t
===================================================================
--- t/06_error.t (.../tags/libdbm-deep-perl/2.0000-1) (revision 62350)
+++ t/06_error.t (.../trunk/libdbm-deep-perl) (revision 62350)
@@ -137,4 +137,22 @@
)1\.0003 to \d/, "Fail if opening a file version 1";
}
+{
+ # Make sure we get the right file name in the error message.
+ throws_ok {
+ eval "#line 1 gneen\nDBM::Deep->new( 't/etc/db-0-99_04' )"
+ or die $@
+ } qr/ at gneen line 1\b/, "File name in error message is correct";
+}
+
+{
+ # Too many transactions.
+ my ($fh, $filename) = new_fh();
+
+ throws_ok {
+ new DBM::Deep $filename =>-> begin_work;
+ } qr/^DBM::Deep: Cannot allocate transaction ID at/,
+ "Error when starting transaction in database with only 1 txn";
+}
+
done_testing;
Index: t/03_bighash.t
===================================================================
--- t/03_bighash.t (.../tags/libdbm-deep-perl/2.0000-1) (revision 62350)
+++ t/03_bighash.t (.../trunk/libdbm-deep-perl) (revision 62350)
@@ -44,7 +44,8 @@
cmp_deeply( \@keys, \@control, "Correct keys are there" );
ok( !exists $foo->{does_not_exist}, "EXISTS works on large hashes for non-existent keys" );
- is( $foo->{does_not_exist}, undef, "autovivification works on large hashes" );
+ $foo->{does_not_exist}{ling} = undef;
+ ok( $foo->{does_not_exist}, "autovivification works on large hashes" );
ok( exists $foo->{does_not_exist}, "EXISTS works on large hashes for newly-existent keys" );
cmp_ok( scalar(keys %$foo), '==', $max_keys + 2, "Number of keys after autovivify is correct" );
Index: t/02_hash.t
===================================================================
--- t/02_hash.t (.../tags/libdbm-deep-perl/2.0000-1) (revision 62350)
+++ t/02_hash.t (.../trunk/libdbm-deep-perl) (revision 62350)
@@ -39,12 +39,9 @@
ok( exists $db->{key2}, "exists() works against tied hash" );
ok( !exists $db->{key4}, "exists() function works for keys that aren't there" );
- is( $db->{key4}, undef, "Autovivified key4" );
- ok( exists $db->{key4}, "Autovivified key4 now exists" );
+ is( $db->{key4}, undef, "Nonexistent key4 is undef" );
+ ok( !exists $db->{key4}, "Simply reading key4 does not autovivify" );
- delete $db->{key4};
- ok( !exists $db->{key4}, "And key4 doesn't exists anymore" );
-
# Keys will be done via an iterator that keeps a breadcrumb trail of the last
# key it provided. There will also be an "edit revision number" on the
# reference so that resetting the iterator can be done.
Index: t/28_index_sector.t
===================================================================
--- t/28_index_sector.t (.../tags/libdbm-deep-perl/2.0000-1) (revision 62350)
+++ t/28_index_sector.t (.../trunk/libdbm-deep-perl) (revision 62350)
@@ -27,7 +27,8 @@
cmp_ok( scalar(@keys), '==', 17, "Right number of keys returned" );
ok( !exists $db->{does_not_exist}, "EXISTS works on large hashes for non-existent keys" );
- is( $db->{does_not_exist}, undef, "autovivification works on large hashes" );
+ $db->{does_not_exist}{ling} = undef;
+ ok( $db->{does_not_exist}, "autovivification works on large hashes" );
ok( exists $db->{does_not_exist}, "EXISTS works on large hashes for newly-existent keys" );
cmp_ok( scalar(keys %$db), '==', 18, "Number of keys after autovivify is correct" );
}
Index: META.yml
===================================================================
--- META.yml (.../tags/libdbm-deep-perl/2.0000-1) (revision 62350)
+++ META.yml (.../trunk/libdbm-deep-perl) (revision 62350)
@@ -23,7 +23,7 @@
provides:
DBM::Deep:
file: lib/DBM/Deep.pm
- version: 2.0000
+ version: 2.0002
DBM::Deep::Array:
file: lib/DBM/Deep/Array.pm
DBM::Deep::Engine:
@@ -81,4 +81,4 @@
perl: 5.008_004
resources:
license: http://dev.perl.org/licenses/
-version: 2.0000
+version: 2.0002
Index: lib/DBM/Deep.pm
===================================================================
--- lib/DBM/Deep.pm (.../tags/libdbm-deep-perl/2.0000-1) (revision 62350)
+++ lib/DBM/Deep.pm (.../trunk/libdbm-deep-perl) (revision 62350)
@@ -6,7 +6,7 @@
use warnings FATAL => 'all';
no warnings 'recursion';
-our $VERSION = q(2.0000);
+our $VERSION = q(2.0002);
use Scalar::Util ();
@@ -504,7 +504,7 @@
my @caller = caller( ++$n );
next if $caller[0] =~ m/^DBM::Deep/;
- die "DBM::Deep: $_[1] at $0 line $caller[2]\n";
+ die "DBM::Deep: $_[1] at $caller[1] line $caller[2]\n";
}
}
Index: lib/DBM/Deep/Internals.pod
===================================================================
--- lib/DBM/Deep/Internals.pod (.../tags/libdbm-deep-perl/2.0000-1) (revision 62350)
+++ lib/DBM/Deep/Internals.pod (.../trunk/libdbm-deep-perl) (revision 62350)
@@ -7,6 +7,8 @@
This document is out-of-date. It describes an intermediate file format used
during the development from 0.983 to 1.0000. It will be rewritten soon.
+So far, the description of the header format has been updated.
+
=head1 DESCRIPTION
This is a document describing the internal workings of L<DBM::Deep>. It is
@@ -43,11 +45,24 @@
=head1 FILE LAYOUT
+This describes the 1.0003 and 2.0000 formats, which internally are numbered
+3 and 4, respectively. The internal numbers are used in this section. These
+two formats are almost identical.
+
DBM::Deep uses a tagged file layout. Every section has a tag, a size, and then
the data.
=head2 File header
+The file header consists of two parts. The first part is a fixed length of
+13 bytes:
+
+ DBDP h VVVV SSSS
+ \ / | \ \
+ \/ '---. \ '--- size of the second part of the header
+ file \ '--- version
+ signature tag
+
=over 4
=item * File Signature
@@ -55,30 +70,80 @@
The first four bytes are 'DPDB' in network byte order, signifying that this is
a DBM::Deep file.
-=item * File tag/size
+=item * File tag
-This is the tagging of the file header. The file used by versions prior to
-1.00 had a different fifth byte, allowing the difference to the determined.
+A literal ASCII 'h', indicating that this is the header. The file used by
+versions prior to 1.00 had a different fifth byte, allowing the difference
+to be determined.
=item * Version
This is four bytes containing the file version. This lets the file format change over time.
+It is packed in network order, so version 4 is stored as "\0\0\0\cD".
+
+=item * Header size
+
+The size of the second part of the header, in bytes. This number is also
+packed in network order.
+
+=back
+
+The second part of the header is as follows:
+
+ S B S T T(TTTTTTTTT...) (SS SS SS SS ...) (continued...)
+ | | | | \ |
+ | | | '----------. \ staleness counters
+ | | '--------. \ txn bitfield
+ | '------. \ number of transactions
+ byte size \ data sector size
+ max buckets
+
+ (continuation...)
+ BB(BBBBBB) DD(DDDDDD) II(IIIIII)
+ | | |
+ | free data |
+ free blist free index
+
+=over
+
=item * Constants
These are the file-wide constants that determine how the file is laid out.
They can only be set upon file creation.
+The byte size is the number of bytes used to point to an offset elsewhere
+in the file. This corresponds to the C<pack_size> option. This and the
+next three values are stored as packed 8-bit integers (chars), so 2 is
+represented by "\cB".
+
+C<max_buckets> and C<data_sector_size> are documented in the main
+L<DBM::Deep> man page. The number stored is actually one less than what is
+passed to the constructor, to allow for a range of 1-256.
+
+The number of transactions corresponds to the C<num_txns> value passed to
+the constructor.
+
=item * Transaction information
-The current running transactions are stored here, as is the next transaction
-ID.
+The transaction bitfield consists of one bit for every available
+transaction ID. It is therefore anywhere from 1 byte to 32 bytes long.
+The staleness counters each take two bytes (packed 32-bit integers), one
+for each transaction, not including the so-called HEAD (the main
+transaction that all processes share I<before> calling C<begin_work>). So
+these take up 0 to 508 bytes.
+
+Staleness is explained in L<DBM::Deep::Engine|DBM::Deep::Engine/STALENESS>.
+
=item * Freespace information
-Pointers into the next free sectors of the various sector sizes (Index,
-Bucketlist, and Data) are stored here.
+Pointers into the first free sectors of the various sector sizes (Index,
+Bucketlist, and Data) are stored here. These are called chains internally,
+as each free sector points to the next one.
+The number of bytes is determined by the byte size, ranging from 2 to 8.
+
=back
=head2 Index
Index: lib/DBM/Deep/Engine/DBI.pm
===================================================================
--- lib/DBM/Deep/Engine/DBI.pm (.../tags/libdbm-deep-perl/2.0000-1) (revision 62350)
+++ lib/DBM/Deep/Engine/DBI.pm (.../trunk/libdbm-deep-perl) (revision 62350)
@@ -83,17 +83,7 @@
});
unless ( $value_sector ) {
- $value_sector = DBM::Deep::Sector::DBI::Scalar->new({
- engine => $self,
- data => undef,
- data_type => 'S',
- });
-
- $sector->write_data({
-# key_md5 => $key_md5,
- key => $key,
- value => $value_sector,
- });
+ return undef
}
return $value_sector->data;
Index: lib/DBM/Deep/Engine/File.pm
===================================================================
--- lib/DBM/Deep/Engine/File.pm (.../tags/libdbm-deep-perl/2.0000-1) (revision 62350)
+++ lib/DBM/Deep/Engine/File.pm (.../trunk/libdbm-deep-perl) (revision 62350)
@@ -154,16 +154,7 @@
});
unless ( $value_sector ) {
- $value_sector = DBM::Deep::Sector::File::Null->new({
- engine => $self,
- data => undef,
- });
-
- $sector->write_data({
- key_md5 => $key_md5,
- key => $key,
- value => $value_sector,
- });
+ return undef
}
return $value_sector->data;
@@ -435,7 +426,7 @@
my @slots = $self->read_txn_slots;
my $found;
- for my $i ( 0 .. $#slots ) {
+ for my $i ( 0 .. $self->num_txns-2 ) {
next if $slots[$i];
$slots[$i] = 1;
Index: lib/DBM/Deep.pod
===================================================================
--- lib/DBM/Deep.pod (.../tags/libdbm-deep-perl/2.0000-1) (revision 62350)
+++ lib/DBM/Deep.pod (.../trunk/libdbm-deep-perl) (revision 62350)
@@ -4,7 +4,7 @@
=head1 VERSION
-2.0000
+2.0002
=head1 SYNOPSIS
Index: Changes
===================================================================
--- Changes (.../tags/libdbm-deep-perl/2.0000-1) (revision 62350)
+++ Changes (.../trunk/libdbm-deep-perl) (revision 62350)
@@ -1,5 +1,20 @@
Revision history for DBM::Deep (ordered by revision number).
+2.0002 Sep 5 12:35:00 2010 PDT
+ - Error messages from DBM::Deep now use the caller’s file name.
+ They used incorrectly to use the name of the program ($0).
+ - begin_work now checks correctly to see whether the new transac-
+ tion exceeds the number the file was created to support. Some-
+ times it would allow a few more transactions, and then proceed
+ to corrupt the database (RT#60903).
+ - The description of the file header in DBM::Deep::Internals has
+ been brought up to date.
+
+2.0001 Aug 22 12:03:00 2010 PDT
+ - Simply reading a hash or array element no longer causes autoviv-
+ ification. (Dereferencing it does still.) This makes DBM::Deep
+ comply with Perl’s behaviour (RT#60391).
+
2.0000 Jul 18 14:30:00 2010 PDT
- THIS VERSION IS NOT FULLY COMPATIBLE WITH 1.002x.
- This version is practically identical to the previous dev
Index: README
===================================================================
--- README (.../tags/libdbm-deep-perl/2.0000-1) (revision 62350)
+++ README (.../trunk/libdbm-deep-perl) (revision 62350)
@@ -3,7 +3,7 @@
transactions
VERSION
- 2.0000
+ 2.0002
SYNOPSIS
use DBM::Deep;
Reply to: