[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Summary: File locking discussion

Hi folks!

I'm glad that so many people are finally involved in our "file locking
discussion". However, we had lots of discussions about this topic in the
past and we have no results, yet. 

That's why I want to try to bring more "structure" into the discussion.
(Please complain if you don't agree :-) And someone else mentioned that we
should write a little assey about the possible decision we'll draw soon,
so that newcomers are informed why we have chosen a specific locking

Thus, I started to write such a document. It's attached below. Please
note, that this is release 0.00001 of the document so feel free to tell me
_any_ comments/critics/suggestions.

I'll try to update this document often and introduce all new arguments
that will show up here. Perhaps we can come to a result this time, and
have the documentation ready when we have a decision.

(Please do not quote the whole document if you are referring to a single
section to reduce the traffic on this list.)


File Locking in the Debian system

Table of Contents

0. References
1. Current situation
2. Locking mechanisms
3. Debian specific ideas
4. Open issues
5. Frequently Asked Questions

Chapter 0: References

- The open(2) describes "dot-locking" and a NFS safe locking
- http://www.swb.de/personal/okir/lockd.html explains rpc.lockd
- several discussions on debian-devel (starting from Mar 97)
- Lars' "publib" library (see publib Debian package)
- Karl's "LockFile.pm"--a Perl implementation of publib's lockfile
  (see http://inetarena.com/~karlheg/Public)

Chapter 1: Current situation

1.1 Debian Policy Manual

The current Debian Poliy Manual ( only mentions file locking
for MUAs (mail-user-agents) and MTAs (mail-transport-agents). In
section 4.3 "Mail processing on Debian systems" it says:

      Mailboxes are locked using the username.lock lockfile
      convention, rather than fcntl, flock or lockf.

Note, that this is obviously not enough information for the
maintainers to adopt their packages. This results in incompatible
locking mechanism. Thus, sometimes problem occur on Debian systems
with mail locking when using NFS.

1.2 Where do we need locking?

We need file locking in several places. For example, the incoming mail
boxes in /var/spool/mail/*, mail folders that are accessed by
"procmail" and "pine", etc.

Chapter 2: Locking mechanisms

2.1 fcntl, flock, lockf

The Linux kernel provides some system calls to do file
locking. However, neither of these functions is working over NFS.

2.2 dot-locking

To perform dot-locking, create the file <mailbox-name>.lock using
open(name, O_CREAT|O_EXCL, 0644). Create with O_EXCL will fail if the
file exists. If you can't create it, sleep for a while and try again.
To unlock, unlink() the file.

This technique is not NFS safe, since the NFS "open" function is not
atomic. The man page open(2) reads:

       O_EXCL     When used with O_CREAT, if the file already  exists
                  it  is  an error and the open will fail.  O_EXCL is
                  broken on NFS file systems, programs which rely  on
                  it for performing locking tasks will contain a race

2.3 rcp.lockd

Please refer to http://www.swb.de/personal/okir/lockd.html for a
more detailed description. 

The idea is simple: Each "server host" runs a rcp.lockd daemon. When a
process wants to lock a file, it contacts that daemon and asks for
locking the file. 

This technique is obviously NFS safe. However, there is no Linux
implementation of rpc.lockd, yet.

2.4 SUN's lockd

(Is this the same as "rpc.lockd"?)

Rumors say, that "lockd" can cause dead-locks when unlocking a file.
And there is no Linux port of this daemon, yet.

2.5 NFS safe locking through "link"

The man page open(2) reads:

       The solution for performing atomic file
       locking using a lockfile is to create a unique file
       on the same fs (e.g.,  incorporating  hostname  and
       pid),  use  link(2)  to make a link to the lockfile
       and use stat(2) on the unique file to check if  its
       link  count  has  increased  to  2.  Do not use the
       return value of the link() call.

This method is currently implemented in Lars' "publib" and has been
ported to a Perl module by Karl. 

There is no documentation available for "publib" yet, but this could
easily be fixed :-)

2.6 Other implementations

A few packages install binaries that handle the locking of files:

   `lockfile' --> procmail
   `shlock'   --> innd
   `newslock' --> mgetty-fax

If we should choose supporting one of these locking mechanisms, we
would have to change these binaries into shared libraries, for

Chapter 3: Debian specific ideas

3.1 Creating a library

When we have chosen a good locking mechanism we should implement it in
a shared library so that the maintainers do not have to hand-code this
themselves. (Since these techniques are a bit "hairy" that would
likely introduce new bugs.)

We'll also provide "modules" for Perl, Python, and other popular
programming languages.

In addition, we'll provide a "binary" that can be called by shell
scripts or other languages to lock/unlock files.

3.2 Creating an upstream library

Since there is no standard way of file locking yet, we should try to
build a new "upstream library". That is, we don't want to create a
Debian specific library, but one that can be used by the other
distributions as well. Perhaps this library can become the new
"standard" in the future.

Perhaps, we could merge this into the new "glibc 2.0". That way, it
will surely become the new standard!

3.3 Policy Manual

The Policy Manual should document the "Right Way of File Locking" in

In addition, there will be a short manual (actually, this document!)
describing the different locking techniques that were discussed.

Chapter 4: Open issues

Who releases a lock when the process, who locked the file, has died?

Chapter 5: Frequently Asked Questions

Q: What's wrong with using dot-locking in conjunction with

A: Provided that everybody enables the two locks in the same order and
   disables them in the opposite order, there is no problem in using
   two different locking mechanisms. However, we wouldn't get any
   advantages from using two mechanisms at the same time.

Q: Why can't we simply fix the Linux kernel to support NFS safe

A: A lot of people use Debian in a network with other UNICES. As such,
   we'll have to choose a locking mechanism which is implemented at
   "application level" so that the involved programs (mail daemons,
   mail clients, etc.) on other platforms can be fixed to apply to our
   mechanism, as well.


--                 Christian Schwarz
Do you know         schwarz@monet.m.isar.de, schwarz@schwarz-online.com,
Debian GNU/Linux?    schwarz@debian.org, schwarz@mathematik.tu-muenchen.de
Visit                  PGP-fp: 8F 61 EB 6D CF 23 CA D7  34 05 14 5C C8 DC 22 BA
http://www.debian.org   http://fatman.mathematik.tu-muenchen.de/~schwarz/

TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word "unsubscribe" to
debian-devel-request@lists.debian.org . 
Trouble?  e-mail to templin@bucknell.edu .

Reply to: