Re: atomically concurrent writes question
On Wed, Jun 23, 2010 at 01:19:01AM -0500, Ron Johnson wrote:
> On 06/23/2010 12:51 AM, Evuraan wrote:
>> I've a script which forks and echo's [appends] multiple long lines to
>> one plain text file. some of the multiple threads also yank lines
>> from the same plain text file. this file is now approx 31M and has
>> about 105000 lines ( growing.) i've been doing this for years now,
>> and have seen nothing go wrong so far. (yay!)
>>
>>
>> my question is rather theoretical - what can go wrong if multiple
>> processes are writing to the same file at multiple instances?
>>
>> i've read elsewhere, that the kernel takes care of this stuff. or is
>> it that i could just not cause finitely concurrent writes to the file
>> and have not seen a corrupt line just by luck?
>>
>
> Not only the kernel, but *bash* (or whatever other lesser shell you
> use...) is what you'd need to look at to see what it does when it sees
> ">>".
It's using the O_APPEND flag on the writes, which guarantees that each
write(2) system call operates atomically. That is, the data on a given
write will be added to the end of the file, and if two or more processes
are writing to the file at the same time, the results will be consistent
with some ordering. (Data will not be mixed or corrupted between writes.)
It's important to make sure that each line does fit in the same call to
write(), though. I'm not sure what's meant by "long lines": If they're
too big to process all at the same time, the guarantees for a single
call to write() no longer apply. But the shell is almost certainly doing
the right thing. (In C code, I'd check it very carefully.)
Jon Leonard
Reply to: