Re: Bypassing the 2/3/4GB virtual memory space on 32-bit ports
On Tue, Aug 20, 2019 at 2:52 PM Sam Hartman <hartmans@debian.org> wrote:
> I think my concern about your approach is that you're trying to change
> how the entire world thinks.
that would be... how can i put it... an "incorrect" interpretation. i
think globally - i always have. i didn't start the NT Domains
Reverse-Engineering "because it would be fun", i did it because,
world-wide, i could see the harm that was being caused by the
polarisation between the Windows and UNIX worlds.
> You're trying to convince everyone to be
> conservative in how much (virtual) memory they use.
not quite: i'm inviting people to become *aware of the consequences*
of *not* being conservative in how much (virtual) memory they use...
when the consequences of their focus on the task that is "today" and
is "right now", with "my resources and my development machine" are
extended to a global scale.
whether people listen or not is up to them.
> Except I think that a lot of people actually only do need to care about
> 64-bit environments with reasonable memory. I think that will increase
> over time.
>
> I think that approaches that focus the cost of constrained environments
> onto places where we need constrained environments are actually better.
>
> There are cases where it's actually easier to write code assuming you
> have lots of virtual memory.
yes. a *lot* easier. LMDB for example simply will not work on files
that are larger than 4GB, because it uses shared-memory copy-on-write
B+-Trees (just like BTRFS).
...oops :)
> Human time is one of our most precious
> resources. It's reasonable for people to value their own time. Even
> when people are aware of the tradeoffs, they may genuinely decide that
> being able to write code faster and that is conceptually simpler is the
> right choice for them.
indeed. i do recognise this. one of the first tasks that i was given
at university was to write a Matrix Multiply function that could
(hypothetically) extend well beyond the size of virtual memory (let
alone physical memory).
"vast matrix multiply" is known to be such a hard problem that you
just... do... not... try it. you use a math library, and that's
really the end of the discussion!
there are several other computer science problems that fall into this
category. one of them is, ironically (given how the discussion
started) linking.
i really wish Dr Stallman's algorithms had not been ripped out of ld.
> And having a flat address space is often
> conceptually simpler than having what amounts to multiple types/levels
> of addressing. In this sense, having an on-disk record store/database
> and indexing that and having a library to access it is just a complex
> addressing mechanism.
>
> We see this trade off all over the place as memory mapped databases
> compete
... such as LMDB...
> with more complex relational databases which compete with nosql
> databases which compete with sharded cloud databases that are spread
> across thousands of nodes. There are trade offs involving complexity of
> code, time to write code, latency, overall throughput, consistency, etc.
>
> How much effort we go to support 32-bit architectures as our datasets
> (and building is just another dataset) grow is just the same trade offs
> in miniture. And choosing to write code quickly is often the best
> answer. It gets us code after all.
indeed.
i do get it - i did say. i'm aware that software libre developers
aren't paid, so it's extremely challenging to expect any change - at
all. they're certainly not paid by the manufacturers of the hardware
that their software actually *runs* on.
i just... it's frustrating for me to think ahead, projecting where
things are going (which i do all the time), and see the train wreck
that has a high probability of occurring.
l.
Reply to: