Memory Leaks or Mem. being misreported

[nito@dismail.de]

Hi everyone,

after upgrading my hardware I started to notice what seemed like memory leaks. 
After booting my desktop machine usually around 450MiB(with i3) or 
650MiB(with MATE) of RAM are being used and after extended use, when closing all
gui-programs except a terminal, based on prior experience I would expect the 
RAM usage to not be more than +120MiB compared to after booting. I am *not* 
counting file cache/buffers in this (or I severely misunderstand how to read
free's and top's output).
But now I would often end up with something between +600MiB and +2.5GiB.
When looking at top or htop no process using nearly that much memory is listed.


I tried to figure this out and will give some more details in a minute, but I'm
left confused as to why this is happening how to continue debugging this and
where I could file a bug report.
Any advice on how to proceed or where to file a bug would be appreciated.
Or maybe I'm just stupid and misreading top's and free's output.

Unfortunately the only reliable way to reproduce this, that I found, involves
manual interaction with some Firefox profiles. I know this means there's
probably an issue with Firefox, but afaik when a user-space process with a
memleak is closed, the leaked memory should be cleaned up as well. My failed
attempt to recreate the leaks with a simple C program seems to support this.
Also this seems to be hardware specific.

Affected are:
  Debian 10 with kernels:
    4.19     (from main)
    5.4.0~bpo.3   5.4.0~bpo.4   (from  backports)
    5.4.23 with Manjaro patches and config
    5.4.26 (vanilla; config based on Debian's config)
    5.6-rc7 (vanilla; config based on Debian's config)

  Manjaro with kernel:
    5.4.23-MANJARO

  Ubuntu-18.04 with kernels:
    5.3
    5.4.xy (vanilla)


In Manjaro and Ubuntu proprietary firmware was used.
In Debian I was first only using the proprietary AMD-GPU firmware but later also
added firmware for the Ethernet-controller and microcode for the CPU without any
change.
dmesg does not seem to show anything relevant*. 
(*or atleast not anymore since cpu-bootcode is installed)

(The following was only tested with kernel 5.6-rc7):
Now yesterday on accident I noticed, that after running a backup, which "filled"
RAM with file cache, the reported memory fell again to a normal level.

With i3:
--After Boot:
$ free -h
              total        used        free      shared  buff/cache   available
Mem:           31Gi       397Mi        30Gi        11Mi       314Mi        30Gi
Swap:         6,0Gi          0B       6,0Gi

--After using and closing Firefox
--(no processes using that much RAM are listed in htop):
$ free -h
              total        used        free      shared  buff/cache   available
Mem:           31Gi       1,1Gi       2,4Gi        13Mi        27Gi        29Gi
Swap:         6,0Gi          0B       6,0Gi


--After Backing up which "filled" RAM with file caches:
$ free -h
              total        used        free      shared  buff/cache   available
Mem:           31Gi       471Mi       328Mi        13Mi        30Gi        30Gi
Swap:         6,0Gi          0B       6,0Gi

The reported memory usage of htop,top and free was all in accordance with each
 other here.

Now as I couldn't find a way to reliably "fill" up my RAM with file caches, I 
tried to recreate this (with MATE) with a C program that allocates memory, 
writes pseudo-random data to it and exits (without freeing). I allocated 30GiB,
so more than MAX-used, but less than MAX-(expected usage). 
After that (h)top and free are disagreeing on how much memory is being used
(see SWAP).
--After Firefox (htop,top and free report more or less the same):
$ free -h
              total        used        free      shared  buff/cache   available
Mem:           31Gi       1,5Gi        27Gi        92Mi       2,7Gi        29Gi
Swap:         6,0Gi          0B       6,0Gi

-- After allocating 30GiB and then exiting:
$ top -b -n 1 | grep "^MiB"
MiB Mem :  32092,0 total,  27747,1 free,    744,3 used,   3600,6 buff/cache
MiB Swap:   6144,0 total,   6143,5 free,      0,5 used.  30458,0 avail Mem
$ free -h
              total        used        free      shared  buff/cache   available
Mem:           31Gi       743Mi        27Gi        93Mi       3,5Gi        29Gi
Swap:         6,0Gi       0,0Ki       6,0Gi

htop was reporting around 500MiB swap usage (before and also after running the
above two commands).


So maybe this is not a memory leak but memory usage is being misreported ?
Where should I file a bug for this; or am I just misreading (h)top's and free's
output and everything is as expected ?
free's manpage states:
>  used   Used memory (calculated as total - free - buffers - cache)
which I take as this being the memory actually in use, that I could not just
allocated in a process without needing to fallback to swap.

Sorry for this wall of text.

Kind Regards
Nils