Linux.

Guys, I need some inspiration. How are you dealing with memory leaks, i. .e identifying which component of the system is leaking memory?

Regular method of dumping ps aux sorted by virtual memory usage is not working as all the processes are using the same amount of memory all the time. This is XEN dom0 memory leak, and I have no more ideas what to do.
Is it possible that guests could be eating the dom0 memory?

Just curious.. Why are you sorting by vsz rather than rss?

@netikras made an typo, RSS is the column which is being sorted, however neither RSS neither VSZ does not show any increase in memory usage.

The only commands which shows shrinking free memory are free and vmstat and any other one.

Valgrind?

@con-fig I would use it if I would know what is leaking that :) furthermore, valgrind requires processes to be started using it as a wrapper as far as I recall.

Will try memleax

@gitversion sorry, didn't realise the scale! My bad.

Are you using gnome? It has a known memory leak that has not been solved yet if I remember well. You can see that by minimizing a maximizing a window repeatedly.

@pwar no, that's an oracle vm server 3.4.5

What's the total count of processes? Does the sum(rss) match output of free? Are you root?

@netikras there are 449 processes, their RSS sums up to 436.421725MB and free shows 3050MB as used.

6047MB are allocated to xen hypervisor out of 261741MB and nothing in the hypervisor itself are using that memory, if that would've been regular linux box it wouldn't take so much time to identify what's wrong. I'll keep digging, unless you have any kind of insights.

Be to, ačiū!

@gitversion oo, kolega :)

and you are on host os, right? No other hypervisor below? Are you measuring stats of physical server?

I know I've seen this exact mismatch dozens of times on vmware VMs - all bcz esx used balooning for ram and all the overcommited memory pool was exhausted. I.E. If esx has 1tb ram, assigns 1.5 tb ram to VMs [with .5 tb overcommit] and all the vms in total actually consume 999gb. This effectively drains all the memory out of all the hosts while some of them could be idling. Free shows all ram is used but ps aux cannot indicate any hi rss usage

but if you are measuring stats on physical server this should not apply

@netikras that's an physical server. I've tried to compare every possible xen hypervisor configuration with the lower version (VM Server 3.4.4) on PROD environment which does not experience this issue, and all of them seems to be the same. Well, I suppose will have to dig deeper into google. Once again, ačiū!

Any warnings in dmesg? Syslog?

Sooo.. The physical machine is low on ram but ps aux as root does not show any hoggers. Damn that's interesting. Could it be some kernel module maybe? I recall once I had a leak on one of veritas vx kernel modules

if you find a culprit, please, share your findings :)

@netikras Physical machine is not low on RAM. Xen Hypervisor is limited to 6GB of RAM and it's running low on that. All other memory is free to be allocated to the guest machines.

Will surely share any findings.

Can't help so i'll just joke about it ... Chrome ... Still ... Good luck on finding it

Top -> ctrl+m? (or was it shift....)