Increasing memory use over time

[stm]

I am using v6.0.6 and I've noticed that over time the memory use of MezzmoMediaServer.exe process just keeps growing. Once I see it nearing 2GB (as it appears to bit a 32bit non LAA exe) I restart the Mezzmo server and it goes back to under 100MB and the cycle repeats over a number of days/weeks.

This installation is almost never used as I've moved my main media type which is photos to IMatch photo web server. This only leaves a few rarely viewed videos and a small amount of music which even when not accessed for weeks on end makes no difference to this behaviour. I don't know if this is a memory leak or whatever memory management it uses just suffering from increased memory fragmentation or not releasing memory back to the OS.

I've not had time to check for more detailed windows resource use yet but is this process expected to max out its process space? As I run other key services on the machine in question I haven't left it longer to see what happens, and I could schedule a service restart every night but that isn't ideal.

Any thoughts?

[Peter]

Mezzmo media server has a number of worker threads that will update the library in the background, when opening files Windows will indicate the memory size of the process including the size of the opened file so possibly Mezzmo server is reading from 2Gb files at this time?

[stm]

Mezzmo media server has a number of worker threads that will update the library in the background, when opening files Windows will indicate the memory size of the process including the size of the opened file so possibly Mezzmo server is reading from 2Gb files at this time?

Just before I last restarted the MezzmoMediaServer.exe process I checked with Process Monitor and Process Explorer and although the process had a number of files open (mostly its database), no file I/O was taking place. The size of all the Mezzmo database files in the root folder combined is just 12MB. The items I have left in Mezzmo are just 200 very small videos (short exported clips of notable events from my home CCTV) and around 3000 mp3 tracks.

When the process size grows again, is there anything useful I can check before restarting the process?

[Peter]

You could create a dump of the process from Windows task manager so that we could try to see where the memory is allocated? Go to task manager and find MezzmoMediaServer.exe in the list and right-click and select to create a dump file. Then zip it up and send it to us.

[stm]

You could create a dump of the process from Windows task manager so that we could try to see where the memory is allocated? Go to task manager and find MezzmoMediaServer.exe in the list and right-click and select to create a dump file. Then zip it up and send it to us.

While I am waiting for the MezzmoMediaServer.exe process to hit nearly 2GB again for a process dump (currently just over 1GB) I've been watching some process resource usage and it has over 5500 file handles open.

Although that's possible for some applications with many data/resource/dll files, what's odd here is there are well over 600 handles for each of the files: Mezzmo.db, Mezzmo.dm-wal, MezzmoPluginMetadata.db, MezzmoPluginMetadata.db-wal, MezzmoStreamingStatus.db, MezzmoStreamingStatus.db-wal, MezzmoTranscode.db. This makes it look like its failing to close the files and release the handles so that could be one issue of increasing memory use.

I'll keep monitoring for anything else that looks suspicious as its memory use grows.

[Peter]

MezzmoMediaServer.exe opens multiple connections to the database and keeps them open from different parts of the server so it is not unusual to have multiple connections open. How long do you have MezzmoMediaServer.exe running before the memory usage increases?

[stm]

MezzmoMediaServer.exe opens multiple connections to the database and keeps them open from different parts of the server so it is not unusual to have multiple connections open. How long do you have MezzmoMediaServer.exe running before the memory usage increases?

I still think 600 file handles per file and growing is excessive. It's only using a bit over 30+ threads.

The current instance has been up 9 days and since my last post another 290 handles in total are now permanently open by that process (so 5790 total) so that's now over 650 for each of the files I listed earlier. It's using 1.1GB memory and still rising.

[stm]

Just an update that memory use is still climbing as are handles for Mezzmo.db, Mezzmo.dm-wal, MezzmoPluginMetadata.db, MezzmoPluginMetadata.db-wal, MezzmoStreamingStatus.db, MezzmoStreamingStatus.db-wal and MezzmoTranscode.db.

Each of those files now has over file 760 handles open on them, and total process memory is 1.3GB. Since the process was started it hasn't served any content so that's no a factor.

Some current process details:
1312

[Peter]

Are you running the latest version of Mezzmo or an older version? We did have handle leaks in the past but have fixed them since then. We will try running the server for an extended period to see if any leaks occur.

[Peter]

I have had MezzmoMediaServer.exe running for several hours now. The memory use is sitting on 43, 476 K and the handles is 2,931 (2,804 was the value when I first checked). Threads is 49, when I first checked it was 47 so there are two new worker threads running. Memory use dropped to 15, 620 K while I was typing this so maintenance or another process just ended.

[stm]

Are you running the latest version of Mezzmo or an older version? We did have handle leaks in the past but have fixed them since then. We will try running the server for an extended period to see if any leaks occur.

As per the opening post I am on v6.0.6 and it shows no updates are available.

Maybe there is something specific to my setup (Mezzmo settings or environment), but its now a very rarely used install as per my earlier posts. In Process Monitor I don't see any odd failures with file access so permissions all seem good (same for process/registry activity) and the installation has been in place for almost two years, just upgraded when applicable.

This could have been happening for some time as the server often reboots every 10 - 14 days for updates so that would mask the problem. MezzmoMediaServer.exe appears to 32bit and not be large address aware so I expect it to have issues as it nears 2GB which will probably occur in another 3 to 5 days if I don't have any reboots for system updates.

If there's anything else I can capture that would help locate the issue then please let me know. I've taken a couple of full process dumps, but I know from experience they're not much fun to debug with.

[Peter]

Sorry I missed seeing the version information in the opening post. Can you please check the interfaces Mezzmo server is listening on in 'Media Server Settings' as the server could be processing a lot of Upnp messages from virtual adapters which may be eating up memory. Try setting the adapters to just the LAN adapter to see if this helps.

[stm]

Sorry I missed seeing the version information in the opening post. Can you please check the interfaces Mezzmo server is listening on in 'Media Server Settings' as the server could be processing a lot of Upnp messages from virtual adapters which may be eating up memory. Try setting the adapters to just the LAN adapter to see if this helps.

It already is bound to the main LAN adapter (an Intel NIC team)

Today the MezzmoMediaServer.exe process is over 1.6GB with 7600 file handles (just more of the same Mezzmo DB files listed ealier making up the ongoing increase). The excessive file handles may not be the cause of all the memory use but its certainly not correct and surely that means some area of code is not releasing them?

I've done a fresh v6.0.6 install in a VM and diff'd that to my current install in case the upgrader had an issue at some point but all files are correct, just with some surplus older versions of FFmpeg dlls that I've checked aren't loaded into the process.

[stm]

As I've already captured a lot of information about this leaking MezzmoMediaServer.exe process I restarted it once it got to 1.86GB memory and 8240 file handles with the only ones that increased relating to the DB files mentioned before. Leaving it would only lead to it locking up/crashing as it exhausted its 2GB memory space.

I am disappointed that this issue doesn't appear to be of interest to the developers despite the hours effort I've gone to as per the posts above (I am a server side software dev). That leaves me with restarting the Mezzmo service every night as a workaround or just going elsewhere for music streaming.

Incidentally the product I switched to a while back for photos and videos runs on the same server and is worked a lot harder yet has stable memory and other Windows resource types. If there was any issue on the server itself then that application would be very likely to have issues as well, but it doesn't, and that's the same for the other services running.

Quick summary before I restarted the Mezzmo service:
1313

[odin]

As I've already captured a lot of information about this leaking MezzmoMediaServer.exe process I restarted it once it got to 1.86GB memory and 8240 file handles with the only ones that increased relating to the DB files mentioned before. Leaving it would only lead to it locking up/crashing as it exhausted its 2GB memory space.

I am disappointed that this issue doesn't appear to be of interest to the developers despite the hours effort I've gone to as per the posts above (I am a server side software dev). That leaves me with restarting the Mezzmo service every night as a workaround or just going elsewhere for music streaming.

Incidentally the product I switched to a while back for photos and videos runs on the same server and is worked a lot harder yet has stable memory and other Windows resource types. If there was any issue on the server itself then that application would be very likely to have issues as well, but it doesn't, and that's the same for the other services running.

Quick summary before I restarted the Mezzmo service:
1313

Could it be possible you have some database corruption have you tried starting fresh with a clean db or at least trying to run the current database through sqlite3 ?
ie ("sqlite3 Mezzmo.db ".dump" | sqlite3 MezzmoNew.db") then move MezzmoNew.db to Mezzmo.db restart mezzmo and see if it fixed.

Make sure you keep your old databbase files so you can revert back if nothing changes.

anyways just something to try since it fixed a problem I had in the past with memory leak and mezzmo becoming unstable.

[jbinkley60]

A similar option would be to run a Mezzmo database backup and then run a full Mezzmo maintenance on the database.


Jeff

[stm]

Could it be possible you have some database corruption have you tried starting fresh with a clean db or at least trying to run the current database through sqlite3 ?
ie ("sqlite3 Mezzmo.db ".dump" | sqlite3 MezzmoNew.db") then move MezzmoNew.db to Mezzmo.db restart mezzmo and see if it fixed.

Make sure you keep your old databbase files so you can revert back if nothing changes.

anyways just something to try since it fixed a problem I had in the past with memory leak and mezzmo becoming unstable.

Thanks, but I've already recreated a fresh database entirely I first spotted this issue when I removed all my photos from Mezzmo and the related complex playlists.

The leaking file handles also applies to four database files equally and three are basically tiny/empty. That makes it more likely a leak in Mezzmo code than whatever sqlite library they're using or corruption, unless they're not handling a DB related exception somewhere in code that opens these regularly.

The server also has had all anti-virus scanning disabled for the Mezzmo files and processes ever since install. I could recreate the setup in a VM to rule our other settings/file related issues as its a 21 month old install but I've already spent quite a bit of time and some steps to narrow down causes from the authors (any specific logs to check etc) would be helpful. Given the specific handle leak pattern they could at least check the code for likely areas/causes instead of just saying Mezzmo opens lots of files as has been stated several times in this thread.

[odin]

Thanks, but I've already recreated a fresh database entirely I first spotted this issue when I removed all my photos from Mezzmo and the related complex playlists.

The leaking file handles also applies to four database files equally and three are basically tiny/empty. That makes it more likely a leak in Mezzmo code than whatever sqlite library they're using or corruption, unless they're not handling a DB related exception somewhere in code that opens these regularly.

The server also has had all anti-virus scanning disabled for the Mezzmo files and processes ever since install. I could recreate the setup in a VM to rule our other settings/file related issues as its a 21 month old install but I've already spent quite a bit of time and some steps to narrow down causes from the authors (any specific logs to check etc) would be helpful. Given the specific handle leak pattern they could at least check the code for likely areas/causes instead of just saying Mezzmo opens lots of files as has been stated several times in this thread.

Hm not sure, here is what mine is using , around 5,222 handles but I don't really have any memory issues, my server is mostly movies, and pictures (no audio/music).
1314

[Peter]

We do have interest in resolving this issue. We have not yet received a dump file of the MezzmoMediaServer.exe process when the memory use is high so if you can try to create this next time the memory use is high and send it to us we can investigate the memory allocation. Also checking for database issues may be useful so if you can enable diagnostic logging http://www.mezzmo.com/wiki/doku.php?id=using_logging&s[]=diagnostic and then after restarting your Mezzmo media server let the server run until the memory use gets high then zip the logs and send them to support [at] conceiva [dot] com