Memory usage too high for 32bit usage

[Katharsas]

So, i switched from ZenLib to ZenKit some months ago for my renderer.

Now in ZenLib, VDF/MOD archives were loaded with PhysFS, which apparently does not keep its archive files in memory and instead keeps only handles to the files inside the archives. When loading a single asset file, the correct archive is opened and the content is read, after which the archive is closed again, every time.

Now in ZenKit, all archives are permanently kept in memory. For grahics mods coming with somewhat big textures (like GRM, which has 1k/2k textures), this essentially multiplies the memory usage during load time by an order of magnitude or more. This makes ZenKit essentially unusable for me for a 32bit target binary. Now, the reason why i want to be able to run a 32bit binary is that im interested in keeping the possibility open of hooking my renderer into the original game (not really on the table yet but still).

In my renderer, i have a certain commit where i added ZenKit without removing ZenLib, that works with both libraries using a simple flag, so i can make an easy comparison of memory usage during loadtime between both libraries.

ZenRen 32bit+LAA, loading GRM WorldMesh, w ZenLib:

322 MB (Commit)

ZenRen 32bit+LAA, loading GRM Worldmesh, w ZenKit:

3512 MB (Commit)

So, the virtual memory space is 10 times bigger during load (close to half of that being memory mapped VDFs).
Note: 32 and 64bit binaries are not comparable under Windows, since for 64bit all VRAM allocation are part of commited memory space, which makes 64bit binaries always commit much more than 32bit for applications with lots of VRAM buffers.

I have to restrict my test to WorldMesh loading only, since loading with ZenKit actually crashes due to OOM (std::bad_alloc) if i try to also load VOBs. This is because the commited memory then reaches 3,8-3,9 GB, at which point any further substantial vector allocations fail, since 32bit+LAA applications have a max virtual memory space of 4 GB.

I have tried disabling ZK_ENABLE_MMAP, which does disable mmaped memory, but does not change the problem, because then all VDFs are just loaded as byte arrays, but they are still kept in memory, so commited memory basically does not change.

So my question is, would it be feasable to add a flag to ZenKit that keeps VDFs out of memory unless a file from that VDF must be actually loaded?
I could also try to implement this, but i am not very good at C++, so i might struggle with that.

[lmichaelis]

Hm, I agree that 4GB is an unreasonable amount of memory to use. It should be possible to add such a flag to the Vfs subsystem only, by making mount_disk store file paths, byte offsets and lengths and then selectively mmapping the file in the given range on a call to open_read.

This would require some rework of the Vfs subsystem by modifying mount_disk and mount_host and adding a new alternative to _m_data of VfsNode and using that when open_read is called. This might also require adding a new implementation of the Read interface which is capable of reading from a file at a given offset.

All in all, this does not sound like a huge amount of work but won't be trivial either.

[Katharsas]

The Win32 API differentiates between memory mapping the actual file and creating a view on it.

I assume that the "expensive" part is the actual mapping, and creating the view should be relatively cheap. Only when the view is created (and it can also cover only a section of the file) does it take up virtual memory address space.

So on windows, we could keep the file mapped but re-create a view for each access.

Now for Unix i have not idea what is possible. If i find time i could try to come up with a prototype, but it might take a while until i can look at this.

[lmichaelis]

Heyo! It's been a while, but I've finally had the time to implement a solution to this problem. The general idea is, then you disable ZenKit's memory mapping system entirely when building for 32-bit by setting ZK_ENABLE_MMAP to OFF. This will now cause the new system to take over, which represents files in VDFs using the path to the VDF file itself and an offset, size pair.

You you try it out to see if this solution works for you? It's in a new feat/reduced-memory-footprint branch.

[Katharsas]

Very cool i will give it a try soon, thanks!

[Katharsas]

The problem for me right now is that due to a bunch of reasons i rely on an older and slightly customized version of ZenKit. I guess it is time to try and upstream some of it and find a solution for everyting. I guess ill just open up some issues if you don't mind^^

Also if you like we can communicate directly (https://forum.worldofplayers.de/forum/members/109293-Kellendil), but here is fine also.

[lmichaelis]

The problem for me right now is that due to a bunch of reasons i rely on an older and slightly customized version of ZenKit. I guess it is time to try and upstream some of it and find a solution for everyting. I guess ill just open up some issues if you don't mind^^

Also if you like we can communicate directly (https://forum.worldofplayers.de/forum/members/109293-Kellendil), but here is fine also.

Feel free submit PRs if you have code you'd like to upstream, I appreciate it :) Please don't make it one huge PR though, rather split it up into multiple smaller ones. I've sent you a friend request on WoG for general questions.