FIFE forums

Please login or register.

Login with username, password and session length
Advanced search  

News:

FIFE 0.4.0 has been released on 15th of January, 2017!

Author Topic: FIFE RAM usage - animation loading  (Read 3081 times)

chewie

  • Developer
  • Full Member
  • *
  • Posts: 123
    • View Profile
    • zero-projekt.net
FIFE RAM usage - animation loading
« on: August 13, 2008, 10:44:51 am »

Moin @all :)

We (Zero) discovered that FIFE eats a lot of RAM when loading multiple maps. (we are talking about ending with 1.5 GB RAM after 3-4 maps). After some testing we discovered the cause for this behaviour.

Major issues are:
 - pool has no cleanup method (spq told me that this is already known)
 - loading huge xml files for agents

Some very simple "benchmarks" had this result:
 - Loading a map with one tile, two trees, the player (13 defined actions, ~2140 png files) and one zombie (6 defined actions, ~600 png files) = ~500 MB RAM usage
 - Loading the same map without the zombie = ~240 MB RAM usage
 - Loading the same map without player & zombie = ~ 60 MB RAM usage

Loading time steps are somehow equal (~ 30, ~15, ~2 seconds).

(my machine: 2.8 Ghz DualCore, 2 GB RAM, Nvidia 8600 GT, Ubuntu 7.10)

As the results may of course vary when using different systems, one thing is very strange:

The zombie + player file size is ~15 MB. Why does that lead to a RAM usage of ~440 MB for these two instances (only)?

We had a little discussion about this at the fife channel:

Quote
<chew-ie> barra, do you happen to know who created / worked on the pools of FIFE? (imagepool, animationpool)
<barra> AFAIR jasoka wrote at least parts of the code
<barra> svn blame could shed some light
<chew-ie> Okay, that's good :) So the person is at least still @FIFE :)
<barra> yep, always hard to get in contact with the people who don't work on the engine anymore
<GreyGhostNew> chew-ie, resource freeing?
<chew-ie> yes... I discovered that rio de hola only needs 160 MB RAM - whereas any map of zero eats ~500 :( (per load)
<GreyGhostNew> :(
<spq> how big are all the imagesyou load?
<chew-ie> that's variable. I set up a testmap with "real" tiles - and they have a size 629x315
<chew-ie> Our trees are huge images, too
<spq> hm
<spq> could you load the map in rio de hola?
<chew-ie> But strange enough - this tiny little map eats the same amount of ram -> http://zero.vaultnet.ath.cx/pictures/1024x_zero_13_08_2008-13_44_04.png
<chew-ie> my only guess is that the animationsets of the player & zombie can cause this ...
<chew-ie> The framework without loaded map only needs 50 MB RAM - with all the gui loaded
<spq> how big in mb are all your images you load in a game?
<chew-ie> In this little map.. wait a second
<chew-ie> about 20 MB - whereas the player character animationset has ~ 13 MB
<chew-ie> holy shit...
<MuteX_> chew-ie, what happened?
<chew-ie> spq, I deactivated the player and the zombie on the little map - and the map is loaded within seconds - RAM usage has little difference to framework startup (~ 60 MB)
<chew-ie> So the player and the zombie causes a RAM usage of ~ 440 MB ...
<GreyGhostNew> chew-ie, you mean .. 1 player and 1 zombie?
<chew-ie> yes ...
<barra> hmm please paste the map markup chew-ie
<GreyGhostNew> O_o ..
<chew-ie> ~ 15 MB (player + zombie animationsets) end up in 440 MB used by FIFE...
<barra> it might be possible that you use batch loading to load ALL agents
<chew-ie> barra, no - I only load object files of zombie and player
<GreyGhostNew> somethings looping in the loader maybe?
<spq> how much images are there in the animations?
<chew-ie> Why doesn't that affect rio de hola then? There are animations loaded, too.
<chew-ie> spq, depends ... a typical idle animation has 20 frames for each direction - whereas reloading has 60 frames.
<chew-ie> I've to correct myself: I am batchloading ->
<chew-ie> "<import dir="../objects/agents/malehuntingclothes001"></import>
<barra> hmm try to just load the animations that are actually needed
<chew-ie> But I'd say that's equivalent to import file (because there is only one object,xml to load)
<barra> I think there should be a feature in the editor tool to " finalize" a map
<barra> could you please paste the map markup chew-ie ?
<barra> or tell me which map is causing the issue, I got a local checkout here
<chew-ie> barra, it's not only a map ...
<barra> so free choice for me?
<chew-ie> I changed import from dir to file now
<chew-ie> Let me explain:
<chew-ie> A map without any agent animationsets loads fast and doesn't cause high RAM usage.
<chew-ie> But as soon as I introduce an agent, the RAM usage is awfully high.
<chew-ie> This is a major showstopper - because these are the critical components of the game :(
<barra> I agree on that one, hopefully somebody could take a look into it, as SVN access is given
<chew-ie> The funny thing is : these animationsets are far from being finished - they are lacking dozens of actions / equipment etc. I don't want to imagine what happens to the RAM if we have 20 different weapons * 10 different armors... -.-
<chew-ie> (so far we have 2 weapons and no armor)
<barra> I would propose to either try to catch a developer here and convince him to look into it or to post about it at the forums
<chew-ie> spq, are you loading animations on demand @openanno?
<dauerflucher> chew-ie, did you and the other zero devs already agree about graphical limitations?
<chew-ie> dauerflucher, what exactly do you mean with that?
<dauerflucher> somethign like all rifles use the same animation
<chew-ie> sure
<chew-ie> The first idea was to provide animations for all weapons - but we dropped that very soon.
<dauerflucher> a good decision :)
<chew-ie> So we now have the "Fallout method"
<barra> yep, I think the weapon class approach used in the old RPG classics of the 90s is the way to go for prerendered art
<chew-ie> yip
<dauerflucher> i think you still _might_ handle a lot of stuff using overlays
<barra> yep, but I think an overlay concept might be complicated
<dauerflucher> but it's still a shitload of work
<dauerflucher> it's sad but true :)
<chew-ie> yep - and you get very generic animations
<barra> that's another drawback of it, yep
<chew-ie> because all movements have to be synced against each other *brr*
<chew-ie> I'd say agents should load the needed animations due to their current equipment - so no huge xml file with all animations, but small ones with e. g. pistol:leatherarmor etc...
<dauerflucher> that's the only way to get stuff done at some time...
<chew-ie> of course this needs also a way to clear animationpool from unused animations
<dauerflucher> chew-ie, wether OA already got something beiong of use for you or it will be there at some time... spq is taking care for a lot of interesting code details others might not have thought about
<chew-ie> dauerflucher :)
<chew-ie> sure, but I guess this one is connected to the xml datasets - which oa doesn't use (I start to understand, why ;-) )
<chew-ie> I'll talk with helios & jasoka - maybe we can develop a solution for zero and hand it back to fife :)
<dauerflucher> maybe... i mean just maybe... you should think about using databases as well :)
<dauerflucher> i can't tell about long time effects, but for now it works pretty fine
<chew-ie> I'm not so convinced yet  - openanno has very low fps on my system here ... (~ 19; zero has 400 on zombie playground)
<chew-ie> I don't know if that's connected to the db approach - but every method has some drawbacks I'd say...
<dauerflucher> hm, dunno if this is cause by the DBs, but you're right... there's stuff to be cleaned out first
<dauerflucher> might as well be the fucking small tile size we use :S
<chew-ie> ^^
<chew-ie> I did some tests in the meanwhile - loading the player agent with 3 actions only (idle, walk, run = 246 files) causes a RAM usage of 80 on the small map. Adding the zombie with 6 Actions (idle, walk, run, hit, dead, death = 642 files) causes a RAM usage of ~240 MB.
<kaelis> insane
<chew-ie> So if we talk about 3 different animationsets for humans and some critters (zombie, boar) you can get past 1GB with only using "default" animations.
<chew-ie> kaelis, indeed ...
<chew-ie> a map with 3 different visuals of humans and 2-3 types of critter is what I'd call the minimum (for a small map)
<kaelis> indeed!
<kaelis> that sounds like.. massive ram reqs

I'm still somehow puzzled about this - as it also is a kind of a real showstopper for using FIFE in a playable demo (at least with the size we planned it). My first idea about this is to develop and introduce a system that allows the agents to load their animations due to their equipment.

So at map startup all agents only "know" the needed actions like idle, walk and run. If an npc or the pc uses a weapon or an item, he loads the proper animations - (and perhaps deletes the old ones).

Still, the amount of needed RAM in relation with the original file size on the HD seems at least to me way too high.

Any thoughts, comments...?


EDIT

Proposal / statement by spq:

Quote
<spq> anyway chew-ie, the problem is not that the xml file is too long
<spq> thats just a definition
<spq> and that you have so much animations is no problem also
<spq> the thing missing is that images (only images, no animations, animations are just definitions of which image is displayed at which time) can be unloaded and how this can be done automatically
<spq> one could either regulary run a garbage collector or have a good policy when something is to be removed from cache
<spq> an easy but probably resource consuming method would be tagging the resources in the pool, at a given time you mark all resources as not used, then every time a resource is used, you mark the resource as used, and at any time later you unload every image that is not used
<spq> if this behaviour could be controlled by the game that would be really good
<spq> you could free the whole memory by calling those two functions directly, or you could collect needed images and some seconds/frames later remove the unneeded ones from the cache
« Last Edit: August 13, 2008, 03:34:24 pm by chewie »
Logged

phoku

  • Developer
  • Full Member
  • *
  • Posts: 102
    • View Profile
    • IZ dev blog
Re: FIFE RAM usage - animation loading
« Reply #1 on: October 10, 2008, 03:51:20 am »

Heya guys, here's an update for you  ;D

First of all the Pool classes had a combination of bugs which would cause
the addResourceFromXYZ functions to add resources although they were already
loaded. This bug has been ironed out as of revision 2613.

Now the getIndex function works exactly as the addResourceFromXYZ functions,
and is now deprecated.

So any major memory leaks are not the pool's fault anymore. Still the loaded resources
will not get purged. I would like to approach that by adding a "usage" property to all
resources and then collect unused resources after some time.

Note that this implies that Image/Animation pointers can be deleted anytime, unless you
properly call addRef and decRef in the using classes. So if and when this change comes
about misbeheaviour in this regard will be punished with segfault.  ::)

-phoku

Logged

vtchill

  • Developer
  • Full Member
  • *
  • Posts: 206
    • View Profile
Re: FIFE RAM usage - animation loading
« Reply #2 on: October 10, 2008, 05:43:22 pm »

so are you proposing a garbage collected approach, where the collector runs at a set interval?

This could potentially lead to non-deterministic  behavior and a drop in performance depending on how many resources are purged at a given time. I think non-deterministic behavior would be undesirable in this case.

Sounds like an awful lot of objects being loaded based on chewie's numbers for a small number of total game objects being loaded into a map.

Maybe we just need to look at the resources that are being loaded and come up with a better way so we don't load so many resources per object.

It is definitely a problem though and I think deserves a good discussion about the different approaches.
« Last Edit: October 10, 2008, 09:29:37 pm by vtchill »
Logged

phoku

  • Developer
  • Full Member
  • *
  • Posts: 102
    • View Profile
    • IZ dev blog
Re: FIFE RAM usage - animation loading
« Reply #3 on: October 11, 2008, 12:10:41 am »

remember that we are talking about the pools, not any and all objects
stored in memory.

i think we can make it non-deterministic but bounded, which is easy enough.
checking and maybe deleting 100 or so images per frame isn't an issue.

and yeah i would like to know the output of a debug run with 'pool' in the
logmodules - for both zero and openanno :)

just to get an idea what numbers we are talking about.

-phoku

ps: and yeah a solid solution to end all issues needs time, but i also see the
potential to shadow bugs in fife without ANY kind of resource deallocation.

so i will write a collector. trust me, you want it  :P
« Last Edit: October 11, 2008, 12:15:33 am by phoku »
Logged

vtchill

  • Developer
  • Full Member
  • *
  • Posts: 206
    • View Profile
Re: FIFE RAM usage - animation loading
« Reply #4 on: October 11, 2008, 11:15:46 am »

Agreed it would be nice to see output from a debug run of the pool to get a better idea as to what is going on.
Logged