Debugging Memory Issues

Overview

Generally there are two types of memory issues we're worried about: leaks and high memory usage. Leaks are references to memory which are completely lost, so they can never be properly deleted. High memory usage is just a result of not freeing memory aggressively enough or inefficient data structures. Leak checking addresses the former, heap profiling addresses the latter.

We use two tools to track memory issues, valgrind and Google's perftools. These tools are incompatible as they both replace the default malloc implementation. Complicating matters further is the fact that Google's perftools are also used in a library we link to, Berkelium (via Chromium), so we need to deal with the presence of two copies of the library in some cases. Also, at least one library (Ogre) is normally built with yet another replacement malloc which is optimized for their heap usage patterns. Generally, these are non-issues for any binary other than cppoh. Read the notes under each section carefully -- if you don't follow the instructions in them, you likely won't get correct results and may not get any results at all.

Valgrind

Valgrind has two tools that are useful: memcheck and massif for heap profiling.

Notes

• Make sure you've disabled and recompiled without perftools support (see next section, but it should be off by default).
• If you're debugging cppoh and don't care about the Ogre plugin, you should:
  • Disable loading of the Ogre plugin (pass --oh.plugins=xxx with the default list, sans ogregraphics).
  • Disable Berkelium (move it somewhere the build can't find it) or disable linking cppoh to it (currently you have to modify CMakeLists.txt manually, removing a line that looks like SET(CPPOH_LINK_LIBRARIES ${CPPOH_LINK_LIBRARIES} ogregraphics)).
• If you're debugging cppoh and care about the Ogre plugin, you should:
  • Disable Berkelium (see above).
• If you're debugging cppoh and care about Berkelium: we're not sure yet.

Google perftools

Google's perftools provide a replacement malloc implementation that can help with debugging memory issues. (It also comes with a CPU profiler which we haven't tested on Sirikata yet). On Linux, perftools is supported directly in the build, but you need to explicitly enable it. First, build the dependency:

 cd dependencies/
 make installed-perftools

When this completes, you'll have a directory dependencies/installed-perftools/. That directory will contain the perftools binaries and libraries. Later on, when we reference perftools binaries, e.g., pprof, you should run them from that directory.

Because perftools can cause problems on some platforms, we turn support for them off by default. While you should be able to use them on any platform for debugging memory issues, you probably don't want them on by default as they can occasionally cause deadlock. To enable support in the build, use ccmake to change the setting GOOGLE_PERFTOOLS_SETTING. The options are:

• NONE - don't use perftools at all, and don't even link to the tcmalloc library
• TCMALLOC - use tcmalloc, but do no memory debugging
• HEAP - enable heap profiling
• CPU - enable CPU profiling

That's it. From there, you should be able to follow the normal perftools instructions to check for leaks and profile the heap.

Notes

Once enabled, perftools should usually just work. The one exception is cppoh:

• The library linked conflicts with another copy of itself in the Ogre plugin via Berkelium via Chromium). To debug cppoh using perftools you need to disable berkelium support -- simply move Chromium so the build won't find it and won't build Berkelium.
• Even with this change, it may not work -- because of a deadlock issue with dlopen (see the perftools README), if you're on 64-bit Linux, you'll likely end up deadlocking. In fact, this seems to happen sometimes even without Ogre, but we've seen it consistently when Ogre is enabled. Valgrind is probably your best option for looking into leaks/memory usage issues involving the Ogre plugin.