Debugging Performance Issues
Sirikata is large, complicated, and uses multiple languages. Debugging performance issues can be tricky. The following tries to describe the available tools, how to use them, and which is best for different situations.
If you're looking for hotspots in C++ code, this is the suggested tool. It works by having the kernel periodically trigger sampling, collecting stacktraces from Sirikata (and other applications). You run opcontrol to enable sampling, run the application, and then turn off opcontrol. A separate program, opreport, will report the collected data. See the OProfile documentation for details.
A simple wrapper script for whichever binary you're targeting can be very handy:
#!/bin/bash sudo opcontrol --no-vmlinux --start sudo opcontrol --reset sudo opcontrol --callgraph=6 ./cppoh_d "$@" sudo opcontrol --stop sudo opcontrol --shutdown
When you run opreport, you'll need to filter out non-Sirikata modules (because OProfile examines the entire system). Make sure you include Sirikata's libraries and plugins as well as the binary you're examining!
Google Perftools is already included as we also use it for Debugging Memory Issues. To use it, set the CMake variable GOOGLE_PERFTOOLS_SETTING=CPU and recompile. When you execute the program you want to examine, set the environment variable CPUPROFILE to a location to save profiling data to, e.g.,
> env CPUPROFILE=/tmp/cppoh.prof /home/me/sirikata/build/cmake/cppoh_d
Then run pprof to analyze the output.
You can build Sirikata with GNU Profiler settings. Set CMAKE_BUILD_TYPE=Profile. This is an instrumentation-based statistical profiler. It uses the same basic idea as OProfile and will report information in a similar manner. However, it's more intrusive and seems to have a higher impact on performance. Therefore, you should generally use it when you can't get the information you need from OProfile, or to double check OProfile's results.
See the GNU Profiler guide for more details.
Course Grained Profiling
Enable existing profilers with --profile=true. The statistics will be reported at exit.
Fine-grained Event Loop Profiling
- Build with the CMake variable SIRIKATA_TRACK_EVENT_QUEUES enabled
- During performance issues, break and call Sirikata::Network::IOService::reportAllStats()