Inspecting a Running Python Process

Prerequisites

# Used by pyrasite to inject code into a running process
sudo apt-get install gdb
 
# Used to allow pyrasite to inspect the internals of a running process
sudo apt-get install python-dbg
 
# Make sure that gdb can find debugging symbols for the virtualenv python
sudo mkdir -p /usr/lib/debug/opt/edx
sudo ln -s /usr/lib/debug/{usr,opt/edx}/bin
 
# Enable ptrace
echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

Meliae

Dumps memory from a process, killing it as well.

To dump memory from and kill a process, run:

kill -SIGPROF <process>

Note: this is only enabled for production environments right now. In order to get this to work for local environments, you need to make sure that the memory dumper is installed on startup. So add

import openedx.core.operations
openedx.core.operations.install_memory_dumper()

to manage.py.

You'll find the memory dump in the /tmp directory.

Pyrasite

A python library that uses gdb to inject code into a running python process. Can be used for viewing memory statistics, and also other live process inspection.

Install Pyrasite

# Install the required python libraries
virtualenv memory
source memory/bin/activate
pip install Cython
pip install pyrasite meliae urwid

Interactively View Process Memory

sudo memory/bin/pyrasite-memory-viewer <PID>

Dump Process Memory

sudo -u www-data memory/bin/pyrasite <PID> dump_memory.py
less /tmp/pyrasite-*-objects.json

Clean Up

echo 1 | sudo tee /proc/sys/kernel/yama/ptrace_scope
sudo apt-get remove gdb python-dbg
rm -rf memory
sudo rm -rf /tmp/pyrasite-*-objects.json

GdbHeap

A gdb extension that allows inspection of memory used by many python internals

Install gdbheap

sudo apt-get install libc-dbg
git clone http://git.fedorahosted.org/git/gdb-heap.git

Debug a Process

> sudo PYTHONPATH=gdb-heap/ gdb attach <PID>
(gdb) python import gdbheap
(gdb) heap