How to Debug Forums in Devstack

Observing Elasticsearch calls (Linux only)

Compatibility

Currently, this section is only tested on Linux workstations. It probably won't work on Mac since docker works differently there. FIXME!

First, launch devstack the usual way. Shell into the elasticsearch container to do some network introspection:

Determining the container interface from the elasticsearch container

tsankey@thinkpad:~/edx/devstack/devstack$ make elasticsearch-shell
root@617717ad5632:~# ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
59: eth0@if60: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:13:00:04 brd ff:ff:ff:ff:ff:ff
    inet 172.19.0.4/16 brd 172.19.255.255 scope global eth0
       valid_lft forever preferred_lft forever

The relevant snippet above is "59: eth0@if60:". We now know to look for local interface 60 on the host:

host interface 60 corresponds to container interface 59

tsankey@thinkpad:~$ ip address
[...]
60: vethe8d3316@if59: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-00dbc45e4883 state UP group default 
    link/ether a2:35:e0:3c:0f:b1 brd ff:ff:ff:ff:ff:ff link-netnsid 4
    inet6 fe80::a035:e0ff:fe3c:fb1/64 scope link 
       valid_lft forever preferred_lft forever
[...]

Now, we know we want to listen on interface vethe8d3316 on the host machine. Start up tcpflow as root:

root@thinkpad:~# tcpflow -i vethe8d3316 -e all -p -c port 9200

Now, do some things in the discussion/forums or send API calls to ES manually (see next section). You will see all the raw HTTP traffic between the forums container and ES container.

For example, this is printed when the forums makes an ES API call to edit an existing forum post (aka comment_thread) while tcpflow is listening:

tcpflow example output

root@thinkpad:~# tcpflow -i vethe8d3316 -e all -p -c port 9200
172.019.000.012.49054-172.019.000.004.09200: POST //content/comment_thread/5b994297219038008f000003/_update HTTP/1.1
User-Agent: Faraday v0.12.1
Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
Accept: */*
Connection: close
Host: edx.devstack.elasticsearch:9200
Content-Length: 51
Content-Type: application/x-www-form-urlencoded


172.019.000.012.49054-172.019.000.004.09200: {"doc":{"last_activity_at":"2018-09-13T19:14:42Z"}}
172.019.000.004.09200-172.019.000.012.49054: HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 92

{"_index":"content","_type":"comment_thread","_id":"5b994297219038008f000003","_version":15}

Sending Elasticsearch API calls

You may want to send ES API calls directly via curl. For example, you may want to search the database to validate the effects of a code change. Here is how you would search for all forum comments from a particular user:

POST to ES from the elasticsearch container

tsankey@thinkpad:~/edx/devstack/devstack$ make elasticsearch-shell
root@617717ad5632:~# curl -X GET 'http://localhost:9200/content/comment/_search?q=author_username:ExampleUser'

Refer to the ES API documentation for more examples.