Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 56 Next »

MA-1099 - Getting issue details... STATUS

Goals:

  1. Understand the load we are able to handle with the discussion API for when the mobile app is released.

    1. What can the server handle?
  2. Understand the the overhead between the discussion API and the ruby forums code.
    1. Does the Discussion API perform better, worse, or on par with the browser's forums?
    2. What does the forums performance look like in general? 

Usage patterns to look out for:

  • Default page size for the browser is 25 while the mobile device will be using 10. It is possible that more requests could be sent for the same amount of information.
  • Push notifications
    • There is a possibility for a different usage pattern to look out for. If there is a popular thread, bursts of requests can be expected.
    • Increased forum usage as there is currently no notifications for the browser.
  • The browser can display the Threads, Response, and Comments all at once. The mobile app treats all three of these as separate views. It is possible that more requests could be sent for the same amount of information.
  • General Usage. Discussions on mobile could naturally increase discussion forums usage.

Endpoints:

 

Testing Strategy:

Originally the plan was to isolate each endpoint and determine what kind of load it can handle, but after analysis of the data, some of these endpoints seem unnecessary to isolate for a load test. These endpoints include DELETE and PATCH which are a significantly small part of the overall load in production. For the isolated test for these endpoints, it will be paired with it's appropriate GET Thread/Comment. For example, every DELETE Thread request requires a thread_id. We obtain this thread_id by calling GET Thread List with randomize parameters, which returns a list of threads where one is then randomly selected. This selected thread is then DELETEd. Below is the chart of the additional request we make. As long the ratio of how many of these requests happen in each task is understood, we can get the desired endpoint distribution. 

Request

RequiresReturnsOrder of requests
GET Threadthread_idThreadTaken from thread_id pool
GET Thread List Thread ListGET Thread List   
GET Comment Listthread_idComment ListGET Thread ListGET Comment List  
POST Threadcourse_idThreadPOST Thread   
POST Responsethread_idCommentGET Thread ListPOST Response  
POST CommentCommentCommentGET Thread ListGET Comment ListPOST comment 
PATCH Threadthread_idThreadGET Thread ListPATCH Thread  
PATCH Commentcomment_idCommentGET Thread ListGET Comment ListPATCH Comment 
DELETE Threadthread_idNo ContentPOST Thread ListGET Thread ListDELETE Thread 
DELETE Responsecomment_idNo ContentGET Thread ListPOST ResponseGET Comment ListDELETE Response*
DELETE Commentcomment_idNo ContentGET Thread ListGET Comment ListPOST CommentDELETE Comment*

*GET Thread List can always return a response (so we delete a random response), but will not always return a comment so the comment created will be the one deleted.

Thread and Comment pool:

    Various methods of select post data were considered.

  • Selecting threads from a smaller pool or selecting the same thread.  Rather than getting the entire list of thread_ids to send requests against, we would just store a random portion of the threads. A test was run to see if matters whether the retrieved thread was random or not, but the sandbox it was run against did not have the correct mongo indexes set up. Regardless, this strategy would not work when trying to DELETE threads as the pool of potential threads would be smaller. Additionally this relies on storing data that must be shared amongst the locust users which could lead to race conditions as a locust user could be trying to GET a thread that another locust user was in the middle of DELETEing. When dealing with much larger file IO operations, it could cause some limitations on the machine that spawns the locusts.
  • Retrieving the list of thread ids when starting locust. This method was effective up until the number of threads in the data set started to increase. As the median number of posts in a course is ~2000, when trying to retrieve 20*(page size max of 100), it would take 20 queries. Additionally, as mentioned in the above strategy, storing data amongst the locust users is not a trivial task. Each locust user would try to generate it's own list of threads which is unacceptable. If a thread was POSTed or DELETEd, only that locust user would have that updated information. Attempts at using the lazy module did not work either as each list of threads was instantiated separately by each locust user. Again, even if the locust users were able to use the same global variables, there would be race conditions. 

  • Calling GET thread_list per DELETE/PATCH/GET_comment. Since the ratio of GET thread_list is significantly higher than any of the other calls except for GET Thread, we can achieve the desired distribution of requests for the discussion API without having to store any of the thread_ids. The table below is a 7 day snapshot on NewRelic for the discussion forums. The only drawback is that in order to GET a single thread, we need to have a thread_id. This issue will be discussed in the next bullet.

    ActionCount Discussion API Call
    .forum.views:single_thread6759804760GET Thread
    .forum.views:forum_form_discussion2347831653GET Thread List
    .forum.views:inline_discussion1551761093GET Thread List
    create_thread31176220POST Thread
    create_comment27438193POST comment
    create_sub_comment14345101POST comment
    users1382097-
    .forum.views:user_profile1233687-
    .forum.views:followed_threads769854GET Thread List
    vote_for_comment673147PATCH Comment
    vote_for_thread624244PATCH Thread
    upload420830-
    update_comment340324PATCH Comment
    follow_thread387027PATCH Thread
    update_thread282720PATCH Thread
    delete_thread209115DELETE Thread
    endorse_comment12329PATCH Comment
    delete_comment7705DELETE Comment
    flag_abuse_for_comment3733PATCH Comment
    flag_abuse_for_thread1421PATCH Thread
  • Using pre-stored thread_id data. Since GET Thread is called more than GET Thread List, we cannot use GET Thread List to get a thread_id. Instead, we can use a pre-defined set of thread_ids as mentioned in the first two bullets. This will allow us to be able to test GET Threads in isolation. Unfortunately the issue of trying to GET a DELETEd thread may still arise. Another option could be to have the locust user only call GET Thread List once and then run multiple GET Thread's. Again, the same issue still arises if one of those Threads happened to get DELETEd. 

Production Spikes in response time:

When running some early tests, it was found that some of the requests that were believed to be slow on production, were not appearing that way on the loadtests.

Looking at the errors that show up on GET Thread, the response time is 20s, which is the time out limit. Other factors that are involved are courses that have many posts, may take longer to GET information from. These courses, although the exception, mixed in with normal courses, could explain the spikes in the data.

 Staff vs. Normal User:

    Using users with staff access was thrown into consideration as it would be make some of the permissions a bit more difficult for some discussion forums actions such as editing the body of a thread. Some tests were ran to see if there was a difference. No difference was found the tests that were designed to check for a difference. 

Pagination:

    There were some concerns that the pagination in the forums code are not working properly. A series of tests will be run against courses of different sizes and compared. The idea is that if the pagination is working correctly, all the courses should be returning threads with similar response times. If it is not working correctly, course with more posts will take longer to return threads than a smaller course. 

Things that were left out:

Moderator actions

  • Pin Thread - Not implemented
  • Open/Close Thread -Not implemented
  • Endorsed - Not Implemented

Course topics - This will be addressed at another time. 

 

/threads/

GET: 

Will also be testing against different course sizes.

 Invalid: Old test that was invalid due to the way mongo is set up

Using 10 locust users with min/max time of 1000/2000ms, GET requests for sent for either a single thread or randomly from a selection of 10% of the threads in a course. This was tested against a sandbox.

  • It seems like getting a random thread vs. the same thread does not make a reliably noticeable difference.
  • As number of posts in the database increase, the response time also increases.
  • Number of posts in a course does not seem to be matter for GETting a post. 

Update: This conclusion is invalid as the sandbox does not have the proper indexes. 

 

Approx Total PostsName# requests# failsMedianAverageMinMaxContent Size# reqs/sec
1000GET a random Thread out of 1000 Posts26710140150125.5698204410.569906211785.8
1000GET a single Thread out of 1000 Posts22620140151126.8491745337.106943111785.9
11000GET a random Thread out of 10000 Posts15220180196164.470911447.613000911786
11000GET a single Thread out of 10000 Posts15930180198164.4868851480.930805211785.8
11100GET a random Thread out of 100 Posts5420180193164.2448902419.183015811785.9
11100GET a single Thread out of 100 Posts7380180199165.0490761434.263944611786
12100GET a random Thread out of 1000 Posts6830180204166.1930084489.537954311785.6
12100GET a single Thread out of 1000 Posts10490180202169.2481041443.414926511786
13100GET a random Thread out of 1000 Posts14730180209171.40197751157.29904211786.1
13100GET a single Thread out of 1000 Posts13170180204170.7370281510.493993811785.8
14100GET a random Thread out of 1000 Posts8550190209175.9641171468.723058711786
14100GET a single Thread out of 1000 Posts75570190213173.6099721970.30401211785.1

POST: 

 Missing some data: After 1,000,000 posts in 24 hours, the response time remain constant.

Unfortunately locust ran into a calculation error when running a post test so there is no table data. After 1,000,000 posts in 24 hours, the response time remain constant. This was tested against https://courses-loadtest.edx.org/ 

/threads/{thread_id}

GET: 

Waiting on Loadtest env to get meaningful results. Refer to /threads/get

PATCH: 

 

 Patch events for the boolean values took more or less the same amount of time.
This was tested against a t2.large sandbox. 

 

TypeName# requests# failsMedianAverageMinMaxContent Size# reqs/sec
PATCHabuse_flagged1814030031596.833944321307.78193520092.5
PATCHfollowing1847030031497.836017611730.39698619391.9
PATCHvoted1875031031997.416877751427.901032104

1.3

 Patch events for being a Staff vs. Non-staff user did not make a difference.
Using 2 locust user intentionally split between staff and non-staff tasks, PATCH requests were sent with a 500ms wait time. This was tested against a t2.large sandbox. 

 

Namereqs#failsAvgMinMaxMedianreq/s
PATCHabuse_flag_thread330(0.00%)3322444703300
PATCHfollowing_thread1200(0.00%)3262414563300.2
PATCHstaff_abuse_flag_thread400(0.00%)3202404023300
PATCHstaff_following_thread1100(0.00%)3332394823300
PATCHstaff_vote_on_thread17740(0.00%)339229176073301.1
PATCHvote_on_thread17870(0.00%)331226145993301
 Different body sized edits did not seem to make a difference in response times.
Namereqs#failsAvgMinMaxMedianreq/s
PATCHabuse_flag_thread160(0.00%)1621172121500
PATCHedit_thread_with_10000char500(0.00%)1991514241900.1
PATCHedit_thread_with_1000char610(0.00%)23413837071700.1
PATCHedit_thread_with_250char540(0.00%)1781364221600.1
PATCHedit_thread_with_4char570(0.00%)1831383311700.1
PATCHedit_thread_with_5000char570(0.00%)1881413411800.1
PATCHfollowing_thread420(0.00%)1681303371600.2
PATCHvote_on_thread6980(0.00%)1601146521501.1

DELETE:

 DELETE will be best tested with the other endpoints

For every DELETE thread, we POST a Thread and then GET a thread from the thread pool. This was tested against a t2.large.sandbox.

Namereqs#failsAvgMinMaxMedianreq/s
DELETEDELETE_thread3051(0.33%)2031373521903
GETGET_thread_list3060(0.00%)1901544201702.9
POSTPOST_thread3060(0.00%)1271022771103

/comments/

GET: 

 mark_as_read and page did not seem to affect the response time at all.
Using 10 locust users with min/max time of 1000/2000ms, GET requests for sent for 1 of 10 threads with 100 responses in increasing increments of 100 responses, each with single comment.  The page_size seemed to be the parameter that affected the response time. This was tested against a t2.large sandbox.

 

TypeName# requests# failsMedianAverageMinMaxContent Size# reqs/sec
GETpage_size=100552012001212689.62907792079.3230531677470.6
GETpage_size=75588010001053570.69206241957.837821258580.7
GETpage_size=505520900927459.97500421813.49206839581.1
GETpage_size=255250790810345.99900251692.13295420581
GETpage_size=15570680710237.98203471673.28906118330.8
 None of the parameters affect the response time. As comments on a response increases, so does the response time.
Using 10 locust users with min/max time of 1000/2000ms, GET requests for sent for 1 of 10 threads where the chosen thread will contain a response of 50, 100, 150, 200... comments increasing in increments of 50.  This was tested against a t2.large sandbox.

 

TypeName# requests# failsMedianAverageMinMaxContent Size# reqs/sec
GETcomments=5003520170018221361.5820413062.2570514068320.1
GETcomments=4503950160017101244.3480493462.923053662320.3
GETcomments=4003870140015521093.0590634270.4479693256320.1
GETcomments=350394013001394972.35107422612.5550272850320.4
GETcomments=300379011001233827.77094842370.137932444320.2
GETcomments=25035309701056708.37497712795.85292038320
GETcomments=2003520830943583.65988732230.6971551632320.1
GETcomments=1503680680785441.07198722042.1819691226320.2
GETcomments=1003420540658323.15111161866.328001820320.2
GETcomments=503900390512194.78607182818.300009414320.3

POST: 

POST should be similar to POSTing threads. 

/comments/comment_id

PATCH: 

 As with PATCHing a thread, body edit size did not seem to matter. Boolean patches also seemed to be the same.
Namereqs#failsAvgMinMaxMedianreq/s
GETGET_comment_list9160(0.00%)2601196432403
GETGET_thread_list9160(0.00%)2751875712602.9
PATCHabuse_flag_comment470(0.00%)2711595132600.4
PATCHedit_comment_with_10000char600(0.00%)3072054983000.2
PATCHedit_comment_with_1000char570(0.00%)2641654482500.2
PATCHedit_comment_with_250char710(0.00%)2641604672500.3
PATCHedit_comment_with_4char760(0.00%)2651664692500.4
PATCHedit_comment_with_5000char600(0.00%)2931955972800.1
PATCHvote_on_comment5470(0.00%)2831306412701.5

DELETE: 

DELETE is best tested with the other endpoints. For every comment delete, we POST a thread, GET a random thread, and then DELETE that random thread.

 

 

MA-1102 - Getting issue details... STATUS

Course Structure Setup:

    A tarfile with a very simple setup will be used for each load test. This course was created in studio and then exported. During the course creating when seeding data, this tarfile will be used as the skeleton.

Forums Data Analysis:

    After some analysis of the forums database, there were some pieces of information that were found to be useful. The link to the google doc is here.

Some key points:

  • We’ve had 7 million forum posts (including both top-level and responses).

  • IDBx+IDB6x+2015_T2 has the most activity of any course, with over 122,000 posts. 

     Course DATA for number of posts

     

    cntcourse_idcntcourse_id
    1UTAustinX/UT.6.02x/1T20151759MITProfessionalX/6.BDX/2T2014
    1course-v1:HarvardX+SPU27x+devops_test1770PekingX/20000001x/2015Q1
    1course-v1:TsinghuaX+30640014x+3T20151781HarvardX/HUM2.1x/3T2014
    1arbi/cs205/1T20141785HarvardX/PH525.4x/1T2015
    1TBRx/EngCompX/1T20151789HarvardX/1368.2x/2T2015
    1course-v1:LouvainX+Louv3x+3T20151796MITx/JPAL101_1x/3T2014
    1course-v1:UBCx+Marketing5501x+2015preview1797RiceX/PHYS102.2x/2015T1
    1course-v1:MandarinX+MX101x+3T20151800BUx/ASTR105x/3T2014
    1course-v1:MITx+14.74x+3T20151809MITx/7.QBW_1x/1T2015
    1course-v1:RiceX+AdvBIO.4x+2016T11809course-v1:TenarisUniversityX+CNC.ETRRx_2+2T2015
    1CarrTest/Test101/2014_T11826MITx/15.390.2x/3T2014
    1UTAustinX/UT.7.01x/3T20141839course-v1:UC3Mx+CEH.1-ESx+2T2015
    1course-v1:GeorgetownX+PHLX101-02x+1T20151843course-v1:IEEEx+CloudIntro.x+2015T2
    1course-v1:Microsoft+DAT203x+1T20161866LouvainX/Louv1.1x/3T2014
    1course-v1:UBCx+Climate101x+3T20151867DavidsonX/D.001x/3T2014
    1BerkeleyX/CS190.1x/1T20151890course-v1:UQx+Denial101x+2T2015
    1TBRx/STATx/1T20151894HKUSTx/ELEC1200.1x/3T2014
    1course-v1:HarvardX+HUM1.5x+3T20151902course-v1:CaltechX+Ec1011x_2+T12015
    1course-v1:MITx+15.390.2x_1_TUR+2T20151911course-v1:UTHealthSPHx+INOV101x+2T2015
    1course-v1:MITx+4.605x_3+3T20151929course-v1:UPValenciaX+GP201x+3T2015
    1MITx/8.851x/3T20141973SchoolYourself/AlgebraX/1T2015
    1BerkeleyX/CS169.1/2013_Spring_SPOC_Binghamton1977course-v1:MITx+7.28.1x_1+2T2015
    1course-v1:CurtinX+DBAEx+3T20151985AdelaideX/HumBio101x/1T2015
    1UTAustinX/UT.6.01x/1T20141998BerkeleyX/ColWri3.3x/1T2015
    1course-v1:Microsoft+DEV205x+3T20152004DelftX/TW.3421x/1T2015
    1course-v1:MITx+7.28.2x+3T20152005HarvardX/AT1x/2T2014
    1course-v1:UBCx+Phot1x+test2006BerkeleyX/CS169.2x/2012_Fall
    1course-v1:AKTEST+INTROSTEEL+8_20152038MITx/21W.789x_2/1T2015
    1course-v1:RiceX+AdvPHY2.2x+2015T32047course-v1:IMFx+MFx+2015T2
    1course-v1:UBCx+CW1.1x+1T20162050BerkleeX/BMPR365x/2T2014
    1course-v1:AA+AA101+2015_T12053RiceX/AdvBIO.1x/2014T3
    1course-v1:MITx+11.127x+1T20152057course-v1:TeachersCollegeX+BDE1x+2T2015
    2course-v1:NYIF+MA1.3x+3T20152068course-v1:UTAustinX+UT.5.03x+2T2015
    2course-v1:TenarisUniversityX+PIPE02x+3T20152092MITx/6.341x/3T2014
    2course-v1:Microsoft+DAT204x+3T20152094OECx/PH241x/3T2014
    2MITx/11.132x/3T20142108course-v1:LouvainX+Louv11x+2T2015
    2course-v1:Microsoft+DEV202.3x+2015_T42136course-v1:UC3Mx+CEH.1-ENx+2T2015
    2course-v1:Microsoft+DAT206x+2T20162155BerkeleyX/ColWri3.2x/1T2015
    2course-v1:RiceX+BIOC300.2x+2T20152157course-v1:HarvardX+HUM2x+2015T3
    2course-v1:ANUx+ANU-INDIA1x+4T20152172course-v1:HKUSTx+COMP102.2x+4T2015
    2course-v1:HarvardX+HDS99.1x+1T20162177BerkeleyX/Stat_2.1x/1T2014
    2course-v1:PurdueX+Aero101+2015_T32177ETHx/AMRx/1T2014
    2course-v1:MITx+7.00x_3+2T20152200MITx/14.73x_1/1T2015
    2course-v1:NotreDameX+TH120.2x+3T20152205course-v1:BerkeleyX+EE40LX+2T2015
    2course-v1:TrinityX+T002x+3T20152207course-v1:DelftX+EX101x+3T2015
    3course-v1:ACCA+Accounting101x+1T20152208UPValenciaX/TE201x/2T2015
    3course-v1:ACCA+FMA-F2.x+3T20152210PekingX/04332960X/3T2013
    4course-v1:MITx+6.002.Ax+3T20152215RiceX/PHYS102.1x/2015T1
    4course-v1:AdelaideX+AddictionX+3T20152219HarvardX/HLS3x/1T2015
    4BerkeleyX/CS188.1x-4/1T20152225course-v1:HarvardX+BUS5.1x+3T2015
    4course-v1:MITx+6.002.Cx+3T20152241TsinghuaX/80000901_1X/3T2013
    4course-v1:NYIF+YCA2015.1x+2015T32262HarveyMuddX/CS001x/1T2015
    4course-v1:UBCx+Marketing5502x+3T20152268NotreDameX/MAT150x/2T2015
    5MITx/15.390.2x_TUR/1T20152284course-v1:GeorgetownX+MEDX202-01+2015_3T
    5course-v1:ANUx+ANU-ActuarialX+3T20152289course-v1:BerkeleyX+ColWri.3.7x+3T2015
    5course-v1:TsinghuaX+60240013x+3T20152291DelftX/CTB3300WCx/2T2014
    5course-v1:DelftX+NGIx+3T20152296course-v1:UBCx+Phot1x+2T2015
    5course-v1:MITx+6.002.Bx+3T20152304course-v1:MITx+15.390.1x_1+1T2015
    5course-v1:TsinghuaX+00690242.2x+3T20152308BUx/PY1x/1T2015
    5course-v1:edX+LG101x+2015_testing2314UQx/TROPIC101x/1T2014
    5andya/AA101/2014_T12330DelftX/CTB3365STx/2T2015
    5course-v1:HarvardX+1368.4x+2T20162341HamiltonX/ENG142x/2015_T1
    6course-v1:Course_Re-runs+LG101x+2014_T1_re-run2355course-v1:DelftX+TPM1x+2T2015
    6course-v1:PrincetonX+CaseStudies101+3T20152365UC3Mx/PCA.1x/1T2015
    6course-v1:HarvardX+1368.3x+2T20162377course-v1:UCSDx+CSE167x+3T2015
    6course-v1:Microsoft+INF202x+3T20152383CornellX/INFO2040x/1T2014
    7course-v1:PekingX+20000001x+2015T32392IDBx/IDB2x/3T2014
    7course-v1:Microsoft+INF201.13x+1T20162403UBCx/Water201x/3T2014
    7course-v1:UTennesseeX+BIRDS101.1x+3T20152412UTArlingtonX/LINK5.10x/3T2014
    7CooperUnion/CS.2x/1T20152423course-v1:UPValenciaX+BSP101x+3T2015
    7BerkeleyX/BJC.1x/3T20152423course-v1:TUMx+QEMx+2T2015
    7course-v1:Microsoft+CLD203x+1T20162431course-v1:MandarinX+MX101x+2T2015
    8course-v1:HarvardX+1368.2x+2T20162435course-v1:MITx+uINOV8x+2T2015
    8course-v1:HarvardX+SPU30x+2T20162465MITx/3.091x_3/3T2014
    8course-v1:Microsoft+CLD1003x+1T20162476HarvardX/PH525.3x/1T2015
    8course-v1:MITx+15.390.2x_SPA+2T20152479DavidsonX/D003x.1/1T2015
    9course-v1:HarvardX+1368.1x+2T20162482course-v1:UQx+World101x+2T2015
    9course-v1:TsinghuaX+00690242.3x+3T20152491course-v1:SMES+PSYCH101x+2T2015
    11course-v1:edX+JOB101x+20152501HKUx/HKU01x/3T2014
    12course-v1:IITBombayX+EE210.2X+1T20162509course-v1:Microsoft+DEV202x+2015_T2
    13RiceX/AdvPHY2.4x/2T20152526HarvardX/PH525x/1T2014
    15course-v1:DartmouthX+DART.MUS.01X+2015_T32533TenarisUniversityX/CNC.ETRRx/3T2014
    15BerkeleyX/CS100.1x/1T20152569McGillX/ATOC185x_2/1T2015
    16course-v1:Microsoft+DEV202.2x+2015_T42596DelftX/DDA691x/3T2014
    17course-v1:TsinghuaX+00690242.1x+3T20152624MITx/3.086-2x/1T2015
    17HarvardX/HLS1.1x-2/1T20152680DelftX/Calc001x/2T2015
    18course-v1:CaltechX+BEM1105x+3T20152694SMES/COL101x/1T2015
    18course-v1:NYIF+MA1.2x+3T20152731DelftX/CTB3365DWx/3T2014
    19course-v1:PekingX+02132750x+2015T32739MITx/16.110x/1T2014
    20course-v1:DavidsonX+D004x+3T20152740PekingX/01339180X/3T2013
    21course-v1:TenarisUniversityX+CNC.ETRRx_3+3T20152749course-v1:LinuxFoundationX+LFS101x.2+1T2015
    22HarvardX/HKS-211.1x/3T20132775MITx/3.091x_2/1T2014
    22course-v1:adam+adam+adam2780course-v1:TenarisUniversityX+STEEL101x+2T2015
    23PekingX/532001x/3T20142800course-v1:MITx+10.03x+2T2015
    23course-v1:WestonHS+BIO101x+2T20152833UC3Mx/CEH.1-ESx/1T2015
    24course-v1:TsinghuaX+30240184x+3T20152849course-v1:SmithsonianX+USHIS1.1x+2015_T2
    25course-v1:PekingX+01233170x+2015T12853course-v1:HKPolyUx+ANA101x+2T2015
    25course-v1:EPFLx+BrainX+3T20152862course-v1:WellesleyX+ANTH207x_2+2T2015
    31course-v1:Microsoft+INF201x+2015_T42875BerkeleyX/EECS149.1x/2T2014
    34course-v1:EPFLx+NMR101x+2015_T32878course-v1:MITx+6.002x_6x+1T2015
    39course-v1:EPFLx+AlgebreX+4T20152881course-v1:MITx+15.662x+1T2015
    42EdX/NAA101/20142936MITx/2.01x/2T2014
    43course-v1:HarveyMuddX+PHYS024.3x+1T20152940course-v1:IMFx+FPP.1x_2015+2015_T2
    45course-v1:TsinghuaX+40040152X+3T20152943UBCx/China300x/3T2014
    46course-v1:RiceX+AdvPHY2.1x+2015T32943course-v1:UQx+Crime101x+2T2015
    49course-v1:MITx+15.390.1x_1_TUR+1T20152948course-v1:IIMBx+QM101.1x+1T2016
    50course-v1:HKPolyUx+EWA.1x+3T20152956RiceX/RELI157x/1T2015
    52RiceX/AdvBIO.5x/2015T12957IITBombayX/ME209x/2T2014
    53course-v1:Microsoft+CLD201x+2015_T42969BerkeleyX/Stat2.3x/2013_SOND
    58MITx/6.00short/2013_IAP2976HarvardX/HKS211.1x/3T2013
    61RiceX/AdvENVSCI.4x/2015T12990LouvainX/Louv3x/1T2015
    61course-v1:LouvainX+ConfX+2T20152998MITx/2.03x/3T2013
    61course-v1:TsinghuaX+30700313x+3T20153056MITx/9.01x/3T2014
    62EPFLx/EE-102B.1x/1T20153078University_of_TorontoX/OEE101x/3T2013
    65UPValenciaX/IQ101.3x/3T20153079BerkeleyX/CS188/fa12
    65MITx/3.091/MIT_2012_Fall3117course-v1:CornellX+HIST1514x+1T2015
    66HarvardX/MCB80.2x/3T20143161UPValenciaX/AIP201x/2T2015
    68course-v1:edX+StudioX+20153164course-v1:Microsoft+DAT204x+1T2016
    70course-v1:TsinghuaX+10610224x+3T20153231RiceX/BIOC372.1x/1T2014
    70course-v1:ANUx+ANU-ASTRO1x+3T20153253course-v1:HarvardX+MCB63X+3T2015
    71course-v1:WestonHS+CALC360x+2T20153286HKUSTx/EBA102x/4Q2015
    71course-v1:KIx+KIUrologyx+2015_T33294MITx/JPAL101x/1T2014
    76course-v1:DelftX+DDA691x+3T20153321course-v1:HKUSTx+COMP102.1x+4T2015
    76course-v1:PekingX+00330280x+2015T13326HarvardX/PH525.1x/1T2015
    79UPValenciaX/FFI101.2x/1T20153333RiceX/BIOC372.1x-F14/2T2014
    81course-v1:EPFLx+TrigoExpX+1T20153345BUx/Math226.1x/1T2015
    83EPFLx/EE-100Bx/3T20143380course-v1:Microsoft+DEV203x+2015_T2
    89course-v1:TsinghuaX+70120073x+3T20153390MITx/6.041x_1/1T2015
    90course-v1:UBCx+SPD1x+1T20163396HarvardX/HKS211.2x/3T2015
    90course-v1:TsinghuaX+00612642x+3T20153430course-v1:UTPermianBasin+SOCI101x+2T2015
    92course-v1:PekingX+18000201X+2015T23443MITx/3.032x/3T2014
    96PekingX/04830260x/2015Q13465MITx/6.832x/3T2014
    96PekingX/18001001x/2015Q13503UTAustinX/UT.5.02x/1T2015
    99WestonHS/MechC101x/2T20153635MITx/EECS.6.002x/3T2013
    100course-v1:WellesleyX+ANTH207x_3+3T-20153653course-v1:WageningenX+NUTR101x+2T2015
    106course-v1:HarvardX+ER22.1x+2015T33657MITx/MAS.S69x/1T2014
    107Ethicon/CSEC2014-1x/3T20143675DelftX/TW3421x/1T2014
    110UPValenciaX/IQ101.1x/2T20153714course-v1:UWashingtonX+ECFS302x+1T2015
    111course-v1:Microsoft+INF201.12x+1T20163748course-v1:ColgateX+GW101x+3T2015
    114IMFx/FPP.1x/1T20143816course-v1:DelftX+ET3034x+3T2015
    121course-v1:MITx+15.390.2x_1+2T20153830DelftX/TPM1x/3T2014
    121course-v1:tennessee+FYS100+F20153887course-v1:AdelaideX+Lang101x+2T2015
    123course-v1:NYIF+MA1.1x+3T20153904HarvardX/AmPoX.1/2014_T3
    123course-v1:DavidsonNext+Cal_APccx+2T20153978MITx/7.QBWx/2T2014
    125course-v1:Microsoft+CLD201x+2015_T23981HarvardX/1368.1x/3T2014
    127PekingX/04830050.2x/2015Q34026course-v1:IEEEx+MTX.1x+2015Q3
    127course-v1:RiceX+AdvBIO.1x+2015T34040CornellX/HIST1514x_Fall2014/3T2014
    131course-v1:IITBombayX+ME209.1x+1T20164053course-v1:AdelaideX+Wine101x+2T2015.2
    133CooperUnion/Chem.2x/1T20154089HarvardX/SPU30x/2T2014
    133UPValenciaX/DC201x/2T20154106MITx/7.28.1x/1T2015
    136course-v1:PekingX+04831750.2x+2015T24134UWashingtonX/AA432x/3T2014
    137course-v1:LinuxFoundationX+LFS201x+2T20154147BerkeleyX/CS.CS169.1x/3T2013
    138course-v1:SMES+ASLCx+2T20154149course-v1:ASUx+MCO425x+2T2015
    140MITProfessionalX/6.BD_GEx/2T20144239KIx/KIeHealthX/2T2015
    141course-v1:TokyoTechX+GeoS101x+2016_T14258course-v1:TUMx+AUTONAVx+2T2015
    142course-v1:PekingX+03530370x+2015T14270CaltechX/Ec1011x/1T2014
    143course-v1:HarvardX+BUS5.1x_Application_Only+3T20154271course-v1:HKUSTx+COMP107x+2016_T1
    146PekingX/00330280x/2015Q14273TsinghuaX/80512073x/1T2014
    151course-v1:W3Cx+HTML5.1x+4T20154316RiceX/ELEC301x/T1_2014
    152course-v1:RiceX+AdvENSCI.1x+2015T34340MITx/21W.789x/1T2014
    152course-v1:IMFx+TADATx+2015T24345UPValenciaX/GM201x/2T2015
    153course-v1:RiceX+BIOC300.1x+3T20154350course-v1:UWashingtonX+COMM220UWx_2+1T2015
    154OECx/Energy103/3T20144352MITx/4.605x_2/3T2014
    154course-v1:HKUx+HKU01x+3T20154378MITx/6.SFMx/1T2014
    154UPValenciaX/FFI101.1x/1T20154391edX/GWPx/2014_T1
    156course-v1:DavidsonNext+Phy_APccx+2T20154430LouvainX/Louv3.02x/3T2014
    161UPValenciaX/BMD101x/2T20154431course-v1:DelftX+AE1110x+2T2015
    164MITx/6.002x-EE98/2012_Fall_SJSU4433KIx/KIexploRx/3T2014
    170course-v1:HarveyMuddX+PHYS024.2x+1T20154446course-v1:IIMBx+OM101.1x+1T2016
    172course-v1:UPValenciaX+ISC101.1x+3T20154449MITx/3.091X/2013_Fall
    173RiceX/AdvPHY2.3x/1T20154463course-v1:Microsoft+DEV204x+2015_T4
    175course-v1:DavidsonNext+Mac_APccx+2T20154463HarvardX/AmPoX.3/1T2015
    176PekingX/02030330x_1/3T20144495BerkeleyX/CS169.1x/2013_Spring
    178course-v1:BerkleeX+MB110x+3T20154505HarvardX/AI12.1x/2013_SOND
    182UPValenciaX/FCO201x/2T20154522course-v1:HarvardX+HKS101A+2015T3
    183HarvardX/PH525.7x/1T20154536course-v1:SmithsonianX+POPX1.2x+2015_T2
    183TsinghuaX/30240184_x/1T20154557course-v1:UBCx+SPD1x+2T2015
    184TsinghuaX/00690242_2x/3T20144574UAMx/Android301x/1T2015
    187UPValenciaX/IQ101.2x/2T20154580CornellX/ENGRI1280x/1T2014
    187course-v1:EPFLx+TrigoExpX+2T20154613HarvardX/AmPoX.2/2014_T3
    187course-v1:BUx+PY1x+3T20154618MITx/6.004.1x/1T2015
    197TsinghuaX/00690242.x/1T20154630University_of_TorontoX/BE101x_2/3T2014
    201TsinghuaX/20220332_2x/1T20144631course-v1:HarvardX+AmPoX.5+2T2015
    201PekingX/02930106x/2015Q14696UTAustinX/UT.2.02x/3T2014
    203course-v1:TsinghuaX+00612642x+1T20154747SMES/ASLCx/1T2015
    203TsinghuaX/30240184.x/3T20144752course-v1:BerkeleyX+ColWri2.1x+3T2015
    205MITx/7.012/MIT_2013_Spring4766MITx/12_340x/1T2014
    209RiceX/AdvENVSCI.3x/2015T14770course-v1:ACCA+FA1-MA1.X+2T2015
    214course-v1:EPFLx+MF201x+1T20154812ColumbiaX/HIST1.3x/1T2015
    214MITx/15.390.1x_TUR/1T20154819GeorgetownX/INFX523-02x/3T2014
    214course-v1:GeorgetownX+GUIX-501-02x+2015_3T4889course-v1:MITx+8.MechCx_2+2T2015
    214HarvardX/HUM2.3X/3T20144917MITx/6.00.2_2x/3T2014
    216BerkeleyX/BFV101x/T120154925course-v1:ANUx+ANU-ASTRO1x+2T2015
    218course-v1:Microsoft+DEV201x+2015_T44938IMFx/ESRx/2T2014
    222course-v1:ANUx+ANU-ASTRO2x+3T20155071CornellX/ENGRI1210x/1T2015
    222GeorgetownX/HUMX423-01x/1T20155087HarveyMuddX/CS005x/2T2015
    222Ethicon/MARS2014-1x/2014_15107LouvainX/Louv8x/1T2015
    231course-v1:Microsoft+DEV208x+3T20155131IITBombayX/CS101.1x/2T2014
    236course-v1:Microsoft+205x+1T20165141BerkeleyX/CS184.1x/2013_Spring
    239BerkeleyX/CS188x_1/1T20135148DavidsonX/D002/2014_T3
    241PekingX/01034040x/3T20145163course-v1:MITx+6.00.2x_3+1T2015
    243RiceX/AdvBIO.4x/2015T15174DavidsonX/001x/1T2014
    243BerkeleyX/CS169.2x/2013_Spring5186VJx/VJx/3T2014
    244course-v1:LouvainX+Louv10x+3T20155188RiceX/PHYS201x/3T2014
    246IITBombayX/EE210.2X/3T20155204course-v1:OECx+TESS101x+2T2015
    247course-v1:TsinghuaX+20220214x+2015_T35252LouvainX/Louv2x/1T2015
    268TsinghuaX/80000901x/3T20145253LouvainX/Louv1.01x/1T2014
    273course-v1:DavidsonNext+TwD_APccx+2T20155327KyotoUx/001x/1T2014
    274UPValenciaX/BMA101x/2T20155331University_of_TorontoX/LA101x_2/1T2015
    280EPFLx/EE-102Bx/1T20145359MITx/14_73x/1T2014
    283RiceX/AdvBIO.3x/2015T15363MITx/8.MechCx/1T2015
    285UPValenciaX/VF201x/2T20155401BerkeleyX/Stat2.2x/2013_April
    289IEEEx/ISSCCx/3T20145426BerkeleyX/CS-184.1x/2013_October
    291RiceX/AdvPHY2.2x/1T20155455HarvardX/GSE3x/1T2015
    292IITBombayX/ME209.1x/2T20155588course-v1:ASUx+AST111+3T2015
    295course-v1:Microsoft+DEV206.1x+3T20155605BerkeleyX/ColWri3.5x/1T2015
    297UPValenciaX/BMN101x/1T20155625BerkeleyX/EE40LX/1T2015
    299course-v1:MITProfessionalX+CSx+2015_T15668HarvardX/EMC2x/1T2015
    300UPValenciaX/BMI101x/2T20155670MITx/16.00x/1T2015
    305course-v1:CurtinX+TBOMx+2T20155673UC3Mx/CEH.1-ENx/1T2015
    316course-v1:BerkleeX+BMPR365x+3T20155703BUx/INTL301x/3T2014
    317course-v1:UBCx+SPD2x+2T20155728course-v1:RiceX+MedDigX_+2015T2
    328EPFLx/BIO465.1x/4T20145812HarvardX/PH210x/1T2014
    329MITx/8.MReVx_T/2T20145817course-v1:MITx+24.118x+2T2015
    330course-v1:Microsoft+DAT201x+2015_T45916DelftX/AE.1110x/3T2014
    331UTokyoX/UTokyo002x/3T20145952MITx/16.101x/2013_SOND
    332course-v1:MITx+3.091x_5+3T20156003BerkeleyX/ColWri3.1x/3T2014
    340TsinghuaX/60240013.x/1T20156026MITx/6.002_4x/3T2014
    345SNUx/SNU034.005.1x/1T20146031DelftX/Frame101x/1T2015
    347course-v1:BerkleeX+MB110x+2T20156056course-v1:DartmouthX+DART.ENGS.01.X+2015_T2
    354BerkeleyX/CS169.1x/2013_March6192BerkeleyX/J4SC101/1T2015
    358course-v1:UTokyoX+UTokyo003x+2015T26222HamiltonX/RELST005.5x/1T2015
    370course-v1:UQx+HYPERS301.x+1T20156231UTAustinX/UT.8.02x/1T2015
    371HarvardX/PH525.8x/1T20156261IMF/FPP.1x/2T2014
    372course-v1:Microsoft+INF201x+2015_T26301BerkleeX/BCM-MB110x/1T2014
    382HarvardX/HUM2.4x/3T20146313NotreDameX/SOC120x/2T2015
    387course-v1:edX+BlendedX+1T20156325MITx/2.01x/2013_Spring
    389HarvardX/PH525.6x/1T20156356ETHx/FC-01x/3T2014
    399RiceX/AdvENVSCI.2x/2015T16369HKUSTx/EBA101x/3T2014
    400HarveyMuddX/PHYS024x/1T20156381BerkeleyX/CS184.1x/2012_Fall
    403course-v1:MITx+3.032.1x+3T20156388RiceX/PHYS102x/2013_Oct
    407TsinghuaX/70167012x/3T20146399University_of_TorontoX/LA101x/1T2014
    415EPFLx/BrainX/1T20156473course-v1:HKUSTx+COMP102.1x+2T2015
    417course-v1:UBCx+China300.1x+3T20156622MITx/6.002x/2013_Spring
    417course-v1:NYIF+YCA2015.1.2x+2015_Q26643ANUx/ANU-ASTRO3x/4T2014
    420course-v1:VJx+VJx+3T20156659course-v1:ETHx+FC-01x+2T2015
    427CooperUnion/Chem.1x/1T20156659MITx/3.091x/2013_Spring
    428HarvardX/HUM2.5x/3T20146672TUMx/AUTONAVx/2T2014
    430course-v1:Microsoft+DEV202.1x+2015_T46746IMFx/DSAx/3T2014
    432EPFLx/MF201x/1T20146762GEMSx/GE001x/1T2015
    438course-v1:DelftX+TXT1x+3T20156899course-v1:W3Cx+W3C-HTML5+2015T3
    440course-v1:BerkleeX+BMPR365x+2T20156942McGillX/CHEM181x_2/3T2014
    443course-v1:IITBombayX+CS101.1x+4T20156948BerkeleyX/CS-191x/2013_August
    462EPFLx/PHYS-209.1x/1T20156994RiceX/PHYS_102x/1T2014
    464HarvardX/PH525.5x/1T20157042course-v1:TsinghuaX+30640014x+2T2015
    469UPValenciaX/DMT201x/2T20157108UTAustinX/UT.5.01x/1T2014
    474course-v1:PekingX+04831750.1x+2015T17145CaltechX/CS_1156x/3T2014
    475GeorgetownX/PHYX152x/1T20157164BUx/SABR101x/2T2014
    476course-v1:PekingX+02132750x+2015T17182course-v1:ANUx+RSIT-01x+3T2015
    488course-v1:PurdueX+nano525x+2015_T37226ANUx/ANU-ASTRO2x/2T2014
    493PekingX/04832430X/3T20137237course-v1:BerkeleyX+ColWri.3.6x+3T2015
    495course-v1:WageningenX+GFFCx+2T20157343ColumbiaX/HIST1.1x/3T2014
    495ETHx/CAMS.1x/1T20157363DelftX/NGI101x/1T2014
    510edX/edX101/20147516ANUx/ANU-ASTRO4x/1T2015
    513course-v1:RiceX+AdvPHY1.1x+3T20157526CornellX/Astro2290x/1T2014
    518PekingX/01718330x/1T20147560MITx/6.041x/1T2014
    521course-v1:IITBombayX+EE210.1X+1T20167582BerkeleyX/CS191x/2013_Spring
    530course-v1:TenarisUniversityX+STEEL101x_1+3T20157622LouvainX/Louv4x/1T2015
    535TsinghuaX/20220214.x/1T20157682LouvainX/Louv3.01x/1T2014
    538BerkeleyX/CS.169.2x/3T20137687McGillX/ATOC185x/2T2014
    539PurdueX/nano520/3T20157718MITx/18.01.1x/2T2015
    545PekingX/04830050x/2T20147784course-v1:ANUx+Igno101x+2T2015
    546Ethicon/MARS2014-2x/2014_27913ColumbiaX/HIST1.2x/3T2014
    551TsinghuaX/30240184x/1T20148094HarvardX/SW47.1x/2014_T3
    553BerkeleyX/Stat_2.2x/1T20148181GeorgetownX/HUMX421-01x/3T2014
    555HarvardX/MCB80.1x/2013_SOND8210UQx/Crime101x/3T2014
    556TsinghuaX/80512073.x/3T20148221DelftX/FP101x/3T2014
    556HarvardX/HUM2.2x/3T20148276UTAustinX/UT.4.01x/2013_Sept
    557CooperUnion/CS.1x/1T20158305SMES/PSYCH101x/1T2015
    558course-v1:PurdueX+PN-15.2+2015_3T8352UTAustinX/UT.3.02x/3T2014
    561course-v1:MITProfessionalX+6.BDx+4T20158416course-v1:HarveyMuddX+CS002x+2T2015
    584SNUx/SNU446.345.2x/2T20148529DelftX/ET.3034TU/3T2014
    587course-v1:UQx+Tropic101+2T20158635course-v1:BerkeleyX+GG101x+3T2015
    597course-v1:Delftx+CTB3300WCx+2015_T38674course-v1:HarvardX+SPU27x+2015_Q2
    600course-v1:Microsoft+CLD202x+1T20168735course-v1:UWashingtonX+ECFS311x+2T2015
    605course-v1:MITx+6.004.1x_2+3T20159063MITx/8.MReVx/2T2014
    610course-v1:OECx+B1156+1T20159072course-v1:DartmouthX+DART.ENVS.02X+2015_T1
    614UQx/HYPERS301x/1T20149227course-v1:HarvardX+USW30x+1T2016
    620course-v1:UC3Mx+HGA.1x+2T20159271LinuxFoundationX/LFS101x/2T2014
    621course-v1:SMES+COL101x+2T20159343MITx/7.00x_2/2T2014
    624GeorgetownX/HUMX422-01x/1T20159402ANUx/ANU-INDIA1x/1T2014
    625course-v1:UQx+BIOIMG101x+1T20159418HarvardX/PH201x/2013_SOND
    634RiceX/AdvBIO.2x/2014T39540course-v1:MITx+CTL.SC1x_1+2T2015
    637RiceX/ELEC301.2x/2015Q39749MITx/6.00.2x/1T2014
    643LouvainX/Louv1.2x/4T20149766course-v1:University_of_TorontoX+INQ101x+2T2015
    660TsinghuaX/00690242_1x/1T20149810NotreDameX/TH120.1x/2T2015
    664course-v1:edX+GFADemox+3T20159955McGillX/Body101x/1T2015
    664HarvardX/HAA1x/1T20149963BerkeleyX/CS188.1x/2013_Spring
    665course-v1:MITx+15.390.1x_SPA+1T101510050course-v1:MITx+6.00.1x_7+3T2015
    666course-v1:McGillX+GROOCx+T3_201510070CornellX/HIST1514x/1T2014
    667RiceX/AdvPHY2.1x/3T201410071course-v1:MITx+15.071x_2a+2T2015
    674course-v1:IMF+Sp_FPP.1x+2015_T310273MITx/15.390x/1T2014
    675PekingX/02132750x/2015Q110406course-v1:AdelaideX+Code101x+2T2015
    685MITx/8.EFTx/3T201410421LouvainX/Louv4.01x/1T2014
    688ETHx/CAMSx/3T201410465CornellX/ENVSCI1500x/1T2015
    702MITx/3.072x/1T201510469DelftX/EX101x/1T2015
    705UPValenciaX/BI101x/2T201510646HarvardX/CB22.1x/2013_SOND
    706course-v1:Microsoft+DAT202.1x+1T201610695course-v1:IDBx+IDB9x+2015_T2
    731PurdueX/pncom201501x/2015_T110870ANUx/ANU-ASTRO1x/1T2014
    732course-v1:UBCx+UseGen.1x+3T201510878MITx/15.390.1x/3T2014
    733UAMx/TxEtj201x/1T201510883MITx/6.00.1-x/1T2014
    746OsakaX/BIO101x/1T201510959MITx/8.05x/1T2015
    757TsinghuaX/80000901_2x/1T201411034HarvardX/AmPoX.4/1T2015
    764course-v1:OECx+RADIO101x+2T201511229WellesleyX/ANTH_207x/2013_SOND
    811course-v1:ETHx+AMRx+1T201511279CaltechX/CS1156x/Fall2013
    814EPFLx/PHYS-209x/4T201411328UQx/World101x/3T2014
    819TsinghuaX/20220214x/3T201411464UTAustinX/UT.8.01x/1T2014
    830TsinghuaX/30700313x/1T201511501MITx/6.00.1_4x/3T2014
    832BerkeleyX/Stat_2.3x/2T201411523HarvardX/SW25x/1T2014
    832course-v1:Microsoft+DEV203x+2015_T411570UWashingtonX/COMM220UWx/1T2014
    836CornellX/INFO2040x_Spring2015/1T201511595course-v1:MITx+JPAL101SPAx+1T2015
    844course-v1:MITx+JPAL101x_3+1T201511742course-v1:BUx+SABR101x+2T2015
    855IMFx/ESRx-June/2T201411747KIx/KIBEHMEDx/3T2014
    861UAMx/Quijote501x/1T201512009MITx/3.086x/2013_SOND
    872course-v1:KIx+KIexploRx+2015T312035HarvardX/SW12.6x/2T2014
    873TsinghuaX/40050455x/1T201512037course-v1:IDBx+IDB8x+2015_T3
    874DelftX/NGI102x/3T201412061course-v1:CatalystX+ILX2+2015_2T
    879HarvardX/PH525.2x/1T201512194BerkeleyX/CS169.1x/2012_Fall
    883UBCx/Forest222x/1T201512671HarvardX/AI12.2x/2013_SOND
    889UTAustinX/UT.PreC.10.01x/3T201512719GeorgetownX/MEDX202-01/2014_SOND
    894UChicagoX/PS280x/2015_T112728course-v1:AdelaideX+Cyber101x+2T2015
    900BerkleeX/MB110_3x/3T201412839MITx/ESD.SCM1x/3T2014
    905course-v1:PekingX+20000001x+2015T112926CornellX/HOSP.101x/1T2015
    905course-v1:HKUSTx+ELEC1200.1x+3T201512994MITx/15.071x/1T2014
    911course-v1:HKPolyUx+ISE101x+2T201513104HarvardX/SW12.5x/2T2014
    912DelftX/TOPOCMx/1T201513292HKUSTx/COMP102x/2T2014
    918SchoolYourself/GeometryX/1T201513325WageningenX/NUTR101x/1T2014
    931OECx/BP111x/3T201413515GeorgetownX/GUIX-501-01x/3T2014
    952course-v1:MITx+3.15x+1T201513729UC3Mx/IT.1.1x/1T2015
    956HKUx/HKU04x/2T201513768HarveyMuddX/CS002x/1T2015
    973course-v1:UC3Mx+PCA.1x+2T201514041DelftX/AE1110x/1T2014
    979course-v1:ACCA+FA2-MA2.X+2T201514229MITx/4.605x/2013_SOND
    987course-v1:UWashingtonX+JSIS123x+1T201514302WellesleyX/SOC108x/2014_SOND
    990course-v1:BerkleeX+OENTR-391x+3T201514563LouvainX/Louv2.01x/1T2014
    996IMFx/FPP.1x_2015/1T201514869HarvardX/HSPH-HMS214x/2013_SOND
    1000PekingX/03131840x/3T201415070MITx/6.00.1_3x/2T2014
    1000HarvardX/HKS211.2x_1/3T201515122CornellX/PHIL2411x/2Q2015
    1008IMFx/OL14.01/2T201415514UBCx/IndEdu200x/1T2015
    1009course-v1:BUx+Math226.3x+3T201515648HarvardX/SW12.4x/1T2014
    1014PurdueX/nano515x/2015_T215828MITx/15.071x_2/1T2015
    1020UTArlingtonX/ENGR1.0x/2T201516332IDBx/IDB1x/2T2014
    1029course-v1:BerkleeX+OHARM-100x+3T201516376DartmouthX/DART.ENVS.01.X/2015_T1
    1031course-v1:IIMBx+IS110x+1T201616514MITx/8.02x/2013_Spring_REMOVED
    1035BerkeleyX/CS-169.2x/2013_Summer16740course-v1:Microsoft+DEV204x+2015_T2
    1035course-v1:TsinghuaX+THU00022X+3T201516836DelftX/CTB3365x/2013_Fall
    1041University_of_TorontoX/D101x/1T201516911HarvardX/HLS2x/1T2015
    1041TsinghuaX/80512073_x/1T201517556course-v1:SmithsonianX+POPX1.1x+2015_T2
    1042CaltechX/Ay1001x/2T201418097HarvardX/SW12.9x/3T2014
    1054BerkleeX/BMPR365_2x/3T201418134MITx/8.MReV/2013_Summer
    1066DavidsonX/D003x.2/2T201518175BerkeleyX/Stat2.1x/2013_Spring
    1070course-v1:BerkleeX+BMPR365_3x+1T201518775UTAustinX/UT.2.01x/2013_Sept
    1075RiceX/BIOC300.1x/1T201518789course-v1:MITx+6.00.1x_6+2T2015
    1083IMFx/FPPx/3T201318839MITx/3.091x/2012_Fall
    1094IITBombayX/CS101.2x/3T201418948MITx/7.00x/2013_SOND
    1110IMFx/FPP.1x_fr/3T201418980BerkeleyX/CS188.1x/2012_Fall
    1110TsinghuaX/60240013x/3T201419975HarvardX/GSE1.1x/3T2014
    1135course-v1:EPFLx+PlasmaX+2T201520176HarvardX/SW12.8x/3T2014
    1137course-v1:Microsoft+DAT201x+2015_T220286GeorgetownX/phlx101-01/1T2014
    1137WestonHS/PFLC1x/3T201520625course-v1:UQx+Think101x+2T2015
    1150course-v1:TsinghuaX+00691153.x+1T201520938MITx/24.00_1x/3T2014
    1182course-v1:AdelaideX+HumBio101x+2T201520951UTAustinX/UT.1.01x/2013_Sept
    1182CaltechX/BEM1105x/1T201520970MITx/14.73x/2013_Spring
    1186course-v1:IMFx+FPP.1x_fr+2015T221743UTAustinX/UT.9.01x/1T2014
    1195UAMx/QuiOrg101x/1T201521761HarvardX/SW12.7x/3T2014
    1217course-v1:ColgateX+PSYCRx+2015_T322127HarvardX/PH278x/2013_Spring
    1232course-v1:TenarisUniversityX+PIPE01x+2T201522345GeorgetownX/INFX523-01/2013_Fall
    1241RiceX/AdvENVSCI.1x/2014T322486UQx/Denial101x/1T2015
    1251course-v1:ChalmersX+ChM002x+2T201522517MITx/6.002x/2012_Fall
    1253BerkleeX/MB110x/3T201422654University_of_TorontoX/BE101x/2013_SOND
    1256course-v1:Microsoft+DIS50.1x+1T201622715HarvardX/SW12.2x/1T2014
    1257course-v1:MITx+3.091x_4+1T201522936course-v1:UQx+Write101x+2T2015
    1271RiceX/ELEC301x_/2015Q322963course-v1:AdelaideX+Wine101x+2T2015
    1272UTokyoX/UTokyo001x/3T201423239MITx/6.00.1x_5/1T2015
    1316course-v1:UBCx+Marketing5501x+2T201523784UWashingtonX/ECFS302x/2T2014
    1325HarvardX/1368.4x/2T201524234DelftX/ET3034TUx/2013_Fall
    1330BerkeleyX/CS-169.1x/2013_Summer24328LouvainX/Louv5x/2T2015
    1354HarvardX/HLS1xC/Copyright24424BerkeleyX/ColWri_2.1x/3T2014
    1359HKUx/HKU02.1x/3T201424708course-v1:IDBx+IDB7x+2015_T1
    1367course-v1:BUx+ASTR105x+2T201524802BerkeleyX/ColWri2.3x_2/1T2015
    1368course-v1:MITProfessionalX+6.BDx+5T201524966CatalystX/ILX1/2015_T1
    1375course-v1:KyotoUx+001x+2T201525530edX/DemoX/Demo_Course
    1382UC3Mx/HGA.1x/1T201525538IDBx/IDB_LSC101x/3T2014
    1390UPValenciaX/RA201x/1T201525755HarvardX/SPU27x/2013_Oct
    1410RiceX/BIOC372.2x/2T201426005HarvardX/PH555x/2014_T2
    1411course-v1:BerkleeX+MB110_4x+1T201526063HarvardX/USW30x/2T2014
    1415DelftX/EconSec101x/1T201526975HarvardX/GSE2x/2T2014
    1430SNUx/SNU216B.226.1x/1T201428203edX/DemoX.1/2014
    1436BerkeleyX/CS_184.1x/3T201429907UWashingtonX/ECFS311x/2T2014
    1438PekingX/18000123x/2015Q130045McGillX/CHEM181x/1T2014
    1438course-v1:Microsoft+DEV201x+2015_T230299MITx/8.01x/2013_SOND_REMOVED
    1442PekingX/02030330X/3T201330569HarvardX/SW12x/2013_SOND
    1448SNUx/SNU446.345.1x/1T201430839WellesleyX/ENG_112x/2014_SOND
    1448HarvardX/HLS1xB/Copyright32112HarvardX/SW12.3x/1T2014
    1452course-v1:BUx+Math226.2x+2T201533096BUx/ARPO222x/3T2014
    1459LouvainX/Louv9x/1T201533115UQx/Write101x/3T2014
    1461course-v1:IMFx+DSAx+2015_SPOC33565MITx/6.00.1x/3T2013
    1474UC3Mx/MMC.1x/1T201534145HarvardX/CB22x/2013_Spring
    1478KIx/KIPractihx/3T201434734course-v1:IDBx+IDB1x+2015_3T
    1480HarvardX/HLS1xA/Copyright35697WellesleyX/HIST229x/2013_SOND
    1484ChalmersX/ChM001x/1T201535988BerkeleyX/ColWri.2.2x/1T2015
    1487course-v1:OECx+2T2015+2T201542663HarvardX/SW12.10x/1T2015
    1505UChicagoX/Ed179x/1T201547226course-v1:BerkeleyX+GG101x-2+1T2015
    1540MITProfessionalX/6.BD_2X/3T201555003BerkeleyX/ColWri2.3x/1T2014
    1553course-v1:LouvainX+Louv5x+3T201556102MITx/7.00x/2013_Spring
    1560HarvardX/1368.3x/2T201559761HarvardX/HDS1544.1x/2013_SOND
    1571course-v1:UWashingtonX+ANTH378x+2T201560490HarvardX/ER22.1x/1T2014
    1602IITBombayX/EE210.1X/3T201563492HarvardX/PH207x/2012_Fall
    1620DelftX/TBP01x/3T201467227MITx/24.00x/2013_SOND
    1633NotreDameX/EG240x/2T201568637MITx/6.00x/2013_Spring
    1667UPValenciaX/EX101x/2T201570250HarvardX/GSE1x/2014_JFMA
    1676UQx/BIOIMG101x/1T201471870UTAustinX/UT.3.01x/2013_Sept
    1698HKUx/HKU03x/1T201576630HarvardX/ER22x/2013_Spring
    1707TsinghuaX/20220332X/3T201378277BerkeleyX/ColWri2.2x/1T2014
    1709HarvardX/HLS1xD/Copyright78903BerkeleyX/ColWri2.1x/3T2013
    1718WageningenX/GFFCx/1T201589639MITx/6.00x/2012_Fall
    1724course-v1:MITx+0.111x+2T2015100492BerkeleyX/GG101x/1T2014
    1726course-v1:UBCx+Water201x_2+2T2015112029UQx/Think101x/1T2014
    1732EPFLx/BIO465x/2013_OND122739course-v1:IDBx+IDB6x+2015_T2
    1740DelftX/RI101x/3T20142198128NULL*

    *Threads are supposed to have course_id's but for some reason, there are some that show up as null. To make sure that the table was not off the mark, queries were made against BerkeleyX/ColWri2.2x/1T2014 to check the integrity of the data.

    db.contents.count({course_id:"BerkeleyX/ColWri2.2x/1T2014"})
78277

    Using the script that collects the list of threads, 32665 threads have been found.

    Update: The null value may have been caused by a bug back in Sept 2013, these entries are not threads. OPS-1062 - Getting issue details... STATUS

  • For comparison, the median course has just 1759 posts.

  • The 95% percentile is 24252

  • The largest forum post is nearly a megabyte. The average is 250 characters long. The median is just 110 characters.

  • According to a 7 day NewRelic table (below), about 20,000 threads, 20,000 responses, and 10,000 comments, are created and about 900 threads/comments are deleted per week.

  • Average comment_count is  1.49194 (when eyeballing, median looks like 1)

  • Largest comment_count is 5907

Using this data, we were able to get an idea of what a course might look like. Most notably, the largest comment_count (comments and responses) for a thread is 5907 and the median seems to be 1. Although that value is an outlier, each course has a "Introduce yourself" topic which would consistently put a thread with a high comment_count in each course. Also, when thinking about mobile usage, push notification could possibly have a different usage pattern where these high comment_count threads could see high spikes in traffic. 

Data Design:

Since the request distribution is very disproportional, the individual endpoint tests are categorized base on how often these requests are hit. 

Important individual endpoint tests:

GET Thread - This test will be for the common case of a thread. ~2000 posts (median) in total will be created as the base course to GET from.
  • Each thread has a ~250character body
  • Of the 1000 threads created
    • 200 have no comments
    • 300 have some sort of flag (abused/voted/following)
    • 100 has a response and a comment
    • 500 have a response
    • 200 will be of the type "question"
  • Of the response heavy threads
    • n threads will be created with a response that has n*20 comments (This could change)

    In addition to this test, different course sizes will be created as well and tested against as we expect the course size to affect the performance. 

GET Comment (Response is depth=1, comment is depth=2) - This test will be for the expected edge cases of a thread. It is important to note that the although the largest comment_count is ~5000, the ratio of responses to comments is unknown. 

  • Each response/comment has a ~250character body
  • Each response will have 20*n comments (could change)

Less important:

POST Thread/Comments - Expected to be constant, this test will just be POSTing threads. 

PATCH Comments/Threads - Will use the same setup as GET thread. This test will modify fields such as "abuse_flagged", "following", "voted", "body"

Insignificant:

DELETE Comment/Thread - These endpoints are hit significantly less than the other endpoints. If running these individually, Threads/Comments will be created to delete. Refer to "Testing Strategy" for more information.

Flowtest:

The flowtest is the testing of all endpoints while trying to simulate the expected usage patterns of production. The data used will be the same as GET Threads where we have ~2000 posts. Threads and Comments will be created and deleted appropriately as all the endpoints are configured to work together.

  • No labels