Notes on using mongodb backed LMS and CMS

These are some random notes for developers, on how things are stored in mongodb, and how to debug mongodb data.

Databases

Two mongodb databases are used:

xmodule: stores module definitions and metadata (modulestore)
xcontent: stores filesystem content, like PDF files

modulestore documents are stored with an _id which has fields like this:

{"_id": {"tag":"i4x","org":"HarvardX","course":"CS50x","category":"chapter","name":"Week_1","revision":null}}

Document fields

Problems

Here is an example showing the fields available in problem documents:

{
"_id" : {
	"tag" : "i4x",
	"org" : "MITx",
	"course" : "6.00x",
	"category" : "problem",
	"name" : "ps03:ps03-Hangman_part_2_The_Game",
	"revision" : null
},
"definition" : {
	"data" : " ..."
},
"metadata" : {
	"display_name" : "Hangman Part 2: The Game",
	"attempts" : "30",
	"title" : "Hangman, Part 2",
	"data_dir" : "6.00x",
	"type" : "lecture"
}
}

Sample interaction with mongodb

"mongo"
"use xmodule"
"show collections" should give "modulestore" and "system.indexes"
'db.modulestore.find( {"_id.org": "MITx"} )' will produce a list of all MITx course documents
'db.modulestore.find( {"_id.org": "MITx", "_id.category": "problem"} )' will produce a list of all problems in MITx courses

Example query for finding all files with "image" in the filename:

use xcontent
db.fs.files.find({'filename': /image/ } )
db.fs.files.find({'filename': /image/ } ).count()

Debugging the mongodb contents

A convenient tool is http://phpmoadmin.com/ (needs php)

Under ubuntu, do:

apt-get install php5-fpm php-pear
pecl install mongo
edit /etc/php5/fpm/php.ini to add "extension=mongo.so"
/etc/init.d/php5-fpm restart

and also setup nginx to run php through fastcgi.

Backing up mongodb

mogodump (dumps all dbs)
mongodump --collection modulestore --db xmodule (dumps just xmodule/modulestore)
mongodump -d xmodule -q '{"_id.org": "MITx"}' (dumps just MITx documents in xmodule)
mongodump -q '{"_id.org": "MITx"}' (dumps all MITx documents)

Deleting course content

Use "remove" instead of "find":

db.modulestore.remove( {"_id.course": "8.01greytak"})

Finding useful information from the mongodb modulestore

Organizations

db.modulestore.distinct( "_id.org") [ "HarvardX", "MITx", "edX", "edx" ]
Courses

db.modulestore.distinct( "_id.course") [ "CS50x", "PH207x", "3.091x", "6.002x", "6.00x", "8.01esg", "8.01rq_MW", "8.02teal", "8.02x", "edx4edx", "toy", "templates" ]
Find a problem which has the word "quantum" in its definition

db.modulestore.findOne( {"definition.data":/quantum/})n
Find Location for all problems with the word "quantum" in its definition

db.modulestore.find( {"definition.data":/quantum/}, {'_id':1})
Number of problems in each course

db.runCommand({ mapreduce: "modulestore", query: { '_id.category': 'problem' }, map: function(){ emit(this._id.course, {count:1}); }, reduce: function(key, values){ var result = {count:0}; values.forEach(function(value) { result.count += value.count; }); return result; }, out: 'pbyc', verbose: true });

produces:

db.pbyc.find() { "_id" : "3.091x", "value" : { "count" : 184 } } { "_id" : "6.002x", "value" : { "count" : 176 } } { "_id" : "6.00x", "value" : { "count" : 147 } } { "_id" : "8.01esg", "value" : { "count" : 184 } } { "_id" : "8.01rq_MW", "value" : { "count" : 73 } } { "_id" : "8.02teal", "value" : { "count" : 5 } } { "_id" : "8.02x", "value" : { "count" : 99 } } { "_id" : "PH207x", "value" : { "count" : 25 } } { "_id" : "edx4edx", "value" : { "count" : 50 } } { "_id" : "templates", "value" : { "count" : 11 } }

3.9 KiB Raw Blame History