From 457b678af714dd50ffe56e1efdf95101aa1409ae Mon Sep 17 00:00:00 2001 From: David Baumgold Date: Thu, 2 May 2013 13:32:25 -0400 Subject: [PATCH 01/17] Wrote a proper README file So that no one else will have to go through what I went through. Hopefully. --- README | 1 - README.md | 120 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 120 insertions(+), 1 deletion(-) delete mode 100644 README create mode 100644 README.md diff --git a/README b/README deleted file mode 100644 index 2ed50ba063..0000000000 --- a/README +++ /dev/null @@ -1 +0,0 @@ -See doc/ for documentation. diff --git a/README.md b/README.md new file mode 100644 index 0000000000..1642539c6b --- /dev/null +++ b/README.md @@ -0,0 +1,120 @@ +This is edX, a platform for online higher education. The project is primarily +written in [Python](http://python.org/), using the +[Django](https://www.djangoproject.com/) framework. We also use some +[Ruby](http://www.ruby-lang.org/) and some [NodeJS](http://nodejs.org/). + +Installation +============ +The installation process is a bit messy at the moment. Here's a high-level +overview of what you should do to get started. + +**TLDR:** There is a `create-dev-env.sh` script that will attempt to set all +of this up for you. If you're in a hurry, run that script. Otherwise, I suggest +that you understand what the script is doing, and why, by reading this document. + +Directory Hierarchy +------------------- +This code assumes that it is checked out in a directory that has three sibling +directories: `data` (used for application data?), `db` (used to hold a +[sqlite](https://sqlite.org/) database), and `log` (used to hold logs). If you +clone the repository into a directory called `edx` inside of a directory +called `dev`, here's an example of how the directory hierarchy should look: + + * dev + \ + * data + * db + * log + * edx + \ + README.md + +Language Runtimes +----------------- +You'll need to be sure that you have Python 2.7, Ruby 1.9.3, and NodeJS +(latest stable) installed on your system. Some of these you can install +using your system's package manager: [homebrew](http://mxcl.github.io/homebrew/) +for Mac, [apt](http://wiki.debian.org/Apt) for Debian-based systems +(including Ubuntu), [rpm](http://www.rpm.org/) or [yum](http://yum.baseurl.org/) +for Red Hat based systems (including CentOS). + +If your system's package manager gives you the wrong version of a language +runtime, then you'll need to use a versioning tool to install the correct version. +Usually, you'll need to do this for Ruby: you can use +[`rbenv`](https://github.com/sstephenson/rbenv) or [`rvm`](https://rvm.io/), but +typically `rbenv` is simpler. For Python, you can use +[`pythonz`](http://saghul.github.io/pythonz/), +and for Node, you can use [`nvm`](https://github.com/creationix/nvm). + +Virtual Environments +-------------------- +Often, different projects will have conflicting dependencies: for example, two +projects depending on two different, incompatible versions of a library. Clearly, +you can't have both versions installed and used on your machine simultaneously. +Virtual environments were created to solve this problem: by installing libraries +into an isolated environment, only projects that live inside the environment +will be able to see and use those libraries. Incompatible dependencies? Use +different virtual environments, and your problem is solved. + +Once again, each language has a different implementation. Python has +[virtualenv](http://www.virtualenv.org/), Ruby has +[`bundler`](http://gembundler.com/), and Node has +[`nave`](https://github.com/isaacs/nave). For each language, decide +if you want to use a virtual environment, or if you want to install all the +language dependencies globally (and risk conflicts). I suggest you start with +installing things globally until and unless things break; you can always +switch over to a virtual environment later on. + +Language Packages +----------------- +The Python libraries we use are listed in `requirements.txt`. The Ruby libraries +we use are listed in `Gemfile`. The Node libraries we use are listed in +`packages.json`. Python has a library installer called +[`pip`](http://www.pip-installer.org/), Ruby has a library installer called +[`gem`](https://rubygems.org/) (or `bundle` if you're using a virtual +environment), and Node has a library installer called +[`npm`](https://npmjs.org/). +Once you've got your languages and virtual environments set up, install +the libraries like so: + + $ pip install -r requirements.txt + $ bundle install + $ npm install + +Configuring Your Project +------------------------ +We use [`rake`](http://rake.rubyforge.org/) to execute common tasks in our +project. The `rake` tasks are defined in the `rakefile`, or you can run `rake -T` +to view a summary. + +Before you run your project, you need to create a sqlite database and create +tables in that database. Fortunately, `rake` will do it for you! Just run: + + $ rake django-admin[syncdb] + $ rake django-admin[migrate] + +If you are running these commands using the [`zsh`](http://www.zsh.org/) shell, +zsh will assume that you are doing +[shell globbing](https://en.wikipedia.org/wiki/Glob_(programming)), search for +a file in your directory named `django-adminsyncdb` or `django-adminmigrate`, +and fail. To fix this, just surround the argument with quotation marks, so that +you're running `rake "django-admin[syncdb]"`. + +Run Your Project +---------------- +To *finally* get up and running, just run: + + $ rake cms + +And `rake` will start up your Django project on the localhost, port 8001. To +view your running project, type `127.0.0.1:8001` into your web browser, and +you should see edX in all its glory! + + +Further Documentation +===================== +Once you've got your project up and running, you can check out the `docs` +directory to see more documentation about how edX is structured. + + + From a58f39695aa55bf80012f9e8a920c9724d456b04 Mon Sep 17 00:00:00 2001 From: David Baumgold Date: Thu, 2 May 2013 13:50:22 -0400 Subject: [PATCH 02/17] Remove install.txt, because all that information is in the README.md file now --- install.txt | 74 ----------------------------------------------------- 1 file changed, 74 deletions(-) delete mode 100644 install.txt diff --git a/install.txt b/install.txt deleted file mode 100644 index 801036af6b..0000000000 --- a/install.txt +++ /dev/null @@ -1,74 +0,0 @@ -This document describes how to set up the MITx development environment -for both Linux (Ubuntu) and MacOS (OSX Lion). - -There is also a script "create-dev-env.sh" that automates these steps. - -1) Make an mitx_all directory and clone the repos - (download and install git and mercurial if you don't have them already) - - mkdir ~/mitx_all - cd ~/mitx_all - git clone git@github.com:MITx/mitx.git - hg clone ssh://hg-content@gp.mitx.mit.edu/data - -2) Install OSX dependencies (Mac users only) - - a) Install the brew utility if necessary - /usr/bin/ruby -e "$(curl -fsSL https://raw.github.com/mxcl/homebrew/master/Library/Contributions/install_homebrew.rb)" - - b) Install the brew package list - cat ~/mitx_all/mitx/brew-formulas.txt | xargs brew install - - c) Install python pip if necessary - sudo easy_install pip - - d) Install python virtualenv if necessary - sudo pip install virtualenv virtualenvwrapper - - e) Install coffee script - curl http://npmjs.org/install.sh | sh - npm install -g coffee-script - -3) Install Ubuntu dependencies (Linux users only) - - sudo apt-get install curl python-virtualenv build-essential python-dev gfortran liblapack-dev libfreetype6-dev libpng12-dev libxml2-dev libxslt-dev yui-compressor coffeescript - - -4) Install rvm, ruby, and libraries - - echo "export rvm_path=$HOME/mitx_all/ruby" > $HOME/.rvmrc - curl -sL get.rvm.io | bash -s stable - source ~/mitx_all/ruby/scripts/rvm - rvm install 1.9.3 - gem install bundler - cd ~/mitx_all/mitx - bundle install - -5) Install python libraries - - source ~/mitx_all/python/bin/activate - cd ~/mitx_all - pip install -r mitx/pre-requirements.txt - pip install -r mitx/requirements.txt - -6) Create log and db dirs - - mkdir ~/mitx_all/log - mkdir ~/mitx_all/db - -7) Start the dev server - - To start using Django you will need - to activate the local Python and Ruby - environment: - - $ source ~/mitx_all/ruby/scripts/rvm - $ source ~/mitx_all/python/bin/activate - - To initialize and start a local instance of Django: - - $ cd ~/mitx_all/mitx - $ django-admin.py syncdb --settings=envs.dev --pythonpath=. - $ django-admin.py migrate --settings=envs.dev --pythonpath=. - $ django-admin.py runserver --settings=envs.dev --pythonpath=. - From a0727ac2261ce5e557a22ce504b20e64abdf1ede Mon Sep 17 00:00:00 2001 From: David Baumgold Date: Thu, 2 May 2013 13:53:42 -0400 Subject: [PATCH 03/17] Virtualenv is a tool, and should be surrounded by backticks --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 1642539c6b..db839cfbfb 100644 --- a/README.md +++ b/README.md @@ -57,7 +57,7 @@ will be able to see and use those libraries. Incompatible dependencies? Use different virtual environments, and your problem is solved. Once again, each language has a different implementation. Python has -[virtualenv](http://www.virtualenv.org/), Ruby has +[`virtualenv`](http://www.virtualenv.org/), Ruby has [`bundler`](http://gembundler.com/), and Node has [`nave`](https://github.com/isaacs/nave). For each language, decide if you want to use a virtual environment, or if you want to install all the From 1bda218e17d2ca6d7212f947e7544c85cce9535c Mon Sep 17 00:00:00 2001 From: David Baumgold Date: Thu, 2 May 2013 16:27:19 -0400 Subject: [PATCH 04/17] Clarify purpose of `data` dir --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index db839cfbfb..498277d646 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ that you understand what the script is doing, and why, by reading this document. Directory Hierarchy ------------------- This code assumes that it is checked out in a directory that has three sibling -directories: `data` (used for application data?), `db` (used to hold a +directories: `data` (used for XML course data), `db` (used to hold a [sqlite](https://sqlite.org/) database), and `log` (used to hold logs). If you clone the repository into a directory called `edx` inside of a directory called `dev`, here's an example of how the directory hierarchy should look: From fccee7a1e320be73b8506d71f5a507bc83ce60e4 Mon Sep 17 00:00:00 2001 From: David Baumgold Date: Thu, 2 May 2013 16:29:26 -0400 Subject: [PATCH 05/17] Node does virtual environments using npm --- README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 498277d646..7d8a61e7b4 100644 --- a/README.md +++ b/README.md @@ -58,12 +58,12 @@ different virtual environments, and your problem is solved. Once again, each language has a different implementation. Python has [`virtualenv`](http://www.virtualenv.org/), Ruby has -[`bundler`](http://gembundler.com/), and Node has -[`nave`](https://github.com/isaacs/nave). For each language, decide -if you want to use a virtual environment, or if you want to install all the -language dependencies globally (and risk conflicts). I suggest you start with -installing things globally until and unless things break; you can always -switch over to a virtual environment later on. +[`bundler`](http://gembundler.com/), and Node's virtual environment support +is built into [`npm`](https://npmjs.org/), it's library management tool. +For each language, decide if you want to use a virtual environment, or if you +want to install all the language dependencies globally (and risk conflicts). +I suggest you start with installing things globally until and unless things +break; you can always switch over to a virtual environment later on. Language Packages ----------------- From 06ade12f6a782e5cb0ca2f47d4f1fde16554b1c6 Mon Sep 17 00:00:00 2001 From: David Baumgold Date: Thu, 2 May 2013 16:42:51 -0400 Subject: [PATCH 06/17] Also need to install Mongo --- README.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/README.md b/README.md index 7d8a61e7b4..05f1eff406 100644 --- a/README.md +++ b/README.md @@ -81,6 +81,18 @@ the libraries like so: $ bundle install $ npm install +Other Dependencies +------------------ +You'll also need to install [MongoDB](http://www.mongodb.org/), since our +application uses it in addition to sqlite. You can install it through your +system package manager, and I suggest that you configure it to start +automatically when you boot up your system, so that you never have to worry +about it again. For Mac, use +[`launchd`](https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man8/launchd.8.html) +(running `brew info mongodb` will give you some commands you can copy-paste.) +For Linux, you can use [`upstart`](http://upstart.ubuntu.com/), `chkconfig`, +or any other process management tool. + Configuring Your Project ------------------------ We use [`rake`](http://rake.rubyforge.org/) to execute common tasks in our From 03470fc48e09eef7b2ba7ba72013fd6799114705 Mon Sep 17 00:00:00 2001 From: Ned Batchelder Date: Thu, 2 May 2013 09:43:18 -0400 Subject: [PATCH 07/17] Add a mention of create-dev-env.sh --- doc/development.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/doc/development.md b/doc/development.md index 95cc32329c..a6a1de4ef7 100644 --- a/doc/development.md +++ b/doc/development.md @@ -31,6 +31,14 @@ Check out the course data directories that you want to work with into the rake resetdb +## Installing + +To create your development environment, run the shell script in the root of +the repo: + + create-dev-env.sh + + ## Starting development servers Both the LMS and Studio can be started using the following shortcut tasks From 98243b2b53f77aca569e50b8d7936f4ae01148d5 Mon Sep 17 00:00:00 2001 From: David Baumgold Date: Fri, 3 May 2013 10:55:28 -0400 Subject: [PATCH 08/17] Document django-admin[update_templates] step --- README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 05f1eff406..2c4d76f8fd 100644 --- a/README.md +++ b/README.md @@ -99,11 +99,13 @@ We use [`rake`](http://rake.rubyforge.org/) to execute common tasks in our project. The `rake` tasks are defined in the `rakefile`, or you can run `rake -T` to view a summary. -Before you run your project, you need to create a sqlite database and create -tables in that database. Fortunately, `rake` will do it for you! Just run: +Before you run your project, you need to create a sqlite database, create +tables in that database, run database migrations, and populate templates for +CMS templates. Fortunately, `rake` will do all of this for you! Just run: $ rake django-admin[syncdb] $ rake django-admin[migrate] + $ rake django-admin[update_templates] If you are running these commands using the [`zsh`](http://www.zsh.org/) shell, zsh will assume that you are doing From fa87cccb9ba7cf99fda9a9785a52b18d866efc95 Mon Sep 17 00:00:00 2001 From: David Baumgold Date: Fri, 3 May 2013 11:16:05 -0400 Subject: [PATCH 09/17] Document old-style `rake lms` command --- README.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/README.md b/README.md index 2c4d76f8fd..9f5d7ba1c1 100644 --- a/README.md +++ b/README.md @@ -124,6 +124,13 @@ And `rake` will start up your Django project on the localhost, port 8001. To view your running project, type `127.0.0.1:8001` into your web browser, and you should see edX in all its glory! +If you need to run old XML-only LMS (which doesn't use the database), run this +instead: + + $ rake lms + +And `rake` will start up the old project on localhost port 8000. + Further Documentation ===================== From 2a09f6bac10fa3bb04479dbb0c5429b2b82d5dd8 Mon Sep 17 00:00:00 2001 From: David Baumgold Date: Fri, 3 May 2013 11:28:05 -0400 Subject: [PATCH 10/17] Clarified based on @shnayder's comment --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 9f5d7ba1c1..80e534730d 100644 --- a/README.md +++ b/README.md @@ -124,8 +124,8 @@ And `rake` will start up your Django project on the localhost, port 8001. To view your running project, type `127.0.0.1:8001` into your web browser, and you should see edX in all its glory! -If you need to run old XML-only LMS (which doesn't use the database), run this -instead: +If you need to run old XML-only LMS (which doesn't use the mongo database for +course content), run this instead: $ rake lms From ee6d68e2e0ecc701f3f97c2110cc671c67c58ed3 Mon Sep 17 00:00:00 2001 From: David Baumgold Date: Fri, 3 May 2013 14:41:58 -0400 Subject: [PATCH 11/17] grammar --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 80e534730d..5ac5f91632 100644 --- a/README.md +++ b/README.md @@ -59,7 +59,7 @@ different virtual environments, and your problem is solved. Once again, each language has a different implementation. Python has [`virtualenv`](http://www.virtualenv.org/), Ruby has [`bundler`](http://gembundler.com/), and Node's virtual environment support -is built into [`npm`](https://npmjs.org/), it's library management tool. +is built into [`npm`](https://npmjs.org/), its library management tool. For each language, decide if you want to use a virtual environment, or if you want to install all the language dependencies globally (and risk conflicts). I suggest you start with installing things globally until and unless things From 234ca0e07616490a0b7f5bc565036abebe4cf2e4 Mon Sep 17 00:00:00 2001 From: David Baumgold Date: Fri, 3 May 2013 14:51:56 -0400 Subject: [PATCH 12/17] Responding to @jzoldak's comments --- README.md | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 5ac5f91632..f1532d53b6 100644 --- a/README.md +++ b/README.md @@ -116,21 +116,27 @@ you're running `rake "django-admin[syncdb]"`. Run Your Project ---------------- -To *finally* get up and running, just run: +edX has two components: Studio, the course authoring system; and the LMS +(leaning management system) used by students. These two systems communicate +through the MongoDB database, which stores course information. + +To run Studio, run: $ rake cms -And `rake` will start up your Django project on the localhost, port 8001. To -view your running project, type `127.0.0.1:8001` into your web browser, and -you should see edX in all its glory! +To run the LMS, run: -If you need to run old XML-only LMS (which doesn't use the mongo database for -course content), run this instead: + $ rake lms[cms.dev] - $ rake lms +Studio runs on port 8001, while LMS runs on port 8000, so you can run both of +these commands simultaneously, using two different terminal windows. To view +Studio, visit `127.0.0.1:8001` in your web browser; to view the LMS, visit +`127.0.0.1:8000`. -And `rake` will start up the old project on localhost port 8000. +There's also an older version of the LMS that saves its information in XML files +in the `data` directory, instead of in Mongo. To run this older version, run: +$ rake lms Further Documentation ===================== From 7b3646b39c798dc6642bcfae61b2c35b9d13a236 Mon Sep 17 00:00:00 2001 From: David Baumgold Date: Mon, 6 May 2013 11:05:06 -0400 Subject: [PATCH 13/17] Need to install `pre-requirements.txt` first --- README.md | 3 ++- pre-requirements.txt | 8 ++++++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index f1532d53b6..76dbe4b150 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ This is edX, a platform for online higher education. The project is primarily written in [Python](http://python.org/), using the -[Django](https://www.djangoproject.com/) framework. We also use some +[Django](https://www.djangoproject.com/) framework. We also use some [Ruby](http://www.ruby-lang.org/) and some [NodeJS](http://nodejs.org/). Installation @@ -77,6 +77,7 @@ environment), and Node has a library installer called Once you've got your languages and virtual environments set up, install the libraries like so: + $ pip install -r pre-requirements.txt $ pip install -r requirements.txt $ bundle install $ npm install diff --git a/pre-requirements.txt b/pre-requirements.txt index 7ecead0ce7..d39199a741 100644 --- a/pre-requirements.txt +++ b/pre-requirements.txt @@ -1,2 +1,10 @@ +# We use `scipy` in our project, which relies on `numpy`. `pip` apparently +# installs packages in a two-step process, where it will first try to build +# all packages, and then try to install all packages. As a result, if we simply +# added these packages to the top of `requirements.txt`, `pip` would try to +# build `scipy` before `numpy` has been installed, and it would fail. By +# separating this out into a `pre-requirements.txt` file, we can make sure +# that `numpy` is built *and* installed before we try to build `scipy`. + numpy==1.6.2 distribute>=0.6.28 From 683906cdaf050ee94fa1e0b3972d6853d1dff8b5 Mon Sep 17 00:00:00 2001 From: David Baumgold Date: Mon, 6 May 2013 11:17:28 -0400 Subject: [PATCH 14/17] "One again" -> "Remember" --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 76dbe4b150..cd6a95cb0a 100644 --- a/README.md +++ b/README.md @@ -56,7 +56,7 @@ into an isolated environment, only projects that live inside the environment will be able to see and use those libraries. Incompatible dependencies? Use different virtual environments, and your problem is solved. -Once again, each language has a different implementation. Python has +Remember, each language has a different implementation. Python has [`virtualenv`](http://www.virtualenv.org/), Ruby has [`bundler`](http://gembundler.com/), and Node's virtual environment support is built into [`npm`](https://npmjs.org/), its library management tool. From b99584287484bf2c8e21ca064388282438d509df Mon Sep 17 00:00:00 2001 From: David Baumgold Date: Mon, 6 May 2013 11:21:05 -0400 Subject: [PATCH 15/17] LMS is not a leaning management system That would be a chair. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index cd6a95cb0a..27494fe6fa 100644 --- a/README.md +++ b/README.md @@ -118,7 +118,7 @@ you're running `rake "django-admin[syncdb]"`. Run Your Project ---------------- edX has two components: Studio, the course authoring system; and the LMS -(leaning management system) used by students. These two systems communicate +(learning management system) used by students. These two systems communicate through the MongoDB database, which stores course information. To run Studio, run: From 58bc0452eba5dc60abd69b63011b472c94cf6195 Mon Sep 17 00:00:00 2001 From: David Baumgold Date: Mon, 6 May 2013 11:25:52 -0400 Subject: [PATCH 16/17] Describe edX better --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 27494fe6fa..4ffbaf9642 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -This is edX, a platform for online higher education. The project is primarily +This is edX, a platform for online course delivery. The project is primarily written in [Python](http://python.org/), using the [Django](https://www.djangoproject.com/) framework. We also use some [Ruby](http://www.ruby-lang.org/) and some [NodeJS](http://nodejs.org/). From a779f6271692c1b68ea602fb972e1bc3aff065b8 Mon Sep 17 00:00:00 2001 From: David Baumgold Date: Mon, 6 May 2013 11:27:06 -0400 Subject: [PATCH 17/17] Make rhetorical question a bit clearer --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 4ffbaf9642..ec17d7c9a4 100644 --- a/README.md +++ b/README.md @@ -53,7 +53,7 @@ projects depending on two different, incompatible versions of a library. Clearly you can't have both versions installed and used on your machine simultaneously. Virtual environments were created to solve this problem: by installing libraries into an isolated environment, only projects that live inside the environment -will be able to see and use those libraries. Incompatible dependencies? Use +will be able to see and use those libraries. Got incompatible dependencies? Use different virtual environments, and your problem is solved. Remember, each language has a different implementation. Python has