diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000000..fa1385d99a --- /dev/null +++ b/.gitattributes @@ -0,0 +1 @@ +* -text diff --git a/.gitignore b/.gitignore index 69bc47afdd..b1a36e5f2e 100644 --- a/.gitignore +++ b/.gitignore @@ -44,3 +44,4 @@ node_modules .prereqs_cache autodeploy.properties .ws_migrations_complete +.vagrant/ diff --git a/README.md b/README.md index 4dbf069da3..e533459c8b 100644 --- a/README.md +++ b/README.md @@ -2,8 +2,136 @@ This is the main edX platform which consists of LMS and Studio. See [code.edx.org](http://code.edx.org/) for other parts of the edX code base. -Installation -============ +Installation - The first time +============================= + +The following instructions will help you to download and setup a virtual machine +with a minimal amount of steps, using Vagrant. It is recommended for a first +installation, as it will save you from many of the common pitfalls of the +installation process. + +1. Make sure you have plenty of available disk space, >5GB +2. Install Git: http://git-scm.com/downloads +3. Install VirtualBox: https://www.virtualbox.org/wiki/Download_Old_Builds_4_2 + (you need version 4.2.12, as later/earlier versions might not work well with + Vagrant) +4. Install Vagrant: http://www.vagrantup.com/ (Vagrant 1.2.2 or later) +5. Open a terminal +6. Download the project: `git clone git://github.com/edx/edx-platform.git` +7. Enter the project directory: `cd edx-platform/` +8. (Windows only) Run the commands to + [deal with line endings and symlinks under Windows](https://github.com/edx/edx-platform/wiki/Simplified-install-with-vagrant#dealing-with-line-endings-and-symlinks-under-windows) +9. Start: `vagrant up` + +The last step might require your host machine's administrator password to setup NFS. + +Afterwards, it will download an image, install all the dependencies and configure +the VM. It will take a while, go grab a coffee. + +Once completed, hopefully you should see a "Success!" message indicating that the +installation went fine. (If not, refer to the +[troubleshooting section](https://github.com/edx/edx-platform/wiki/Simplified-install-with-vagrant#troubleshooting).) + +Note: by default, the VM will get the IP `192.168.20.40`. If you need to use a +different IP, you can edit the file `Vagrantfile`. If you have already started the +VM with `vagrant up`, see "Stopping and restarting the VM" below to take the change +into account. + +Accessing the VM +---------------- + +Once the installation is finished, to log into the virtual machine: + +``` +$ vagrant ssh +``` + +Note: This won't work from Windows, install install PuTTY from +http://www.chiark.greenend.org.uk/%7Esgtatham/putty/download.html instead. Then +connect to 127.0.0.1, port 2222, using vagrant/vagrant as a user/password. + +Using edX +--------- + +Once inside the VM, you can start Studio and LMS with the following commands +(from the `/opt/edx/edx-platform` folder): + +Learning management system (LMS): + +``` +$ rake lms[cms.dev,0.0.0.0:8000] +``` + +Studio: + +``` +$ rake cms[dev,0.0.0.0:8001] +``` + +Once started, open the following URLs in your browser: + +* Learning management system (LMS): http://192.168.20.40:8000/ +* Studio (CMS): http://192.168.20.40:8001/ + +You can develop by editing the files directly in the `edx-platform/` directory you +downloaded before, you don't need to connect to the VM to edit them (the VM uses +those files to run edX, mirroring the folder in `/opt/edx/edx-platform`). + +You may also want to create a super-user with: + +``` +$ rake django-admin["createsuperuser"] +``` + +Also note that if you register a new user through the web interface, +the activiation email will be posted to your VM's terminal window (search for +lines similar to): + +``` +Subject: Your account for edX Studio +From: registration@edx.org +``` + +and find the activation URL for the account you've created. + +See the [Frequently Asked Questions](https://github.com/edx/edx-platform/wiki/Frequently-Asked-Questions) +for more usage tips. + +Stopping & starting +------------------- + +To stop the VM (from your `edx-platform/` directory): + +``` +$ vagrant halt +``` + +To restart: + +``` +$ vagrant up +``` + +or, to start without attempting to update the dependencies: + +``` +$ vagrant up --no-provision +``` + +Troubleshooting +--------------- + +If anything doesn't work as expected, see the +[troubleshooting section](https://github.com/edx/edx-platform/wiki/Simplified-install-with-vagrant#troubleshooting). + +Installation - Advanced +======================= + +Note: The following installation instructions are for advanced users & developers +who are familiar with setting up Python, Ruby & node.js virtual environments. +Even if you know what you are doing, edX has a large code base with multiple +dependencies, so you might still want to use the method described above the +first time, as Vagrant helps avoiding issues due to the different environments. There is a `scripts/create-dev-env.sh` that will attempt to set up a development environment. diff --git a/Vagrantfile b/Vagrantfile new file mode 100644 index 0000000000..0d409cc408 --- /dev/null +++ b/Vagrantfile @@ -0,0 +1,33 @@ +# -*- mode: ruby -*- +# vi: set ft=ruby : + +Vagrant.configure("2") do |config| + config.vm.box = "precise32" + config.vm.box_url = "http://files.vagrantup.com/precise32.box" + + config.vm.network :forwarded_port, guest: 8000, host: 9000 + config.vm.network :forwarded_port, guest: 8001, host: 9001 + + # Create a private network, which allows host-only access to the machine + # using a specific IP. + config.vm.network :private_network, ip: "192.168.20.40" + + nfs_setting = RUBY_PLATFORM =~ /darwin/ || RUBY_PLATFORM =~ /linux/ + config.vm.synced_folder ".", "/opt/edx/edx-platform", id: "vagrant-root", :nfs => nfs_setting + + # Make it so that network access from the vagrant guest is able to + # use SSH private keys that are present on the host without copying + # them into the VM. + config.ssh.forward_agent = true + + config.vm.provider :virtualbox do |vb| + # Use VBoxManage to customize the VM. For example to change memory: + vb.customize ["modifyvm", :id, "--memory", "1024"] + + # This setting makes it so that network access from inside the vagrant guest + # is able to resolve DNS using the hosts VPN connection. + vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"] + end + + config.vm.provision :shell, :path => "scripts/vagrant-provisioning.sh" +end diff --git a/rakelib/workspace.rake b/rakelib/workspace.rake index c705899f58..5a928388d5 100644 --- a/rakelib/workspace.rake +++ b/rakelib/workspace.rake @@ -7,10 +7,11 @@ namespace :ws do task :migrate => MIGRATION_MARKER_DIR do Dir['ws_migrations/*'].select{|m| File.executable?(m)}.each do |migration| completion_file = File.join(MIGRATION_MARKER_DIR, File.basename(migration)) - if ! File.exist?(completion_file) + is_excluded = File.basename(migration).start_with?("README") + if ! File.exist?(completion_file) && ! is_excluded sh(migration) File.write(completion_file, "") end end unless SKIP_MIGRATIONS end -end \ No newline at end of file +end diff --git a/scripts/create-dev-env.sh b/scripts/create-dev-env.sh index 56fd744972..3083ddadba 100755 --- a/scripts/create-dev-env.sh +++ b/scripts/create-dev-env.sh @@ -49,8 +49,11 @@ usage() { Usage: $PROG [-c] [-v] [-h] + -y non interactive mode (no prompt, proceed immediately) -c compile scipy and numpy + -n do not attempt to pull edx-platform -s give access to global site-packages for virtualenv + -q be more quiet (removes info at beginning & end) -v set -x + spew -h this @@ -62,7 +65,7 @@ info() { cat</dev/null) 2>/dev/null) || echo -n "" @@ -126,9 +131,16 @@ BASE="${PROJECT_HOME:-$(set_base_default)}" # unless you've already got one set up with virtualenvwrapper. PYTHON_DIR=${WORKON_HOME:-"$HOME/.virtualenvs"} -# RVM defaults its install to ~/.rvm, but use the overridden rvm_path -# if that's what's preferred. -RUBY_DIR=${rvm_path:-"$HOME/.rvm"} +# Find rbenv root (~/.rbenv by default) +if [ -z "${RBENV_ROOT}" ]; then + RBENV_ROOT="${HOME}/.rbenv" +else + RBENV_ROOT="${RBENV_ROOT%/}" +fi +# Let the repo override the version of Ruby to install +if [[ -r $BASE/edx-platform/.ruby-version ]]; then + RUBY_VER=`cat $BASE/edx-platform/.ruby-version` +fi LOG="/var/tmp/install-$(date +%Y%m%d-%H%M%S).log" @@ -149,7 +161,7 @@ if [[ "x$VIRTUAL_ENV" != "x" ]]; then fi # Read arguments -ARGS=$(getopt "cvhs" "$*") +ARGS=$(getopt "cvhsynq" "$*") if [[ $? != 0 ]]; then usage exit 1 @@ -170,6 +182,18 @@ while true; do verbose=true shift ;; + -y) + noninteractive=true + shift + ;; + -q) + quiet=true + shift + ;; + -n) + nopull=true + shift + ;; -h) usage exit 0 @@ -181,7 +205,8 @@ while true; do esac done -cat< $HOME/.rvmrc - fi +if [[ ! -d $RBENV_ROOT ]]; then + output "Installing rbenv" + git clone https://github.com/sstephenson/rbenv.git $RBENV_ROOT +fi +if [[ ! -d $RBENV_ROOT/plugins/ruby-build ]]; then + output "Installing ruby-build" + git clone https://github.com/sstephenson/ruby-build.git $RBENV_ROOT/plugins/ruby-build +fi +shelltype=$(basename $SHELL) +if ! hash rbenv 2>/dev/null; then + output "Adding rbenv to \$PATH in ~/.${shelltype}rc" + echo "export PATH=\"$RBENV_ROOT/bin:\$PATH\"" >> $HOME/.${shelltype}rc + echo 'eval "$(rbenv init -)"' >> $HOME/.${shelltype}rc + export PATH="$RBENV_ROOT/bin:$PATH" + eval "$(rbenv init -)" fi -# rvm has issues in debian family, this is taken from stack overflow -case `uname -s` in - Darwin) - curl -sL get.rvm.io | bash -s -- --version 1.15.7 - ;; - - [Ll]inux) - warning "Setting up rvm on linux. This is a known pain point. If the script fails here - refer to the following stack overflow question: - http://stackoverflow.com/questions/9056008/installed-ruby-1-9-3-with-rvm-but-command-line-doesnt-show-ruby-v/9056395#9056395" - sudo apt-get --purge remove ruby-rvm - sudo rm -rf /usr/share/ruby-rvm /etc/rvmrc /etc/profile.d/rvm.sh - curl -sL https://get.rvm.io | bash -s stable --ruby --autolibs=enable --auto-dotfiles - ;; -esac - - -# Ensure we have RVM available as a shell function so that it can mess -# with the environment and set everything up properly. The RVM install -# process adds this line to login scripts, so this shouldn't be necessary -# for the user to do each time. -if [[ `type -t rvm` != "function" ]]; then - source $RUBY_DIR/scripts/rvm +if [[ ! -d $RBENV_ROOT/versions/$RUBY_VER ]]; then + output "Installing Ruby $RUBY_VER" + rbenv install $RUBY_VER + rbenv global $RUBY_VER fi -# Ruby doesn't like to build with clang, which is the default on OS X, so -# use gcc instead. This may not work, since if your gcc was installed with -# XCode 4.2 or greater, you have an LLVM-based gcc, which also doesn't -# always play nicely with Ruby, though it seems to be better than clang. -# You may have to install apple-gcc42 using Homebrew if this doesn't work. -# See `rvm requirements` for more information. -case `uname -s` in - Darwin) - export CC=gcc - ;; -esac - -# Let the repo override the version of Ruby to install -if [[ -r $BASE/edx-platform/.ruby-version ]]; then - RUBY_VER=`cat $BASE/edx-platform/.ruby-version` +if ! hash bundle 2>/dev/null; then + output "Installing gem bundler" + gem install bundler fi - -# Current stable version of RVM (1.19.0) requires the following to build Ruby: -# -# autoconf automake libtool pkg-config libyaml libxml2 libxslt libksba openssl -# -# If we decide to upgrade from the current version (1.15.7), can run -# -# LESS="-E" rvm install $RUBY_VER --autolibs=3 --with-readline -# -# to have RVM look for a package manager like Homebrew and install any missing -# libs automatically. RVM's --autolibs flag defaults to 2, which will fail if -# any required libs are missing. -LESS="-E" rvm install $RUBY_VER --with-readline - -# Create the "edx" gemset -rvm use "$RUBY_VER@edx-platform" --create -rvm rubygems latest - -output "Installing gem bundler" -gem install bundler +rbenv rehash output "Installing ruby packages" bundle install --gemfile $BASE/edx-platform/Gemfile - # Install Python virtualenv - output "Installing python virtualenv" case `uname -s` in @@ -426,14 +404,14 @@ fi # Create edX virtualenv and link it to repo # virtualenvwrapper automatically sources the activation script if [[ $systempkgs ]]; then - mkvirtualenv -a "$HOME/.virtualenvs" --system-site-packages edx-platform || { + mkvirtualenv -q -a "$HOME/.virtualenvs" --system-site-packages edx-platform || { error "mkvirtualenv exited with a non-zero error" return 1 } else # default behavior for virtualenv>1.7 is # --no-site-packages - mkvirtualenv -a "$HOME/.virtualenvs" edx-platform || { + mkvirtualenv -q -a "$HOME/.virtualenvs" edx-platform || { error "mkvirtualenv exited with a non-zero error" return 1 } @@ -447,8 +425,8 @@ SCIPY_VER="0.10.1" if [[ -n $compile ]]; then output "Downloading numpy and scipy" - curl -sL -o numpy.tar.gz http://downloads.sourceforge.net/project/numpy/NumPy/${NUMPY_VER}/numpy-${NUMPY_VER}.tar.gz - curl -sL -o scipy.tar.gz http://downloads.sourceforge.net/project/scipy/scipy/${SCIPY_VER}/scipy-${SCIPY_VER}.tar.gz + curl -sSL -o numpy.tar.gz http://downloads.sourceforge.net/project/numpy/NumPy/${NUMPY_VER}/numpy-${NUMPY_VER}.tar.gz + curl -sSL -o scipy.tar.gz http://downloads.sourceforge.net/project/scipy/scipy/${SCIPY_VER}/scipy-${SCIPY_VER}.tar.gz tar xf numpy.tar.gz tar xf scipy.tar.gz rm -f numpy.tar.gz scipy.tar.gz @@ -467,7 +445,7 @@ DISTRIBUTE_VER="0.6.28" output "Building Distribute" SITE_PACKAGES="$HOME/.virtualenvs/edx-platform/lib/python2.7/site-packages" cd "$SITE_PACKAGES" -curl -OL http://pypi.python.org/packages/source/d/distribute/distribute-${DISTRIBUTE_VER}.tar.gz +curl -sSLO http://pypi.python.org/packages/source/d/distribute/distribute-${DISTRIBUTE_VER}.tar.gz tar -xzvf distribute-${DISTRIBUTE_VER}.tar.gz cd distribute-${DISTRIBUTE_VER} python setup.py install @@ -503,7 +481,6 @@ pip install -r $BASE/edx-platform/requirements/edx/pre.txt output "Installing edX requirements" # Install prereqs cd $BASE/edx-platform -rvm use "$RUBY_VER@edx-platform" rake install_prereqs # Final dependecy @@ -518,7 +495,7 @@ mkdir -p "$BASE/log" mkdir -p "$BASE/db" mkdir -p "$BASE/data" -rake django-admin[syncdb] +rake django-admin[syncdb,lms,dev,--noinput] rake django-admin[migrate] rake cms:update_templates # Configure Git @@ -529,15 +506,15 @@ git config --global push.default current ### DONE -cat< +# +# Authors: Xavier Antoviaque +# David Baumgold +# +# This software's license gives you freedom; you can copy, convey, +# propagate, redistribute and/or modify this program under the terms of +# the GNU Affero General Public License (AGPL) as published by the Free +# Software Foundation (FSF), either version 3 of the License, or (at your +# option) any later version of the AGPL published by the FSF. +# +# This program is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero +# General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with this program in a file in the toplevel directory called +# "AGPLv3". If not, see . + + +############################################################################### + +# vagrant-provisioning.sh: +# +# Script to setup base environment on Vagrant, based on `precise32` image +# Runs ./scripts/create-dev-env.sh for the actual setup +# +# This script is ran by `$ vagrant up`, see the README for more explanations + + +# APT - Packages ############################################################## + +apt-get update +apt-get install -y python-software-properties vim + + +# Curl - No progress bar ###################################################### + +[[ -f ~vagrant/.curlrc ]] || echo "silent show-error" > ~vagrant/.curlrc +chown vagrant.vagrant ~vagrant/.curlrc + + +# SSH - Known hosts ########################################################### + +# Github +([[ -f ~vagrant/.ssh/known_hosts ]] && grep "zBX7bKA= ssh" ~vagrant/.ssh/known_hosts) || { + echo "|1|4DtBcMsTM4zgl/jTS7h3ZkmS/Vc=|XkRnn2xEhr8ixOxeskJAzBX7bKA= ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==" >> ~vagrant/.ssh/known_hosts +} +([[ -f ~vagrant/.ssh/known_hosts ]] && grep "jO3J5bvw= ssh" ~vagrant/.ssh/known_hosts) || { + echo "|1|9rANf/qOAPgKH/TXpGuZCAgGxMs=|x9VYWEDI8kiotbhhNXqjO3J5bvw= ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==" >> ~vagrant/.ssh/known_hosts +} +chown vagrant.vagrant ~vagrant/.ssh/known_hosts + + +# edX - Development environment ############################################### + +# Node modules require a filesystem with symlinks (Windows support) +mkdir -p /opt/edx/node_modules /opt/edx/edx-platform/node_modules +mount -o bind /opt/edx/node_modules /opt/edx/edx-platform/node_modules + +# Force rechecking all prerequisites (could have been fetched outside of the VM) +rm -rf /opt/edx/edx-platform/.prereqs_cache + +# Permissions +chown vagrant.vagrant /opt/edx /opt/edx/node_modules /opt/edx/edx-platform/node_modules + +# For convenience with `vagrant ssh`, the `edx-platform` virtualenv is always +# loaded after the first run, so we need to deactivate that behavior to run +# `create-dev-env.sh`. +[[ -f ~vagrant/.bash_profile ]] && { + mv ~vagrant/.bash_profile ~vagrant/.bash_profile.bak +} +sudo -u vagrant -i bash -c "cd /opt/edx/edx-platform && PROJECT_HOME=/opt/edx ./scripts/create-dev-env.sh -ynq" + +# Load .bashrc ################################################################ +([[ -f ~vagrant/.bash_profile ]] && grep ".bashrc" ~vagrant/.bash_profile) || { + echo -e "\n. /home/vagrant/.bashrc\n" >> ~vagrant/.bash_profile +} + + +# Virtualenv - Always load #################################################### + +([[ -f ~vagrant/.bash_profile ]] && grep "edx-platform/bin/activate" ~vagrant/.bash_profile) || { + echo -e "\n. /home/vagrant/.virtualenvs/edx-platform/bin/activate\n" >> ~vagrant/.bash_profile +} + + +# Directory ################################################################### + +grep "cd /opt/edx/edx-platform" ~vagrant/.bash_profile || { + echo -e "\ncd /opt/edx/edx-platform\n" >> ~vagrant/.bash_profile +} + + +# End ######################################################################### + +cat << EOF +============================================================================== +Success! +============================================================================== + +Now, from the virtual machine (connect with "vagrant ssh" if vagrant didn't +log you in already), you can start Studio & LMS with the following commands: + +- Learning management system (LMS): + $ rake lms[cms.dev,0.0.0.0:8000] + + => http://192.168.20.40:8000/ + +- Studio: + $ rake cms[dev,0.0.0.0:8001] + + => http://192.168.20.40:8001/ + + +See the README for details. + + +EOF + +