The goal of the project is to gain hands-on experience in building and deploying a scalable web service on the Internet. Using the latest web technologies while learning how to tackle the scalability and fault-tolerance concerns. This is a “learn by doing” course: the course project will form the primary focus of the course with the lectures and discussion of research papers providing background material. Each project will be conducted in an agile team where students will build their own scalable, redundant web site using fundamental web technologies and the Ruby on Rails framework.
Throughout the project, you might find it helpful to go through a ruby on rails tutorial. The Ruby on Rails Tutorial (see sidebar) is an amazing resource. Rail’s own tutorial is quite good as well: Getting Started with Rails
Submit a report describing your project, and your data-driven approach to load testing. Consider answering the following questions:
Record a video presentation of your final project to share with the class. The video must be under 10 minutes in duration.
The video should emphasize your key features of your application.
All sprints end and begin with each week’s lab session, except for the last sprint which does not have an ending lab session.
At the end of each sprint you will:
N user stories, where N is the number of people on your team.Assuming you have docker installed locally, follow the steps below to create a new rails project using docker.
mkdir TEAMNAME
cd TEAMNAME
Assuming you have rails >=7.1.2 installed on your system:
rails new . --force --database=postgresql --skip-action-cable --skip-turbolinks --skip-jbuilder --skip-system-test
This should generate a Dockerfile, Gemfile and Gemfile.lock in addition to a standard rails project.
touch docker-compose.yml
Now you should have the Dockerfile, docker-compose.yml, Gemfile and Gemfile.lock files at the root of your project. (You can tough the files if they were not created automatically)
Copy the following contents into Dockerfile (replace all content):
# syntax = docker/dockerfile:1
# Make sure RUBY_VERSION matches the Ruby version in .ruby-version and Gemfile
ARG RUBY_VERSION=3.2.2
FROM ruby:$RUBY_VERSION
# throw errors if Gemfile has been modified since Gemfile.lock
RUN bundle config --global frozen 1
WORKDIR /app
# Install packages needed to build gems
RUN apt-get update -qq && \
apt-get install --no-install-recommends -y build-essential git libpq-dev libvips pkg-config
RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add \
&& echo "deb https://dl.yarnpkg.com/debian/ stable main" > /etc/apt/sources.list.d/yarn.list \
&& apt-get update && apt-get install -y nodejs yarn --no-install-recommends
# Install application gems
COPY Gemfile Gemfile.lock ./
RUN bundle install
# Copy application code
COPY . .
# Start the server by default, this can be overwritten at runtime
# EXPOSE 3000
CMD ["/bin/bash"]
Copy the following contents into docker-compose.yml:
version: "3"
services:
db:
environment:
POSTGRES_PASSWORD: postgres
image: postgres
volumes:
- ./tmp/db:/var/lib/postgresql/data
web:
build: .
command: bash -c "rm -f tmp/pids/server.pid && bundle exec rails s -p 3000 -b '0.0.0.0'"
depends_on:
- db
ports:
- "3000:3000"
volumes:
- .:/app:delegated
Replace the contents of the Gemfile with:
source "https://rubygems.org"
ruby "3.2.2"
gem "rails", "~> 7.1.2"
gem "sprockets-rails"
gem "pg", "~> 1.1.0"
# Check the latest supported [https://docs.aws.amazon.com/elasticbeanstalk/latest/platforms/platforms-supported.html#platforms-supported.ruby]
gem "puma", ">= 5.0"
gem "importmap-rails"
gem "turbo-rails"
gem "stimulus-rails"
gem "tzinfo-data", platforms: %i[ windows jruby ]
gem "bootsnap", require: false
group :development, :test do
gem "debug", platforms: %i[ mri windows ]
end
group :development do
gem "web-console"
end
Every time you make changes to the Gemfile, you will need to create a new Gemfile.lock:
docker run --rm -v "$PWD":/app -w /app ruby:3.2.2 bundle install
Initialize your git repository and make an initial commit:
git init
git add .
git commit -m "Prepare the project directory"
If on M1 MAC run the following command
export DOCKER_DEFAULT_PLATFORM=linux/amd64
Build the web container image using docker-compose:
docker-compose build web
Add the following to lines to the default section of config/database.yml:
host: <%= ENV.fetch("PGHOST") { "db" } %>
password: postgres
username: postgres
Make a commit:
git add .
git commit -m "Configure the project to talk to the database container"
First start up the database container:
docker-compose up --detach db
Then verify that the container is running:
docker-compose ps
The output should look like:
NAME COMMAND SERVICE STATUS PORTS
TEAMNAME-db-1 "docker-entrypoint.s…" db running 5432/tcp
Run the following to create and the database:
docker-compose run web rails db:create
Run the following to create and commit the empty schema file.
docker-compose run web rails db:migrate
git add db/schema.rb
git commit -m "Add empty schema file"
Periodically run the following two commands to ensure your ruby and node dependencies are up to date:
docker-compose run web bundle install
docker-compose run web yarn install
If your project has outstanding changes as shown via git status, consider making a
commit at this time.
docker-compose up
At this point you should be able to access the “Rails” page via http://localhost:3000.
If you haven’t already, add GitHub as a remote:
git remote add origin git@github.com:scalableinternetservices/TEAMNAME.git
If this is your first time pushing to GitHub, then run:
git branch -M main
git push -u origin HEAD
Otherwise, you can simply run:
git push
To trigger automated test runs everytime you push to main, or create a pull request, or push an update to a branch associated with a pull request do the following.
Run the following:
mkdir -p .github/workflows
touch .github/workflows/ruby.yml
Then copy the following contents into the file .github/workflows/ruby.yml:
name: Ruby
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
test:
runs-on: ubuntu-latest
services:
postgres:
env:
POSTGRES_PASSWORD: postgres
image: postgres
ports:
- 5432:5432
steps:
- uses: actions/checkout@v2
- name: Install PostgreSQL client
run: sudo apt-get -yqq install libpq-dev
- name: Set up Ruby
uses: ruby/setup-ruby@v1
with:
bundler-cache: true
- name: Set up yarn
run: |
yarn install --pure-lockfile
- name: Prepare database
env:
PGHOST: localhost
RAILS_ENV: test
run: bin/rails db:setup
- name: Run tests
env:
PGHOST: localhost
run: bin/rails test
Add, commit, and push these changes:
git add .
git commit -m "Configure GitHub actions"
git push
At this point you should have a Rails project that you can successfully run in development locally using Docker. In the following steps we’ll make the necessary adjustments to configure the application for Amazon’s Elastic Beanstalk, and then deploy it.
Update the lines in the production section of config/database.yml to include:
database: <%= ENV['RDS_DB_NAME'] %>
host: <%= ENV['RDS_HOSTNAME'] %>
password: <%= ENV['RDS_PASSWORD'] %>
port: <%= ENV['RDS_PORT'] %>
username: <%= ENV['RDS_USERNAME'] %>
Create the directory and file
mkdir .ebextensions
touch .ebextensions/01_install_dependencies.config
touch .ebextensions/10_nginx_add_packs.config
Copy the following contents into .ebextensions/01_install_dependencies.config:
commands:
install_nodejs:
command: |
curl --silent --location https://rpm.nodesource.com/setup_14.x | bash -
yum install -y nodejs
install_yarn:
command: |
curl --silent --location https://dl.yarnpkg.com/rpm/yarn.repo | tee /etc/yum.repos.d/yarn.repo
yum install -y yarn
Copy the following contents into .ebextensions/10_nginx_add_packs.config:
commands:
add_packs_location_to_nginx:
command: |
grep packs /opt/elasticbeanstalk/config/private/nginx/webapp.conf || sed -i '$a\\nlocation /packs {\n alias /var/app/current/public/packs;\n gzip_static on;\n gzip on;\n expires max;\n add_header Cache-Control public;\n}' /opt/elasticbeanstalk/config/private/nginx/webapp.conf
Inform elasticbeanstalk to use your chosen version of puma.
touch Procfile
Copy the following contents in Procfile:
web: bundle exec puma -C /opt/elasticbeanstalk/config/private/pumaconf.rb
git add .
git commit -m "Prepare the application to deploy to Amazon's Elastic Beanstalk"
git push
In order to most easily create an elastic beanstalk deployment, we need to SSH
into ec2.cs291.com. You should have received the file TEAMNAME.pem via your
UCSB Google Drive. Assuming that file is in your downloads folder run the
following:
ssh -i ~/Downloads/TEAMNAME.pem TEAMNAME@ec2.cs291.com
Once logged in, setup your ssh keys to access GitHub repo. Generate the key pair:
ssh-keygen -t ed25519 -C "your_email@example.com"
Then upload the key pair with read-only permissions to GitHub.
For more info see Adding a new SSH key to your GitHub account
After uploading your public key to GitHub, clone your repository using SSH (this will be a read-only version of the project):
git clone git@github.com:scalableinternetservices/TEAMNAME.git
For each copy of your repository, you’ll need to do the following only once:
cd TEAMNAME
eb init --keyname $(whoami) \
--platform "64bit Amazon Linux 2023 v4.0.1 running Ruby 3.2" \
--region us-west-2 TEAMNAME
eb create --envvars SECRET_KEY_BASE=BADSECRET \
-db.engine postgres -db.i db.t3.micro -db.user u \
-i t3.micro --single YOURNAME
Enter a database password at the prompt (twice) and then take a break as creating a deployment will take about ten minutes (the database is slow to create).
Run eb status to see the state of your deployment. The output should look
something like the following:
Environment details for: YOURNAME
Application name: TEAMNAME
Region: us-west-2
Deployed Version: app-f1ab-221021_194424258658
Environment ID: e-7fm2cwv55t
Platform: arn:aws:elasticbeanstalk:us-west-2::platform/Ruby 3.0 running on 64bit Amazon Linux 2/3.5.0
Tier: WebServer-Standard-1.0
CNAME: TEAMNAME.eba-6k3duymc.us-west-2.elasticbeanstalk.com
Updated: 2022-10-21 19:45:40.487000+00:00
Status: Ready
Health: Green
The two most important parts are that Status is Ready, and Health is
Green. If not consult the logs eb logs.
To test if the deployment is successful copy the CNAME, and paste it into your browser: http://YOURNAME.yxhf954iam.us-west-2.elasticbeanstalk.com
If you get a page stating The page you were looking for doesn't exist., that
likely means things are working, and you have yet to set up a root_route on
your site (the “Yay! You’re on Rails!” doesn’t show up in production mode).
After making changes and verifying they work with locally, push your changes to
GitHub, pull them on ec2.cs291.com and then update your deployment via:
eb deploy
Note: Only commited changes are pushed on deployment, so ensure git
status is clean. (You can run eb deploy --staged to include staged files,
but it’s preferrable to deploy code that has been pushed to GitHub.
To view the logs run:
eb logs | less -R
eb ssh -i "ssh -i ~/$(whoami).pem"
When you know you’re done, clean up your deployment:
eb terminate
Note: Deployments will automatically be cleaned up ~110 minutes after their last update.
Please select from one of the following project ideas. You are free to modify them as you wish, and can even come up with another project idea, but the complexity must remain the same.
Please note all the user stories start off with unauthenticated users. That’s because you want to deliver functionality first. Implementing authentication first does not provide any value, if that authentication has no features behind it. Complete a set of unauthenticated stories first, prior to introducing authentication.
Minimum necessary models:
Initial user stories:
As an unauthenticated user I can make a post so that I can share whatever cool things I want.
As an unauthenticated user I can view all the posts on the front page in descending order so that I can see what new things people are sharing.
As an unauthenticated user, I can comment on a post so that I can add updates to existing content.
As an authenticated user, my front page shows only posts made on my profile so that I can see content specific to me.
As as authenticated user, my posts and comments are attributable to me so that others know what I’ve shared.
Minimum necessary models:
Initial user stories:
As an unauthenticated user I can list my item on the store so that I can sell my products.
As an unauthenticated user I can purchase an item from the store so that I can obtain the things I desire.
As an unauthenticated user I can view all the purchase histories so that I can see what others have bought.
As an unauthenticated user, I can rate an item that has been purchased so that I can share my opinions of the item with others.
As an authenticated user, only I can see my own orders in my order history because I don’t want others to see what I’ve purchased.
Minimum necessary models:
Initial user stories:
As an unauthenticated user, I can create an event so that I can share with others the details of the event I am hosting.
As an unauthenticated user, I can comment on an event to share my enthusiasm for said event.
As an authenticated user, I can RSVP yes/no to events that I do [not] intend to attend so that the host can better estimate how many people will show up.
As an authenticated user, I can see the list of events that I have previously attended so that I can recall my fond memories.
As an event host, I can proactively invite users to my event, so that I can help spread the word.