Installing CMS (Contest Management System) on multiple servers
For the needs of the 24th BOI (Balkan Olympiad in Informatics) that will be held in Cyprus in June 2016 we setup CMS (Contest Management System) on multiple servers.
We used Ubuntu 14.04 LTS (Trusty Tahr) for the OS of the servers, the installations were made with encrypted LVM.
We used 3 servers in total, the first (alpha
) would serve as a master that holds all services for the competition except for the workers that test the submissions. The other two (beta
and gamma
) were setup to hold the workers.
On Alpha:
We installed various packages that are needed for the installation of the system
sudo apt-get install build-essential fpc postgresql postgresql-client gettext python2.7 python-setuptools python-tornado python-psycopg2 python-sqlalchemy python-psutil python-netifaces python-crypto python-tz python-six iso-codes shared-mime-info stl-manual python-beautifulsoup python-mechanize python-coverage python-mock cgroup-lite python-requests python-werkzeug python-gevent patool;
We downloaded the CMS and created the basic configuration.
#Download stable version from GitHub wget https://github.com/cms-dev/cms/releases/download/v1.2.0/v1.2.0.tar.gz #Extract the archive tar -xf v1.2.0.tar.gz cd cms #Copy the sample configuration files to be modified cp config/cms.conf.sample config/cms.conf cp config/cms.ranking.conf.sample config/cms.ranking.conf
cms.conf
Using a text editor we modified file config/cms.conf
. The changes we did are the following:
- We replaced
"database": "postgresql+psycopg2://cmsuser:password@localhost/database"
With the username, the password and database name we will use for our database configuration in a while. e.g.
"postgresql+psycopg2://myuser:myPassword@alpha/mydatabase"
. Note: we chose a username and database name with no capital letters. - We replaced
"secret_key": "8e045a51e4b102ea803c06f92841a1fb"
with another random 32 character string containing a-z and 0-9 (hex characters) - We replaced
"rankings": ["http://usern4me:passw0rd@alpha:8890/"]
to some other username password configuration. e.g."rankings": ["http://myUsern4me:myPassw0rd@localhost:8890/"]
- We changed the
"_section": "AsyncLibrary",
to hold the configuration of the services as follows
"_section": "AsyncLibrary", "core_services": { "LogService": [["alpha", 29000]], "ResourceService": [["alpha", 28000], ["beta", 28000], ["gamma", 28000]], "ScoringService": [["alpha", 28500]], "Checker": [["alpha", 22000]], "EvaluationService": [["alpha", 25000]], "Worker": [["beta", 26000], ["beta", 26001], ["beta", 26002], ["beta", 26003], ["gamma", 26000], ["gamma", 26001], ["gamma", 26002], ["gamma", 26003]], "ContestWebServer": [["alpha", 21000]], "AdminWebServer": [["alpha", 21100]], "ProxyService": [["alpha", 28600]], "PrintingService": [["alpha", 25123]] }, "other_services": { "TestFileCacher": [["alpha", 27501]] },
We replaced all instances of localhost
with alpha
. We did not put any workers on alpha
for fairness.
cms.ranking.conf
Using a text editor we modified file config/cms.ranking.conf
. The changes we did were to set
"username": "usern4me", "password": "passw0rd",
to the values we set in the file cms.conf
at the section rankings
. In our example the values would be myUsern4me
and myPassw0rd
.
Setting up the system
After we were done with the configuration changes, we executed the following ./setup.py build
so that the system would perform the basic checks and setup. Then, we executed sudo ./setup.py install
to make the installation.
Following, we added the user we will use to start the competition system to the user group of the cms sudo usermod -a -G cmsuser george
.
And we switched to the DB user to take the necessary actions.
Database Configuration
sudo su - postgres
to switch to the DB user.
We created the DB user we defined in the cms.conf
file before (myuser
) with the password we defined using the command createuser myuser -P
, the password in our case is myPassword
.
After that, we created our database (mydatabase
) and we assigned our user to be the owner with full access rights of that database using the following commands: createdb -O myuser mydatabase
then psql mydatabase -c 'ALTER SCHEMA public OWNER TO myuser'
finally psql mydatabase -c 'GRANT SELECT ON pg_largeobject TO myuser'
.
Allowing connections from other machines
We needed to find the hba
file for psql to allow incoming connections from other machines. We got the location of the file using psql -t -P format=unaligned -c 'show hba_file';
. In our case the location was /etc/postgresql/9.3/main/pg_hba.conf
. Using a text editor we added at the end of the file the following line:
host mydatabase myuser 0.0.0.0/0 md5
The above line instructs the system to allow our user to connect from any IP to our database.
For better security, you can use the following lines instead:
host mydatabase myuser beta md5 host mydatabase myuser gamma md5
Then we used psql -t -P format=unaligned -c 'show config_file';
which gave use the file /etc/postgresql/9.3/main/postgresql.conf
, using a text editor we changed #listen_addresses = 'localhost'
to listen_addresses = '*'
to enable listening on all network devices.
On the terminal, we exited the console of the database user, we re-installed the cms using sudo ./setup.py install
to create all database tables and we rebooted the machine using sudo reboot
to apply group changes to the user as well.
Once the system reboots, execute cmsInitDB
to create the tables for our database.
Create a file named updateHosts.sh
and copy the following code in it. We will use it to update the host file of the machine so that it will identify the external IP of the machine first.
#!/bin/bash FILE="/etc/hosts"; BACKUP="$FILE.original"; if [ -f "$BACKUP" ]; then echo "--- --- --- BackUp exists --- --- ---"; else cp "$FILE" "$BACKUP"; if [ $? -ne 0 ]; then echo -e "--- --- --- Could not create backUp. Terminating! --- --- ---"; exit -1; fi echo "--- --- --- BackUp Created --- --- ---"; fi echo "--- --- --- Original '$FILE' --- --- ---"; cat "$BACKUP"; DEFAULT_DEVICE="p7p1"; DEVICE=${1:-$DEFAULT_DEVICE}; IP=`ifconfig "$DEVICE" | grep "inet addr" | cut -d ':' -f 2 | cut -d ' ' -f 1`; HOST=`hostname`; > "$FILE"; while read LINE; do if [[ "$LINE" == *"$HOST"* ]]; then echo -ne "$IP\t" >> "$FILE"; echo "$LINE" | cut -f2- >> "$FILE"; else echo "$LINE" >> "$FILE"; fi done < "$BACKUP"; echo "--- --- --- Updated '$FILE' --- --- ---"; cat "$FILE"; exit 0;
Execute chmod +x updateHosts.sh
to make the script into an executable one.
The script needs to know the name of the network device it is supposed to tamper with. Type ifconfig
in your terminal to find the available device names. The one you most likely are interested in would have an IP similar to the one of your PC or smartphone e.g. inet addr:192.168.10.5
. If the device is an ethernet device, then you will find the following next to the device name Link encap:Ethernet
. The name you need to copy could be eth0
or p7p1
or something similar to these.
Once you get the name execute sudo ./updateHosts.sh DEVICE_NAME
and replace DEVICE_NAME
with the name you copied in the previous step.
Starting the system on Alpha:
In a terminal execute cmsAdminWebServer 0
to start the administration server and use it to create a new contest from http://alpha:8889/.
Once you created you competition from http://alpha:8889/, open up 3 terminals to alpha
.
On the first one execute cmsLogService 0
to start the logging service.
On the second execute cmsRankingWebServer
to start the Ranking Server that can be reached at http://alpha:8890/.
And on the third one cmsResourceService 0 -a 1
to start the competition we just created along with all services that are needed for the competition from alpha
. The competition can be reached at http://alpha:8888/.
Setting up the live replication of the database:
Switch to the postgres user sudo su - postgres
and create an ssh key ssh-keygen
.
Copy on both servers the key you just created using ssh-copy-id contest@beta
and ssh-copy-id contest@gamma
.
Next we will create a user called replicator
that can be used solely for the replication process and set a random password of it:
psql -c "CREATE USER replicator REPLICATION LOGIN CONNECTION LIMIT -1 ENCRYPTED PASSWORD 'randomPassword';"
Then edit the file /etc/postgresql/9.3/main/pg_hba.conf
(we found the location of pg_hba.conf
using the command psql -t -P format=unaligned -c 'show hba_file';
) and add the following lines:
host replication replicator beta md5 host replication replicator gamma md5
This will allow user replicator to connect from the machines beta
and gamma
.
You could give the following line, instead of the two above:
host replication replicator 0.0.0.0/0 md5
This will allow user replicator to connect from ANY IP.
Also, if you know from beforehand the IPs that beta
and gamma
are going to have, it is best to replace the above with two lines containing the IPs of the two machines as it is more secure.
host replication replicator IP_address_of_beta/32 md5 host replication replicator IP_address_of_gamma/32 md5
Following, using psql -t -P format=unaligned -c 'show config_file';
we got the path to the file postgresql.conf
(/etc/postgresql/9.3/main/postgresql.conf
) which we edited using a text editor. The following changes were made to the file:
- We uncommented and changed
#wal_level = minimal
towal_level = hot_standby
- We uncommented and changed
#archive_mode = off
toarchive_mode = on
- We uncommented and changed
#archive_command = ''
toarchive_command = 'cd .'
- We uncommented and changed
#max_wal_senders = 0
tomax_wal_senders = 10
- We uncommented and changed
#hot_standby = off
tohot_standby = on
Save the file and execute service postgresql restart
to apply the changes.
Execute psql -c "ALTER USER replicator WITH CONNECTION LIMIT -1";
to remove the limit of connections for our replication user.
You can verify the settings using psql -c "SELECT rolname, rolconnlimit FROM pg_roles";
, this command will show you the connection limits per user. If the value is -1
then it means that there is no restriction.
On Beta and Gamma:
We installed various packages that are needed for the installation of the system, we intended in using these machines both as the workers for the competition but also as live backups in case alpha
goes down. Later on, we will enable live copying of the database in alpha
to these machines.
sudo apt-get install build-essential fpc postgresql postgresql-client gettext python2.7 python-setuptools python-tornado python-psycopg2 python-sqlalchemy python-psutil python-netifaces python-crypto python-tz python-six iso-codes shared-mime-info stl-manual python-beautifulsoup python-mechanize python-coverage python-mock cgroup-lite python-requests python-werkzeug python-gevent patool;
We switched to the postgres user sudo su - postgres
and we stopped the postgres service service postgresql stop
.
Then edit the file /etc/postgresql/9.3/main/pg_hba.conf
(we found the location of pg_hba.conf
using the command psql -t -P format=unaligned -c 'show hba_file';
) and add the following line:
host replication replicator alpha md5
Following, using psql -t -P format=unaligned -c 'show config_file';
we got the path to the file postgresql.conf
(/etc/postgresql/9.3/main/postgresql.conf
) which we edited using a text editor. The following changes were made to the file:
- We uncommented and changed
#listen_addresses = 'localhost'
tolisten_addresses = 'localhost,beta'
forbeta
and tolisten_addresses = 'localhost,gamma'
forgamma
- We uncommented and changed
#wal_level = minimal
towal_level = hot_standby
- We uncommented and changed
#archive_mode = off
toarchive_mode = on
- We uncommented and changed
#archive_command = ''
toarchive_command = 'cd .'
- We uncommented and changed
#max_wal_senders = 0
tomax_wal_senders = 1
- We uncommented and changed
#hot_standby = off
tohot_standby = on
Execute pg_basebackup -h alpha -D /var/lib/postgresql/9.3/main/ -U replicator -v -P --xlog-method=stream
to copy the database from Alpha. If you get the error pg_basebackup: directory "/var/lib/postgresql/9.3/main/" exists but is not empty
then delete the contents of the folder with the command rm -rf /var/lib/postgresql/9.3/main/*
and try again. You will be prompted for a password, use the random password we assigned the user replicator
before. In this scenario it will be randomPassword
.
Afterwards, create the file /var/lib/postgresql/9.3/main/recovery.conf
and add the following content:
standby_mode = 'on' primary_conninfo = 'host=alpha port=5432 user=replicator password=randomPassword sslmode=require' trigger_file = '/tmp/postgresql.trigger'
Finally, start the service using service postgresql start
and exit
to stop using the postgres user.
Starting the workers on beta and gamma:
First, we downloaded the CMS:
#Download stable version from GitHub wget https://github.com/cms-dev/cms/releases/download/v1.2.0/v1.2.0.tar.gz #Extract the archive tar -xf v1.2.0.tar.gz
Then, we copied the configuration files from alpha
. Specifically, we copied the configuration files from alpha
to 'cms/config/cms.conf'
and 'cms/config/cms.ranking.conf'
using the following commands:
scp contest@alpha:~/cms/config/cms.conf ~/cms/config/cms.conf scp contest@alpha:~/cms/config/cms.ranking.conf ~/cms/config/cms.ranking.conf
Afterwards, we entered the CMS folder cd cms
, build it ./setup.py build
and installed it sudo ./setup.py install
.
Following, we added the user we will use to start the competition system to the user group of the cms sudo usermod -a -G cmsuser george
. We rebooted the machine using sudo reboot
to apply all changes including the group change to the user.
Then we copied the updateHosts.sh
file from alpha
using scp contest@alpha:~/updateHosts.sh ~/updateHosts.sh
, executed chmod +x updateHosts.sh
to make the script into an executable one. And, we executed sudo ./updateHosts.sh DEVICE_NAME
. Like before you need to replace DEVICE_NAME
with the network device name that will be used for the communication.
To start the workers, execute cmsResourceService 1 -a 1
on beta
and cmsResourceService 2 -a 1
on gamma
.
Next steps
- How to change the ranking system logo
- How to assign photographs to user profiles
- How to create and assign teams with their flags to users
In case you have DNS problems, modify the file /etc/hosts
and add the following entries to it:
10.1.10.3 alpha 10.1.10.4 beta 10.1.10.5 gamma
In the end it should look something similar to this:
127.0.0.1 localhost 10.1.10.3 alpha 10.1.10.4 beta 10.1.10.5 gamma # The following lines are desirable for IPv6 capable hosts ::1 localhost ip6-localhost ip6-loopback ff02::1 ip6-allnodes ff02::2 ip6-allrouters