PREREQUISITES have putty portable configured and ppk key have winscp portable configured and ppk key Tim's mysql scripts SHARED\DB scripts (will be unique folders by SVN version number)
Staging app.rb (or deployer.rb) Chef client only deploys code to a server based on Roles (e.g. RN1 doesn't have a bunch so only gets some databag stuff) (service naming convention oneword-thrift-server) manage.opscode.com (chef server)
2 RDS SERVERS FOR SANDBOX test..us-east-1.rds.amazonaws.com testrn2..us-east-1.rds.amazonaws.com
3 RDS SERVERS FOR PRODUCTION
OVERVIEW
- SSH into machine(s) and open screen/byobu
- Start tailing catalina.out
- Upload Scripts
- Stage Chef (version comparison)
- Stop Tomcat
- run chef-client (central chef servers Prod & Sandbox)
- RUN MYSQL SCRIPTS (if necessary)
- Start Tomcat
- Watch for errors
- Use PsiProbe to see that all the apps are running
- Test
STEP BY STEP
hostname = sandbox..com user = ubuntu with .ppk
SSH -> Tunnels
Source Port 8080 (could be anything) Destination localhost:80
byobu
control + a then type 0 (change to session 0)
0 = usually used to tail tomcat logs 1 = mysql 2 = mysql 3 = chef client
tail -f /var/log/tomcat6/catalina.out
get the mysql scripts from SVN VERSION folder from Tim's MySQL upload the SVN VERSION folder using WINSCP into the folder:
/home/ubuntu/deploy
WINSCP sftp with scp fallback
- Refresh, 2. DEPLOY ALL TO PREPARE STAGING http://10.10.10.230:9294 (ruby app to see versions for staging)
( http://10.10.10.230:9294/#/ to start Refresh directly )
Will use java script to get "build version" on D2 (Maven metadata file on metadata.xml, parses the xml to decide whether it's a new version - databags with JSON to help decide)
Red = Out of date (Build versions does not match Chef version) Build version = available on D2 Chef Version = what's avaible to deploy using Chef
STOP TOMCAT (e.g. byobu screen 3) (COMMUNICATE TO USERS THAT SERVICE WILL BE GOING DOWN)
sudo /etc/init.d/tomcat6 stop
USE EITHER byobu screen 0 with the tail of log or ps aux | grep java to VERIFY tomcat has stopped
sudo chef-client
sudo chef-client -d = DEBUG = A LOT OF NOISE IF nodename is wrong key mismatch error
" [date] INFO: Will update: servicename"
deletes previous stuff from /tmp
version compare (from staging to production), if different put in /tmp
extracts .war, svn to grab app.properties and log4j.xml
deletes temp stuff from /tmp
(Sandbox does not pull svn changes, it has default templates so for app.properties and log4j.xml we have to manually update it) sandbox..com (update cookbook to get it)
RUN MYSQL SCRIPTS (can be done in parallel with chef client command above) byobu 2 (back to mysql stuff) usually stage separate mysql commands to go faster scripts need to be run in chronological order oldest to newest DO NOT RUN A SCRIPT TWICE!!!!
ctrl+r to get the recent command mysql host user pass database < scriptname.sql
mysql -htest..us-east-1.rds.amazonaws.com -uUSER -pPASSWORD repositorynode < repositorynode.2011030100.sql
(Needs to run about 6 times - if you see errors CALL TIM!)
FOR Repository Node 2 server =
mysql -htestrn2.h.us-east-1.rds.amazonaws.com -uUSER -pPASSWORD repositorynode < repositorynode.2011030100.sql
DO NOT RUN A SCRIPT TWICE!!!!
SANDBOX SPECIFIC STEP rm -rf repositorynode2 cp -r repository repositorynode2 (could be -a to preserve owner) chown -R tomcat6:tomcat6 repositorynode2
cp /mnt/*.xml . (copy the backup of xml files back into repositorynode2 cp /mnt/app.properties .
sudo /etc/init.d/tomcat6 start tail the tomcat6 log (about 2 mins on sandbox, 40 secs on Prod)
INFO: Server startup in 126976 ms (need to wait a little till it starts)
DEBUG WITH TIM IF YOU SEE: IGNORE COULD NOT CONNECT TO SQL DATABASE XML ERROR
PSI PROBE (for tomcat6)
http://localhost:8080/probe (tunneling to get this more secure web app) All apps running = GOOD (disabled means error) System Information -> Memory Utilization If CMS Perm Gen memory > 512MB server crashes (stop / start tomcat allows you to clear memory) Can force garbage collection if necessary
TEST with adminconsole, client, systemconsole
stop tailling the tomcat log catalina.out Detach Session: control + a + d screen -ls (should only show 1 Detached session) exit