PeopleSoft Performance Metrics
850 likes | 1.18k Vues
PeopleSoft Performance Metrics. nalin.n.patel@abbey.com david.kurtz@go-faster.co.uk. Abbey key facts 1. Sixth largest bank by assets in the UK Founded in 1944 Currently have approximately 18m customers 741 branches across UK. Abbey key facts 2.
PeopleSoft Performance Metrics
E N D
Presentation Transcript
PeopleSoftPerformance Metrics nalin.n.patel@abbey.com david.kurtz@go-faster.co.uk
Abbey key facts 1 • Sixth largest bank by assets in the UK • Founded in 1944 • Currently have approximately 18m customers • 741 branches across UK
Abbey key facts 2 • Abbey's main offices are in London, Milton Keynes, Bradford, Glasgow and Belfast. • We have around 26,000 people (full time equivalent) • We have about 1.8 million shareholders • Assets at 30 June 2004 - £171 billion • Personal Financial Services trading profit before tax for 6 months to 30 June 2004 - £340 million
History of PeopleSoft at Abbey • PeopleSoft HRMS acquired for recruitment in 1994 • Implemented PeopleSoft HRMS in 1997 • Recruitment, Personnel & Training • Paylink used to send data from HRMS to payroll • Workflow and self-service with v7.5 in 2000 • JAVA HTML Clients • PeopleSoft HRMS upgraded to 8 SP1 in 2001 • Implemented PeopleSoft Payroll in August 2003 • Project initiated to upgrade to HCM 8.8
Current Platform • AppServer runs on SUN E4500 • Database runs on SUN E10000 • Both boxes are shared with other applications • Tier 1 mirrored disks • Oracle 9.2.04
Why PeopleSoft Payroll ? • Integrated HRMS • Common infrastructure • Web enabled • Automate administrative functions • Manager Self-Service • Absence and maternity input • Employee Self-Service • Overtime input • On-line payslips • Real-time data input • Increase system availability
PeopleSoft Payroll Implementation • Development commenced in January 2002 • In-house IT Project team • Project delays due to re-scoping and internal re-structure • Streamed payroll during parallel run tests • Went live with payroll and absence in August 2003 • 30,000 staff and 7,000 pensioners • 12 streams introduced in February 2004 • Introduced hash partition in July 2004 due to increased run times • Identify and calculate taking 2.5 hours • But we had to tune it…
Agenda • What is Performance Tuning? • On-Line • Batch • What has been achieved at Abbey? • PeopleSoft Performance Monitor • The shape of things to come! • Questions?
Rules of Engagement • If you can’t hear me - say so now! • Feel free to ask on-topic questions during the presentation. • This presentation (with notes) will be available on the Conference website on Customer Connection and on • www.go-faster.co.uk
Independent consultant System Performance Tuning Oracle Databases Unix Tuxedo PeopleSoft Apps Book www.psftdba.com Who am I?
Aphorism • Performance is exactly what the user perceives it to be. No more, no less. • Poor performance is when the user’s perception does not match their expectation.
Performance Tuning is a search for lost time. Optimizing Oracle Performance Cary Millsap & Jeff Holt Aphorism • See • ‘Yet Another Performance Profiling Method’ (1999) • by Anjo Kolk, Shari Yamaguchi & Jim Viscusi • http://www.oraperf.com/whitepapers.html
When you have eliminated the impossible, whatever remains, however improbable, must be the truth. A Study in Scarlet, Arthur Conan-Doyle Aphorism • It is a capital mistake to theorise before one has data. The Memoirs of Sherlock Holmes, Arthur Conan-Doyle
Aphorism • Detection is, or ought to be, an exact science. It should be be treated in the same cold and unemotional manner. • The Sign of Four, Arthur Conan-Doyle
PIA (On-Line) Performance • Architecture • brief overview • Sources of Metrics • Graphical Techniques • Tracing • Techniques
It isn’t always the database! • Understand architecture • Web front end • Network • Application server • Database • What can you measure?
Java Servlet (presentation logic) Tuxedo Application Server (application logic) DBMS (application data & meta-data http / https Tuxedo Message SQL PeopleSoft Internet Architecture • PIA is 4-tier architecture • Each component can contribute to response time • Need to measure each component
Sources of Metrics (1) • Browser • Proxy Server • 3rd Party software • Local agent on PC • PeopleSoft Ping • From PT8.19, PT8.4 • Web Server • Access Log • Weblogic & Apache • Websphere does not generate a log
Sources of Metrics (2) • BEA/Tuxedo • Service Trace • TMADMIN • LogFence 4 • Oracle Database • SQL Trace (event 10046) • PeopleTools SQL Trace • Large overhead
Hint • All the data collected on a time base. • Therefore, make sure that the clocks on all machines are synchronised at least with each other. • PC clocks are particularly liable to drift.
Java Servlet (presentation logic) Tuxedo Application Server (application logic) DBMS (application data & meta-data http / https Tuxedo Message SQL Sources of Metrics Proxy Server Access Log Webserver Access Log
W3C Standard format for access log www.w3.org/pub/WWW/TR/logfile.html Used by Weblogic, IIS Apache is different WebSphere doesn’t have an access log! Web/Proxy Access Log
Web/Proxy Access Log • Fields • Date, Time • Time went log entry emitted • (End time not start time) • Time Taken • accurate to OS time units (1/100ths or 1/1000ths) • only 1s on Apache • Message size bytes • Client IP addresses & DNS names • URI stem & Query
How to Enable WebLogic 5.1 Log #weblogic.properties #required weblogic.httpd.enableLogFile=true weblogic.httpd.logFileName=access.log weblogic.httpd.logFileFormat=extended #recommended weblogic.httpd.logRotationType=date weblogic.httpd.logRotationPeriodMins=1440 weblogic.httpd.logRotationBeginTime=11-24-2000-00:00:00
Sample WebLogic Access Log #Version: 1.0 #Fields: date time time-taken bytes c-ip c-dns cs-method sc-status cs-uri-stem cs-uri-query 2002-03-15 21:51:03 0.11 2103 10.0.0.2 go-faster-1 GET 200 /peoplesoft8/signin.css - 2002-03-15 21:51:04 0.05 1186 10.0.0.2 go-faster-1 GET 200 /peoplesoft8/images/secondaryPSFTLogo.gif - 2002-03-15 21:51:10 3.414 8101 10.0.0.2 go-faster-1 POST 200 /servlets/iclientservlet/peoplesoft8/ ICType=Panel&Menu=ADMINISTER_WORKFORCE_(GBL)&Market=GBL&PanelGroupName=PERSONAL_DATA&RL=&target=main2
Apache Access Log #httpd.conf LogFormat "%{%Y.%m.%d %H:%M:%S}t|%T|%B|%u|%h|%{User-Agent}i|%>s|%m|%U|%q" monitoring CustomLog logs/access.log monitoring
Sample Apache Access Log 2002.02.26 09:57:06|0|275|-|127.0.0.1|Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Q312461)|200|GET|/peoplesoft8/cache/PT_NEXTTAB_ENG_1.gif| 2002.02.26 09:57:17|0|31847|-|127.0.0.1|Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Q312461)|200|POST|/servlets/iclientservlet|?ICType=Panel&Menu=ADMINISTER_WORKFORCE_(GBL)&Market=GBL&PanelGroupName=JOB_DATA 2002.02.26 09:57:26|0|30959|-|127.0.0.1|Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Q312461)|200|POST|/servlets/iclientservlet|?ICType=Panel&Menu=ADMINISTER_WORKFORCE_(GBL)&Market=GBL&PanelGroupName=JOB_DATA
What does the access log tell us? • A date and time for every request • Who made the request? • How long did it take to serve? • What was requested? • Which component (panel group)? • Indicate possible bandwidth problems.
Download Bandwidth #version: 1.0 #Fields: date time time-taken bytes c-ip c-dns cs-method sc-status cs-uri-stem cs-uri-query 2002-09-10 11:29:52 0.359 15832 141.93.249.74 141.93.249.74 GET 200 /XXXXXXXX/images/topPSFTLogo.gif -
Java Servlet (presentation logic) Tuxedo Application Server (application logic) DBMS (application data & meta-data http / https Tuxedo Message SQL Sources of Metrics Tuxedo Service Trace
BEA/Tuxedo Service Trace • Edit psappsrv.ubx file • CLOPT=“-r -e APPQ.stderr…” • -r enables trace to stderr file • -e qualifies name of stderr file
Edit psappsrv.cfg • Add a new variable to the trace section • Thus service trace can be enabled/disabled from configuration file • New variable will appear in the configuration dialogue [Trace] TuxedoServiceTrace=-r
Edit psappsrv.ubx • Change every CLOPT entry in psappsrv.ubx. • File name is set manually. • Same as queue name RQADDR="APPQ" CLOPT="{$Trace\TuxedoServiceTrace} -e {LOGDIR}{FS}{MACH}_{$Domain Settings\Domain ID}_APPQ.stderr {$PSAPPSRV\Spawn Server} -s@..{FS}psappsrv.lst -- -C {CFGFILE} -D {$Domain Settings\Domain ID} -S PSAPPSRV"
Resultant psappsrv.ubb • When domain is configured .ubb file is generated. All variables are fully resolved. CLOPT=”-r -e D:\ps\hr88\appserv\hr88\LOGS\HR88_APPQ.stderr -p 1,600:3,10 -s@..\psappsrv.lst -s@..\psqcksrv.lst -sICQuery -sSqlQuery:SqlRequest -- -C psappsrv.cfg -D HR88 -S PSAPPSRV"
Sample Tuxedo Service Trace • Each service called to domain is logged SERVICE PID SDATE STIME EDATE ETIME ------- --- ----- ----- ----- ----- @ICScript 390 1014717320 4586495 1014717322 4588397 @ICScript 390 1014717322 4588467 1014717322 4588537 @ICScript 390 1014717322 4588557 1014717323 4589259 @ICScript 390 1014717332 4598021 1014717332 4598171 @ICScript 390 1014717355 4621745 1014717361 4627554 @ICScript 390 1014717365 4631329 1014717366 4632511 @ICScript 390 1014717375 4641944 1014717377 4643306 @ICPanel 390 1014717380 4646761 1014717381 4647783 @ICPanel 390 1014717388 4654202 1014717391 4657567 @ICPanel 390 1014717396 4662614 1014717401 4667271
Tuxedo Service Trace • Service - Name of Service • Pid - An OS PID of listener or handler • Sdate/Edate - Start/End Date • Seconds since 00:00hrs GMT 1st January 1970. • Stime/Etime • Start/End Time in OS time units (1/ 100ths or 1/1000ths) • Hence calculate accurate service duration • Time on Server - not queuing
Txrpt Utility SERVICE SUMMARY REPORT SVCNAME 9a-10a TOTALS Num/Avg Num/Avg --------------- -------- ------- ICScript 8/1.43 8/1.43 ICPanel 6/3.47 6/3.47 JavaMgrGetObj 1/0.04 1/0.04 GetCertificate 1/0.73 1/0.73 --------------- ------- ------- TOTALS 16/2.06 16/2.06
Java Servlet (presentation logic) Tuxedo Application Server (application logic) DBMS (application data & meta-data http / https Tuxedo Message SQL Sources of Metrics Tuxedo tmadmin script
BEA/Tuxedo TMADMIN Utility • BEA Interactive command line interface • Can be included in scripts • -r for read only in monitoring scripts • (this is a different –r) • pq (printqueue) • psr (printserver) • pclt (printclient)
PQ • What queues in domain • How many servers on queue • How many requests queued • (How much work on queue) Prog Name Queue Name # Serve Wk Queued # Queued Ave. Len Machine --------- ------------------- --------- -------- -------- ------- JSL.exe 00095.00200 1 - 0 - GO-FASTER+ JREPSVR.exe 00094.00250 1 - 0 - GO-FASTER+ PSSAMSRV.exe SAMQ 1 - 0 - GO-FASTER+ BBL.exe 54455 1 - 0 - GO-FASTER+ WSL.exe 00001.00020 1 - 0 - GO-FASTER+ PSAPPSRV.exe APPQ 1 - 0 - GO-FASTER+
PSR • What servers exist? • How many requests have they handled? • What are they doing right now? Prog Name Queue Name Grp Name ID RqDone Load Done Current Service --------- ---------- -------- -- ------ --------- --------------- BBL.exe 54455 GO-FAST+ 0 3 150 ( IDLE ) PSAPPSRV.exe APPQ APPSRV 1 9 450 ICPanel PSSAMSRV.exe SAMQ APPSRV 100 0 0 ( IDLE ) WSL.exe 00001.00020 BASE 20 0 0 ( IDLE ) JSL.exe 00095.00200 JSLGRP 200 0 0 ( IDLE ) JREPSVR.exe 00094.00250 JREPGRP 250 5 250 ( IDLE )
PCLT • Who is connected? • Are they doing something? • When did they connect? LMID User Name Client Name Time Status Bgn/Cmmt/Abrt --------------- --------------- --------------- -------- ------- ------------- GO-FASTER-1 NT WSH 0:21:14 IDLE 0/0/0 GO-FASTER-1 NT JSH 0:21:13 IDLE 0/0/0 GO-FASTER-1 NT tmadmin 0:00:00 IDLE 0/0/0 GO-FASTER-1 PS JavaClient 0:02:30 BUSY/W 0/0/0
Java Servlet (presentation logic) Tuxedo Application Server (application logic) DBMS (application data & meta-data http / https Tuxedo Message SQL Sources of Metrics Oracle SQL*Trace
Oracle SQL Trace • Tuning tool • Session trace • Format it with TKPROF • (Or the www.HOTSOS.com profiler) • Find long running SQL statements • Total SQL Time for session call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 72289 173.06 175.45 7 14654 94 0 Execute 77140 64.02 67.37 205495 766343 170611 78630 Fetch 50599 33.20 34.28 17302 971624 80146 143218 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 200028 270.28 277.10 222804 1752621 250851 221848
Small Application Servers • Create small application with just a single handler process • Enable trace on that process • Trace sample transaction (only one user) sys.dbms_system.set_sql_trace_in_session(<sid>,<serial>,TRUE);
Java Servlet (presentation logic) Tuxedo Application Server (application logic) DBMS (application data & meta-data http / https Tuxedo Message SQL Sources of Metrics Proxy Server Access Log Tuxedo tmadmin script Tuxedo Service Trace Oracle SQL*Trace Webserver Access Log
Graphical Analysis • Don’t drown in numbers • Draw a picture • Load Data into Database • SQL_Loader (Oracle 9i External Table) • Process Data • Matching, Aggregation • Oracle Analytic Function • Load data into Excel • Draw Graph
Sample SQL*Loader File LOAD DATA INFILE 'APPQ.stderr' REPLACE INTO TABLE txrpt WHEN (1) = '@' FIELDS TERMINATED BY WHITESPACE TRAILING NULLCOLS (service "substr(:service,2)" -- remove leading @ ,pid ,stimestamp ":stimestamp/86400+2/24+TO_DATE('01011970','DDMMYYYY')" ,stime ":stime/100" ,queue ”’APPQ’" ,etime ":etime/100")