Download
1 / 12
120 likes | 368 Vues
웹서비스 장애 관련. 강호성. 목 차. 서버 시스템 구성도 서버 다운 증상 서버 정보 (vmstat ) 서버 정보 (netstat ) 서버 정보 (netstat ) 서버 다운 정보 (startup.sh 옵션 : GC 정보 ) 서버 정보 ( ps -aux ) 서버 정보 ( 메모리 heap 변화 모니터링 ) 4. 질문사항 5. 예상 장애 요인. 1. 서버 시스템 구성도. 연동방법
E N D
웹서비스 장애 관련 강호성
목 차 • 서버 시스템 구성도 • 서버 다운 증상 • 서버 정보 (vmstat ) • 서버 정보 (netstat ) • 서버 정보 (netstat ) • 서버 다운 정보 (startup.sh 옵션 : GC 정보) • 서버 정보 (ps -aux ) • 서버 정보 (메모리 heap 변화 모니터링) • 4. 질문사항 • 5. 예상 장애 요인
1. 서버 시스템 구성도 연동방법 Thumnail List출력 : URLConnection 이용 Apache : Redirect 를 이용 무룡이미지다운서비스 : Socket통신,다운로드 File Server (웹용 썸네일 이미지 파일들) Env : Windows Tomcat 4.1.18 Apache2.0.43 IP : 생략 위치 : 생략 사양 : CUP 2.4 G / 550MHZ 2G Memory 실제 원본 Data Server 사용용도 : Web Server와 암호모듈 통해서 다운로딩 서비스 기존부터잘사용됨 Web Server Env : Linux Redhat 9.0 Tomcat 4.1.18 Apache2.0.43 IP : 생략 위치 : 생략 상세내용 : FileServer와연동(URLConnection 이용) 사양 : LGIBM intel pentium 3 Xeon 550 Dual 1G Memory 장애발생서버 DataBase Env : windows + Mssql2000 IP : 생략 위치 : 생략 사양 : HP DL380 모델 CUP 3.06 G / 550MHZ 2G Memory 사용자 연동방법 DNS : search.XXX.co.kr Apache : Redirect 를 이용 Search Server Env : Linux Redhat 9.0 Tomcat 4.1.18 Apache2.0.43 IP : 생략 위치 : 생략 상세내용 : JSP Servlet C 검색모듈 사양 : CPU - Xeon 2.6 GHz * EA Board - Intel SE7501BR2 Mem - 1G SDRAM * 4 EA 10 M 단독라인
* Tomcat 로그 (catalina) May 19, 2004 7:02:28 PM org.apache.tomcat.util.log.CommonLogHandler log INFO: All threads are busy, waiting. Please increase maxThreads or check the servlet status150 150 TOP 명령어를 이용한 총 프로세스는 평상시 200개 이하.서버 장애 시점에는 갑자기 8개이상씩 증가하면서 5~10분여 만에 총 프로세스 400개이상.서버 장애 시점에서의 초기에는 JSP 페이지는 안뜨나, html 페이지는 열립니다.서버 장애 후반에서는 JSP및html 페이지가 뜨지 않는 현상. 2. 서버 다운 증상
* vmstat (평상시)procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 0 0 1360 127668 45360 263056 0 0 3 7 9 10 4 2 18 0 0 0 1360 127668 45360 263056 0 0 0 0 728 390 3 1 96 0 0 0 1360 127668 45360 263060 0 0 0 0 877 523 5 2 93 0 0 0 1360 127668 45360 263060 0 0 0 0 656 303 1 1 98 0 0 0 1360 127668 45360 263060 0 0 0 0 727 402 6 1 93 1 0 0 1360 127668 45360 263060 0 0 0 100 835 472 6 2 92 0 0 0 1360 127668 45360 263064 0 0 0 0 860 440 3 4 93 0 0 0 1360 127660 45360 263064 0 0 0 0 883 456 5 2 93 0 0 0 1360 127660 45360 263064 0 0 0 0 658 283 2 0 98 0 0 0 1360 127660 45360 263064 0 0 0 0 779 441 1 2 96 0 0 0 1360 127660 45360 263064 0 0 0 104 697 352 1 1 98 0 0 0 1360 127668 45360 263064 0 0 0 0 573 190 0 4 95 0 0 0 1360 127660 45360 263064 0 0 0 0 873 477 8 2 90 0 0 0 1360 127660 45360 263068 0 0 0 0 943 479 1 2 97 1 0 0 1360 127660 45360 263068 0 0 0 0 942 716 8 2 89 0 0 0 1360 127660 45364 263068 0 0 0 132 732 393 2 2 96 0 0 0 1360 127660 45364 263068 0 0 0 0 562 179 1 0 99 1 0 0 1360 127660 45364 263080 0 0 12 0 567 192 2 1 96 0 0 0 1360 127660 45364 263080 0 0 0 0 779 389 8 1 90 0 0 0 1360 127660 45364 263080 0 0 0 0 740 413 1 6 93 0 0 0 1360 127660 45364 263080 0 0 0 76 690 327 5 2 92 0 0 0 1360 127660 45364 263084 0 0 0 0 790 492 4 1 95 0 0 0 1360 127660 45364 263084 0 0 0 0 689 372 4 3 93 0 0 0 1360 127652 45364 263084 0 0 0 0 962 646 6 6 88 0 0 0 1360 127652 45364 263088 0 0 0 0 877 529 2 1 97 0 0 0 1360 127652 45364 263088 0 0 0 108 768 450 4 2 94 0 0 0 1360 127644 45364 263088 0 0 0 0 669 287 1 1 98 0 0 0 1360 127644 45364 263088 0 0 0 0 680 644 8 1 90 0 0 0 1360 127636 45364 263088 0 0 0 0 803 510 4 2 94 0 0 0 1360 127636 45364 263088 0 0 0 0 647 301 3 1 95 0 0 0 1360 127644 45368 263152 0 0 68 96 745 460 6 2 92 1 0 0 1360 127652 45368 263164 0 0 12 0 782 428 3 2 95 3. 서버 정보 (vmstat ) * vmstat (서버장애시) procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 0 0 1360 92372 44664 278980 0 0 0 0 644 327 2 1 97 0 0 0 1360 92096 44664 278980 0 0 0 0 584 177 1 1 98 0 0 0 1360 92064 44668 278980 0 0 0 16 563 178 4 2 94 0 0 0 1360 91752 44668 278980 0 0 0 0 601 228 0 2 98 0 0 0 1360 91384 44668 278980 0 0 0 0 578 191 1 1 97 0 0 0 1360 90500 44668 278980 0 0 0 0 593 241 2 3 95 0 0 0 1360 89320 44668 278980 0 0 0 0 572 212 1 4 95 0 0 0 1360 89188 44672 278976 0 0 0 44 589 208 2 3 94 0 0 0 1360 89064 44672 278976 0 0 0 0 580 181 0 2 97 0 0 0 1360 88860 44672 278976 0 0 0 0 582 207 2 3 95 0 0 0 1360 88792 44672 278976 0 0 0 0 565 173 1 4 94 0 0 0 1360 88728 44672 278976 0 0 0 32 594 186 0 1 98 0 0 0 1360 88552 44672 278976 0 0 0 0 607 216 1 2 96 0 0 0 1360 88164 44672 278976 0 0 0 0 566 170 1 1 98 0 0 0 1360 88156 44672 278976 0 0 0 0 580 172 0 1 98 0 0 0 1360 87848 44672 278976 0 0 0 0 618 213 1 2 97 0 0 0 1360 87356 44672 278976 0 0 0 28 615 266 4 3 93 0 0 0 1360 86848 44672 278976 0 0 0 0 575 180 1 2 97 0 0 0 1360 86848 44672 278976 0 0 0 0 570 166 1 5 94 0 0 0 1360 86756 44672 278976 0 0 0 0 587 193 1 2 97 0 0 0 1360 86700 44672 278976 0 0 0 0 587 173 1 2 97 0 0 0 1360 86492 44672 278976 0 0 0 24 577 205 2 2 95 0 0 0 1360 86300 44672 278976 0 0 0 0 569 179 2 6 92 0 0 0 1360 85988 44672 278976 0 0 0 0 582 188 1 1 98 0 0 0 1360 85540 44672 278976 0 0 0 0 606 260 3 2 95 0 0 0 1360 85416 44672 278976 0 0 0 0 571 170 1 2 97 0 0 0 1360 84944 44672 278976 0 0 0 40 609 270 2 2 96 0 0 0 1360 83992 44672 278976 0 0 0 0 653 289 2 3 95 1 0 0 1360 82340 44672 278980 0 0 0 0 640 294 3 2 94 0 0 0 1360 80124 44672 278980 0 0 0 0 590 250 1 4 95 0 0 0 1360 79724 44672 278980 0 0 0 48 605 202 2 6 92
* netstat -n | grep EST (평상시)tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52493 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52492 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52486 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:22 xxx.xxx.29.4:4586 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52575 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52571 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:80 xxx.xxx.152.9:4608 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52567 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52553 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:52493 xxx.xxx.xx.xx:8009 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:52492 xxx.xxx.xx.xx:8009 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:52486 xxx.xxx.xx.xx:8009 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:52571 xxx.xxx.xx.xx:8009 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:52575 xxx.xxx.xx.xx:8009 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:52567 xxx.xxx.xx.xx:8009 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:52553 xxx.xxx.xx.xx:8009 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:52475 xxx.xxx.xx.xx:8009 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:52478 xxx.xxx.xx.xx:8009 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:52464 xxx.xxx.xx.xx:8009 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52478 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52475 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52464 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:80 xxx.xxx.9.59:1625 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:51512 database-ip:1433 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:51513 database-ip:1433 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:51778 database-ip:1433 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:51778 database-ip:1433 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:50330 database-ip:1433 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:50742 database-ip:1433 ESTABLISHED tcp 0 68 xxx.xxx.xx.xx:50914 database-ip:1433 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:50913 database-ip:1433 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:50952 database-ip:1433 ESTABLISHED tcp 0 904 xxx.xxx.xx.xx:22 xxx.xxx.xx.x:1112 ESTABLISHED-------------------------------------------------------------------------------------------------총 ESTABLISHED 개수 : 33개Database 연동관련 : 9개8009 포트 기타 포트 : 10 개기타 포트 8009 포트 : 10 개80 포트 client : 2 개 3. 서버 정보 (netstat : Established) * netstat -n | grep EST (서버장애시) Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 webserver-ip:80 xxx..xxx.49.174:19848 ESTABLISHED tcp 0 0 webserver-ip:8009 webserver-ip:36670 ESTABLISHED tcp 0 0 webserver-ip:80 xxx.xxx.242.71:2976 ESTABLISHED tcp 0 0 webserver-ip:22 xxx.xxx.29.4:4586 ESTABLISHED tcp 0 0 webserver-ip:8009 webserver-ip:36703 ESTABLISHED tcp 0 0 webserver-ip:80 xxx.xxx.197.146:1682 ESTABLISHED tcp 0 0 webserver-ip:8009 webserver-ip:36696 ESTABLISHED : : : : 기타 15개 동일 내용 (상동) tcp 0 0 webserver-ip:80 xxx.xxx.xxx.xx:2177 ESTABLISHED tcp 0 0 webserver-ip:8009 webserver-ip:36717 ESTABLISHED : : : : 기타 28개 동일 내용 (상동) tcp 0 0 webserver-ip:80 xxx.xxx.158.13:3051 ESTABLISHED tcp 0 0 webserver-ip:8009 webserver-ip:36744 ESTABLISHED : : : : 기타 36개 동일 내용 (상동) tcp 0 0 webserver-ip:80 xxx.xxx.xxx.107:2604 ESTABLISHED tcp 0 0 webserver-ip:8009 webserver-ip:36774 ESTABLISHED : : : : 기타 6개 동일 내용 (상동) tcp 0 0 webserver-ip:36817 webserver-ip:8009 ESTABLISHED : : : : 기타 106개 동일 내용 (상동) tcp 0 0 webserver-ip:8009 webserver-ip:36822 ESTABLISHED : : : : 기타 20개 동일 내용 (상동) tcp 0 0 webserver-ip:80 xx8.xxx.31.12:3735 ESTABLISHED tcp 0 0 webserver-ip:80 xxx.xxx.2.102:2354 ESTABLISHED tcp 0 0 webserver-ip:35038 database-ip:1433 ESTABLISHED tcp 0 0 webserver-ip:35213 database-ip:1433 ESTABLISHED tcp 0 0 webserver-ip:35214 database-ip:1433 ESTABLISHED tcp 0 0 webserver-ip:33816 database-ip:1433 ESTABLISHED tcp 0 0 webserver-ip:80 xxx.215.xx5.55:3877 ESTABLISHED : : : : 기타 23개 동일 내용 (상동) ------------------------------------------------------------------------------------------------- 총 ESTABLISHED 개수 : 256개 Database 연동관련 : 4개 8009 포트 기타 포트 : 102 개 기타 포트 8009 포트 : 107 개 80 포트 client : 32 개
* GC (평상시)[GC 39885K->37285K(40124K), 0.0071930 secs][GC 39887K->37605K(40324K), 0.0100080 secs][Full GC 37605K->16401K(40324K), 0.7960270 secs][GC 19328K->17062K(40124K), 0.0126660 secs][GC 19892K->17599K(40124K), 0.0111760 secs][GC 20432K->18012K(40124K), 0.0408080 secs][GC 20850K->18626K(40124K), 0.0109500 secs][GC 21463K->18830K(40124K), 0.0118570 secs][GC 21451K->19196K(40124K), 0.0095420 secs][GC 22037K->19555K(40124K), 0.0093910 secs][GC 22399K->20149K(40124K), 0.0098520 secs][GC 22854K->20379K(40124K), 0.0083420 secs][GC 22996K->20612K(40124K), 0.0076090 secs][GC 23455K->20845K(40124K), 0.0063710 secs][GC 23655K->21308K(40124K), 0.0075850 secs][GC 24370K->21892K(40124K), 0.0072240 secs][GC 24732K->22138K(40124K), 0.0060080 secs][GC 24949K->22380K(40124K), 0.0061300 secs][GC 39762K->37170K(40124K), 0.0058780 secs][GC 39727K->37429K(40300K), 0.0068420 secs][Full GC 37429K->16191K(40300K), 0.7953360 secs][GC 39679K->37276K(40124K), 0.0069350 secs][GC 39925K->37610K(40300K), 0.0076660 secs][Full GC 37610K->14276K(40300K), 0.8748960 secs][GC 17581K->15373K(40124K), 0.0095520 secs][GC 18449K->15861K(40124K), 0.0104900 secs][GC 18483K->16526K(40124K), 0.0137420 secs][GC 19146K->17209K(40124K), 0.0346310 secs][GC 19832K->17766K(40124K), 0.0206310 secs][GC 20389K->17794K(40124K), 0.0103710 secs][GC 20391K->17813K(40124K), 0.0061950 secs][GC 20584K->17906K(40124K), 0.0054300 secs][GC 20498K->17998K(40124K), 0.0061370 secs][GC 20617K->18008K(40124K), 0.0054560 secs][GC 20730K->18018K(40124K), 0.0055230 secs][GC 20619K->18137K(40124K), 0.0055160 secs][GC 20735K->18145K(40124K), 0.0058770 secs] 3. 서버 다운 정보 (startup.sh 옵션 : GC 정보) • GC (서버장애시) • [GC 35807K->32869K(44368K), 0.0074160 secs] • [GC 44010K->41290K(44368K), 0.0071010 secs] • [Full GC 41290K->16677K(44368K), 0.9458450 secs] • [GC 22642K->20059K(44368K), 0.0078300 secs] • [GC 33637K->30787K(44368K), 0.0072790 secs] • [GC 43817K->41049K(44368K), 0.0063940 secs] • [GC 43865K->41657K(44496K), 0.0124660 secs] • [Full GC 41657K->17253K(44496K), 0.9132000 secs] • [GC 23044K->20879K(44368K), 0.0138540 secs] • [GC 23691K->21423K(44368K), 0.0141100 secs] • [GC 24234K->21868K(44368K), 0.0131710 secs] • [GC 24684K->23311K(44368K), 0.0262750 secs] • [GC 26126K->23893K(44368K), 0.0163500 secs] • [GC 26708K->24277K(44368K), 0.0124560 secs] • [GC 27081K->24749K(44368K), 0.0143000 secs] • [GC 27563K->25908K(44368K), 0.0191340 secs] • [GC 28708K->26359K(44368K), 0.0139570 secs] • [GC 29175K->27535K(44368K), 0.0185610 secs] • GC 30351K->27820K(44368K), 0.0125210 secs] • [GC 30635K->28441K(44368K), 0.0281460 secs] • [GC 31247K->28785K(44368K), 0.0128440 secs] • [GC 31600K->29805K(44368K), 0.0275490 secs] • [GC 32621K->31303K(44368K), 0.0394960 secs] • [GC 34118K->33026K(44368K), 0.0486900 secs] • [GC 35836K->34692K(44368K), 0.0554940 secs] • [GC 37508K->36243K(44368K), 0.0576330 secs] • [GC 39056K->37789K(44368K), 0.0641460 secs] • [GC 40605K->39362K(44368K), 0.0687840 secs] • [GC 42178K->41334K(44368K), 0.0804310 secs] • [GC 44149K->42679K(45520K), 0.0803990 secs] • --------------------------------------------- 여기부터 서버 느려서 페이지가 뜨지않음 • [Full GC 42679K->31065K(45520K), 1.3358060 secs] • -------------------------------------------- 페이지 안뜬상태에서 조금있다가 로그 남음 • [GC 34649K->33041K(55748K), 0.0914980 secs] • -------------------------------------------- 페이지 안뜬상태에서 조금있다가 로그 남음 • [GC 36624K->35406K(55748K), 0.1062460 secs] • -------------------------------------------- 웹서비스를 위해 톰캣을 재시작처리함
* ps 내용 (평상시)USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMANDroot 1 0.0 0.0 1400 468 ? S May15 0:19 initroot 2 0.0 0.0 0 0 ? SW May15 0:00 [migration_CPU0]root 3 0.0 0.0 0 0 ? SW May15 0:00 [migration_CPU1]root 4 0.0 0.0 0 0 ? SW May15 0:00 [keventd]root 5 0.0 0.0 0 0 ? SWN May15 0:00 [ksoftirqd_CPU0]root 6 0.0 0.0 0 0 ? SWN May15 0:00 [ksoftirqd_CPU1]root 7 0.0 0.0 0 0 ? SW May15 2:43 [kswapd]root 8 0.0 0.0 0 0 ? SW May15 0:05 [bdflush]root 9 0.0 0.0 0 0 ? SW May15 0:10 [kupdated]root 10 0.0 0.0 0 0 ? SW May15 0:00 [mdrecoveryd]root 16 0.0 0.0 0 0 ? SW May15 0:00 [scsi_eh_0]root 19 0.0 0.0 0 0 ? SW May15 1:08 [kjournald]root 75 0.0 0.0 0 0 ? SW May15 0:00 [khubd]root 167 0.0 0.0 0 0 ? SW May15 0:00 [kjournald]root 441 0.0 0.0 1472 524 ? S May15 0:02 syslogd -m 0root 445 0.0 0.0 1400 444 ? S May15 0:00 klogd -xroot 483 0.0 0.1 3356 1260 ? S May15 0:18 /usr/sbin/sshdroot 494 0.0 0.0 2036 608 ? S May15 0:00 xinetd -stayalive -reuse -pidfile /var/runroot 513 0.0 0.1 5604 1512 ? S May15 0:43 [sendmail]smmsp 522 0.0 0.1 4952 1372 ? S May15 0:00 [sendmail]root 532 0.0 0.0 1460 580 ? S May15 0:01 cronddaemon 550 0.0 0.0 1444 544 ? S May15 0:00 [atd]root 561 0.0 0.0 3432 544 ? S May15 0:00 rhnsd --interval 120root 567 0.0 0.0 1376 368 tty1 S May15 0:00 /sbin/mingetty tty1root 3974 0.0 0.2 6936 2156 ? S May24 0:29 /usr/sbin/sshdroot 4059 0.0 0.1 3228 1176 ? S May24 0:01 /usr/libexec/openssh/sftp-serverroot 4087 0.0 0.1 6868 2048 ? S May24 0:01 /usr/sbin/sshdroot 4089 0.0 0.1 3208 1108 ? S May24 0:00 /usr/libexec/openssh/sftp-serverroot 26908 8.2 57.8 728308 596340 ? S 08:58 10:17 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 26925 0.0 0.2 4972 2344 ? S 08:58 0:00 /usr/local/httpd/bin/httpd -k startroot 27415 0.0 0.1 4400 1464 pts/0 S 11:02 0:00 -bashnobody 27502 0.0 0.2 5080 2556 ? S 11:08 0:00 [httpd] nobody 27511 0.0 0.2 5076 2524 ? S 11:09 0:00 [httpd]nobody 27527 0.0 0.2 5080 2552 ? S 11:11 0:00 [httpd]nobody 27528 0.0 0.2 5080 2552 ? S 11:11 0:00 [httpd]nobody 27530 0.0 0.2 5076 2520 ? S 11:12 0:00 [httpd]nobody 27531 0.0 0.2 5072 2544 ? S 11:12 0:00 [httpd]nobody 27533 0.0 0.2 5076 2516 ? S 11:12 0:00 [httpd]nobody 27535 0.0 0.2 5068 2512 ? S 11:14 0:00 [httpd]nobody 27540 0.0 0.2 5068 2508 ? S 11:17 0:00 [httpd]nobody 27543 0.0 0.2 5068 2504 ? S 11:17 0:00 [httpd]nobody 27544 0.0 0.2 5068 2504 ? S 11:17 0:00 [httpd]root 27545 0.0 0.0 2776 836 pts/0 R 11:17 0:02 ps -aux 3. 서버 정보 (ps -aux ) * ps 내용 (평상시) USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 1400 468 ? S May15 0:19 init root 2 0.0 0.0 0 0 ? SW May15 0:00 [migration_CPU0] root 3 0.0 0.0 0 0 ? SW May15 0:00 [migration_CPU1] root 4 0.0 0.0 0 0 ? SW May15 0:00 [keventd] root 5 0.0 0.0 0 0 ? SWN May15 0:00 [ksoftirqd_CPU0] root 6 0.0 0.0 0 0 ? SWN May15 0:00 [ksoftirqd_CPU1] root 7 0.0 0.0 0 0 ? SW May15 2:43 [kswapd] root 8 0.0 0.0 0 0 ? SW May15 0:05 [bdflush] root 9 0.0 0.0 0 0 ? SW May15 0:10 [kupdated] root 10 0.0 0.0 0 0 ? SW May15 0:00 [mdrecoveryd] root 16 0.0 0.0 0 0 ? SW May15 0:00 [scsi_eh_0] root 19 0.0 0.0 0 0 ? SW May15 1:08 [kjournald] root 75 0.0 0.0 0 0 ? SW May15 0:00 [khubd] root 167 0.0 0.0 0 0 ? SW May15 0:00 [kjournald] root 441 0.0 0.0 1472 524 ? S May15 0:02 syslogd -m 0 root 445 0.0 0.0 1400 444 ? S May15 0:00 klogd -x root 483 0.0 0.1 3356 1260 ? S May15 0:18 /usr/sbin/sshd root 494 0.0 0.0 2036 608 ? S May15 0:00 xinetd -stayalive -reuse -pidfile /var/run root 513 0.0 0.1 5604 1512 ? S May15 0:44 [sendmail] smmsp 522 0.0 0.1 4952 1372 ? S May15 0:00 [sendmail] root 532 0.0 0.0 1460 580 ? S May15 0:01 crond daemon 550 0.0 0.0 1444 544 ? S May15 0:00 [atd] root 561 0.0 0.0 3432 544 ? S May15 0:00 rhnsd --interval 120 root 567 0.0 0.0 1376 368 tty1 S May15 0:00 /sbin/mingetty tty1 root 3974 0.0 0.2 6936 2156 ? S May24 0:29 /usr/sbin/sshd root 4059 0.0 0.1 3228 1176 ? S May24 0:01 /usr/libexec/openssh/sftp-server root 4087 0.0 0.1 6868 2048 ? S May24 0:01 /usr/sbin/sshd root 4089 0.0 0.1 3208 1108 ? S May24 0:00 /usr/libexec/openssh/sftp-server root 26908 5.8 58.5 738628 603804 ? S 08:58 9:33 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 26925 0.0 0.2 4972 2344 ? S 08:58 0:01 /usr/local/httpd/bin/httpd -k start root 27066 0.3 58.5 738628 603804 ? S 10:07 0:16 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27067 0.3 58.5 738632 603816 ? S 10:07 0:19 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27072 0.4 58.5 738628 603816 ? S 10:07 0:22 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27073 0.4 58.5 738628 603816 ? S 10:07 0:23 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27074 0.2 58.5 738628 603816 ? S 10:07 0:13 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27075 0.2 58.5 738628 603816 ? S 10:07 0:13 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27079 0.0 58.5 738628 603816 ? S 10:07 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27080 0.0 58.5 738628 603816 ? S 10:07 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27081 0.0 58.5 738628 603816 ? S 10:07 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27082 0.0 58.5 738628 603816 ? S 10:07 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27087 0.0 58.5 738628 603816 ? S 10:07 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27088 0.0 58.5 738628 603816 ? S 10:07 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27089 0.0 58.5 738628 603816 ? S 10:07 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27090 0.3 58.5 738628 603816 ? S 10:07 0:15 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27415 0.0 0.1 4400 1464 pts/0 S 11:02 0:00 -bash nobody 27620 0.0 0.2 5068 2512 ? S 11:27 0:00 [httpd] nobody 27622 0.0 0.2 5076 2524 ? S 11:27 0:00 [httpd] nobody 27624 0.0 0.2 5076 2516 ? S 11:28 0:00 [httpd] nobody 27633 0.0 0.2 5080 2552 ? S 11:30 0:00 [httpd] nobody 27637 0.0 0.2 5080 2552 ? S 11:30 0:00 [httpd] nobody 27638 0.0 0.2 5080 2552 ? S 11:30 0:00 [httpd] nobody 27642 0.0 0.2 5080 2548 ? S 11:30 0:00 [httpd] nobody 27643 0.0 0.2 5068 2508 ? S 11:30 0:00 [httpd] nobody 27645 0.0 0.2 5068 2496 ? S 11:31 0:00 [httpd] nobody 27646 0.0 0.2 5068 2488 ? S 11:31 0:00 [httpd] nobody 27647 0.0 0.2 5052 2480 ? S 11:31 0:00 [httpd] nobody 27648 0.0 0.2 5052 2480 ? S 11:31 0:00 [httpd] nobody 27649 0.0 0.2 5052 2480 ? S 11:31 0:00 [httpd] nobody 27650 0.0 0.2 5052 2480 ? S 11:31 0:00 [httpd] nobody 27651 0.0 0.2 5052 2488 ? S 11:31 0:00 [httpd] nobody 27652 0.0 0.2 5052 2492 ? S 11:31 0:00 [httpd] nobody 27653 0.0 0.2 5044 2472 ? S 11:32 0:00 [httpd] nobody 27654 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd] nobody 27655 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd] nobody 27656 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd] nobody 27657 0.0 0.2 5052 2488 ? S 11:32 0:00 [httpd] nobody 27658 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd] nobody 27659 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd] nobody 27660 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd] nobody 27661 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd] nobody 27662 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd] nobody 27663 0.0 0.2 5060 2496 ? S 11:32 0:00 [httpd] nobody 27664 0.0 0.2 5060 2484 ? S 11:32 0:00 [httpd] nobody 27665 0.0 0.2 5052 2492 ? S 11:32 0:00 [httpd] nobody 27666 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd] nobody 27667 0.0 0.2 5060 2484 ? S 11:32 0:00 [httpd] nobody 27668 0.0 0.2 5068 2500 ? S 11:32 0:00 [httpd] nobody 27669 0.0 0.2 5068 2496 ? S 11:32 0:00 [httpd] nobody 27670 0.0 0.2 5068 2488 ? S 11:32 0:00 [httpd] nobody 27671 0.0 0.2 5068 2496 ? S 11:32 0:00 [httpd] nobody 27672 0.0 0.2 5060 2492 ? S 11:32 0:00 [httpd] nobody 27673 0.0 0.2 5068 2484 ? S 11:32 0:00 [httpd] nobody 27674 0.0 0.2 5076 2500 ? S 11:32 0:00 [httpd] nobody 27675 0.0 0.2 5060 2484 ? S 11:32 0:00 [httpd] nobody 27676 0.0 0.2 5076 2496 ? S 11:32 0:00 [httpd] nobody 27677 0.0 0.2 5052 2488 ? S 11:32 0:00 [httpd] nobody 27678 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd] nobody 27679 0.0 0.2 5068 2504 ? S 11:32 0:00 [httpd] nobody 27680 0.0 0.2 5060 2488 ? S 11:32 0:00 [httpd] nobody 27681 0.0 0.2 5060 2488 ? S 11:33 0:00 [httpd] nobody 27682 0.0 0.2 5060 2484 ? S 11:33 0:00 [httpd] nobody 27683 0.0 0.2 5052 2488 ? S 11:33 0:00 [httpd] nobody 27684 0.0 0.2 5068 2496 ? S 11:33 0:00 [httpd] nobody 27685 0.0 0.2 5060 2484 ? S 11:33 0:00 [httpd] nobody 27686 0.0 0.2 5052 2488 ? S 11:33 0:00 [httpd] root 27687 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27688 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27689 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27690 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m nobody 27691 0.0 0.2 5052 2484 ? S 11:33 0:00 [httpd] root 27692 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27693 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27694 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27695 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m nobody 27696 0.0 0.2 5060 2492 ? S 11:33 0:00 [httpd] nobody 27697 0.0 0.2 5076 2500 ? S 11:33 0:00 [httpd] nobody 27698 0.0 0.2 5060 2492 ? S 11:33 0:00 [httpd] nobody 27699 0.0 0.2 5060 2492 ? S 11:33 0:00 [httpd] nobody 27700 0.0 0.2 5076 2508 ? S 11:33 0:00 [httpd] root 27701 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27702 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27703 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27704 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m nobody 27705 0.0 0.2 5068 2508 ? S 11:33 0:00 [httpd] nobody 27706 0.0 0.2 5068 2496 ? S 11:33 0:00 [httpd] nobody 27707 0.0 0.2 5044 2472 ? S 11:33 0:00 [httpd] nobody 27708 0.0 0.2 5076 2508 ? S 11:33 0:00 [httpd] nobody 27709 0.0 0.2 5068 2496 ? S 11:33 0:00 [httpd] nobody 27710 0.0 0.2 5104 2540 ? S 11:33 0:00 [httpd] nobody 27711 0.0 0.2 5068 2496 ? S 11:33 0:00 [httpd] nobody 27712 0.0 0.2 5044 2476 ? S 11:33 0:00 [httpd] root 27713 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27714 0.1 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27715 0.1 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27716 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27717 0.2 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27718 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27719 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27720 0.2 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 27721 0.0 0.0 2844 904 pts/0 R 11:33 0:02 ps -aux nobody 27722 0.0 0.2 4996 2384 ? S 11:33 0:00 [httpd]
* 메모리 heap (평상시)Tue May 25 11:47:28 KST 2004F:443MB T:508 MB F:469MB T:508 MB Tue May 25 11:49:23 KST 2004F:466MB T:508 MB F:491MB T:508 MB Tue May 25 11:51:11 KST 2004F:466MB T:508 MB F:495MB T:508 MB Tue May 25 11:53:27 KST 2004F:434MB T:508 MB F:476MB T:508 MBTue May 25 12:08:09 KST 2004F:347MB T:508 MB F:471MB T:508 MB Tue May 25 12:08:25 KST 2004F:452MB T:508 MB F:468MB T:508 MB Tue May 25 12:10:32 KST 2004F:481MB T:508 MB F:487MB T:508 MBTue May 25 12:15:33 KST 2004F:431MB T:508 MB F:470MB T:508 MBTue May 25 13:47:02 KST 2004F:148MB T:508 MB F:462MB T:508 MB Tue May 25 14:38:21 KST 2004F:77MB T:508 MB F:461MB T:508 MB 3. 서버 정보 (메모리 heap 변화 모니터링) • 메모리 heap (서버장애시) • JSP 페이지 자체가 뜨지 않아 파악하지 못함
4. N/W 정보 (10M 단독 라인) [ 주간 ] [ 월간 ] [ 년간 ]
* Runtime.exec()를 통해 JVM 외부의 shell script나 c program 을 호출하는 부분이 있는지요?현재 다운되는 서버 내의 JVM에서는 shell이나 c를 호출하는 부분이 없습니다.* 프로그램 어디선가 Socket 혹은 HttpUrlConnection으로 "localhost:80" 으로 접속을 시도하는 부분이 있는지요?어떤 용도로 쓰였는지요? Socket –파일 서버 와 연동중 무료이미지 관련 부분이 사용되고 있습니다. (첨부파일참조) URLConnection –파일서버와 연동중 폴더내 파일리스트를 가져오는 부분이 있습니다. (첨부파일참조)* java process 개수가 80+1 에서 80 이 갖는 의미가 뭘까요? 어떤 파라메터가 80인가요? 저도 정확히는 모르겠습니다만 아파치 프로세스중에 좀비가 가끔씩 뜨지만 잠시 떴다가 금방 사라지며, 이부분에 대해 자바서비스넷 자료를 찾는중에 금방 사라지는 좀비프로세스는 괜찮다는 글을 읽은걸로 알고있습니다.* dump파일 생성 관련 ibm JDK 가 아닌 sun JDK사용으로 인하여 dump 파일이 생기지 않습니다.* server.xml의 내용<Host name="test.test.co.kr" debug="0" appBase=“appbase" liveDeploy="true" unpackWARs="true" autoDeploy="true"><Context cookies="true" docBase=“test" path="/test" reloadable="true" debug="0" crossContext="false"> <Logger className="org.apache.catalina.logger.FileLogger" prefix="localhost_test_log." suffix=".txt" timestamp="true"/> <Realm className="org.apache.catalina.realm.JDBCRealm" connectionName="sa" connectionPassword=“xxxxx" connectionURL="jdbc:microsoft:sqlserver://database-ip:1433;DatabaseName=test;SelectMethod=cursor;SendStringParametersAsUnicode=true" debug="0" driverName="com.microsoft.jdbc.sqlserver.SQLServerDriver" userTable="T_testmember" userNameCol=“userid" userCredCol=“pwd" userRoleTable="T_룰" roleNameCol="U_룰"/> <ResourceLink global="jdbc/TESTDB" name="jdbc/TESTDB"/> </Context> ~~~~~~~~~~~~이하생략 4. 질문 사항
5. 예상 장애 요인 1 –이미지 다운로드 관련 로직 (첨부파일 참조 ) : 웹서버 파일서버 소켓연동 ( 파일서버에서 socket은 체크후 해제 처리를 합니다. 현재 첨부파일소스는 웹서버의 client소스 입니다.) 2 –폴더내 이미지 리스트 호출 관련 로직 : 웹서버 파일서버 URLConnection 이용 아래 소스 참고 FileServer페이지 <%@page 선언부 %> <%@page import=“~~”%> <% String fpath=""; fpath=(String)(request.getParameter("fpath")); //---request받음 int filesize=0; try { File file = new File(fpath); filesize=(int)file.length(); if(filesize<1){ String fileList[]=new String[filesize]; if(file.list()!=null){ fileList=file.list(); for(int k=0;k<fileList.length;k++){ File file_List = new File(fpath+fileList[k]); out.println(file_List.getName()); }//-------end for for(int k=0;k< }else{//---if(file.list()!=null out.println(""); } file_List=null; }else{ out.println(""); } } catch(Exception e) { e.printStackTrace(); }finally{ System.out.println("fpath==>"+fpath); } %> WebServer페이지 public static Vector mbJspFileList(String path){ Vector list = new Vector(); BufferedInputStream is = null; try { URL url = new URL("http://FileServer페이지주소?fpath="+path); URLConnection conn = url.openConnection(); if(conn!=null){ is = new BufferedInputStream(conn.getInputStream()); int _value; char value; StringBuffer sb = new StringBuffer(); while((_value = is.read())!=-1){ value = (char)_value; if(value != '\n') sb.append(value); else { list.addElement(sb.toString()); sb = new StringBuffer(); } } }else{ list=null; } }catch(Exception e){ e.printStackTrace(); }finally { if(is!=null){try {is.close();}catch(Exception e){}} } return list; }
