Download
1 / 24
240 likes | 363 Vues
This guide explores strategies for identifying hidden file system issues, particularly in Lustre environments, where problems may exist without explicit errors. It outlines methods for quickly detecting non-error problems through command-line tools (e.g., grep) and provides practical API calls to assess system health. The document also emphasizes best practices for benchmarking and ensuring optimal performance, alongside wisdom from industry leaders, to help enhance file system reliability and efficiency.
E N D
Detecting Hidden File System Problems Nicholas P. Cardo National Energy Research Scientific Computing Center Lawrence Berkeley National Laboratory cardo@nersc.gov
The Dilemma • File system problems can exist • Sometimes there are no errors • How to detect a non-error problem • How to do it quickly # grep -ilustre messages | grep -i error | wc –l 5836 (6 days)
Words of Wisdom “Strive for perfection in everything. Take the best that exists and make it better. If it does not exist, create it. Accept nothing nearly right or good enough.” Sir Henry Royce co-founder of Rolls-Royce
Useful API Calls • llapi_ping ….. ping lustre components • llapi_file_create … create a file w/stripes • llapi_file_get_stripe … read stripe info
Wanted: Dead or Alive hc=/proc/fs/lustre/health_check for node in $mds $oss do xx=`ssh $node cat $hc` if [ “$xx” != “healthy” ] then echo “$node not healthy” fi done
OST Ping /proc/fs/lustre/osc Y N ., .., num_refs readdir DT_DIR Y N N Y rc=0 llapi_ping Y N ping failed
How About dir = opendir("/proc/fs/lustre/osc"); while((d = readdir(dir)) != NULL) { if (!strcmp(d->d_name,".")) continue; if (!strcmp(d->d_name,"..")) continue; if (!strcmp(d->d_name,"num_refs")) continue; if ( d->d_type == DT_DIR ) { if ((llrc=llapi_ping("osc”,d->d_name)) != 0) fprintf(stderr,”problem %s\n”,osc); } }
Benchmarking “benchmark: to study (as a competitor’s product or business practices) in order to improve the performance of one’s own company.” www.webster.com “Benchmarking is a process of comparing one’s business processes and performance metrics to industry bests and/or best practices from other industries. Dimensions typically measured are quality, time and cost, improvements from learning mean doing things better, faster, and cheaper.” www.wikipedia.org Nick’s Corollary BM1: benchmarks can make anything look good.
Create File llapi_file_create llapi_file_get_stripe stripe cnt ost match rc=0 Y Y Y N N N ost mismatch cnt mismatch get failed
Create Striped File • rc = llapi_file_create(fname,0,ostnum,1,0); • lum = malloc(LOV_EA_MAX(lum)); • /* read back the stripe data */ • rc = llapi_file_get_stripe(fname,lum); • if (!rc) { • /* check the stripe count */ • if (lum->lmm_stripe_count != 1) { • fprintf(stderr,"%s: stripe count mismatch: %s\n", • ProgName,fname); • } else { • /* check the ost number */ • oi = lum->lmm_objects[lum->lmm_stripe_offset].l_ost_idx; • if (oi != ostnum){ • fprintf(stderr,"%s: ost mismatch: %s\n", • ProgName,fname); • } • } • }
Write Test Start time size End time Y N write MB/s
time(&b_timval); /* start the clock */ • for (cnt=0;cnt<fcnt;cnt++) • write(ifd,pattern,sizeof(pattern)); • time(&e_timval); /* stop the clock */ • fsync(ifd); /* flush the cache */ • /* • * calculate write mega bytes per second • */ • wmbs=(float)fcnt*sizeof(pattern)/(float)(e_timval-b_timval)/1048576;
Read Test Start time size End time Y N read MB/s
lseek(ifd,0,SEEK_SET); /* rewind */ • time(&b_timval); /* start the clock */ • for(cnt=0;cnt<fcnt;cnt++) • read(ifd,buf,1024); • time(&e_timval); /* stop the clock */ • /* • * calculate read mega bytes per second • */ • rmbs=(float)fcnt*sizeof(pattern)/(float)(e_timval-b_timval)/1048576.0;
Data Check seek 0 size Y N read Pattern mismatch pattern N Y
lseek(ifd,0,SEEK_SET); /* rewind */ • /* • * repeat the read, but this time check the results • * to make sure that what we read matches what we wrote • */ • for(cnt=0;cnt<fcnt;cnt++) { • read(ifd,buf,sizeof(buf)); • if(strncmp(buf,pattern,sizeof(pattern))) { • fprintf(stderr,"%s: read/write mismatch: %s\n", ProgName,fname); • rc = 1; • break; • } • }
Get OSTs dp = opendir("/proc/fs/lustre/lov"); while((de=readdir(dp)) != NULL) { /* we only want the requested file system */ if (strncmp(de->d_name,fsp,strlen(fsp))) continue; sprintf(rc.fsname,"%s\0",fsname); /* get the number of osts */ sprintf(pname,"/proc/fs/lustre/lov/%s/numobd\0",de->d_name); ffd = fopen(pname,"r"); fscanf(ffd,"%d",&rc.numobd); fclose(ffd); break; }
In a Nutshell MPI_Init Create Get OSTs Write Scale OK Read Verify
MPI_Init (&argc, &argv); MPI_Comm_rank (MPI_COMM_WORLD, &rank); MPI_Comm_size (MPI_COMM_WORLD, &size); /* Parse the command line options */ while ((optchr=getopt(argc,argv,"f:s:t:")) != EOF) { switch (optchr) { case 'f': fsname = optarg; break; case 's': fsize = atoi(optarg); break; case 't': tstdir = optarg; break; default : MPI_Abort(MPI_COMM_WORLD,1); } } fs = getfsinfo(fsname); if (size != fs->numobd) { fprintf(stderr,"%s: tasks(%d) != osts(%d)\n", ProgName,size,fs->numobd); MPI_Abort(MPI_COMM_WORLD,1); } /* construct the name of the test file */ sprintf(fname,"%s/testfile.%s.%d\0",tstdir,fs->fsname,rank); rc = StripeFile(fname,rank); /*create the striped file */ if(!rc) rc = RDWR(fname,rank,fsize,fsname);/* read/write test */ if (!rc) unlink (fname); /* delete the test file */ MPI_Finalize();
