1 / 15

Introduction to File Systems

This course presents an overview of various file systems utilized in computer environments, detailing their structures, types, and usage policies. Key topics include the difference between permanent and temporary file systems, particularly focusing on user directories like $HOME, $SCRATCH, and $TMPDIR. The course emphasizes the importance of managing disk usage and understanding backup protocols. Users will learn practical skills related to file handling, quota checking, and space management on ECFS and HPCA systems to optimize their workflows in a supercomputing environment.

emmy
Télécharger la présentation

Introduction to File Systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Introduction to File Systems Computer User Training Course 2004 Carsten Maaß User Support

  2. Overview • Workstation server ecgate • HOME • SCRATCH • SCRATCHDIR • (TMPDIR) • Supercomputer: hpca • HOME • TEMP • TMPDIR • File systems cross-mounted • Important remarks - Summary

  3. File system types • Permanent • $HOME is the only permanent file system • backed up daily • Temporary • files are kept as long as possible • no backups • Automatically deleted • files are deleted at the end of your job or interactive session • no backups • ECFS (archive) • long term archive • for excess data/files

  4. Structure of ECMWF file systems

  5. ecgate: $HOME NFS mounted from HP High Availability servers, ≈ 14GB • backups • permanent files: profile files + e.g. utilities, source, jobs • quota of 40 MB • ecquota–v will show disk usage $HOME examples /home/$GROUP/$USER /home/ms/$GROUP/$USER /home/ectrain/tra /home/ms/it/cnv /home/ms/spfrpech/aum

  6. ecgate: $SCRATCH (locally) mounted, currently 1 file system, ≈ 530 GB • to be used for temporary data • quota: 2 GB (soft limit 1.5 GB) • select/delete is running: • when 80% full, file system cleans up to 50% • files newer than 3 days not removed • Please remove all files no longer needed! $SCRATCH examples /scratch/$GROUP/$USER /scratch/ms/$GROUP/$USER /scratch/ectrain/tra /scratch/ms/it/cn0 /scratch/ms/spfrpech/aum

  7. ecgate: $SCRATCHDIR Part of $SCRATCH, ≈ 530 GB • automatically deleted at the end of job • suitable for temporary data $SCRATCHDIR examples: interactive: /scratch/$GROUP/$USER/scratchdir/$HOST.PID /scratch/us/usc/scratchdir/ecgate.144600 batch: /scratch/$GROUP/$USER/scratchdir/$HOST.JobID.StepID /scratch/us/usc/scratchdir/ecgate.31487.0

  8. ecgate: $TMPDIR Local file system, ≈ 2.0 GB • Don’t use it! • used by the system for temporary file storage • automatically deleted at the end of job/session $TMPDIR examples: • interactive: • /var/tmp/tmpdir/$USER/jtmp.PID • /var/tmp/tmpdir/usc/jtmp.14460 • batch: • /var/tmp/nqstmp/$USER/host.JobID.StepID • /var/tmp/nqstmp/usc/ecgate.31489.0

  9. hpca: $HOME General Parallel File System (GPFS), ≈ 8.7 GB • permanent files: profile files + e.g. utilities, source, jobs • block size 64 KB • quota of 80 MB • quota command will show disk usage $HOME examples /home/$GROUP/$USER /home/ms/$GROUP/$USER /home/ectrain/tra /home/ms/it/cn0 /home/ms/spfrpech/aum more permanent disk space for common library or data installation on request

  10. hpca: $TEMP GPFS, ≈ 1.1 TB (shared between all MS users) • to be used for temporary data • block size 256 KB • select/delete is running • Please remove all files no longer needed! $TEMP examples /hpca/tmp/$GROUP/$USER /hpca/tmp/ms/$GROUP/$USER /hpca/tmp/ectrain/tra /hpca/tmp/ms/it/cn0 /hpca/tmp/ms/spfrpech/aum

  11. hpca: $TMPDIR Part of $TEMP, ≈ 1.1 TB (shared between all MS users) • automatically deleted at the end of job or interactive login • can be used for data $TMPDIR examples: interactive: /hpca/tmp/$GROUP/TMP/JTMP/#/$HOST_$USER_PID /hpca/tmp/us/TMP/JTMP/7/hpca2501_usc_p77840 batch: /hpca/tmp/$GROUP/TMP/BATCH/#/$HOST.JobID.StepID /hpca/tmp/us/TMP/BATCH/2/hpca2401.36350.0

  12. Cross mounted file systems Purpose: facilitate commands like ls etc. on remote machines, not suitable for large data transfers, preferably not to be used in batch on hpca on ecgate

  13. Important remarks – Summary Use only the following file systems

  14. Important remarks – Summary • Only $HOME is backed up • All $HOME file systems are different • A select/delete policy is applied on all temporary file systems • Make use of the environment variables $HOME etc. • Check your quota with • ecquota –v # on ecgate • quota # on hpca • Clear your space as soon as possible • Do not rely on select/delete • ECFS is accessible from both hpca and ecgate

  15. Exercise On Linux desktop or ecgate: • List for your training uid the paths for the following file systems: $HOME, $TMPDIR, $SCRATCH, $SCRATCHDIR. • Check your quotas. • Which is your largest directory/file (in kb)? • Find any file in your directories larger than 100 kb! • How much disk space (in kb) is available in the /home/ectrain file system? • List your hpca HOME directory! • How many directories with the name “Trash” do you find under your HOME directory? • In your HOME directory create a directory called public with the following permissions: rwxrwxr-x.

More Related