Traditionally, users who wanted to run workflows on multiple HCC clusters had to manually transfer the data between resources. While Globus and the high-speed transfer nodes have done much to alleviate the headache associated with transferring files between clusters, users were still often forced to choose between using a cluster which offered the hardware configuration they needed and accessibility to files that could not be easily moved. This issue will forever be a thing of the past due to the new $COMMON file system.
In addition to their $HOME (/home/group/user/) and $WORK (/work/group/user/) directories, all HCC users now also have $COMMON (/common/group/user/) directories. This allows users to access files stored on $COMMON regardless of what cluster they are working on. The $COMMON file system is available on all of HCC’s traditional clusters; Crane, Tusker and Sandhills.
While $COMMON functions most like $WORK, there are a few fundamental differences. Quotas on $COMMON are individually based like $HOME but with larger quotas per user. However, unlike both $HOME and $WORK, additional storage can be purchased on $COMMON if necessary. Like $WORK, $COMMON is not backed up. But, unlike $WORK, $COMMON will not be subject to purging. To prevent file loss, users are strongly encouraged to backup files stored on $COMMON on a regular basis.
Like all file systems, $COMMON has its limitations. Please consider these best practices when using your new directories:
- Do not use $COMMON for high I/O workflows. Active job output should still be directed to $WORK directories for best job performance.
- Limit the number of files in your $COMMON directory. Storing excessively large numbers of files will cause slow access for all users.
- Do not expect your compiled program binaries to work everywhere! $COMMON is available on machines with different CPU architecture, different network connections, etc.
For information on how to access $COMMON and details on its use, please visit Using the /common file system. Any questions, comments and concerns are welcome and should be directed to hcc-support@unl.edu