How to Use Storage Areas?

KUACC

Home Folder Usage on KUACC Parallel File System

The Home folder ( /kuacc/users/username or /scratch/users/username) is located on a parallel file system and is not designed for heavy or long-term storage. To prevent data loss and system performance issues, the following precautions and policies are in place:

What NOT to Do

Do not download or store datasets directly under your Home folder.
Avoid storing completed or inactive project data—move it to your personal storage (e.g., external hard drives or Google Drive).

Disk Usage Limits

Each user is allocated 500 GiB of Home directory storage.
This limit may change at any time without prior notice, depending on overall system usage.
If your usage exceeds the quota, you will be notified via e-mail. Immediate action is required to reduce disk usage.
If no action is taken after notifications, your account will be suspended without warning.

Data Retention and Cleanup

Users with files older than one year will receive e-mail reminders to move or delete data.
Regularly review and clean up your Home folder to stay within quota.

No Backups

The system does not perform backups of the Home folder.
Users are solely responsible for backing up their data (e.g., via Google Drive, external hard drives, etc.).

Account Suspension and Re-activation

If your account is suspended:
- Submit a re-activation request via IT Trackit.
- Inform your supervisor and add them as a participant in your Trackit request.

DC2

Home and Scratch Folder Usage on DC2 Cluster

The DC2 cluster provides access to two parallel file systems: /home and /scratch

These storage areas are designed for different purposes and must be used accordingly to ensure optimal performance and data reliability.

Storage Usage Policy

/home Directory

Intended for code, scripts, and long-term storage of important data.
Each user is allocated 1 TB of storage under /home
This limit may be adjusted at any time without prior notice based on overall system usage.
If your usage exceeds the quota, you will be notified via e-mail. Immediate action is expected to reduce usage.
Failure to take action after notification may result in account suspension without warning.
Users with files older than 1 year will be contacted via e-mail and asked to clean up their data.
There is no backup for this directory. Users are responsible for backing up their own data using personal storage solutions such as external HDDs or Google Drive.

/scratch Directory

Designed for temporary data storage and job execution.
You must run your jobs on your personal /scratch folder, not under /home
After your job completes, all relevant output must be moved back to your personal directory under /home
Data left on /scratch will be deleted at any time without notice by Research Computing administrators.
/scratch is not intended for long-term or backup storage. Do not rely on it to retain any project data.

Account Suspension and Re-activation

If your account is suspended due to policy violations:
- Submit a re-activation request via IT Trackit.
- Inform your supervisor and add them as a participant in your Trackit request.

Joint Storage Areas (KUACC & DC2)

Datasets Folder (/datasets)

The /datasets folder hosts a collection of general-purpose datasets used across various research and machine learning projects. It serves as a centralized, read-only repository accessible from all compute nodes.

Access and Usage Guidelines

Read-Only Access:
The /datasets folder is mounted as read-only on all nodes. Users do not have write permissions and cannot modify its contents.
Avoid Downloading to Home Directory:
Do not download or copy datasets to your home directory, as this can lead to quota issues and negatively affect overall system performance.
Dataset Requests:
If you require a dataset that is not currently available, you can request it by:
- Sending an e-mail to it@ku.edu.tr or submitting a request via IT-Trackit. Please include the dataset name, download URLs, and set the subject line to "HPC Dataset Request".
Dataset Structure & README Files:
Most datasets are extracted from compressed archives (e.g., .zip) and kept in their original structure. If any modifications are made to the folder structure, the details will be documented in a README file within the dataset directory. It is highly recommended to read the README file before using the dataset.

Using Symbolic Links

If you need to rename or restructure dataset folders for convenience, you can create symbolic (soft) links in your home directory. For example, the following command creates a symbolic link to a dataset file:

CODE

cd ~
mkdir CID
ln -s /datasets/cid/CID_train_set/2019_01_01_01_20_04_277/JPEG_2019_01_01_01_20_08_673.jpg CID/image1.jpeg

This will create a link under ~/CID/named image1.jpeg, pointing to the original file in /datasets.

Userfiles Folder (/userfiles)

The /userfilesdirectory provides slow disk storage intended primarily for private datasets and inactive research data. It is not meant for high-performance computing tasks but serves as auxiliary storage to reduce clutter on shared scratch spaces.

Access and Usage Policy

Access Request Required:
A personal directory under /userfiles is created upon request via Trackit. The request must include advisor approval or notification.
Purpose of Use:
- Store private datasets not suitable for the general /datasets repository
- Archive inactive research data, meaning data not currently involved in active computation
- Helps keep the scratch space clean and efficient
Storage Limits and Retention:
- The maximum allowed usage is 500 GB per user
- Inactive data older than 6 months must be removed by the user
- No backup is provided; users are responsible for securing critical data
Group Storage Extensions:
Research groups may request additional storage by adding dedicated disks to the system. This also allows for group-specific directories (e.g., /userfiles/ai).

Frozen Folder (/frozen)

The /frozen directory on the KUACC and DC2 cluster is a long-term archival storage area intended for users with large, infrequently accessed datas that require extended retention periods.

Access and Request Process

Access to /frozen is not granted by default. Users must submit a storage request form. The request form can be obtained by sending an e-mail to it@ku.edu.tr or submitting a request via IT-Trackit.
Requests involving large volumes of data or long retention periods are reviewed by the Research Computing Committee. Upon approval, a dedicated space is provisioned.
Read/write permissions are assigned based on the nature of the request.

Usage and Responsibilities

This area is intended for archived, finalized datas only. It is not suitable for job execution or temporary storage.
There is no backup policy for /frozen. Users are responsible for maintaining their own data redundancy and backups as needed.
Data cleanup may be initiated by Research Computing administrators if required due to storage constraints or policy changes. Users will be notified accordingly if such action is planned.
A record of all /frozen allocations is maintained, and usage is monitored by Research Computing administrators.
Users are required to provide details about the data contents as part of the request process to ensure proper tracking.

Kuttam Folder (/kuttam_fg)

The /kuttam_fg directory is a dedicated storage area reserved exclusively for academic staff members of KUTTAM (Koç University Research Center for Translational Medicine).

Purpose and Access

This area is built on NetApp ONTAP infrastructure and is accessible only to authorized users through the HPC systems.
It is designed to support data storage, sharing, and analysis workflows for KUTTAM-affiliated researchers.
Access is granted upon request. Interested users must contact it@ku.edu.tr to initiate the access process.

Usage Guidelines

The /kuttam_fg directory is intended for use by KUTTAM researchers in the context of their scientific activities.
Users are expected to manage their data responsibly, ensuring compliance with general Research Computing policies.
Access and usage are monitored and maintained by Research Computing administrators.

Please use the storage systems responsibly and follow these guidelines to ensure a stable and efficient computing environment for all users.