IT Help

How do I copy files/data to the HPC Machines?

You do not need to be logged in to your HPC Cluster account while using these commands.


If you are using zsh terminal on your Mac, you can have this error:

/etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory

Despite this error the commands work. But if you want to solve the problem, you can follow the steps in here.

If you are on campus:

1) Windows

Use MobaXterm. This is a GUI-based scp client for MS Windows-based computers that has a drag-and-drop facility and an inbuilt file editor. If you have cygwin installed, you can open a cygwin-terminal and then use ssh the same as Linux and Mac below.
-> Download MobaXterm

2) Linux and Mac

scp on terminal (KUACC):

scp file-to-name USERNAME@login.kuacc.ku.edu.tr:/HOME_DIR/SUB_FOLDER/new-filename

scp on terminal (VALAR):

scp file-to-name USERNAME@login.valar.ku.edu.tr:/HOME_DIR/SUB_FOLDER/new-filename

This will copy the file to a SUB_FOLDER and rename it to new-filename. The destination folder must be a valid address. So SUB_FOLDER must be an existing folder.


As an example, let's copy a file from the Desktop to the home folder of the HPC Cluster:

In my cluster account, I have Morse.jl, logs, and workfolder folders.

Screen-Shot-2021-11-18-at-13.09.01-1024x43.png


Then, on my local computer’s terminal I am writing scp command and copying file named main.py to my home folder on cluster renamed as geohash.py. By the way since I am a Mac user, I see the error, which was mentioned before.

Screen-Shot-2021-11-18-at-13.10.00-1024x68.png


Then, I login to the HPC Cluster again and check whether file exist or not.

Screen-Shot-2021-11-18-at-13.10.22-1024x40.png

rsync on terminal:

rsync is a fast and versatile command-line utility for synchronizing files and directories between two locations over a remote shell, or from/to a remote Rsync daemon.

It provides fast incremental file transfer by transferring only the differences between the source and the destination.

https://www.geeksforgeeks.org/rsync-command-in-linux-with-examples/

Local to Local:

rsync [OPTION]... [SRC]... DEST


Local to Remote:

rsync [OPTION]... [SRC]... [USER@]HOST:DEST


Remote to Local:

rsync [OPTION]... [USER@]HOST:SRC... [DEST]


Zip the transferred file from remote to local:

rsync [OPTION]... [USER@]HOST:SRC... [DEST]


Transfer all files from remote to local:

rsync -a USERNAME@host:relative path of the file/folder destination


Now, lets copy Morse.jl folder which is on cluster to the Destop under the folder which is named as rsync-test.

In my cluster account, this time, I have Morse.jl, logs, work folder and goehash.py

Transfer all files from remote to local.png


Then I use the rsync command and since I want all files I use -a option. I give the path of Morse.jl under cluster folder and destination as rsync-test. Since I do not have rsync-test folder, it will create automatically.

Transfer all files from remote to local-2.png


Then I use the rsync command and since I want all files I use -a option. I give the path of Morse.jl under cluster folder and destination as rsync-test. Since I do not have rsync-test folder, it will create automatically.

3.png


Recommended sftp or ftp clients:

Windows: MobaXterm, WinSCP, FileZilla
Linux: sftp, FileZilla, Rclone
Mac: Cyberduck, FileZilla

 
You should use following parameters for KUACC.

Host: login.kuacc.ku.edu.tr
Username: KU-username
Password: KU-password
Port: 22


You should use following parameters for VALAR.

Host: login.valar.ku.edu.tr
Username: KU-username
Password: KU-password
Port: 22


If you are off campus:

1) Google Drive/Dropbox:

If your data is big and you are using VPN, previous solutions can be slow to copy files. You can use Cyberduck (a graphical file transfer client) or rclone (a command-line tool) to transfer files between your computer, cloud storage (such as Google Drive, Dropbox, OneDrive), and the HPC clusters.


Cyberduck:Provides an easy-to-use drag-and-drop interface over SFTP and supports direct connections to cloud storage services. You can download it from the Cyberduck Official Websiteand check the Cyberduck Documentationfor SFTP and cloud connection setups.


rclone:Ideal for large datasets, automated transfers, and syncing directly from the cluster terminal without downloading files to your local machine first. Please refer to the rclone Documentationfor installation and configuration details (such as rclone config).