Transfer of files¶

You can transfer files to/from Arrhenius, you can use any SSH or SFTP-based tool (more file transfer mechanisms might be implemented later):

Graphical drag-and-drop tool

• Transfer files to/from Arrhenius using FileZilla

Command-line tools

• Transfer files to/from Arrhenius using rsync
  • In many cases the best option
• Transfer files to/from Arrhenius using scp
  • Simple tool for just transferring a file or directory
• Transfer files to/from Arrhenius using sftp
  • Like FileZille, a two-way portal, but on the command-line (no drag-and-drop).

Introduction¶

Let's start with some priciples and definitions!

• The principle is that the computer you start FileZilla or run a file transfer command on is the local host.
• Remote is the server you connect to. The remote requires hostaname, like an IP address or URL, like *.arrhenius.naiss.se

Since you probably do not have a server address to your computer you need to run the commands from there. An exception is when you transfer files between servers, for instance clusters. Then it does not matter where you run the command or start FileZilla from (if it is installed).

The Arrhenius hostname to use login.hpc.arrhenius.naiss.se.

rsync basics¶

When transferring large amounts of data, we recommend that you use a tool (e.g rsync) that can resume an interrupted transfer

If you have copied data to or from e.g Tetralith, this should already be familiar to you. If not, there are many rsync guides available online for beginners and pros alike, e.g 1, 2, 3, 4, 5 or 6.

If you want to make rsync do something other than the "preserve mostly everything" that the -a option does, read the man page (man rsync on the cluster).

Basic example 1: push data from Tetralith to Arrhenius

[x_examp@tetralith1 ~]$ rsync -av /proj/somedirectory/somesubdirectory/ myarrheniususer@login.hpc.arrhenius.naiss.se:/nobackup/proj/disk/somedirectory/somewhere/
(myarrheniususer@login.hpc.arrhenius.naiss.se) Password: <ENTER YOUR ARRHENIUS PASSWORD HERE>
(myarrheniususer@login.hpc.arrhenius.naiss.se) Verification code: <ENTER YOUR ARRHENIUS 2FA CODE HERE>
sending incremental file list
created directory /nobackup/proj/disk/somedirectory/somewhere
[...]
[x_examp@tetralith1 ~]$

Basic example 2: pull data from Tetralith to Arrhenius, preserve hard links and handle sparse files efficiently:

[myarrheniususer@arrhenius11 ~]$ rsync -av -H --sparse x_examp@tetralith:/proj/somedirectory/somesubdirectory/ /nobackup/proj/disk/somedirectory/somewhere/
(x_examp@tetralith.nsc.liu.se) Password: <ENTER YOUR TETRALITH PASSWORD HERE>
(x_examp@tetralith.nsc.liu.se) Verification code: <ENTER YOUR TETRALITH 2FA CODE HERE>
sending incremental file list
created directory /nobackup/proj/disk/somedirectory/somewhere/
[...]
[myarrheniususer@arrhenius11 ~]$

scp basics¶

This is how you copy a file from your local computer directly to your HOME folder (~/):

scp my_file.txt nisse@arrhenius1.hpc.arrhenius.naiss.se:~/

To copy a file from Arrhenius to your local computer (and present folder), do the command above in reverse order:

scp nisse@login.hpc.arrhenius.naiss.se:~/remote_file.txt .

sftp basics¶

SFTP is an abbreviation of 'SSH File Transfer Protocol' and 'SSH' is an abbreviation of 'Secure Shell protocol'.

• The difference from rsync and scp is that you login to a SFTP session
  • just like a log in to a server with SSH.
• You use the terminal to transfer the files.
• You need the credentials only when establishing the connection.

In sftp session, go to right folders and upload/download files to/from the server. You can do this as long as the connection is active!

• pwd/cd/ls - browse and list files in server file system
• lpwd/lcd/lls - browse and list files in local file system
• put <file/folder> - Copy a file/folder from the local computer to the remote host
• get <file/folder> - Copy a file/folder from the remote host to the local computer

Connect to SFTP session

In the terminal, run sftp to connect to Arrhenius by doing:

sftp nisse@login.hpc.arrhenius.naiss.se

If asked, give your center's password. You can get rid of this prompt if you have setup SSH keys

Upload/download files to/from Arrhenius

In sftp session, upload/download files/folders to/from the server.

- ``put localfile.txt`` - Copy a file from the local computer to the remote host
- ``get serverfile.txt`` - Copy a file from the remote host to the local computer

Moving data within Arrhenius¶

When moving data within Arrhenius, e.g between two project storage directories, you can use any normal Linux commands, but you should be aware that e.g mv is not instantaneous since the project quota counters needs to be updated which makes the process a little slower than you might expect.

Large or many files¶

Compress¶

• Shorten download/upload time by reducing the size of a file!
  • A common tool in Linux environments is gzip.
  • Usage: gzip <filename>. You'll get a gzfile ending
  • Decompress: gunzip <filename>
  • gzip manual

Archive many files¶

• Transferring many files will create so called overhead
  • each file has to be addressed individually.
• Solution is to gather the files in an archive, like tar.
  • The content then behaves like ONE file.
  • Usage: tar -cf archive.tar /path/files or tar -cf archive.tar /path/folder

• While TARing you may compress the data as well!

  • tar -czf archive.tar.gz [/path/files]

• tar manual