Data backup and transport

Beamtime for macromolecular crystallography at MacCHESS is likely to generate lots of data which needs to be transported back to the home lab. This is usually done by backing the data up to a portable drive or by transferring it over the Internet. The preferred software for this is RSYNC.

SOFTWARE

We recommend RSYNC for data transfers. It works with both portable drives and network transfers. It has many features which make it useful for beamlne backup. Perhaps the most important is the ability to do incremental transfers. Before actually copying any files, it will compare the source with the target and only copy files which are new or altered.

Here are some examples of rsync commands:

To a portable drive:

FAT file system:

rsync -rltv --modify-window=1 /A1a/smith /media/disk1/chess_20170812/.

To other file systems:

rsync -av /A1a/smith /media/disk1/chess_20170812/.

PORTABLE DRIVES

USB 2.0+ drives are most common, although MAcCHESS still supports FireWire. SSD (solid state drives) are typically faster and more shock-resistant than hard disk drives. Make sure you have sufficient capacity for the data you plan to collect. Designate a drive for data transfer, and do not use that same drive for permanent storage.

NETWORK TRANSFERS

MacCHESS data is located inside the CHESS firewall, so it is expected that all data transfer connections will be outgoing connections made from the inside. If you plan to do this you should check into your home IP address and firewall status. If you need to make a connection from your home lab to CHESS, look into getting CLASSE credentials so that you can connect using VPN. If you need access to a small number of files, MacCHESS maintains a file server outside the CHESS firewall.

OTHER

 Additional possibilities for handling MacCHESS data include burning DVDs. A single-sided single-layer DVD holds only 4.7 GB of data.

You might bring your own computer. Plugging in to a MacCHESS network switch will provide up to a 1 Gbps connection. Wi-fi connections do not work as well; CHESS outsources wi-fi support to the campus IT service, so that wi-fi connections originate outside the CHESS firewall.