Processing crystallography data

MacCHESS provides a full suite of software for processing crystallography data. We also include automated processing for getting fast results.

info graphics: software

ID7B2 users who are collecting data may run assorted crystallographic processing and post-processing software on Landline, the data collection workstation, during their allotted time.

During the run your collected data will be reachable on Landline at: /nfs/chess/daq/current/id7b2/<your group name>

After the run, it can be accessed in read-only form at: /nfs/chess/raw/current/id7b2

We recommend doing data processing in the following folder: /nfs/chess/aux/cycles/current/id7b2

We recommend organizing your directories by PI name and date, as we do for data collection.

After the data collection period, any additional processing should be moved to other computing resources to make way for the next user. We have two computing resources available.
 
1) CLASSE Computing Network
 
CLASSE has a general purpose Linux server available, which can be reached by SSH or X2Go remote virtual desktop using the same CLASSE ID and password required for remote data collection. From there, intensive computing tasks may be submitted as batch jobs to the CLASSE Compute Farm.
 
To make the MacCHESS software depository accessible to your CLASSE account, place the following in the .bashrc file of your home directory:
if [ -e /nfs/chess/sw/macchess/BASHRC ]; then
  source /nfs/chess/sw/macchess/BASHRC
fi
Source the file, or logout and log back in to execute the change.
 
2) Jawline: For HKL2000 Use

MacCHESS also has a dedicated Linux computing resource for ID7B2 users, jawline.classe.cornell.edu, which is reachable by SSH or by NoMachine or X2Go remote virtual desktop. The NoMachine console is reached through the CLASSE NoMachine cloud server nomachine.classe.cornell.edu.
Landline and jawline are both licensed for the HKL2000 package, so if HKL2000 is your choice, then a graphical connection to jawline using either X2Go or NoMachine is probably what you want for post-processing. 
 
For data collection on landline, you connect to the physical desktop which is run under the data collection account. For post-processing with NoMachine on jawline, you will instead have a virtual desktop running under your own CLASSE account.

Fully Automated Data Processing

fast_dp is a program which uses XDS, CCP4, and CCTBX to deliver data processing results very quickly. This will be especially useful for users who are looking for fast processing results to monitor how to use their beam time most effectively. However, it is also robust and intelligent enough to deliver publication-quality reduced data for most datasets.

fast_dp uses XDS for indexing and integration, then runs Pointless for space group determination. XDS is then rerun with the chosen space group, and the data scaled with Aimless.

ADX will run fast_dp on our dedicated processing server immediately after a run if Auto-process is turned on in the Run(s) window (see below). fast_dp output will be saved in the 'fast_dp' folder of the data directory. A typical run with fast_dp delivers reduced data in CCP4 .mtz format in three minutes or less.

adx_fast_dp

Running fast_dp manually

fast_dp can be run manually from a terminal window. To see options, type:

$ fast_dp --help

xia2: Automated Processing with DIALS

xia2 is another automated processing routine which uses DIALS, an open-source processing package based on cctbx. The best aspect of xia2 is XTriage, which can identify potential problems in the dataset such as twinning, reflection outliers, missed symmetry, or missed anomalous signal.

xia2 is especially recommended for problematic data sets that may fail with other methods.

xia2 can be accessed on CHESS computers by entering `xia2` on a terminal window, which will return a list of options and parameters. Make sure to set reverse_phi=True when running xia2 with data collected at ID7B2.

A typical xia2 run on data from ID7B2 look something like this:

$ xia2 pipeline=dials-aimless reverse_phi=True /path/to/images

XDS

XDS is the preferred data reduction package for crystallographic data collected at CHESS.

XDS.INP file

The best method for generating the XDS.INP file is to use the program generate_XDS.INP, which is available on the CHESS network. From a terminal window, type:

$ generate_XDS.INP /path/to/file_master.h5 # for EIGER HDF5 data
$ generate_XDS.INP /path/to/file.cbf # for PILATUS or other detector data

The program will automatically fill in the data collection parameters from the detector metadata, along with the best default settings.

XDS input file templates for MacCHESS detectors can be found here: XDS Input Files

Processing EIGER Data With XDS: Neggia Plugin

In order to process EIGER HDF5 data with XDS, a library that can read the data into XDS is required. The Neggia plugin for XDS reads EIGER HDF5 data efficiently and is recommended. The plugin is available for Linux and Mac. To use the library, add the following line to your XDS.INP file:

LIB=/path/to/dectris-neggia.so

generate_XDS.INP and fast_dp insert the correct LIB= path for Neggia for processing on the CHESS network automatically.

Download the neggia plugin for Linux and Mac here.

XDSGUI

XDSGUI is a nice GUI for processing XDS data. It is especially useful for looking at plots of data statistics. It is available on the terminal by typing

$ xdsgui

Additional information

For further information on XDS, go to the XDS Documentation. The XDS Wiki is an excellent resource for XDS, full of useful hints and advice for data processing by the authors of XDS.

HKL2000 and HKL (xdisp/denzo/scalepack)

HKL2000 and HKL (xdisp/denzo/scalepack), a graphical interface to Denzo, XdisplayF, Scalepack, and associated routines, is available, through a collaborative arrangement with HKL Research, on all MacCHESS computers. All detectors in use on MacCHESS beamlines are supported.

Experimenters from academic and non-commercial institutions may use the HKL programs for data evaluation and processing, while collecting data at the beamline. Users, holding a non-profit HKL license, can obtain, from HKL Research, Inc., the program upgrade to process data collected at CHESS.

Users who do not have a permanent HKL license can obtain, free of charge, a temporary license from HKL Research, Inc. to (re)process data collected at CHESS.

Users from commercial and for-profit institutions must have an appropriate commercial HKL license to use the program for data evaluation and processing, both when collecting data at the beamline and when processing data at home. The license can be obtained from HKL Research, Inc.

HKL2000 Site Definition Files 

To launch HKL2000 from a terminal window:

$ hkl2000

Manual 

A few copies of the HKL Manual are on hand at CHESS. An on-line version is also available.

CCP4/Mosflm Data Processing Suite

CCP4

CCP4 suite documentation

For interface to CCP4 programs & file display, from a terminal window:

$ ccp4i    # for the 'classic' GUI
$ ccp4i2   # for the 'new' GUI

iMosflm

iMosflm is the graphical user interface for the data reduction program Mosflm, and a part of the CCP4 suite of programs.

To launch iMosflm independently of CCP4 from a terminal window:

$ imosflm

Phenix: Phasing and Refinement

phenix is a crystallographic program suite based on Python. It has powerful tools for both crystallographic and cryo-EM data, with an especially nice GUI.

For connecting to the phenix GUI, from a terminal window:

$ phenix

Laue processing software

Download updated Daresbury suite

For help in processing Laue data, contact Marian Szebenyi

Coot

coot is a popular model building and refinement tool. From a terminal window:

$ coot

Pymol

pymol

Fancy graphics: movies, rendering, and so much more..., type:

$ pymol