Python Interface to NeXus

The Python interface to NeXus is provided by the nexusformat package, which is distributed separately from NeXpy.

The Python API can be used within a standard Python or IPython shell:

$ python
Python 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:41:54) [Clang 13.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from nexusformat.nexus import *

Note

Although wildcard imports are usually discouraged in Python, all the imported functions and variables start with ‘nx’ or ‘NX’, so the risk of namespace conflicts should be small.

See also

A Jupyter notebook provides a tutorial for the Python API. It can be run in Google Colaboratory.

Loading NeXus Data

The entire tree structure of a NeXus file can be loaded by a single command:

>>> a=nxload('sns/data/ARCS_7326_tof.nxs')

The assigned variable now contains the entire tree structure of the file, which can be displayed by printing the ‘tree’ property:

>>> print(a.tree)
root:NXroot
 @HDF5_Version = '1.8.2'
 @NeXus_version = '4.2.1'
 @file_name = 'ARCS_7326_tof.nxs'
 @file_time = '2010-05-05T01:59:25-05:00'
 entry:NXentry
   data:NXdata
     @axes = ['rotation_angle' 'tilt_angle' 'sample_angle' 'time_of_flight']
     @signal = 'data'
     data = float32(631x461x4x825)
     rotation_angle = float32(632)
       @units = 'degree'
     sample_angle = float32(5)
       @units = 'degree'
     tilt_angle = float32(462)
       @units = 'degree'
     time_of_flight = float32(826)
       @units = 'microsecond'
   run_number = '7326'
   sample:NXsample
     pulse_time = 2854.947473649946
       @units = 'microsecond'

Individual data items are immediately accessible from the command-line:

>>> print(a.entry.run_number)
7326

Only the tree structure and the values of smaller data sets are read from the file to avoid using up memory unnecessarily. In the above example, only the types and dimensions of the larger data sets are displayed in the tree. Data is loaded only when it is needed, for plotting or calculations, either as a complete array, if memory allows, or as a series of slabs (see below).

Note

The maximum size of data that will be read from a file into memory can be configured using nxsetconfig. Details of other configuration variables are described later.

There is a second optional argument to the load module that defines the access mode for the existing data. For example, the following opens the file in read/write mode:

>>> a=nxload('chopper.nxs', mode='rw')

The default mode is ‘r’, i.e., readonly access. The nxload function will accept any mode values allowed when opening h5py files, such as ‘r+’, ‘w’, ‘w-‘, and ‘a’ (see the h5py documentation for more details), but once open, the mode values are stored as ‘r’ or ‘rw’.

Warning

If the file is opened in read/write mode, any changes are made automatically to the file itself. In particular, any deletions of file objects will be irreversible. If necessary, a backup of the file can be made using the backup function.

See also

nexusformat.nexus.tree.NXroot.backup

Creating NeXus Data

It is just as easy to create new NeXus data sets from scratch using NumPy arrays. The following example shows the creation of a simple function, which is then saved to a file:

>>> import numpy as np
>>> x=y=np.linspace(0,2*np.pi,101)
>>> X,Y=np.meshgrid(x,y)
>>> z=np.sin(X)*np.sin(Y)
>>> a=NXdata(z,[y,x])
>>> a.save('function.nxs')

This file can then be loaded again:

>>> b=nxload('function.nxs')
>>> print(b.tree)
root:NXroot
  @HDF5_Version = '1.12.2'
 @file_name = '/home/username/function.nxs'
 @file_time = '2023-02-10T15:50:17.419158'
 @h5py_version = '3.7.0'
 @nexusformat_version = '1.0.0'
 entry:NXentry
   data:NXdata
     @axes = ['axis1', 'axis2']
     @signal = 'signal'
     axis1 = float64(101)
     axis2 = float64(101)
     signal = float64(101x101)

Note

The save() method automatically wraps any valid NeXus data in an NXentry group, in order to produce a standard-compliant file. See Saving NeXus Data for more details.

NeXus Objects

NeXus data are stored in a hierarchical tree structure, much like a computer file system. NeXus data structures consist of groups, with base class NXgroup, which can contain fields, with base class NXfield, and/or other groups.

NeXus Fields

NeXus data values are stored in NeXus objects of class ‘NXfield’. The NXfield class wraps standard NumPy arrays, scalars, and Python strings so that additional metadata (or attributes) and methods can be associated with them.

There are three ways to create an NXfield.

  1. Direct assignment:

    >>> x = NXfield(np.linspace(0,2*np.pi,101), units='degree')

The data value is given by the first positional argument, and may be a Python scalar or string, or a NumPy array. In this method, keyword arguments can be used to define NXfield attributes.

  1. Dictionary assignment to the NeXus group:

    >>> a['entry/sample/temperature']=40.0

  2. Attribute assignment as the child of a NeXus group:

    >>> a.entry.sample.temperature=40.0

The assigned values are automatically converted to an NXfield:

>>> a.entry.sample.temperature
NXfield(40.0)

Dictionary and attribute assignments are equivalent, but dictionary assignments should always be used if there is a danger of a name clash with, for example, group or field methods. They are also recommended when writing scripts. Attribute assignments are allowed because they are much faster to type in interactive sessions.

Note

When using the NeXpy GUI shell (see Python Graphical User Interface), it is possible to use tab completion to check for possible name clashes with NXfield methods. Autocompletion can be added as an extension to IPython sessions as well:

>>> from nexusformat.nexus.completer import load_ipython_extension
>>> load_ipython_extension(get_ipython())

The data in an NXfield can be of type integer, float, or character. The type is normally inherited automatically from the data type of the Python object, although it is possible to define alternative (but compatible) datatypes. For example, a float64 array can be converted to float32 on assignment:

>>> x=np.linspace(0, 2*np.pi,101)
>>> x.dtype
dtype('float64')
>>> a=NXfield(x, dtype='float32')
>>> a.dtype
dtype('float32')
>>> b=NXfield('Some Text')
>>> b.dtype, b.shape
(dtype('O'), ())

Note

Numeric dtypes can be defined either as a string, e.g., ‘int16’, ‘float32’, or using the NumPy dtypes, e.g., np.int16, np.float32.

Warning

By default, Python strings are stored as variable-length strings in the HDF5 file. These use a special object dtype defined by h5py (see the h5py documentation). If you wish to store fixed length strings, specify a dtype of kind ‘S’, e.g., ‘S10’ when creating the NXfield.

Warning

If you wish to store an array of strings containing Unicode characters as fixed-length strings, convert them to byte strings first using UTF-8 encoding, e.g.:

>>> text_array = ['a', 'b', 'c', 'd', 'é']
>>> a=NXfield([t.encode('utf8') for t in text_array], dtype='S')
>>> a
NXfield(['a', 'b', 'c', 'd', 'é'])
>>> a.dtype
dtype('S2')

The shape and dimension sizes of an integer or float array are inherited from the assigned NumPy array. It is possible to initialize an NXfield array without specifying the data values in advance, e.g., if the data is too large to be stored in memory and has to be written as slabs:

>>> a=NXfield(dtype=np.float32, shape=[2048,2048,2048])
>>> a
NXfield(shape=(2048, 2048, 2048), dtype=float32)

More details of handling large arrays are given below.

NeXus attributes

The NeXus standard allows additional attributes to be attached to NXfields to contain metadata

>>> a['entry/sample/temperature'].units='K'

These have a class of NXattr. They can be defined using the ‘attrs’ dictionary if necessary to avoid name clashes:

>>> a['entry/sample/temperature'].attrs['units']='K'

Other common attributes include the ‘signal’ and ‘axes’ attributes used to define the plottable signal and independent axes, respectively, in a NXdata group.

When a NeXus tree is printed, the attributes are prefixed by ‘@’:

>>> print(a.entry.sample.tree)
sample:NXsample
  temperature = 40.0
    @units = 'K'

Masked Arrays

NumPy has the ability to store arrays with masks to remove missing or invalid data from computations of, e.g., averages or maxima. Since Matplotlib is able to handle masked arrays and removes masked data from plots, this is a convenient way of preventing bad data from contaminating statistical analyses, while preserving all the data values, good and bad, i.e., masks can be turned on and off.

NeXpy uses the same syntax as NumPy for masking and unmasking data.

>>> z = NXfield([1,2,3,4,5,6], name='z')
>>> z[3:5] = np.ma.masked
>>> z
NXfield(masked_array(data=[1, 2, 3, --, --, 6],
             mask=[False, False, False,  True,  True, False],
       fill_value=999999))
>>> z.mask
array([False, False, False,  True,  True, False])
>>> z.mask[3] = np.ma.nomask
>>> z
NXfield(masked_array(data=[1, 2, 3, 4, --, 6],
             mask=[False, False, False, False,  True, False],
       fill_value=999999))

Warning

If you perform any operations on a masked array, those operations are not performed on the masked values. It is not advisable to remove a mask if you have modified the unmasked values.

If the NXfield does not have a parent group, the mask is stored within the field as in NumPy arrays. However, if the NXfield has a parent group, the mask is stored in a separate NXfield that is generated automatically by the mask assignment or whenever the masked NXfield is assigned to a group. The mask is identified by the ‘mask’ attribute of the masked NXfield.

>>> print(NXlog(z).tree)
log:NXlog
z = [1 2 3 4 -- 6]
 @mask = 'z_mask'
z_mask = [False False False False  True False]

The mask can then be saved to the NeXus file if required.

Warning

In principle, the NXfield containing the mask can be modified manually, but it is recommended that modifications to the mask use the methods described above.

Masks can also be set using the Projection panel in the Python Graphical User Interface.

Large Arrays

If the size of an array is too large to be loaded into memory (see Loading NeXus Data), the NXfield can be created without any initial values, and then filled incrementally as slabs:

>>> entry['data/z'] = NXfield(shape=(1000,1000,1000), dtype=np.float32)
>>> for i in range(1000):
        entry.data.z[i,:,:] = np.ones(shape=(1000,1000), dtype=np.float32)
            ...

If entry in the above example is already stored in a NeXus file (with write access), then entry['data/z'] is automatically updated in the file. If it is not stored in a file, the field is stored in an HDF5 core memory file that will be copied to the NeXus file when it is saved.

When initializing the NXfield, it is possible to specify a number of HDF5 attributes that specify how the data are stored.

  • Compression:

    >>> z = NXfield(shape=(1000,1000,1000), dtype=np.float32, compression='lzf')

    This specifies the compression filter used. For large arrays, the data are compressed with the gzip filter by default.

  • Chunk size:

    >>> z = NXfield(shape=(1000,1000,1000), dtype=np.float32, chunks=(1,100,100))

    If chunk sizes are not specified, e.g., with chunks=True, HDF5 will choose default sizes.

  • Maximum array shape:

    >>> z = NXfield(shape=(10,1000,1000), dtype=np.float32, maxshape=(1000,1000,1000))

    The initial shape is defined by the shape attribute, but it will be automatically expanded up to a limit of maxshape if necessary using the NXfield resize function.

    >>> z.resize((100,1000,1000))
>>> z.shape
(100, 1000, 1000)

See also

nexusformat.nexus.tree.NXfield.resize

  • Fill value:

    >>> z = NXfield(shape=(1000,1000,1000), dtype=np.float32, fillvalue=np.nan)

    Slabs that are not initialized will contain the specified fill value. This is normally set to zero by default.

All these values can be adjusted at the command line until the first slab has been written, whether to a file or in core memory, using the compression, chunks, maxshape or fillvalue properties, e.g.

>>> z = NXfield(shape=(1000,1000,1000), dtype=np.float32)
>>> z.compression = 'lzf'

NeXus Groups

NeXus groups are defined as subclasses of the NXgroup class, with the class name defining the type of information they contain, e.g., the NXsample class contains metadata that define the measured sample, such as its temperature or lattice parameters. The initialization parameters can be used to populate the group with other predefined NeXus objects, either groups or fields:

>>> temperature = NXfield(40.0, units='K')
>>> sample = NXsample(temperature=temperature)
>>> print(sample.tree)
sample:NXsample
  temperature = 40.0
    @units = 'K'

In this example, it was necessary to use the keyword form to add the NXfield ‘temperature’ since its name is otherwise undefined within the NXsample group. However, the name is set automatically if the NXfield is assigned to the group:

>>> sample = NXsample()
>>> sample['temperature']=NXfield(40.0, units='K')
>>> print(sample.tree)
sample:NXsample
  temperature = 40.0
    @units = 'K'

The NeXus objects in a group (NXfields or NXgroups) can be accessed as dictionary items:

>>> sample['temperature'] = 40.0
>>> sample.keys()
dict_keys(['temperature'])

Note

It is also possible to reference objects by their complete paths with respect to the root object, e.g., root[‘/entry/sample/temperature’].

If a group is not created as another group attribute, its internal name defaults to the class name without the ‘NX’ prefix. This can be useful in automatically creating nested groups:

>>> a=NXentry(NXsample(temperature=40.0),NXinstrument(NXdetector(distance=10.8)))
>>> print(a.tree)
entry:NXentry
  instrument:NXinstrument
    detector:NXdetector
      distance = 10.8
  sample:NXsample
    temperature = 40.0

See also

Existing NeXus objects can also be inserted directly into groups. See nexusformat.nexus.tree.NXgroup.insert

NXdata Groups

NXdata groups contain data ready to be plotted. That means that the group should consist of an NXfield containing the signal and one or more NXfields containing the axes. NeXus defines a method of associating axes with the appropriate dimension, but NeXpy provides a simple constructor that implements this method automatically. This was already demonstrated in the example above, reproduced here:

>>> import numpy as np
>>> x=y=np.linspace(0,2*np.pi,101)
>>> X,Y=np.meshgrid(x,y)
>>> z=np.sin(X)*np.sin(Y)
>>> a=NXdata(z,[y,x])

The first positional argument is an NXfield or NumPy array containing the data, while the second is a list containing the axes, again as NXfields or NumPy arrays. In this example, the names of the arrays have not been defined within an NXfield so default names were assigned:

>>> print(a.tree)
data:NXdata
  @axes = ['axis1' 'axis2']
  @signal = signal
  axis1 = float64(101)
  axis2 = float64(101)
  signal = float64(101x101)

Note

The plottable signal and axes are identified by the ‘signal’ and ‘axes’ attributes of the NXdata group. The ‘axes’ attribute defines the axes as a list of NXfield names. The NXdata constructor sets these attributes automatically.

Warning

NumPy stores arrays by default in C, or row-major, order, i.e., in the array ‘signal(axis1,axis2)’, axis2 is the fastest to vary. In most image formats, e.g., TIFF files, the x-axis is assumed to be the fastest varying axis, so we are adopting the same convention and plotting as signal[y,x]. The Python Graphical User Interface allows the x and y axes to be swapped.

Names can be assigned explicitly when creating the NXfield through the ‘name’ attribute:

>>> phi=NXfield(np.linspace(0,2*np.pi,101), name='polar_angle')
>>> data=NXfield(np.sin(phi), name='intensity')
>>> a=NXdata(data,(phi,))
>>> print(a.tree)
data:NXdata
  @axes = 'polar_angle'
  @signal = 'intensity'
  intensity = float64(101)
  polar_angle = float64(101)

Note

In the above example, the x-axis, phi, was defined as a tuple in the second positional argument of the NXdata call. It could also have been defined as a list. However, in the case of one-dimensional signals, it would also have been acceptable just to call NXdata(data, phi), i.e., without embedding the axis in a tuple or list.

It is also possible to define the plottable signal and axes using the nxsignal and nxaxes properties, respectively:

>>> phi=np.linspace(0,2*np.pi, 101)
>>> a=NXdata()
>>> a.nxsignal=NXfield(np.sin(phi), name='intensity')
>>> a.nxaxes=NXfield(phi, name='polar_angle')
>>> print(a.tree)
data:NXdata
  @axes = 'polar_angle'
  @signal = 'intensity'
  intensity = float64(101)
  polar_angle = float64(101)

Similarly, signal errors can be added using the nxerrors property:

>>> a.nxerrors = np.sqrt(np.abs(np.sin(phi)))
>>> print(a.tree)
data:NXdata
  @axes = 'polar_angle'
  @signal = 'intensity'
  intensity = float64(101)
  intensity_errors = float64(101)
  polar_angle = float64(101)

Note

In a NXdata group, errors for each field are defined by another field with ‘_errors’ appended to the name.

Plotting NeXus Data

NXdata, NXmonitor, and NXlog groups all have a plot method, which automatically determines what should be plotted:

>>> data.plot()
_images/simple-plot.png

Note that the plot method uses the NeXus attributes within the groups to determine automatically which NXfield is the signal, what its rank and dimensions are, and which NXfields define the plottable axes. The same command will work for one-dimensional or two-dimensional data. If you plot higher-dimensional data, the top two-dimensional slice is plotted. Alternative two-dimensional slices can be specified using slice indices on the NXdata group.

Note

If the interpretation attribute is set to ‘rgb’ or ‘rgba’ and the final dimension is of size 3 or 4, the NXdata group will be plotted as an image using the colors defined by the final dimension. By default, images are displayed with the origin in the top-left corner.

If the data is one-dimensional, it is possible to overplot more than one data set using ‘over=True’. By default, each plot has a new color, but conventional Matplotlib keywords can be used to change markers and colors:

>>> data.plot(log=True)
>>> data.plot('r-')
>>> data.plot(over=True, log=True, color='r')

If the NXdata group contains RGB(A) image data, i.e., the signal is a three-dimensional array, in which the fastest varying dimension, which should be of size 3 or 4, contains the RGB(A) values for each two-dimensional pixel, then the image can be plotted using the ‘image=True’.

>>> data.plot(image=True)

By convention, the first pixel of an image is in the upper-left corner, rather than the lower-left used in other two-dimensional plots.

Note

The plot method also works on NXroot and NXentry groups, if they are able to identify plottable data. If the default attribute is set, the default NXentry and/or NXdata groups are used. Otherwise, the first valid NXdata group found in an iterative search is used.

Additional Plot Methods

As a convenience, additional plot methods can be used instead of adding extra keywords.

>>> data.oplot()
>>> data.logplot()
>>> data.implot()

These are equivalent to setting the ‘over’, ‘log’, and ‘image’ keywords to True when invoking the plot method.

Manipulating NeXus Data

Arithmetic Operations

NXfield

NXfields usually consist of arrays of numeric data with associated metadata, the NeXus attributes (the exception is when they contain character strings). This makes them similar to NumPy arrays, and this module allows the use of NXfields in numerical operations as if they were NumPy ndarrays:

>>> x = NXfield((1.0,2.0,3.0,4.0))
>>> print(x+1)
[ 2.  3.  4.  5.]
>>> print(2*x)
[ 2.  4.  6.  8.]
>>> print(x/2)
[ 0.5  1.   1.5  2. ]
>>> print(x**2)
[  1.   4.   9.  16.]
>>> x.reshape((2,2))
NXfield([[ 1.  2.]
[ 3.  4.]])
>>> y = NXfield((0.5,1.5,2.5,3.5))
>>> x+y
NXfield(name=x,value=[ 1.5  3.5  5.5  7.5])
>>> x*y
NXfield(name=x,value=[  0.5   3.    7.5  14. ])
>>> (x+y).shape
(4,)
>>> (x+y).dtype
dtype('float64')

Such operations return valid NXfield objects containing the same attributes as the first NXobject in the expression. The ‘reshape’ and ‘transpose’ methods also return NXfield objects.

NXfields can be compared to other NXfields (this is a comparison of their NumPy arrays):

>>> y=NXfield(np.array((1.5,2.5,3.5)),name='y')
>>> x == y
True

NXfields are technically not a sub-class of the NumPy ndarray class, but they are cast as NumPy arrays when required by NumPy operations, returning either another NXfield or, in some cases, an array that can easily be converted to an NXfield:

>>> x = NXfield((1.0,2.0,3.0,4.0))
>>> x.size
4
>>> x.sum()
10.0
>>> x.max()
4.0
>>> x.mean()
2.5
>>> x.var()
1.25
>>> x.reshape((2,2)).sum(1)
array([ 3.,  7.])
>>> np.sin(x)
array([ 0.84147098,  0.90929743,  0.14112001, -0.7568025 ])
>>> np.sqrt(x)
array([ 1.        ,  1.41421356,  1.73205081,  2.        ])
>>> print(NXdata(np.sin(x), (x)).tree)
data:NXdata
  @axes = 'x'
  @signal = 'signal'
  signal = [ 0.84147098  0.90929743  0.14112001 -0.7568025 ]
  x = [ 1.  2.  3.  4.]

Note

If a function will only accept a NumPy array, use the nxvalue attribute, which returns the stored NumPy array.

>>> x.nxvalue
array([1., 2., 3., 4.])

NXdata

Similar operations can also be performed on whole NXdata groups. If two NXdata groups are to be added, the rank and dimensions of the main signal array must match (although the names could be different):

>>> y=NXfield(np.sin(x),name='y')
>>> y
NXfield(name=y,value=[ 0.99749499  0.59847214 -0.35078323])
>>> a=NXdata(y,x)
>>> print(a.tree)
data:NXdata
  @axes = 'x'
  @signal = 'y'
  x = [ 1.5  2.5  3.5]
  y = [ 0.99749499  0.59847214 -0.35078323]
>>> print((a+1).tree)
data:NXdata
  @axes = 'x'
  @signal = 'y'
  x = [ 1.5  2.5  3.5]
  y = [ 1.99749499  1.59847214  0.64921677]
>>> print((2*a).tree)
data:NXdata
  @axes = 'x'
  @signal = 'y'
  x = [ 1.5  2.5  3.5]
  y = [ 1.99498997  1.19694429 -0.70156646]
>>> print((a+a).tree)
data:NXdata
  @axes = 'x'
  @signal = 'y'
  x = [ 1.5  2.5  3.5]
  y = [ 1.99498997  1.19694429 -0.70156646]
>>> print((a-a).tree)
data:NXdata
  @axes = 'x'
  @signal = 'y'
  x = [ 1.5  2.5  3.5]
  y = [ 0.  0.  0.]
>>> print((a/2).tree)
data:NXdata
  @axes = 'x'
  @signal = 'y'
  x = [ 1.5  2.5  3.5]
  y = [ 0.49874749  0.29923607 -0.17539161]

If data errors are included in the NXdata group, then the errors are propagated according to the operand:

>>> print(a.tree)
data:NXdata
  @axes = 'x'
  @signal = 'y'
  x = [ 1.5  2.5  3.5]
  y = [ 0.99749499  0.59847214  0.35078323]
  y_errors = [ 0.99874671  0.77360981  0.59226956]
>>> print((a+a).tree)
data:NXdata
  @axes = 'x'
  @signal = 'y'
  x = [ 1.5  2.5  3.5]
  y = [ 1.99498997  1.19694429  0.70156646]
  y_errors = [ 1.41244114  1.09404949  0.83759564]

Some statistical operations can be performed on the NXdata group.

  • NXdata.sum(axis=None):

    Returns the sum of the NXdata signal data. If the axis is not specifed, the total is returned. Otherwise, it is summed along the specified axis. The result is a new NXdata group containing a copy of all the metadata contained in the original NXdata group:

    >>> x=np.linspace(0, 3., 4)
>>> y=np.linspace(0, 2., 3)
>>> X,Y=np.meshgrid(x,y)
>>> a=NXdata(X*Y,(y,x))
>>> print(a.tree)
data:NXdata
  @axes = ['axis1' 'axis2']
  @signal = 'signal'
  axis1 = [ 0.  1.  2.  3.]
  axis2 = [ 0.  1.  2.]
  signal = float64(3x4)
>>> a.nxsignal
NXfield([[ 0.  0.  0.  0.]
 [ 0.  1.  2.  3.]
 [ 0.  2.  4.  6.]])
>>> a.sum()
18.0
>>> a.sum(0).nxsignal
NXfield([ 0.  3.  6.  9.])
>>> a.sum(1).nxsignal
NXfield([  0.   6.  12.])
  • NXdata.average(axis=None):

    Returns the average of the NXdata signal data. This is identical to the sum method, but the result is divided by the number of data elements in the summation:

    >>> a.average()
1.5
>>> a.average(0).nxsignal
NXfield([ 0.,  1.,  2.,  3.])
>>> a.average(1).nxsignal
NXfield([ 0. ,  1.5,  3. ])
  • NXdata.moment(order=1):

    Returns an NXfield containing the first moment of the NXdata group assuming the signal is one-dimensional:

    >>> x=np.linspace(0, 10., 11)
>>> y=np.exp(-(x-3)**2)
>>> a=NXdata(y,x)
>>> a.moment()
3.0000002539776141

Slicing

NXfield

A slice of an NXfield can be obtained using the usual Python indexing syntax:

>>> x=NXfield(np.linspace(0,2*np.pi,101))
>>> print(x[0:51])
[ 0.          0.06283185  0.12566371 ...,  3.01592895  3.0787608 3.14159265]

If either of the indices are floats, then the limits are set by the values themselves (assuming the array is monotonic):

>>> print(x[0.5:1.5])
[ 0.50265482  0.56548668  0.62831853 ...,  1.38230077  1.44513262 1.50796447]

NXdata

It is also possible to slice whole NXdata groups. In this case, the slicing works on the multidimensional NXfield, but the full NXdata group is returned with both the signal data and the associated axes limited by the slice parameters. If either of the limits along any one axis is a float, the limits are set by the values of the axis:

>>> a=NXdata(np.sin(x),x)
>>> a[1.5:2.5].x
NXfield(name=x,value=[ 1.57079633  1.72787596  1.88495559 ...,  2.19911486  2.35619449])

Unless the slice reduces one of the axes to a single item, the rank of the data remains the same. To project data along one of the axes, and so reduce the rank by one, the data can be summed along that axis using the sum() method. This employs the NumPy array sum() method:

>>> x=y=NXfield(np.linspace(0,2*np.pi,41))
>>> X,Y=np.meshgrid(x,y)
>>> a=NXdata(np.sin(X)*np.sin(Y), (y,x))
>>> print(a.tree)
data:NXdata
  @axes = ['axis1' 'axis2']
  @signal = 'signal'
  axis1 = float64(41)
  axis2 = float64(41)
  signal = float64(41x41)
>>> print(a.sum(0).tree)
data:NXdata
  @axes = 'axis2'
  @signal = 'signal'
  axis2 = float64(41)
  signal = float64(41)
    @long_name = 'Integral from 0.0 to 6.28318530718'

This can be extended to higher dimensions, using a tuple as the sum() argument. The following code projects a NXdata group, whose signal is a three-dimensional array, down to a one-dimensional NXdata group. The average values of the summed axes are stored as fields, with attributes showing the range of the summation.

>>> signal=NXfield(np.arange(60).reshape((3,4,5)), name='v')
>>> x=NXfield(np.arange(5.0), name='x')
>>> y=NXfield(np.arange(4.0), name='y')
>>> z=NXfield(np.arange(3.0), name='z')
>>> d=NXdata(signal, (z, y, x))
>>> print(d.tree)
data:NXdata
  @axes = ['z', 'y', 'x']
  @signal = 'v'
  v = int64(3x4x5)
  x = float64(5)
  y = float64(4)
  z = [0. 1. 2.]
>>> print(d.sum((0,1)).tree)
data:NXdata
  @axes = 'x'
  @signal = 'v'
  @summed_bins = 12
  title = 'data/data'
  v = int64(5)
  x = float64(5)
  y = 1.5
    @maximum = 3.0
    @minimum = 0.0
    @summed_bins = 4
  z = 1.0
    @maximum = 2.0
    @minimum = 0.0
    @summed_bins = 3

The Python Graphical User Interface provides a menu-based approach to simplify the plotting of 1D and 2D data projections of multidimensional data.

Saving NeXus Data

Every NeXus object, whether it is a group or a field, has a save() method as illustrated in Creating NeXus Data.:

>>> root.save(filename='example.nxs')

NXroot Groups

If the NeXus object is a NXroot group, the save() method saves the whole NeXus tree. The filename can only be omitted if the tree is being saved to a file that was loaded with read/write access. In this case, the format argument is ignored. If the tree was loaded with readonly access, any modifications must be saved to a new file specified by the filename argument.

Other Objects

If the object is not a NXroot group, a new file will be created containing the selected object and its children. A filename must be specified. Saving non-NXroot data allows parts of a NeXus tree to be saved for later use, e.g., to store an NXsample group that will be added to other files. The saved NeXus object is wrapped in an NXroot group and an NXentry group (with name ‘entry’), if necessary, in order to produce a valid NeXus file.

NeXus File Operations

Changes to a NeXus tree that has been loaded from disk or saved to a file are automatically updated in the HDF5 file, assuming that it is opened with read/write permissions. This means that the tree is always an accurate representation of the current state of the NeXus file, unless it has been modified by an external process, in which case, the file should be reloaded.

Note

In the Python Graphical User Interface, the lock icon color for an externally modified file changes to red.

When a file is loaded, using the nxload function, the nxfile attribute of the root group is an NXFile object, which is thin wrapper over the underlying h5py.File object:

>>> root = nxload('chopper.nxs', 'r')
>>> root['entry']
NXentry('entry')
>>> root.nxfile['/entry']
<HDF5 group "/entry" (10 members)>

The nxload function can also be used to create a new file with the mode set to ‘w’. Any keywords accepted by h5py.File can be used to customize the new HDF5 file, e.g., to turn on SWMR mode.

Warning

There is usually no need to call the nxfile attribute except to invoke the context manager (see next section). If it is referenced, the underlying h5py.File object is left open. It should be explicitly closed by calling root.nxfile.close(). The current status of the file can be determined by calling root.nxfile.is_open().

Multiple operations

When a change is made to a NeXus file, which is open with read/write access, it is automatically opened, updated, and then closed to ensure that any changes are flushed to the file and other processes can read the file if necessary. When writing or modifying multiple items in the file, it is possible to use a context manager to prevent multiple open/close operations:

>>> with root.nxfile:
>>>     root['entry/sample'] = NXsample()
>>>     root['entry/sample/temperature'] = NXfield(40.0, units='K')
>>>     root['entry/sample/mass'] = NXfield(5.0, units='g')

The file will be opened at the start of the of the with clause and closed automatically at the end.

Note

This context manager can be nested so it is safe to add a with clause within a function that might, in some implementations, be embedded in another with clause. The file is only closed when the outermost context manager is exited.

In v0.7.7, the ability to use a context manager directly with NXroot objects, rather than its associated NXfile, was added. This allows the use of a similar syntax to the Python open function, in which a with clause ensuring that the file is opened and closed, before and after the file access, respectively. To make this analogy clearer, nxopen was added as an alias to nxload.

In the following code, a NeXus file is created, filled with NeXus objects and then closed.

>>> with nxopen('nexus_file.nxs', 'w') as root:
>>>     root['entry'] = NXentry()
>>>     root['entry/sample'] = NXsample()
>>>     root['entry/sample/temperature'] = NXfield(40.0, units='K')

File Locking

The context manager can also be used to lock the NeXus file to prevent other processes from accessing the file. According to the HDF5 documentation, concurrent read access is supported if the HDF5 library has been built as thread-safe. This appears to be the default with conda installations, for example. However, concurrent read and write access is only allowed when using SWMR mode. To prevent issues with multiple processes accessing the same file, nexusformat contains a simple file-locking mechanism, which is designed to work even when the processes are running on separate nodes and when other file-locking mechanisms might prove unreliable (e.g., on NFS-mounted disks).

Warning

Unfortunately, the word ‘lock’ can cause confusion because it is commonly used to refer to two different operations. The other one is to switch a file from read/write to read-only mode, e.g.,

>>> root.lock()

This operation will prevent the current process from writing to the file, but it does not add a file lock to prevent I/O conflicts with other processes.

A new file is created with the same name as the NeXus file, with the additional extension ‘.lock’. Other processes using the nexusformat package will wait until the lock is cleared before performing any further I/O operations. By default, this lock file is created in the same directory as the NeXus file, but this will fail if the user does not have sufficient permissions to create the file in that directory. For this reason, it is possible to define another directory with relaxed group and/or world permissions to store the lock files.

Configuring File Locks

File-locking is configured using nxgetconfig and nxsetconfig (see next section). File locking is enabled by setting a non-zero value for the lock parameter, which defines the length of time the process will wait before triggering a NXLockException exception. Then, the context manager described above will create and remove the lock file at the beginning and end of the with clause, respectively.

>>> nxgetconfig('lock')
0
>>> nxsetconfig(lock=10)
>>> with root.nxfile:
>>>     root['entry/sample'] = NXsample()
>>>     root['entry/sample/temperature'] = NXfield(40.0, units='K')

The lock file name is the name of the NeXus file with .lock appended. If a stale lock is encountered, it may be cleared by calling clear_lock:

>>> root.nxfile.is_locked()
True
>>> root.nxfile.clear_lock()
>>> root.nxfile.is_locked()
False

Note

This lock is advisory. It is only guaranteed to work if the external process is also using nexusformat.

Configuration Parameters

The nexusformat package uses a number of parameters to configure its default behavior. These are stored internally in a dictionary, which may be read or modified using the nxgetconfig and nxsetconfig functions, respectively.

>>> nxgetconfig()
{'compression': 'gzip',
 'encoding': 'utf-8',
 'lock': 0,
 'lockexpiry': 28800,
 'lockdirectory': None,
 'maxsize': 10000,
 'memory': 2000,
 'recursive': False}
>>> nxsetconfig(memory=4000)
>>> nxgetconfig('memory')
4000

Here is a list of the current configuration parameters and their defaults.

  • compression:

    This sets the default HDF5 compression filter. The default is ‘gzip’.

  • encoding:

    This sets the default encoding for input strings. The default is ‘utf-8’.

  • lock:

    This sets the number of seconds before an attempted file lock acquisition times out. The default is 10 seconds. If set to 0, file locking is disabled (but see below).

  • lockexpiry:

    This sets the number of seconds before a file lock is considered stale. If the lock file is older than this value, a new lock can be acquired. The default is 28,800 seconds (8 hours).

  • lockdirectory:

    This defines the path to a directory, in which to store the lock files. The directory should be set to allow users to create files. The default is None, in which case, file locks are stored in the same directory as the NeXus file to be locked.

    Note

    If lockdirectory is defined, the lock parameter is automatically set to 10 seconds if the currently set value is 0, i.e., defining a lock directory is enough to enable file locking.

  • maxsize:

    This sets the maximum size of an array before HDF5 chunking and compression is turned on by default. The default is 10,000.

  • memory:

    This sets the memory limit (in MB) for loading arrays into memory. If a field contains data that is larger than this limit, it can only be accessed as a series of smaller slabs using the standard slicing syntax. The default is 2000 MB.

  • recursive:

    This sets the default method of loading NeXus files. If the value is set to True, all objects in the file are loaded (lazily) into memory. If set to False, only the first two levels of hierarchy are initially loaded. Lower levels are loaded when they are referenced. This includes tests for the existence of object paths in the file. The default is False.

Environment variables

The configuration parameters can also be set by defining environment variables, defined either in a user’s login files or by a system administrator. This is particularly useful for setting a system-wide lock-file directory for all users accessing the same data.

When the nexusformat package is loaded, environment variables take precedence over the package defaults. The user can still override them manually by calling nxsetconfig.

All of the configuration parameters defined in the previous section can be defined. The equivalent environment variable name is constructed by prefixing the parameter name in upper case by ‘NX_’, e.g., ‘NX_COMPRESSION’, ‘NX_ENCODING’, etc.