History#

As a regular user of MaskedArray, I (Pierre G.F. Gerard-Marchant) became
increasingly frustrated with the subclassing of masked arrays (even if
I can only blame my inexperience). I needed to develop a class of arrays
that could store some additional information along with numerical values,
while keeping the possibility for missing data (picture storing a series
of dates along with measurements, what would later become the TimeSeries
Scikit
(dead link).

I started to implement such a class, but then quickly realized that
any additional information disappeared when processing these subarrays
(for example, adding a constant value to a subarray would erase its
dates). I ended up writing the equivalent of numpy.core.ma for my
particular class, ufuncs included. Everything went fine until I needed to
subclass my new class, when more problems showed up: some attributes of
the new subclass were lost during processing. I identified the culprit as
MaskedArray, which returns masked ndarrays when I expected masked
arrays of my class. I was preparing myself to rewrite numpy.core.ma
when I forced myself to learn how to subclass ndarrays. As I became more
familiar with the __new__ and __array_finalize__ methods,
I started to wonder why masked arrays were objects, and not ndarrays,
and whether it wouldn’t be more convenient for subclassing if they did
behave like regular ndarrays.

The new maskedarray is what I eventually come up with. The
main differences with the initial numpy.core.ma package are
that MaskedArray is now a subclass of ndarray and that the
_data section can now be any subclass of ndarray. Apart from a
couple of issues listed below, the behavior of the new MaskedArray
class reproduces the old one. Initially the maskedarray
implementation was marginally slower than numpy.ma in some areas,
but work is underway to speed it up; the expectation is that it can be
made substantially faster than the present numpy.ma.

Note that if the subclass has some special methods and
attributes, they are not propagated to the masked version:
this would require a modification of the __getattribute__
method (first trying ndarray.__getattribute__, then trying
self._data.__getattribute__ if an exception is raised in the first
place), which really slows things down.

Main differences#

• The _data part of the masked array can be any subclass of ndarray (but not recarray, cf below).

• fill_value is now a property, not a function.

• in the majority of cases, the mask is forced to nomask when no value is actually masked. A notable exception is when a masked array (with no masked values) has just been unpickled.

• I got rid of the share_mask flag, I never understood its purpose.

• put, putmask and take now mimic the ndarray methods, to avoid unpleasant surprises. Moreover, put and putmask both update the mask when needed. * if a is a masked array, bool(a) raises a ValueError, as it does with ndarrays.

• in the same way, the comparison of two masked arrays is a masked array, not a boolean

• filled(a) returns an array of the same subclass as a._data, and no test is performed on whether it is contiguous or not.

• the mask is always printed, even if it’s nomask, which makes things easy (for me at least) to remember that a masked array is used.

• cumsum works as if the _data array was filled with 0. The mask is preserved, but not updated.

• cumprod works as if the _data array was filled with 1. The mask is preserved, but not updated.

New features#

This list is non-exhaustive…

• the mr_ function mimics r_ for masked arrays.

• the anom method returns the anomalies (deviations from the average)

Using the new package with numpy.core.ma#

I tried to make sure that the new package can understand old masked
arrays. Unfortunately, there’s no upward compatibility.

For example:

>>> import numpy.core.ma as old_ma
>>> import maskedarray as new_ma
>>> x = old_ma.array([1,2,3,4,5], mask=[0,0,1,0,0])
>>> x
array(data =
 [     1      2 999999      4      5],
      mask =
 [False False True False False],
      fill_value=999999)
>>> y = new_ma.array([1,2,3,4,5], mask=[0,0,1,0,0])
>>> y
array(data = [1 2 -- 4 5],
      mask = [False False True False False],
      fill_value=999999)
>>> x==y
array(data =
 [True True True True True],
      mask =
 [False False True False False],
      fill_value=?)
>>> old_ma.getmask(x) == new_ma.getmask(x)
array([True, True, True, True, True])
>>> old_ma.getmask(y) == new_ma.getmask(y)
array([True, True, False, True, True])
>>> old_ma.getmask(y)
False

Using maskedarray with matplotlib#

Starting with matplotlib 0.91.2, the masked array importing will work with
the maskedarray branch) as well as with earlier versions.

By default matplotlib still uses numpy.ma, but there is an rcParams setting
that you can use to select maskedarray instead. In the matplotlibrc file
you will find:

#maskedarray : False       # True to use external maskedarray module
                           # instead of numpy.ma; this is a temporary #
                           setting for testing maskedarray.

Uncomment and set to True to select maskedarray everywhere.
Alternatively, you can test a script with maskedarray by using a
command-line option, e.g.:

python simple_plot.py --maskedarray

Masked records#

Like numpy.ma.core, the ndarray-based implementation
of MaskedArray is limited when working with records: you can
mask any record of the array, but not a field in a record. If you
need this feature, you may want to give the mrecords package
a try (available in the maskedarray directory in the scipy
sandbox). This module defines a new class, MaskedRecord. An
instance of this class accepts a recarray as data, and uses two
masks: the fieldmask has as many entries as records in the array,
each entry with the same fields as a record, but of boolean types:
they indicate whether the field is masked or not; a record entry
is flagged as masked in the mask array if all the fields are
masked. A few examples in the file should give you an idea of what
can be done. Note that mrecords is still experimental…

Optimizing maskedarray#

Should masked arrays be filled before processing or not?#

In the current implementation, most operations on masked arrays involve
the following steps:

• the input arrays are filled

• the operation is performed on the filled arrays

• the mask is set for the results, from the combination of the input masks and the mask corresponding to the domain of the operation.

For example, consider the division of two masked arrays:

import numpy
import maskedarray as ma
x = ma.array([1,2,3,4],mask=[1,0,0,0], dtype=numpy.float64)
y = ma.array([-1,0,1,2], mask=[0,0,0,1], dtype=numpy.float64)

The division of x by y is then computed as:

d1 = x.filled(0) # d1 = array([0., 2., 3., 4.])
d2 = y.filled(1) # array([-1.,  0.,  1.,  1.])
m = ma.mask_or(ma.getmask(x), ma.getmask(y)) # m =
array([True,False,False,True])
dm = ma.divide.domain(d1,d2) # array([False,  True, False, False])
result = (d1/d2).view(MaskedArray) # masked_array([-0. inf, 3., 4.])
result._mask = logical_or(m, dm)

Note that a division by zero takes place. To avoid it, we can consider
to fill the input arrays, taking the domain mask into account, so that:

d1 = x._data.copy() # d1 = array([1., 2., 3., 4.])
d2 = y._data.copy() # array([-1.,  0.,  1.,  2.])
dm = ma.divide.domain(d1,d2) # array([False,  True, False, False])
numpy.putmask(d2, dm, 1) # d2 = array([-1.,  1.,  1.,  2.])
m = ma.mask_or(ma.getmask(x), ma.getmask(y)) # m =
array([True,False,False,True])
result = (d1/d2).view(MaskedArray) # masked_array([-1. 0., 3., 2.])
result._mask = logical_or(m, dm)

Note that the .copy() is required to avoid updating the inputs with
putmask. The .filled() method also involves a .copy().

A third possibility consists in avoid filling the arrays:

d1 = x._data # d1 = array([1., 2., 3., 4.])
d2 = y._data # array([-1.,  0.,  1.,  2.])
dm = ma.divide.domain(d1,d2) # array([False,  True, False, False])
m = ma.mask_or(ma.getmask(x), ma.getmask(y)) # m =
array([True,False,False,True])
result = (d1/d2).view(MaskedArray) # masked_array([-1. inf, 3., 2.])
result._mask = logical_or(m, dm)

Note that here again the division by zero takes place.

A quick benchmark gives the following results:

• numpy.ma.divide : 2.69 ms per loop

• classical division : 2.21 ms per loop

• division w/ prefilling : 2.34 ms per loop

• division w/o filling : 1.55 ms per loop

So, is it worth filling the arrays beforehand ? Yes, if we are interested
in avoiding floating-point exceptions that may fill the result with infs
and nans. No, if we are only interested into speed…

Thanks#

I’d like to thank Paul Dubois, Travis Oliphant and Sasha for the
original masked array package: without you, I would never have started
that (it might be argued that I shouldn’t have anyway, but that’s
another story…). I also wish to extend these thanks to Reggie Dugard
and Eric Firing for their suggestions and numerous improvements.

Revision notes#

• 08/25/2007 : Creation of this page

• 01/23/2007 : The package has been moved to the SciPy sandbox, and is regularly updated: please check out your SVN version!