DOLfYN

The DOLfYN module contains a set of functions to analyze binary Nortek or TRDI files. Instrument datafiles are processed and returned as MATLAB structures.

Note

DOLfYN support within MHKiT-MATLAB is a work in progress. Please submit any issues or improvements to the MHKiT-MATLAB GitHub Repository

Note

The names of the functions below are of the convention path.path.function. Only the function name is used when calling the function in MATLAB. For example, to call on mhkit.dolfyn.io.read_nortek simply use read_nortek.

IO

The io submodule contains the following functions to read binary Nortek (e.g., .VEC, .wpr, .ad2cp, etc.) or TRDI (.000, .PD0, .ENX, etc.) data files.

mhkit.dolfyn.io.dolfyn_read(filename, options)

Read a binary Nortek (e.g., .VEC, .wpr, .ad2cp, etc.) or RDI

(.000, .PD0, .ENX, etc.) data file.

Parameters:

  • filename (string) – Filename of instrument file to read.

  • userdata (bool or string (optional)) – true, false, or string of userdata.json filename (default true) Whether to read the ‘<base-filename>.userdata.json’ file.

  • nens (nan, int, or 2-element array (optional)) – nan (default: read entire file), int, or 2-element tuple (start, stop) Number of pings to read from the file.

  • dolfyn_read(filename (call with options ->)

  • 'userdata'

  • false

  • 'nens'

  • 12)

Returns:

ds (structure) – Structure from the binary instrument data

mhkit.dolfyn.io.nortek_defs

Run this file to generate the nortek_defs.mat file

mhkit.dolfyn.io.read_h5(filename)

Read H5 data structure.

Parameters:

filename (string) – Filename of NetCDF file to read.

Returns:

ds (structure) – Structure from the binary instrument data

mhkit.dolfyn.io.read_netcdf(filename)

Read NetCDF data structure.

Parameters:

filename (string) – Filename of NetCDF file to read.

Returns:

ds (structure) – Structure from the binary instrument data

mhkit.dolfyn.io.read_nortek(filename, options)

Read a classic Nortek (AWAC and Vector) datafile

Parameters:

  • filename (string) – Filename of Nortek file to read.

  • userdata (bool or string (optional)) – true, false, or string of userdata.json filename (default true) Whether to read the ‘<base-filename>.userdata.json’ file.

  • do_checksum (bool (optional)) – Whether to perform the checksum of each data block. (default false)

  • nens (nan, int, or 2-element array (optional)) – nan (default: read entire file), int, or 2-element tuple (start, stop) Number of pings to read from the file.

  • read_nortek(filename (call with options ->)

  • 'userdata'

  • false

  • 'do_checksum'

  • true

  • 'nens'

  • 12)

Returns:

ds (structure) – Structure from the binary instrument data

mhkit.dolfyn.io.read_rdi(filename, options)

Read a TRDI binary data file.

Parameters:

  • filename (string) – Filename of TRDI file to read.

  • userdata (bool or string (optional)) – true, false, or string of userdata.json filename (default true) Whether to read the ‘<base-filename>.userdata.json’ file.%

  • nens (nan, int, or 2-element array (optional)) – nan (default: read entire file), int, or 2-element tuple (start, stop) Number of pings to read from the file.

  • read_rdi(filename (call with options ->)

  • 'userdata'

  • false

  • 'do_checksum'

  • true

  • 'nens'

  • 12)

Returns:

ds (structure) – Structure from the binary instrument data

mhkit.dolfyn.io.read_signature(filename, options)

Read a Nortek Signature (.ad2cp) datafile

Parameters:

  • filename (string) – Filename of Nortek file to read.

  • userdata (bool or string (optional)) – true, false, or string of userdata.json filename (default true) Whether to read the ‘<base-filename>.userdata.json’ file.

  • nens (nan, int, or 2-element array (optional)) – nan (default: read entire file), int, or 2-element tuple (start, stop) Number of pings to read from the file.

  • read_signature(filename (call with options ->)

  • 'userdata'

  • false

  • 'nens'

  • 12)

Returns:

ds (structure) – Structure from the binary instrument data

mhkit.dolfyn.io.read_userdata(filename, userdata)

Reads a userdata.json file and returns the data it contains as a structure

Parameters:

  • filename (string) – Filename of Nortek file to read.

  • userdata (bool or string) – true, false, or string of userdata.json filename

Returns:

userdata (structure)

mhkit.dolfyn.io.write_netcdf(ds, filename)

Write Dolfyn data set to NetCDF data structure.

Parameters:

  • ds (structure) – Structure from the binary instrument data

  • filename (string) – Filename of NetCDF file to read.

ADP (Acoustic Doppler Profiler)

The ADP module contains routines for reading and working with ADP/ADCP data. It contains:

mhkit.dolfyn.adp.correlation_filter(ds, options)

Filters out velocity data where correlation is below a threshold in the beam correlation data.

Parameters:

  • ds (Dataset) – The adcp dataset to clean.

  • thresh (numeric) – The maximum value of correlation to screen, in counts or %, default is 50

  • val (numeric) – Value to set masked correlation data to, default is nan

Returns:

ds (xarray.Dataset) – Velocity data with low correlation values set to val

Notes

Does not edit correlation or amplitude data.

mhkit.dolfyn.adp.find_surface_from_P(ds, options)

Calculates the distance to the water surface. Temperature and salinity are used to calculate seawater density, which is in turn used with the pressure data to calculate depth.

Parameters:

  • ds (Dataset) – The full adcp dataset

  • salinity (numeric) – Water salinity in psu

Returns:

out (Dataset) – adds the variables “water_density” and “depth” to the input dataset.

Notes

Requires that the instrument’s pressure sensor was calibrated/zeroed before deployment to remove atmospheric pressure.

Calculates seawater density at normal atmospheric pressure according to the UNESCO 1981 equation of state. Does not include hydrostatic pressure.

mhkit.dolfyn.adp.nan_beyond_surface(ds, options)

Mask the values of 3D data (vel, amp, corr, echo) that are beyond the surface.

Parameters:

  • ds (Dataset) – The adcp dataset to clean

  • val (nan or numeric) – Specifies the value to set the bad values to (default np.nan).

Returns:

ds (Dataset) – The adcp dataset where relevant arrays with values greater than depth are set to NaN

Notes

Surface interference expected to happen at distance > range * cos(beam angle) - cell size

mhkit.dolfyn.adp.set_range_offset(ds, h_deploy)

Adds an instrument’s height above seafloor (for an up-facing instrument) or depth below water surface (for a down-facing instrument) to the range coordinate. Also adds an attribute to the Dataset with the current “h_deploy” distance.

Parameters:

  • ds (xarray.Dataset) – The adcp dataset to ajust ‘range’ on

  • h_deploy (numeric) – Deployment location in the water column, in [m]

Returns:

None, operates “in place”

Notes

Center of bin 1 = h_deploy + blank_dist + cell_size

Nortek doesn’t take h_deploy into account, so the range that DOLfYN calculates distance is from the ADCP transducers. TRDI asks for h_deploy input in their deployment software and is thereby known by DOLfYN.

If the ADCP is mounted on a tripod on the seafloor, h_deploy will be the height of the tripod +/- any extra distance to the transducer faces. If the instrument is vessel-mounted, h_deploy is the distance between the surface and downward-facing ADCP’s transducers.

Rotate

The rotate submodule contains tools to rotate a dataset to different coordinate systems. When a file is read into dolfyn, the data will be stored in the same coordinate system it was saved in. The different coordinate systems are “beam” <-> “inst” <-> “earth” <-> “principal”. “Beam” and “inst” are manufacturer/instrument specific and refer to the along-beam (either 3 or 4 beams) and instrument (XYZ) reference frame. Instrument reference frames differ across sensors, but the rotate code takes this into account when rotating into the “earth” frame, defined as East-North-Up (ENU). The earth frame can then be rotated to the principal axes, defined as streamwise-cross_stream-vertical.

mhkit.dolfyn.rotate.beam2inst(dat, reverse, rmod)

Rotate velocities from beam to instrument coordinates.

datThe adv object containing the data.

The ADCP dataset

reversebool

If True, this function performs the inverse rotation (inst->beam).

rmod: string

awac, vector, signature, rdi

mhkit.dolfyn.rotate.calc_beam_orientmat(theta, convex, degrees)

Calculate the rotation matrix from beam coordinates to instrument head coordinates for an RDI ADCP.

Parameters:

  • theta (is the angle of the heads (usually 20 or 30 degrees))

  • convex (is a flag for convex or concave head configuration.)

  • degrees (is a flag which specifies whether theta is in degrees) – or radians (default: degrees=True)

Returns:

out (array) – orientation matrix

mhkit.dolfyn.rotate.calc_omat(hh, pp, rr, orientation_down)
mhkit.dolfyn.rotate.calc_orientmat(adcpo)

Calculate the orientation matrix using the raw heading, pitch, roll values from the RDI binary file.

Parameters:

  • adcpo (The ADP object containing the data.)

  • 08 (## RDI-ADCP-MANUAL (Jan)

  • 18) (section 5.6 page)

  • same (The internal tilt sensors do not measure exactly the)

  • in (pitch as a set of gimbals would (the roll is the same). Only)

  • (EZxxx1xxx) (the case of the internal pitch sensor being selected)

:param : :param the measured pitch is modified using the following algorithm.: P = arctan[tan(Tilt1)*cos(Tilt2)] (Equation 18) :param Where: :type Where: Tilt1 is the measured pitch from the internal sensor, and :param Tilt2 is the measured roll from the internal sensor The raw pitch: :param (Tilt 1) is recorded in the variable leader. P is set to 0 if the: :param “use tilt” bit of the EX command is not set.”””:

Returns:

rotmat (array) – orientation matrix

mhkit.dolfyn.rotate.calc_principal_heading(vel, tidal_mode)

Compute the principal angle of the horizontal velocity.

Parameters:

  • vel (array) – The 2D or 3D velocity array

  • tidal_mode (bool)

Returns:

p_heading (float or ndarray) – The principal heading in degrees clockwise from North.

Notes

When tidal_mode=True, this tool calculates the heading that is aligned with the bidirectional flow. It does so following these steps:

  1. rotates vectors with negative velocity by 180 degrees

  2. then doubles those angles to make a complete circle again

  3. computes a mean direction from this, and halves that angle (to undo the doubled-angles in step 2)

  4. The returned angle is forced to be between 0 and 180. So, you may need to add 180 to this if you want your positive direction to be in the western-half of the plane.

Otherwise, this function simply computes the average direction using a vector method.

mhkit.dolfyn.rotate.earth2principal(advo, reverse)

Rotate data in an ADV dataset to/from principal axes. Principal heading must be within the dataset.

All data in the advo.attrs[‘rotate_vars’] list will be rotated by the principal heading, and also if the data objet has an orientation matrix (orientmat) it will be rotated so that it represents the orientation of the ADV in the principal (reverse:earth) frame.

Parameters:

  • advo (The adv object containing the data.)

  • reverse (bool (default: False)) – If True, this function performs the inverse rotation (principal->earth).

mhkit.dolfyn.rotate.euler2orient(heading, pitch, roll, options)

For Nortek data only. The heading, pitch, roll used here are from the Nortek binary files.

Parameters:

  • heading (array) – Heading input is clockwise from North

  • pitch (array)

  • roll (array)

  • units (string (optional)) – radians or degrees (default degrees) Whether heading, pitch, and roll are in degrees or radians.

Returns:

omat – a rotation matrix that rotates earth (ENU) -> inst. This is based on the Nortek Transforms.m file, available in the refs folder.

mhkit.dolfyn.rotate.inst2earth(advo, reverse, make)

Rotate data in an ADV object to the earth from the instrument frame (or vice-versa).

Parameters:

  • advo (The adv object containing the data.)

  • reverse (bool (default: False)) – If True, this function performs the inverse rotation (earth->inst).

  • make (char) – awac, vector, signature, rdi

mhkit.dolfyn.rotate.inst2head(advo, reverse)
mhkit.dolfyn.rotate.orient2euler(omat)

Calculate DOLfYN-defined euler angles from the orientation matrix.

Parameters:

omat (array) – The orientation matrix

Returns:

  • heading (|ndarray|) – The heading angle. Heading is defined as the direction the x-axis points, positive clockwise from North (this is opposite the right-hand-rule around the Z-axis), range 0-360 degrees.

  • pitch (ndarray) – The pitch angle (degrees). Pitch is positive when the x-axis pitches up (this is opposite the right-hand-rule around the Y-axis).

  • roll (ndarray) – The roll angle (degrees). Roll is positive according to the right-hand-rule around the instrument’s x-axis.

mhkit.dolfyn.rotate.quaternion2orient(quaternions)
Calculate orientation from Nortek AHRS quaternions, where

q = [W, X, Y, Z] instead of the standard q = [X, Y, Z, W] = [q1, q2, q3, q4]

Parameters:

quaternions (Structure) – Quaternion structure from the raw dataset

Returns:

orientmat (|ndarray|) – The inst2earth rotation maxtrix as calculated from the quaternions

See also

scipy.spatial.transform.Rotation

mhkit.dolfyn.rotate.rotate2(data_set, out_frame)

Rotate a dataset to a new coordinate system.

Parameters:

  • ds (xr.Dataset) – The dolfyn dataset (ADV or ADCP) to rotate.

  • out_frame (string {'beam', 'inst', 'earth', 'principal'}) – The coordinate system to rotate the data into.

Returns:

ds (xarray.Dataset) – The rotated dataset

Notes

This function rotates all variables in ds.attrs.rotate_vars

mhkit.dolfyn.rotate.set_declination(data_set, declin)

Set the magnetic declination

Parameters:

declination (float) – The value of the magnetic declination in degrees (positive values specify that Magnetic North is clockwise from True North)

Returns:

ds (dataset structure) – Dataset adjusted for the magnetic declination

Notes

This method modifies the data object in the following ways:

  • If the dataset is in the earth reference frame at the time of setting declination, it will be rotated into the “True-East, True-North, Up” (hereafter, ETU) coordinate system

  • dat[‘orientmat’] is modified to be an ETU to instrument (XYZ) rotation matrix (rather than the magnetic-ENU to XYZ rotation matrix). Therefore, all rotations to/from the ‘earth’ frame will now be to/from this ETU coordinate system.

  • The value of the specified declination will be stored in dat.attrs[‘declination’]

  • dat[‘heading’] is adjusted for declination (i.e., it is relative to True North).

  • If dat.attrs[‘principal_heading’] is set, it is adjusted to account for the orientation of the new ‘True’ earth coordinate system (i.e., calling set_declination on a data object in the principal coordinate system, then calling dat.rotate2(‘earth’) will yield a data object in the new ‘True’ earth coordinate system)

mhkit.dolfyn.rotate.set_inst2head_rotmat(ds, rotmat)

Set the instrument to head rotation matrix for cable head Nortek ADVs.

Parameters:

rotmat (float) – 3x3 rotation matrix

Returns:

data_set (structure) – Dataset with rotation matrix applied

mhkit.dolfyn.rotate.tensorproduct(A, B, subscripts) %ind_R, A, ind_A, B, ind_B)

TENSORPRODUCT Implementation of Einstein summation convention for multidimensional matlab arrays, where repeated indices sum over.

Call:

R = tensorproduct(ind_R,A,ind_A,B,ind_B);

Inputs:
ind_Rstring with indices of the output

A : (non-empty) (multidimensional) matlab array

ind_Astring with indices of array A

B : (non-empty) (multidimensional) matlab array

ind_B : string with indices of array B

Supports multiple: - Outer products - Inner products - Singleton dimensions - Pages

Does not support (yet): - Self-contraction

Example

A = rand(5, 1,4,8); B = rand(4,10,5 ); R = tensorproduct(‘jzgi’,A,’gxki’,B,’kjg’); % Outer: ‘i’,’j’

% Inner: ‘k’ % Page: ‘g’ % Singleton: ‘x’,’z’ % size(R) = [10,1,5,8]

Note

Provided indices in (ind_R), (ind_A) and (ind_B) are case-sensitive, i.e. ‘a’ and ‘A’ are different indices.

Version compatibility:

This implementation makes use of Matlab built-in function pagemtimes, introduced in Matlab version R2020b. To make use of this implementation in previous Matlab releases, comment out the statement:

R = pagemtimes(A,B);

and uncomment the following lines:

R = zeros(size(A,1),size(B,2),size(A,3)); for i = 1:size(A,3)

R(:,:,i) = A(:,:,i) * B(:,:,i);

end

While it will still work, performance (wall-clock time) might be degraded.

Benchmarking:

The performance of tensorproduct can be assessed by calling the function tensorproduct_benchmark.

mhkit.dolfyn.rotate.tensorproduct_core(A, B, A_permute, B_permute, R_permute, A_reshape, B_reshape, R_reshape)

Rearrangement of input

mhkit.dolfyn.rotate.tensorproduct_initialize(ind_A, ind_B, ind_R, sizeA, sizeB)

TENSORPRODUCT_INITIALIZE Part of the tensorproduct utility Copyright 2021 - by David Codony, PhD (dcodony@cimne.upc.edu) This software is distributed without any warranty. Permission is granted for anyone to copy, use, or modify this software for any uncommercial purposes, provided this copyright notice is retained, and note is made of any changes that have been made. Assert empty inputs

Tools

The clean submodule contains basic quality control tools specific to ADCP and ADVs.

mhkit.dolfyn.tools.bluewhitered(m)

BLUEWHITERED Blue, white, and red color map. BLUEWHITERED(M) returns an M-by-3 matrix containing a blue to white to red colormap, with white corresponding to the CAXIS value closest to zero. This colormap is most useful for images and surface plots with positive and negative values. BLUEWHITERED, by itself, is the same length as the current colormap.

Examples:

figure imagesc(peaks(250)); colormap(bluewhitered(256)), colorbar

figure imagesc(peaks(250), [0 8]) colormap(bluewhitered), colorbar

figure imagesc(peaks(250), [-6 0]) colormap(bluewhitered), colorbar

figure surf(peaks) colormap(bluewhitered) axis tight

See also HSV, HOT, COOL, BONE, COPPER, PINK, FLAG, COLORMAP, RGBPLOT.

mhkit.dolfyn.tools.create_dataset(data)

Creates final dataset from structure created from binary readers. It is meant to try and approximate an xarray dataset Direction ‘dir’ coordinates get reset in set_coords

mhkit.dolfyn.tools.dolfyn_plot(ds, field, options)

Plot fields from Dolfyn generated data structures

Parameters:

  • ds (structure) – Structure from the binary instrument data

  • field (string) – field from ds to be plotted (ex: ‘vel’)

  • dim (int (optional)) – for higher dimension fields, which dimension should be plotted

  • title (string (optional)) – Title for the plot

  • dolfyn_plot(ds (call with options ->)

  • 'vel'

  • 'dim'

  • 1

  • 'title'

  • plot') ('My)

mhkit.dolfyn.tools.fillgaps(a, extrapFlg)

Linearly fill NaN value in an array.

Parameters:

  • a (array) – The array to be filled.

  • extrapFlg (bool) – Whether to extrapolate if NaNs are found at the ends of the array.

Notes

This function interpolates assuming spacing/timestep between successive points is constant. If the spacing is not constant, use interpgaps.

mhkit.dolfyn.tools.fill_time_gaps(epoch, sample_rate_hz)

Fill gaps (NaN values) in the timeseries by simple linear interpolation. The ends are extrapolated by stepping forward/backward by 1/sample_rate_hz.

mhkit.dolfyn.tools.py_struct_2_bytes_format(py_format, size_only)
mhkit.dolfyn.tools.set_coords(ds, ref_frame)

Checks the current reference frame and adjusts xarray coords/dims as necessary. Makes sure assigned dataarray coordinates match what DOLfYN is reading in.