Example: MHKiT-MATLAB Quality Control Module
The following example runs a simple quality control analysis on wave elevation data using the MHKiT QC module. The data file used in this example is stored in the \MHKiT\examples\data directory. Import Data
Load the data into a table. The data includes several issues, including timestamps that are out of order, corrupt data with values of -999, data outside expected range, and stagant data.
data=readtable('../examples/data/qc/wave_elevation_data.csv');
disp(data)
Time probe1 probe2 probe3
______ ______ ______ ______
10 24.48 28.27 1.3
10.002 34.48 40.27 -8.7
10.004 30.48 38.27 -13.7
10.006 12.48 24.27 -32.7
10.008 13.48 22.27 -21.7
10.01 7.48 15.27 -13.7
10.012 4.48 13.27 -19.7
10.014 -9.52 -0.73 -11.7
10.016 8.48 19.27 -16.7
10.018 -4.52 8.27 -34.7
10.02 16.48 27.27 5.3
10.022 -13.52 -0.73 -36.7
10.024 -13.52 0.27 -39.7
10.026 -15.52 -3.73 -32.7
10.028 -22.53 -17.73 -23.7
10.03 -22.52 -16.73 16.3
10.032 -30.52 -23.73 1.3
10.034 -30.52 -16.73 11.3
10.036 -36.52 -24.73 13.3
10.038 -26.52 -13.73 23.3
10.04 -28.52 -14.73 25.3
10.046 -14.52 -11.73 33.3
10.048 -10.52 -1.73 36.3
10.05 -19.52 -10.73 28.3
10.052 -3.52 7.27 39.3
10.054 7.48 20.27 26.3
10.056 -2.52 14.27 11.3
10.058 4.48 19.27 13.3
10.06 2.48 17.27 16.3
10.062 9.48 22.27 2.3
10.064 17.48 23.27 19.3
10.066 16.48 27.27 26.3
10.068 33.48 42.27 5.3
10.07 28.48 36.27 -6.7
10.072 20.48 36.27 2.3
10.074 17.48 30.27 -7.7
10.076 11.48 23.27 -14.7
10.078 16.48 24.27 2.3
10.08 4.48 12.27 -8.7
10.082 15.48 22.27 -9.7
10.084 9.48 21.27 -15.7
10.086 -2.52 7.27 -14.7
10.088 -7.52 9.27 -31.7
10.09 3.48 17.27 -40.7
10.092 -2.52 8.27 -46.7
10.094 -13.52 -2.73 -19.7
10.096 -4.52 3.27 -13.7
10.098 -14.52 -7.73 NaN
10.1 -18.52 -11.73 NaN
10.102 -40.52 -22.73 NaN
10.104 -33.52 -18.73 NaN
10.106 -40.52 -23.73 NaN
10.108 -34.52 -18.73 NaN
10.11 -999 -27.73 NaN
10.112 -999 -19.73 NaN
10.114 -999 -2.73 NaN
10.116 -999 -21.73 46.3
10.118 -999 -26.73 21.3
10.12 -999 -14.73 31.3
10.122 -999 -13.73 30.3
10.124 -999 -16.73 15.3
10.126 -999 -15.73 34.3
10.128 -999 -18.73 47.3
10.13 -999 -0.73 31.3
10.132 -999 -1.73 48.3
10.134 -999 4.27 44.3
10.136 4.48 13.27 22.3
10.138 15.48 20.27 10.3
10.14 4.48 20.27 13.3
10.142 18.48 27.27 21.3
10.144 23.48 34.27 -1.7
10.146 20.48 26.27 -7.7
10.148 13.48 21.27 17.3
10.15 27.48 36.27 -1.7
10.152 20.48 31.27 -1.7
10.154 16.48 28.27 -17.7
10.156 12.48 27.27 -27.7
10.158 24.48 31.27 -35.7
10.16 10.48 20.27 -15.7
10.162 10.48 17.27 -25.7
10.164 11.48 16.27 -3.7
10.166 3.48 7.27 -25.7
10.168 -14.52 -7.73 -24.7
10.17 -5.52 6.27 -24.7
10.172 -7.52 1.27 -24.7
10.174 -13.52 -2.73 -42.7
10.176 -13.52 -3.73 -19.7
10.178 -20.52 -11.73 -8.7
10.18 -11.52 -10.73 -21.7
10.182 -27.52 -22.73 2.3
10.184 -28.52 -21.73 17.3
10.186 -29.52 -17.73 11.3
10.188 -28.52 -17.73 0.3
10.19 -41.52 -30.73 30.3
10.192 -35.52 -26.73 18.3
10.194 -33.52 -26.73 10.3
10.196 -20.52 -17.73 33.3
10.198 -17.52 -11.73 29.3
10.2 -10.52 -8.73 49.3
10.202 4.48 13.27 28.3
10.204 2.48 12.27 25.3
10.206 0.48 14.27 21.3
10.208 -7.52 3.27 NaN
10.21 -6.52 1.27 NaN
10.212 16.48 19.27 18.3
10.214 9.48 16.27 19.3
10.216 12.48 19.27 26.3
10.218 24.48 31.27 14.3
10.22 20.48 31.27 -3.7
10.222 18.48 31.27 -0.7
10.224 15.48 27.27 13.3
10.226 30.48 34.27 -11.7
10.228 16.48 23.27 -15.7
10.288 28.48 23.27 5.3
10.23 26.48 23.27 -0.7
10.232 23.48 28.27 8.3
10.234 13.48 16.27 -36.7
10.236 1.48 8.27 -17.7
10.238 16.48 23.27 -24.7
10.24 -7.52 4.27 -50.7
10.242 -12.52 -2.73 -21.7
10.244 -2.52 1.27 -4.7
10.246 -6.52 -9.73 -19.7
10.248 -17.52 -16.73 -21.7
10.25 -12.52 -13.73 1.3
10.252 -22.52 -17.73 0.3
10.254 -24.52 -21.73 -20.7
10.256 -38.52 -30.73 21.3
10.258 -34.52 -24.73 10.3
10.26 -22.52 -18.73 30.3
10.262 -24.52 -24.73 21.3
10.264 -25.52 -23.73 42.3
10.266 -18.52 -16.73 15.3
10.268 -20.52 -19.73 17.3
10.27 -29.52 -26.73 23.3
10.272 -21.52 -18.73 26.3
10.274 -3.52 5.27 30.3
10.276 4.48 7.27 34.3
10.278 5.48 8.27 45.3
10.28 7.48 7.27 44.3
10.282 26.48 24.27 22.3
10.284 17.48 16.27 9.3
10.286 23.48 24.27 5.3
10.29 33.48 36.27 -9.7
10.292 28.48 35.27 -0.7
10.294 31.48 30.27 1.3
10.296 19.48 32.27 -4.7
10.298 18.48 20.27 -12.7
10.3 7.48 15.27 -14.7
10.302 8.48 9.27 -25.7
10.304 10.48 8.27 -24.7
10.306 12.48 13.27 -25.7
10.308 8.48 16.27 -39.7
10.31 4.48 14.27 -34.7
10.312 17.48 11.27 -23.7
10.314 1.48 16.27 -23.7
10.316 -2.52 -6.73 -28.7
10.318 -9.52 -1.73 -19.7
10.32 -13.52 -3.73 -28.7
10.322 -17.52 -14.73 -11.7
10.324 -19.52 -11.73 -11.7
10.326 -19.52 -16.73 -14.7
10.328 -13.52 -15.73 -4.7
10.33 -11.52 -13.73 17.3
10.332 -19.52 -19.73 13.3
10.334 -14.52 -15.73 18.3
10.336 -20.52 -18.73 23.3
10.338 -25.52 -22.73 10.3
10.34 -24.52 -18.73 10.3
10.34 -24.52 -18.73 10.3
10.342 -21.52 -19.73 -3.7
10.344 -34.52 -29.73 16.3
10.346 -21.52 -22.73 33.3
10.348 0.48 -3.73 39.3
10.35 -1.52 -1.73 30.3
10.352 4.48 3.27 19.3
10.354 8.48 12.27 32.3
10.356 5.48 13.27 17.3
10.358 15.48 17.27 -0.7
10.36 13.48 16.27 2.3
10.362 18.48 14.27 23.3
10.364 24.48 21.27 -0.7
10.366 28.48 31.27 -4.7
10.368 20.48 23.27 5.3
10.37 36.48 30.27 -13.7
10.372 20.48 29.27 -6.7
10.374 19.48 22.27 -34.7
10.376 11.48 12.27 -14.7
10.378 23.48 22.27 -17.7
10.38 13.48 9.27 -22.7
10.382 8.48 5.27 -34.7
10.384 15.48 9.27 -23.7
10.386 5.48 1.27 -40.7
10.388 -3.52 -3.73 -38.7
10.39 -13.52 -13.73 -31.7
10.392 0.48 -0.73 -21.7
10.394 -2.52 -3.73 -24.7
10.396 -14.52 -17.73 -29.7
10.398 -12.52 -18.73 -6.7
10.4 -10.52 -15.73 -14.7
10.402 -30.52 -15.73 -8.7
10.404 -28.52 -15.73 5.3
10.406 -24.52 -15.73 11.3
10.408 -29.52 -15.73 0.3
10.41 -26.52 -15.73 21.3
10.412 -28.52 -15.73 40.3
10.414 -25.52 -15.73 36.3
10.416 -12.52 -15.73 24.3
10.418 -20.52 -15.73 1.3
10.42 -25.52 -15.73 9.3
10.422 4.48 -15.73 8.3
10.424 -1.52 -15.73 -4.7
10.426 -6.52 -15.73 21.3
10.428 21.48 -15.73 9.3
10.43 24.48 -15.73 22.3
10.432 20.48 -15.73 26.3
10.434 12.48 -15.73 5.3
10.436 20.48 -15.73 3.3
10.438 28.48 -15.73 -0.7
10.44 32.48 -15.73 -4.7
10.442 27.48 -15.73 -0.7
10.444 36.48 -15.73 -9.7
10.446 43.48 -15.73 -23.7
10.448 26.48 -15.73 1.3
10.45 24.48 -15.73 -21.7
10.452 17.48 -15.73 -13.7
10.454 16.48 -15.73 -39.7
10.456 1.48 -15.73 -15.7
10.458 11.48 -15.73 -23.7
10.46 19.48 -15.73 -19.7
10.462 7.48 -15.73 -36.7
10.464 10.48 -15.73 -42.7
10.466 2.48 -15.73 -22.7
10.468 -3.52 -15.73 -58.7
10.47 -12.52 -15.73 -42.7
10.472 -10.52 -15.73 -29.7
10.474 -21.52 -15.73 -16.7
10.476 -25.52 -15.73 -11.7
10.478 -32.52 -15.73 9.3
10.48 -21.52 -15.73 11.3
10.482 -17.52 -15.73 2.3
10.484 -21.52 -15.73 13.3
10.486 -22.52 -15.73 6.3
10.488 -11.52 -15.73 19.3
10.49 -9.52 -15.73 14.3
10.492 -29.52 -15.73 10.3
10.494 -17.52 -15.73 15.3
10.496 -4.52 -15.73 30.3
10.498 -6.52 -15.73 18.3
10.5 -6.52 -15.73 22.3
10.502 17.48 -15.73 10.3
10.504 18.48 -15.73 19.3
10.506 9.48 -15.73 -9.7
10.508 18.48 -15.73 11.3
10.51 16.48 -15.73 -1.7
10.512 28.48 -15.73 -23.7
10.514 17.48 -15.73 -8.7
10.516 25.48 -15.73 -11.7
10.518 25.48 -15.73 -15.7
10.52 31.48 -15.73 -13.7
10.522 15.48 -15.73 -15.7
10.524 11.48 -15.73 -22.7
10.526 21.48 -15.73 -15.7
10.528 17.48 -15.73 -22.7
10.53 10.48 -15.73 -23.7
10.532 9.48 -15.73 -29.7
10.534 17.48 -15.73 -30.7
10.536 0.48 -15.73 -39.7
10.538 -4.52 -15.73 -38.7
10.54 0.48 -15.73 -24.7
10.542 -10.52 -15.73 -34.7
10.544 -7.52 -15.73 -31.7
10.546 -12.52 -18.73 9.3
10.548 -10.52 -18.73 5.3
10.55 -21.52 -27.73 -16.7
10.552 -30.52 -33.73 1.3
10.554 -34.52 -34.73 -6.7
10.556 -24.52 -23.73 -20.7
10.558 -28.52 -31.73 9.3
10.56 -34.52 -36.73 8.3
10.562 -25.52 -30.73 23.3
10.564 -12.52 -22.73 42.3
10.566 -22.52 -29.73 38.3
10.568 -29.52 -31.73 28.3
10.57 -2.52 -5.73 36.3
10.572 -13.52 -11.73 25.3
10.574 -9.52 -10.73 16.3
10.576 -9.52 -11.73 1.3
10.578 -2.52 -5.73 13.3
10.58 28.48 17.27 10.3
10.582 17.48 7.27 26.3
10.584 0.48 -6.73 14.3
10.586 8.48 6.27 1.3
10.588 15.48 14.27 -5.7
10.59 9.48 12.27 5.3
10.592 7.48 7.27 6.3
10.594 32.48 28.27 -3.7
10.596 25.48 20.27 -27.7
10.598 23.48 17.27 -12.7
10.6 24.48 20.27 -21.7
10.602 25.48 20.27 -39.7
10.604 2.48 3.27 -19.7
10.606 10.48 8.27 -31.7
10.608 2.48 3.27 -17.7
10.61 -3.52 -13.73 -24.7
10.612 -6.52 -10.73 -7.7
10.614 3.48 -1.73 -6.7
10.616 -2.52 -8.73 -23.7
10.618 11.48 6.27 -36.7
10.62 -1.52 -3.73 -29.7
10.622 -9.52 -7.73 -35.7
10.624 -5.52 -3.73 -21.7
10.626 -27.52 -26.73 -27.7
10.628 -29.52 -33.73 -17.7
10.63 -24.52 -27.73 -13.7
10.632 -21.52 -26.73 8.3
10.634 -32.52 -33.73 1.3
10.636 -41.52 -38.73 25.3
10.638 -28.52 -27.73 9.3
10.64 -29.52 -25.73 11.3
10.642 -28.52 -24.73 23.3
10.644 -22.52 -22.73 23.3
10.646 1.48 -6.73 16.3
10.648 -6.52 -13.73 48.3
10.65 11.48 6.27 28.3
10.652 15.48 9.27 16.3
10.654 15.48 12.27 5.3
10.656 12.48 17.27 18.3
10.658 21.48 23.27 7.3
10.66 15.48 12.27 8.3
10.662 18.48 15.27 8.3
10.664 18.48 12.27 -7.7
10.666 20.48 15.27 -15.7
10.668 16.48 12.27 -20.7
10.67 9.48 12.27 -22.7
10.672 10.48 13.27 -34.7
10.674 8.48 9.27 -22.7
10.676 9.48 6.27 -35.7
10.678 -1.52 -5.73 -23.7
10.68 3.48 0.27 -7.7
10.682 3.48 -1.73 -23.7
10.684 -9.52 -13.73 -39.7
10.686 -11.52 -10.73 -24.7
10.688 -9.52 -8.73 -24.7
10.69 -4.52 0.27 -28.7
10.692 -26.52 -23.73 -20.7
10.694 -20.52 -21.73 10.3
10.696 -24.52 -29.73 9.3
10.698 -26.52 -34.73 18.3
10.7 -41.52 -44.73 22.3
10.702 -38.52 -38.73 17.3
10.704 -37.52 -34.73 -0.7
10.706 -38.52 -39.73 5.3
10.708 -38.52 -33.73 13.3
10.71 -19.52 -24.73 26.3
10.712 -15.52 -24.73 33.3
10.714 -20.52 -25.73 48.3
10.716 -20.52 -31.73 68.3
10.718 -15.52 -18.73 46.3
10.72 -9.52 -13.73 22.3
10.722 -10.52 -9.73 38.3
10.724 9.48 11.27 39.3
10.726 16.48 15.27 22.3
10.728 24.48 16.27 15.3
10.73 27.48 23.27 21.3
10.732 27.48 21.27 6.3
10.734 27.48 23.27 -5.7
10.736 13.48 16.27 -8.7
10.738 18.48 17.27 -3.7
10.74 16.48 16.27 -11.7
10.742 16.48 14.27 -24.7
10.744 4.48 5.27 -7.7
10.746 18.48 14.27 -19.7
10.748 25.48 20.27 -19.7
10.75 13.48 11.27 -16.7
10.752 1.48 -0.73 -29.7
10.754 9.48 9.27 -29.7
10.756 11.48 9.27 -45.7
10.758 -15.52 -11.73 -30.7
10.76 1.48 1.27 -13.7
10.762 0.48 -3.73 -8.7
10.764 -9.52 -13.73 -29.7
10.766 -18.52 -22.73 -21.7
10.768 -17.52 -14.73 5.3
10.77 -12.52 -13.73 -8.7
10.772 -20.52 -16.73 -5.7
10.774 -22.52 -19.73 2.3
10.776 -34.52 -30.73 -3.7
10.778 -19.52 -21.73 8.3
10.78 -27.52 -32.73 13.3
10.782 -32.52 -34.73 9.3
10.784 -32.52 -29.73 38.3
10.786 -21.52 -21.73 33.3
10.788 -32.52 -26.73 10.3
10.79 -37.52 -33.73 41.3
10.792 -21.52 -18.73 22.3
10.794 -18.52 -21.73 23.3
10.796 -5.52 -6.73 19.3
10.798 -2.52 -3.73 23.3
10.8 11.48 12.27 9.3
10.802 18.48 17.27 28.3
10.804 9.48 14.27 25.3
10.806 23.48 26.27 10.3
10.808 15.48 22.27 2.3
10.81 18.48 17.27 7.3
10.812 11.48 13.27 11.3
10.814 31.48 27.27 5.3
10.816 26.48 27.27 8.3
10.818 21.48 21.27 -9.7
10.82 11.48 15.27 10.3
10.822 26.48 29.27 -31.7
10.824 10.48 16.27 -36.7
10.826 0.48 1.27 -13.7
10.828 8.48 5.27 -13.7
10.83 12.48 5.27 -3.7
10.832 1.48 -2.73 5.3
10.834 1.48 -0.73 -999
10.836 -4.52 -5.73 -999
10.838 -20.52 -13.73 -999
10.84 -13.52 -8.73 -999
10.842 -27.52 -22.73 -999
10.844 -22.52 -21.73 -999
10.846 -18.52 -23.73 -999
10.848 -26.52 -26.73 -999
10.85 -24.52 -29.73 21.3
10.852 -33.52 -32.73 18.3
10.854 -30.52 -33.73 25.3
10.856 -35.52 -32.73 28.3
10.858 -33.52 -26.73 36.3
10.86 -21.52 -22.73 38.3
10.862 -22.52 -24.73 34.3
10.864 -11.52 -14.73 39.3
10.866 -18.52 -19.73 39.3
10.868 -15.52 -15.73 14.3
10.87 -4.52 -1.73 19.3
10.872 -2.52 5.27 21.3
10.874 2.48 7.27 23.3
10.876 12.48 11.27 17.3
10.878 24.48 24.27 46.3
10.88 9.48 8.27 39.3
10.882 31.48 30.27 15.3
10.884 26.48 24.27 13.3
10.886 20.48 21.27 6.3
10.888 21.48 26.27 2.3
10.89 33.48 40.27 -20.7
10.892 20.48 22.27 0.3
10.894 19.48 19.27 -16.7
10.896 24.48 23.27 0.3
10.898 25.48 21.27 42.3
10.9 16.48 13.27 -13.7
10.902 9.48 14.27 -8.7
10.904 9.48 9.27 -20.7
10.906 10.48 13.27 -17.7
10.908 11.48 9.27 -13.7
10.91 4.48 6.27 -29.7
10.912 9.48 6.27 -19.7
10.914 12.48 9.27 -24.7
10.916 13.48 8.27 -25.7
10.918 -3.52 -8.73 -22.7
10.92 -5.52 -3.73 -11.7
10.922 -5.52 -2.73 -14.7
10.924 -13.52 -11.73 -19.7
10.926 -12.52 -11.73 -8.7
10.928 -20.52 -21.73 -14.7
10.93 -19.52 -21.73 1.3
10.932 -17.52 -21.73 3.3
10.934 -21.52 -22.73 30.3
10.936 -11.52 -11.73 32.3
10.938 -12.52 -11.73 25.3
10.94 -27.52 -18.73 31.3
10.942 -24.52 -18.73 30.3
10.944 -21.52 -18.73 19.3
10.946 -15.52 -15.73 24.3
10.948 3.48 0.27 39.3
10.95 13.48 14.27 34.3
10.952 21.48 21.27 9.3
10.954 20.48 24.27 14.3
10.956 34.48 37.27 15.3
10.958 28.48 30.27 -6.7
10.96 19.48 22.27 -8.7
10.962 27.48 28.27 8.3
10.964 33.48 27.27 9.3
10.966 18.48 17.27 2.3
10.968 20.48 20.27 8.3
10.97 15.48 16.27 -13.7
10.972 19.48 21.27 -21.7
10.974 8.48 13.27 -13.7
10.976 11.48 14.27 -29.7
10.978 18.48 16.27 -11.7
10.98 2.48 -5.73 -9.7
10.982 4.48 1.27 -17.7
10.984 -3.52 -8.73 -13.7
10.986 -3.52 -5.73 -28.7
10.988 -10.52 -10.73 -20.7
10.99 -26.52 -22.73 -28.7
10.992 -12.52 -11.73 -25.7
10.994 -11.52 -10.73 2.3
10.996 -12.52 -16.73 -4.7
10.998 -19.52 -23.73 19.3
11 -14.52 -19.73 18.3
set(gcf, 'Position', [100, 100, 2000, 600]); % setting the size of the plots
plot(data.Time,[data.probe1,data.probe2,data.probe3]);
legend('probe 1', 'probe 2', 'probe 3');
The MHKiT-Matlab QC module utilizes structures. WE need to convert the table of data into a structure.
datast.values = table2array(data(:,2:4));
datast
datast =
time: [500×1 double]
values: [500×3 double]
Quality control tests
The following quality control tests are used to identify timestamp issues, corrupt data, data outside expected range, and stagnant data.
Each quality control tests results in the following information in a results structure:
- Cleaned data that has NaN in place of data that did not pass the quality control test
- Boolean mask with True/False that indicates if each data point passed the quality control test
Check timestamp
Quality control analysis generally starts by checking the timestamp index of the data.
The follow test checks to see if 1) the data contains duplicate timestamps, 2) timestamps are not monotonically increasing, and 3) timestamps occur at irregular intervals (an interval of 0.002s is expected for this data).
If duplicate timestamps are found, the resulting structure keeps the first occurrence. If timestamps are not monotonic, the timestamps in the resulting structure are reordered.
% define expected frequency of the data, in seconds
% run the timestamp quality control test
results = check_timestamp(datast,frequency);
The cleaned data and boolean mask are shown below.
disp(results)
values: [501×3 double]
mask: [501×3 int64]
time: [1×501 datetime]
plot(results.time,results.values);
legend('probe 1', 'probe 2', 'probe 3');
Check for corrupt data
In the following qualtiy control tests, the cleaned data from the previous test are used as input to the subsequent test. For each quality control test, a plot of the cleaned data is shown.
The quality control test below checks for corrupt data, indicated by a value of -999.
% run the corrupt data QC test
results = check_corrupt(results,corrupt_values);
plot(results.time,results.values);
legend('probe 1', 'probe 2', 'probe 3');
Check for data ourside expected range
The next quality control test checks for data that is greater than 50 or less than -50. Note that expected range tests can also be used to compare measured values to a model, or analyze the expected relationships between data columns.
% define expected lower and upper bound
expected_bounds = [-50,50];
% run expected range QC test
results = check_range(results,expected_bounds);
plot(results.time,results.values);
legend('probe 1', 'probe 2', 'probe 3');
Check for stagnant data
The final quality control test checks for stagnant data by looking for data that changes by less than 0.001 within a 0.02 second moving window.
% define the lower bound (no upper bound is specified in this example)
expected_bound = {0.001,py.None};
% Define the moving window, in seconds
% input the data, boundaries, py.None indicates apply to all columns, and window
results = check_delta(results,expected_bound,window);
plot(results.time,results.values);
legend('probe 1', 'probe 2', 'probe 3');
