Specdal Pipeline Script¶
Specdal provides a command line script specdal_pipeline
for batch
processing of spectral data files in a directory. A typical input to
specdal_pipeline
is a directory containing spectral files
(i.e. .asd files), which will be converted into .csv files and figures
of spectra. User can provide arguments to customize the processing
operations (i.e. jump correction, groupby) and output (i.e. .csv file
of group means). This page describes the usage and provides examples.
Usage¶
usage: specdal_pipeline [-h] [--proximal_reference PATH] [-o PATH]
[-op PREFIX] [-of] [-od] [-oi] [-i {slinear,cubic}]
[-is SPC] [-s {mean,median,min,max}] [-j {additive}]
[-js WVL [WVL ...]] [-jr REF] [-g] [-gs S]
[-gi [I [I ...]]] [-gmean] [-gmedian] [-gstd]
[-fstd wl0 wl1 n_std [wl0 wl1 n_std ...]]
[-fthresh wl0 wl1 LO HI [wl0 wl1 LO HI ...]] [-fwhite]
[-fg method] [-fo set] [-yl ymin ymax] [-q] [-f]
INPUT_PATH
Command Line Arguments¶
- positional arguments:
INPUT_PATH
directory containing input files- optional arguments:
-h, --help
show this help message and exit
--proximal_reference PATH
- directory containing proximal reference spectral files
-o PATH, --output_dir PATH
- directory to store the csv files and figures
-op PREFIX, --prefix PREFIX
- option to specify prefix for output dataset files
-of, --omit_figures
option to omit output png figures
-od, --omit_data
option to omit output csv files
-oi, --omit_individual
- option to omit output of individual csv file for each spectrum file
-i {slinear,cubic}, --interpolate {slinear,cubic}
- specify the interpolation method. method descriptions can be found on scipy docs: https://docs.scipy.org/doc/scipy-0.19.1/reference/generated/scipy.interpolate.interp1d.html
-is SPC, --interpolate_spacing SPC
- specify desired spacing for interpolation in nanometers
-s {mean,median,min,max}, --stitch {mean,median,min,max}
- specify overlap stitching method; not necessary if data at detector edges does not overlap
-j {additive}, --jump_correct {additive}
- specify jump correction method;
-js WVL [WVL ...], --jump_correct_splices WVL [WVL ...]
- wavelengths of jump locations
-jr REF, --jump_correct_reference REF
- specify the reference detector (e.g. VNIR is 1, SWIR1 is 2)
-g, --group_by
create groups using filenames``
-gs S, --group_by_separator S
- specify filename separator character to define groups
-gi [I [I ...]], --group_by_indices [I [I ...]]
- specify the indices of the split filenames to define a group
-gmean, --group_mean
calculate group means and append to group figures
-gmedian, --group_median
- calculate group median and append to group figures
-gstd, --group_std
calculate group standard deviation and append to group figures
-fstd wl0 wl1 n_std [wl0 wl1 n_std ...], --filter_std wl0 wl1 n_std [wl0 wl1 n_std ...]
- Remove spectra from dataset with a pct_reflect over n_std away from the mean between wavelengths wl0 and wl1. Can specify multiple sets of wavenumber ranges and thresholds
-fthresh wl0 wl1 LO HI [wl0 wl1 LO HI ...], --filter_threshold wl0 wl1 LO HI [wl0 wl1 LO HI ...]
- Remove spectra from the dataset with a pct_reflect outside (LO,HI)in the wavenumber range wl0 wl1. Can specify multiple sets of wavenumber ranges and thresholds
-fwhite, --filter_white
- Remove white reference spectra from dataset
-fg method, --filter_group method
- How to combine the wavelengths selected by –filter_group.
-fo set, --filter_on set
- What subset of the data to apply filter on (collection, group or both)
-yl ymin ymax, --ylim ymin ymax
- Force the y axis of plots to display between ymin and ymax
-q, --quiet
-f, --force
if output path exists, remove previous output and run
Examples¶
For a description of all command line arguments: specdal_pipeline --help
.
To produce an individual plot and textfile for every spectrum file
in directory /path/to/spectra/
and store the results in specdal_output/
:
specdal_pipeline -o specdal_output /path/to/spectra/
To only output whole-dataset images and files:
specdal_pipeline -oi -o specdal_output /path/to/spectra/
To only output images, with no data files:
specdal_pipeline -od -o specdal_output /path/to/spectra/
To group input files by the first 3 underscore-separated components
of their filename (such that foo_bar_baz_001.asd
and
foo_bar_baz_002.asd
will appear in one group, and
foo_bar_qux_001.asd
in another):
specdal_pipeline -g -gi 0 1 2 -- /path/to/spectra/
To also output the mean and median of every group of spectra:
specdal_pipeline -g -gi 0 1 2 -gmean -gmedian /path/to/spectra/
To remove all white reference spectra from the output dataset (leaves input files intact):
specdal_pipeline --filter_white /path/to/spectra/
Filtering (WIP)¶
specdal_pipeline also provides the option to automatically filter spectra out of the dataset. This feature is not fully tested and may cause issues.
To remove all spectra with a 750-1200 nm reflectance that is greater than 1 standard deviation from the mean, or with a 500-600 nm reflectance that is greater than 2 standard devations from the mean:
specdal_pipeline --filter_std 750 1200 1 500 600 2 -- /path/to/spectra/
To perform the filtering above, and then group the remaining spectra by filename:
specdal_pipeline --filter_std 750 1200 1 500 600 2
-g -gi 0 1 2 /path/to/spectra/
To group the spectra by filename, and then perform filtering on each group:
specdal_pipeline --filter_std 750 1200 1 500 600 2
-g -gi 0 1 2 --filter_on group /path/to/spectra/