3.4.7. Batch

_images/batch_tab.jpg

batch_tool Batch

This tab allows for the automatic execution (batch) of several SCP functions using a scripting interface.

3.4.7.1. Batch

Enter a batch expression; each function must be in a new line. Functions have the following structure:

function name; function options

Each function has options, identified by a name, with the following structure:

option name: option argument

Options must be separated by the character ; . Each function option represents an option in the corresponding interface of SCP; option arguments of type text must be between the character ' ; in case of checkboxes, the value 1 represents checked, while the value 0 represents unchecked. A new line beginning with # can be used for commenting.

According to the function, some of the options are mandatory while other options can be omitted from the expression. Option names that contain path require the full path to a file.

Some options require multiple arguments such as lists; lists must be separated by , .

  • reset: clear the expression;
  • export: export the batch expression to a file;
  • import: import a previously saved batch expression from file;

A panel Functions is displayed at the right side; double click to insert a function in the expression; the following functions are available with the corresponding options:

3.4.7.1.1. Band set

  • add_new_bandset: add a new empty band set; optionally it is possible to specify the position of the added band set (e.g. band_set : 1):

    add_new_bandset;band_set : 1
    
  • add_raster: add a raster to QGIS, and optionally to an existing band set defining the wavelength:

    add_raster;input_raster_path : '';input_raster_name : '';band_set : 1;center_wavelength : 1
    
  • create_bandset: create a Band set assigning bands; it is possible to enter the satellite name as center_wavelength option (satellite name as displayed in Wavelength quick settings e.g. center_wavelength : 'sentinel-2' or center_wavelength : 'landsat 8') to set the corresponding band wavelength center automatically for all the bands; raster_path_list option accepts a directory and a file name filter separated by comma (e.g. raster_path_list : '/home/user, tif') to load all the raster inside the directory:

    create_bandset;raster_path_list : '';center_wavelength : '';wavelength_unit : 1;multiplicative_factor : '';additive_factor : '';date : ''
    
  • open_training_input: open a training input file:

    open_training_input;training_file_path : ''
    
  • remove_band_from_bandset: remove a list band from band set by the number thereof (e.g. remove_band_from_bandset;band_set : 1;band_list : '1, 2'):

    remove_band_from_bandset;band_set : 1;band_list : ''
    
  • remove_bandset: remove a band set by the number thereof:

    remove_bandset;band_set : 1
    
  • select_bandset: select a band set (active band set) by the number thereof:

    select_bandset;band_set : 1
    

3.4.7.1.2. Band calculation

  • Band calc: band calculation:

    band_calc;expression : '';output_raster_path : '';extent_same_as_raster_name : '';align : 1;extent_intersection : 1;input_nodata_as_value : 0;use_value_nodata : 0;output_nodata_value : -32768;data_type : 'Float32';scale_value : 1;offset_value : 0;band_set : 1
    

3.4.7.1.3. Preprocessing

  • ASTER: ASTER conversion:

    aster_conversion;input_raster_path : '';celsius_temperature : 0;apply_dos1 : 0;use_nodata : 1;nodata_value : 0;create_bandset : 1;output_dir : '';band_set : 1
    
  • Clip multiple rasters: clip multiple rasters:

    clip_multiple_rasters;band_set : 1;output_dir : '';use_vector : 0;vector_path : '';use_vector_field : 0;vector_field : '';ul_x : '';ul_y : '';lr_x : '';lr_y : '';nodata_value : 0;output_name_prefix : 'clip'
    
  • Cloud masking: cloud masking:

    cloud_masking;band_set : 1;input_raster_path : '';class_values : '';use_buffer : 1;size_in_pixels : 1;nodata_value : 0;output_name_prefix : 'mask';output_dir : ''
    
  • GOES: GOES conversion:

    goes_conversion;input_dir : '';use_nodata : 1;nodata_value : 0;create_bandset : 1;output_dir : '';band_set : 1
    
  • Landsat: Landsat conversion:

    landsat_conversion;input_dir : '';mtl_file_path : '';celsius_temperature : 0;apply_dos1 : 0;use_nodata : 1;nodata_value : 0;pansharpening : 0;create_bandset : 1;output_dir : '';band_set : 1
    
  • MODIS: MODIS conversion:

    modis_conversion;input_raster_path : '';reproject_wgs84 : 1;use_nodata : 1;nodata_value : -999;create_bandset : 1;output_dir : '';band_set : 1
    
  • Mosaic band sets: mosaic band sets:

    mosaic_bandsets;band_set_list : '';output_dir : '';output_name_prefix : 'mosaic'
    
  • Neighbor pixels: neighbor pixels:

    neighbor_pixels;band_set : 1;matrix_size : 1;circular : 0;matrix_file_path : '';output_name_prefix : 'neighbor';statistic : 'sum';stat_value : 50;output_dir : ''
    
  • Reproject raster bands: reproject raster bands:

    reproject_raster_bands;band_set : 1;output_dir : '';align_raster_path : '';same_extent_reference : 0;epsg : '';x_resolution : '';y_resolution : '';resample_pixel_factor : '';resampling_method : 'near';output_nodata_value : -32768;data_type : 'auto';output_name_prefix : 'reproj'
    
  • Sentinel-1: Sentinel-1 conversion:

    sentinel1_conversion;input_raster_path : '';xml_file_path : '';vh : 1;vv : 1;raster_project : 0;raster_projections_band_set : 1;convert_to_db : 1;use_nodata : 1;nodata_value : 0;create_bandset : 1;output_dir : '';band_set : 1
    
  • Sentinel-2: Sentinel-2 conversion:

    sentinel2_conversion;input_dir : '';mtd_safl1c_file_path : '';apply_dos1 : 0;preprocess_bands_1_9_10 : 0;use_nodata : 1;nodata_value : 0;create_bandset : 1;output_dir : '';band_set : 1
    
  • Sentinel-3: Sentinel-3 conversion:

    sentinel3_conversion;input_dir : '';apply_dos1 : 0;use_nodata : 1;nodata_value : 0;create_bandset : 1;output_dir : '';band_set : 1
    
  • Vector to raster: convert vector to raster:

    vector_to_raster;vector_file_path : '';use_value_field : 1;vector_field_name : '';constant_value : 1;reference_raster_path : '';type_of_conversion : 'Center of pixels';output_raster_path : ''
    
  • Split raster bands: split raster to single bands:

    split_raster_bands;input_raster_path : '';output_dir : '';output_name_prefix : 'split'
    
  • Stack raster bands: stack rasters into a single file:

    stack_raster_bands;band_set : 1;output_raster_path : ''
    

3.4.7.1.4. Band processing

  • Band combination: band combination:

    band_combination;band_set : 1;output_raster_path : ''
    
  • Classification: perform classification:

    classification;band_set : 1;use_macroclass : 0;algorithm_name  : 'Minimum Distance';use_lcs : 0;use_lcs_algorithm : 0;use_lcs_only_overlap : 0;apply_mask : 0;mask_file_path : '';vector_output : 0;classification_report : 0;save_algorithm_files : 0;output_classification_path : ''
    
  • Clustering: clustering:

    clustering;band_set : 1;clustering_method : 1;use_distance_threshold : 1;threshold_value : 0.0001;number_of_classes : 10;max_iterations : 10;isodata_max_std_dev : 0.0001;isodata_min_class_size : 10;use_nodata : 0;nodata_value : 0;seed_signatures : 1;distance_algorithm : 1;save_signatures : 0;output_raster_path : ''
    
  • PCA: Principal Component Analysis:

    pca;band_set : 1;use_number_of_components : 0;number_of_components : 2;use_nodata : 1;nodata_value : 0;output_dir : ''
    
  • Random Forest: Random Forest classification:

    random_forest;band_set : 1;use_macroclass : 1;number_training_samples : 5000;number_trees : 10;evaluate_classifier : 0;evaluate_feature_power_set : 0;min_power : 2;max_power : 7;save_classifier : 0;classifier_file_path : '';output_classification_path : ''
    
  • Spectral distance: spectral distance of band sets:

    spectral_distance;first_band_set : 1;second_band_set : 2;distance_algorithm  : 1;use_distance_threshold : 1;threshold_value : 0.1;output_raster_path : ''
    

3.4.7.1.5. Post processing

  • Accuracy: calculate accuracy:

    accuracy;classification_file_path : '';reference_file_path : '';vector_field_name : '';output_raster_path : '';use_value_nodata : 0
    
  • Class signature: class signature:

    class_signature;input_raster_path : '';band_set : 1;save_signatures : 1;output_text_path : ''
    
  • Classification dilation: dilation of a classification:

    classification_dilation;input_raster_path : '';class_values : '';size_in_pixels : 1;circular : 0;output_raster_path : ''
    
  • Classification erosion: erosion of a classification:

    classification_erosion;input_raster_path : '';class_values : '';size_in_pixels : 1;circular : 0;output_raster_path : ''
    
  • Classification report: report of a classification:

    classification_report;input_raster_path : '';use_nodata : 0;nodata_value : 0;output_report_path : ''
    
  • Classification sieve: classification sieve:

    classification_sieve;input_raster_path : '';size_threshold : 2;pixel_connection : 4;output_raster_path : ''
    
  • Classification to vector: convert classification to vector:

    classification_to_vector;input_raster_path : '';use_signature_list_code : 0;code_field : 'C_ID';dissolve_output : 0;output_vector_path : ''
    
  • Cross classification: cross classification:

    cross_classification;classification_file_path : '';use_nodata : 0;nodata_value : 0;reference_file_path : '';vector_field_name : '';output_raster_path : ''
    
  • Edit raster: edit raster values using a vector:

    edit_raster_using_vector;input_raster_path : '';input_vector_path : '';vector_field_name : '';constant_value : 0;expression :  'where(raster == 1, 2, raster)'
    
  • Land cover change: calculate land cover change:

    land_cover_change;reference_raster_path : '';new_raster_path : '';output_raster_path : ''
    
  • Reclassification: raster reclassification:

    reclassification;input_raster_path : '';value_list : 'oldVal_newVal,oldVal_newVal';use_signature_list_code : 1;code_field : 'MC_ID';output_raster_path : ''
    
  • Zonal stat raster: calculate zonal statistics related to an input raster:

    zonal_stat_raster;input_raster_path : '';reference_file_path : '';use_nodata : 0;nodata_value : 0;vector_field_name : '';statistic : 'sum';stat_value : 50;output_raster_path : ''
    

3.4.7.1.6. Variables

Working directory

  • !working_dir!: set a working directory (argument is the path to a directory):

    !working_dir!;''
    

If a working directory is defined, !working_dir! can be entered in other functions where a path is required (e.g. add_raster;input_raster_path : '!working_dir!/raster1.tif';input_raster_name : 'raster1.tif'). An example of batch expression is:

!working_dir!; '/home/user/Desktop/temp/'
add_raster;input_raster_path : '!working_dir!/raster1.tif';input_raster_name : 'raster1.tif'
band_calc;expression : 'where("raster1.tif" > 1, 1,0)';output_raster_path : '!working_dir!/calc1.tif';input_nodata_as_value : 1;use_value_nodata : 0
band_calc;expression : '"raster1.tif" * "calc1.tif"';output_raster_path : '!working_dir!/calc2.tif';extent_intersection : 0

Iterate directories

  • !for_directory_in!: function to iterate commands for all the directories in a directory path (e.g. !for_directory_in!;'directoryPath'):

    !for_directory_in!;''
    

The variables !directory_name! can be used to reference the current directory name and !directory! for the current directory path.

The iterated commands must terminate with a line !end_for_directory!; for instance the following script performs Sentinel-2 conversion of all the subdirectories of /home/user/:

!for_directory_in!;'/home/user/'
sentinel2_conversion;input_dir : '!directory!';create_bandset : 1;output_dir : '!directory!_converted'
!end_for_directory!

Optionally a filter (i.e. only directories matching the filter will be processed) can be defined for the subdirectory level; the following examples stop the iteration at level 2:

!for_directory_in!;'/home/user/';2

A name filter can be set; for instance the following script creates a band set for every subdirectory having rt in the name:

!for_directory_in!;'/home/user/';'rt'
add_new_bandset;
create_bandset;raster_path_list : '!directory!, tif';center_wavelength : 'landsat 8';
!end_for_directory!

Multiple name filters can be set separated by character | such as:

!for_directory_in!;'/home/user/';3;'name1|name2')

It is possible to filter by date if the directory name ends with a date (format yyyy-mm-dd) using a range separated by colon:

!for_directory_in!;'/home/user/';2020-01-01:2020-03-31
sentinel2_conversion;input_dir : '!directory!';create_bandset : 1;output_dir : '!directory!_converted'
!end_for_directory!
  • !directory!: variable to be used with !for_directory_in! to reference current directory path:

    !directory!
    
  • !directory_name!: variable to be used with !for_directory_in! to reference current directory name:

    !directory_name!
    
  • !end_for_directory!: ending line to be used with !for_directory_in! to close the iterative commands:

    !end_for_directory!
    

Iterate files

  • !for_file_in!: function to iterate commands for all the files in a directory path (e.g. !for_file_in!;'directoryPath'):

    !for_file_in!;''
    

The variable !file! can be used to reference the current file path.

The iterated commands must terminate with a line !end_for_file!, for instance:

!for_file_in!;'/home/user/'
create_bandset;raster_path_list : '!file!';
!end_for_file!

Optionally a filter (i.e. only directories matching the filter will be processed) can be defined for the subdirectory level; the following examples stop the iteration at level 0:

!for_file_in!;'/home/user/';0

Also, a file name filter can be defined:

!for_file_in!;'/home/user/';0;'tif'

Multiple name filters can be set separated by character | such as:

!for_file_in!;'/home/user/';0;'jp2|tif'

It is possible to filter by date if the file name ends with a date (format yyyy-mm-dd) using a range:

!for_file_in!;'/home/user/';2020-01-01:2020-03-31

Also !file_directory! can be used to refer to the parent directory of the file.

  • !file!: variable to be used with !for_file_in! to reference current file path:

    !file!
    
  • !file_directory!: variable to be used with !for_file_in! to reference the directory containing current file path:

    !file_directory!
    
  • !end_for_file!: ending line to be used with !for_file_in! to close the iterative commands:

    !end_for_file!
    

Iterate band sets

  • !for_band_set!: function to iterate commands for all the band sets in a range (e.g. !for_band_set!;bandsetNumbers):

    !for_band_set!;''
    

It is possible to enter a list of numbers (separated by commas), a range separated by colon, or a list of ranges, for instance:

!start_for_band_set!;'2:3'
add_new_bandset;band_set : !bandset_number!
!end_for_band_set!
  • !bandset_number!: variable to be used with !for_band_set! to reference band set number;
  • !end_for_band_set!: ending line to be used with !for_band_set!;

Custom variables

  • !customVariableName!: it is possible to create custom variables entering a line such as !customVariableName! = value (value can be either number or text), that will be replaced in the following parts of the batch script (the variable name cannot be any of the program Batch variables); for instance:

    !number! = 20
    !path_to_file! = /home/file.tif
    add_new_bandset;band_set : !number!
    band_combination;band_set : 1;output_raster_path : '!path_to_file!'
    

Temporary output

  • !temp_raster_1!: create a temporary raster path (saved in the temporary directory) to be used with other functions:

    !temp_raster_1!
    

!temp_raster_1! can be used as temporary output for other functions such as:

band_calc;expression : '"raster1"';output_raster_path : '!temp_raster_1!'

Additional temporary output can be entered manually, replacing 1 with any number or string (e.g. !temp_raster_2! or !temp_raster_calculation! ). The actual raster name will be the string between ! , for instance temp_raster_1 is the raster name of variable !temp_raster_1!.

3.4.7.1.7. QGIS Processing

  • qgis_processing: use a QGIS Processing algorithm entering a command (algorithm name) and parameters (without brackets) as described in the QGIS console

    qgis_processing;command : '';parameters : '';load_results : 1
    

For instance the following command executes GRASS vector dissolve:

qgis_processing;command : 'grass7:v.dissolve';parameters : 'input': '/home/user/vector.shp','column':'DN','output':'/home/user/output_vector.shp','GRASS_OUTPUT_TYPE_PARAMETER':0

It is possible to use the Batch Variables for setting input or output paths.

3.4.7.1.8. Run

  • CHECK check: check if expression syntax is correct; if expression is correct then the text color is green, red otherwise; in case of error, a label is displayed with a brief description of the error;
  • RUN run: check if expression syntax is correct and start the batch processes;