Custom data type is a data type used on a step property. A given step may have many # of inputs, each input is rendered using a data type. For example, if you look at the inputs for the "Decision" step:
The decision step has an input called "Condition to evaluate", this input is rendered using the "multilineTextBox" data type. So, each input must have a data type, by default most inputs use the "string" data type which renders as a text box.
See the full Step by Step guide HERE.