MAP

Configuration

The configuration of this operator is one single expression with no constraints on the output type. The output type does not have to match with the input type.

Examples

A basic case is to compute something using existing values of the element. For example, if the input elements are of type Tuple<income:Float, taxes_rate:Float>, yu can output a new Tuple with the net_income added to the income and taxes_rate:

{income:element.income, taxes_rate:element.taxes_rate, net_income:income*(1-taxes_rate)}

Note: Using the spread operator, you can write this in a shorter way:

{...element,
          net_income:income*(1-taxes_rate)}

Using more advanced features from the expression language, you can configure the MAP operator to do powerful things.

In the following example, we use the MAP operator to make a JOIN (LEFT) through a reference between a manager and its employees, and to sort employees by birth date:

{manager:manager, employees:List.sort(manager<-[corporate.Employee.manager], (employee)->employee.birth_date)}

In this example, we want to create a map for salaries.

Use the Operator to Clean Elements

You can use the MAP operator to "clean" elements by removing attributes that are not used further in the graph. However, to reduce memory usage, keep the Item type as is, as much as possible.

An Item in a stream is just a reference inside the Index data structure. When you put an Item attribute inside a Tuple, it is stored in RAM a second time (Index + processing engine).

Sub Processing Graph

The sub processing graph option defines a sub graph that will be applied on each input element. It allows you to define more complex operations.

Stateful Mode

The stateful MAP iterates over each element sequentially, and keeps previous results to process the next element. The stream order is preserved and impacts the result.

Use this operator to compute things like moving average, derivation, etc.

The pseudo code for this operator is the following:

var element;

var var1 = ...// default value

var var2 = ...// default value

var var3 = ...// default value

while((element = getNextValue()) != null && $termination_expression == false) {

  var1 = ... //update var

  var2 = ... //update var

  var3 = ... //update var

  emit($output_expression);

}

Note: As for now, the operator emits an element for every element of the input stream. If you need less elements, the best practice is to emit NULL and filter NULL values afterwards.

Table 1. Configuration
Parameter	Type	Comment
Variables declaration	List of Variable declaration	List of variables that will be available during the iteration and then in the Ouput expression.
Variables update	List of variable update	List of assignments used to update the variables. Note: You can assign a variable more than one time in the same iteration. The order matters as the last version of the variable will be available in the next assignments.
Output Expression	Expression	Expression used to build the element that will be emitted. The variables declared above are available in this expression.

Table 2. Variable Declaration
Parameter	Type	Comment
Name	String	Name of the variable.
Type	String	Type of the variable.
Default value	Expression	Default value of the variable.

Table 3. Variable Update
Parameter	Type	Comment
Name	String	Name of the variable to update.
Expression	Expression	Expression used to update the variable