Company Hierarchy Example

We want to connect each employee to the service, and the service to the appropriate company with the following data model.

The code for such a transformation may look like the following:

@CVComponentConfigClass(configClass=CVComponentConfigNone.class)
public class EmployeeTransformationProcessor implements IJavaAllUpdatesTransformationProcessor {
    public EmployeeTransformationProcessor(final CVComponentConfigNone config) {
    }

    @Override
    public String getTransformationDocumentType() {
        return "employee";
    }    

    @Override
    public void process(final IJavaAllUpdatesTransformationHandler handler, 
      final IMutableTransformationDocument document) throws Exception {
        final String companyName = document.getMeta("company_name");
        final String serviceName = document.getMeta("service_name");
        if ((companyName != null) && (! companyName.isEmpty()) && (serviceName != null) && 
(! serviceName.isEmpty())) {
          final ITransformationDocument serviceDoc = addService(handler, document, serviceName, companyName);
          document.addArcTo("employee", serviceDoc.getUri());
        }
    }
    private ITransformationDocument addService(final IJavaAllUpdatesTransformationHandler handler,
      final IMutableTransformationDocument document, final String serviceName, final String companyName) {
        final IMutableTransformationDocument newDoc = handler.createDocument("service=" + serviceName + "&",
 "service");
        newDoc.addArcTo("service", "company=" + companyName + "&");
        handler.yield(newDoc);
           // Note that the yield here is required because it is a document created
           // during the Transformation phase
           return newDoc;
    }
}

...    
@Override
    public void process(final IJavaAllUpdatesTransformationHandler handler, 
       final IMutableTransformationDocument document) throws Exception {
        final String serviceName = document.getMeta("service_name");
        if ((companyName != null) && (! companyName.isEmpty()) && (serviceName != null) && 
(! serviceName.isEmpty())) {
            document.addArcTo("employee", "service=" + serviceName + "_" + companyName + "&");
        }
    }
...

Sometimes, you might want to keep the business logic within your connector, even if it is not recommended. You can do that using the com.exalead.cloudview.consolidationapi.PushAPITransformationHelpers. The sample below shows how to embed the graph modeling done by the EmployeeTransformationProcessor directly within your connector.

final PushAPI employeePushAPI = CloudviewAPIClientsFactory.newInstance(GATEWAY_URL).newPushAPI
(PUSH_API_SERVER, CONNECTOR_NAME);
final List<Document> employees = new ArrayList<Document>();

Document employee = new Document("employee=Alice&");
employee.addMeta("company_name", "3ds");
employee.addMeta("service_name", "RandD");

employees.add(employee);

employee = new Document("employee=Bob&");
employee.addMeta("company_name", "3ds");
employee.addMeta("service_name", "RandD");

employees.add(employee);

employee = new Document("employee=John&");
employee.addMeta("company_name", "3ds");
employee.addMeta("service_name", "Sales");

employees.add(employee);
final Iterator<Document> employeesIt = employees.iterator();
while (employeesIt.hasNext()) {employee = employeesIt.next();
final String serviceURI = getServiceURI(employee.getMetaContainer().getMeta("service_name").getValue());
// Create service managed document
PushAPITransformationHelpers.createDocument(employee, serviceURI, "service");

// Add arc from employee to service
PushAPITransformationHelpers.addArcTo(employee, "employee", serviceURI);

// Add arc from service to company
final String companyURI = getCompanyURI(employee.getMetaContainer().getMeta("company_name").getValue());
PushAPITransformationHelpers.addArcTo(employee, "service", serviceURI, companyURI);
employeePushAPI.addDocument(employee);
}

Now, for each company document, we want to add a nb_employees meta that counts the total number of employees, and push updated document to the Indexing Server. You can perform this kind of task during the aggregation phase.

A possible implementation could be:

@CVComponentConfigClass(configClass=CVComponentConfigNone.class)
public final class CompanyAggregationProcessor implements IJavaAllUpdatesAggregationProcessor {
    public CompanyAggregationProcessor(final CVComponentConfigNone config) {
    }
    @Override
    public String getAggregationDocumentType() {
       return "company";
    }
   
    @Override
    public void process(final IJavaAllUpdatesAggregationHandler handler, final IAggregationDocument document)
 throws Exception {
        int nbEmployees = 0;
        for (final IAggregationDocument serviceDoc : GraphMatchHelpers.getPathsEnd(handler.match(document,
 "-service"))) {
            nbEmployees += handler.match(serviceDoc, "-employee").size();
        }
        document.withMeta("nb_employees", String.valueOf(nbEmployees));
    }
}

We first retrieve all the services that belong to a given company with the following call:

handler.match(document, "-service")

This returns all the paths starting from the company document that follow the service edge in reverse order.

We know by design, and also from the match query, that such paths contain only one document, the neighbors of the company document. So GraphMatchHelpers.getPathsEnd is responsible for accessing it. The Java code for such helper method must be equal (or equivalent) to:

public static <T> List<T> getPathsEnd(final List<List<T>> paths) {
    return Lists.transform(paths, new Function<List<T>, T>() {
       @Override
       public T apply(final List<T> path) { 
           return Iterables.getLast(path);
        }
    });
}

Then for each service document:

handler.match(serviceDoc, "-employee").size()

Returns all the paths leading to a unique employee in the service. We need to get the number of paths to get the number of employees in the service. The company document is then enriched with the nb_employee meta with the variable that allowed us to sum up all the different paths that were found.

A better and simpler implementation is:

@CVComponentConfigClass(configClass=CVComponentConfigNone.class)
public final class CompanyAggregationProcessor implements IJavaAllUpdatesAggregationProcessor {
    public CompanyAggregationProcessor(final CVComponentConfigNone config) {
    }

    @Override
    public String getAggregationDocumentType() {
        return "company";
    }

    @Override
    public void process(final IJavaAllUpdatesAggregationHandler handler, final IAggregationDocument document)
 throws Exception {
        int nbEmployees = handler.match(document, "-service.-employee").size()
        document.withMeta("nb_employees", String.valueOf(nbEmployees));
    }
}

The graph matching expression language allows us to specify an arbitrary long path, with various quantifiers (see Appendix - Matching Expressions Grammar). We can therefore reach the employee documents from the company document directly, with the expression:

handler.match(document, "-service.-employee")

We could also want, as a refinement, to push the number of employees present in each service. Writing the following code would then be enough:

@CVComponentConfigClass(configClass=CVComponentConfigNone.class)
public final class ServiceAggregationProcessor implements IJavaAllUpdatesAggregationProcessor {
        public ServiceAggregationProcessor(final CVComponentConfigNone config) {
    }

    @Override
    public String getAggregationDocumentType() {
        return "service";
    }

    @Override
    public void process(final IJavaAllUpdatesAggregationHandler handler, final IAggregationDocument document)
 throws Exception {
        document.withMeta("nb_employees", String.valueOf(handler.match(document, "-employee").size()));
    }

}

@CVComponentConfigClass(configClass=CVComponentConfigNone.class)
public final class CompanyAggregationProcessor implements IJavaAllUpdatesAggregationProcessor {
        public CompanyAggregationProcessor(final CVComponentConfigNone config) {
    }

    @Override
    public String getAggregationDocumentType() {
        return "company";
    }
    @Override
    public void process(final IJavaAllUpdatesAggregationHandler handler, final IAggregationDocument document)
 throws Exception {
        int nbEmployees = 0;
        for (final IAggregationDocument serviceDoc : GraphMatchHelpers.getPathsEnd(handler.match(document,
 "-service"))) {
           final String nbServiceEmployees = serviceDoc.getMeta("nb_employees");
           if ((nbServiceEmployees != null) && (! nbServiceEmployees.isEmpty()) {
               nbEmployees += Integer.valueOf(nbServiceEmployees);
           }
        }
        document.withMeta("nb_employees", String.valueOf(nbEmployees));
    }
}

The code above collects all service documents, and for each of them, sums up the values coming from its nb_employees meta.

This code works because even if we sum up the services meta values while more documents are still arriving, the impact detection ensures that the processor for these specific company documents is re-evaluated.

What may prevent this code from working properly is that the data visible during the Aggregation phase comes from the data pushed to the Consolidation store only, and nothing more! In other words, all document modifications and newly created custom documents that occur during the Aggregation phase are not visible to one another. So the company meta does not have any visibility on the new nb_employees meta created during the aggregation phase by the service processor.