Implementing the Connector

To manage the connector’s configuration, you must first extend the ConnectorConfig class. Your class must follow the recommendations described in Top level configuration class(es).

A basic filesystem connector needs to know where to start crawling for files. By specifying a root folder, the connector will recursively crawl all the files located in this folder.

See the DemoFileSystemConnectorConfig sample code:

package com.exalead.papi.framework.connectors.example.filesystem;

import com.exalead.papi.framework.connectors.ConnectorConfig;
import com.exalead.config.bean.IsMandatory;
import com.exalead.config.bean.PropertyLabel;

import java.io.File;
/**
  * The configuration class for the filesystem connector.
  */
public class DemoFileSystemConnectorConfig extends ConnectorConfig {
     private File startingFolder = null;
     @IsMandatory(false)
     @PropertyLabel("My starting folder")
     public void setStartingFolder(final File startingFolder) {
         this.startingFolder = startingFolder;
     }
     public File getStartingFolder() {
         return startingFolder;
     }
     /**
      * Optional method listing all option names as they will be displayed in the UI
      **/
     public static String[] getMethods() {
         return new String[] { "StartingFolder" };
     }
}

If your connector requires an encrypted password to connect to the source (for example, a database), then you must define a Password property in the configuration. You add a setter method as follows.

...import com.exalead.config.security.Crypter;

public class MyDataBaseConf extends ConnectorConfig{
    private String password;

   // setter to define Password property
    @BeeKeyValueType("encrypted")
    public void setPassword(final String password) {
      this.password = Crypter.getInstance().decrypt(password);
    }
...

To implement a Java connector, you must extend the Connector class. The constructor of your class must have your ConnectorConfig-derived class as sole parameter.

The second class is your ConnectorConfig, in this case, BasicFilesystemConnectorConfig

public class BasicFilesystemConnector extends Connector {
    private final BasicFilesystemConnectorConfig config;
    public BasicFilesystemConnector(final BasicFilesystemConnectorConfig config)
         throws Exception {
        super(config);
        this.config = config;
    }

The following methods must be implemented:

Method	Description
public void scan(PushAPI papi, String mode, Object modeConfig) throws Exception;	This method is used to scan/synchronize all documents. The provided PushAPI object is to be used for synchronization operations. The optional mode parameter, and its optional configuration object, can be used in specialized scan cases defined with the @ConnectorCapabilities annotation. You can also set the @ConnectorCapabilities annotation to get a continuous scan. See Implement a continuous scan.
public Document fetch(String uri) throws Exception	This method is used to retrieve a document from the source, for example, to create a thumbnail or when a user clicks on a search result. uri is the URI of the indexed document.
public MetaContainer getDocumentSecurityTokens(String uri);	This method is used to retrieve the security tokens of a document. This method is not used during indexing but when Exalead CloudView needs to retrieve security tokens at real time. uri is the URI of the indexed document. When you push documents, you must add a security meta: document.addMeta("security", "mytoken");

You can use the following helper to self-abort a synchronization in progress:

/**
     * Helper which can be called by the connector code to self-abort.
     * @param reason the reason why the abort was issued ; may be null
*/
     @Override
     public final void selfAbortScan(String reason);

You can also use the following triggers, which are called upon aborted scan, suspended scan, or resumed scan, to execute additional operations:

/**
            * Called by the framework when scan abort is requested.
            * This method must not change the state of the connector, only wake up some calls
            * which might be blocking.
            * The scan will only be considered as aborted when the scan() method has returned.
        */
     @Override
     public void onScanAborted() throws Exception;

/**
            * Called by the framework when scan suspend is requested.
            * This method must not change the state of the connector, only wake up some calls
            * which might be blocking.
            * It is then the responsibility of the connector to set the status of the connector
            * to SUSPENDED when effectively taken into account.
        */
     @Override
     public void onScanSuspended() throws Exception;

/**
            * Called by the framework when scan resume is requested.
            * This method must not change the state of the connector, only wake up some calls
            * which might be blocking.
            * It is then the responsibility of the connector to set the status of the connector
            * to WORKING when effectively taken into account.
        */
     @Override
     public void onScanResumed() throws Exception;

You should also call the following helper regularly inside long worker loops (long enumeration of documents) to be able to exit gracefully if an abort has been requested by the user, or by the internal framework:

/**
            * Checks the current connector status and, if an abort command was
            * sent, throws an ConnectorAbortingException exception.
            *
            * This function is a helper which can be called by connector. It is not
            * called by the framework.
        **/
     @Override
     public void checkAbortingOperation() throws ConnectorAbortingException;

See the sample Java code for a basic filesystem connector (DemoFileSystemConnector.java) located in <INSTALLDIR>\sdk\cloudview-sdk-java-connectors\samples\fsbasic.

Scan modes describe how connectors index documents. By default, a connector has one scan mode called full which starts when you click the Scan button in the Administration Console. Additional scan modes can be developed and provided with the connector. They are started when you select CONNECTOR NAME > Operation > More actions and click their corresponding Run buttons.

Scan modes are described by a ScanModeDefinition. A ScanModeDefinition contains a workflow which describes how a connector will process the indexing. The workflow can be:

• a scan-based indexing (Wokflow.SCAN_BASED)

• or a permanent scan (Workflow.PERMANENT_WORK).

A scan-based indexing exits when the scan is done. The connector starts, scans and indexes documents in the scan() method, and quits. This method is either called periodically to index new documents automatically, or manually by clicking the Scan button.

A permanent-work indexing (also called continuous scan) does not exit when a first scan is done. The connector loops forever in the scan() method, indexing new documents permanently. This method is started automatically by Exalead CloudView just after the connector's initialization, so that when Exalead CloudView starts, your connector starts running immediately. When a connector is set to permanent-work mode, we recommend implementing an "abort" command, to let users click the Abort scan button, when they want the connector to terminate its job and exit.

The following code sample show how to implement the continuous scan.

@ConnectorCapabilities(
    scanModes = {
        @ScanModeDefinition(
            name = "full",
            workflow = ConnectorCapabilities.Workflow.PERMANENT_WORK)
    }
)public class ContinuousScanConnector extends Connector implements CVComponent {
    public ContinuousScanConnector(ContinuousScanConfig config) throws Exception {
        super(config);
         }

         @Override
    public void scan(final PushAPI papi, final String scanMode, final Object scanModeConfig)
            throws Exception {

                  while( true ) {
            try {
                // do the job, index documents
                // ...
                // check for an abort
                if( getStatus(scanMode) == ConnectorStatus.ABORTING ) {
                    // log the abort and quit
                    break;
                }
            }
            catch(final Exception e) {
                // log the exception
                // handle the problem
            }
            finally {
                // clean everything
                // be ready for a next run
            }
        }
   }
}

By default, the connector framework is set to launch scans one after the other. You cannot run several scan operations at the same time mainly for thread safety.

For example, let’s say that your connector is scheduled to launch full scan operations on a regular basis (for example, every 5 minutes) and that you want to run another scan operation once a week to update the index regarding what was deleted in the source content. The two scan operations may conflict at a given time and the second operation may not even be triggered.

To tackle this issue, you can set your connector behavior to allow concurrent scan modes using the ConnectorCapabilities#isReentrant() annotation property set to false.

You can write a config check class to validate the configuration of your connector. This class must implement the CVComponentConfigCheck interface and override the check() method. In this method, you should check whether all parameters of your configuration contain valid values.

If a parameter contains an invalid value, you should throw a ConfigurationException with a message describing the problem. The check() method is either called when you click the Check config button in the connector Configuration tab, or when you click the Apply button. Throwing a ConfigurationException will stop the validation process and you can't apply an invalid configuration.

The check method may also be called by buildgct when Exalead CloudView is not started.

Let's suppose that your connector is in the MyConnector class and its configuration in the MyConnectorConfig class.

public class MyConnectorConfigCheck implements CVComponentConfigCheck<MyConnectorConfig> {
 @Override
 public void check(final MyConnectorConfig config, final boolean useNow) throws ConfigurationException, 
Exception {

 // use the MyParam getter
 final int myParam = config.getMyParam();

 // let's check the parameter value
 if( myParam < 0) {
 final ConfigurationException e = new ConfigurationException("Invalid MyParam value (must be >= 0)");
 e.setConfigKey("MyParam"); throw e;
  }
 }
}

And you should annotate your connector with a @CVComponentConfigClass.

@CVComponentConfigClass(configClass = MyConnectorConfig.class,
    configCheckClass = MyConnectorConfigCheck.class)public class MyConnector extends Connector {}

When a connector runs many things can happen. Knowing which exceptions where met by your connector is necessary to fix issues. This is why it is better to avoid creating your own loggers using either the standard java logger or the Log4J Logger.getLogger().

We recommend using the Connector.getLogger() method, or even better, the Connector.getLogger(String suffix) method. These methods create loggers with the connector instance name. This allows you to differentiate multiple instances of the same connector. Use the method with the suffix argument when you have multiple classes.

public class MyConnectorClass extends Connector implements CVComponent {
    Logger thisClassLogger = getLogger();
    MyConnectorSubClass sub = new MyConnectorSubClass(getLogger("MyConnectorSubClass"));
    // ...
}
public class MyConnectorSubClass {
     public MyConnectorSubClass(Logger logger) {
          // ...
     }
}

When performing tasks within other tasks, you can use com.exalead.log4ng.Log4NGContext which will append pushed contexts before each logging message in the same order they were pushed. This context stack is different for each thread.

For example:

Log4NGContext.push("State1");
getLogger().error("FIRST LOGGER");
Log4NGContext.push("State2");
getLogger("myLogger").error("SecondLogger");
Log4NGContext.pop();

Thread t = new Thread(new Runnable() {

 @Override
 public void run() {
 Log4NGContext.push("Thread");
 getLogger("myLogger2").error("ThrirdLogger");
 }
});
t.start();
t.join();
getLogger().error("FIRST LOGGER AGAIN");

Log4NGContext.pop();
getLogger().error("FIRST LOGGER ONE LAST TIME");

Would print something like:

State1: FIRST LOGGER
State1: State2: SecondLogger
Thread: ThrirdLogger
State1: FIRST LOGGER AGAIN
FIRST LOGGER ONE LAST TIME

To update the connector scan status, that is to say the number of documents pushed, deleted and scanned, you must call the following methods during the scan:

getState(“scan mode”).incPushed();

getState(“scan mode”).incDeleted();

getState(“scan mode”).incScanned();

Where “scan mode” is the name of the scan passed to the scan method.