In this article you’ll find:

Goal

The goal of this article is to show you how to create additional data sources which are a part of the indexing process as well as how the system handles data source failures.

Prerequisites

Overview

You can add additional data sources for specific indexes which become a part of the indexing process. When the system begins a full indexing scheduled task, it sends a post request to each data source with a predefined API Key as well as the new Hawksearch index name so they can begin indexing as well. Those data sources are then responsible for sending status update requests back which determine if indexing fails or not based on settings. Explained in more detail below.

Adding additional data sources

1. Open the side menu application selector (F2 or button top left) and go to Custom / Hawksearch.

2. Go to Indexing Api / Data Sources.

3. Click New data source.

4. Name - choose a unique name for this data source.

5. API Key - choose a unique random API Key which will be used for this data source. This will be sent to the data source to ensure the validity of the request so we recommend choosing a random GUID.

6. Index name - choose the index for which you wish this data source to be. Predefined names coming from https://bridgeline.atlassian.net/wiki/spaces/CON/pages/3468461011 .

7. Url - the url to which the system will send a post request along with the API Key and the new Hawksearch index name.

8. Severity - you can choose between Low & Critical. There are differences in the way the system handles failure based on this value. Explained further below.

9. Timeout interval - choose a duration in minutes. This duration determines how long the system will wait after notifying the data source before it is marked at timed out.

Workflow

When a full indexing scheduled task runs, the system notifies external data sources by sending the API Key for that data source as well the new Hawksearch index name. Each data source is then responsible for sending status update requests back to the system. If a data source doesn’t respond for a period longer than Timeout interval (in minutes), it is marked as timed out. If a data source times out or has a failing status update, indexing can fail based on the severity of the data source.

There are different workflows based on the severity of the data source:

• Low Severity:

  • Success - the indexing procedure will continue as usual and the indexing will not fail.

  • Fail/Timeout - the indexing procedure will continue as usual and the indexing will not fail.

• Critical Severity:

  • Success - the indexing procedure will continue as usual and the indexing will not fail.

  • Fail/Timeout:

    • Use deferred indexing is disabled - the indexing procedure will continue as usual and the indexing will not fail.

    • Use deferred indexing is enabled - while deferred indexing is enabled, the connector will create the new index while keeping the old one so that there is no downtime while the indexes are swapped. If the Critical Severity data source fails to index or the Timeout interval is exceeded, the newly created index will be removed and the system will continue to use the old one.

Next steps