Working with Export API

1 Goal
2 Prerequisite
3 How is Export made behind the scenes?
- 3.1 What happens in the Hawksearch project?
- 3.2 What happens in the Hawksearch.SDK project?
4 Setting a Schedule Export
5 How to integrate the Hawksearch.SDK with any CMS?
- 5.1 Creating SubmitDocument

Goal

In this article, you will find information about working with the Exporter, detailed information about the Exporter API.

Prerequisite

Installed .NET SDK NuGet package

How is Export made behind the scenes?

What happens in the Hawksearch project?

In the Hawksearch Admin Panel (your-website-url/Sitefinity/Administration/hawksearchconfig), there are 2 buttons: “Run Full Export Now“ and “Run Delta Export Now“.

When one of the buttons is hit the “hawk/export/run” is called. This is a Http Post request where we pass as а parameter an integer number depending on which kind of export is being triggered. This number is later resolved from the ExportType enumerator, where:

FullExport = 0
DeltaExport = 1

[HttpPost]
[Route("run")]
public JsonResult<OperationStatus> Run([FromBody] int exportType)
{
    var exportModel = new ExportModel();
    var result = exportModel.Run(exportType);

    return this.Json(result);
}

The ExportModel is being initialized and the Run method is being called. There we call the Export method that is part of the Exporter.cs in the Hawksearch.SDK.

In the Export method, a List<string> with the content type names is being passed. These content types are resolved from the GetItems() and GetDeltaItems() methods that are part of the DataLoader class. There the string values (the names of the content types) are resolved and the proper records are returned in form of SubmitDocuments.

Resolving the types happens in the DataLoader class, part of the Hawksearch project the following way: first it is checked if this type is Hierarchical Item, Role Item, FlatTaxonomy Item, Page Node, or none of these. Next, the corresponding method is called and the Sitefinity managers are being used to return the records. These records are passed to the Export method in the SDK for further processing.

In the ExportModel there is also the GetExportDetails() method, where the information about the Full and Delta export is being resolved. After the Export, an object ExportResult is being returned to the ExportModel and this object provides the whole details about the export.

What happens in the Hawksearch.SDK project?

The further steps that are related to the export are part of the SDK project that is CMS independent, but still a .NET SDK.

There we have a class Exporter.cs that implements the IExport interface.

The IExport has one method called Export. This method takes as a parameter an object of type ExportInfo and returns an object of type ExportResult.

ExportInfo is a class that gives the whole needed information for an Export:

· List of the content types we want to export. Those are all types (dynamic and static) we have selected in the Hawksearch Administration. It’s a List<string> with the content type names.

· The ExportType – enumerator, giving information about what kind of export we want to run.

· The information that we have given as configurations, like the destination of our files (LogDestination), our host (Host), is the Elastic version enabled (IsEnabled) etc.

ExportResult is the return type and provides info about the passed Export. It contains the following properties:

String Message: this message appears under the two run buttons and shows a user-friendly message about the export process.
ExportStatus Status: this is an enum value, with 3 options Running, Failed, Succeeded. Provides information about the status of the export.
ExportType ExportType: enumerator, giving information about what kind of export we want to run.
String DeltaExportTime: at what time was the delta export done
String FullExportTime: the time of the full export
String FullExportDuration: the time it took to complete the full export process.
String DelaExportDuration: the time it took to complete the delta export process.

The entry point in the export functionality is the Export method, that is implemented in the Exporter.cs:

public ExportResult Export(ExportInfo exportInformation)
        {
            var itemsSb = new StringBuilder();
            var contentSb = new StringBuilder();
            var attributesSb = new StringBuilder();
            var hierarchySb = new StringBuilder();
            var exportResult = new ExportResult();

            DestinationPath = exportInformation.DestinationPath;

            Stopwatch sw = new Stopwatch();
            sw.Start();

            foreach (var contentType in exportInformation.ContentTypes)
            {
                var items = new List<SubmitDocument>();

                switch (exportInformation.ExportType)
                {
                    case ExportType.Full:
                        items = dataLoader.GetItems(contentType);
                        exportResult = this.DoFullExport(items, exportInformation, ref itemsSb, ref contentSb, ref attributesSb, ref hierarchySb);
                        break;
                    case ExportType.Delta:
                        items = dataLoader.GetDeltaItems(contentType);
                        exportResult = this.DoDeltaExport(items, exportInformation, ref itemsSb, ref contentSb, ref attributesSb, ref hierarchySb);
                        break;
                }
                if (exportResult.Status == ExportStatus.Failed)
                {
                    break;
                }
            }
            sw.Stop();

            if (exportInformation.ExportType == ExportType.Full)
            {
                exportResult.FullExportDuration = string.Format("{0}:{1}:{2}:{3}", sw.Elapsed.Days.ToString(), sw.Elapsed.Hours.ToString(), sw.Elapsed.Minutes.ToString(), sw.Elapsed.Seconds.ToString());
            } 
            else
            {
                exportResult.DeltaExportDuration = string.Format("{0}:{1}:{2}:{3}", sw.Elapsed.Days.ToString(), sw.Elapsed.Hours.ToString(), sw.Elapsed.Minutes.ToString(), sw.Elapsed.Seconds.ToString());
            }

            return exportResult;
        }

In the code above we create 4 StringBuilders that are going to hold the information that will be written in the txt files. Then we start counting the time that the export will need. We make a loop, through all content types that we want to export (The ones selected in the Hawksearch Administration). In the loop we fetch all existing items of this content type by calling the methods GetItems or GetDeltaItems. That’s how the Exporter receives the records in form of SubmitDocuments.

The SubmitDocument class is also part of the SDK and is CMS independent. It contains a Dictionary<string, string>, called IdentityFields and List<SubmitField>, called Fields. Using the SubmitDocument is a mechanism for passing the records that are taken from Sitefinity in a way that does not depend on a particular CMS.

We pass this list of items in form of SubmitDocuments to the DoFullExport or DoDeltaExport .

In these two methods, the information about these items is being extracted and written in the corresponding file. It happens with the FileWriter and the Writer.

The Writer is implementing the interface IWriter:

using Hawksearch.SDK.Indexing;
using System.Collections.Generic;
using System.Text;

namespace Hawksearch.SDK.Export.IO
{
    public interface IWriter
    {
        void Write(ref StringBuilder itemsSb, ref StringBuilder contentSb, ref StringBuilder attributesSb, ref StringBuilder hierarchySb, List<SubmitDocument> items);
    }
}

The method Write is responsible for writing the headers in the files by calling the FileWriter method WriteHeaders and for writing each item in the proper file:

There is an AttributeField class that is extending the SubmitField. This is a mechanism for resolving which fields should be added in the attribute.txt.

After all items of all chosen content types are being exported and the information about them.

As a result a ExportResult class is being created, its fields have been set and have been returned, in order the information about the export to be visualized. This info is part of the upper Hawksearch Administration block, where the backend user can read what have been the parameters about full and delta export.

The ExportResult is returned to the Run method and base on it the information below is shown. For resolving and displaying these parameters, the GetExportDetails method is used. The method is places in the ExportModel.

You can find your files with the exported items in the folder that you’ve set in the Configurations.

Setting a Schedule Export

Setting a Schedule for Export is an important part of Export functionalities.
When Scheduling an export a HttpPost request is made.

The method Schedule is triggered. As an argument, it takes the ExportScheduler DTO. This object provides information about the kind of the Export and the time it should be executed. Depending on the type there are two other methods: the ScheduleDeltaExport and ScheduleFullExport. Both of them set the time and create a task. When the time comes the overridden method ExecuteTask is being triggered.

Below is the ExecuteTask method for the Full Export. The one for the Delta Export is analogical.

How to integrate the Hawksearch.SDK with any CMS?

The power of the Export now is that it can be used with all kind of CMS. The Export functionality is part of the SDK and does not depend on Sitefinity.

The Exporter can be accessed though the IExportr interface and the Export method. This method needs as parameter an object of type ExportInfo. As described above the ExportInfo has properties for the type of the export which is required and a list of content types, that’s also required. This list is a List<string>, that holds the names of the types selected for export. Next the ExportInfo requires some configuration information.

As soon as an ExportInfo is initialized, the Export method can be triggered. During the Export, the records for each content types should be taken. A for-each loop goes through the list with content types, that are a property of the ExportInfo and takes all records of a type. This happens by using the methods GetDeltaItems and GetItems.

These methods return a list of SubmitDocuments. One SubmitDocument holds the information about one record. The GetItemsInternal is a protected virtual method that can be overridden and a new custom logic can be implemented.

The goals of the method GetItemsInternal are to resolve the content type, that is passed as a string, to check if the site works in multisitecontext, next based on the type to get all records.

As shown above based on the type a method is called. All this methods: GetPageNodes, GetContentItems, GetRoles, GetTaxonomies are also protected virtual.

Example with the method GetPageNodes:

One of the arguments we pass is the ISite site. This is the current site that is observed. Also DateTime parameter is passed. This is an optional parameter and is important when we do a Delta export, in order to check which records are needed.

Creating SubmitDocument

The SubmitDocuments are the items that the Exporter expects and can process. These Documents contain the base information about an record. A document can be initialized with an empty constructor and different Fields can be added. An SubmitDocument can have IdentityFields and List<SubmitFields>.

The IdentityFields is Dictonary<string, string> that is used to hold identity information about the item.

The SubmitFields is a class with 2 properties: string Name and List<string> Values and is used to hold specific information about the item. An SubmitField can be added this way:

There are also the AttributeField, that’s a class that inherits from the SubmitField and is used as a flag to show that this information should be part of the attribute.txt.

AttributeField should have always a contenttypetitle (ExportConstants.ContentTypeTitle) that holds the full name of the type and a contenttypename that holds the name. This two fields are required for all records!

Using this 2 properties a SubmitDocument can be created. This SubmitDocument should serve the export documents. It should have all fields required for export. More information about the fields needed can be read here:

It is important to add as a SubmitField the name of the field as Name and a Value as List<string> { value}.

In the attributes.txt are added all AttributesFields in form of id + fieldName + fieldValue.

In the content.txt are added the id + title + relative url + imageUrl + content + action. For the content.txt should be created a SubmitDocument that has:

In the hierarchy.txt are added the id + title + parentid. For the hierarchy item should be created a SubmitDocument that has:

In the item.txt are added the unique_id + name + url_detail + image + price_retail + price_sale + price_sort + group_id + description_short + description_long + sku + sort_default + item_operation. For the content item should be created a SubmitDocument that has:

If any of the fields needed in the text document does not exist as SubmitField in the SubmitDocument, then an empty string is added.

One also important detail when creating SubmitDocument, no matter in which file the content will be written in, is the Action. This field provides information about the type of operation and is important when a Delta Export is executed, to show how the item has been changed. For action should be set one of this constants:

public const string OperationAdd = "A";
public const string OperationDelete = "D";
public const string OperationUpdate = "U";