A primer on Sitecore Data Exchange Framework

On Nov 8, 2016 there was a live session on Sitecore Data Exchange Framework with Nate Barad (Senior Director of Product Marketing) and Adam Conn (Director of Technical Product Management) on Sitecore YouTube channel. It was a great session demonstrating the flexibility of Sitecore supporting complex applications. In this case it no less than developing an ETL on Sitecore.

Here is the link to YouTube Video

What is Sitecore Data Exchange Framework (DEF)


Data Exchange Framework: 

"At the highest level DEF is a Product that aims to make it as easy as possible for the people to be able to connect different systems for the purpose of sharing data." - Adam Conn, Director of Technical Product Management Sitecore.

Sitecore can take the role of both Source and Destination as per the need.

Need for DEF


partners and clients are developing integration solutions and multiple ways. This leads to inconsistencies and one-time solutions. So, DEF was created to give a consistent approach for 
  • Reading, Writing, mapping data (ETL)
  • Maintain flexibility of current approaches
  • Minimal dependency on Sitecore at runtime
  • Minimal Sitecore expertise needed to develop provider

Difference between DataProvider, DEF  and xConnect


DataProvider: Was available in older versions of Sitecore. Helps create or display external data in Sitecore as Items. Lot of plumbing code to be developed for implementing each DataProvider. Mapping, reading and writing logic all handled by same component.

DEF: Dependent on DataProvider logic for retrieving the data. Has Sitecore Item based configuration model for performing the mapping logic and execution of data exchange process. Compatible with Sitecore 8.1 and above only.

xConnect: An abstraction layer for interacting with Sitecore Items. Using xConnect, developers need to know only about the external system and its schema. DEF can relay the data to xConnect to handle the creation, edit and delete actions. Not yet available from Sitecore. May release with Sitecore 8.3.


Installation steps


Download the below files and install in same order from dev.sitecore.net
  1. Data Exchange Framework 1.1
  2. Sitecore Provider for Data Exchange Framework 1.1
  3. Sitecore Provider for Data Exchange Framework 1.1 Hotfix 128342


Installed Items


Under System a new  Data Exchange is created which has insert options to create a new Data Exchange Tenant. All the action happens under Tenants. Once a new tenant is created below items are automatically created.
  • Endpoints: For defining the source and destination providers
  • Data Access: For defining the value accessor sets
  • Value Mapping Sets: For defining the mapping between the source and destination value accessor sets
  • Pipelines: For configuring the flow of logic. 
  • Pipeline Batches: Object to run the synchronization
  • Queues
  • Tenant Settings

Items created for Data Exchange Framework

Things to remember with DEF

By default the pipelines batches are not enabled. This helps prevent accidentally running the batch in design stage. Verify the below before running the pipeline batch for success
  1. Tenant is enabled
  2. Pipeline batch is enabled - Will be eligible for enable only if the Tenant is enabled
  3. Pipeline should be enabled
  4. Ensure the sort order of the pipeline processors


Advantages


  • Promotes a component design
  • Reuse of pipelines, endpoints, value accessor and mapping sets
  • Easy to accomdate any changes in Source and Destination schema changes
  • User interface validation for datatypes and fields


Cons

  • Not an intuitive process. So, need developers for defining process, mapping and value sets
  • No drag-and-drop UI for creating source and destination mappings
  • Need to know the source and destination schema first
  • Lot of manual item creation for simple process also

In conclusion, DEF is a powerful tool which give the development team the ability to perform ETL activities without much coding. Tool being in it's early stages and no UI similar to other BI/ETL tools may have less adoption. For projects with requirements to connect to only single system for data exchange this can become a handy tool but enterprise projects that have multiple integrations will incline towards a commercial ETL tool like Informatica, Scribe and similar.

Comments

  1. Great overview of the product!

    While your conclusion is not incorrect about the single/multi-system integration, DEF has an important role to play even in multiple integration scenarios.

    If you need to integrate with multiple systems, an ETL tool like Informatica, Scribe, etc. are a great option. But you still need a connector to Sitecore. I would argue you get more value building a DEF provider for your ETL tool and doing your Sitecore integration through that rather than a custom connector that goes directly from Sitecore APIs to the ETL tool.

    The benefit is that the Sitecore integration is already written for the Sitecore part. You only need to worry about mapping the DEF API to the ETL tool. After you do that, you'll automatically get all of the value of DEF without having to do anything extra.

    Future DEF Hangouts will cover provider development. I look forward to being able to demonstrate these benefits. I hope you'll join us!

    ReplyDelete
  2. Thanks for the feedback Adam. This means a lot.
    Absolutely second your thought on "I would argue you get more value building a DEF provider for your ETL tool and doing your Sitecore integration through that rather than a custom connector that goes directly from Sitecore APIs to the ETL tool."
    This in fact stirred some conversations internally.

    A drag and drop GUI that aids in automatic item generation for Value accessor and mapper sets is something we are discussing about.

    Will look forward for future sessions to learn more.

    ReplyDelete

Post a Comment