On Nov 8, 2016 there was a live session on Sitecore Data Exchange Framework with Nate Barad (Senior Director of Product Marketing) and Adam Conn (Director of Technical Product Management) on Sitecore YouTube channel. It was a great session demonstrating the flexibility of Sitecore supporting complex applications. In this case it no less than developing an ETL on Sitecore.
Here is the link to YouTube Video
partners and clients are developing integration solutions and multiple ways. This leads to inconsistencies and one-time solutions. So, DEF was created to give a consistent approach for
Here is the link to YouTube Video
What is Sitecore Data Exchange Framework (DEF)
Data Exchange Framework:
"At the highest level DEF is a Product that aims to make it as easy as possible for the people to be able to connect different systems for the purpose of sharing data." - Adam Conn, Director of Technical Product Management Sitecore.
Sitecore can take the role of both Source and Destination as per the need.
Need for DEF
- Reading, Writing, mapping data (ETL)
- Maintain flexibility of current approaches
- Minimal dependency on Sitecore at runtime
- Minimal Sitecore expertise needed to develop provider
Difference between DataProvider, DEF and xConnect
DataProvider: Was available in older versions of Sitecore. Helps create or display external data in Sitecore as Items. Lot of plumbing code to be developed for implementing each DataProvider. Mapping, reading and writing logic all handled by same component.
DEF: Dependent on DataProvider logic for retrieving the data. Has Sitecore Item based configuration model for performing the mapping logic and execution of data exchange process. Compatible with Sitecore 8.1 and above only.
xConnect: An abstraction layer for interacting with Sitecore Items. Using xConnect, developers need to know only about the external system and its schema. DEF can relay the data to xConnect to handle the creation, edit and delete actions. Not yet available from Sitecore. May release with Sitecore 8.3.
Download the below files and install in same order from dev.sitecore.net
- Data Exchange Framework 1.1
- Sitecore Provider for Data Exchange Framework 1.1
- Sitecore Provider for Data Exchange Framework 1.1 Hotfix 128342
Under System a new Data Exchange is created which has insert options to create a new Data Exchange Tenant. All the action happens under Tenants. Once a new tenant is created below items are automatically created.
- Endpoints: For defining the source and destination providers
- Data Access: For defining the value accessor sets
- Value Mapping Sets: For defining the mapping between the source and destination value accessor sets
- Pipelines: For configuring the flow of logic.
- Pipeline Batches: Object to run the synchronization
- Tenant Settings
Things to remember with DEF
By default the pipelines batches are not enabled. This helps prevent accidentally running the batch in design stage. Verify the below before running the pipeline batch for success
- Tenant is enabled
- Pipeline batch is enabled - Will be eligible for enable only if the Tenant is enabled
- Pipeline should be enabled
- Ensure the sort order of the pipeline processors
- Promotes a component design
- Reuse of pipelines, endpoints, value accessor and mapping sets
- Easy to accomdate any changes in Source and Destination schema changes
- User interface validation for datatypes and fields
- Not an intuitive process. So, need developers for defining process, mapping and value sets
- No drag-and-drop UI for creating source and destination mappings
- Need to know the source and destination schema first
- Lot of manual item creation for simple process also
In conclusion, DEF is a powerful tool which give the development team the ability to perform ETL activities without much coding. Tool being in it's early stages and no UI similar to other BI/ETL tools may have less adoption. For projects with requirements to connect to only single system for data exchange this can become a handy tool but enterprise projects that have multiple integrations will incline towards a commercial ETL tool like Informatica, Scribe and similar.