Generic Storage Extension Solution with ElasticSearch,MongoDB and SOLR

In my last couple of blogs we have discussed on how to extend storage capability of SDL WEB 8. Today we are going to create one common solution where we can decide out of the following which one we want to use as a extended storage medium by just updating the configuration file.This will give you the flexability to select the storage medium you wanted to enable.

1. Elastic Search
2. MongoDB
3. SOLR 
4. You can also use Custom DB.

Let's discuss what is storage extension?

Storage Extension :-Is the capability of using custom storage medium to store the data.When we publish the data it goes in to the BrokerDB or on FileSystem as per the setting but we can extend this functionality by using storage extension technique.

How we do that :-We need to override methods which published and un-published(create,update and remove) component presentation to the Broker DB and need to update the cd_storage_conf.xml file to inject your custom storage extension code.

High-Level GenericStorageExtension Architecture Diagram

You can download the sample code from here

Happy Coding and Keep Sharing !!!!

Bypass Model dependency while indexing data in ElasticSearch

Based on the feedback given in SDL Web Dev Summit in Delhi,India.

This is in continuation of my previous blog where we have used ElasticSearch as storage while publishing components data is getting saved in it.

Earlier whenever we need to index any component we first need to create schema model,that creates model based dependency to bypass that we now Serialize the DCP (Dymanic Component preseatation)which is in XML format into JSON and Elastic Search .NET based NEST API has this capability to index JSON directly.

Now, we just need to allow the schema in Dynamic Component Template which we have created to index the data in ElasticSearch.

In NEST API we have used

         var json = @"{

    ""BookName"": ""Book1"",

    ""ISBN"": ""978-3-16-148410-0"",

    ""chapter"" : [

        {

            ""chapter_name"": ""Chapter1"",

            ""chapter_desc"": ""Before getting into computer programming, let us first                     understand computer programs and what they...""

        },

        {

            ""chapter_name"": ""Chapter2"",

            ""chapter_desc"": ""Today computer programs are being used in almost every field,               household, agriculture, medical, entertainment, defense..""

        },

        {

            ""chapter_name"": ""Chapter3"",

            ""chapter_desc"": ""MS Word, MS Excel, Adobe Photoshop, Internet Explorer,                     Chrome, etc., are...""

        },

        {

            ""chapter_name"": ""Chapter4"",

            ""chapter_desc"": ""Computer programs are being used to develop graphics and                   special effects in movie...""

        }

    ]

}";

var indexResponse = client.LowLevel.Index<string>("fromelasticstoweb8", "esnews", json);

And Link Resolver TBB is used to generate the Link e;g i have multimedia component attached ,when you published it will resolve the link below is the DCP XML.

<blog Title='Testing of model' Id='tcm:6-13290' xmlns:xlink='http://www.w3.org/1999/xlink' xmlns:tcm='http://www.tridion.com/ContentManager/5.0'>

  <blogTitle>Testing of model</blogTitle>

  <blogDescription>Remove model dependency</blogDescription>

  <image Title='skyline' Id='tcm:6-291' Path='/DigitalAssets/skyline.jpg' xmlns:tridion='http://www.tridion.com/ContentManager/5.0'></image>

  <publication Id='tcm:0-6-1' Title='400 Example Site'></publication></blog>

Happy coding and keep sharing !!!

Mircoservice over SDL WEB 8 Coreservice

Self Hosted REST Microservice using OWIN and ASP.NET Web API 2

Open Web Interface for .NET (OWIN) defines an abstraction between .NET web servers and web applications. OWIN decouples the web application from the server, which makes OWIN ideal for self-hosting a web application in your own process, outside of IIS.

ASP.NET Web API - CORS Support in ASP.NET Web API 2. Cross-origin resource sharing (CORS) is a World Wide Web Consortium (W3C) specification (commonly considered part of HTML5) that lets JavaScript overcome the same-origin policy security restriction imposed by browsers.

For more details on how to setup and create Microservice read it here.

This service is used to get the data from SDL WEB 8 CM Database

Methods which are available 

1. GetComponentByTcmUri
• http://127.0.0.1:8080/Coreservice/getComponentByTcm/{tcmuri}
2. GetSchemaByTcmUri
• http://127.0.0.1:8080/Coreservice/getSchemaByTcm/{tcmuri}
3. GetAllCategoriesWithInPubByTcmUri
• http://127.0.0.1:8080/Coreservice/GetAllCategoriesWithInPubByTcmUri/{tcmuri}
4. GetKeywordByCategoryID
• http://127.0.0.1:8080/Coreservice/GetKeywordByCategory/{tcmuri}
5. GetPageTempletByPubID
• http://127.0.0.1:8080/Coreservice/GetPageTempletByPubID/{tcmuri}
6. GetComponentTemplateByPubID
• http://127.0.0.1:8080/Coreservice/GetComponentTemplateByPubID/{tcmuri}
7. GetTemplateBuildingBlockByPubID
• http://127.0.0.1:8080/Coreservice/GetTemplateBuildingBlockByPubID/{tcmuri}
8. GetPageByPubID
• http://127.0.0.1:8080/Coreservice/GetPageByPubID/{tcmuri}
9. GetStructureGroupByPubID
• http://127.0.0.1:8080/Coreservice/GetStructureGroupByPubID/{tcmuri}
10. GetMultimediaComponentByPubID
• http://127.0.0.1:8080/Coreservice/GetMultimediaComponentByPubID/{tcmuri}
11. GetPublicationList
• http://127.0.0.1:8080/Coreservice/GetPublicationList
12. GetUserList 
• http://127.0.0.1:8080/Coreservice/GetUserList

You can download code from here.

Happy Codeing and keep Sharing !!!1

ElasticSearch Module in DXA

In my last two blogs we have discussed how to create components by consuming the data from Elasticsearch and then how to index data in Elasticsearch when publishing components.

Today,We are going to discuss and create Elasticsearch module in DXA which will have following features .DXA Modules allows us to extend its Functionality via creating new modules. Here we can create new modules which gives us flexibility & re-usability.

Features and Advantages :- 

• Free text search .
• Tag based filtering.
• Lighting fast response powered by Elasticsearch.
• Pagination and sorting based on fields.
• Article detail page.

In order to use the Elasticsearch module we need to setup the storage extension and Elasticsearch which we already discussed in my previous blog here.
Once that is done using Elasticsearch module in DXA we can search the content from Elasticsearch and render on DXA based application .

Let's discuss the CMS part first.

Elasticsearch Module structure 

1. Define module's folder structure
2. DXA gives us predefined structure where we can keep all the building blocks of the module.
3. For each new DXA module, we need to create a new folder in Modules Folder
4. The name of folder is usually module name.

Next is creating the Schema and Component Templates. I have create dynamic components and index them in Elasticsearch and render on the DXA based web application.

1. Publish all the dynamic components and index them in Elastic.
2. Create new area in you DXA web Application.

   

   DXA Module

3. We can call Elastic RESTFul API to get the data.
4. Data will be returned in the form of JSON.
5. Create model to deserialize the JSON.
6. Let's search for the text SDL.
   
7. Result can be further filtered based on tagging as well.
8. Compare the search result with the data index in Elastic

   

   Elastic Search RESTFul API

.Happy Coding and Keep Sharing !!!

ElasticSearch Integration 4 WEB 8

In my previous post we have discussed on how to create components by consuming data from Elastic Search.Today we are going to discuss how to index data in ElasticSearch when publishing components from WEB8.

To know more about the ElasticSearch advantages and how to setup please click here.

Steps to setup Storage extension and Generic Index service which will read data from and push in to the Elastic search.

1. CMS setup
• Copy and paste the templating building block (TBB) to a location on your SDL WEB 8 CM Server
• Upload ESI4TIndexing.Templating.dll TBB to WEB 8 CMS using TcmUploadAssembly.exe
• Create a Component Template with following attributes
• Output Format – XML Fragment
• Add GetComponentAsXML TBB ,Publish Binaries in Package, Link Resolver and Cleanup Template.
2. Setup ElasticSearch
• Download ES from here and unzip.
• Run bin/elasticsearch (or bin\elasticsearch.bat on Windows).
• Run http://localhost:9200.
• Default Elasticsearch instance is up and running now.
• With the help of Elasticsearch .NET high level client ,we can do all the CRUD operations in Elasticsearch.
3. Setup Deployer service
• Open the cd_storage_config.xml Storage Configuration file from the /bin/config folder and add following node under the Storages section:
• <StorageBindings><Bundle src="CustomStorageDAOBundles.xml"/></StorageBindings>
• Copy and paste CustomStorageConfig.xml file to change the value of following nodes
• ServiceEndPoint - URL of the IndexService
• TemplateIdToIndex - Tcm Id Of component Template which we have created in step 1 CMS setup. 
• Copy the CustomStorageDAOBundles.xml XML file in the Content Delivery /bin/config folder
4. ElasticSearch Index Service
• Copy and paste ElasticSearch IndexService on your server host it in IIS 
• I have used Strongly typed interface to Elasticsearch. Fluent and classic object initializer mappings of requests and responses. Uses and exposes Elasticsearch.Net
• Run Install-Package NEST
• Copy the configuration folder as well .
• You can update the log files path from Logging.config inside the configuration folder
• Update the path of configuration\logging.config file in web.config of index service
5. Here, I have created a default index  fromelasticstoweb8 (should be in all lower case other-wise you will get the error Invalid request) and Type esnews.
6. Once the data is indexed in the ES .
7. REST service will return data in JSON format.
• To get all the records http://localhost:9200/fromelasticstoweb8/_search?q=*:*

To test the index and search services

1. Index service


1. You can use fiddler for debugging 
2. I have created a sample schema article
1. title
2. description
3. imageurl
3. Run index service on fiddler
• http://localhost/Service1.svc/AddDocument
• Input JSON which will generated by custom storage 
• {"ServicePayload":{"DCP":"<esnews Title='About SDL WEB 8' Id='tcm:6-12273' xmlns:xlink='http://www.w3.org/1999/xlink' xmlns:tcm='http://www.tridion.com/ContentManager/5.0'><Title>About SDL WEB 8</Title><Description>SDL recently released the latest version of their web content management SDL WEB 8 an all-in-one global Web Experience Management solution</Description><shortTitle>About SDL WEB 8</shortTitle><authorName>SDL</authorName><Category>WEB 8</Category><relasedDate>2017-02-03T23:44:08.261+05:30</relasedDate><imageUrl>Images/SDL.png</imageUrl><publication Id='tcm:0-6-1' Title='400 Example Site'></publication></esnews>","LanguageInRequest":"en"}}Method Name: AddDocument
4. Execute this request in Fidler and verify the results returned
5. Check in ElasticSearchas well inside the collection\document which you have created and entered in the index service 
6. Result as 0 for success and 1 for failure

•  {"ResponseContext":{"EnvironmentContext":null,"FaultCollection":[]},"ServicePayload":{"ErrorMessage":"","Result":0}}

7. Test this by publishing the Component as well 
8. Log File Where Response Result :0 means successfully published and data is index in ElasticSearch.
2. Search Service
• To get all data http://localhost:9200/fromelasticstoweb8/_search?q=*:*
• To get the data based on the title http://localhost:9200/fromelasticstoweb8/esnews/AVo8K0bDJVbTUFCj_OQW
  
3. Here fromelasticstoweb8 is the node and esnews is type.

You can download the code here

Happy coding and keep Sharing !!!!

Connect SDL WEB 8 with Elasticsearch

What is ElasticSearch ?

Elasticsearch is an Apache Lucene-based search server. It was developed by Shay Banon and published in 2010.

Elasticsearch is a real-time distributed and open source full-text search and analytics engine. It is accessible from RESTful web service interface and uses schema less JSON (JavaScript Object Notation) documents to store data. It is built on Java programming language, which enables Elasticsearch to run on different platforms. It enables users to explore very large amount of data at very high speed.

Features of Elasticsearch are as follows:- 

1. Elasticsearch is scalable up to petabytes of structured and unstructured data.
2. Elasticsearch is open source and available under the Apache license version 2.0.
3. Elasticsearch uses denormalization to improve the search performance.
4. Elasticsearch can be used as a replacement of document stores like MongoDB.
5. Elasticsearch is one of the popular enterprise search engines, which is currently being used by many big organizations like Wikipedia, The Guardian, StackOverflow, GitHub etc.

key concepts of Elasticsearch:-

Node:- It refers to a single running instance of Elasticsearch. Single physical and virtual server accommodates multiple nodes depending upon the capabilities of their physical resources like RAM, storage and processing power.

Cluster:- It is a collection of one or more nodes. Cluster provides collective indexing and search capabilities across all the nodes for entire data.

Index:- It is a collection of different type of documents and document properties. Index also uses the concept of shards to improve the performance. For example, a set of document contains data of a social networking application.

Type/Mapping:- It is a collection of documents sharing a set of common fields present in the same index. For example, an Index contains data of a social networking application, and then there can be a specific type for user profile data, another type for messaging data and another for comments data.

Replicas:- Elasticsearch allows a user to create replicas of their indexes and shards. Replication not only helps in increasing the availability of data in case of failure, but also improves the performance of searching by carrying out a parallel search operation in these replicas.

Shard:-  Indexes are horizontally subdivided into shards. This means each shard contains all the properties of document, but contains less number of JSON objects than index. The horizontal separation makes shard an independent node, which can be store in any node. Primary shard is the original horizontal part of an index and then these primary shards are replicated into replica shards.

Advantages

Elasticsearch is developed on Java, which makes it compatible on almost every platform.

Elasticsearch is real time, in other words after one second the added document is searchable in this engine.

Elasticsearch is distributed, which makes it easy to scale and integrate in any big organization.

Creating full backups are easy by using the concept of gateway, which is present in Elasticsearch.

Now, Lets create connect  ElasticSearch  and SDL WEB 8 and  components in CMS.

1. Let's create an console based application using .NET 4.5.
2. Run the following command in the Package manager console .Install-Package NEST .Install Elasticsearch .NET high level client.
3. Download Elasticsearch click and unzip the Elasticsearch 
4. Run ../bin/elasticsearch
   
5. Run  http://localhost:9200/
   
6. Default Elasticsearch instance is up and running now .
7. With the help of Elasticsearch .NET high level client ,we can do all the CRUD operations in Elasticsearch.
8. Here, I have created a default index  fromelasticstoweb8 (should be in all lower case other-wise you will get the error Invalid resquest) and Type esnews.
9. So the data URL would be  http://localhost:9200/fromelasticstoweb8/esnews/AVoFKrUbqeqnhk6aE3Ca  the last parameter is the ID
   
10. To get all the records http://localhost:9200/fromelasticstoweb8/_search?q=*:*
    
11. Code snippet to write data in Elasticsearch node
    
12. Let read all the data from node fromelasticstoweb8
    
13. Using Coreservice we can create components in WEB 8 
1. Create coreservice client using ISessionAwareCoreService.
2. Create model Serialize that model into XML.
   
3. Using coreservice client.create method we can create the components in WEB 8.
   
4. I have created all the items available in ES node fromelasticstoweb8  into SDL WEB 8 as components .
   
14. To know more on how to use coreservice to interact with SDL WEB 8 and create CMS items such as components ,Keywords etc you can refer my previous blogs also .

Happy Coding and Keeping Sharing !!!!!

Salesƒorce Integration with SDL WEB8

Salesƒorce-CRM

A CRM system is a business tool that allows you to manage all your customers, partners, and prospects information all in one place. The Sales Cloud (Salesforce.com’s CRM system) is a cloud based CRM system.

For example, it helps:

1.       sales teams close deals faster

2.       marketing manage campaigns and track lead generation

3.       service call centres reduce the time to resolve customer complaints

SDL WEB 8-CMS

                SDL Web covers four main core functional areas:

1.       Web Content Management

2.       Experience Optimization (including personalization)

3.       Digital Media

4.       Localization

Steps to get the SF API and how to create .NET based client

1.       Go to ap2.salesforce.com and create your developer account.

2.       Click on Setup and from quick find search for API.

3.       Select the API, Salesforce’s WSDL allows you to easily integrate salesforce.com with your applications, and to build new applications that work with salesforce.com. To get started, download a WSDL file to a place accessible to your development environment. For complete documentation, sample code, and developer community, visit

4.       http://developer.salesforce.com/

5.       I used Enterprise WSDL 

a.       A strongly typed WSDL for customers who want to build an integration with their salesforce.com organization only.

SF API

Create dummy data in SF

Dummy data is created in SF

 I have created some dummy data in Lead object which we will accessing using Enterprise WSDL  API

Read Data using .NET Client and SF API

Add SF Enterprise API as service reference in your project 

1. Login to the Force.com API
2. Change the binding to the new endpoint
3. Create a new session header object and set the session id to that returned by the login
4. Queries are executed against the Force.com API using SOQL (Salesforce.com Object Query Language).

In the screenshot, we can see that the data created in SF is accessible using .Net client.

Write Data in SF using .Net Client

1. Instantiate the Object Being Created: Before we can execute a create() against the API, we must establish the type of object being created. In this case, we will create a new lead.
   
2. Establish Field Values and Object Properties
   
3. Execute Create Call and Capture Save Results
   
4. Working with the Results: The create call will obviously result in success or failure. Successful create calls will return the ID value of the resultant record created. Fortunately, the API is built to allow us to handle both scenarios within our application.
   
5. Data is successfully created
   
6. Let’s go and check in SF online. Here you can see the details which we have supplied via .NET client is saved in SF.
   

Let’s integrate SF with SDL WEB 8

1. We already have SF API and before we start the integration of SF with SDL WEB 8 we need to first create WEB 8 Core service client to get the object.
2. Core Service is WCF based webService.
3. I have already created core service client using ISessionAwareCoreService.
   
4. After this we need to read Component data using.
   
5. Deserialize the data in to the object, and send to SF API to create the records in SF.
   
6. Know we have the components List from WEB 8 , run SF API create method to create new records in SF.
   
7. I have created sample component in WEB 8 which is pushed in to SF
• Below image is CMS component which is pushed to SF.
  
• Salesforce UI here we can see the data which was pushed from WEB 8 is successfully created in SF.
• So, we are successfully able to push content from WEB 8 to SF lets work on the other way around so that we can say SF and WEB 8 are in sync.
8. To create content in WEB 8 
• Need core service client I have re-used the same core service utility.
• Folder Location where you want to create content.
• Read data from SF you can check Read data from SF section for more information
• Transform the object into XML.
• We need to call coreService create method.
  

Run this service in background as Windows service or Task Scheduler  to keep Salesƒorce and Tridion in a sync also if you want to have, create custom Salesƒorce objects, and add custom fields you can also do that, in this demo I have used the default object LEAD provided by Salesƒorce.

Prerequisites to create custom objects in SF:

A Salesƒorce account in a sandbox Professional, Enterprise, Performance, or Unlimited Edition org, or an account in a Developer org.

Happy  Coding and keep Sharing !!