Generic Storage Extension Solution with ElasticSearch,MongoDB and SOLR

In my last couple of blogs we have discussed on how to extend storage capability of SDL WEB 8. Today we are going to create one common solution where we can decide out of the following which one we want to use as a extended storage medium by just updating the configuration file.This will give you the flexability to select the storage medium you wanted to enable.

1. Elastic Search
2. MongoDB
3. SOLR 
4. You can also use Custom DB.

Let's discuss what is storage extension?

Storage Extension :-Is the capability of using custom storage medium to store the data.When we publish the data it goes in to the BrokerDB or on FileSystem as per the setting but we can extend this functionality by using storage extension technique.

How we do that :-We need to override methods which published and un-published(create,update and remove) component presentation to the Broker DB and need to update the cd_storage_conf.xml file to inject your custom storage extension code.

High-Level GenericStorageExtension Architecture Diagram

You can download the sample code from here

Happy Coding and Keep Sharing !!!!

MongoDB Integration 4 Tridion-Mi4T

Introduction

MongoDB Integration 4 Tridion-Mi4T is intent to provide the Tridion integration with MongoDB

Let's see some of the features advantages of using MongoDB

Advantages

1. Schema less : MongoDB is document database in which one collection holds different different documents. Number of fields, content and size of the document can be differ from one document to another.
2. Structure of a single object is clear
3. Deep query-ability. MongoDB supports dynamic queries on documents using a document-based query language

Why should use MongoDB

1. Document Oriented Storage : Data is stored in the form of JSON style documents
2. Index on any attribute
3. Replication & High Availability
4. Rich Queries

Where should use MongoDB?

1. Big Data
2. Content Management and Delivery
3. Mobile and Social Infrastructure
4. User Data Management

How to Set-up and Configure Mi4T

We have five different modules in the Mi4T

1. MongoDB - Setup and configuration
2. Template Bulding Block:- C# TBB is used to get the component DCP in XMLformat after some changes . 
3. Custom Storage Extension – A JPAComponentPresentationDAO based custom storage extension to manipulate the dynamic component presentations
4. MongoDBIndexService- WCFRestFul Service which get invoked by custom storage extension and take DCP as input
5. MongoDBSearchService - WCFRestFul service to get the data from MongoDB and take input query in JSON format .

SETUP

Below are various setup steps

1. CMS Setup
• Copy and paste the templating building block (TBB) to a location on your Tridion CM Server 
• Upload MI4TIndexing.Templating.dll TBB to Tridion CMS 
• Create a Component Template with following attributes
• Output Format – XML Fragment
• Add GetComponentAsXML TBB ,Publish Binaries in Package, Link Resolver and Cleanup Template 
2. MongoDB Setup
• You Can download the MongoDB link
• To start MongoDB
• Create folder C:/servers and copy the bin from MongoDB installed Path
• Create data/db folders in C:/servers 
• Run >> mongod --dbpath c:\servers\data --port 27017
• click here now find out more on MongoDB Installation and setup 
3. Content Delivery Setup
• Open the cd_storage_config.xml Storage Configuration file from the /bin/config folder and add following node under the Storages section:
• <StorageBindings><Bundle src="CustomStorageDAOBundles.xml"/></StorageBindings>
• Copy and paste CustomStorageConfig.xml file to change the value of following nodes
• ServiceEndPoint - URL of the IndexService
• TemplateIdToIndex - Tcm Id Of component Template which we have created in step 1 CMS setup. 
• Copy the CustomStorageDAOBundles.xml XML file in the Content Delivery /bin/config folder
4. MongoDB Index Service
• Copy and paste MongoDBIndexService on your server host it in IIS 
• Copy the configuration folder as well .
• You can update the log files path from Logging.config inside the configuration folder
• Update the path of configuration\logging.config file in web.config of index service
5. MongoDB Search Service
• Copy and paste MongoDBIndexService on your server host it in IIS 
• Copy the configuration folder as well 
• Update the log files path in Logging.config inside the configuration folder
• Update the path of configuration\logging.config file in web.config of index service
6. Index service support Publishing ,Re-publishing and un-publishing
7. Search service will get the data in JSON format. 

To test the index and search services

1. Index service


1. You can use fiddler for debugging 
2. I have created a sample schema article
1. title
2. description
3. imageurl
3. Run index service on fiddler
• http://localhost/Service1.svc/AddDocument
• Input JSON which will generated by custom storage 
• {"ServicePayload":{"DCP":"<Content xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:xsd='http://www.w3.org/2001/XMLSchema' xmlns:xlink='http://www.w3.org/1999/xlink' xmlns:tcm='http://www.tridion.com/ContentManager/5.0' Title='Copy of Demo Of MongoDB' Id='tcm:2073-13667'><title>Demo Of MongoDB</title><description><![CDATA[Demo of component creation in Tridion using MongoDB]]></description><imageurl>/images/demo.png</imageurl><publication Id='tcm:0-2073-1' Title='03 Content Master' /></Content>","LanguageInRequest":"en"}} 
4. Execute this request in Fidler and verify the results returned
5. Check in mongoDB as well inside the collection\document which you have created and entered in the index service 
6. Result as 0 for success and 1 for failure

•  {"ResponseContext":{"EnvironmentContext":null,"FaultCollection":[]},"ServicePayload":{"ErrorMessage":"","Result":0}}

7. Test this by publishing the Component as well 
8. Log File Where Response Result :0 means successfully published and data is index in MongoDB

1. Search Service
1. You use the fiddler for debugging 
2. Run the search service on fiddler
• Url:- http://localhost/SearchSvc.svc/GetContentFromMongoDB
• {"ServicePayload":{"ContentType":"Content","Filters":[{"Key":"ItemURI","Value":"tcm:2073-13667"},{"Key":"publicationID","Value":"tcm:0-2073-1"}]}
• Execute the request in fiddler and verify the results returned
• This service will get you the result based on Filters you provide and  use //MAP/REDUCE  

•  //Map/Reduce            
                      var map =
                          "function() {" +
                          "    for (var key in this) {" +
                          "        emit(key, { count : 1 });" +
                          "    }" +
                          "}";
  
  
                      var reduce =
                          "function(key, emits) {" +
                          "    total = 0;" +
                          "    for (var i in emits) {" +
                          "        total += emits[i].count;" +
                          "    }" +
                          "    return { count : total };" +
                          "}";

• Output will be in the JSON format 
• { "_id" : ObjectId("5741539eef525465db9eb131"), "title" : "Demo Of MongoDB", "description" : "Demo of component creation in Tridion using MongoDB", "imageUrl" : "/images/demo.png", "ItemURI" : 
 
• "tcm:2073-13667", "publicationID" : "tcm:0-2073-1" }
• You can have AND,OR and NOT as a logical operators to query the data from MongoDB
• {"ServicePayload":{"ContentType":"Content","Filters":[{"Key":"ItemURI","Value":"tcm:278-13667"},{"Key":"publicationID","Value":"tcm:0-278-1"}],"MongoDatabase":"customerDatabase","Table":"article","QueryType":"OR"}
2. You can download the code here

Happy Coding & Keep Sharing !!!

MongoDB Integration 4 Tridion-Mi4T

Introduction

MongoDB Integration 4 Tridion-Mi4T is intent to provide the Tridion integration with MongoDB

Let's see some of the features advantages of using MongoDB

Advantages

1. Schema less : MongoDB is document database in which one collection holds different different documents. Number of fields, content and size of the document can be differ from one document to another.
2. Structure of a single object is clear
3. Deep query-ability. MongoDB supports dynamic queries on documents using a document-based query language

Why should use MongoDB

1. Document Oriented Storage : Data is stored in the form of JSON style documents
2. Index on any attribute
3. Replication & High Availability
4. Rich Queries

Where should use MongoDB?

1. Big Data
2. Content Management and Delivery
3. Mobile and Social Infrastructure
4. User Data Management

How to Set-up and Configure Mi4T

We have five different modules in the Mi4T

1. MongoDB - Setup and configuration
2. Template Bulding Block:- C# TBB is used to get the component DCP in XMLformat after some changes . 
3. Custom Storage Extension – A JPAComponentPresentationDAO based custom storage extension to manipulate the dynamic component presentations
4. MongoDBIndexService- WCFRestFul Service which get invoked by custom storage extension and take DCP as input
5. MongoDBSearchService - WCFRestFul service to get the data from MongoDB and take input query in JSON format .

SETUP

Below are various setup steps

1. CMS Setup
• Copy and paste the templating building block (TBB) to a location on your Tridion CM Server 
• Upload MI4TIndexing.Templating.dll TBB to Tridion CMS 
• Create a Component Template with following attributes
• Output Format – XML Fragment
• Add GetComponentAsXML TBB ,Publish Binaries in Package, Link Resolver and Cleanup Template 
2. MongoDB Setup
• You Can download the MongoDB link
• To start MongoDB
• Create folder C:/servers and copy the bin from MongoDB installed Path
• Create data/db folders in C:/servers 
• Run >> mongod --dbpath c:\servers\data --port 27017
• click here now find out more on MongoDB Installation and setup 
3. Content Delivery Setup
• Open the cd_storage_config.xml Storage Configuration file from the /bin/config folder and add following node under the Storages section:
• <StorageBindings><Bundle src="CustomStorageDAOBundles.xml"/></StorageBindings>
• Copy and paste CustomStorageConfig.xml file to change the value of following nodes
• ServiceEndPoint - URL of the IndexService
• TemplateIdToIndex - Tcm Id Of component Template which we have created in step 1 CMS setup. 
• Copy the CustomStorageDAOBundles.xml XML file in the Content Delivery /bin/config folder
4. MongoDB Index Service
• Copy and paste MongoDBIndexService on your server host it in IIS 
• Copy the configuration folder as well .
• You can update the log files path from Logging.config inside the configuration folder
• Update the path of configuration\logging.config file in web.config of index service
5. MongoDB Search Service
• Copy and paste MongoDBIndexService on your server host it in IIS 
• Copy the configuration folder as well 
• Update the log files path in Logging.config inside the configuration folder
• Update the path of configuration\logging.config file in web.config of index service
6. Index service support Publishing ,Re-publishing and un-publishing
7. Search service will get the data in JSON format. 

To test the index and search services

1. Index service


1. You can use fiddler for debugging 
2. I have created a sample schema article
1. title
2. description
3. imageurl
3. Run index service on fiddler
• http://localhost/Service1.svc/AddDocument
• Input JSON which will generated by custom storage 
• {"ServicePayload":{"DCP":"<Content xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:xsd='http://www.w3.org/2001/XMLSchema' xmlns:xlink='http://www.w3.org/1999/xlink' xmlns:tcm='http://www.tridion.com/ContentManager/5.0' Title='Copy of Demo Of MongoDB' Id='tcm:2073-13667'><title>Demo Of MongoDB</title><description><![CDATA[Demo of component creation in Tridion using MongoDB]]></description><imageurl>/images/demo.png</imageurl><publication Id='tcm:0-2073-1' Title='03 Content Master' /></Content>","LanguageInRequest":"en"}} 
4. Execute this request in Fidler and verify the results returned
5. Check in mongoDB as well inside the collection\document which you have created and entered in the index service 
6. Result as 0 for success and 1 for failure

•  {"ResponseContext":{"EnvironmentContext":null,"FaultCollection":[]},"ServicePayload":{"ErrorMessage":"","Result":0}}

7. Test this by publishing the Component as well 
8. Log File Where Response Result :0 means successfully published and data is index in MongoDB

1. Search Service
1. You use the fiddler for debugging 
2. Run the search service on fiddler
• Url:- http://localhost/SearchSvc.svc/GetContentFromMongoDB
• {"ServicePayload":{"ContentType":"Content","Filters":[{"Key":"ItemURI","Value":"tcm:2073-13667"},{"Key":"publicationID","Value":"tcm:0-2073-1"}]}
• Execute the request in fiddler and verify the results returned
• This service will get you the result based on Filters you provide and  use //MAP/REDUCE  

•  //Map/Reduce            
                      var map =
                          "function() {" +
                          "    for (var key in this) {" +
                          "        emit(key, { count : 1 });" +
                          "    }" +
                          "}";
  
  
                      var reduce =
                          "function(key, emits) {" +
                          "    total = 0;" +
                          "    for (var i in emits) {" +
                          "        total += emits[i].count;" +
                          "    }" +
                          "    return { count : total };" +
                          "}";

• Output will be in the JSON format 
• { "_id" : ObjectId("5741539eef525465db9eb131"), "title" : "Demo Of MongoDB", "description" : "Demo of component creation in Tridion using MongoDB", "imageUrl" : "/images/demo.png", "ItemURI" : 
• "tcm:2073-13667", "publicationID" : "tcm:0-2073-1" }
• You can have AND,OR and NOT as a logical operators to query the data from MongoDB
• {"ServicePayload":{"ContentType":"Content","Filters":[{"Key":"ItemURI","Value":"tcm:278-13667"},{"Key":"publicationID","Value":"tcm:0-278-1"}],"MongoDatabase":"customerDatabase","Table":"article","QueryType":"OR"}
2. You can download the code here

Happy Coding & Keep Sharing !!!

Creating Components in SDL Tridion using MongoDB Data Feed

In one of my previuos post we've discussed how to create components in SDL Tridion using MSSQLServer as a DataSoucre but today we are going to create components in SDL Tridion using MongoDB as a DataSource

We are going to use SDL Tridion CoreService,MonoDB 3.2.6 ssl-signed and C# as a language

You can download MongoDB from here and to know more about MongoDB here

Create an empty solution

1. MongoDb.Client
• C# Console Based Application Running on .NET 4.5
• which will get the data from mongoDB
2. MongoDBToTridion.BAL
• In this Class Liabrary  i have written all the logic in order to create component.
3. Install MongoDB Drivers
• Install-Package MongoDB.Driver
4. Create SDL Tridion CoreService Client
• ISessionAwareCoreService
5. Create a model name Article as Serializable
6. I have also create a Database called customerDatabase in MongoDB
7. In my customerDatabase i have create document name article which we will use to create component
8. Main Method which is starting point of this application 

     

      9.Update appsettings key in app.config file 

     10. Process List<Article>

   11.Serialize article model

  12.Create or update Component in Tridion 

 Compare your Data in MongoDB with components created in Tridion

All the data which are available in Article Document in MongoDB has been successfully  created as Components in Tridion .

Open Component and check 

You can download the code from here

Happy Coding and keep Sharing !!!!