Extending Content Delivery Storage in SDL WEB 8.5 - Part 3

In the last blog, we saw what happened when we publish a new component, it only calls the create method, but when we re-publish an item it first calls the remove method and then create.

Remove Method

public void remove(int publicationId, int componentId, int componentTemplateId, ComponentPresentationTypeEnum componentPresentationType)

throws StorageException

{

log.debug("Custom storage remove Method");

super.remove(publicationId, componentId, componentTemplateId, componentPresentationType);

log.debug("Custom storage remove Method :- "+componentId);

 

}

Re-Publish an item

As you can see when I re-published an item, we have remove method invoked first and then create method. Next, is un-publish an item and see how it works/ in what sequence.

Un-Publish an Item

When we publish a new item only create method is called, re-publishing will call the remove method and then create and finally in un-publish only remove method is called.   

In the last two blogs, we learned how to set up and configure the project to build storage extension and Publishing a new item and in this blog, we saw other functionalities as well and how they are invoked, in what sequence/order, with this approach we can add/update/remove DCPs in custom storage e:g SOLR, ElasticSearch, MongoDB, etc.

This data can further be used for analytics and for third-party applications.

Happy Coding and Keep Sharing !!!

Extending Content Delivery Storage in SDL WEB 8.5 - Part 2

In the previous blog, we set up the project and configured all the required config and based on that we published a DCP and saw custom logs are available which means everything is working fine.

Today we are going to read the content of DCP and will see how Storage Extension work when we publish, re-publish and un-publish an item.

The Content is available in the form of Bytes and we need to convert that, below is the code snippet that will allow you to access the DCP content in your custom code.

public void create(ComponentPresentation itemToCreate, ComponentPresentationTypeEnum componentPresentationType)

throws StorageException

{

          super.create(itemToCreate,componentPresentationType);

          log.debug("Custom Code Create Method COMID :- " + itemToCreate.getComponentId());

          log.debug("Custom Storage create Method:- "+ itemToCreate.getContent());

          log.debug(String.valueOf(itemToCreate.getComponentId()));

          byte[] dcpComponent= itemToCreate.getContent();

          try

          {

                  componentPresentation = new String(dcpComponent,"UTF-8");

                  log.debug("Custom Storage create DCP :- " + componentPresentation);

          }

          catch (UnsupportedEncodingException ex)

          {

                  log.error("Custom Storage create Unsupported " + ex);

          }

}

DCP content

Publish a new component and in the deployer log file, you will see the DCP content but what will happen when we re-publish the same component or un-publish this. How we are going to manage that in the custom code, will see in the next blog, until then.

Happy Coding and Keep Sharing !!!

Generic Storage Extension Solution with ElasticSearch,MongoDB and SOLR

In my last couple of blogs we have discussed on how to extend storage capability of SDL WEB 8. Today we are going to create one common solution where we can decide out of the following which one we want to use as a extended storage medium by just updating the configuration file.This will give you the flexability to select the storage medium you wanted to enable.

1. Elastic Search
2. MongoDB
3. SOLR 
4. You can also use Custom DB.

Let's discuss what is storage extension?

Storage Extension :-Is the capability of using custom storage medium to store the data.When we publish the data it goes in to the BrokerDB or on FileSystem as per the setting but we can extend this functionality by using storage extension technique.

How we do that :-We need to override methods which published and un-published(create,update and remove) component presentation to the Broker DB and need to update the cd_storage_conf.xml file to inject your custom storage extension code.

High-Level GenericStorageExtension Architecture Diagram

You can download the sample code from here

Happy Coding and Keep Sharing !!!!

MongoDB Integration 4 Tridion-Mi4T

Introduction

MongoDB Integration 4 Tridion-Mi4T is intent to provide the Tridion integration with MongoDB

Let's see some of the features advantages of using MongoDB

Advantages

1. Schema less : MongoDB is document database in which one collection holds different different documents. Number of fields, content and size of the document can be differ from one document to another.
2. Structure of a single object is clear
3. Deep query-ability. MongoDB supports dynamic queries on documents using a document-based query language

Why should use MongoDB

1. Document Oriented Storage : Data is stored in the form of JSON style documents
2. Index on any attribute
3. Replication & High Availability
4. Rich Queries

Where should use MongoDB?

1. Big Data
2. Content Management and Delivery
3. Mobile and Social Infrastructure
4. User Data Management

How to Set-up and Configure Mi4T

We have five different modules in the Mi4T

1. MongoDB - Setup and configuration
2. Template Bulding Block:- C# TBB is used to get the component DCP in XMLformat after some changes . 
3. Custom Storage Extension – A JPAComponentPresentationDAO based custom storage extension to manipulate the dynamic component presentations
4. MongoDBIndexService- WCFRestFul Service which get invoked by custom storage extension and take DCP as input
5. MongoDBSearchService - WCFRestFul service to get the data from MongoDB and take input query in JSON format .

SETUP

Below are various setup steps

1. CMS Setup
• Copy and paste the templating building block (TBB) to a location on your Tridion CM Server 
• Upload MI4TIndexing.Templating.dll TBB to Tridion CMS 
• Create a Component Template with following attributes
• Output Format – XML Fragment
• Add GetComponentAsXML TBB ,Publish Binaries in Package, Link Resolver and Cleanup Template 
2. MongoDB Setup
• You Can download the MongoDB link
• To start MongoDB
• Create folder C:/servers and copy the bin from MongoDB installed Path
• Create data/db folders in C:/servers 
• Run >> mongod --dbpath c:\servers\data --port 27017
• click here now find out more on MongoDB Installation and setup 
3. Content Delivery Setup
• Open the cd_storage_config.xml Storage Configuration file from the /bin/config folder and add following node under the Storages section:
• <StorageBindings><Bundle src="CustomStorageDAOBundles.xml"/></StorageBindings>
• Copy and paste CustomStorageConfig.xml file to change the value of following nodes
• ServiceEndPoint - URL of the IndexService
• TemplateIdToIndex - Tcm Id Of component Template which we have created in step 1 CMS setup. 
• Copy the CustomStorageDAOBundles.xml XML file in the Content Delivery /bin/config folder
4. MongoDB Index Service
• Copy and paste MongoDBIndexService on your server host it in IIS 
• Copy the configuration folder as well .
• You can update the log files path from Logging.config inside the configuration folder
• Update the path of configuration\logging.config file in web.config of index service
5. MongoDB Search Service
• Copy and paste MongoDBIndexService on your server host it in IIS 
• Copy the configuration folder as well 
• Update the log files path in Logging.config inside the configuration folder
• Update the path of configuration\logging.config file in web.config of index service
6. Index service support Publishing ,Re-publishing and un-publishing
7. Search service will get the data in JSON format. 

To test the index and search services

1. Index service


1. You can use fiddler for debugging 
2. I have created a sample schema article
1. title
2. description
3. imageurl
3. Run index service on fiddler
• http://localhost/Service1.svc/AddDocument
• Input JSON which will generated by custom storage 
• {"ServicePayload":{"DCP":"<Content xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:xsd='http://www.w3.org/2001/XMLSchema' xmlns:xlink='http://www.w3.org/1999/xlink' xmlns:tcm='http://www.tridion.com/ContentManager/5.0' Title='Copy of Demo Of MongoDB' Id='tcm:2073-13667'><title>Demo Of MongoDB</title><description><![CDATA[Demo of component creation in Tridion using MongoDB]]></description><imageurl>/images/demo.png</imageurl><publication Id='tcm:0-2073-1' Title='03 Content Master' /></Content>","LanguageInRequest":"en"}} 
4. Execute this request in Fidler and verify the results returned
5. Check in mongoDB as well inside the collection\document which you have created and entered in the index service 
6. Result as 0 for success and 1 for failure

•  {"ResponseContext":{"EnvironmentContext":null,"FaultCollection":[]},"ServicePayload":{"ErrorMessage":"","Result":0}}

7. Test this by publishing the Component as well 
8. Log File Where Response Result :0 means successfully published and data is index in MongoDB

1. Search Service
1. You use the fiddler for debugging 
2. Run the search service on fiddler
• Url:- http://localhost/SearchSvc.svc/GetContentFromMongoDB
• {"ServicePayload":{"ContentType":"Content","Filters":[{"Key":"ItemURI","Value":"tcm:2073-13667"},{"Key":"publicationID","Value":"tcm:0-2073-1"}]}
• Execute the request in fiddler and verify the results returned
• This service will get you the result based on Filters you provide and  use //MAP/REDUCE  

•  //Map/Reduce            
                      var map =
                          "function() {" +
                          "    for (var key in this) {" +
                          "        emit(key, { count : 1 });" +
                          "    }" +
                          "}";
  
  
                      var reduce =
                          "function(key, emits) {" +
                          "    total = 0;" +
                          "    for (var i in emits) {" +
                          "        total += emits[i].count;" +
                          "    }" +
                          "    return { count : total };" +
                          "}";

• Output will be in the JSON format 
• { "_id" : ObjectId("5741539eef525465db9eb131"), "title" : "Demo Of MongoDB", "description" : "Demo of component creation in Tridion using MongoDB", "imageUrl" : "/images/demo.png", "ItemURI" : 
 
• "tcm:2073-13667", "publicationID" : "tcm:0-2073-1" }
• You can have AND,OR and NOT as a logical operators to query the data from MongoDB
• {"ServicePayload":{"ContentType":"Content","Filters":[{"Key":"ItemURI","Value":"tcm:278-13667"},{"Key":"publicationID","Value":"tcm:0-278-1"}],"MongoDatabase":"customerDatabase","Table":"article","QueryType":"OR"}
2. You can download the code here

Happy Coding & Keep Sharing !!!

MongoDB Integration 4 Tridion-Mi4T

Introduction

MongoDB Integration 4 Tridion-Mi4T is intent to provide the Tridion integration with MongoDB

Let's see some of the features advantages of using MongoDB

Advantages

1. Schema less : MongoDB is document database in which one collection holds different different documents. Number of fields, content and size of the document can be differ from one document to another.
2. Structure of a single object is clear
3. Deep query-ability. MongoDB supports dynamic queries on documents using a document-based query language

Why should use MongoDB

1. Document Oriented Storage : Data is stored in the form of JSON style documents
2. Index on any attribute
3. Replication & High Availability
4. Rich Queries

Where should use MongoDB?

1. Big Data
2. Content Management and Delivery
3. Mobile and Social Infrastructure
4. User Data Management

How to Set-up and Configure Mi4T

We have five different modules in the Mi4T

1. MongoDB - Setup and configuration
2. Template Bulding Block:- C# TBB is used to get the component DCP in XMLformat after some changes . 
3. Custom Storage Extension – A JPAComponentPresentationDAO based custom storage extension to manipulate the dynamic component presentations
4. MongoDBIndexService- WCFRestFul Service which get invoked by custom storage extension and take DCP as input
5. MongoDBSearchService - WCFRestFul service to get the data from MongoDB and take input query in JSON format .

SETUP

Below are various setup steps

1. CMS Setup
• Copy and paste the templating building block (TBB) to a location on your Tridion CM Server 
• Upload MI4TIndexing.Templating.dll TBB to Tridion CMS 
• Create a Component Template with following attributes
• Output Format – XML Fragment
• Add GetComponentAsXML TBB ,Publish Binaries in Package, Link Resolver and Cleanup Template 
2. MongoDB Setup
• You Can download the MongoDB link
• To start MongoDB
• Create folder C:/servers and copy the bin from MongoDB installed Path
• Create data/db folders in C:/servers 
• Run >> mongod --dbpath c:\servers\data --port 27017
• click here now find out more on MongoDB Installation and setup 
3. Content Delivery Setup
• Open the cd_storage_config.xml Storage Configuration file from the /bin/config folder and add following node under the Storages section:
• <StorageBindings><Bundle src="CustomStorageDAOBundles.xml"/></StorageBindings>
• Copy and paste CustomStorageConfig.xml file to change the value of following nodes
• ServiceEndPoint - URL of the IndexService
• TemplateIdToIndex - Tcm Id Of component Template which we have created in step 1 CMS setup. 
• Copy the CustomStorageDAOBundles.xml XML file in the Content Delivery /bin/config folder
4. MongoDB Index Service
• Copy and paste MongoDBIndexService on your server host it in IIS 
• Copy the configuration folder as well .
• You can update the log files path from Logging.config inside the configuration folder
• Update the path of configuration\logging.config file in web.config of index service
5. MongoDB Search Service
• Copy and paste MongoDBIndexService on your server host it in IIS 
• Copy the configuration folder as well 
• Update the log files path in Logging.config inside the configuration folder
• Update the path of configuration\logging.config file in web.config of index service
6. Index service support Publishing ,Re-publishing and un-publishing
7. Search service will get the data in JSON format. 

To test the index and search services

1. Index service


1. You can use fiddler for debugging 
2. I have created a sample schema article
1. title
2. description
3. imageurl
3. Run index service on fiddler
• http://localhost/Service1.svc/AddDocument
• Input JSON which will generated by custom storage 
• {"ServicePayload":{"DCP":"<Content xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:xsd='http://www.w3.org/2001/XMLSchema' xmlns:xlink='http://www.w3.org/1999/xlink' xmlns:tcm='http://www.tridion.com/ContentManager/5.0' Title='Copy of Demo Of MongoDB' Id='tcm:2073-13667'><title>Demo Of MongoDB</title><description><![CDATA[Demo of component creation in Tridion using MongoDB]]></description><imageurl>/images/demo.png</imageurl><publication Id='tcm:0-2073-1' Title='03 Content Master' /></Content>","LanguageInRequest":"en"}} 
4. Execute this request in Fidler and verify the results returned
5. Check in mongoDB as well inside the collection\document which you have created and entered in the index service 
6. Result as 0 for success and 1 for failure

•  {"ResponseContext":{"EnvironmentContext":null,"FaultCollection":[]},"ServicePayload":{"ErrorMessage":"","Result":0}}

7. Test this by publishing the Component as well 
8. Log File Where Response Result :0 means successfully published and data is index in MongoDB

1. Search Service
1. You use the fiddler for debugging 
2. Run the search service on fiddler
• Url:- http://localhost/SearchSvc.svc/GetContentFromMongoDB
• {"ServicePayload":{"ContentType":"Content","Filters":[{"Key":"ItemURI","Value":"tcm:2073-13667"},{"Key":"publicationID","Value":"tcm:0-2073-1"}]}
• Execute the request in fiddler and verify the results returned
• This service will get you the result based on Filters you provide and  use //MAP/REDUCE  

•  //Map/Reduce            
                      var map =
                          "function() {" +
                          "    for (var key in this) {" +
                          "        emit(key, { count : 1 });" +
                          "    }" +
                          "}";
  
  
                      var reduce =
                          "function(key, emits) {" +
                          "    total = 0;" +
                          "    for (var i in emits) {" +
                          "        total += emits[i].count;" +
                          "    }" +
                          "    return { count : total };" +
                          "}";

• Output will be in the JSON format 
• { "_id" : ObjectId("5741539eef525465db9eb131"), "title" : "Demo Of MongoDB", "description" : "Demo of component creation in Tridion using MongoDB", "imageUrl" : "/images/demo.png", "ItemURI" : 
• "tcm:2073-13667", "publicationID" : "tcm:0-2073-1" }
• You can have AND,OR and NOT as a logical operators to query the data from MongoDB
• {"ServicePayload":{"ContentType":"Content","Filters":[{"Key":"ItemURI","Value":"tcm:278-13667"},{"Key":"publicationID","Value":"tcm:0-278-1"}],"MongoDatabase":"customerDatabase","Table":"article","QueryType":"OR"}
2. You can download the code here

Happy Coding & Keep Sharing !!!