OutScale Publishing in SDL Tridion

This was my topic in this year's SDL Tridion DX DEV India summit. We recently upgraded one of our client's system from SDL Tridion 2013 to SDL WEB 8.5, with more than 700 Websites and 200K items to publish, We need to build a system that supports the above scenarios with the high-performance rate.

So to maintain scalability we decided to use Deployer-Endpoint and Deployer-Worker rather than going with Deployer-Combined, we also used REDIS for Binary Storage and ActiveMQ for Massage queue/Notifications. Below is the Architecture Diagram.

Let understand the difference between Split Deployer and Deployer Combined

• Deployer Endpoint:- Receive the package sent by the Transport Service and index it in Binary Storage. 
• Deployer Worker:-  Process the package and saves the content in Broker DB or on the File System depending upon the configuration.  

Next, let's see how this architecture works step by step.

• Content is passed to the Deployer after a user Publishes an item in the CME.
• The Deployer Endpoint passes the Transport Package (.zip file) to the defined Binary Storage (File System or Redis Database). We are using Redis right now.
• The Deployer Endpoint also passes the item to the Queue in ActiveMQ (JMS).
• ActiveMQ triggers an event that informs the Worker Deployers that a new package has been received.
• The first available Worker Deployer picks up the job from the Queue and contacts the Binary Storage to get the respective Transport Package.
• After rendering the Transport Package the Worker Deployer passes the item to the Broker Database.
• The Deployer then gets the status of the job from the Broker Database, which is updated by the Worker Deployer responsible for that job.

Below is the screenshot of Redis Consumes the "Transport Package" and in the ActiveMQ we have "Queue/notifications" Enqueued and Dequeued.



 Happy Coding and Keep Sharing !!!

Configure Workers, ActiveMQ and Redis for Scalable Deployment in SDL WEB 8.5

This is in continuation of my previous blogs where we discussed how to install CM and publisher on a dedicated machine and then, we saw how we can implement scale out content deployment using workers, ActiveMQ and Redis. Today, we are going to see how to configure Workes, ActiveMQ and Redis.

To read more about last two blogs in this series:-

1. Scaling SDL WEB 8.5 Installing CM and Publisher on Dedicated Machines
2. Scaling Deployers in SDL WEB 8.5

In order to configure the scalable content deployment, we need Deployer(Endpoint) and Deployer-Worker.

Deployer Installation Media

Pre-requisites, we need to install ActiveMQ and Redis

1. How to install Redis please check my previous blogs.
2. Download the Apache ActiveMQ
1. Unzip the package in a suitable location and from command line execute command activemq start.
2. Open your browser and navigate to the Admin Console at http://localhost:8161

Next step is to configure Deployer-Endpoint and Deployer-Workers Microservices.

1. Both these services are required to update the deployer_conf.xml and Deployer-Workers required cd_storage_conf.xml.
2. In the Deployer-Endpoint installation media open deployer_conf.xml.
1. Here we need to update the <BinaryStorage> node and point this to Redis data store.

   

   Configure Redis in Deployer-Endpoint

2. we need to configure the <State> node. so that workers know where to update the status of a job and the endpoint knows from where to get the status of the job.

   

   Configure State

3. Next, is we need to configure ActiveMQ, we also need to make sure that all the other <Queue> entries related to FileSystem are commented out.

   

   Configure Apache ActiveMQ JMS

4. Save and close the deployer-conf.xml file and use this same file for Workers as well.
5. Next step is to configure the cd_storage_conf.xml for Deployer-Workers only.
1. Here we need to update the <Storage> node with the Broker Database details. The worker will deploy the content on this DB.
2. Update the License file path.
3. Save and close the file and we can use this same file on the second Deployer-Worker.
6. Now, we need to configure the ports for Deployer-Endpoint and Deployer-Workers.
7. Last run the installService.ps1 from the Deployer-Endpoint folder and from Deployer-Worker. You need to run the command more than once depending upon the number of Deployer-Workers you want.

   

   Deployer-Endpoint and Deployer-Workers are installed  

8. We have all the services configure and up-running.
9. The final step is to register your Deployer (Endpoint) with your Discovery Service and run the discovery-registration.jar.

Happy Coding and Keep Sharing

Scaling Deployers in SDL WEB 8.5

This is in continuation of my previous post where we discussed how to set up Publisher and CM on a dedicated machine to improve the publishing efficiency click here to Read More.

Today we are going to see how to scale out Deployers, using multiple deployers and workers.

High-Level Achrciture diagram 

Steps:

1. Content is passed to the Deployer after a user Publishes an item in the CME.
2. The Deployer Endpoint passes the Transport Package (.zip file) to the defined Binary Storage (File System or Redis Database). We are using Redis right now.
3. The Deployer Endpoint also passes the item to the Queue in ActiveMQ (JMS).
4. ActiveMQ triggers an event that informs the Worker Deployers that a new package has been received.
5. The first available Worker Deployer picks up the job from the Queue and contacts the Binary Storage to get the respective Transport Package.
6. After rendering the Transport Package the Worker Deployer passes the item to the Broker Database.
7. The Deployer then gets the status of the job from the Broker Database, which is updated by the Worker Deployer responsible for that job.

In the next blog, we'll see how to configure Workers, ActiveMQ and Redis.

Happy Coding and Keep Sharing