March 24, 2017
The purpose of this blog is to provide guidelines for maintaining Sitecore Experience Database (xDB) components – in particular Mongo and Solr - after a successful go-live.
The ideal scenario would be for customers with a scaled environment using Sitecore XP. The most common level architecture we see is as follows:
- Two Content Delivery (CD) instances
- Dedicated Content Management (CM) instance
- Dedicated Processing Server
- MongoDB replica set
- SOLR for search and analytics index
- SQL Server for Master + Web + Core + Reporting Databases
Below are few recommendations pertaining to MongoDB, SOLR, and Sitecore.
MongoDB
The MongoDB administration documentation addresses the ongoing operation and maintenance of MongoDB instances and deployments. This documentation includes both high level overviews of these concerns as well as tutorials that cover specific procedures and processes for operating MongoDB.
Couple of key sections are mentioned below for reference.
Operations Checklist
Reference: https://docs.mongodb.com/manual/administration/production-checklist-operations/
Configuration and Maintenance
Reference: https://docs.mongodb.com/manual/administration/configuration-and-maintenance/
Backup
Reference: https://docs.mongodb.com/manual/core/backups/
Security
Reference: https://docs.mongodb.com/manual/security/
Monitoring
MongoDB includes a number of tools out of the box. One can run these against a live MongoDB server and report stats in real time.
serverStatus
The serverStatus command returns a document that provides an overview of the database’s state. Monitoring applications can run this command at a regular interval to collect statistics about the instance. The serverStatus command, or db.serverStatus() from the shell, returns a general overview of the status of the database, detailing disk usage, memory use, connection, journaling, and index access. The command returns quickly and does not impact MongoDB performance.
See the manual for details: https://docs.mongodb.com/manual/reference/command/serverStatus/#dbcmd.serverStatus
mongostat
This shows key metrics like opcounts, memory usage and replica set status updating every second. It is useful for real time troubleshooting because you can see what is going on right now.
See the manual for details: https://docs.mongodb.com/manual/reference/program/mongostat/
mongotop
Whereas mongostat shows global server metrics, mongotop looks at the metrics on a collection level, specifically in relation to reads and writes. This helps to show where the most activity is.
See the manual for details: https://docs.mongodb.com/manual/reference/program/mongotop/
rs.status()
This shows the status of the replica set from the viewpoint of the member you execute the command on. It’s useful to see the state of members and their oplog lag.
See the manual for details: https://docs.mongodb.com/manual/reference/command/replSetGetStatus/#dbcmd.replSetGetStatus
sh.status()
This shows the status of your sharded cluster, in particular the number of chunks per shard so you can see if things are balanced or not.
See the manual for details: https://docs.mongodb.com/manual/reference/method/sh.status/
Additional Help
Custom Monitoring Solution
One could also build a custom monitoring solution using open-source monitoring software such as Nagios.
Performance
Reference: https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/
Compacting
Running a big Mongo DB installation requires a certain amount of routine maintenance. Over time, collections in a Mongo DB can become fragmented. This can be a serious problem if your data usage patterns are relatively unstructured. In the long term, this can result in your databases taking up more space on disk and in RAM to hold the same amount of data, it can make many database operations noticeably slower, and it can reduce your overall query capacity significantly.
Conveniently, MongoDB provides two different ways to compact your data and restore optimal performance:
repairDatabase
repairDatabase checks and repairs errors and inconsistencies in data storage. repairDatabase is analogous to a fsck command for file systems. Run the repairDatabase command to ensure data integrity after the system experiences an unexpected system restart or crash, if:
The mongod instance is not running with journaling enabled. When using journaling, there is almost never any need to run repairDatabase. In the event of an unclean shutdown, the server will be able to restore the data files to a pristine state automatically. There are no other intact replica set members with a complete data set.
WARNING
During normal operations, only use the repairDatabase command and wrappers including db.repairDatabase() in the mongo shell and mongod --repair, to compact database files and/or reclaim disk space. Be aware that these operations remove and do not save any corrupt data during the repair process.
If you are trying to repair a replica set member, and you have access to an intact copy of your data (e.g. a recent backup or an intact member of the replica set), you should restore from that intact copy, and not use repairDatabase.
repairDatabase takes the following form:
{ repairDatabase: 1 }
Example: db.runCommand( { repairDatabase: 1 } )
Using repairDatabase to Reclaim Disk Space
You should not use repairDatabase for data recovery unless you have no other option.
However, if you trust that there is no corruption and you have enough free space, then repairDatabase is the appropriate and the only way to reclaim disk space.
RepairDatabase is appropriate if your databases are relatively small, or you can afford to take a node out of rotation for quite a long time.
See the manual for details: https://docs.mongodb.com/manual/reference/command/repairDatabase/
compact
Rewrites and defragments all data and indexes in a collection. On WiredTiger databases, this command will release unneeded disk space to the operating system.
compact has the following form:
{ compact: <collection name> }
compact only blocks operations for the database it is currently operating on. Only use compact during scheduled maintenance periods.
See the manual for details: https://docs.mongodb.com/manual/reference/command/compact/
Additional Help
- http://highscalability.com/blog/2014/3/5/10-things-you-should-know-about-running-mongodb-at-scale.html
- https://docs.mongodb.com/manual/introduction/
Minimum Recommendation for Regular Maintenance
Based on the above, the table below states the minimum items for regular maintenance.
| Mongo DB | |
| Scheduled Back Ups | Schedule periodic tests of your backup and restore process to have time estimates on hand, and to verify its functionality |
| Security Patches | Follow guidance from https://docs.mongodb.com/manual/administration/security-checklist/ |
| Monitoring | https://docs.mongodb.com/manual/administration/monitoring/ |
| Component | One Time Basic Setup | Nightly/Weekly Maintenance | Quarterly Maintenance | Ad Hoc Maintenance |
| MONGO DB | Monitoring Scheduled Backups |
|
|
|
SOLR
The SOLR wiki suggests various techniques to keep your SOLR instance healthy. Several key sections are mentioned below for reference.
Operations and Production
SolrCaching
Built in SolrRequestHandler based SolrReplication
Unix script based CollectionDistribution
Reference: https://wiki.apache.org/solr
Monitoring
Reference: https://wiki.apache.org/solr/SolrMonitoring
Minimum Recommendation for Regular Maintenance
Based on the above, the table below states the minimum items for regular maintenance.
| SOLR | |
| Security Patches | https://wiki.apache.org/solr/SolrSecurity |
| Monitoring | https://wiki.apache.org/solr/SolrMonitoring |
| Component | One Time Basic Setup | Nightly/Weekly Maintenance | Quarterly Maintenance | Ad Hoc Maintenance |
| SOLR |
|
Scheduled Backups |
|
|
xDB Sitecore
Upgrades
Keep Sitecore up to date. We recommend you plan and budget for a Sitecore upgrade every 12 months, 18 at the most. In addition to newer versions providing new fixes and features, a more frequent upgrade schedule reduces the overall risk and cost of upgrades over the course of several years.
Patches
If you encounter what you believe to be a bug in Sitecore, you can contact Sitecore support team and provide details. Please note you need to be a Sitecore certified developer to contact Sitecore support.
Security
From time to time, there are security vulnerabilities, for which Sitecore implements patches and fixes them in next updates. For example, Sitecore suggests applying the following fixes:
- https://kb.sitecore.net/articles/496731
- https://kb.sitecore.net/articles/039942
- https://kb.sitecore.net/articles/547255
Performance
To review solution performance, Sitecore recommends the following for consideration
- https://sdn.sitecore.net/reference/sitecore%207/cms%20performance%20tuning%20guide.aspx
- https://sdn.sitecore.net/upload/sitecore6/66/cache_configuration_reference_sc66-a4.pdf
Minimum Recommendation for Regular Maintenance
Based on the above, the table below states the minimum items for regular maintenance.
| Sitecore | |
| Security Patches | Apply the required security patches for Sitecore version installed on premise |
| Upgrades | Upgrade Mongo and SOLR to the appropriate recommended versions that work in conjunction with the Sitecore version installed on premise |
| Component | One Time Basic Setup | Nightly/Weekly Maintenance | Quarterly Maintenance | Ad Hoc Maintenance |
| SITECORE | Monitor Sitecore Analytics Dashboard. |
|
|
