diff --git a/cloudhub-2/modules/ROOT/pages/ch2-ha-dr.adoc b/cloudhub-2/modules/ROOT/pages/ch2-ha-dr.adoc index 1b066621..27492b46 100644 --- a/cloudhub-2/modules/ROOT/pages/ch2-ha-dr.adoc +++ b/cloudhub-2/modules/ROOT/pages/ch2-ha-dr.adoc @@ -1,205 +1,184 @@ = CloudHub 2.0 High Availability and Disaster Recovery -CloudHub 2.0 provides high availability (HA) and disaster recovery (DR) capabilities for applications and protection against hardware failures. +CloudHub 2.0 provides high availability (HA) and disaster recovery (DR) for applications and protects against hardware failures. -CloudHub 2.0 leverages Amazon AWS for its cloud infrastructure, so availability is directly dependent on Amazon services. CloudHub 2.0 deployments and availability are region-based and correspond to Amazon regions. If an Amazon region goes down, the applications within that region are unavailable and not automatically replicated in other regions. +For an overview of HA and DR across all Mule deployment models, and to compare CloudHub 2.0 with other options, see xref:mule-runtime::hadr-guide.adoc[High Availability and Disaster Recovery]. -In the event of a network partition between the control plane and the runtime plane, applications in the runtime plane continue to run. The runtime plane continues to locally buffer log and telemetry data until control plane availability is restored. +CloudHub 2.0 runs on Amazon AWS, so its availability depends on Amazon services. CloudHub 2.0 bases deployments and availability on regions that correspond to Amazon regions. If an Amazon region goes down, applications in that region become unavailable, and CloudHub 2.0 doesn't automatically replicate them to other regions. -CloudHub 2.0 leverages external messaging services, such as Anypoint MQ, to ensure message reliability through persistent queues. These external queues are highly available within a region, but they can‌ be inaccessible for short periods, generally seconds to minutes, during a regional outage leading to some data loss. Communication with the queues resumes after the region becomes available. For more information, see xref:mq::mq-failover.adoc[]. - -Anypoint Object Store v2 enables CloudHub 2.0 applications to store data and states across various components and applications. Object Store v2 is maintained in the same region as the deployed CloudHub 2.0 application. Data persists and becomes available after the region returns to service. However, Object Store v2 is a regional service. It doesn't provide any region failover functionality. You can't access data in Object Store v2 from an application in a different region. Implement your own external storage solution for cross-region data persistence and access. Object Store v2 doesn't provide periodic backups. +During a network partition between the control plane and the runtime plane, applications in the runtime plane keep running. The runtime plane buffers log and telemetry data locally until control plane availability returns. == High Availability Versus Disaster Recovery -High availability (HA) is the measure of a system’s ability to remain accessible despite a system component failure. You generally implement HA by building in multiple levels of fault tolerance or load balancing capabilities into a system. In CloudHub 2.0, you can achieve high availability by deploying your application with multiple replicas. +High availability (HA) measures a system's ability to remain accessible when a component fails. You implement HA by building multiple levels of fault tolerance and load balancing into a system. On CloudHub 2.0, you achieve HA by deploying your application with multiple replicas. -Disaster recovery (DR) refers to the process of restoring a system to an acceptable previous state after a natural or man-made disaster, such as flooding, tornadoes, earthquakes, fires, power failures, server failures, and misconfigurations. +Disaster recovery (DR) restores a system to an acceptable previous state after a natural or man-made disaster, such as flooding, tornadoes, earthquakes, fires, power failures, server failures, or misconfigurations. -While they both increase overall availability, the difference is that with HA there's generally no loss of service. HA retains the service, and DR retains the data, but with DR, there's usually a slight loss of service while the DR plan executes and the system restores. +Both strategies increase availability but differ in scope: on CloudHub 2.0, HA keeps your application serving requests through replica and availability-zone failures with no loss of service, whereas DR restores service and data after a region-level disruption and usually incurs a brief loss of service while the DR plan runs. -These key terms are essential to understanding HA and DR strategies in CloudHub 2.0: +Two measurable objectives drive your HA and DR strategies on CloudHub 2.0: -Recovery Time Objective (RTO):: The maximum amount of downtime a business can tolerate. The RTO represents the time it takes for the system to recover after a business disruption. +Recovery Time Objective (RTO):: The maximum downtime a business tolerates. RTO is the time the system takes to recover after a disruption. -Recovery Point Objective (RPO):: The maximum amount of time acceptable for data loss after a disaster. The RPO influences the frequency of data backups. +Recovery Point Objective (RPO):: The maximum data loss a business accepts after a disaster. RPO drives how often you back up data. == High Availability in CloudHub 2.0 -CloudHub 2.0 implements high availability through built-in mechanisms, which MuleSoft manages automatically, and require no additional configuration. +CloudHub 2.0 builds high availability into the platform through mechanisms that MuleSoft manages automatically and that require no additional configuration. + +Multiple-replica deployment:: When an application uses multiple replicas, CloudHub 2.0 deploys those replicas across two or more availability zones (AZs) by default. If an AZ fails, CloudHub 2.0 automatically restarts the application in a different AZ to maintain availability. + +=== CloudHub 2.0 Improvements Over CloudHub -Multiple-Replica Deployment:: If an application uses multiple replicas, CloudHub 2.0 deploys these replicas across two or more availability zones (AZs) by default. In case of an AZ failure, CloudHub 2.0 automatically restarts the application in a different AZ to maintain availability. +CloudHub 2.0 improves on CloudHub in these areas: + +* Decouples the runtime plane region from the control plane region through a service-oriented architecture. +* Supports clustering, which lets you configure HA while running the process on a single primary replica. +* Optimizes usage through elastic scaling with Horizontal Pod Autoscaling (HPA) for warm standby when a failover is required. +* Manages applications in active/passive mode, which lets you set schedulers and listening connectors—such as Anypoint MQ, Database, FTP, and Salesforce listeners—to a stopped or inactive state. Externalize DR flags and expose REST endpoints to enable and disable flows through a Groovy script as needed. == Disaster Recovery in CloudHub 2.0 -Disaster recovery in CloudHub 2.0 focuses on the process of restoring systems after significant disruptions. +Disaster recovery in CloudHub 2.0 focuses on restoring systems after significant disruptions. MuleSoft manages in-region infrastructure, and you own your cross-region DR strategy. === Global Distribution -You can deploy applications to CloudHub 2.0 in various global regions, including North America, South America, the European Union, and Asia-Pacific. - -If you have the Global Deployment entitlement, and are in the US Cloud or EU cloud, you can deploy applications to CloudHub 2.0 in more than one runtime plane region. +You can deploy applications to CloudHub 2.0 in global regions across North America, South America, the European Union, and Asia-Pacific. -You can host integrations in a runtime plane region that is closer to your services to reduce latency. +With the Global Deployment entitlement, and in the US Cloud or EU Cloud, you can deploy applications to more than one runtime plane region. -For a list of available runtime plane regions based on the control plane region where your organization is provisioned, see: xref:ch2-architecture.adoc#regions-and-dns-records[Runtime Plane Regions and DNS Records]. +To reduce latency, host integrations in a runtime plane region closer to your services. -The runtime plane region is the region where you deploy your CloudHub 2.0 applications and create CloudHub 2.0 private spaces. +For the runtime plane regions available for your control plane region, see xref:ch2-architecture.adoc#regions-and-dns-records[Runtime Plane Regions and DNS Records]. The runtime plane region is where you deploy your CloudHub 2.0 applications and create CloudHub 2.0 private spaces. === Regional Infrastructure You can create your private space in a region of your choice. -=== Customer Responsibility for Disaster Recovery - -If your organization has cross-region DR requirements for your applications, build your applications accordingly and consider deploying them across multiple regions. You can use a load balancer, either cloud-based or on-premises, for applications deployed to different regions to switch traffic to your backup region as part of your DR strategy. - -For more information about how to deploy for HA and DR strategies, see xref:mule-runtime::hadr-guide.adoc[High Availability and Disaster Recovery]. - -== Disaster Recovery Use Cases and Architecture for the Runtime Plane - -CloudHub 2.0 supports various multi-region disaster recovery strategies based on application impact and statefulness. These strategies typically involve a bring-your-own (BYO) global load balancer to manage traffic distribution and failover across regions. - -To decouple your runtime plane application availability from the control plane region, configure your primary and backup runtime plane regions to be different from the region where your Anypoint Platform control plane is located. This is applicable to EU, with the control plane in eu-central-1, and US, with the control plane in us-east-1. - -You can use a load balancer, either cloud-based or on-premises, for applications deployed to different regions to provide a better disaster recovery strategy. - -image::hadr-load-balancer.png["Diagram showing basic disaster recovery configuration"] - -=== Use Case 1: Active-Active Configuration - -==== When to Use - -High-impact, stateless applications that require continuous availability with minimal downtime. - -==== Application State - -* Set up the infrastructure with applications in different regions, with these applications running simultaneously and fully scaled up. +=== Your Responsibility for Disaster Recovery -==== Traffic Management +If your organization has cross-region DR requirements, build your applications accordingly and consider deploying them across multiple regions. For applications deployed to different regions, use a cloud-based or on-premises load balancer to switch traffic to your backup region as part of your DR strategy. -* Use a BYO global load balancer to distribute traffic between both regions. -* Perform health checks to detect disasters. -* Applications continue to run even if the control plane region is unavailable, as long as these applications are deployed in different regions than the control plane. +For more about how to deploy for HA and DR strategies, see xref:mule-runtime::hadr-guide.adoc[High Availability and Disaster Recovery]. -==== Licensing +=== Disaster Recovery Use Cases for the Runtime Plane -* For organizations using UBP, this configuration distributes xref:general::usage-metrics.adoc[Mule messages and Data throughput] between the two regions. It consumes Mule flows based on the number of running replicas per application deployment. -* For organizations not using UBP, this configuration consumes additional vCores. +CloudHub 2.0 supports multiple multiregion DR strategies based on application impact and statefulness. These strategies typically use a bring-your-own (BYO) global load balancer to manage traffic distribution and failover across regions. +To decouple your runtime plane application availability from the control plane region, configure your primary and backup runtime plane regions to differ from the region that hosts your Anypoint Platform control plane. This guidance applies to the EU, with the control plane in eu-central-1, and to the US, with the control plane in us-east-1. -=== Use Case 2: Warm Standby Configuration +image::hadr-load-balancer.png["A bring-your-own global load balancer routing traffic across primary and backup CloudHub 2.0 runtime plane regions"] -==== When to Use +==== Active-Active Configuration -Medium-impact stateless APIs that tolerate brief downtime during failover or stateful APIs. +*When to use:* Choose this configuration for high-impact, stateless applications that require continuous availability with minimal downtime. -==== Application State +*Application state:* Run applications in different regions simultaneously and fully scaled up. -* Set up the infrastructure across multiple regions. In the passive region, you can deploy your backup applications either in a fully scaled-up state or in a scaled-down state, with fewer replicas or smaller replica sizes. Scaled-down applications increase reliance on the control plane to redeploy and scale up during an outage in the primary runtime plane region. -* You can configure CPU-based Horizontal Pod Autoscaling (HPA) where applicable if available to your organization. For more information, see xref:ch2-configure-horizontal-autoscaling.adoc[]. +*Traffic management:* -==== Traffic Management +* Distribute traffic between both regions with a BYO global load balancer. +* Run health checks to detect disasters. +* Keep applications running even when the control plane region is unavailable, as long as you deploy them in different regions than the control plane. -* Use a BYO global load balancer to route traffic to only one region at a time. -* Perform health checks to determine DR and switch routing. -* For non-HTTP use cases, such as Schedulers, externalize a DR flag in your application and use a Groovy script to enable and disable scheduler flows directly in the runtime plane. This reduces reliance on the control plane to trigger your backup applications via a cold standby approach. +*Licensing:* -==== Licensing +* With usage-based pricing (UBP), this configuration distributes xref:general::usage-metrics.adoc[Mule messages and data throughput] between the two regions and consumes Mule flows based on the number of running replicas per application deployment. +* Without UBP, this configuration consumes additional vCores. -* For organizations using UBP, this configuration doesn’t consume messages or throughput. The configuration consumes fewer flows if you configure the application with xref:ch2-configure-horizontal-autoscaling.adoc[CPU-based HPA]. -* For organizations not using UBP, this configuration consumes lower vCores if the backup deployment is in a scaled-down state before it scales up in response to an outage in the primary region. +==== Warm Standby Configuration -==== Recovery Time Objective +*When to use:* Choose this configuration for medium-impact stateless APIs that tolerate brief downtime during failover, or for stateful APIs. -If the backup application is in a fully scaled state, you experience lower disruption in the event of an outage. If the backup application is in a scaled-down state, the RTO depends on the time it takes to scale the application up either through control-plane-triggered redeployments or through xref:ch2-configure-horizontal-autoscaling.adoc[CPU-based HPA]. +*Application state:* -For integrating systems to resend events or data of in-flight transactions, capture recovery points in logs and monitoring. This builds reliability in your applications. +* Set up infrastructure across multiple regions. In the passive region, deploy backup applications either fully scaled up or scaled down, with fewer replicas or smaller replica sizes. Scaled-down applications rely more on the control plane to redeploy and scale up during an outage in the primary runtime plane region. +* Configure CPU-based Horizontal Pod Autoscaling (HPA) where your organization supports it. See xref:ch2-configure-horizontal-autoscaling.adoc[]. -=== Use Case 3: Cold Standby Configuration +*Traffic management:* -==== When to Use +* Route traffic to only one region at a time with a BYO global load balancer. +* Run health checks to determine DR and switch routing. +* For non-HTTP use cases such as schedulers, externalize a DR flag in your application and use a Groovy script to enable and disable scheduler flows directly in the runtime plane, which reduces reliance on the control plane to trigger your backup applications through a cold-standby approach. -Low-impact, stateless, or stateful applications that tolerate longer downtime during recovery. +*Licensing:* -==== Application State +* With UBP, this configuration consumes no messages or throughput, and it consumes fewer flows when you configure xref:ch2-configure-horizontal-autoscaling.adoc[CPU-based HPA]. +* Without UBP, this configuration consumes fewer vCores when the backup deployment stays scaled down before it scales up in response to an outage in the primary region. -* Set up the infrastructure, but configure your application to a stopped state in the passive region. -* Start the application only if you detect an outage. +*Recovery time objective:* A fully scaled backup application experiences lower disruption during an outage. A scaled-down backup application has an RTO that depends on the time to scale up, either through control-plane-triggered redeployments or through xref:ch2-configure-horizontal-autoscaling.adoc[CPU-based HPA]. To let integrating systems resend events or in-flight transaction data, capture recovery points in logs and monitoring, which builds reliability into your applications. -==== Traffic Management +==== Cold Standby Configuration -* Use a BYO global load balancer to route traffic only to the region that is up. -* Perform health checks to determine DR, and use scripts to start your applications in backup state and switch traffic routing. -* If the control plane is unavailable, you can't start the backup application in the runtime plane. +*When to use:* Choose this configuration for low-impact, stateless or stateful applications that tolerate longer downtime during recovery. -==== Licensing +*Application state:* -Because the backup application is in a stopped state, this configuration doesn't consume additional vCores, Mule flows, Mule messages, or Data throughput until the backup application starts in the event of a disaster. +* Set up infrastructure, but keep your application stopped in the passive region. +* Start the application only when you detect an outage. -==== Recovery Time Objective +*Traffic management:* -The RTO depends on the time it takes to start applications via the control plane. +* Route traffic only to the region that is up with a BYO global load balancer. +* Run health checks to determine DR, and use scripts to start your backup applications and switch traffic routing. +* If the control plane is unavailable, you cannot start the backup application in the runtime plane. -== Runtime Plane High Availability and Disaster Recovery Summary +*Licensing:* Because the backup application stays stopped, this configuration consumes no additional vCores, Mule flows, Mule messages, or data throughput until the backup application starts during a disaster. +*Recovery time objective:* Your RTO depends on the time to start applications through the control plane. -High-Impact Stateless API:: Use an active-active setup with active replicas in two regions and an external load balancer. +=== Runtime Plane High Availability and Disaster Recovery Summary -High-Impact Stateful API:: Use an external load balancer to route traffic, and perform health checks to determine DR and switch traffic. +High-impact stateless API:: Use an active-active setup with active replicas in two regions and an external load balancer. -Medium-Impact Stateless and Stateful API:: -+ -* Use cold/warm standby, with applications deployed to both regions. -* Stop or scale down the applications or replicas in the secondary region. +High-impact stateful API:: Use an external load balancer to route traffic, and run health checks to determine DR and switch traffic. +Medium-impact stateless and stateful API:: Use cold or warm standby with applications deployed to both regions, and stop or scale down the applications or replicas in the secondary region. -== CloudHub 2.0 High Availability and Disaster Recovery Improvements +=== Disaster Recovery for Regional Services -CloudHub 2.0 improvements over CloudHub include: +Some MuleSoft services are regional, which shapes their DR strategy. -* Loose coupling of the runtime plane region and control plane region due to a service-oriented architecture. -* Clustering, which allows you to configure HA while running the process on a single primary replica. -* Usage optimization by elastic scaling with Horizontal Pod Autoscaling (HPA) for warm standby if a failover is required. -* Active/passive application management, which enables you to set schedulers and listening connectors, such as Anypoint MQ, DB, FTP, and Salesforce listeners, to a stopped or inactive state. -+ -Externalize DR flags and expose REST endpoints to enable and disable flows via Groovy script as needed. +Anypoint MQ:: This regional service runs in the same region as the application and provides out-of-the-box region failover to a fallback region for standard queues. When your application's RTO tolerates processing in-flight messages only after the primary site recovers, Anypoint MQ failover suffices. Otherwise, configure a backup application in an active/passive setup and resend in-flight events and data to the failover queue during a traffic switch. During backup with VM queues and Anypoint MQ, you can lose data for a few minutes, depending on your configuration. -== Disaster Recovery for Regional Services +Object Store v2:: This regional service runs in the same region as the application and provides no region failover. An application deployed in a different region cannot access it. Bring your own external storage to access or back up data across regions. Object Store v2 doesn't retrieve data for regular backups. Caching in the object store, which you use for token management between calls, is region-specific, and you cannot transfer it to a secondary region. -Certain MuleSoft services are regional, which impacts their DR strategy. +=== Disaster Recovery for the Anypoint Platform Control Plane -Anypoint MQ:: This regional service runs in the same region as the application and provides OOTB region failover to a fallback region for standard queues. If an application's RTO tolerates processing in-flight messages only after the primary site is backed up, Anypoint MQ failover is sufficient. Otherwise, configure a backup application in an active/passive setup, and resend in-flight events and data to the failover queue during a traffic switch. During backup with VM Queues and Anypoint MQ, data can be lost for a few minutes, depending on the configuration. +The Anypoint control plane covers additional Anypoint services beyond CloudHub 2.0. For details, see the https://compliance.salesforce.com/en/documents/a005A00000vvLBjQAM[DR Test Summary]. -Object Store v2:: This regional service runs in the same region as the application and doesn't provide any region failover functionality. An application deployed in a different region can't access it. Bring your own external storage to access or back up data from different regions. Object Store v2 doesn't retrieve data for regular backups. Caching in the object store, used for token management between calls, is region-specific, and you can't transfer it to a secondary region. +== Considerations and Limitations -== Considerations for Disaster Recovery Implementations +=== Considerations for Disaster Recovery Implementations -Implementing a Disaster Recovery (DR) strategy for applications deployed to CloudHub 2.0 requires careful consideration of several key factors. +A DR strategy for applications deployed to CloudHub 2.0 requires you to weigh several factors. -Control Plane Availability:: The CloudHub 2.0 management UI and control plane are available in the US East region of North America. They're also available in EU, Canada, and Japan. For more information, see xref:hosting-home::index.adoc#control-plane-and-runtime-plane-support-matrix[Control Plane and Runtime Plane Support Matrix]. +Control plane availability:: The CloudHub 2.0 management UI and control plane run in the US East region of North America, and also in the EU, Canada, and Japan. See xref:hosting-home::index.adoc#control-plane-and-runtime-plane-support-matrix[Control Plane and Runtime Plane Support Matrix]. -Caching:: Use caching in the object store to share the tokens among various calls. Object stores are only region-specific, and you can't transfer them to a secondary region. +Caching:: Use caching in the object store to share tokens among calls. Object stores are region-specific, and you cannot transfer them to a secondary region. -Rate-Limiting API Policies:: Rate-limiting policies are currently set at the replica level. If you require rate limiting for in-flight transactions during DR, define custom policies using an external multiregion shared object store. Include a DR playbook with steps to adjust rate limits during and after DR. +Rate-limiting API policies:: Rate-limiting policies apply at the replica level. When you need rate limiting for in-flight transactions during DR, define custom policies that use an external multiregion shared object store, and include a DR playbook with steps to adjust rate limits during and after DR. -Schedulers:: Many applications run on a scheduled basis to read from databases or files. In this case, schedulers continue to run even if the control plane is briefly unavailable. +Schedulers:: Many applications run on a schedule to read from databases or files. In this case, schedulers keep running even when the control plane is briefly unavailable. -Listeners and Watermarks:: For listeners like DB Table Row and Salesforce Events, use an external persistent store, like DB or Redis, across replicas and regions for polling and watermarks. This is crucial only if connectors use watermarks directly or indirectly. +Listeners and watermarks:: For listeners such as Database Table Row and Salesforce Events, use an external persistent store, such as a database or Redis, across replicas and regions for polling and watermarks. This matters only when connectors use watermarks directly or indirectly. -== Regional Differences and Disaster Recovery for Specific Regions +=== Regional Differences for Specific Regions -Some control plane regions have limitations for multiregion DR. +Some control plane regions limit multiregion DR. -Control Plane to Runtime Plane Ratio:: Unlike the US (1:12) and EU (1:2) regions, Canada and Japan have a 1:1 control plane region to runtime plane region ratio. +Control plane to runtime plane ratio:: Unlike the US (1:12) and EU (1:2) regions, Canada and Japan use a 1:1 control plane region to runtime plane region ratio. -== Cross-Region Disaster Recovery Limitations +=== Cross-Region Disaster Recovery Limitations -* Cross-region runtime plane DR isn't possible in Canada and Japan. +* Canada and Japan don't support cross-region runtime plane DR. -== Disaster Recovery for the Anypoint Platform Control Plane +== Best Practices -The Anypoint Control Plane encompasses additional Anypoint services apart from CloudHub 2.0. For more details, refer to the https://compliance.salesforce.com/en/documents/a005A00000vvLBjQAM[DR Test Summary]. +* Keep your integrations stateless. +* Don't share transactional information between client invocations or scheduled executions. +* When the middleware must retain data because of a system limitation, persist it in an external store, such as a database or a messaging queue, rather than in the middleware infrastructure or memory. +* Keep each replica's state and resources independent so that your applications scale and recover reliably across zones and regions. == See Also diff --git a/hybrid-standalone/modules/ROOT/nav.adoc b/hybrid-standalone/modules/ROOT/nav.adoc index 287b42fd..c0f3db5e 100644 --- a/hybrid-standalone/modules/ROOT/nav.adoc +++ b/hybrid-standalone/modules/ROOT/nav.adoc @@ -10,6 +10,7 @@ ** xref:managing-servers.adoc[Managing Servers] ** xref:managing-servers-groups.adoc[Managing Server Groups] ** xref:managing-clusters.adoc[Managing Clusters] +** xref:hybrid-standalone-hadr.adoc[High Availability and Disaster Recovery] * xref:monitoring-hybrid-standalone.adoc[Monitoring Hybrid Standalone Instances] ** xref:creating-alerts-hybrid.adoc[Creating Alerts for Server, Server Groups or Clusters] ** xref:monitoring-dashboards-hybrid.adoc[Monitoring Dashboards for Server, Server Groups and Clusters] diff --git a/hybrid-standalone/modules/ROOT/pages/hybrid-standalone-hadr.adoc b/hybrid-standalone/modules/ROOT/pages/hybrid-standalone-hadr.adoc new file mode 100644 index 00000000..9135c97c --- /dev/null +++ b/hybrid-standalone/modules/ROOT/pages/hybrid-standalone-hadr.adoc @@ -0,0 +1,67 @@ += Hybrid Standalone High Availability and Disaster Recovery + +Hybrid Standalone lets you host Mule runtime engine on servers or cloud instances that you control, and manage them centrally through Anypoint Platform. You achieve high availability (HA) by clustering multiple Mule runtime instances, so that another node takes over when one fails. You own disaster recovery (DR) under a shared-responsibility model: because you control the infrastructure, you back up, restore, and recover it with your own tooling, while MuleSoft provides HA best practices and keeps the Anypoint Platform control plane available. + +For an overview of HA and DR across all Mule deployment models, see xref:mule-runtime::hadr-guide.adoc[High Availability and Disaster Recovery]. + +== High Availability Versus Disaster Recovery + +High availability (HA) measures a system's ability to remain accessible when a component fails. On Hybrid Standalone, you achieve HA by clustering Mule runtime instances, so that another node takes over when one fails. + +Disaster recovery (DR) restores a system to an acceptable previous state after a natural or man-made disaster, such as flooding, power failures, server failures, or misconfigurations. + +Both strategies increase availability but differ in scope: on Hybrid Standalone, HA keeps your application serving requests when a server in the cluster fails with no loss of service, whereas DR restores service and data after an infrastructure disaster and usually incurs a brief loss of service while the DR plan runs. + +Two measurable objectives drive your DR plan: + +Recovery Time Objective (RTO):: The maximum downtime a business tolerates. RTO is the time the system takes to recover after a disruption. + +Recovery Point Objective (RPO):: The maximum data loss a business accepts after a disaster. RPO drives how often you back up data. + +== High Availability in Hybrid Standalone + +You achieve high availability in Hybrid Standalone by grouping servers into a cluster. + +Clustering:: A cluster groups up to eight servers that act as a single deployment target. The nodes communicate, coordinate their status, and share information. When one server fails, another server in the cluster automatically takes over, which provides failover with no loss of service. + +Shared memory:: A cluster uses a distributed shared-memory grid that replicates data across nodes and keeps it available when a node fails. The shared components include object stores, VM queues, and a distributed lock factory. MuleSoft recommends memory-based storage for performance. Choose database-based (JDBC) storage when you need persistence beyond the cluster lifecycle. + +Node discovery:: A cluster discovers nodes through one of two methods. With unicast, you configure the IP addresses of the nodes, and adding a server restarts the cluster. With multicast, nodes discover each other automatically, but many networks block multicast traffic. + +To create and manage clusters, see xref:creating-clusters.adoc[] and xref:managing-clusters.adoc[]. For a deeper reference on Mule runtime clustering, see xref:mule-runtime::mule-high-availability-ha-clusters.adoc[]. + +== Disaster Recovery in Hybrid Standalone + +Because you control the infrastructure in Hybrid Standalone, you and MuleSoft share disaster recovery responsibilities. + +Your responsibility:: You back up and restore the runtime plane, fail over applications, and run DR procedures with your own tooling and infrastructure. Align your cluster DR with standby-site strategies. Back up your platform configurations, such as connections and properties, with the Anypoint CLI or Anypoint Platform APIs. + +MuleSoft responsibility:: MuleSoft keeps the Anypoint Platform control plane available across multiple availability zones. The control plane stores only operational metadata and configuration, not your business data. For a full-region control-plane loss, MuleSoft maintains documented recovery objectives. + +== Considerations and Limitations + +Account for these factors when you design a clustered Hybrid Standalone topology: + +Network topology:: Clustering across availability zones or regions can degrade performance because of network latency. Where possible, place cluster nodes on the same subnet or availability zone. + +Simultaneous node restarts:: Don't restart multiple cluster nodes at the same time, such as during operating system patching. Concurrent restarts can trigger cluster-membership eviction and distributed-lock partition migration, which can take applications offline. + +Cluster rejoin behavior:: After a network interruption, evicted nodes might not rejoin the cluster automatically and can require a manual restart. Account for this behavior in your operational runbooks. + +Object Store persistence:: Object Store persistence doesn't support sharing the same database across different clusters. Plan your persistence strategy accordingly. + +Object Store locking:: For Object Store lock handling in clusters, use an Object Store Connector version that supports configurable lock timeouts, which prevents indefinite lock hangs. + +== Best Practices + +* Keep your integrations stateless, and externalize state to an external store, such as a database or a messaging queue. +* Run identical Mule runtime, Runtime Manager agent, and Java versions on all cluster nodes. +* Schedule cluster changes during maintenance windows. +* Test your failover and restore procedures regularly. + +== See Also + +* xref:mule-runtime::hadr-guide.adoc[] +* xref:creating-clusters.adoc[] +* xref:managing-clusters.adoc[] +* xref:mule-runtime::mule-high-availability-ha-clusters.adoc[] diff --git a/hybrid-standalone/modules/ROOT/pages/managing-hybrid-standalone.adoc b/hybrid-standalone/modules/ROOT/pages/managing-hybrid-standalone.adoc index 68676832..e51d5bdb 100644 --- a/hybrid-standalone/modules/ROOT/pages/managing-hybrid-standalone.adoc +++ b/hybrid-standalone/modules/ROOT/pages/managing-hybrid-standalone.adoc @@ -13,6 +13,8 @@ Individual server administration including health monitoring, configuration upda Coordinated management of multiple servers for load distribution and workload balancing * xref:managing-clusters.adoc[Managing Clusters] + High-availability cluster administration with member coordination and shared resource management +* xref:hybrid-standalone-hadr.adoc[High Availability and Disaster Recovery] + +Design hybrid standalone deployments to stay available during component failures and to recover from disasters while minimizing downtime and data loss == Management Operations