Additional Zone vs Region - US East and West

[justinestruch]

Hi All. I have a question on architecture for our Cloudstack setup. Currently we are setup in a datacenter on the US East Coast in NYC. We are starting to setup a new environment on the US West Coast in LA.

There is about 70 ms of latency between US East and West.
My goal is to be able to manage the entire cloud from one Management Console GUI. I'm trying to avoid having separate consoles for US East and US West but I am worried about the latency slowing things down between the 2 DCs.

Which option from below would you recommend or think is best practice for us?

1. Deploy a new management server and Region. I think when doing this, cloudstack can see all resources in both regions? As described here https://docs.cloudstack.apache.org/en/4.22.0.0/installguide/configuration.html#adding-regions-optional
2. Deploy a new Zone and deploy a secondary management server in that new zone. This will give us redundancy if the primary mgmt server goes down as described here https://docs.cloudstack.apache.org/en/4.22.0.0/conceptsandterminology/choosing_deployment_architecture.html#multi-site-deployment
3. Just deploy a new zone. Don't deploy a secondary management server.
4. Other better option?

[NuxRo]

Regions are a bit useless and I would ignore the concept. There is nothing shared between them.

Depending on your requirements either could work:
1 - deploy a new zone under existing CloudStack; also deploy a new mgmt server and a DB slave, but keep the mgmt server stopped - the latency is too high between the 2 sites to make this work without problems. In the event of DR you can start the management server and have it talk to the local DB slave (which you will turn into a master) so you can manage the local zone.

2 - even "simpler", just deploy a new CloudStack, completely independent. This will make things a little bit complicated if you want to have the same users, offerings, templates etc, you'll need to think of a way to sync these between the two. SAML or LDAP could be employed here for the users part. The upside here is that an outage in one will not affect you in any way in the other, hence more robustness.

My 2 cents.

[justinestruch]

Thanks @NuxRo For this info it's very helpful. It's good to know regions are useless. It sort of seemed like You can see all resources from all regions when logged into 1 Cloudstack Mgmt instance. But the 2 instances would be independent of each other.

I will likely go with option 2 as you mentioned. I agree keeping the instances in sync with templates, offerings, etc. will be a pain. But this will make both DCs independent which is a big bonus.

I was hoping there was an easy way to have "single pain of glass" view of the entire infra of both DCs but it sort of sounds like it's not possible due to the latency. Most of the info I've seen out there about multiple sites, is dealing with much lower latency than US East to West. I appreciate the help here!

If you happen to know of any good architecture guides or white papers of multiple locations that would be great. I'll keep searching and testing on my own.

[NuxRo]

@justinestruch

I was hoping there was an easy way to have "single pain of glass" view of the entire infra of both DCs but it sort of sounds like it's not possible due to the latency.

Well, you've ignored option 1 in my reply then. You just need to be a little careful with the way you use the DB, but pretty much the gist is in my message. The weak point is the latency between your primary site and its DB and using it from the secondary site - if you avoid/workaround that, then it should work.

You should be able to manage one site from the other's management server, but do test thoroughly.

Do you have any idea how many resources (hosts & VMs) you will be ending up managing with CloudStack?

[daviftorres]

Dear @justinestruch and @NuxRo ,

This is a discussion that we circle back every week. How to expand geographically maintaining the latency as low as possible and add resiliency to our infrastructure.

Currently, we have one primary region with the core management of CloudStack, and multiple satellite (remote) Zones geographically dispersed. Here are my observations:

Facts:

• We use 4 management servers where:
  • 1st primarily responds to UI/API. If unhealthy, it fails over to 3rd, 4th, and 2nd.
  • 2nd first priority for Agents to connect. Followed by 4th, 3rd, and 1st.
    • Note:
      • 1st and 2nd have dedicated host (metal),
      • 3rd and 4th shared computing resources (virtualized) in a different failure domain.
• Primary -> Secondary database replication.
  • The Primary has dedicated hardware,
  • Secondary share computing resources in a different failure domain.
  • Additionally, encrypted database snapshot every 15 min two different geographies for DR.

Observations:

• Even with multiple Zones across North America connected to the same core management, we have:
  • No latency issue between management servers and database (all in same geography),
  • No perceptible latency to the satellite Zones (~5,000 km apart),
• Resiliency to a datacenter (region) blackout.:
  • With automation, a new CloudStack core management can be deployed in minutes,
  • Only the DB needs to be restored. Which is similar to the proposal number 1 from @NuxRo.
    • Note: promoting a DB into primary is not trivial and has the risk of cause split brains.

Reflections:

• The focus of the architecture has to be on the DB.
  • Replication, Galera, InnoDB Cluster, you name it!
  • Point in time recovery: Snapshot, Dumps, Replication, etc.
• Management servers and databases in different failure domains addresses partial DC outages.
• Automation plus Snapshots resolve the need for a DR from a total DC outage.

There are many additional topics we can explore, such as serving ACS at the edge (similar to a CDN) with caching and BGP advertising in multiple locations, or placing one dedicated management server in each region to serve only that Zone’s Agents.

Please share your thoughts, this is a highly relevant subject that warrants thorough discussion.