How does Azure DevOps work with on premise databases hosted in two continents?

Question

question

Winston101 asked Jan 6, '19

How does Azure DevOps work with on premise databases hosted in two continents?

Hi

DevOps newbie scenario and questions:

There are two teams of Developers/ Testers in NZ and the UK who regularly check in SSDT code using on premise TFS 2013 on NZ server. Issue is the number of merge conflict, slow gets, slow checkins, builds and deployments to the UK from NZ. Looking to upgrade SQL to 2016 on the TFS server in NZ, upgrade to TFS 2018 (latest SP) and then migrate to the cloud in a pre-prod environment and test.

However noted that due to the large SSDT solutions, I will require on premise private agents in NZ and the UK, which then begs the question if a TFS Admins is logging onto VS in NZ but deploying to London, would the agent used be the London Agent or the NZ agent? If the Agent used in on prem NZ I suspect it won't be much faster to deploy in the UK.

Alternatively If I set up a TFS proxy server in London and upgrade the disks to SSD and up the vCPU's how would this work on the TFS Proxy server in London, i.e. can I deploy the solution in an on premise set up from the proxy server in the UK? Or does the deployment have to be from the main TFS server in NZ?

Thanks

Win

sql server ssdt azure tfs proxy

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Answer 1 · 2019-01-08T22:35:05Z

Steve Jones - Editor answered Jan 8, '19

So a), deploy from proxy. AFAIK, yes, but I'm not a TFS expert. My reading of how this works is that the proxy server looks like a normal TFS server and handles the caching/replication of vcs data to/from the main tfs server. So I'd have my UK release agents just using the TFS Proxy and that should work fine. The deploy is usually just a checkout/artifact pull, so assuming there's some dFS or other file xfer under the hood, this should work.

b) the deployment is based on the target. If I have UKSQL01 in the UK and NZSQL01 in NZ, and I click a button in TFS/ADO to say "release to NZSQL01", it's not relevant where I am. The TFS/ADO system contacts the agent in the release pool, which is where you set it. If you have one pool of agents, then you could get a UK agent or an NZ agent. If you have a UK pool, then that's the agent process that will run your release. Same for NZ, so you want to set your agent pools to be separate.

I don't think you can override a pool on release, at least, I don't see an option in ADO.

Orchestrator - a process that decides how to run other processes. K8s has this, and TFS/ADO are the orchestrator for your DevOps process. They decide how to execute a process on which agents that you've defined. The ADO/TFS server itself doesn't really do any work. It farms this out to an agent process via some message. Just as if your boss told you to compile the code. Your boss is the ADO/TFS/orchestrator and you are the agent. The actual place the compilation takes place depends on which machine you log into and click compile. Could be your laptop, could be RDP to a server, or a message send to a machine in AWS in the UK or NZ. You aren't doing the work, and your machine may or may not be. That's why you want to separate your architecture out and think about how you want things to work.

In your scenario, if the TFS admin in the UK tells TFS to do something (proxy or no), the thing gets started where the agent lives. That's the user. Could be UK or NZ, depending on your config. Where it executes, that's different. The agent can run things by connecting to machines in the UK or NZ, just as you can, so you want to ensure that you set up an agent to connect to local resources.

Make sense?

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Answer 2 · 2019-01-08T15:55:19Z

Steve Jones - Editor answered Jan 8, '19

A couple things. First, you want to separate out the VCS/orchestrator from the environments. Those are two different hings.

For TFS/Azure DevOps, these are the remote repos for code, as well as the master orchestrator/scheduler. They are two versions of the same thing. Azure DevOps slightly rev's ahead of TFS, but they're the same codebase. If you use Azure DevOps, then everyone is connecting to a remote TFS install at a Microsoft region rather than in one of your offices. This means that half of your people will get a local connection now, but a remote one later.

The get/commit speed comes from network and host resources. If things are slow, you have an issue with one or both of those. Azure DevOps (ADO) might improve things, as I think MS has fairly large resources underlying most of these, but networking is still networking. It's hard to know what's better, but what I'd do is set up an ADO project with 2 of your devs that get frustrated, 1 in UK, 1 in NZ. Then I'd copy the SSDT project over there and do some get/commits with them, trying to understand if this is better or worse.

For the environments, you could move environments local to each side. Build UK in UK, test UK in UK, build NZ in NZ, test NZ in NZ. In this way you'd have people pull the deployment to their local test environment rather than having "a test environment". At some point, merge things to a single location to verify that nothing is lost.

For deployments, this is again like a user making a get. The agent is deployed to where the environment is located. If this is UK, then the agent acts as a user, doing a get and deploy to an instance there. Same for NZ. For Azure DevOps, nothing changes, other than the agent needs to now "get" from the ADO server. A Proxy should make this faster, but depending on rate of change and networking, this can still be slow. Same issues with ADO, now everyone is remote.

If the deployments are slow, your agents and the machines they deploy to are likely short resources. If you need this stuff to go fast, it needs hardware.

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Answer 3 · 2019-01-08T21:47:06Z

Winston101 answered Jan 8, '19

Thank you Steve.

I think the on premise will be faster than Azure, especially if we up the disk to SSD, and increase the CPU count + RAM. However as we may go to Azure for the DWH solution, we need to test the ADO scenario for TFS.

I am planning on a test scenario where we run the checkins/gets from NZ and the UK, and compare results to existing on premise TFS 2013.

However in terms of an on premise configuration with a proxy TFS server in the UK, and the main TFS server in NZ, functionally is it possible to deploy from the proxy TFS server' SQL solution to UK, even though it's the proxy and not the main TFS box?

Also in terms of the deployments with two local Agents one in NZ and the 2nd in the UK.

So let's say a TFS admin is in VS, and attempts publishing a SQL solution to NZ from ADO, logged into ADO from the UK, would the build and publish call the Agent on the NZ server or would it call the available agent, which could be the UK Agent? If the deployment to NZ, calls the UK Agent I'm relatively confident the performance will be in the realms of the current on premise solution. However if it calls the NZ Agent, I think ADO is worth a good go.

Lastly, could you elaborate on "VCS/orchestrator from the environments", not sure what VCS/Orchestrator is in the ADO or TFS context.

Many thanks

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Answer 4 · 2019-01-08T23:27:05Z

Winston101 answered Jan 8, '19

Excellent, that gives me a lot of confidence.

a) Here I can set up an Agent on the proxy server in it's own pool in the UK, with the existing agent in the NZ pool, and if deploy from the UK TFS proxy it will call the UK Agent, and if a deployment is executed from NZ, from the on premise main TFS box it would use it's own local agent. Great.

b) So I will test this scenario, and is my preferred choice we will be going to Azure for the DWH solution at some point. So, as long as we have a separate UK & NZ Agents in separate pools, this will facilitate local builds and deploys.

Cheers

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

question