Blue Green with multiple microservices with internal calls
I have 8 spring boot microservices that call each other internally. DNS callers of other microservices define in the application.properties file of each service.
Suppose microservice A is represented by A-> a.mydns.com and B-> b.mydns.com etc.
So basically each microservice consists of an ELB and two HA Proxies (distributed in two zones) and 4 application servers (distributed in two zones).
I am currently creating new Green servers (application servers only) and switching direct traffic from the HA proxy layer. In this case, while the new version of microservices is being tested, it is also being rolled out to live clients.
Ideally, the approach should be creating the entire server structure, including ELB and HA Proxies, for each microservice, right?
But then when I am faced with testing problem with test dns. I can match ELB to test dns. But then what about an external dns microservice that is hardcoded into the side of the application.properties file?
What would be the approach I should take in such a scenario?
source to share
Ideally, the approach should be creating the entire server structure including ELB and HA Proxies for each microservice?
This is not necessarily the case. The deployment (blue-green or canary, whatever your deployment strategy) should be transparent to it to consumers (in your case, 7 more microservices). This means that your DNS service names (or IPs) that other services communicate with must remain the same. IMHO, in the case of a microservice deployment, you don't need to think about other services in the ecosystem as long as you keep your part of the contract; because the whole point of "micro" services. As another SOer pointed out, if you cannot deploy your one microservice without making changes to other services, it is not a microservice, it is just a monolith speaking http.
I suggest you read this article https://www.thoughtworks.com/insights/blog/implementing-blue-green-deployments-aws
I am quoting relevant parts here
Multiple EC2 instances behind ELB
If you are serving content through a load balancer, then the same technique will not work because you cannot bind Elastic IPs to SOB. In this case, the current blue environment is a pool of EC2 instances and the load balancer routes requests to any usable instances in the pool. To execute the blue-green switch behind the same, you must replace the entire pool with a new set of EC2 Instances containing the new software version. There are two ways to do this - automating a series of API calls or using the AutoScaling Group.
There are other types of ads like this one.
DNS redirection using Route53
Instead of exposing Elastic IPs or ELB long hostnames to your users, you can have a domain name for all of your public URLs. Outside AWS, you can perform a blue-green switch by modifying the CNAME records in DNS. On AWS, you can use Route53 to achieve the same result. With Route53, you create a hosted zone and define a recordset resource to tell the Domain Name System how traffic is routed for that domain.
To answer another question.
But what about an external dns microservice that is hardcoded into the side of the application.properties file?
If you do, I suggest you read about 12factor ; especially config . You should also take a look at the service discovery options if you haven't already.
I have a feeling that you have spaghetti not-so-micro-services. If this is a green box project, and if your timeline budget allows, I would suggest you explore a container application along with its infrastructure (one word: "Docking") and use any container orchestration technology like Kubernetes, Docker swarm or AWS ECS (easiest if you are already on AWS-land) I know this is beyond the scope of this question, just a suggestion.
source to share
I would suggest subrequesting your microservices (easily with spring-boot) and then using ECS (Elastic Container Service) and ELB (Elastic Load Balancer) with application load balancers. (can be an internal or internet person).
ECS and ELB then leverage your microservices /health
when deploying new versions.
You can then implement a more complex one HealthIndicator
in spring-boot to determine if the application is healthy or not (and therefore ready to receive incoming requests). Only when the new application is healthy will it be put into production and the old one will be strewn about.
Then check all your business logic on test environment
, and because of Docker you are using the same image throughout your environment, you don't need to run (any) tests when deploying to production. (Because it's already tested and if it boots, you're good to go).
source to share
Typically, for B / G testing, you will not use different dns for new features, but define the rules, for example, every 100th user submits a new feature, or only ips from a certain region or office have access to new features, etc.
Assuming you are using AWS, you should be able to create an ALB before the ELB for context-based routing, in which you should be able to define the rules for your routing either B or G. In this case, you need separate environments to run independently (perhaps using the same database).
For more complex rules, you can use tools like leanplum or omniture inside your spring boot application. With this approach, you have one single environment in which to host old and new functionality, and then you remove the obsolete code.
source to share
I personally would go the simpler route, using a green deployment test DNS record, which is then dumped for a realtime DNS record when you've fully verified that your green deployment is good.
So what I mean:
You declare that your current deployments have the following DNS records:
- a.mydns.com
- b.mydns.com
I would suggest that you are creating a template where every microservice deployment also receives a test dns entry:
- test.a.mydns.com
- test.b.mydns.com
When you deploy the green version of your microservice, you deploy everything (including the ELB) and map the CNAME ELB to the test DNS record in Route 53. This means you have the green version ready to go but not in use by your live application. The green version has its own DNS record, so you can run a full test suite against the test.a.mydns.com domain.
If (and only if) the test package passes, you change the CNAME record for a.mydns.com to be the ELB that was created as part of your green deployment. This means that your existing microservices just start talking to your green deployment after DNS propagation. If there is a problem, simply reverse the DNS update to the old CNAME record and you are completely rolled back.
It takes a bit of coordination here, but you should be able to automate the whole thing with something like Jenkins and the AWS CLI.
source to share