AKS - Application Gateway Ingress Controller management

As discussed in my post about AKS planning there are a couple of choices for what device you use in conjunction with your Azure Kubernetes Service (AKS) ingress controller. The default scenario is you deploy a Azure Load Balancer and then you install your ingress controller of choice. The ingress controller then works in unison with the Load balancer. 

Another option now available it the Application Gateway Ingress Controller (AGIC) this ingress controller allows you to use the Azure Application Gateway Layer 7 device to expose your services to the internet. As you add services to your cluster it automatically updates the Azure Application gateway. 

For information on how to implement AGIC checkout the links on this page .

In this article we are going to discuss AGIC and some of the practicalities of working with this configuration.

Managed Identities

To use AGIC you need to use AAD pod identity. We configure a managed identity with permissions to manage and update the Application Gateway. We use AAD Pod identity to allow AGIC to use this managed identity. For more information on AKS and managed identities checkout my previous post on this topic.

Application Gateway Managed Identity

• Reader permissions on Application Gateway Resource Group
• Contributor permissions on Application Gateway resource
• Managed Identity Operator permissions on KeyVaults if used for SSLs

AKS permissions

AKS kubelet identity needs;

• Managed Identity Operator over the Application Gateway managed identity for use with the AAD pod identity

Deployments

You can deploy the initial Application Gateway configuration using your IaC tool of choice.  Once you have configured the Application Gateway ingress controller this takes over management of the Gateway and any further deployments of your IaC would wipe out this configuration.  

You will see on the Application Gateway tags that after the AGIC updates the Gateway it applies tags which show when the configuration was last updated by the AGIC. This can be useful in troubleshooting issues as you can see when the Application Gateway was last updated.

You need to manage this conflict for you deployments to avoid potential downtime for your services.  One suggestion is to periodically get the AGIC to push updates.  To avoid any potential impact or down time I have taken an approach using configuration in the IaC code to manage this. I am using Terraform for my deployments so will talk around this method.  I deploy the Application Gateway with a base configuration of some place holder HTTP settings, Listeners and Backend configuration.  

I then use a lifecycle block with an "ignore_changes" list, this means the first deployment will create the Application Gateway but from that point on other changes are made by the AKS cluster and Application Gateway Ingress Controller (AGIC).

Lifecycle gist code

There are a couple of exceptions for configuration that we can change via our Terraform templates. This includes the SSL certificate, which will need updating when it expires and the SSL policy which may need changing as new TLS versions or other policy changes are need.

Annotations

This site contains the annotations that can be used in your ingress config to manage the way the application gateway behaves. You will always need to set the ingress class so the application knows to use AGIC `"kubernetes.io/ingress.class" = "azure/application-gateway"`. In addition the applications are configured to use the certificate that has been deployed to the application gateway. 

Annotation gist code

The name is the friendly name you use when you deploy the SSL to the Application Gateway. If you add new certs or update these after a renewal ensure you are referencing the correct certificate. Below is how the annotation block may look on a common AKS ingress deployment.

Portal Bug 

When you use the annotation for hostname on your application configuration this should be configured on the Application Gateway as a mutli-site listener.  Currently if you check the configuration in the portal you will see an error as shown below;

This should be showing as a mutli-site listener but will show as a basic listener;

This is a bug in the portal and can safely be ignored.  However, if you want to confirm the right hostname is showing on the listener you can run the following command. This shows it is assigned to the value "Hostname", the portal uses the value from "Hostnames" which is where the issues appears.

Hostnames gist code

Conclusion

There is lots to take into consideration when implementing the Application Gateway Ingress Controller.  I hope this article has helped you with your configuration and avoiding some common issues with the deployment of the Application Gateway.