Kubecost on AKS Part 02

 Hi all, continuing from previous post on part 1, this blog post will focus on enabling kubecost with cloud integration . 

First, do the Azure Cost Management export to a Storage Account 

fill in all the required details especially on the storage account . The report in this storage account alter will be access by kubecost to view in on the kubecost dashboard . 

do note this may takes hours to be populated. 

Step 2 -Create A custom role and assigned a SPN with that custom role to your subscription

# Create kubecost custom role

   resource "azurerm_role_definition" "kubecost" {

   name        = "kubecost_rate_card_query"

   scope       = "/subscriptions/${var.sub-id}"

   description = "kubecost Rate Card query role"

 

  permissions {

    actions     = [

     "Microsoft.Compute/virtualMachines/vmSizes/read",

      "Microsoft.Resources/subscriptions/locations/read",

      "Microsoft.Resources/providers/read",

      "Microsoft.ContainerService/containerServices/read",

      "Microsoft.Commerce/RateCard/read",

    ]

    not_actions = []

  }

  assignable_scopes = [

    "/subscriptions/${var.sub-id}"

  ]

}

#Assign Role to SPN at Subcription level

resource "azurerm_role_assignment" "kubecost" {

  scope                = "/subscriptions/${var.sub-id}"

  role_definition_name = azurerm_role_definition.kubecost.name

  principal_id         = var.spn-id

}

you may also use create this using powershell or az cli, link in the reference

Step 3 - For this one it can divided into two the hard way or the gui way . dont worry i will cover both 

Let get to know the easier way first , hard requiredment is it must be running kubecost version 1.96 which was release few day ago as the blog is drafted. 

Once u deploy the kubecost with just setting up with the kubetoken. Access the kubecost dashboard by port forward to port 9090 and go to the setting. 

scroll until u find this option 

Click update and fill the details 

After click on submit, wait for sometimes and the data on cloud integration to be populated. 

So here come the hard way , all the component in easier method will be converted into line of code start with creating a kubernetes secret containing all the details 

resource "kubernetes_secret" "kubecost_sec" {

  metadata {

    name      = "kubecost-sec"

    namespace = kubernetes_namespace.kubecost.metadata[0].name

  }

  data = {

      "cloud-integration.json" = "\r\n{\r\n    \"azure\": [\r\n        {\r\n          \"azureSubscriptionID\": \"${var.sub-id}\",\r\n          \"azureStorageAccount\": \"${var.saname}\",\r\n          \"azureStorageAccessKey\": \"${var.sakey}\",\r\n          \"azureStorageContainer\": \"${var.sacontainer}\",\r\n          \"azureContainerPath\": \"${var.sapath}\",\r\n          \"azureCloud\": \"${var.azcloud}\"\r\n        }\r\n    ]\r\n}"

   

  }

  type = "Opaque"

}

on the helm value , point the cloud integration value to the secret that been created . 

  set {

    name  = "kubecostProductConfigs.cloudIntegrationSecret"

    value = kubernetes_secret.kubecost_sec.metadata[0].name

  }

Both of the method will get you to have kubecost cloud integration with azure 

now it is all concluded, list of the reference as below 

1. Deploy AKS + Kubecost with Terraform - Code it Yourself... (mendible.com)

2. Azure Config – Kubecost

3. poc-common-configurations/cloud-integration.json at main · kubecost/poc-common-configurations (github.com)

4. Kubernetes secret with json · Issue #1801 · hashicorp/terraform-provider-kubernetes (github.com)

5.Cost governance with Kubecost - Cloud Adoption Framework | Microsoft Docs

6. Kubecost team who has provide a good insight in their slack to the community 

enjoy the learning and thanks for reading, the sample code is available on my github 

Kubecost on AKS Part 01

 Hi, for this blog post , i will share my journey on enabling kubecost with cloud integration on my kubernetes cluster . 

Prerequisite 

1. Kubernetes cluster ; my case will be AKS 

2. Azure AD SPN 

3. Storage Account 

4.Tools
 4.1 Helm
 4.2 Terraform(Optional)

5. Ingress Controller (Optional)

Step 1- Get kubecost running on your AKS cluster 

There are few guide available even on microsoft document tation , but i will share on utilizing helm chart provider in terraform. 

 1.1 - Create a kubecost namespace

 resource "kubernetes_namespace" "kubecost" {

  metadata {

    annotations = {

      name = "kubecost-annotation"

    }

    name = "kubecost"

  }

}

1.2 install the kubecost helm chart 

resource "helm_release" "kubecost-helm" {

   name       = "kubecost"

   repository = "https://kubecost.github.io/cost-analyzer/"

   chart      = "cost-analyzer"

   namespace = kubernetes_namespace.kubecost.metadata[0].name

  

    set {

    name  = "kubecostToken"

    value = "aGVsbUBrdWJlY29zdC5jb20=xm343yadf98"

    #value = "YWJkdWwubXVuaXI5NEBvdXRsb29rLmNvbQ==xm343yadf98"

   } 

}

For kubetoken , u may use either one as both are working as per my testing

Then u can apply the terraform code and if you want to add  other parameter , you may add other like what has been done here - kubecost helm

set {

    name  = "kubecostProductConfigs.clusterName"

    value = var.aks-name

  }

  set {

    name  = "ingress.enabled"

     value = true

   }

  set {

     name  = "ingress.hosts"

     value = "kubecost.munirtajudin.xyz"

   }

 

  #Set the currency

 set {

    name  = "kubecostProductConfigs.currencyCode"

    value = "USD"

  }

# 

  # Set the region

  set {

    name  = "kubecostProductConfigs.azureBillingRegion"

    value = "US"

  }

  

  # Generate a secret based on the Azure configuration provided below

  set {

    name  = "kubecostProductConfigs.createServiceKeySecret"

    value = true

  }

  # Azure Subscription ID

  set {

    name  = "kubecostProductConfigs.azureSubscriptionID"

    value = var.sub-id

  }

  # Azure Client ID

  set {

    name  = "kubecostProductConfigs.azureClientID"

    value = var.client-id

  }

  # Azure Client Password

  set {

    name  = "kubecostProductConfigs.azureClientPassword"

    value = var.client-sec

  }

  # Azure Tenant ID

  set {

    name  = "kubecostProductConfigs.azureTenantID"

    value = var.tenant-id

  }

you may do port forward to 9090 and access the kubecost dashboard. do give it a few minute because kubecost will take sometimes to collect the metric. 

see you in part 2 where it will focus on enabling cloud integration with azure cost management. 

thanks for reading and do provide feedback if any 

Nginx Ingress on AKS via Terraform

 let's  continue playing with AKS and this time around , i will be sharing the how to install nginx ingress controller on AKS, this is not new but it is more on my sharing and finding during the process. 

First of all ,  you need to have AKS to begin with , and an app to be publish , it can be a test app some sample app or your own app. i will be utilizing terraform to deploy it . 

Prerequisite 

1. AKS cluster 

2.Terraform installed 

3.Helm installed. 

Some special requirement that found during the deployment as below 

i. Public IP for nginx ingress must be in precreated RG that hold the AKS resource ; MC_rg_aks-name_region

ii. It does not need SPN like traefik and agic deployment 

so now we can deploy nginx ingress into our aks, here is my helm setup in terraform 

resource "helm_release" "nginx" {

  depends_on = [kubernetes_namespace.nginx]

  namespace = kubernetes_namespace.nginx.metadata[0].name

  name       = "nginx-ingress-controller"

  repository = "https://kubernetes.github.io/ingress-nginx"

  chart      = "ingress-nginx"

  timeout    = 300

  set {

    name  = "controller.service.type"

    value = "LoadBalancer"

  }  

  set {

    name  = "controller.service.loadBalancerIP"

    value = azurerm_public_ip.ngxip.ip_address

  }

  set {

    name  = "controller.service.annotations.service\\.beta\\.kubernetes\\.io/azure-load-balancer-health-probe-request-path"

    value = "/healthz"

  }

}

 here is how i cater the public ip resource group 

resource "azurerm_public_ip" "ngxip" {

  name                = var.ngxip

  resource_group_name = "MC_${var.aks-rg}_${var.aks-name}_${var.loc}"

  location            = var.loc

  allocation_method   = "Static"

  sku = "Standard"

  sku_tier = "Regional"

  availability_zone = "Zone-Redundant"

}

After you apply your terraform code, then you can test your nginx using the sample app that i used for agic just need to change some parameter on the code 

and here is the result 

the format may be off but it does conclude that our nginx is running fine. 

Thanks for reading and see you on next post 

here is the refence link that i refer while figuring this 

1. GitHub - kubernetes/ingress-nginx: Ingress-NGINX Controller for Kubernetes

2. Create an ingress controller in Azure Kubernetes Service (AKS) - Azure Kubernetes Service | Microsoft Docs

3.Service | Kubernetes

AKS AGIC Addon via Azure Portal

 The Previous post on enabling agic was done via terraform but let make it easier to follow which by using Azure Portal. 

Please make sure you have all this component running . 

1. AKS 

2. Application Gateway (needed for utilizing the existing app gw )

Step One . Go to AKS > Networking > Application Gateway ; enable it  

Step Two, Monitor the deployment 

Step Three, Test AGIC using sample app 

kubectl apply -f https://raw.githubusercontent.com/Azure/application-gateway-kubernetes-ingress/master/docs/examples/aspnetapp.yaml 

Step Four, Check the ingress creation and backend pool

Sample app running 

Thanks for reading and see you .. 

AKS Ingress with Traefik via Terraform

 Hi All, 

continue from my previous post on AKS AGIC addon, you can utilize azure application gateway to do so but if you like kind of open source project , from what i know , there are two option which is traefik and nginx . 

As for this post , i will be explaining on how to deploy traefik in aks by utilizing helm chart for traefik and terraform for easier deployment. 

These are some component and tools involved in this deployment 

1. Azure Kubernetes Service 

2. Service Principle from Azure AD 

3. Traefik helm chart 

4. Terraform 

Let start, At first please make sure u have azure virtual network for AKS to "sit" in or u may deploy a new one 

module "vnet00" {

  source        = "../modules/network/vnet"
  vnet-location = "eastus"
  vnet-rg       = "DEV-VNET-EASTUS"
  vnet-name     = "DEV-VNET-EASTUS"
  vnet-range    = ["172.25.0.0/20"]
  subnets-name  = ["subnet01", "aks-subnet"]
  subnets-range = ["172.25.0.0/24", "172.25.1.0/24"]
}

following with Service Principle 

module "aks-spn" {
source      = "../modules/aad-spn"
spn-name    = "AKSDEV-SPN"
secret-name = "aksdev-spn-secret"

}

we gonna use of reference for this SPN later ... after both of this is deployed, AKS can be deployed referencing to both of the output from this vnet and spn module. 

module "myaks" {
source = "../modules/k8s/dev"
aks-name = "aks01"
aks-dns = "aks01-dns"
aks-version = "1.24.0"
aks-region = "eastus"
aks-subnet-id = module.vnet00.vnet_subnets.1
admin = "adminlogin"
ssh = "ssh-rsa AAAAB3N"
winpass = "P@ssw0rd1234"
client_id = module.aks-spn.client_id
client_sec = module.aks-spn.client_secret
depends_on = [module.aks-spn]
}

then arrive to the main topic of today which is traefik deployment 

module "traefik" {

  source    = "../modules/network/traefik"
  loc       = "eastus"
  aks-rg    = module.myaks.aks-rg.id
  aks-name  = module.myaks.aks-name
  aks-spnid = module.aks-spn.object_id-entapp
  depends_on = [module.myaks]

}

this module will do few thing like creating public ip for traefik and assign permission accordingly . Traefik public ip will be bind to the same load balancer that aks deployed 

permission 

then u may do terraform apply and once completed , you may browse your traefik dashboard by http://<yourtraefikoublicip>:9000/dashbaord nad here is mine

your deployment is completed and u may use traefik as ingress for your app in aks 

Thanks for reading , please leave a comment if you have some doubt and here are some reference that i use to complete traefik deployment 

1. https://stackoverflow.com/questions/69269097/unable-to-pass-service-annotations-when-deploying-helm-chart-via-terraform

2. https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release 

AKS AGIC addon via Terrafrom

 Hello All, 

my testing has been alot with aks lately  and surely i cannot run away on setting up an ingress controller for all my test application on aks , So i would like to share my journey on setting Appling gateway Ingress controller (AGIC) using aks addon via terraform. 

There are two method on setting AGIC which is via aks addon or helm chart. There are pro and cons for each method but if your Application Gateway just for AKS to be use , using addon will be better. 

More details on the different can be read from here - Microsoft Docs 

I will be using terraform module that i created for this example . 

 Component Involved 

1. Azure Kubernetes Service (AKS)

2. Application Gateway 

3. Service Principle 

4. Azure Virtual Network 

you may download my module to test but we will start as below 

1.  Virtual Network creation 

module "vnet00" {
  source             = "./modules/network/vnet"
  vnet-name       = "vNet-00-aks"
  vnet-range       = ["192.168.16.0/24", "172.16.0.0/22"]
  subnets-name  = ["subnet01", "aks-subnet", "ApplicationGatewaySubnet"]
  subnets-range  = ["192.168.16.0/28", "172.16.0.0/24", "192.168.16.128/26"]
}

2. SPN creation 

module "aks-spn" {
  source             = "./modules/aad-spn"
  spn-name        = "AKSAGIC-SPN"
  secret-name    = "akstfk-spn-secret"
}

3. Application Gateway 

module "appgw01" {
  source               = "./modules/network/appgw"
  agsnetname       = module.vnet00.vnet_subnets_name.2
  agsnetid            = module.vnet00.vnet_subnets.2
  
  depends_on = [module.vnet00]
}

4. AKS 

 module "myaks" {
  source        = "./modules/k8s/dev-appgw"
  aks-name      = "aks01"
  aks-dns       = "aks01-dns"
  aks-version   = "1.23.3"
  aks-subnet-id = module.vnet00.vnet_subnets.1
  admin         = "username"
  ssh           = "ssh-rsa xx"
  winpass       = "password"
  client_id     = module.aks-spn.client_id
  client_sec    = module.aks-spn.client_secret
  aks-spn = module.aks-spn.object_id-entapp
  appgwid = module.appgw01.appgwid
  appgwrg = module.appgw01.appgwrg
  #appgwname = module.appgw01.appgwname
  #appgwnetid =  module.vnet00.vnet_subnets.2
  #appgwnetcidr = xxx
  depends_on = [module.aks-spn,module.appgw01] 
}

Notice there are some line  is commented. this is because u can either use the existing Application gateway as AGIC or You let the AKS create it for you but i choose on the first one. 

Reason for this structure because the same spn will be use in AKS setup and for AGIC which is the read permission on AGIC resource group and contributor to AGIC. 

after all the component has been deployed, you may download the kubeconfig file and test with this command 

1.  kubectl apply -f https://raw.githubusercontent.com/Azure/application-gateway-kubernetes-ingress/master/docs/examples/aspnetapp.yaml

2. Kubectl get ingress 

Then the sample aspnet app details will appear as the setup was done successfully. 

Link to my terraform github repo is here with sample module call  -  GitHub - munir94/TFLAB

That all for now , im planning to cover traefik and nginx ingress deployment also in future, thanks for reading and you may leave comment for enquiry or improvement .