Kubecost on AKS Part 02

 Hi all, continuing from previous post on part 1, this blog post will focus on enabling kubecost with cloud integration . 

First, do the Azure Cost Management export to a Storage Account 

fill in all the required details especially on the storage account . The report in this storage account alter will be access by kubecost to view in on the kubecost dashboard . 

do note this may takes hours to be populated. 

Step 2 -Create A custom role and assigned a SPN with that custom role to your subscription

# Create kubecost custom role

   resource "azurerm_role_definition" "kubecost" {

   name        = "kubecost_rate_card_query"

   scope       = "/subscriptions/${var.sub-id}"

   description = "kubecost Rate Card query role"

 

  permissions {

    actions     = [

     "Microsoft.Compute/virtualMachines/vmSizes/read",

      "Microsoft.Resources/subscriptions/locations/read",

      "Microsoft.Resources/providers/read",

      "Microsoft.ContainerService/containerServices/read",

      "Microsoft.Commerce/RateCard/read",

    ]

    not_actions = []

  }

  assignable_scopes = [

    "/subscriptions/${var.sub-id}"

  ]

}

#Assign Role to SPN at Subcription level

resource "azurerm_role_assignment" "kubecost" {

  scope                = "/subscriptions/${var.sub-id}"

  role_definition_name = azurerm_role_definition.kubecost.name

  principal_id         = var.spn-id

}

you may also use create this using powershell or az cli, link in the reference

Step 3 - For this one it can divided into two the hard way or the gui way . dont worry i will cover both 

Let get to know the easier way first , hard requiredment is it must be running kubecost version 1.96 which was release few day ago as the blog is drafted. 

Once u deploy the kubecost with just setting up with the kubetoken. Access the kubecost dashboard by port forward to port 9090 and go to the setting. 

scroll until u find this option 

Click update and fill the details 

After click on submit, wait for sometimes and the data on cloud integration to be populated. 

So here come the hard way , all the component in easier method will be converted into line of code start with creating a kubernetes secret containing all the details 

resource "kubernetes_secret" "kubecost_sec" {

  metadata {

    name      = "kubecost-sec"

    namespace = kubernetes_namespace.kubecost.metadata[0].name

  }

  data = {

      "cloud-integration.json" = "\r\n{\r\n    \"azure\": [\r\n        {\r\n          \"azureSubscriptionID\": \"${var.sub-id}\",\r\n          \"azureStorageAccount\": \"${var.saname}\",\r\n          \"azureStorageAccessKey\": \"${var.sakey}\",\r\n          \"azureStorageContainer\": \"${var.sacontainer}\",\r\n          \"azureContainerPath\": \"${var.sapath}\",\r\n          \"azureCloud\": \"${var.azcloud}\"\r\n        }\r\n    ]\r\n}"

   

  }

  type = "Opaque"

}

on the helm value , point the cloud integration value to the secret that been created . 

  set {

    name  = "kubecostProductConfigs.cloudIntegrationSecret"

    value = kubernetes_secret.kubecost_sec.metadata[0].name

  }

Both of the method will get you to have kubecost cloud integration with azure 

now it is all concluded, list of the reference as below 

1. Deploy AKS + Kubecost with Terraform - Code it Yourself... (mendible.com)

2. Azure Config – Kubecost

3. poc-common-configurations/cloud-integration.json at main · kubecost/poc-common-configurations (github.com)

4. Kubernetes secret with json · Issue #1801 · hashicorp/terraform-provider-kubernetes (github.com)

5.Cost governance with Kubecost - Cloud Adoption Framework | Microsoft Docs

6. Kubecost team who has provide a good insight in their slack to the community 

enjoy the learning and thanks for reading, the sample code is available on my github 

Kubecost on AKS Part 01

 Hi, for this blog post , i will share my journey on enabling kubecost with cloud integration on my kubernetes cluster . 

Prerequisite 

1. Kubernetes cluster ; my case will be AKS 

2. Azure AD SPN 

3. Storage Account 

4.Tools
 4.1 Helm
 4.2 Terraform(Optional)

5. Ingress Controller (Optional)

Step 1- Get kubecost running on your AKS cluster 

There are few guide available even on microsoft document tation , but i will share on utilizing helm chart provider in terraform. 

 1.1 - Create a kubecost namespace

 resource "kubernetes_namespace" "kubecost" {

  metadata {

    annotations = {

      name = "kubecost-annotation"

    }

    name = "kubecost"

  }

}

1.2 install the kubecost helm chart 

resource "helm_release" "kubecost-helm" {

   name       = "kubecost"

   repository = "https://kubecost.github.io/cost-analyzer/"

   chart      = "cost-analyzer"

   namespace = kubernetes_namespace.kubecost.metadata[0].name

  

    set {

    name  = "kubecostToken"

    value = "aGVsbUBrdWJlY29zdC5jb20=xm343yadf98"

    #value = "YWJkdWwubXVuaXI5NEBvdXRsb29rLmNvbQ==xm343yadf98"

   } 

}

For kubetoken , u may use either one as both are working as per my testing

Then u can apply the terraform code and if you want to add  other parameter , you may add other like what has been done here - kubecost helm

set {

    name  = "kubecostProductConfigs.clusterName"

    value = var.aks-name

  }

  set {

    name  = "ingress.enabled"

     value = true

   }

  set {

     name  = "ingress.hosts"

     value = "kubecost.munirtajudin.xyz"

   }

 

  #Set the currency

 set {

    name  = "kubecostProductConfigs.currencyCode"

    value = "USD"

  }

# 

  # Set the region

  set {

    name  = "kubecostProductConfigs.azureBillingRegion"

    value = "US"

  }

  

  # Generate a secret based on the Azure configuration provided below

  set {

    name  = "kubecostProductConfigs.createServiceKeySecret"

    value = true

  }

  # Azure Subscription ID

  set {

    name  = "kubecostProductConfigs.azureSubscriptionID"

    value = var.sub-id

  }

  # Azure Client ID

  set {

    name  = "kubecostProductConfigs.azureClientID"

    value = var.client-id

  }

  # Azure Client Password

  set {

    name  = "kubecostProductConfigs.azureClientPassword"

    value = var.client-sec

  }

  # Azure Tenant ID

  set {

    name  = "kubecostProductConfigs.azureTenantID"

    value = var.tenant-id

  }

you may do port forward to 9090 and access the kubecost dashboard. do give it a few minute because kubecost will take sometimes to collect the metric. 

see you in part 2 where it will focus on enabling cloud integration with azure cost management. 

thanks for reading and do provide feedback if any 

Nginx Ingress on AKS via Terraform

 let's  continue playing with AKS and this time around , i will be sharing the how to install nginx ingress controller on AKS, this is not new but it is more on my sharing and finding during the process. 

First of all ,  you need to have AKS to begin with , and an app to be publish , it can be a test app some sample app or your own app. i will be utilizing terraform to deploy it . 

Prerequisite 

1. AKS cluster 

2.Terraform installed 

3.Helm installed. 

Some special requirement that found during the deployment as below 

i. Public IP for nginx ingress must be in precreated RG that hold the AKS resource ; MC_rg_aks-name_region

ii. It does not need SPN like traefik and agic deployment 

so now we can deploy nginx ingress into our aks, here is my helm setup in terraform 

resource "helm_release" "nginx" {

  depends_on = [kubernetes_namespace.nginx]

  namespace = kubernetes_namespace.nginx.metadata[0].name

  name       = "nginx-ingress-controller"

  repository = "https://kubernetes.github.io/ingress-nginx"

  chart      = "ingress-nginx"

  timeout    = 300

  set {

    name  = "controller.service.type"

    value = "LoadBalancer"

  }  

  set {

    name  = "controller.service.loadBalancerIP"

    value = azurerm_public_ip.ngxip.ip_address

  }

  set {

    name  = "controller.service.annotations.service\\.beta\\.kubernetes\\.io/azure-load-balancer-health-probe-request-path"

    value = "/healthz"

  }

}

 here is how i cater the public ip resource group 

resource "azurerm_public_ip" "ngxip" {

  name                = var.ngxip

  resource_group_name = "MC_${var.aks-rg}_${var.aks-name}_${var.loc}"

  location            = var.loc

  allocation_method   = "Static"

  sku = "Standard"

  sku_tier = "Regional"

  availability_zone = "Zone-Redundant"

}

After you apply your terraform code, then you can test your nginx using the sample app that i used for agic just need to change some parameter on the code 

and here is the result 

the format may be off but it does conclude that our nginx is running fine. 

Thanks for reading and see you on next post 

here is the refence link that i refer while figuring this 

1. GitHub - kubernetes/ingress-nginx: Ingress-NGINX Controller for Kubernetes

2. Create an ingress controller in Azure Kubernetes Service (AKS) - Azure Kubernetes Service | Microsoft Docs

3.Service | Kubernetes

AKS AGIC Addon via Azure Portal

 The Previous post on enabling agic was done via terraform but let make it easier to follow which by using Azure Portal. 

Please make sure you have all this component running . 

1. AKS 

2. Application Gateway (needed for utilizing the existing app gw )

Step One . Go to AKS > Networking > Application Gateway ; enable it  

Step Two, Monitor the deployment 

Step Three, Test AGIC using sample app 

kubectl apply -f https://raw.githubusercontent.com/Azure/application-gateway-kubernetes-ingress/master/docs/examples/aspnetapp.yaml 

Step Four, Check the ingress creation and backend pool

Sample app running 

Thanks for reading and see you .. 

Deploy Arc Data Controller - Az command

 Here are some az command to deploy your azure arc data controller 

Prerequisite :

1. Az cli is installed 

2. Storage class ext4 deployed 

3. AKS HCI cluster deployed 

4. kubectl is installed 

here is the command 

az extension add --name k8s-extension

az extension add --name connectedk8s

az extension add --name k8s-configuration

az extension add --name customlocation

$namespace = "arc-dc"

$location = "arcdc" #custom location 

$region = "eastus"          #Azure region 

$subscription = "subname"   #subscription name 

#$subscriptionid = "abcdefg" #subscription id 

$rg = "RG"                  #RG name 

$cluster = "cluster01"      #cluster name 

az login 

az account set --subscription "subname"

az connectedk8s connect --subscription $subscription --resource-group $rg --name $cluster --location $region

az connectedk8s enable-features -n $cluster -g $rg --features cluster-connect custom-locations

az k8s-extension create --cluster-name $cluster --resource-group $rg --name $namespace --cluster-type connectedClusters --extension-type microsoft.arcdataservices --auto-upgrade true --scope cluster --release-namespace $namespace --config Microsoft.CustomLocation.ServiceAccount=sa-arc-bootstrapper

$akshciid = az connectedk8s show -n $cluster -g $rg  --query id -o tsv

$extensionid = az k8s-extension show --name arc-dc --cluster-type connectedClusters -c $cluster -g $rg  --query id -o tsv

az customlocation create --resource-group $rg --name $location --namespace $namespace --host-resource-id $akshciid --cluster-extension-ids $extensionid --location $region

#use portal to create data controller 

#Example:

#az arcdata dc create --profile-name azure-arc-aks-premium-storage --k8s-namespace arc --name arc --subscription xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx --resource-group my-resource-group --location eastus --connectivity-mode direct --use-k8s

#check deployment status 

kubectl get datacontrollers --namespace arc-dc

thanks for reading 

AKS Ingress with Traefik via Terraform

 Hi All, 

continue from my previous post on AKS AGIC addon, you can utilize azure application gateway to do so but if you like kind of open source project , from what i know , there are two option which is traefik and nginx . 

As for this post , i will be explaining on how to deploy traefik in aks by utilizing helm chart for traefik and terraform for easier deployment. 

These are some component and tools involved in this deployment 

1. Azure Kubernetes Service 

2. Service Principle from Azure AD 

3. Traefik helm chart 

4. Terraform 

Let start, At first please make sure u have azure virtual network for AKS to "sit" in or u may deploy a new one 

module "vnet00" {

  source        = "../modules/network/vnet"
  vnet-location = "eastus"
  vnet-rg       = "DEV-VNET-EASTUS"
  vnet-name     = "DEV-VNET-EASTUS"
  vnet-range    = ["172.25.0.0/20"]
  subnets-name  = ["subnet01", "aks-subnet"]
  subnets-range = ["172.25.0.0/24", "172.25.1.0/24"]
}

following with Service Principle 

module "aks-spn" {
source      = "../modules/aad-spn"
spn-name    = "AKSDEV-SPN"
secret-name = "aksdev-spn-secret"

}

we gonna use of reference for this SPN later ... after both of this is deployed, AKS can be deployed referencing to both of the output from this vnet and spn module. 

module "myaks" {
source = "../modules/k8s/dev"
aks-name = "aks01"
aks-dns = "aks01-dns"
aks-version = "1.24.0"
aks-region = "eastus"
aks-subnet-id = module.vnet00.vnet_subnets.1
admin = "adminlogin"
ssh = "ssh-rsa AAAAB3N"
winpass = "P@ssw0rd1234"
client_id = module.aks-spn.client_id
client_sec = module.aks-spn.client_secret
depends_on = [module.aks-spn]
}

then arrive to the main topic of today which is traefik deployment 

module "traefik" {

  source    = "../modules/network/traefik"
  loc       = "eastus"
  aks-rg    = module.myaks.aks-rg.id
  aks-name  = module.myaks.aks-name
  aks-spnid = module.aks-spn.object_id-entapp
  depends_on = [module.myaks]

}

this module will do few thing like creating public ip for traefik and assign permission accordingly . Traefik public ip will be bind to the same load balancer that aks deployed 

permission 

then u may do terraform apply and once completed , you may browse your traefik dashboard by http://<yourtraefikoublicip>:9000/dashbaord nad here is mine

your deployment is completed and u may use traefik as ingress for your app in aks 

Thanks for reading , please leave a comment if you have some doubt and here are some reference that i use to complete traefik deployment 

1. https://stackoverflow.com/questions/69269097/unable-to-pass-service-annotations-when-deploying-helm-chart-via-terraform

2. https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release 

AKS AGIC addon via Terrafrom

 Hello All, 

my testing has been alot with aks lately  and surely i cannot run away on setting up an ingress controller for all my test application on aks , So i would like to share my journey on setting Appling gateway Ingress controller (AGIC) using aks addon via terraform. 

There are two method on setting AGIC which is via aks addon or helm chart. There are pro and cons for each method but if your Application Gateway just for AKS to be use , using addon will be better. 

More details on the different can be read from here - Microsoft Docs 

I will be using terraform module that i created for this example . 

 Component Involved 

1. Azure Kubernetes Service (AKS)

2. Application Gateway 

3. Service Principle 

4. Azure Virtual Network 

you may download my module to test but we will start as below 

1.  Virtual Network creation 

module "vnet00" {
  source             = "./modules/network/vnet"
  vnet-name       = "vNet-00-aks"
  vnet-range       = ["192.168.16.0/24", "172.16.0.0/22"]
  subnets-name  = ["subnet01", "aks-subnet", "ApplicationGatewaySubnet"]
  subnets-range  = ["192.168.16.0/28", "172.16.0.0/24", "192.168.16.128/26"]
}

2. SPN creation 

module "aks-spn" {
  source             = "./modules/aad-spn"
  spn-name        = "AKSAGIC-SPN"
  secret-name    = "akstfk-spn-secret"
}

3. Application Gateway 

module "appgw01" {
  source               = "./modules/network/appgw"
  agsnetname       = module.vnet00.vnet_subnets_name.2
  agsnetid            = module.vnet00.vnet_subnets.2
  
  depends_on = [module.vnet00]
}

4. AKS 

 module "myaks" {
  source        = "./modules/k8s/dev-appgw"
  aks-name      = "aks01"
  aks-dns       = "aks01-dns"
  aks-version   = "1.23.3"
  aks-subnet-id = module.vnet00.vnet_subnets.1
  admin         = "username"
  ssh           = "ssh-rsa xx"
  winpass       = "password"
  client_id     = module.aks-spn.client_id
  client_sec    = module.aks-spn.client_secret
  aks-spn = module.aks-spn.object_id-entapp
  appgwid = module.appgw01.appgwid
  appgwrg = module.appgw01.appgwrg
  #appgwname = module.appgw01.appgwname
  #appgwnetid =  module.vnet00.vnet_subnets.2
  #appgwnetcidr = xxx
  depends_on = [module.aks-spn,module.appgw01] 
}

Notice there are some line  is commented. this is because u can either use the existing Application gateway as AGIC or You let the AKS create it for you but i choose on the first one. 

Reason for this structure because the same spn will be use in AKS setup and for AGIC which is the read permission on AGIC resource group and contributor to AGIC. 

after all the component has been deployed, you may download the kubeconfig file and test with this command 

1.  kubectl apply -f https://raw.githubusercontent.com/Azure/application-gateway-kubernetes-ingress/master/docs/examples/aspnetapp.yaml

2. Kubectl get ingress 

Then the sample aspnet app details will appear as the setup was done successfully. 

Link to my terraform github repo is here with sample module call  -  GitHub - munir94/TFLAB

That all for now , im planning to cover traefik and nginx ingress deployment also in future, thanks for reading and you may leave comment for enquiry or improvement . 

Azure Arc Data Controller on Azure Stack HCI - ext4 storage creation

 Hi all, continue on hybrid deployment for related topic on Azure Arc, this post will be focusing on creating a storage with fstype: ext4 in order to complete Azure Arc Data Controller deployment. 

Some common issue if this requirement is not meet is , it will cause the agent cannot be initialize. example here - click here 

The official guide is available at the end of the post but will show here how i made it possible in my test lab . 

1. Locate the config on yaml for default storage class. 

2. Take note on the group and hostname using tools or run this command - kubectl get storageclass default -o yaml

3. Three things needed to be apply in your new yaml to create that fstype:ext4 is 

  3.1 Name : this will appear as the name of your  new storage class 
  3.2 Container ; the file path for storing your persistence volume

  3.3 Group

  3.4 Hostname ; this is created based on your pre-stage your dns on deploying Azure Stack HCI at the beginning  

4. for my case, i did not create another custom storage container , i just use the same container as my default storage class 

5. you may deploy by using kubectl apply -f  /examplepathext4.yaml

6. After that you may try to deploy data controller again and put the name of your new storage class . 

reference link is here 

1. Use the AKS on Azure Stack HCI disk Container Storage Interface (CSI) drivers - AKS-HCI | Microsoft Docs

2. Deploy Azure Arc enabled data services in Azure Kubernetes Service (AKS) on Azure Stack HCI - AKS-HCI | Microsoft Docs

hope this will help in your journey to with azure arc data controller , thanks for reading and see you again 

 

empty_dir = {} is not expected here - Terraform

 Hi all, i did some conversion from yaml to tf using tool called k2tf . As part of code conversion and i use to be in my terraform module. so if you have this line with "empty_dir = {}" terraform cannot read as it is not part of the syntax. 

as first i was just commenting the line like below 

 spec {

        volume {

          name      = "tmp-dir"

          #empty_dir = {}

        }

turn out everytime i run terraform plan it will be part of changes . 

so i was trying to ignore this line code using a ignore_changes function but could find the right class to put into. so i look back the line like picture above and change  from empty_dir = {} to empty_dir {} and it workkkk.. 

so my lesson to me here is other than commenting, i should also pay attention to syntax to replace it .

btw if any of you interested in using k2tf, i already made container image , you may pull it using this command - docker pull pra8/k2tf 

* i have just compile the tool and make it a container 

that all for now and see you guys on next post

AKS HCI Access Token - Azure Arc

Hello All, 

Having to play with kubernetes or K8s is great plus point and now, public cloud provider like microsoft has their manages service k8 offering; AKS to be running on Azure Stack HCI . 

As service is now running on your datacenter, but with azure arc integration , it also can be view from your azure portal with some parameter need to be pass to get the view 

i was abit struggling to find the token in order to view the k8 properties on portal but after sometimes, i manage to pull it. i will share the step here so it will be easier for you to get that token to be use in few place . Here is what you need

1. AKS running on HCI with azure arc enable 

2. try and error ; but i will make easier 

For me, i use free software to access my k8s cluster which is Lens , so browse through the secret in the k8 and look for some key word, one thing that caught my eyes is azure arc, so i click to look into the content of the secret. 

so i unhide the token and copy over the token and paste over in azure portal like the first picture and walla , all the namespace appear. 

i have also incorperate the token into my kube config because i want to use the same kube config to acess k8 either via lens or k8-dashbaord. here how u add into the existing kube config 

add the extra line with token : <your token> and pay attention on the allignment due to format. 

that all for now, thanks for reading and stay safe.