Using AWS ECS Fargate with Cloudfront and WAF

Introduction

I have already presented the concepts of AWS ECS in my previous post, which you can review for more information. In this article, I will guide you on how to deploy a docker image with AWS ECS on Cloudfront using WAF, monitored by Cloudwatch. Additionally, we will setup alerts to automatically send emails and notifications to Telegram when a WAF rule is matched.

Prerequisites

You can continue using the NestJS source code that I guided you through in previous articles or use your own project. After pushing the docker image to ECR, please proceed to the following sections.


Detail

The workflow will be as follows:

  1. Requests are sent to Cloudfront.
  2. Here, the rules in WAF take effect to block requests with security issues, preventing them from reaching our Load Balancer.
  3. Cloudwatch will aggregate results based on metrics; if the required threshold is reached, it will send an email and a notification to Telegram for alerting.
  4. If the request has no issues, Cloudfront will attach a secret header to that request and point it to the Load Balancer; only then will the request be processed to return the corresponding response.


First, create a .env file with the following information:

CDK_DEFAULT_REGION = ap-southeast-1
CDK_US_REGION = us-east-1

BUCKET = <BUCKET>
IMAGE = <IMAGE>

CUSTOM_HEADER = <CUSTOM_HEADER>
VERIFY_SECRET = <VERIFY_SECRET>

TELEGRAM_TOKEN = <TELEGRAM_TOKEN>
CHAT_ID = <CHAT_ID>
EMAIL = <EMAIL>


The information regarding CDK_DEFAULT_REGION, CDK_US_REGION, BUCKET, IMAGE, CUSTOM_HEADER, and VERIFY_SECRET is similar to what I mentioned in previous posts; please replace them with your corresponding values.

TELEGRAM_TOKEN, CHAT_ID: This is the Telegram information used to send messages.

EMAIL: Any email address of yours.


Use AWS CDK to create the file lib/ecs-fargate-stack.ts

import * as cdk from "aws-cdk-lib"
import * as ec2 from "aws-cdk-lib/aws-ec2"
import * as ecs from "aws-cdk-lib/aws-ecs"
import * as ecs_patterns from "aws-cdk-lib/aws-ecs-patterns"
import * as iam from "aws-cdk-lib/aws-iam"
import * as s3 from "aws-cdk-lib/aws-s3"
import * as cr from "aws-cdk-lib/custom-resources"
import { Construct } from "constructs"

export class EcsFargateStack extends cdk.Stack {
  loadBalancer: cdk.aws_elasticloadbalancingv2.ApplicationLoadBalancer

  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props)

    const vpc = new ec2.Vpc(this, "NestVpc", {
      maxAzs: 2,
      natGateways: 0,
      subnetConfiguration: [
        { name: "Public", subnetType: ec2.SubnetType.PUBLIC },
      ],
    })

    const imageUri = process.env.IMAGE || ""
    const bucket = process.env.BUCKET || ""
    const originVerifySecret = process.env.VERIFY_SECRET || ""

    const fargateService =
      new ecs_patterns.ApplicationLoadBalancedFargateService(
        this,
        "NestService",
        {
          vpc,
          cpu: 256,
          memoryLimitMiB: 512,
          assignPublicIp: true,
          circuitBreaker: { rollback: true },
          capacityProviderStrategies: [
            {
              capacityProvider: "FARGATE_SPOT",
              weight: 1,
            },
          ],
          taskImageOptions: {
            image: ecs.ContainerImage.fromRegistry(imageUri),
            containerPort: 3000,
            environment: {
              REGION: process.env.CDK_DEFAULT_REGION || "",
              BUCKET: bucket,
              VERIFY_SECRET: originVerifySecret,
            },
          },
          healthCheckGracePeriod: cdk.Duration.seconds(120),
        },
      )

    fargateService.taskDefinition.addToExecutionRolePolicy(
      new iam.PolicyStatement({
        actions: [
          "ecr:GetAuthorizationToken",
          "ecr:BatchCheckLayerAvailability",
          "ecr:GetDownloadUrlForLayer",
          "ecr:BatchGetImage",
        ],
        resources: ["*"],
      }),
    )

    fargateService.targetGroup.configureHealthCheck({
      path: "/health",
      port: "3000",
      healthyThresholdCount: 2,
      unhealthyThresholdCount: 5,
      interval: cdk.Duration.seconds(60),
      timeout: cdk.Duration.seconds(5),
    })

    const myBucket = s3.Bucket.fromBucketName(this, "ExistingBucket", bucket)
    fargateService.taskDefinition.addToTaskRolePolicy(
      new iam.PolicyStatement({
        actions: ["s3:PutObject", "s3:GetObject", "s3:ListBucket"],
        resources: [
          myBucket.bucketArn,
          myBucket.arnForObjects("*"),
        ],
      }),
    )

    const getPrefixList = new cr.AwsCustomResource(this, "GetCFPrefixList", {
      onUpdate: {
        service: "EC2",
        action: "describeManagedPrefixLists",
        parameters: {
          Filters: [
            {
              Name: "prefix-list-name",
              Values: ["com.amazonaws.global.cloudfront.origin-facing"],
            },
          ],
        },
        region: this.region,
        physicalResourceId: cr.PhysicalResourceId.of("cf-prefix-list-lookup"),
      },
      policy: cr.AwsCustomResourcePolicy.fromSdkCalls({
        resources: cr.AwsCustomResourcePolicy.ANY_RESOURCE,
      }),
    })

    this.loadBalancer = fargateService.loadBalancer
    const cloudFrontPrefixListId = getPrefixList.getResponseField(
      "PrefixLists.0.PrefixListId",
    )

    this.loadBalancer.connections.allowFrom(
      ec2.Peer.prefixList(cloudFrontPrefixListId),
      ec2.Port.tcp(80),
      "Allow HTTP traffic from CloudFront only",
    )
  }
}

  • Parts such as creating the vpc, fargateService, granting permissions for the task, health checks, and granting permissions for the S3 Bucket are also similar to the previous article I used.
  • GetCFPrefixList: We will still only allow Cloudfront's prefix list to access the Load Balancer.
  • The only new part is saving this.loadBalancer to be used when creating the next stack.


Next is the file lib/telegram-notifier-lambda.ts used to handle sending notifications to Telegram.

import { SNSEvent, SNSHandler } from "aws-lambda"
import * as https from "https"

const TELEGRAM_TOKEN = process.env.TELEGRAM_TOKEN || ""
const CHAT_ID = process.env.CHAT_ID || ""

export const handler: SNSHandler = async (event: SNSEvent): Promise<void> => {
  try {
    const snsMessage = event.Records[0].Sns.Message
    let alarmName = "Unknown Alarm"
    let newState = "UNKNOWN"
    let reason = snsMessage

    try {
      const parsedMessage = JSON.parse(snsMessage)
      alarmName = parsedMessage.AlarmName || alarmName
      newState = parsedMessage.NewStateValue || newState
      reason = parsedMessage.NewStateReason || reason
    } catch (e) {
      console.log("SNS Message is not JSON, sending as raw text.")
    }

    const text =
      `🚨 *WAF Security Alert* 🚨\n\n` +
      `*Alarm:* ${alarmName}\n` +
      `*Status:* ${newState}\n` +
      `*Reason:* ${reason}\n\n` +
      `👉 [Mở AWS Console](https://console.aws.amazon.com/wafv2/home)`

    const data = JSON.stringify({
      chat_id: CHAT_ID,
      text: text,
      parse_mode: "Markdown",
    })

    const options: https.RequestOptions = {
      hostname: "api.telegram.org",
      port: 443,
      path: `/bot${TELEGRAM_TOKEN}/sendMessage`,
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        "Content-Length": Buffer.byteLength(data),
      },
    }

    await sendTelegramRequest(options, data)
    console.log("Notification sent successfully")
  } catch (error) {
    console.error("Error sending notification:", error)
    throw error
  }
}

function sendTelegramRequest(
  options: https.RequestOptions,
  data: string,
): Promise<void> {
  return new Promise((resolve, reject) => {
    const req = https.request(options, (res) => {
      let responseBody = ""
      res.on("data", (chunk) => (responseBody += chunk))
      res.on("end", () => {
        if (res.statusCode && res.statusCode >= 200 && res.statusCode < 300) {
          resolve()
        } else {
          reject(
            new Error(
              `Telegram API error: ${res.statusCode} - ${responseBody}`,
            ),
          )
        }
      })
    })

    req.on("error", (e) => reject(e))
    req.write(data)
    req.end()
  })
}

  • The main content of this part is simply using Telegram information to call the API and send messages to the group chat.
  • You can freely change the content in the text section as appropriate; this is the message content that will show up on Telegram.


Next is the file lib/waf-cloudfront-cloudwatch-stack.ts

import * as cdk from "aws-cdk-lib"
import * as cloudfront from "aws-cdk-lib/aws-cloudfront"
import * as origins from "aws-cdk-lib/aws-cloudfront-origins"
import * as cloudwatch from "aws-cdk-lib/aws-cloudwatch"
import * as cw_actions from "aws-cdk-lib/aws-cloudwatch-actions"
import * as lambda from "aws-cdk-lib/aws-lambda"
import * as nodejs from "aws-cdk-lib/aws-lambda-nodejs"
import * as sns from "aws-cdk-lib/aws-sns"
import * as subs from "aws-cdk-lib/aws-sns-subscriptions"
import * as wafv2 from "aws-cdk-lib/aws-wafv2"
import { Construct } from "constructs"
import * as path from "path"

export class WafCloudfrontCloudwatchStack extends cdk.Stack {
  readonly WAF_NAME = "AppWebAcl"
  readonly WAF_METRIC_NAME = "AppWafMetric"
  readonly RATE_LIMIT_RULE_NAME = "GeneralRateLimitRule"

  constructor(scope: Construct, id: string, props: WafCloudfrontCloudwatchProps) {
    super(scope, id, props)

    const managedRules = [
      { priority: 1, name: "AWSManagedRulesCommonRuleSet" },
      { priority: 2, name: "AWSManagedRulesSQLiRuleSet" },
      { priority: 3, name: "AWSManagedRulesAmazonIpReputationList" },
      { priority: 4, name: "AWSManagedRulesKnownBadInputsRuleSet" },
    ]

    const webAcl = new wafv2.CfnWebACL(this, "AppWaf", {
      name: this.WAF_NAME,
      defaultAction: { allow: {} },
      scope: "CLOUDFRONT",
      visibilityConfig: {
        cloudWatchMetricsEnabled: true,
        metricName: this.WAF_METRIC_NAME,
        sampledRequestsEnabled: true,
      },
      rules: [
        ...managedRules.map((rule) => ({
          name: rule.name,
          priority: rule.priority,
          statement: {
            managedRuleGroupStatement: {
              vendorName: "AWS",
              name: rule.name,
            },
          },
          overrideAction: { none: {} },
          visibilityConfig: {
            cloudWatchMetricsEnabled: true,
            metricName: `${rule.name}Metric`,
            sampledRequestsEnabled: true,
          },
        })),
        {
          name: this.RATE_LIMIT_RULE_NAME,
          priority: 5,
          action: { block: {} },
          statement: {
            rateBasedStatement: {
              limit: 10,
              aggregateKeyType: "IP",
              // evaluationWindowSec: 60, // default 300s = 5m
            },
          },
          visibilityConfig: {
            sampledRequestsEnabled: true,
            cloudWatchMetricsEnabled: true,
            metricName: this.RATE_LIMIT_RULE_NAME,
          },
        },
      ],
    })

    const customHeader = process.env.CUSTOM_HEADER || ""
    const verifySecret = process.env.VERIFY_SECRET || ""
    const cfDist = new cloudfront.Distribution(this, "AppDistribution", {
      defaultBehavior: {
        origin: new origins.LoadBalancerV2Origin(props.loadBalancer, {
          protocolPolicy: cloudfront.OriginProtocolPolicy.HTTP_ONLY,
          customHeaders: { [customHeader]: verifySecret },
        }),
        viewerProtocolPolicy: cloudfront.ViewerProtocolPolicy.REDIRECT_TO_HTTPS,
        allowedMethods: cloudfront.AllowedMethods.ALLOW_ALL,
        cachePolicy: cloudfront.CachePolicy.CACHING_DISABLED,
        originRequestPolicy:
          cloudfront.OriginRequestPolicy.ALL_VIEWER_EXCEPT_HOST_HEADER,
      },
      webAclId: webAcl.attrArn,
    })

    cfDist.node.addDependency(webAcl)

    const telegramLambda = new nodejs.NodejsFunction(this, "TelegramNotifier", {
      runtime: lambda.Runtime.NODEJS_LATEST,
      entry: path.join(__dirname, "telegram-notifier-lambda.ts"),
      environment: {
        TELEGRAM_TOKEN: process.env.TELEGRAM_TOKEN || "",
        CHAT_ID: process.env.CHAT_ID || "",
      },
      bundling: { minify: true },
      memorySize: 512,
    })

    const topic = new sns.Topic(this, "WafAlertTopic")
    topic.addSubscription(new subs.LambdaSubscription(telegramLambda))
    topic.addSubscription(
      new subs.EmailSubscription(process.env.EMAIL || "", {
        json: false,
      }),
    )

    const bruteForceAlarm = new cloudwatch.Alarm(this, "BruteForceAlarm", {
      metric: new cloudwatch.Metric({
        namespace: "AWS/WAFV2",
        metricName: "BlockedRequests",
        dimensionsMap: {
          WebACL: this.WAF_NAME,
          Rule: this.RATE_LIMIT_RULE_NAME,
        },
        statistic: "Sum",
        period: cdk.Duration.minutes(1),
      }),
      threshold: 1,
      evaluationPeriods: 1,
      datapointsToAlarm: 1,
      comparisonOperator:
        cloudwatch.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
      treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING,
    })

    bruteForceAlarm.addAlarmAction(new cw_actions.SnsAction(topic))

    new cdk.CfnOutput(this, "CloudFrontDomain", {
      value: `https://${cfDist.distributionDomainName}`,
    })
  }
}

interface WafCloudfrontCloudwatchProps extends cdk.StackProps {
  readonly loadBalancer: cdk.aws_elasticloadbalancingv2.ApplicationLoadBalancer
}

  • wafv2.CfnWebACL: Creates a Web Access Control List for the Cloudfront scope including free Managed Rules developed by AWS itself. Adding them to the Web ACL will provide your NestJS application with a comprehensive "shield" from the exploitation layer to the behavioral layer.
    1. AWSManagedRulesCommonRuleSet
      • This is the most important and mandatory rule set.
      • Function: Protects against common vulnerabilities in the OWASP Top 10 category. It blocks requests with unusual sizes, strange control characters, or file system intrusion attempts (Local File Inclusion).
    2. AWSManagedRulesSQLiRuleSet
      • Protects your data layer from code injection attacks.
      • Function: Detects and stops SQL Injection (SQLi) attempts in the Query String, Body, or Header. Very important if your NestJS app connects to MySQL, PostgreSQL, or MongoDB.
    3. AWSManagedRulesAmazonIpReputationList
      • Uses "collective intelligence" from Amazon's massive network.
      • Function: AWS collects a list of IP addresses acting as Botnets, malware distribution sources, or IPs that have previously attacked other AWS customers. This rule automatically blocks these "tainted" IPs before they reach your CloudFront.
    4. AWSManagedRulesKnownBadInputsRuleSet
      • Prevents automated vulnerability scanning tools.
      • Function: Hackers often use automated scripts to find .env files, /admin, or sensitive configuration files. This rule set identifies patterns of common attack tools and blocks them immediately.
    5. Custom rule GeneralRateLimitRule
      • Used to block IP Addresses that access too many times within a fixed period. Here I use 10 requests in 5 minutes for easy testing; you can change it according to your needs.
      • I have specifically explained how this rule works in previous posts; you can review them for more information.
  • cfDist: This creates the Cloudfront distribution, which needs the loadBalancer created from the previous stack, and adds a dependency to ensure Cloudfront is only created after the WAF has been created.
  • telegramLambda: Note to update the path to the telegram file created above.
  • sns.Topic: Creates an SNS Topic and adds 2 subscriptions so that when a message is sent to the topic, it will trigger an email and call the lambda function to send a message to Telegram.
  • cloudwatch.Alarm:
    • metricName: "BlockedRequests" only triggers when a request is blocked.
    • The parameters below mean it will calculate the total number of blocked requests in 1 minute; if it is greater than or equal to 1, it will alert. This is the configuration I use for testing; you can update it according to your needs.
      • statistic: "Sum"
      • period: cdk.Duration.minutes(1)
      • threshold: 1
      • comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD
    • addAlarmAction: When the alert is activated, it will send a message to the SNS Topic.


You can use AWS Cli to see the managed rules that AWS supports for Cloudfront.

$ aws wafv2 list-available-managed-rule-groups --scope CLOUDFRONT --region us-east-1 --query 'ManagedRuleGroups[*].Name'
[
    "AWSManagedRulesCommonRuleSet",
    "AWSManagedRulesAdminProtectionRuleSet",
    "AWSManagedRulesKnownBadInputsRuleSet",
    "AWSManagedRulesSQLiRuleSet",
    "AWSManagedRulesLinuxRuleSet",
    "AWSManagedRulesUnixRuleSet",
    "AWSManagedRulesWindowsRuleSet",
    "AWSManagedRulesPHPRuleSet",
    "AWSManagedRulesWordPressRuleSet",
    "AWSManagedRulesAmazonIpReputationList",
    "AWSManagedRulesAnonymousIpList",
    "AWSManagedRulesBotControlRuleSet",
    "AWSManagedRulesATPRuleSet",
    "AWSManagedRulesACFPRuleSet",
    "AWSManagedRulesAntiDDoSRuleSet"
]


Next is the file lib/ecs-fargate-waf-cloudfront-stack.ts to create 2 stacks in 2 different regions.

import * as cdk from "aws-cdk-lib/core"
import { EcsFargateStack } from "./ecs-fargate-stack-waf"
import { WafCloudfrontCloudwatchStack } from "./waf-cloudfront-cloudwatchstack"

export const EcsFargateWafCloudfrontStack = (app: cdk.App) => {
  const eksManifestStack = new EcsFargateStack(app, "EcsFargateStack", {
    env: { region: process.env.CDK_DEFAULT_REGION },
  })
  const wafCloudfrontStack = new WafCloudfrontCloudwatchStack(
    app,
    "WafCloudfrontCloudwatchStack",
    {
      env: { region: process.env.CDK_US_REGION },
      crossRegionReferences: true,
      loadBalancer: eksManifestStack.loadBalancer,
    },
  )
  wafCloudfrontStack.addDependency(eksManifestStack)
}


Update the file bin/aws-cdk.ts

#!/usr/bin/env node
import 'dotenv/config';
import * as cdk from "aws-cdk-lib/core"
import { EcsFargateWafCloudfrontStack } from '../lib/ecs-fargate-waf-cloudfront-stack';

const app = new cdk.App()
EcsFargateWafCloudfrontStack(app)


You need to run bootstrap again first because we need to deploy stacks into 2 different regions.

$ cdk bootstrap --all
View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/peuyc1mcoznltz2a0yex2n1ht
 ⏳  Bootstrapping environment aws://347116763491/ap-southeast-1...
 ⏳  Bootstrapping environment aws://347116763491/us-east-1...
Trusted accounts for deployment: (none)
Trusted accounts for lookup: (none)
Using default execution policy of 'arn:aws:iam::aws:policy/AdministratorAccess'. Pass '--cloudformation-execution-policies' to customize.
✅  Environment aws://347116763491/ap-southeast-1 bootstrapped (no changes).
Trusted accounts for deployment: (none)
Trusted accounts for lookup: (none)
Using default execution policy of 'arn:aws:iam::aws:policy/AdministratorAccess'. Pass '--cloudformation-execution-policies' to customize.
✅  Environment aws://347116763491/us-east-1 bootstrapped (no changes).


After deploying, the result looks like this:

✨  Synthesis time: 13.87s
EcsFargateStack
EcsFargateStack: deploying...

✅  EcsFargateStack
✨  Deployment time: 1.08s
Outputs:
EcsFargateStack.NestServiceLoadBalancerDNSDB906E33 = EcsFar-NestS-ouIAbwyzmqBs-852307268.ap-southeast-1.elb.amazonaws.com  
EcsFargateStack.NestServiceServiceURLA979D4F3 = http://EcsFar-NestS-ouIAbwyzmqBs-852307268.ap-southeast-1.elb.amazonaws.com
Stack ARN:
arn:aws:cloudformation:ap-southeast-1:347116125752:stack/EcsFargateStack/75bf05f0-1a13-11f1-a6ae-06660817cb33
✨  Total time: 14.95s

✅  WafCloudfrontStack
✨  Deployment time: 31.94s
Outputs:
WafCloudfrontStack.CloudFrontDomain = https://d1h41kt9t5n8wk.cloudfront.net
Stack ARN:
arn:aws:cloudformation:us-east-1:347116125752:stack/WafCloudfrontStack/699967e0-1a16-11f1-96a1-0e792ab92505
✨  Total time: 45.81s

✅  CloudwatchLambdaStack
✨  Deployment time: 43.94s
Stack ARN:
arn:aws:cloudformation:ap-southeast-1:347116125752:stack/CloudwatchLambdaStack/ce5c5190-1a18-11f1-8f6e-023d362914e3        
✨  Total time: 57.82s


Note that after deploying the stack, an email will be sent to the address you used to subscribe to the SNS Topic; you need to confirm it to receive other emails when there are notifications (which is when a request is blocked).



The resources have been successfully created on the AWS Console.




Checking the API shows the following results.



When a request is blocked, there will be corresponding notifications sent to email and Telegram.




Happy coding!

See more articles here.

Comments

Post a Comment

Popular posts from this blog

All practice series

Image
Introduction This is a comprehensive page about the technologies I have shared in series format. You can view brief introductions and links to directly access each series you are interested in. In the field of software development, to deploy a product from the initial idea to its release, the standard process typically involves several stages as follows: Database : Designing and implementing the database according to business requirements, storing data during the system's operation. Backend : Handling the main logic of the system, communicating with the database and services. Frontend : Building the interface for users to interact with the system, which could be a desktop, mobile, or web application. This usually includes implementing UI/UX and integrating APIs from the backend. DevOps : Deploying the system for use, which can be done on a server or in the cloud. Testing : Applying testing methods to ensure the product meets the standards for release. Of course, these are just stan...
Read more

Deploying a NodeJS Server on Google Kubernetes Engine

Image
Introduction to GKE Google Kubernetes Engine (GKE) is a managed Kubernetes service provided by Google Cloud Platform , facilitating simple and efficient deployment of Docker images. We only need to provide some configuration for the number of nodes, machine types, and replicas to use. Some Concepts Cluster A Cluster is a collection of Nodes where Kubernetes can deploy applications. A cluster includes at least one Master Node and multiple Worker Nodes . The Master Node is used to manage the Worker Nodes . Node A Node is a server in the Kubernetes Cluster. Nodes can be physical servers or virtual machines. Each Node runs  Kubernetes , which is responsible for communication between the Master Node and Worker Node , as well as managing Pods and containers running on it. Pod A Pod is the smallest deployable unit in Kubernetes . Each Pod contains one or more containers, typically Docker containers. Containers in the same Pod share a network namespace, meaning they have the ...
Read more

Setting up Kubernetes Dashboard with Kind

Image
Introduction In a previous article, I guided you through using Helm to deploy on Google Kubernetes Engine . However, if you want to cut down costs by using Kubernetes in your local environment instead of relying on a cloud provider during development, then Kind is your go-to. There are several tools to help set up Kubernetes locally, such as MiniKube , Kind , K3S , KubeAdm , and more. Each tool has its own pros and cons. In this article, I'll walk you through using Kind to quickly set up a Kubernetes cluster on Docker . Kind stands out for its compactness, making Kubernetes start up quickly, being user-friendly, and supporting the latest Kubernetes versions. Working with Kind Firstly, follow the instructions here to install Kind according to your operating system. If you're using Ubuntu , execute the command: [ $( uname -m ) = x86_64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.23.0/kind-linux-amd64 chmod +x ./kind sudo mv ./kind /usr/local/bin/ki...
Read more

Using Kafka with Docker and NodeJS

Image
Introduction to Kafka Kafka is an open-source , distributed messaging system that functions on a publish/subscribe model. It is widely used by numerous large companies for high-performance , real-time data streaming. Developed by LinkedIn since 2011, Kafka has grown into the most popular distributed streaming platform. It can handle vast amounts of records with high efficiency. Advantages of Kafka Open-source : Freely available and continuously improved by a large community. High-throughput, high-frequency : Capable of processing large volumes of data across topics continuously. Automatic message storage : Allows for easy message retrieval and verification. Large user community : Offers extensive support and shared resources. Basic Concepts If you're new to Kafka and Message Queues, here are some key concepts to understand: Producer : Creates and sends data to the Kafka server, where data is sent as messages in byte array format. Consumer : One or more consumers subscribe to...
Read more

Monitoring with cAdvisor, Prometheus and Grafana on Docker

Image
Introduction Monitoring a system is crucial after deploying a product to a production environment. Keeping an eye on system metrics like logs , CPU , RAM , disks , etc, helps identify the system's status, performance issues, and provides timely solutions to ensure stable operations. While cloud providers like Google , Amazon , or Azure offer built-in monitoring systems, if your company needs to manage multiple applications/systems/containers and desires a centralized monitoring system for easier management, using cAdvisor , Prometheus , and Grafana is a sensible choice. These three popular open-source tools are widely used by DevOps teams, especially for monitoring container applications. cAdvisor Developed by Google , cAdvisor is an open-source project used to analyze resource usage, performance, and other metrics from container applications, providing an overview of all running containers. Find more details here Prometheus Prometheus is a toolkit for system monitoring and a...
Read more

Kubernetes Practice Series

Image
Introduction Kubernetes (often abbreviated as K8s ) is an open-source system designed to automate the deployment, scaling, and management of containerized applications. This page serves as a collection of articles related to Kubernetes (K8s) , including theoretical concepts and practical guides on using Kubernetes to set up essential tools for the software development process. I will continue to update this series with new articles as ideas come to mind, to ensure the series becomes more comprehensive. The articles are arranged in increasing order of difficulty to make it easier for you to follow. If you have time, it's recommended that you start from the beginning of the series to acquire the necessary knowledge and information that will prepare you for the subsequent articles. Key Topics Covered in This Series Basic Knowledge : Fundamental concepts, common commands, etc. Resources : Pod, Deployment, Service, StatefulSet, Ingress, etc. Pod Autoscaler : Horizontal and Vertical sc...
Read more

Kubernetes Deployment for Zero Downtime

Image
Introduction In Kubernetes (K8s) , a Pod is the smallest resource unit used to run one or more containers during deployment. There are several ways to create a Pod : you can create it directly, use a ReplicationController , or a ReplicaSet . However, the most commonly used resource for managing Pods is the Deployment . When you use a Deployment , it actually creates a ReplicaSet to manage the Pods but comes with many additional benefits that support the deployment process. Some Advantages of Deployment : Ensures Pod Availability : It guarantees that the specified number of Pods are always running according to the configuration, automatically deploying additional Pods if any failures occur. Supports Restart and Undo Deployment : Allows you to easily restart or roll back to previous versions of your Deployment . Zero Downtime Deployment : When updating configurations or scaling Deployments , zero downtime is crucial. This means that new Pods are created while the old Pods are stil...
Read more

Practicing with Google Cloud Platform - Google Kubernetes Engine to deploy nginx

Image
Introduction This article provides simple step-by-step instructions for those who are new to Google Cloud Platform (GCP) and Google Kubernetes Engine (GKE) . I'll guide you through using  GKE  to create clusters and deploy nginx . The instructions below will primarily use gcloud  and kubectl  to initialize the cluster, which is more convenient than manual management on the Google Cloud interface. Prerequisites First, you need to prepare the following: Have a GCP account with permission to use Cloud services. If you're new, you'll get a $300 free trial to use for 90 days . Create a new GCP project. Enable Compute Engine and Kubernetes Engine services. Install Google Cloud SDK and kubectl For this installation step, refer to the GCP documentation for instructions tailored to your operating system. Once installed, execute the following commands to check if gcloud  and kubectl  are installed: gcloud version kubectl version If you see the version resul...
Read more

NodeJS Practice Series

Image
Introduction NodeJS is an open-source and cross-platform JavaScript runtime environment . Here are some key points about NodeJS : V8 Engine : NodeJS runs on the V8 JavaScript engine , which is also the core of Google Chrome . This allows NodeJS to be highly performant. Asynchronous and Non-Blocking : NodeJS  uses an event-driven, non-blocking I/O model. It’s lightweight and efficient, making it ideal for data-intensive real-time applications. Single-Threaded : NodeJS  runs in a single process, handling multiple requests without creating new threads. It eliminates waiting and continues with the next request. Common Language : Frontend developers who write JavaScript for browsers can use the same language for server-side code in NodeJS . You can even use the latest ECMAScript standards without waiting for browser updates. This page is designed to compile articles related to NodeJS , including how to integrate it with various libraries and relevant tech stacks. I will cont...
Read more

Helm for beginer - Deploy nginx to Google Kubernetes Engine

Image
Introduction Helm is a package manager for Kubernetes , which simplifies the process of deploying and managing applications on Kubernetes clusters. Helm uses a packaging format called charts , which are collections of files that describe a related set of Kubernetes resources. Key Components of Helm Charts : Helm packages are called charts. A chart is a collection of files that describe a related set of Kubernetes resources. A single chart might be used to deploy something simple, like a memcached pod, or something complex, like a full web app stack with HTTP servers , databases, caches, and so on. Values : Charts can be customized with values, which are configuration settings that specify how the chart should be installed on the cluster. These values can be set in a ` values.yaml ` file or passed on the command line. Releases : When you install a chart, a new release is created. This means that one chart can be installed multiple times into the same cluster, and each can be indep...
Read more