Splunk Observability APM with AWS ECS EC2

이전 글에서 AWS ECS의 Splunk Observability Infrastructure Monitoring 연동을 하였습니다.

이번에는 Splunk Observability APM을 연동을 해보겠습니다.

방법은 여러가지가 있을 것 같지만, docker image에 Java APM instrument 라이브러를 추가 합니다.

Splunk O11y Cloud 에서

Data Management > Java (OpenTelemetry) 세팅

 

아래 스크립트를 이용해서 기존 어플리케이션의 dockerfile 을 업데이트 합니다.

splunk-otel-javaagent.jar (Java Instrument)를 기존 어플리케이션과 같이 기동할 수 있도록 하는 것입니다.

Java가 기동되는 환경값에 아래 내용을 추가합니다. myapp.jar은 샘플 application 입니다.

java -javaagent:./splunk-otel-javaagent.jar \
-Dsplunk.profiler.enabled=true \
-Dsplunk.profiler.memory.enabled=true \
-Dsplunk.metrics.enabled=true \
-jar myapp.jar

 

 

두번째는 ECS Cluster task의 환경변수를 셋업합니다.

            "environment": [
                {
                    "name": "OTEL_SERVICE_NAME",
                    "value": "InhoFactorialService"
                },
                {
                    "name": "OTEL_RESOURCE_ATTRIBUTES",
                    "value": "deployment.environment=TEST,service.version=1.0.0"
                },
                {
                    "name": "OTEL_EXPORTER_OTLP_ENDPOINT",
                    "value": "http://172.17.0.1:4317",
                }, 
               {
                    "name": "SPLUNK_METRICS_ENDPOINT",
                    "value": "http://172.17.0.1:9943"
                }

            ],

{
    "taskDefinitionArn": "arn:aws:ecs:us-east-1:432274630114:task-definition/factorial-task:24",
    "containerDefinitions": [
        {
            "name": "factorial",
            "image": "public.ecr.aws/e7b5k4u7/factorial",
            "cpu": 1024,
            "links": [
                "splunk-otel-collector"
            ],
            "portMappings": [
                {
                    "name": "forwader8080",
                    "containerPort": 8080,
                    "hostPort": 8080,
                    "protocol": "tcp",
                    "appProtocol": "http"
                },
                {
                    "name": "forwarder80",
                    "containerPort": 80,
                    "hostPort": 80,
                    "protocol": "tcp",
                    "appProtocol": "http"
                }
            ],
            "essential": true,
            "environment": [
                {
                    "name": "OTEL_SERVICE_NAME",
                    "value": "InhoFactorialService"
                },
                {
                    "name": "OTEL_RESOURCE_ATTRIBUTES",
                    "value": "deployment.environment=TEST,service.version=1.0.0"
                },
                {
                    "name": "OTEL_EXPORTER_OTLP_ENDPOINT",
                    "value": "http://172.17.0.1:4317"
                },
                {
                    "name": "SPLUNK_METRICS_ENDPOINT",
                    "value": "http://172.17.0.1:9943"
                }                
            ],
            "mountPoints": [],
            "volumesFrom": [],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-create-group": "true",
                    "awslogs-group": "/ecs/factorial-task",
                    "awslogs-region": "us-east-1",
                    "awslogs-stream-prefix": "ecs"
                }
            },
            "systemControls": []
        },
        {
            "name": "splunk-otel-collector",
            "image": "quay.io/signalfx/splunk-otel-collector:latest",
            "cpu": 512,
            "memory": 512,
            "portMappings": [
                {
                    "containerPort": 4317,
                    "hostPort": 4317,
                    "protocol": "tcp"
                }
            ],
            "essential": true,
            "environment": [
                {
                    "name": "METRICS_TO_EXCLUDE",
                    "value": "[]"
                },
                {
                    "name": "HOST_DEV",
                    "value": "/hostfs/dev"
                },
                {
                    "name": "HOST_ETC",
                    "value": "/hostfs/etc"
                },
                {
                    "name": "SPLUNK_CONFIG",
                    "value": "/etc/otel/collector/ecs_ec2_config.yaml"
                },
                {
                    "name": "SPLUNK_REALM",
                    "value": "us1"
                },
                {
                    "name": "HOST_PROC",
                    "value": "/hostfs/proc"
                },
                {
                    "name": "SPLUNK_ACCESS_TOKEN",
                    "value": "RBde2oHa77ZPu95DWI9Jyw"
                },
                {
                    "name": "HOST_SYS",
                    "value": "/hostfs/sys"
                },
                {
                    "name": "HOST_RUN",
                    "value": "/hostfs/run"
                },
                {
                    "name": "ECS_METADATA_EXCLUDED_IMAGES",
                    "value": "[\"quay.io/signalfx/splunk-otel-collector\"]"
                },
                {
                    "name": "HOST_VAR",
                    "value": "/hostfs/var"
                }
            ],
            "mountPoints": [
                {
                    "sourceVolume": "hostfs",
                    "containerPath": "/hostfs",
                    "readOnly": true
                }
            ],
            "volumesFrom": [],
            "systemControls": []
        }
    ],
    "family": "factorial-task",
    "taskRoleArn": "arn:aws:iam::432274630114:role/ecsTaskExecutionRole",
    "executionRoleArn": "arn:aws:iam::432274630114:role/ecsTaskExecutionRole",
    "networkMode": "bridge",
    "revision": 24,
    "volumes": [
        {
            "name": "hostfs",
            "host": {
                "sourcePath": "/"
            }
        }
    ],
    "status": "ACTIVE",
    "requiresAttributes": [
        {
            "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
        },
        {
            "name": "ecs.capability.execution-role-awslogs"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
        },
        {
            "name": "com.amazonaws.ecs.capability.task-iam-role"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.29"
        }
    ],
    "placementConstraints": [],
    "compatibilities": [
        "EC2"
    ],
    "requiresCompatibilities": [
        "EC2"
    ],
    "cpu": "2048",
    "memory": "6072",
    "runtimePlatform": {
        "cpuArchitecture": "X86_64",
        "operatingSystemFamily": "LINUX"
    },
    "registeredAt": "2024-03-22T00:01:25.878Z",
    "registeredBy": "arn:aws:sts::432274630114:assumed-role/AWSReservedSSO_SPLKAdministratorAccess_06c8810013bb6e59/ijung@splunk.com",
    "tags": []
}

task update 하신 후에, 서비스를 업데이트 합니다.

 

테스트: 아래 n의 상수값을 바꾼다. -1일 경우는 exception도 발생 함

http://{Container host IP}:8080/factorial?n=5

http://{Container host IP}:8080/factorial?n=-1

 

 

전수 traces 수집한 화면