Hands On AWS X-Ray: Analyzing ‘Tarces’ and ‘Service map’ by Deploying AWS sample application using Cloudformation, X-Ray daemon on Amazon ECS, Amazon Dynamodb for data persistence, and Amazon SNS for notification.

What are X-Ray traces?

In nutshell, AWS X-Ray receives data from services as segments. X-Ray then groups segments that have a common request into traces. X-Ray processes the traces to generate a service graph that provides a visual representation of your application. For more details What is AWS X-Ray? — AWS X-Ray (amazon.com).

This tutorial uses the xray-gettingstarted branch of the Scorekeep sample application, which uses AWS CloudFormation to create and configure the resources that run the sample application and X-Ray daemon on Amazon ECS. The application uses the Spring framework to implement a JSON web API and the AWS SDK for Java to persist data to Amazon DynamoDB. A servlet filter in the application instruments all incoming requests served by the application, and a request handler on the AWS SDK client instruments downstream calls to DynamoDB.

https://docs.aws.amazon.com/images/xray/latest/devguide/images/architecture-dataflow.png

x-ray agent on EC2 and EKS/ECS node

Key learning and takeaways:

In this demo, we will learn how to analyze and interpret data using “trace” and “service map” in Amazon X-Ray. Then trace and issue caused by one of the nodes using service map, trace ID and exception to spot the error. At the end we look at the solution provided.

Implementation details:

1. Prerequisites

2. Deploy the Scorekeep application with CloudFormation

3. Generate trace data via the application.

4. Explore the sample application

5. Analyzing traces.

6. Analyzing service map.

7. Configuring SNS notification.

8. Clean up

9. References

1. Prerequisites:

This tutorial uses AWS CloudFormation to create and configure the resources that run the sample application and X-Ray daemon. The following prerequisites are required to install and run through the tutorial:

1. If you use an IAM user with limited permissions, add the following user policies in the IAM console:

· AWSCloudFormationFullAccess – to access and use CloudFormation

· AmazonS3FullAccess – to upload a template file to CloudFormation using the AWS Management Console

· IAMFullAccess – to create the Amazon ECS and Amazon EC2 instance roles

· AmazonEC2FullAccess – to create the Amazon EC2 resources

· AmazonDynamoDBFullAccess – to create the DynamoDB tables

· AmazonECS_FullAccess – to create Amazon ECS resources

· AmazonSNSFullAccess – to create the Amazon SNS topic

· AWSXrayReadOnlyAccess – for permission to view the trace map and traces in the X-Ray console

2. To run through the tutorial using the AWS CLI, install the CLI version 2.7.9 or later, and configure the CLI with the user from the previous step. Make sure the region is configured when configuring the AWS CLI with the user. If a region is not configured, you will need to append --region AWS-REGION to every CLI command.

3. Ensure that Git is installed, to clone the sample application repo.

4. Use the following code example to clone the xray-gettingstarted branch of the Scorekeep repository:

git clone https://github.com/aws-samples/eb-java-scorekeep.git xray-scorekeep -b xray-gettingstarted

2. Install the Scorekeep application using CloudFormation

· In AWS console, search X-Ray.

• Service map
• Click on ‘Read more about X-Ray’: Link
• More resources: Getting started with AWS X-Ray: Link
• Sample application: Link

Install the sample application using the AWS Management Console

1. Open the CloudFormation console
2. Choose Create stack and then choose With new resources from the drop-down menu.
3. In the Specify template section, choose Upload a template file.
4. Select Choose file, navigate to the xray-scorekeep/cloudformation folder that was created when you cloned the git repo, and choose the cf-resources.yaml file.
5. Choose Next to continue.
6. Enter scorekeep into the Stack name textbox, and then choose Next at the bottom of the page to continue. Note that the rest of this tutorial assumes the stack is named scorekeep.
7. Scroll to the bottom of the Configure stack options page and choose Next to continue.
8. Scroll to the bottom of the Review page, choose the checkbox acknowledging that CloudFormation may create IAM resources with custom names, and choose Create stack.
9. The CloudFormation stack is now being created. The stack status will be CREATE_IN_PROGRESS for about five minutes before changing to CREATE_COMPLETE. The status will refresh periodically, or you can refresh the page.

2- Generate trace data

Find the application URL using the AWS Management Console

1. Open the CloudFormation console
2. Choose the scorekeep stack from the list.
3. Choose the Outputs tab on the scorekeep stack page, and choose the LoadBalancerUrl URL link to open the web application.

Play the game.

· Click on “Instructions”

· Click on “Create”

· Username: tcbUser

· Game Name: tcbGame

· Rules: Tic Tac Toe

· Play

5. Analyzing “Traces”

Click on “Traces”, and set “Refine query by: Node”, you will see every component (node) is being created.

• Refine query by: “Node”, “HTTP Method”, “URL”…

Refining query by URL:

Every trace gets and ID, click on each for further analysis.

Each trace ID provides graphical map and details of the trace.

Trace ID with raw data, that provides rich details. It will provide nodes metadata, user IP, service ID etc.

“Segments Timeline” provides time stamp and duration of each trace.

Every trace carries a Segment Timeline where an error or an event can be traced using timeline.

6. Analyzing “Service Map”

Service map provides visual map of traces for easier tracing. Service map provides two option to analyze and interpret the data. Map view and List view.

Go back to service map, Map view.

Legend and options: provides legend of the map with different colors.
Metrics or Icons size: you can view the map using Icons or Metrics.

Service map provides multiple options such as viewing using Metrics or Icons. Viewing using Icon provides service name and its status. Using Metrics, the size of the icon is also presented with numbers. Issues such as latency or failure can be spotted easier.

Map view: Icons allows to see the nodes with icons for faster tracing.

Map view: Metrics allows to see detailed information of the nodes.

Clicking on each service map, it will provide more information. We can analyze the node (service) using Metric data, by Alerts or Response time. Below shows the SNS service metrics.

Service map List View.

List view provides the details of nodes in list format.

By clicking on user nodes “scorekeep-user” which is the Dynamodb table, we can analyze the latency, requests, and faults in details.

Tracing the error.

• Click on Service Map
• Select the affected Node (SNS in this case)

• Click on View traces to find the trace ID.

• Select the trace ID from the list.

• Segments Timeline: select the line with the issue.

• Exceptions “To find the cause of the error”. In this case, the error is caused by invalid parameter in the email (wrong email address).

7. Configuring Amazon SNS notification.

Open the CloudFormation console

1. Choose the radio button next to the scorekeep stack name in the list, and then choose Update.
2. Make sure that Use current template is chosen, and then click Next on the Update stack page.
3. Find the Email parameter in the list, and replace the default value with a valid email address.

1. Scroll to the bottom of the page and choose Next.
2. Scroll to the bottom of the Review page, choose the checkbox acknowledging that CloudFormation may create IAM resources with custom names, and choose Update stack.
3. The CloudFormation stack is now being updated. The stack status will be UPDATE_IN_PROGRESS for about five minutes before changing to UPDATE_COMPLETE. The status will refresh periodically, or you can refresh the page.

8. Clean up

• AWS CloudFormation
• xray-sample
• Delete [ Pause…it can take a while…. and result in error after ~30 minutes! ]
• Some resources may not be deleted!
• You can retry “Delete”

Deleting Resources Manually

• Check it and proceed:
• ASG
• TG
• LB
• EC2
• VPC
• ECS ( Cluster | Task Definitions )
• DynamoDB
• SNS
• CloudFormation
• xray-sample
• Delete

References:

https://docs.aws.amazon.com/xray/latest/devguide/aws-xray.html

https://docs.aws.amazon.com/xray/latest/devguide/scorekeep-tutorial.html#xray-gettingstarted-prereqs