What are VPS Endpoint and Flow Logs in AWS?

VPC Endpoint

A VPC endpoint is used to allow users to privately connect the VPC to the AWS resources. It also helps connect VPC endpoint services that are powered by PrivateLink to AWS services without the need of an internet gateway, NAT gateway, VPN connection or an AWS Direct Connect connection.

The VPC instance don’t need a public IP address so as to communicate with the resources present in the service. The traffic between VPC and the other services stays within the Amazon network only. Endpoints can be understood as virtual devices that are horizontally scaled, redundant, and highly-available VPC components. These VPC components help in the communication between instances in the VPC and the services, without causing any availability risks or bandwidth constraints with regards to the network traffic.

There are two types of VPC endpoints:

1. Interface endpoints
2. Gateway endpoints

Note: The VPC endpoints can be created based on the requirement and the service which supports this.

Interface Endpoints

An interface VPC endpoint is used to connect to services which use AWS PrivateLink. These services include Amazon services, services hosted by other AWS customers, partners of the user’s own VPCs, and AWS Marketplace partner services. The owner of the service refers to the service provide, and the user is the person who creates these interface endpoints, who are known as ‘service consumers’.

Gateway Endpoints

A gateway endpoint is a gateway that is specified by the user as a target to the route in the route table so that it follows the traffic it is assigned to the Amazon service that it supports. It supports Amazon S3 and DynamoDB services.

Controlling Usage of VPC Endpoints

IAM users don’t have the permission to work with endpoints by default. The user has to create an IAM policy that is used to grant users the permission to create, change, describe and delete endpoints. The user can’t create an IAM policy to grant permission to a specific endpoint or prefix list.

When an endpoint is created, the user can attach the endpoint policy to the endpoint which has control access to the service it connects to. Endpoint policies are written in JSON format.

If the user is using an endpoint to connect to Amazon S3, S3s bucket policies also need to be used to control access to these buckets from specific endpoints or VPCs.

Usage of VPC Endpoint Policy

VPC endpoint is an IMA policy which is attached to the endpoint when an endpoint is created or modified. If such a policy is not attached when an endpoint is created, Amazon, by default, attaches a policy that allows complete access to the service. No endpoint policy overrides or replaces the IAM user policies or other policies which are specific to the service.

Key Aspects

1. More than one policy can’t be attached to an endpoint, but a policy can be modified at any point in time. Once a policy is modified, it takes a few minutes for the changes to show up.
2. The part of the policy which is related to the specific service will work. An endpoint policy can’t be used to allow resources present in the VPC to perform other actions.
3. The endpoint policy must contain a principal element.
4. The size of the endpoint policy shouldn’t exceed 20480 characters, including whitespaces.

VPC Flow Logs

VPC Flow Logs is a feature provided by Amazon that helps the user capture information regarding IP traffic that goes to and comes from network interfaces in the VPC. Flow Log data which is captured can be published to Amazon CloudWatch Log and Amazon S3. Once the user creates a flow Log, they can retrieve and view the data in the destination of their choice.

Flow Logs can be used with multiple tasks, and some of them have been listed below:

1. Traffic monitoring which reaches the user instance.
2. The over restrictive security group rules can be diagnosed and worked on.
3. Determining the direction from where traffic come to and goes from the network interfaces.

Flow Log data that is outside the network traffic is collected, and hence, it doesn’t affect the network’s throughput or latency. Flow Logs can be created or deleted without worrying about its effect on the impact on network performance.

When Flow Log data is placed inside CloudWatch logs, it is chargeable.

A Flow Log can be created for a VPC, a subnet or a network interface. When a Flow Log is created for a subnet or a VPC, every network interface which is present within that subnet or VPC is considered for the process of monitoring.

The monitored network’s Flow Log data is recorded as ‘flow log record’, which logs events that consist of fields which describe the flow of traffic.

When a Flow Log has to be created, the below mentioned attributes are specified:

1. The resource for which the Flow Log needs to be created.
2. The type of traffic which it will be used to capture (Accepted traffic, rejected traffic or all kinds of traffic).
3. The destination wherein this Flow Log data is to be published.

Once a Flow Log has been created, it takes a few minutes for data to be collected and then publish to the chosen destinations. Flow Logs can’t capture real-time log streams for the user’s network interfaces.

If the user launches more than one instance in the subnet once the flog log has been created for subnet/VPC, a new log stream (for CloudWatch Logs) or a log file object (for Amazon S3) gets created for every new network interface. This operation happens as soon as the network traffic gets recorded for that specific network interface.

Flow Log for network interfaces can be created by other Amazon services which have been listed below:

1. Elastic Load Balancing
2. Amazon RDS
3. Amazon ElastiCache
4. Amazon Redshift
5. NAT gateways
6. Transit gateways
7. Amazon WorkSpaces

Irrespective of the network interface type, the Amazon EC2 console or Amazon EC2 API has to be used to create a Flow Log for the network interface.

When a Flow Log is created, the default format can be used for its Flow Log record or a customized format can be used (Amazon S3 only).

When the user no longer requires the Flow Log, it can be deleted. When a Flow Log is deleted, it disables the Flow Log service for that specific resource, and no new Flow Log gets created or published to CloudWatch or Amazon S3.

When a Flow Log is deleted, it doesn’t delete any existing Flow Log record or log stream or log file objects for a network interface.

If an existing log stream needs to be deleted, the CloudWatch Log console needs to be used. If an existing log file object needs to be deleted, the Amazon S3 console needs to be used.

Flow Log Records

A Flow Log record is used to represent a network flow for the VPC. Every record is used to capture a network internet protocol traffic by default. This traffic is present within a ‘capture window’. Capture window is a time period of about 10 minutes during which the data flow is captured. ‘Aggregation period’ refers to the total amount of time it takes to capture, process and publish the flow data. The aggregation period can take up to 15 minutes.

The record includes values of different components present in the IP flow by default, and this includes the source, destination and the protocol.