AWS Solution Architect Certification Notes – Part 2

My AWS certification notes part 2 from A Cloud Guru material

EFS – Elastic File System

  • File storage service
  • Elastic – grow and shrink based on the need.
  • Can share EFS across multiple EC2. EC2 instances should be in same security group as EFS
  • Block based storage (S3 is object based storage)
  • Read after write consistency
  • Pay only what you use
  • Scale upto petabytes
  • Supports NFSv4
  • Supports 1000 of concurrent NFS connections
  • Data stored in multiple AZ within the region
  • EFS can be mounted in multiple AZ
  • EFS can be used only within one VPC at a time
  • with EFS, both file system and VPC should be in the same region
  • EFS will not be available for all VPC’s within region by default.
  • EFS does not support cross region peered VPC’s
  • For EFS, can work with VPC peering within single AWS region when using C5 or M5 EC2 instances
  • Security group to open NFS port 2049 to access EFS
  • Encryption at rest can be enabled only at the time of EFS creation
  • NFS not an encrypted protocol
  • Encryption at transit canNOT be enabled only at the time of EFS creation
  • To enable encryption at Transit,
  • Enable encryption during mounting on EC2 using amazon mount helper
  • unmount unencrypted mount
  • remount using mount helper encryption during transit option
  • encryption at rest cannot be done during mounting. it can be done only during craetion.
  • Performance Mode
  • General purpose – recommended for most file system, low latency
  • Max I/O – recommended for tens or hundreds or thousands of EC2 sharing EFS. slightly higher latency. big data
  • Throughput mode
  • Provisioned – can configure specific throughput irrespective of EFS data size
  • Bursting – throughput on EFS scales as file system grows
  • Usecases
  • Bigdata and analytics
  • media processing workflows
  • content managment
  • web serving
  • home directories

AWS – CLI

  • aws configure
  • cd ~, cd .aws, credentials and config file
  • aws help
  • aws ec2 describe-instances  will show terminated accounts also
  • aws ec2 describe-images  will show all the available images (both public and private)
  • aws ec2 run-instances  To provision the instance
  • aws ec2 terminate-instances –-instance-ids
  • role can be assigned to running instance
  • With role, acceskey and password not required. Just configure with region
  • aws s3 cp –recursive s3:// –region
  • Best practice is to use region. Some will work without region
    Bash Scripting

!/bin/bash

yum update -y
yum install httpd -y
service httpd start
chkconfig httpd on
aws s3 cp s3://indexbucket-186/index.html /var/www/html –region ap-south-1

Meta Data

Cloud Watch

  • Monitoring, Cloud watch – basic monitoring, CPU. Disk,Nw
  • Standard monitoring – 5 mins
  • Detailed monitoring – for every 1 min (not free)
  • Cloud Watch – performance monitoring
  • Cloud Trail – Audit trail
  • Dashboards
  • Alarms
  • Events
  • Logs
  • Basic monitoring – 5 mins – default
  • Detailed monitoring – 1 min
  • Dashboards
  • Widget
  • Text
  • Line
  • Stacked Area
  • Number – current cpu utilization
  • Default metric for EC2
  • CPU related
  • Disk related
  • Network related
  • Status check related
  • No RAM related – custom metric
  • Alarm
  • create topic, notification list and confirm subscription
  • Events
  • event triggers lambda function to update DNS with public ip when EC2 becomes running state from stop
  • Logs
  • Install agent, monitor and access
  • Metrics

Elastic Load Balancer – ELB

  • No public IP. Only DNS name. Amazon manages public IP
  • Health check. In service, out of service
  • It will cost if its running
  • Classic load balancer – TCP/IP layer or Http/Https, layer 7 or 4, legacy, not recommended. Important for exam
  • Application Load balancer – application layer, operates at layer 7, best suited for http/https, Can look at till application level to decide routing
  • Network Load balancer – level 4, most expensive, TCP, extreme performance, can handle millions of requests per second, supports only TCP. doesnt support http/https
  • Healthy threshold – no of success healthy checks before declare the instance as in service
  • UnHealthy threshold – no of failure healthy checks before declare the instance as out of service
  • Timeout
  • Interval
  • Error code 504 – Gateway Time out error. Issue is with application (web server or DB)
  • X-forwarded-For Header – Load balancer sending user’s source IP to EC2 in X forwarded for header.
  • If you ipv4 public address of end user, look for x-forwarded-for header in classic load balancer.
  • To maintain session state or even distribution session in ELB – use Elastic Cashe to store transient session data
  • ELB use ports from 1024 to 65535
  • You can specify only one subnet per AZ
  • Atleast 2 subnets must be specified
  • Subnet should have internet access. Private Subnet will not be allowed
  • If doing load test on ELB,
  • Ensure to re-resolve DNS before every request
  • Send requests from globally distributed clients or multiple test clients
  • To split the traffic accross the AZ – Enable Cross Zone load balancing (for classic)
  • Cross Zone load balancing
  • Enabled in Application and Network load balancer by default
  • Not Enabled in classic load balancer by default
  • To monitor application load balancers
  • Cloudwatch metrics
  • Access logs
  • request tracing
  • CloudTrail logs
  • Target Type
  • instance Id
  • Ip address
  • ELB can distribute the traffic only in one region. not across the region

SDK

  • Software Development Kit
  • https://aws.amazon.com/tools/
  • Java, .Net, Node.js, PHP, Python, Ruby, Browser, GO, C++
  • AWS IOT
  • Android, IOS, React Native, Mobile web, AWS Mobile
  • Default Region US-East-1, North Virginia
  • Some have default regions (java)
  • Some do not (Node.js)

Lambda

  • Serverless (lambda, S3, DynamoDB, Api gateway). EC2 & RDS is not
  • Its compute service, where you can upload the code and create lambda function
  • Aws lambda takes care of provisioning and managing servers
  • We don’t have to worry about OS, patching, scaling etc
  • AWS Lambda – data center, hardware, protocols, OS
  • Lamda events can trigger other lamda events
  • Can communicate to aws services
  • Scale up – increase RAM, memory, storage
  • Scale out – spawn EC2 instance based on load
  • Lambda scales out automatically. If your lambda run out of memory, you need to upgrade the configuration
  • Lambda triggers (eg: api gatway, S3, Cloudfront, dynamoDB) important exam topic, memories all the triggers
  • Event Driven – when u upload Meme to S3 lambda can be triggered to add text. After adding text another lambda can be triggered to show to user and another lambda to replicate the meme in another S3
  • Api Gateway – when http request comes to api gateway, lambda can be triggered to give the response
  • If 1 million requests to API gateway, I million lambda functions will be triggered. Automatically scale out
  • Languages – Java, Node.js, python, Go and C#
  • Lambda priced based on no of requests
  • 1st 1 million requests are free. $.20 per 1 million there after
  • Also priced based on duration. Max threshold is 5 mins. (300 seconds) If the function takes more than 5 mins, split into multiple functions default timeout is 3 seconds
  • No servers
  • Super cheap
  • Amazon echo is lambda
  • AWS x-ray to debug lambdas
  • Lambda can do things globally
  • Lambda versioning – version control
  • Can have multiple versions of lambda function
  • Lambda versioning immutable. Once published can’t be changed
  • $LATEST – latest version
  • Qualified ARN – the function ARN with the version suffix
  • UnQualified ARN – the function ARN without the version suffix
  • Alias – can create alias and map the version behind the screen
  • Can split traffic using alias to different version. Alias can 50 % to v1 and 50% traffic to v2
  • Cannot spilt traffic with $latest
  • Lambda default timeout – 3 seconds
  • Max execution duration per request – 300 seconds (5 mins)
  • You can set memory in 64 MB increments from 128 MB to 3 GB
  • Minimum 128 MB and Max is 3008 MB (64 MB increments)
  • ephemeral disk capacity (/tmp space ) = 512 MB
  • If you choose 256 MB, it allocates approximately twice as much CPU power to your lambda function as requesting 128 MB of memory and half as much CPU power as choosing 512 MB of memory
  • Lambda function can reach across the region as lomg as it as internet access
  • Lambda function is set to run in private VPC, then to access S3, it need NAT Gateway or S3 VPC endpoint. it requires either internet or internal access.
  • Lambda can run function securely with in VPC by default. But to enable your lamabda to access the resources within your VPC, you must provide additional VPC specific configuration (subnet ID’s, security groups). if you didnt do this, lambda cannot connect to the resources in your vpc. this means lamabda running in no vpc mode.
  • Lambda Triggers
  • S3
  • DynamoDB
  • Kinesis
  • SNS, SES, SQS
  • Cognito
  • CloudFormation
  • Cloudwatch logs and Events
  • CodeCommit
  • AWS Config
  • Lex
  • IOT button
  • CloudFront
  • API GateWay
  • Lambda supports to forward non-processed payloads (timeout or memory out) to Dead Letter Queue using SQS, SNS for notification.
  • Automatically retries failed executions for asynchronous invocations.
  • SQS default and maximum batch size = 10
  • lamabda supports only standard queue. FIFO queue is not supported.
  • Cloudfront events that can trigger lamabda
  • Viewer request – request from end user
  • Viewer response
  • Origin request – request from edge to origin (EC2 or S3) when edge doesnt have the data in cache
  • Origin response
  • Version
  • when you update the code, it updates the version as latest
  • no need to define the code, which version needs to be used. it uses latest by default. however you can mention the version if you want
  • if new version is updated, requests might be served by old or new version for few seconds.
  • Poll based event sources for lamabda to trigger
  • SQS
  • Kinesis
  • DynamoDB
  • Lambda function policy cant be edited from console. yo should use CLI or SDK only
  • Function policy to grant cross account permissions. Not the execution policy.
  • Lambda in VPC, VPC should have sufficient ENI capacity to support scaling
  • ENI Capacity = peak concurrent exeutions * (Memory in GB / 3GB)
  • if the peak concurrent request is 100, memory is 1 GB then = 100* (1 GB/ 3 GB) = 33 ~ 30
  • /24 will have 256 IP and 251 will be available
  • /25 will have 128 IP and 123 will be available
  • /26 will have 64 IP and 59 will be available
  • /27 will have 32 IP and 27 will be available
  • so /26 will be the minimum CIDR block, that lamabda function requires to work without any issues
  • Lambda function uses environement varible to store the configuration. lambda encrypts them using KMS by default and those values are decrypted and made availiable for lamabda code durimg the execution
  • for any sensitive data like DB credetials, data from environment variables is decrypted during execution. so its not secure
  • aws recommends to encrypt the data before deploying by your own KMS key. Lambda provides encryption helper and decryption helper to make it easier.
  • Version of the lamabda function
  • getFunctionVersion() from context object
  • AWS_LAMBDA_FUNCTION_VERSION – environement variable

API Gateway

  • Fully manages service used to publish, maintain, monitor, and secure api’s at any scale \
  • Low cost and scales automatically
  • Can throttle API gateway to prevent attacks
  • Can log results to cloudwatch
  • Configure API gateway :
  • Define an API (container)
  • Define resources and nested resources (URL paths)
  • For each resource, select HTTP method, security, target (EC2,lamabda) and request response transformation
  • Deploy API to STAGE & PROD
  • Uses API gateway domain by default. Can use custom domain
  • Now supports AWS certificate manager (free SSL TLS certs)
  • API caching – cache your endpoint response for the specific TTL (time to live) period in seconds
  • Same Origin policy – under the policy, web browser permits, scripts in first web page to access data in second web page. But only if both the web pages have the same origin. This is to prevent cross site scripting (XSS)
  • CORS policy – CORS is enforced by the client
  • Can import API from an external definition file
  • Currently (import) supports swagger v2.0
  • API Throttling
  • By default API gateway limits 10000 rps (requests per second)
  • By default Max concurrent requests is 5000 across all api’s with in aws account
  • If you go over 10000 rps or 5000 concurrent requests – you will get 429 –Too many requests
  • if the caller sends 10000 requests in the first milli seconds, API gateways serves 5000 of those requests and throttles the rest in the one second period.
  • API gateway can be configured as soap webservice pass through
  • Integration sources
  • Lambda
  • Http
  • Mock
  • AWS service
  • VPC link
  • Use VPC Link to integrate on premises backed solutions through direct connect and private VPC. i.e, Rest api’s are exposed to internet and hosted in on-premises. using VPCLink the api’s can be integrated with api gateway. (VPCLink not used to connect internet)
  • Controlling access to api in api gateway
  • resource policy – add policy to allow or deny access from specified IP addresses
  • IAM roles and policies
  • CORS
  • lambda autorizers
  • cognito user pools
  • client side SSL certs
  • usage plans
  • Automatically protect your backend systems from Distributed denial of service (DDoS) attacks.
  • Access logging – logs who has accessed your API and how they accessed.
  • Automatically integrates with CloudFront to ensure better response to the calls made to the API

Step Function

  • Allows to visualize and test your serverless applications
  • Automatically triggers and tracks each step
  • Retries if error
  • Logs each step – diagnosis and debug easy
  • Sequential steps
  • Branching steps
  • Parallel steps

X-Ray

  • AWS X-Ray makes it easy for developers to analyze the behavior of their distributed applications by providing request tracing, exception collection, and profiling capabilities.
  • Debugging
  • Tracing
  • Service Map – Visual map
  • X-Ray SDK provides,
  • Interceptor – To add your code to trace incoming http requests
  • Client Handler – to call other AWS services
  • Http Client – to call other internal or external http webservices
  • X-Ray integrates with following AWS services
  • Elastic Load balancer
  • Lambda
  • API Gateway
  • Ec2
  • Elastic beanstalk
  • X-Ray languages supported – Java, Go, Node.js, Python, Ruby, .Net

S3

  • Simple storage service
  • Object based
  • Files can be 0 bytes to 5 TB
  • Unlimited storage
  • Files stored in buckets
  • Not suitable for OS or DB
  • S3 is universal. Bucket name should be unique
  • https://s3-us-west-1.amazonaws.com/aacloudguru
  • Read after write consistency for PUTS of new objects – for new uploaded file, can be accessed immediately
  • Eventual consistency for overwrite PUTS and DELETES – for update and delete files, retrieve is will take little time.
  • After update or delete, retrieve is atomic. Which means it will return either updated file or old file. it will not return partial or corrupted file (reason, sync with all AZ)
  • S3 contains
  • Key – name of the object
  • Value –data (sequence of bytes)
  • Version Id – versioning
  • Meta data
  • Sub resources – bucket specific configuration (policies, access control lists), CORS, transfer acceleration
  • Torrent
  • Built for 99.99 % availability
  • Amazon guarantees 99.99% availability
  • Amazon guarantees 99.999999999% durability (remember 11 x 9 s)
  • Tiered storage available
  • Life cycle management
  • Versioning
  • Encryption
  • Secure your data – access control list and bucket policies
    S3 storage tiers / classes
  • S3
  • 99.99 % availability
  • 99.9999999999 % durability
    • Designed to sustain the loss of 2 facilities concurrently
  • Currently cost effective
  • S3 – IA
  • Infrequently accessed
  • But rapid access when needed
  • Lower fee then S2 and but charged retrieval fee (every retrieval)
  • S3 – One Zone IA
  • Same has IA but data stored in single AZ
  • 99.5 % availability only
  • 99.9999999999 % durability
  • Cost is 20% less the S3-IA
  • Reduced Redundancy Storage (legacy – one zone IA)
  • 99.99 % availability
  • 99.99 % durability only
  • Used for data that can be created if lost
  • Ex: thumbnails
  • Low cost
  • Not recommended
  • Glacier
  • Archiving
  • Not part of S3, but linked
  • Very cheap
  • 99.99 % availability
  • 99.9999999999 % durability
  • Infrequently accessed
  • No real time access
  • it takes 3 to 5 hours to restore data from glacier
  • 3 types of retrievals
    • Expedited retrievals (1 -5 mins)
    • standard ( 3 – 5 hours)
    • Bulk (5 – 12 hours) – cheapest option
  • charged for
  • Storage per GB
  • No of request (get, put, copy , delete etc
  • Storage management – inventory, Analytics, Object tags
  • Data Management pricing – Data transferred out of S3
  • Transfer Acceleration – cloudfront
  • S3 – Security
  • BY default – private
  • Access control
    • Bucket policies – applied at bucket level, json, applicable to objects in bucket
    • Access control lists – applied at object level
  • Can configure to create access logs, which log all requests to S3. These logs can be written into another bucket
  • S3 Policies – policy generator
  • S3 Encryption
  • In Transit – SSL/TLS
  • Type of block cypher – Advanced encryption standard 256 – AES 256
  • At Rest, Server side encryption
    • SSE – S3 – S3 managed keys, AWS manage the keys and enc/dec – AES256
  • Strong multi factor encryption
  • S3 encrypts each object with unique key
  • It encrypts key itself with master key that it regularly rotates
    • S3-KMS – Key Management Service, AWS manage the keys. Additional benefits. Can track the encryption, decryption activities
    • SSE-C – Customer provided keys. AWS will do enc/dec based on the keys provided by customer

  • To enforce encryption, add bucket policy to deny all PUT requests if the header doesn’t have x-amz-server-side-encryption tag
  • Every upload is PUT request to S3
  • x-amz-server-side-encryption – if this tag is included in header of PUT request, it tells s3 to encrypt the object with the specified encryption method.
  • x-amz-server-side-encryption:AES256 – this means SSE-S3, S3 managed keys
  • x-amz-server-side-encryption:ams:kms – this means SSE-KMS, KMS managed keys
  • Expect:100-continue – S3 will reject if the content type is invalid
  • CORS – Cross Orgin Resource Sharing. To share the objects from one bucket to another
  • Requires versioning enabled
  • Static website hosting – .s3-website-.amazonaws.com
  • S3 versioning
  • Once enabled ypu cant disable. You can only suspend it
  • Version no for objects before enabling versioning will be null
  • Can enable MFA for any deletes
  • each version is charageble
  • S3 life cycle management
  • Eg : Move the file to S3-IA after 30 days and move to Glacier after 60 days
  • Can delete permenantly after x days
  • Can be applied to current as well as previous version
  • Tp upload file more than 5 GB – Multipart upload API (recommended for more then 100MB)
  • Multipart allows stop and resume uploads
  • Single put max size is 5 GB
  • Multi Object Delete – to delete large number of objects, no additional charge
  • By default 100 buckets per aws account. Contact aws for more
  • S3 Naming
  • 3 to 63 chars long
  • Only lower case
  • Number, period, dash allowed
  • Must start with lower case letter or number
  • Underscore, end with dash, consecutive period, dash adjacent to period not allowed
  • IP address format not allowed
  • Key names are stored lexicographically (alphabetical order)
  • S3 bucket ownership is NOT transferable. Owner can grant access to others
  • Referrer policy – to allows access from specific domains only
  • 409 conflict error – when bucket you trying to delete is not empty through api. But through console, you can delete the non-empty bucket
  • S3 does support redirects
  • Costs are as follows( from most expensive to least)
  • 1. S3-RRS .024
  • 2. S3 .023
  • 3. S3-IA .0125
  • 4. S3OneZone IA .01.

Leave a comment