Databases
AWS Databases
- Relational Database Service RDS
- Oracle, SQL Server, MySQL, PostgreSQL, Aurora, MariaDB
- Multi-AZ - for disaster recovery - exact copy in another AZ, AWS will change the endpoint during failover - no intervention (not available for Aurora b/c it’s fault tolerant by design). no charge for data transfer. Can use Reserved DB Instances
- Read Replicas - for performance - async replication from primary. RR can be promoted to standalone DB (breaks replication)
- Not for DR
- Must have automatic backups turned on
- Up to 5 read replicas (but an of RR of RR)
- Each RR has its own DNS endpoint
- RR can be Multi-AZ
- Can be in a second region
- Not serverless (except Aurora)
- Runs on VMs - can’t log in - Amazon responsible for patching
- Automated Backups are enabled by default
- Point in time recovery w/in retention period (1-35 days)
- Full daily snapshots plus transaction logs throughout day
- May experience latency during window
- Deleted with the RDS db
- DB Snapshots are done manually - stored even after you delete original RDS db
- Restore an instance to a new RDS instance, which means new endpoint (change your apps)
- Encryption at rest - KMS - backups, etc also
- RDS for SQL Server max is 16TB when using Provisioned IPS and General Purpose (SSD) storage types
- Non relational NoSQL DynamoDB
- Collection = table; Document = row; Key Value Pairs = fields
- Data Warehouse - Redshift
- OLTP - online transaction processing
- OLAP - online analytics processing
- ElastiCache - in memory cache
- Redis
- memcached
DynamoDB
- SSD storage - single digit millisecond latency - document and k-v data models
- Across 3 Geo-distinct data
- Eventual consistent reads (default = ~1sec) vs. strongly consistent reads
- Accelerator - DAX : in-memory cache. Can write-through cache
- Fully managed, HA. 10x performance improvement
- Request time down to microseconds
- Transactions -
- Multiple “all or nothing” ops
- 2 underlying reads/write - prepare/commit
- Up to 25 items or 4MB of data
- On-Demand Capacity - pay per request pricing (balance cost and performance) - pay more per request than provisioned
- On-Demand Backup and Restore - full backup at any time, zero impact on perf or availability. Consistent w/in seconds and retained until deleted
- Point in Time Recovery (PITR) - protect against accidental writes or deletes, up to 35 days, incremental backups, not enabled by default. Latest restorable = 5 mins in past
- Stream - time-ordered sequence of item-level changes - stored 24 hrs. Inserts, updates, deletes. Combine w/ lambda’s to simulate stored procedures
- Global Tables - managed multi-master, multi-region replication - based on streams, multi-region redundancy for DR or HA. No application requires. Replication latency under one second.
- Database Migration Service (target, not source)
- Encrypted at rest using KMS. Site to Site VPN, Direct Connect (DX), IAM, Fine-grained access, VPC endpoints, Cloudwatch and CloudTrail
- Strongly Consistent Reads can be used but may have higher latency
- DynamoDB allows for the storage of large text and binary objects, but there is a limit of 400 KB.
Redshift
- Fully managed petabyte-scale data warehouse
- no commitments/upfront costs - 0.25/hr -> 1K per TB per year
- Single Node (160Gb)
- Multi-Node
- Leader Node - manages client conx and receives queries
- Compute Node - store data and perform queries and computations (up to 128 compute nodes)
- Advanced Compression - columnar data stores can be compressed much more than row-based. doesn’t require materialized views, etc
- Massively parallel processing
- Backups - enabled by default w/ a 1 day retention (max 35)
- attempts to maintain at least 3 copies of data (original, replica on compute nodes, and backup in s3)
- async replicate your snapshots to s3 in another region for DR
- Charges - compute node only - 1 unit per node per hour + backup + data transfer w/in VPC
- Security - transit + rest. KMS but can use HSM
- Currently only available in 1 AZ
- Can restore snapshots in another AZ
Aurora
- MySQL & PostgreSQL compatible
- Up to 5x perf over MySQL; 3x over PGSQL
- 10GB, scale in 10GB increments to 64TB
- Compute can scale to 32vcpu and 244 gb memory
- 2 copies of data in each AZ, minimum 3 az = 6 copies of your data
- Transparently handle the loss of up to 2 copies of data w/o affecting write avail. up to 3 copies w/o affecting read availability
- Self healing - data blocks and disks are scanned for errors and repaired automatically
- Aurora Replicas - max 15 - in region replications
- Mysql Read Replicas - max 5
- PGSQL read Replicas - max 1
- Backups are always enabled. Snapshots can be shared w/ other AWS accounts.
- Aurora Serverless - automatically starts/stops/scales - for infrequent, intermittent or unpredictable workloads
ElastiCache
- In memory cache in the cloud - increase DB and web application performance
- memcached - simple
- Redis - pub sub, complex data types, backup and restores, multi-az
Database Migration Service (DMS)
- migrate to cloud, between on-premises, combos
- Server that runs replication software - source/target
- Will create tables and primary keys, but can pre-create manually or use AWS Schema Conversion Tool (SCT) needed for heterogeneous
- Supports homogenous / heterogeneous migrations
- Sources - usual DBs, Azure SQL, RDS, S3
- Targets - usual DBs, RDS, Redshift, DynamoDB, S3, Elasticsearch, Kinesis data streams, DocumentDB
Caching Services
- CloudFront
- API Gateway
- ElastiCache - Redis / memcached
- DynamoDB Application Accelerator - DAX
Amazon EMR
- Elastic Map Reduce
- Spark, Hive, HBase, Flink, Hudi, Presto
- Petabyte-scale analysis at 1/2 cost of traditional on-prem and 3x faster than Apache Spark
- Cluster - nodes w/ roles
- Master node - status of tasks & monitors health - every cluster has a master node
- Core node - runs tasks and stores data in Hadoop Distributed File System (HDFS) - multi-node clusters have at least one core node
- Task node - only runs task and does NOT store data in HDFS - optional
- Log files stored on master node at /mnt/var/log
- configure to archive logs to S3 at 5-minute intervals in case of normal shutdown or error
- must be done when first set up cluster