Cloud Storage

Changing Default Storage Classes

  • When an object is uploaded, the default class is applied to the object, unless a storage class is specified
  • The default storage class of a bucket can be changed
  • The default storage class of an object can be changed as well
  • Objects can be moved from bucket to bucket

Access Control

  • IAM roles and permissions
  • ACLs: Access Control Lists
  • Signed URL: timed access to a bucket or object
  • Signed Policy Document
  • ACLs:
    • A mechanism of defining who has access to objects and what actions can it perform
    • Maximum number of ACLs per object is 100
    • An ACL contain a Scope (who) and a Permissions (what)
  • Signed URLs:
    • For some application it is easier and more efficient to grant timed access to an object
    • Generate signed url:

      gsutil signurl -d 10 m <path-to-private-key> gs://bucket/object
      

Cloud Storage Features

  • Customer-supplied encryption keys (CSEK)
  • Object Lifecycle Management for automatically deleting or archiving objects
  • Object versioning
  • Directory synchronization: sync a VM with a bucket
  • Object change notification
  • Data import
  • Strong consistency

Object Versioning

  • Objects are immutable
  • With versioning we can maintain a history of modifications of an object
  • We can list archived versions of an object, restore an object to an older state or delete a specific version
  • Versioning can be turned on or off

Lifecycle Management Policies

  • They specify actions to be performed on objects that meet certain rules
  • Examples:
    • Downgrade storage class on objects older than a year
    • Delete objects created before specific date
    • Keep only the most 3 recent versions of an object
  • Object inspection occurs in async batches
  • Changes can take up to 24 hours to be applied

Object Change Notification

  • Can be used to notify an application when an object is updated or added to a bucket
  • Recommended: use Pub/Sub Notifications for Cloud Storage changes

Data Import Services

  • Transfer Appliance: hardware appliance, rack, used to migrate large amount of data. Transfer appliances are shipped to Google Cloud with data on it from on-premises
  • Storage Transfer Service: import online data from another bucket or other source
  • Offline Media Import: third-party provider uploads the data from a physical media

Cloud Storage Consistency

  • Cloud Storage offers strong global consistency for:
    • Read-after-write
    • Read-after-metadata-update
    • Read-after-delete
    • Bucket listing
    • Object listing

Choosing Storage Class

  • Archive storage: read data less than once per year
  • Coldline storage: read data less than once per 90 days
  • Nearline storage: read data less than once per 30 days
  • Standard storage: hot data
  • Location type:
    • Single region for optimizing latency and bandwidth within a region
    • Dual-region: similar performance for both regions, high availability and geo-redundancy
    • Multi-region: distribute data globally or high availability and geo-redundancy

Uniform Bucket-Level Access

  • In order to support a uniform permissioning system, Cloud Storage has uniform bucket-level access
  • Using this feature on a bucket disables ACLs for all Cloud Storage resources in the bucket; access to Cloud Storage resources then is granted exclusively through IAM
  • After you enable uniform bucket-level access, you can reverse your decision for 90 days

Cloud Storage IAM Roles

  • Storage Object Creator (roles/storage.objectCreator):
    • Allows users to create objects
    • Does not give permission to view, delete, or replace objects
  • Storage Object Viewer (roles/storage.objectViewer):
    • Grants access to view objects and their metadata, excluding ACLs
    • Can also list the objects in a bucket
  • Storage Object Admin (roles/storage.objectAdmin):
    • Grants full control over objects, including listing, creating, viewing, and deleting objects
  • Storage HMAC Key Admin (roles/storage.hmacKeyAdmin):
    • Full control over HMAC keys in a project
    • This role can only be applied to a project
  • Storage Admin (roles/storage.admin):
    • Grants full control of buckets and objects
    • When applied to an individual bucket, control applies only to the specified bucket and objects within the bucket