Google Cloud

Introduction

gcloud command line tool

https://cloud.google.com/sdk/downloads

https://cloud.google.com/sdk/gcloud/reference/

Kubernetes Engine

https://cloud.google.com/kubernetes-engine/docs/https://kubernetes.io/docs/reference/kubectl/cheatsheet/#kubectl-context-and-configuration https://cloud.google.com/kubernetes-engine/docs/tutorials/installing-istio

k8s login GCP preprod

gcloud container clusters get-credentials prod --zone europe-west1-b --project project-name

https://kubernetes.io/docs/reference/kubectl/cheatsheet/

https://eu.udacity.com/course/scalable-microservices-with-kubernetes–ud615

Big Table

https://
cloud.google.com/bigtable/docs/go/cbt-overview
/Users/username/.cbtrc
project = project-name
instance = instance-name
$cbt ls

Storage

https://console.cloud.google.com/storage/browser/

https://cloud.google.com/storage/

Load Balancing

"By default, to distribute traffic to instances, the Session Affinity is set to NONE. Google Cloud Load Balancing picks an instance based on a hash of the source IP and port, destination IP and port, and protocol. This means that incoming TCP connections are spread across instances and each new connection may go to a different instance. All packets for a connection are directed to the same instance until the connection is closed. Established connections are not taken into account when balancing.

Regardless of the session affinity setting, all packets for a connection are directed to the chosen instance until the connection is closed and have no impact on load balancing decisions for new incoming connections. This can result in imbalance between backends if long-lived TCP connections are in use.

You can choose a different Session Affinity setting if you need multiple connections from a client to go to the same instance. See sessionAffinity in the Target Pools documentation for more information." "In any of these proxy model, any traffic bound for the Service's IP:Port is proxied to an appropriate backend without the clients knowing anything about Kubernetes or Services or Pods. Client-IP based session affinity can be selected by setting service.spec.sessionAffinity to "ClientIP" (the default is "None"), and you can set the max session sticky time by setting the field service.spec.sessionAffinityConfig.clientIP.timeoutSeconds if you have already set service.spec.sessionAffinity to "ClientIP" (the default is "10800")."

Preemptible VM Instances

https://cloud.google.com/compute/docs/instances/preemptible

Deployment Manager

Deployment Manager is an infrastructure deployment service that automates the creation and management of Google Cloud Platform resources for you.

Write flexible template and configuration files and use them to create deployments that have a variety of Cloud Platform services, such as Google Cloud Storage, Google Compute Engine, and Google Cloud SQL, configured to work together.

  1. Preview Configuration https://cloud.google.com/deployment-manager/docs/configuration/preview-configuration-file (project with activated billing is needed)

  2. Alternative to deployment-manager is https://www.terraform.io/docs/providers/google/index.html

custom role

### jinja
resources:
- name: custom-role
type: gcp-types/iam-v1:projects.roles
properties:
parent: projects/{{ env["project"] }}
roleId: {{ properties["roleId"] }}
role:
title: {{ properties["title"] }}
description: {{ properties["description"] }}
stage: {{ properties["stage"] }}
includedPermissions: {{ properties["includedPermissions"] }}
### yaml
- name: app-bigtable-role
type: roles/project_custom_role.jinja
properties:
roleId: bigtable.app
title: App Bigtable role
description: Role for managing BigTable
stage: GA
includedPermissions:
- bigtable.tables.create
- bigtable.tables.update
- bigtable.tables.checkConsistency
- bigtable.tables.generateConsistencyToken
- bigtable.tables.get
- bigtable.tables.list
- bigtable.tables.mutateRows
- bigtable.tables.readRows
- bigtable.tables.sampleRowKeys
- bigtable.appProfiles.get
- bigtable.appProfiles.list
- bigtable.clusters.get
- bigtable.clusters.list
- monitoring.metricDescriptors.get
- monitoring.metricDescriptors.list
- monitoring.timeSeries.list
- resourcemanager.projects.get

service account

### jinja
resources:
- name: {{ env['name'] }}
type: iam.v1.serviceAccount
properties:
accountId: {{ properties['name'] }}
displayName: serviceAccount-{{ properties['name'] }}
### yaml
- name: app-bigtable-sa-prod
type: accounts/service_account.jinja
properties:
name: app-prod

add roles to service accounts

- name: get-iam-policy
action: gcp-types/cloudresourcemanager-v1:cloudresourcemanager.projects.getIamPolicy
properties:
resource: project-id
- name: add-iam-policy
action: gcp-types/cloudresourcemanager-v1:cloudresourcemanager.projects.setIamPolicy
properties:
resource: project-id
policy: $(ref.get-iam-policy)
gcpIamPolicyPatch:
add:
- role: projects/project-name/roles/bigtable.app # custom role
members:
- serviceAccount:app-prod@project-id.iam.gserviceaccount.com
- serviceAccount:app-stg@project-id.iam.gserviceaccount.com
- role: roles/bigtable.reader
members:
- serviceAccount:app-prod-reader@project-id.iam.gserviceaccount.com
- serviceAccount:app-stg-reader@project-id.iam.gserviceaccount.com

Makefile deployment

DRY_RUN=true
ifeq ($(DRY_RUN), false)
EXTRA_ARGS=
else
EXTRA_ARGS=--preview
endif
# preprod
.PHONY: create-deployment
create-deployment:
gcloud deployment-manager deployments create deployment-name --config config.yaml --project project-id $(EXTRA_ARGS)
.PHONY: update-deployment
update-deployment:
gcloud deployment-manager deployments update deployment-name --config config.yaml --project project-id $(EXTRA_ARGS)
.PHONY: delete-deployment
delete-deployment:
gcloud deployment-manager deployments delete deployment-name --project project-id

Problems encountered

  1. Created a role bigtable.some_name , few weeks later that bigtable prefix became reserved name

  2. Created service accouunt manually, later created account and assigned role using deployment manager. Older role was also assigned and was acting like the old one. Unassigned and assigned correct role and started working.

  3. Was hard to find correct api resource for IAM. Cloud Resource Manager was the answer.

Terraform

Roles, IAM, ServiceAccounts, BigTable done with Terraform

provider "google" {
project = "project-name"
}
### Big Table
resource "google_bigtable_instance" "bigtable-name" {
name = "bigtable-name"
cluster_id = "bigtable-name-cluster"
zone = "europe-west1-b"
num_nodes = 3
storage_type = "SSD"
}
### Custom role
resource "google_project_iam_custom_role" "app-bigtable-role" {
role_id = "bigtable.app"
title = "App BigTable Role"
description = "Role for managing BigTable"
permissions = ["bigtable.tables.create",
"bigtable.tables.update",
"bigtable.tables.checkConsistency",
"bigtable.tables.generateConsistencyToken",
"bigtable.tables.get",
"bigtable.tables.list",
"bigtable.tables.mutateRows",
"bigtable.tables.readRows",
"bigtable.tables.sampleRowKeys",
"bigtable.appProfiles.get",
"bigtable.appProfiles.list",
"bigtable.clusters.get",
"bigtable.clusters.list",
"monitoring.metricDescriptors.get",
"monitoring.metricDescriptors.list",
"monitoring.timeSeries.list",
"resourcemanager.projects.get",]
}
### service account
resource "google_service_account" "app-bigtable-sa-prod" {
account_id = "app-prod"
display_name = "serviceAccount-app-prod"
}
### service account
resource "google_service_account" "app-bigtable-sa-prod-reader" {
account_id = "app-prod-reader"
display_name = "serviceAccount-app-prod-reader"
}
### assigned custom role
resource "google_project_iam_binding" "app-prod-role" {
role = "projects/project-name/roles/bigtable.app"
members = ["serviceAccount:app-prod@project-name.iam.gserviceaccount.com"]
}
### assigned predefined BigTable role
resource "google_project_iam_binding" "app-prod-role-reader" {
role = "roles/bigtable.reader"
members = ["serviceAccount:app-prod-reader@project-name.iam.gserviceaccount.com"]
}