Customizing Kubeflow on GKE

This guide describes how to customize your deployment of Kubeflow on Google Kubernetes Engine (GKE) in Google Cloud Platform (GCP).

Before you start

This guide assumes you have already set up Kubeflow with GKE. If you haven’t done so, follow the guide to deploying Kubeflow on GCP.

Customizing Kubeflow

You can use kustomize to customize Kubeflow.

The deployment process is divided into two steps, generate and apply, so that you can modify your deployment before actually deploying.

To customize GCP resources (such as your Kubernetes Engine cluster), you can modify the deployment manager configs in ${KFAPP}/gcp_config.

Many changes can be applied to an existing configuration in which case you can run:

cd ${KFAPP}
kfctl apply platform

or using Deployment Manager directly:

cd ${KFAPP}/gcp_config
gcloud deployment-manager --project=${PROJECT} deployments update ${DEPLOYMENT_NAME} --config=cluster-kubeflow.yaml

• PROJECT Name of your GCP project. You could find it in ${KFAPP}/app.yaml.
• DEPLOYMENT_NAME Name of your Kubeflow app. You could also find it in ${KFAPP}/app.yaml. In specific, .metadata.name

Some changes (such as the VM service account for Kubernetes Engine) can only be set at creation time; in this case you need to tear down your deployment before recreating it:

cd ${KFAPP}
kfctl delete all
kfctl apply all

To customize the Kubeflow resources running within the cluster you can modify the kustomize manifests in ${KFAPP}/kustomize. For example, to modify settings for the Jupyter web app:

cd ${KFAPP}/kustomize
gvim jupyter-web-app.yaml

Find and replace the parameter values:

apiVersion: v1
data:
  ROK_SECRET_NAME: secret-rok-{username}
  UI: default
  clusterDomain: cluster.local
  policy: Always
  prefix: jupyter
kind: ConfigMap
metadata:
  labels:
    app: jupyter-web-app
    kustomize.component: jupyter-web-app
  name: jupyter-web-app-parameters
  namespace: kubeflow

You can then redeploy using kfctl:

cd ${KFAPP}
kfctl apply k8s

or using kubectl directly:

cd ${KFAPP}/kustomize
kubectl apply -f jupyter-web-app.yaml

Common customizations

Add GPU nodes to your cluster:

• Set gpu-pool-initialNodeCount here.

Add Cloud TPUs to your cluster:

• Set enable_tpu:true here.

Add VMs with more CPUs or RAM:

• Change the machineType.
• There are two node pools:
  • one for CPU only machines here.
  • one for GPU machines here.
• When making changes to the node pools you also need to bump the pool-version here before you update the deployment.

Add users to Kubeflow:

• To grant users access to Kubeflow, add the “IAP-secured Web App User” role on the IAM page in the GCP console. Make sure you are in the same project as your Kubeflow deployment.

• You can confirm the update by inspecting the IAM policy for your project:

gcloud projects get-iam-policy ${PROJECT}

• In the output from the above command, users able to access Kubeflow have the following role: roles/iap.httpsResourceAccessor.

More customizations

Refer to the navigation panel on the left of these docs for more customizations, including using your own domain, setting up Cloud Filestore, and more.

Feedback

Was this page helpful?

Yes No

Glad to hear it! Please tell us how we can improve.

Sorry to hear that. Please tell us how we can improve.