Using Incremental Learning Job in Helmet Detection Scenario on S3

This example is based on the example: Using Incremental Learning Job in Helmet Detection Scenario.

Prepare Nodes

Assume you have created a KubeEdge cluster that have two cloud nodes(e.g., cloud-node1, cloud-node2) and one edge node(e.g., edge-node).

Create a secret with your S3 user credential.

kubectl create -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: mysecret
  annotations:
    s3-endpoint: play.min.io # replace with your s3 endpoint
    s3-usehttps: "1" # by default 1, if testing with minio you can set to 0
stringData:
  ACCESS_KEY_ID: Q3AM3UQ867SPQQA43P2F
  SECRET_ACCESS_KEY: zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG
EOF

Prepare Model

• Download models.

• Put the unzipped model file into the bucket of your cloud storage service.

• Attach the created secret to the Model and create Model.

kubectl create -f - <<EOF
apiVersion: sedna.io/v1alpha1
kind: Model
metadata:
  name: initial-model
  spec:
    url : "s3://kubeedge/model/base_model"
    format: "ckpt"
    credentialName: mysecret
EOF

kubectl $action -f - <<EOF
apiVersion: sedna.io/v1alpha1
kind: Model
metadata:
  name: deploy-model
spec:
  url: "s3://kubeedge/model/deploy_model/saved_model.pb"
  format: "pb"
  credentialName: mysecret
EOF

Prepare Dataset

• Download dataset.

• Put the unzipped dataset file into the bucket of your cloud storage service.

• Attach the created secret to Dataset and create Dataset.

kubectl $action -f - <<EOF
apiVersion: sedna.io/v1alpha1
kind: Dataset
metadata:
  name: incremental-dataset
spec:
  url: "s3://kubeedge/data/helmet_detection/train_data/train_data.txt"
  format: "txt"
  nodeName: cloud-node1
  credentialName: mysecret
EOF

Prepare Image

This example uses the image:

kubeedge/sedna-example-incremental-learning-helmet-detection:v0.3.1

This image is generated by the script build_images.sh, used for creating training, eval and inference worker.

Prepare Job

• Inference/Train/Eval worker now can be deployed by nodeName and nodeSelector on multiple nodes.

• Make sure to follow the local dir which exists in edge side.

mkdir -p /incremental_learning/video/

• Download video, unzip video.tar.gz, and put it into /incremental_learning/video/.

cd /incremental_learning/video/
wget https://kubeedge.obs.cn-north-1.myhuaweicloud.com/examples/helmet-detection/video.tar.gz
tar -zxvf video.tar.gz

• Attach the created secret to the Job and create Job.

IMAGE=kubeedge/sedna-example-incremental-learning-helmet-detection:v0.3.1

kubectl create -f - <<EOF
apiVersion: sedna.io/v1alpha1
kind: IncrementalLearningJob
metadata:
  name: helmet-detection-demo
spec:
  initialModel:
    name: "initial-model"
  dataset:
    name: "incremental-dataset"
    trainProb: 0.8
  trainSpec:
    template:
      spec:
        nodeName: cloud-node1
        containers:
          - image: $IMAGE
            name:  train-worker
            imagePullPolicy: IfNotPresent
            args: ["train.py"]
            env:
              - name: "batch_size"
                value: "32"
              - name: "epochs"
                value: "1"
              - name: "input_shape"
                value: "352,640"
              - name: "class_names"
                value: "person,helmet,helmet-on,helmet-off"
              - name: "nms_threshold"
                value: "0.4"
              - name: "obj_threshold"
                value: "0.3"
    trigger:
      checkPeriodSeconds: 60
      timer:
        start: 02:00
        end: 20:00
      condition:
        operator: ">"
        threshold: 500
        metric: num_of_samples
  evalSpec:
    template:
      spec:
        nodeName: cloud-node2
        containers:
          - image: $IMAGE
            name:  eval-worker
            imagePullPolicy: IfNotPresent
            args: ["eval.py"]
            env:
              - name: "input_shape"
                value: "352,640"
              - name: "class_names"
                value: "person,helmet,helmet-on,helmet-off"
  deploySpec:
    model:
      name: "deploy-model"
    trigger:
      condition:
        operator: ">"
        threshold: 0.1
        metric: precision_delta
    hardExampleMining:
      name: "IBT"
      parameters:
        - key: "threshold_img"
          value: "0.9"
        - key: "threshold_box"
          value: "0.9"
    template:
      spec:
        nodeName: edge-node
        containers:
          - image: $IMAGE
            name:  infer-worker
            imagePullPolicy: IfNotPresent
            args: ["inference.py"]
            env:
              - name: "input_shape"
                value: "352,640"
              - name: "video_url"
                value: "file://video/video.mp4"
              - name: "HE_SAVED_URL"
                value: "/he_saved_url"
            volumeMounts:
              - name: localvideo
                mountPath: /video/
              - name: hedir
                mountPath: /he_saved_url
            resources:  # user defined resources
              limits:
                memory: 2Gi
        volumes:   # user defined volumes
          - name: localvideo
            hostPath:
              path: /incremental_learning/video/
              type: DirectoryOrCreate
          - name: hedir
            hostPath:
              path:  /incremental_learning/he/
              type: DirectoryOrCreate
  credentialName: mysecret
  outputDir: "s3://kubeedge/incremental_learning/output"
EOF