huawei-cloud-cloudrobo-dataset

Windows / PowerShell: Examples use bash syntax. To run on Windows PowerShell:

• Flatten \ line continuations to a single line, or end lines with a backtick.
• Set env vars with $env:NAME="value" instead of export NAME="value".
• Single-quoted JSON '{"a":"b"}' works as-is.

Overview

The cloudrobo-dataset skill manages the full lifecycle of CloudRobo data processing (proc-tasks) and data evaluation (eval-tasks). It covers algorithm query, task CRUD operations, status polling, log retrieval (system + job) and download, output data preview, frame extraction, algorithm discovery from the asset marketplace, processing→evaluation pipeline orchestration, batch task management, and failure diagnosis.

Applicable scenarios: Task management (CRUD + restart), troubleshooting (log retrieval + error analysis), algorithm discovery, result verification (preview + frames), pipeline orchestration, batch management, analytics.

Architecture:

Agent / LLM
    │
    ├── CLI  →  cloudrobo dataset proc <command>     (proc-tasks)
    │           cloudrobo dataset eval <command>     (eval-tasks)
    ├── SDK  →  DatasetClient (Python)
                    │
                    ▼
              cloudrobo-service (REST API)
              /v1/data-eng/proc-tasks/*
              /v1/data-eng/eval-tasks/*

All operations target the cloudrobo-service backend and require a workspace_id (default workspace is used unless --workspace-id overrides it). Algorithm and dataset discovery is a cross-package operation that calls the cloudrobo-asset-manager service via cloudrobo asset.

Prerequisites

See references/cli-installation-guide.md for CLI installation, AK/SK authentication, and workspace configuration. All dataset operations require a valid workspace_id (default or explicit --workspace-id).

Workflow

Interactive Task Creation

All interactive creation workflows follow the Interaction Rules: at most one tool call per turn, display query results before asking the next question.

Proc-Task Creation (10 steps): Task name → Description → Algorithm type (PRESET_ASSETS/WORKSPACE_ASSETS/OBS_ASSETS) → Algorithm configuration with field mapping → Environment variables → Job type and resources (CCE/K8S or CCE_RAY/RAY, resource pool, instance spec) → Dynamic storage (public pool only) → Input datasets (multi-dataset loop) → Output configuration → Confirm and submit. See references/interactive-workflow.md for detailed steps.

Eval-Task Creation (7 steps): Task name → Description → Resource pool and instance spec → Evaluation algorithm → Dataset (single) → Robot config → Confirm and submit. See references/interactive-workflow.md for detailed steps.

Task Polling and Result Handling

1. Poll status — dataset proc wait-task (recommended 15s interval), report each state change to user, until terminal state (SUCCEEDED/FAILED/DELETED)
2. On completion:
   • SUCCEEDED: view processing result via get-preview, output system log
   • FAILED: ask user whether to view logs, output system and job logs

Eval tasks: After SUCCEEDED, extract report path from target_report_path in task detail, use dataset eval get-preview --task-id <task-id> --file-name <obs-path-without-bucket> to get OBS temporary link.

Pipeline Workflow (Task Orchestration)

Scenario: data conversion → data evaluation → download artifact

1. Run processing task, wait for SUCCEEDED
2. Extract target_path and target_asset_id from task detail
3. Construct dataset_configs entry with obs_path = target_path, asset_id = target_asset_id
4. Create evaluation task, wait for SUCCEEDED
5. Get report: dataset eval get-preview --file-name <path-from-target_report_path> (remove obs://<bucket>/ prefix)
6. Download artifacts via download-asset

Batch Workflow (Batch Tasks)

Scenario: run same algorithm across multiple datasets

1. Get dataset list (asset list-assets --type dataset)
2. Create processing task per dataset (reuse algorithm/resource config from first; only input dataset differs)
3. Collect task_ids, poll status
4. Summarize success/failure/in-progress counts
5. For failed tasks: view logs or restart

Diagnosis Workflow (Failure Diagnosis)

Scenario: task FAILED → analyze logs, suggest fixes

1. Get system and user logs
2. Analyze error patterns: OOM → increase worker_spec.memory; path not found → check dataset_configs; image pull failure → check image; parameter error → check algo_entrance/envs; access denied → check catalog_id/permissions
3. Output diagnosis and fix suggestions
4. After confirmation, modify via update-task or restart via restart-task

Analytics Workflow (Data Analysis)

Aggregate task stats by status, algorithm ranking, time trend, failure rate. Output analysis report with root-cause suggestions.

Long-Running Task Workflow

Start wait-task with timeout. On timeout, report status and elapsed time; user decides: continue / view logs / terminate.

CLI Command Format Standard

cloudrobo dataset proc <command> [OPTIONS]
cloudrobo dataset eval <command> [OPTIONS]

Core Commands

Task Management (proc-tasks)

Create a data processing task

cloudrobo dataset proc create-task --name <task-name> --algo-type PRESET_ASSETS --task-config '<json-config>' [--workspace-id <id>] [--wait] [--timeout 1800] [--dry-run]

• SDK: client.create_task(task_config: dict)
• API: POST /v1/data-eng/proc-tasks

Required task_config fields: name, algo_type (PRESET_ASSETS / WORKSPACE_ASSETS / OBS_ASSETS, based on Step 3 selection), algo_name, algo_entrance, image, algo_id, catalog_id (workspace's), resource_pool_type, cluster_type, task_framework_type, dataset_configs (JSON string array, multi-dataset input), output_type, output_path, output_name, head_spec (must include cpu/memory/gpu/npu), worker_spec (must include cpu/memory/gpu/npu), worker_num, evs_spec. See references/task-config-catalog.md for full field mapping.

⚠️ Non-Empty Validation: Except description, all parameters cannot be empty values (empty string "", None, empty dict {}). The SDK validates this before API submission. head_spec/worker_spec must include all four keys: cpu, memory, gpu, npu (value 0 is valid). dataset_configs must contain at least 1 dataset entry; empty array "[]" is invalid.

List tasks

cloudrobo dataset proc list-tasks [--status RUNNING|SUCCEEDED|FAILED] [--algo-type <type>] [--name <name>] [--order-by start_at|update_at|finish_at] [--order DESC|ASC] [--offset <n>] [--limit <n>] [--user-id <id>] [--algo-name <name>] [--output-name <name>] [--workspace-id <id>]

• SDK: client.list_tasks(workspace_id=None, statuses=..., algo_type=..., name=..., order_by=..., ...)
• API: GET /v1/data-eng/proc-tasks?workspace_id=<id>&statuses=<status>

Show task detail

cloudrobo dataset proc show-task --task-id <task-id>

• SDK: client.get_task_detail(task_id)
• API: GET /v1/data-eng/proc-tasks/{task_id}

Update a task (SDK only)

• SDK: client.update_task(task_id, task_config: dict)
• API: PATCH /v1/data-eng/proc-tasks/{task_id}

Delete tasks (SDK only)

• SDK: client.delete_tasks([task_id_1, task_id_2])
• API: DELETE /v1/data-eng/proc-tasks?ids=id1,id2

Restart a task

cloudrobo dataset proc restart-task --task-id <task-id>

• SDK: client.restart_task(task_id)
• API: POST /v1/data-eng/proc-tasks/{task_id}/restart

Task Monitoring

Wait for task to reach terminal state

cloudrobo dataset proc wait-task --task-id <task-id> [--timeout 1800] [--interval 10]

Terminal states: SUCCEEDED, FAILED, DELETED

• SDK: client.wait_task(task_id, timeout=1800, interval=10, on_status=callback)
• API: GET /v1/data-eng/proc-tasks/{task_id} (polled)

Retrieve task logs

# Step 1: List log files to get file_path
cloudrobo dataset proc get-log --task-id <id> --is-system true
cloudrobo dataset proc get-log --task-id <id> --is-system false

# Step 2: Get log content (default: latest 64KB tail)
cloudrobo dataset proc get-log --task-id <id> --file-name <name> --file-path <path>

# Get full log (not just tail)
cloudrobo dataset proc get-log --task-id <id> --file-name <name> --file-path <path> --all

• SDK: client.list_log_files(task_id, is_system=True) → client.get_task_log(task_id, file_name, file_path, start_byte, end_byte) or client.get_task_log_tail(task_id, file_name)
• API:
  • GET /v1/data-eng/proc-tasks/{task_id}/logs?is_system=true|false
  • GET /v1/data-eng/proc-tasks/{task_id}/logs/{file_name}?start_byte=&end_byte=&file_path=&job_id=

Download a log file

cloudrobo dataset proc download-log --task-id <id> --file-name <name> --file-path <path>

• SDK: client.download_task_log(task_id, file_name, file_path)
• API: GET /v1/data-eng/proc-tasks/{task_id}/logs/{file_name}/download

Get task resource usage

cloudrobo dataset proc get-resource-usage --task-id <task-id> --metric CPU_UTIL|CPU_USED_CORE|MEM_UTIL|MEM_USED_MB|NETWORK_TX_RATE|NETWORK_RX_RATE|DISK_READ_KB|DISK_WRITE_KB --start <unix-ts-sec> --end <unix-ts-sec> --step <10-3600>

• SDK: client.get_task_resource_usage(task_id, metric, start, end, step)
• API: GET /v1/data-eng/proc-tasks/{task_id}/resource-usage?metric=&start=&end=&step=

Data Preview

Preview task output data

Get OBS temporary download link for a dataset file. file_name = OBS path excluding bucket name (e.g., cloudrobo/f91cee72-.../1ddf1498-.../data/chunk-000/file-000.parquet).

cloudrobo dataset proc get-preview --task-id <task-id> --file-name <obs-path-without-bucket>

• SDK: client.get_task_preview(task_id, file_name)
• API: GET /v1/data-eng/proc-tasks/{task_id}/preview?file_name=<file_name>

Get task frames

Query directory file list of task input/output datasets. prefix = dataset OBS path excluding bucket name (e.g., cloudrobo/f91cee72-.../1ddf1498-.../).

cloudrobo dataset proc get-frames --task-id <task-id> --prefix <obs-path-without-bucket>

• SDK: client.get_task_frames(task_id, prefix)
• API: GET /v1/data-eng/proc-tasks/{task_id}/frames?prefix=<prefix>

Evaluation Tasks (eval-tasks)

Evaluation tasks differ from processing tasks in field names, deletion granularity, and supported operations. See the full comparison table in references/task-config-catalog.md.

Create an evaluation task

cloudrobo dataset eval create-task --name <task-name> --task-config '<json-config>' [--workspace-id <id>] [--wait] [--timeout 1800]

• SDK: client.create_eval_task(task_config: dict)
• API: POST /v1/data-eng/eval-tasks

task-config field description:

The eval-task task-config requires two sets of dataset fields: top-level single-value fields (dataset_type, dataset_id, dataset_name, dataset_path) + dataset_configs array (single dataset, format same as proc-task).

Required fields: name, algo_type, algo_id, algo_name, algo_entrance, image, catalog_id (workspace's), cluster_type, task_framework_type, dataset_type/dataset_id/dataset_name/dataset_path (top-level), dataset_configs, robot_config, resource_pool_type, head_spec, worker_spec, worker_num, evs_spec, output_type, output_path, output_name.

Optional: description, resource_id, dedicated_pool_name.

⚠️ Key reminders:

• Both top-level dataset fields and dataset_configs array required
• dataset_name cannot be empty (extract from OBS path last directory for UDF_OBS_ASSET)
• dataset_type values: BUILD_IN_ASSET / UDF_OBS_ASSET

See references/task-config-catalog.md for full field mapping and eval-task field differences.

List evaluation tasks

cloudrobo dataset eval list-tasks [--status <status>] [--name <name>] [--workspace-id <id>]

• SDK: client.list_eval_tasks(workspace_id=None, **params)
• API: GET /v1/data-eng/eval-tasks?workspace_id=<id>

Show evaluation task detail

cloudrobo dataset eval show-task --task-id <task-id>

• SDK: client.get_eval_task_detail(task_id)
• API: GET /v1/data-eng/eval-tasks/{task_id}

Update an evaluation task (SDK only)

• SDK: client.update_eval_task(task_id, task_config: dict)
• API: PATCH /v1/data-eng/eval-tasks/{task_id}

Delete an evaluation task (single granularity, SDK only)

• SDK: client.delete_eval_task(task_id)
• API: DELETE /v1/data-eng/eval-tasks/{task_id}

Note: eval-tasks deletion is single-task granularity (delete-task), same as proc-tasks (delete-task). eval-tasks do not support restart.

Retrieve evaluation task logs

# Step 1: List log files
cloudrobo dataset eval get-log --task-id <id> --is-system true
cloudrobo dataset eval get-log --task-id <id> --is-system false

# Step 2: Get log content
cloudrobo dataset eval get-log --task-id <id> --file-name <name> --file-path <path>

• SDK: client.list_eval_log_files(task_id, is_system=True) + client.get_eval_task_log(task_id, file_name, ...)
• API:
  • GET /v1/data-eng/eval-tasks/{task_id}/logs?is_system=true|false
  • GET /v1/data-eng/eval-tasks/{task_id}/logs/{file_name}?file_path=&job_id=

Get evaluation task preview

file_name is the OBS path of the report file (excluding bucket name). Extract from task detail's target_report_path by removing obs://<bucket-name>/ prefix.

For example: if target_report_path is obs://cloudrobo-test-203/eval_task/report/abc-123/diversity_evaluation_report.pdf, then file_name = eval_task/report/abc-123/diversity_evaluation_report.pdf.

cloudrobo dataset eval get-preview --task-id <task-id> --file-name <obs-path-without-bucket> [--is-download]

• SDK: client.get_eval_task_preview(task_id, file_name, is_download=False)
• API: GET /v1/data-eng/eval-tasks/{task_id}/preview?file_name=<obs-path-without-bucket>&isDownload=

Returns OBS temporary URL. Use --is-download for download link, omit for preview link. Save or download promptly.

Algorithm Discovery

List available algorithms

cloudrobo asset list-publication-assets --type algorithm --tags "Data Processing" [--name <fuzzy-name>] [--limit 20]

• SDK (cross-package): asset_client.list_publication_assets(type="algorithm", sub_type="data_processing", limit=20)
• API: Cross-package — calls the asset service, not the dataset service directly.

Each algorithm includes ext_metadata with engine.image_url, command, and environment_variables needed for task creation. Use --tags "Data Evaluation" to discover evaluation algorithms.

Reference Documents

• CLI Installation Guide — cloudrobo CLI installation and configuration
• IAM Policies — Least-privilege credential model
• Verification Method — Verification method details
• Dataflow Diagram — Mermaid data flow diagram
• Acceptance Criteria — Acceptance criteria
• Task Config Reference — Algorithm field mapping, required fields template, envs format, eval-tasks field differences
• Interactive Workflow — Detailed step-by-step interactive creation workflows for proc-tasks and eval-tasks

Edge Cases

Verification Method

Specification Compliance Verification

bash scripts/test-cli-commands.sh -s . -e cli

Functional Testing

# CLI / SDK
bash scripts/test-cli-commands.sh -s . -e {cli|sdk}

API-level verification is manual — replay the API paths in references/task-config-catalog.md in API Explorer.

Test Cases

See templates/test-vars.json for the full test case list covering proc-tasks, eval-tasks, algorithm discovery, and pipeline scenarios.

Verification Checklist

• After creating a task, poll status (recommended 15s interval), report each state change to user
• On SUCCEEDED: verify output via get-preview, output system log
• On FAILED: ask user whether to view logs, fetch system and job logs per the two-step flow (list files → get content)
• For pipeline: verify processing target_path is wrapped into dataset_configs for the eval task, then eval SUCCEEDED → get-preview returns OBS link
• For eval tasks: verify report preview after SUCCEEDED

Best Practices

• Always run list-algorithms first to discover available operators before creating a task
• Use --dry-run with create-task to validate task config before actual execution
• Use wait-task with --timeout to avoid indefinite polling; report each status transition
• On task failure, retrieve both system logs (--is-system true) and job logs (--is-system false), analyze error patterns (OOM / path not found / image pull failure / OBS access) and suggest fixes before restarting
• For pipeline orchestration, wait for processing SUCCEEDED before creating evaluation task
• Before batch delete, confirm task IDs to avoid irreversible deletion
• Clean up completed/failed tasks with delete-task to free resources
• Set CLOUDROBO_DEBUG=1 for verbose error output during troubleshooting
• Evaluation report links are OBS temporary URLs — save or download promptly