Report on the Recent
WestGrid Related Projects

Daniel Graves and Mina Nozar

TWIST Collaboration meeting

December 12th, 2004

Motivation

Increase ease of WG job submission

Reduce the frequency and the amount of operator monitoring time of the WG jobs

Reduce node dependencies on each other

Maximize node usage per WG job

Add visual tools for tracking available WG resources and for monitoring the progress of job executions

WG processing info

Available processing resources at UBC

504 compute nodes each with dual 3.06 GHz Intel xeon processors

MC Jobs

100K events/run(file)

Generation (Geant3): ~2 hrs

Analysis (Mofia): ~1 hr

Data Jobs

~800K events/2GB run(file)

Analysis (Mofia): ~8 hrs

Possible Solutions

For MC jobs

Combine generation and analysis of a given job

Chain multiple generation and analysis jobs together

Increase MP spectrum sample size

Add the capability of using more than one sample per geant run during the generation

For Data jobs

Chain multiple analysis jobs together

Possible Solutions

Visual tools

Use Const's examples on TWIST Cluster disk space monitoring web pages.

Combine updated files provided by Roman from WG side with cron jobs and databases to make similar monitoring pages on our side.

Completed Projects

New Scripts and modifications/updates of existing ones

tbsub.pl

tbsub_checkMC_and_resubmit.pl

tbdel.pl

Implementation of web-based databases

Job status

Disk space usage

Web-based database for job-specific parameters

Documentation for completed projects

New Scripts: tbsub.pl

Purpose

Maximizes efficiency of resource usage

Uniformity in running scripts and saving parameter settings for jobs

Adds simplicity to job resubmission

Increases automation

Capabilities

Submission of GEANT & MOFIA together

Submission of GEANT only or MOFIA only

Chaining multiple GEANT and/or MOFIA jobs

New Scripts: tbsub.pl

Improvements

Addition of run number to PBS job names to provide the capability for job resubmission while jobs in the queue

Automatic checking of GEANT output of MC prior to MOFIA execution

Automatic checking of MOFIA trees upon completion of MOFIA execution

Bug fixes in the original tbsub_geant.pl script.

eg: failure of jobs at submission time due to a race condition related to the job directory tree creation

New Scripts: tbsub_checkMC_and_resubmit.pl

Purpose

GEANT resubmission of failed MC jobs

Capabilities

Resubmission while GEANT jobs are still running or queued

Resubmission of aborted GEANT jobs because of PBS specific reasons, network problems or node failures

New Scripts: tbdel.pl

Purpose

Deletion of all submitted PBS jobs of a given gen or set using qdel

Quick recovery from user mistakes at the time of job submissions.

Web-based databases: WG Disk Space Usage

Purpose

Reporting e614 usage on local nodes '/data' filesystem

Using df

Displaying disk usage on '/global/scratch' and '/global/home' filesystems

Using Westgrid-provided cron job outputs (du)

Features

Sorting by node name or percentage of free space

Filtering by nodes down

Filtering by nodes < 30% free space

Updated daily

http://e614db.triumf.ca/private/sysman/wg_disk_db/websummary.pl

Web-based databases: WG Job Status reporting

Purpose

Reporting all PBS jobs in the queue on Westgrid

Features

Job summary categorized by user name

Detailed information of user jobs by clicking on user name

Updated every 30 minutes

http://e614db.triumf.ca/private/sysman/wg_jobsrunning_db/websummary.pl

Web-based database: Job specific parameters

Purpose

Book-keeping of parameters used for a given MC generation/analysis and Data analysis

Updates are kept on the corresponding 2004 Analysis web pages:

http://e614db.triumf.ca/private/TWIST_2004_Analysis/DATA_Analysis.html

http://e614db.triumf.ca/private/TWIST_2004_Analysis/MC_Analysis.html

http://e614db.triumf.ca/private/TWIST_2004_Analysis/TWIST_04_Analysis.html

http://e614db.triumf.ca/private/TWIST_2004_Analysis/DATA_Analysis.html

http://e614db.triumf.ca/private/TWIST_2004_Analysis/MC_Analysis.html

Documentation

Update of the existing WestGrid notes under:

http://tw04.triumf.ca/private/sysman/Westgrid.html

Addition of notes on:

Description on the usage of scripts, tbsub.pl, tbsub_checkMC_and_resubmit.pl, and tbdel.pl

Description of the 'inner-workings' of the disk usage and job status databases

2004-11-12T11:25:59