Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,21 +13,21 @@ The package provides two categories of programs:
|---------|-------|-------------|
| `bashqsub` | | Submit a job as a bash batch job on a PBS node via qsub |
| `tcshqsub` | | Submit a job as a tcsh batch job on a PBS node via qsub |
| `rdasub` | `gdexsub` | Submit a command as a nohup background process on the local machine |
| `gdexsub` | `rdasub` | Submit a command as a nohup background process on the local machine |
| `pgwget` | | Download remote files by root name pattern, combining downloaded parts into a single file |
| `gdexls` | | List local files and directories with matching metadata from the GDEX database |
| `rdaps` | `gdexps` | Show process status for local or PBS batch processes, with filtering by PID, owner, or name |
| `rdazip` | `gdexzip` | Compress or uncompress files using a supported format |
| `rdaown` | `gdexown` | Change ownership of files and directories to rdadata (must be run as root) |
| `gdexps` | `rdaps` | Show process status for local or PBS batch processes, with filtering by PID, owner, or name |
| `gdexzip` | `rdazip` | Compress or uncompress files using a supported format |
| `gdexown` | `rdaown` | Change ownership of files and directories to gdexdata (must be run as root) |
| `pgrst` | | Convert .usg files to RST and push to gdex-docs-* repos on GitHub for readthedocs.io |

**Run as gdexdata via setuid (requires setup below):**

| Command | Alias | Connector script | Description |
|---------|-------|-----------------|-------------|
| `rdacp` | `gdexcp` | `setuid_rdacp` / `setuid_gdexcp` | Copy files and directories across local, remote, Object Store, or Globus endpoints |
| `rdakill` | `gdexkill` | `setuid_rdakill` / `setuid_gdexkill` | Kill local processes and their children, or cancel PBS batch jobs |
| `rdamod` | `gdexmod` | `setuid_rdamod` / `setuid_gdexmod` | Change permission modes for files and directories owned by rdadata |
| `gdexcp` | `rdacp` | `setuid_gdexcp` / `setuid_rdacp` | Copy files and directories across local, remote, Object Store, or Globus endpoints |
| `gdexkill` | `rdakill` | `setuid_gdexkill` / `setuid_rdakill` | Kill local processes and their children, or cancel PBS batch jobs |
| `gdexmod` | `rdamod` | `setuid_gdexmod` / `setuid_rdamod` | Change permission modes for files and directories owned by gdexdata |

## Environment setup

Expand Down Expand Up @@ -86,7 +86,7 @@ pip install rda_python_miscs

## Setuid Setup

The setuid programs (`rdacp`, `rdakill`, `rdamod` and their `gdex*` aliases)
The setuid programs (`gdexcp`, `gdexkill`, `gdexmod` and their `rda*` aliases)
execute as the common user `PGLOG['COMMONUSER']` (default `gdexdata`) via
the `rda_python_setuid` mechanism, which is pulled in automatically as a
dependency. After `pip install` above, choose one of the wiring options
Expand Down
30 changes: 15 additions & 15 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta"

[project]
name = "rda_python_miscs"
version = "3.0.1"
version = "3.0.2"
authors = [
{ name="Zaihua Ji", email="zji@ucar.edu" },
]
Expand Down Expand Up @@ -35,20 +35,20 @@ pythonpath = [
[project.scripts]
"bashqsub" = "rda_python_miscs.bashqsub:main"
"tcshqsub" = "rda_python_miscs.tcshqsub:main"
"rdasub" = "rda_python_miscs.rdasub:main"
"gdexsub" = "rda_python_miscs.rdasub:main"
"gdexsub" = "rda_python_miscs.gdexsub:main"
"rdasub" = "rda_python_miscs.gdexsub:main"
"pgwget" = "rda_python_miscs.pgwget:main"
"gdexls" = "rda_python_miscs.gdexls:main"
"rdaps" = "rda_python_miscs.rdaps:main"
"gdexps" = "rda_python_miscs.rdaps:main"
"rdazip" = "rda_python_miscs.rdazip:main"
"gdexzip" = "rda_python_miscs.rdazip:main"
"rdaown" = "rda_python_miscs.rdaown:main"
"gdexown" = "rda_python_miscs.rdaown:main"
"setuid_rdacp" = "rda_python_miscs.rdacp:main"
"setuid_gdexcp" = "rda_python_miscs.rdacp:main"
"setuid_rdakill" = "rda_python_miscs.rdakill:main"
"setuid_gdexkill" = "rda_python_miscs.rdakill:main"
"setuid_rdamod" = "rda_python_miscs.rdamod:main"
"setuid_gdexmod" = "rda_python_miscs.rdamod:main"
"gdexps" = "rda_python_miscs.gdexps:main"
"rdaps" = "rda_python_miscs.gdexps:main"
"gdexzip" = "rda_python_miscs.gdexzip:main"
"rdazip" = "rda_python_miscs.gdexzip:main"
"gdexown" = "rda_python_miscs.gdexown:main"
"rdaown" = "rda_python_miscs.gdexown:main"
"setuid_gdexcp" = "rda_python_miscs.gdexcp:main"
"setuid_rdacp" = "rda_python_miscs.gdexcp:main"
"setuid_gdexkill" = "rda_python_miscs.gdexkill:main"
"setuid_rdakill" = "rda_python_miscs.gdexkill:main"
"setuid_gdexmod" = "rda_python_miscs.gdexmod:main"
"setuid_rdamod" = "rda_python_miscs.gdexmod:main"
"pgrst" = "rda_python_miscs.pg_rst:main"
26 changes: 13 additions & 13 deletions src/rda_python_miscs/rdacp.py → src/rda_python_miscs/gdexcp.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
#!/usr/bin/env python3
##################################################################################
# Title: rdacp
# Title: gdexcp
# Author: Zaihua Ji, zji@ucar.edu
# Date: 10/24/2020
# 2025-03-10 transferred to package rda_python_miscs from
# https://github.com/NCAR/rda-utility-programs.git
# Purpose: copy files locally and remotely by 'rdadata'
# Purpose: copy files locally and remotely by 'gdexdata'
# Github: https://github.com/NCAR/rda-python-miscs.git
##################################################################################
import re
Expand All @@ -14,16 +14,16 @@
from os import path as op
from rda_python_common.pg_file import PgFile

class RdaCp(PgFile):
"""Copy files and directories locally or between remote hosts via 'rdadata'.
class GdexCp(PgFile):
"""Copy files and directories locally or between remote hosts via 'gdexdata'.

Supports local-to-local, local-to-remote, remote-to-local, and Object Store /
Globus transfers. Target files are owned by 'rdadata' and created with
Globus transfers. Target files are owned by 'gdexdata' and created with
configurable permission modes. Recursive copying is controlled by -r / -R.
"""

def __init__(self):
"""Initialize RdaCp with default copy options and runtime state."""
"""Initialize GdexCp with default copy options and runtime state."""
super().__init__()
self.RDACP = {
'fh': None, # from host name, default to localhost
Expand Down Expand Up @@ -64,8 +64,8 @@ def read_parameters(self):
argv = sys.argv[1:]
self.set_suid(self.PGLOG['EUID'])
self.set_help_path(__file__)
self.PGLOG['LOGFILE'] = "rdacp.log" # set different log file
self.cmdlog("rdacp {} ({})".format(' '.join(argv), self.CINFO['curdir']))
self.PGLOG['LOGFILE'] = "gdexcp.log" # set different log file
self.cmdlog("gdexcp {} ({})".format(' '.join(argv), self.CINFO['curdir']))
defopt = option = 'f'
for arg in argv:
if re.match(r'-(h|-help)$', arg, re.I):
Expand Down Expand Up @@ -95,7 +95,7 @@ def read_parameters(self):
elif option == 'fh':
self.CINFO['fhost'] = arg + '-'
option = defopt
if dohelp or not self.RDACP['f']: self.show_usage("rdacp")
if dohelp or not self.RDACP['f']: self.show_usage("gdexcp")

# function to start actions
def start_actions(self):
Expand All @@ -106,7 +106,7 @@ def start_actions(self):
when specified, then calls copy_top_list. Logs a summary count on completion.
"""
self.dssdb_dbname()
self.validate_decs_group('rdacp', self.PGLOG['CURUID'], 1)
self.validate_decs_group('gdexcp', self.PGLOG['CURUID'], 1)
if not self.RDACP['R'] and self.RDACP['r']: self.RDACP['R'] = 1000
if not self.RDACP['t']:
self.CINFO['tpath'] = self.RDACP['t'] = "."
Expand Down Expand Up @@ -232,10 +232,10 @@ def copy_file(self, fromfile, isfile):

# main function to execute this script
def main():
"""Entry point: instantiate RdaCp, parse arguments, run, and exit."""
"""Entry point: instantiate GdexCp, parse arguments, run, and exit."""
from rda_python_setuid.setup_guide import show_setup_guide
object = RdaCp()
show_setup_guide(object, 'rda_python_miscs', ['rdacp', 'rdakill', 'rdamod'])
object = GdexCp()
show_setup_guide(object, 'rda_python_miscs', ['gdexcp', 'gdexkill', 'gdexmod'])
object.read_parameters()
object.start_actions()
object.pgexit(0)
Expand Down
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@

Copy files and directories to a target location. Source and target may each
reside on the local host, a remote host, an Object Store bucket, or a Globus
endpoint. Target files are owned by 'rdadata' and created with configurable
endpoint. Target files are owned by 'gdexdata' and created with configurable
permission modes.

Usage: rdacp [-f] FromDirectories/Files [-t ToDirectory/FileName] \
Usage: gdexcp [-f] FromDirectories/Files [-t ToDirectory/FileName] \
[-fh FromHostName] [-th ToHostName] \
[-fb FromBucket] [-tb ToBucket] \
[-fp FromGlobusEndpoint] [-tp ToGlobusEndpoint] \
Expand All @@ -14,7 +14,7 @@
default option, so paths may be given without the -f flag.
Shell wildcards are supported. Use './' or '*' to copy everything
in the current directory. Source paths must be readable by user
'rdadata'; rdacp will attempt to fix the mode if they are not;
'gdexdata'; gdexcp will attempt to fix the mode if they are not;

- Option -t, target directory or file name. Defaults to '.' (current
directory). Multiple source files cannot be copied to a single
Expand Down Expand Up @@ -57,21 +57,21 @@
1. Copy all files and subdirectories under the current directory to a
remote host:

rdacp -r -f * -t /PathTo/d277006/ -th castle
gdexcp -r -f * -t /PathTo/d277006/ -th castle

2. Copy the contents of a specific local directory to a remote location
(trailing '/' on source omits the directory entry itself):

rdacp -r -f /PathTo/DirectoryName/ -t /PathTo/d277006/ -th castle
gdexcp -r -f /PathTo/DirectoryName/ -t /PathTo/d277006/ -th castle

Without the trailing '/', DirectoryName itself is also copied:

rdacp -r -f /PathTo/DirectoryName -t /PathTo/d277006/ -th castle
gdexcp -r -f /PathTo/DirectoryName -t /PathTo/d277006/ -th castle

3. Copy a single file to an Object Store bucket:

rdacp -f /PathTo/myfile.nc -tb my-bucket -t myfile.nc
gdexcp -f /PathTo/myfile.nc -tb my-bucket -t myfile.nc

4. Copy files from a remote host to the local current directory:

rdacp -fh castle -f /PathTo/d277006/myfile.nc
gdexcp -fh castle -f /PathTo/d277006/myfile.nc
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
#!/usr/bin/env python3
##################################################################################
# Title: rdakill
# Title: gdexkill
# Author: Zaihua Ji, zji@ucar.edu
# Date: 10/24/2020
# 2025-03-10 transferred to package rda_python_miscs from
# https://github.com/NCAR/rda-utility-programs.git
# Purpose: kill a local or batch process and its child processes for a given
# running process ID by 'rdadata'
# running process ID by 'gdexdata'
# Github: https://github.com/NCAR/rda-python-miscs.git
##################################################################################
import re
import sys
import time
from rda_python_common.pg_file import PgFile

class RdaKill(PgFile):
class GdexKill(PgFile):
"""Kill local processes or PBS batch jobs by process ID, parent PID, or status.

For local processes, sends SIGKILL (-9) to the matched process and all its
Expand All @@ -24,7 +24,7 @@ class RdaKill(PgFile):
"""

def __init__(self):
"""Initialize RdaKill with default kill options."""
"""Initialize GdexKill with default kill options."""
super().__init__()
self.RDAKILL = {
'a': None, # application name
Expand Down Expand Up @@ -52,8 +52,8 @@ def read_parameters(self):
self.dssdb_dbname()
self.set_suid(self.PGLOG['EUID'])
self.set_help_path(__file__)
self.PGLOG['LOGFILE'] = "rdakill.log" # set different log file
self.cmdlog("rdakill {}".format(' '.join(argv)))
self.PGLOG['LOGFILE'] = "gdexkill.log" # set different log file
self.cmdlog("gdexkill {}".format(' '.join(argv)))
for arg in argv:
ms = re.match(r'-([ahpPqsu])$', arg)
if ms:
Expand All @@ -79,7 +79,7 @@ def read_parameters(self):
optcnt += 1
else:
self.pglog(arg + ": pass in value without Option", self.LGEREX)
if not optcnt: self.show_usage("rdakill")
if not optcnt: self.show_usage("gdexkill")

# function to start actions
def start_actions(self):
Expand Down Expand Up @@ -257,10 +257,10 @@ def record_dscheck_interrupt(self, pid, host):

# main function to execute this script
def main():
"""Entry point: instantiate RdaKill, parse arguments, run, and exit."""
"""Entry point: instantiate GdexKill, parse arguments, run, and exit."""
from rda_python_setuid.setup_guide import show_setup_guide
object = RdaKill()
show_setup_guide(object, 'rda_python_miscs', ['rdacp', 'rdakill', 'rdamod'])
object = GdexKill()
show_setup_guide(object, 'rda_python_miscs', ['gdexcp', 'gdexkill', 'gdexmod'])
object.read_parameters()
object.start_actions()
object.pgexit(0)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
other filter criteria; or cancel one or more PBS batch jobs by job ID or by
job status. For PBS operations the -h option must specify the PBS node name.

Usage: rdakill [-h HostName] [-p ProcessID] [-P ParentProcessID] \
Usage: gdexkill [-h HostName] [-p ProcessID] [-P ParentProcessID] \
[-s BatchStatus] [-q BatchQueue] [-r] \
[-u ProcessOwner] [-a ApplicationName]

Expand Down Expand Up @@ -36,31 +36,31 @@
Without this option all users are included;

For local processes, all child processes are also terminated recursively.
Usage is displayed if rdakill is run without any options or arguments.
Usage is displayed if gdexkill is run without any options or arguments.

Examples:

1. Kill a local process by PID:

rdakill -p 13199
rdakill 13199
gdexkill -p 13199
gdexkill 13199

2. Kill a local process by PID and verify it matches a specific application:

rdakill -p 13199 -a dsrqst
gdexkill -p 13199 -a dsrqst

3. Kill all local processes (and their children) owned by a user:

rdakill -u zji -a dsrqst
gdexkill -u zji -a dsrqst

4. Kill a PBS batch job by job ID:

rdakill -h casper-pbs -p 334323
gdexkill -h casper-pbs -p 334323

5. Kill all running PBS batch jobs for a user in the default queue:

rdakill -h casper-pbs -s RUN -u zji
gdexkill -h casper-pbs -s RUN -u zji

6. Kill all pending PBS batch jobs in a specific queue:

rdakill -h casper-pbs -s PEND -q gdex
gdexkill -h casper-pbs -s PEND -q gdex
Loading
Loading