Dependency check when scripts are triggered in background bash

I have a below code block in bash script. In this block of code I am triggering scripts in parallel and storing the process id of that scripts.

The script names are in a file called session_details.txt file.

Contents of the file are below

s01_test_abc
s02_run_cde
s02_skip_xyz
s03_failed_123
s03_success_999
s04_done_111

Bash script code

#!/bin/bash

wf_name=$1

# Create logs directory if not exists for that workflow
mkdir -p logs/${wf_name}

# Create run_share directory if not exists for that workflow
mkdir -p run_share/${wf_name}

# Date on which the workflow is running
run_date=$(date '+%Y-%m-%d')
run_time=$(date '+%Y-%m-%d-%H-%M-%S')

# directory where current run status files are stored
share_dir=run_share/${wf_name}

# directory where current run logs are stored
logs_dir=logs/${wf_name}/${run_date}
number_of_jobs=0

PID_FILE=${logs_dir}/session_pid.txt
if [ -f ${PID_FILE} ]; then
  rm ${PID_FILE}
fi

## parallel call #####

while read session; do
   echo "processing started for ${session}"
   session_name=$(echo ${session%% })

    start_status_file=${share_dir}/${session_name}_start_${run_date}.txt
    end_status_file=${share_dir}/${session_name}_end_${run_date}.txt


    if [ -f ${start_status_file} ] & [ -f ${end_status_file} ] ; then
        echo "session --> ${session_name} has completed run for ${run_date}"
    elif [ -f ${start_status_file} ] ; then
        echo "Session --> ${session_name} has failed run for ${run_date} so restarting the failed job"
        sh ${wf_name}/${session_name}.sh ${wf_name} ${session_name} ${share_dir} ${run_date} ${logs_dir} > ${logs_dir}/${session_name}.log 2>&1 &
    else
        echo "Session --> ${session_name} has not run for ${run_date} so triggering new run"
        sh ${wf_name}/${session_name}.sh ${wf_name} ${session_name} ${share_dir} ${run_date} ${logs_dir} > ${logs_dir}/${session_name}.log 2>&1 &
    fi

   #sh ${wf_name}/${session_name}.sh ${wf_name} > ${logs_dir}/$session_name.log 2>&1 &
    pid=$!
    echo "${pid}-${session_name}" >> ${PID_FILE}
    echo "----------------"
    if [ ${number_of_jobs} -eq 0 ] ; then
        pid_list=${pid}
    else
        pid_list="${pid_list},${pid}"
    fi
    echo "PID list - ${pid_list}"
    number_of_jobs=$((number_of_jobs + 1))
    echo "${number_of_jobs} job(s) submitted so far"
   echo "done"
done < ${wf_name}/session_details.txt


## wait for parallel jobs completion

PID_OUT_FILE=${logs_dir}/session_pid_out.txt
if [ -f ${PID_OUT_FILE} ]; then
  rm ${PID_OUT_FILE}
fi

while read p; do
   pid=$(echo ${p} | cut -d'-' -f1)
   session_id=$(echo ${p} | cut -d'-' -f2)
   wait ${pid}
   echo $?"-"${p} >> ${PID_OUT_FILE}
done < ${PID_FILE}

## verify the pids completion status
while read res; do
   pidres=$(echo ${res} | cut -d'-' -f1)
   sess_id=$(echo ${res} | cut -d'-' -f3)
   if [ ${pidres} -ne 0 ]; then
      status_msg="FAILED"
      echo "job failed for session -${sess_id} failed"
   else
      echo "job for ${sess_id} Completed"
      status_msg="SUCCESS"
   fi
done < ${PID_OUT_FILE}

## fail the script if any parallel run job failed.
while read res; do
   pidres=$(echo ${res} | cut -d'-' -f1)
   sess_id=$(echo ${res} | cut -d'-' -f3)
   if [ ${pidres} -ne 0 ]; then
      exit -1
   fi
done < ${PID_OUT_FILE}

The code is working fine. I am achieving my result.

Now what I want to do is I want to trigger see how can I replicate this code when there is script level dependency.

For example.

Run the below script first 
s01_test_abc 

Then run below two in parallel after s01_test_abc is success if failed then exit the whole script
s02_run_cde
s02_skip_xyz

Then run below two scripts after above two parallel scripts are success if any one failed then exit the whole script
s03_failed_123
s03_success_999

Then run below script after above two parallel scripts are success if any one failed then exit the whole script
s04_done_111

How can I achieve that



Read more here: https://stackoverflow.com/questions/66269033/dependency-check-when-scripts-are-triggered-in-background-bash

Content Attribution

This content was originally published by nmr at Recent Questions - Stack Overflow, and is syndicated here via their RSS feed. You can read the original post over there.

%d bloggers like this: