Sep 30, 2016

Redirected Recovery of IBM DB2 Recovery Expert for z/OS v3.2


In our ERP DB2 environment on z/OS, we perform frequent Data migrations. Most of the time it’ll be whole Database (of size ~3 TB) migrated from Prod to QA or DEV sub-systems. My DBA team was using a Vendor Tool (with REXX customizations) for the Data migrations. But, that was unable to handle the volume of the Data migrated every time and also had some Performance issues. So, we needed a better Tool for Data migrations.

IBM DB2 Recovery Expert for z/OS is in market since many years. It didn’t had the component that can be used for Data migrations from one DB2 sub-system to other. IBM released Redirected Recovery component in 2015, which can perform the Data migrations with better Performance.

We are one of the early adopters of the new component. And, I was assigned to test and implement the Tool in our DB2 environment. After 8 months of rigorous testing, I implemented Redirected Recovery successfully in our DB2 for z/OS environments.

and my colleague are going to present about our Redirected Recovery Implementation experiences in IBM WoW Conference. Our Presentation is scheduled on 27-Oct-16 from 1 – 1.45 PM. Below are our Session details.
Session ID:       DMT-1112
Session Name: Stories from the Front Line on Improving DB2 for z/OS Availability and Operations 







Sep 29, 2016

Caché Database Size Monitoring


We use a vendor supplied application called Artiva for maintaining Medical bills collection details, hosted on AIX servers. Artiva uses InterSystems Caché as the DBMS.

There is AIX File systems (on which Caché Databases are mounted) size monitoring alerts mechanism, but no Caché Databases size monitoring alerts mechanism in place when I started as DBA. InterSystems and the Artiva’s vendor didn’t provide any straight forward approach for Databases size monitoring.

All Caché Databases have the following Size details.
  1. Current size of the Database
  2. Expansion size                          – how much MB can the Database expand further
  3. Maximum size of the Database – with expansion, Database can grow till this size

I coded a Shell script that will do the following.
  1. Search for the Artiva-Caché Databases
  2. Calculate the Total Free Storage percentage, available for the Database to grow
  3. Send an email if any Database with Total Free Storage percentage is less than 10%
This Script is scheduled to be executed every 15 minutes, every day.

The Script is as follows.
 #!/bin/ksh

######################################################
DB_FREE_SPC=/cache/cachedba/scripts/db_free_spc
DB_FREE_SPC_MOD=/cache/cachedba/scripts/db_free_spc_mod
DB_FREE_REPORT=/cache/cachedba/scripts/db_free_report
######################################################

RC=0

#####Getting Database Size Details from Caché######
csession cache "ALL^%ZUTFREE" > $DB_FREE_SPC
#############################################

cat $DB_FREE_SPC | grep -i / | grep -v Auto     > $DB_FREE_SPC
cat $DB_FREE_SPC | sed -n 's/%//p' > $DB_FREE_SPC_MOD
echo "--------------------------------------------------------------------------------------------------------------" > $DB_FREE_REPORT
echo " DB_NAME \t\t  CUR_DB_SIZE \t\t FREE_SPC_PCT_IN_DB         MAX_DB_SIZE \t      TOT_FREE_SPC_PCT \t      FREE_SPC_IN_FS" >> $DB_FREE_REPORT
echo "--------------------------------------------------------------------------------------------------------------"  >> $DB_FREE_REPORT

cat $DB_FREE_SPC_MOD |
while read LINE
do
  DB_NM=$(echo $LINE | awk '{print $2}' )
  DB_CUR_SIZ=$(echo $LINE | awk '{print $5}')
  DB_FREE_PCT=$(echo $LINE | awk '{print $4}')
  DB_MAX_SIZ=$(echo $LINE | awk '{print $6}')
  FS_FREE_SIZ=$(echo $LINE | awk '{print $8}')
  DB_CUR_SIZ=$(echo ${DB_CUR_SIZ%?})
  DB_MAX_SIZ=$(echo ${DB_MAX_SIZ%?})
  ((DB_TOT_FREE_PCT = 100 - DB_CUR_SIZ / DB_MAX_SIZ *100))

 if (( $DB_TOT_FREE_PCT < 10 )) then
      (( RC += 1 ))
      echo " $DB_NM  \t\t\t $DB_CUR_SIZ   \t\t\t $DB_FREE_PCT  \t\t\t $DB_MAX_SIZ  \t\t\t $DB_TOT_FREE_PCT  \t\t\t $FS_FREE_SIZ"  >> $DB_FREE_REPORT
 fi
done

#####Email step#####
if (( $RC > 0 )) then
    mail -s "DATABASE SIZE ALERT ON $HOSTNAME"  <email ID> < $DB_FREE_REPORT
    exit $RC
  else exit $RC
fi
########################


Sep 24, 2016

InfoSphere Optim Data Growth for Archiving on z/OS


We have ~3 TB of operational data in ERP DB2 Prod environment. It’s complicated to apply Application upgrades with this volume of data.  
So, we decided to Archive and Delete the data from Prod Operational Database. We are going to use InfoSphere Optim Data Growth for Archiving on z/OS for Archive and Delete.

The Optim Archive works in the following sequence of steps.
1. Define the DASD needed to hold the Archive Datasets
2. Define Tables Relationships within the Optim Directory
    Note: Tables Relationships will hold the Tables to be archived that make a whole unit.
3. Define Access Definitions within the Optim Directory
    Note: Access Definitions will hold the criteria to be used for archiving the Tables
4. Define the Target for holding the Archive Data
5. Archive the DB2 Operational Data from the Tables into Archive Datasets
6. Verify the Archive Datasets
7. Transfer the Data from Archive Datasets to Target
8. Delete the archived data from DB2 Operational Database

Optim Archive provides multiple options for archiving the DB2 Operational data on z/OS. 
Below are the options provided based on the Target.

1. z/OS as the Target for the Archive Data    
 
  
2. Hadoop as the Target for the Archive Data     


3. Distributed Platform as the Target for the Archive Data


Note: 
ODM stands for Open Data Manager. Its collection of Drives needed for Reporting Tools to connect to Archive Datasets on z/OS. 
Its included in the Optim Archive for z/OS installation kit.



Sep 12, 2016

DB2 MSTR log processing


DB2 MSTR logs have all kinds of DB2 Error messages, which can provide more insight into the Application performance & activity.

We use Data-Sharing setup and have 13 MSTR logs in DEV/QA/Prod regions combined. There is no way of getting customized details from the MSTR logs, except for going through the complete logs.

I developed a new Process that’ll access the MSTR logs, parse the Data and provide the details needed for Application monitoring.
The Process includes 2 Steps.


        Step 1: This Step will be executed on Daily basis.

1.   Connect to SDSF
I coded a REXX routine that connects to SDSF and gets the DB2 MSTR logs.

ISFIN_PARA:                                                     
 "ALLOC F(ISFIN) UNIT(VIO) NEW REU",                           
  "CYLINDERS SPACE(10,10) RECFM(F,B) LRECL(80) DSORG(PS)"      
 "ALLOC FI(ISFOUT) NEW DELETE REU ",                           
  "CYLINDERS SPACE(10,10) LRECL(133) RECFM(F,B,A) DSORG(PS)"   
 "ALLOC F(OUTP) UNIT(VIO) NEW REU",                            
  "CYLINDERS SPACE(100,100) LRECL(133) RECFM(F,B,A) DSORG(PS)" 
 QUEUE "ST"                     /* OPEN SDSF STATUS QUEUE     */
 QUEUE "OWNER *"                /* OPEN SDSF STATUS QUEUE     */
 QUEUE "PRE "A""                /* OPEN SDSF STATUS QUEUE     */
 QUEUE "FIND "A""               /* LOCATE YOUR JOB            */
 QUEUE "++S"                    /* BROWSE JESYSMSG DATASET    */
 QUEUE "PRINT FILE OUTP"        /* PRINT JESYSMSG TO DATASET  */
 QUEUE "PRINT 1 999999"         /* PRINT ALL RECS OF JESYSMSG */
 QUEUE "PRINT CLOSE"            /* CLOSE PRINT FILE           */
 QUEUE "END"                    /* END SDSF SESSION           */
 QUEUE "EXIT"                   /* EXIT SDSF                  */
 "EXECIO" QUEUED()" DISKW ISFIN (FINIS"                        
RETURN;   
                                                      
2.  Next REXX routine I coded will parse the output captured by above REXX routine to find out details of below mentioned DSN messages.
DSNL030I
DSNI031I
DSNT375I
DSNT376I
DSNT501I
DSNB260I
DSNJ139I

3.    Finally, the details parsed by above REXX routine will be inserted into a DB2 Table by another REXX routine I coded.
I created Views for each DSN message. So, the corresponding Details will be inserted into the respective Views.




   Step 2: This Step will be executed on Weekly basis. 
                I setup a JCL that executes the below SQL query and emails the details to my Team.

          SELECT
              STC_REGION
             ,DSN_MSG
             ,COUNT(*) AS OCCURENCES
          FROM DBMSTRLG.MSTRLOG
          WHERE WEEK(ROW_ENTRY_TS) = WEEK(CURRENT TIMESTAMP) - 1
          GROUP BY
              STC_REGION
             ,DSN_MSG
          WITH UR;

This SQL query will gives the no.of occurrences of each DSN message during last week.
For more details about the DSN messages, we can query the base DB2 Table.