Veritas Netback - Drives come up in AVR rather than ACS.

Veritas products have always been confusing. I'm writing here to archive an experience that we had with netbackup and ACSLS (Automated Cartridge System Library Software) in our data center. Essentially, all of the drives in our media library were up as AVR (Automatic Volume Recognition) as opposed to ACS (Automated Cartridge System). This means that the netbackup server was not communicating correctly with the ACSLS server, and unable to control the tape library robot.

(These examples have been edited.)


A drive that is in Automatic Volume Recognition mode (AVR) is not controlled by a robot. This means that once the tape is in the drive and written to.. It just stays there until someone takes it out. Obviously this is not optimal for a backup infrastructure that does thousands of backups. Especially when there just happens to be a giant robotic arm sitting there doing nothing.
 
The issue was first noticed by our backup team who use the NetBackup GUI to schedule and monitor backups. I was able to see the issue on the UNIX side by typing:


#vmoprcmd
Drv Type Control User Label RecMID ExtMID Ready Wr.Enbl. ReqId
0   hcart3  AVR     root Yes      -          -         No        No       2


Generally, you would note that Drive 0 would be under the control of ACS rather than AVR. We were able to note the following errors in /usr/openv/volmgr/debug/acsssi/event.log


07-07-07 14:07:50 SSI[0]: ONC RPC: csi_freeqmem(): status:STATUS_QUEUE_FAILURE; Dropping from Queue: Remote Internet address: 0.0.0.0,Port: 0 , ssi_identifier: 1304, Protocol: 822083584, Connect type: 0;

07-07-07 14:13:10 SSI[0]: ONC RPC: csi_rpccall(): status:STATUS_NI_FAILURE; failed: clntudp_create() RPC UDP client connection failed, RPC: Rpcbind failure Remote Internet address:x.x.x.x, Port: 0;

So we can see here that there's an issue with the UDP connection. We also noticed entries like this in /var/adm/messages

Nov 15 11:28:06 hoehpt07 acsd[8807]: ACS(0) Response has not been returned by Mount command sequence 4434, ACS status = 72, STATUS_PENDING


So this shows us that a response has been sent, but never returned. All of our querys on the ACSLS server returned sane. Veritas support agreed. The issue actually turned out to be on the ACSLS server.

A few days prior, a change had been made enabling a firewall on the ACSLS server. Reversing the change resolved the issue. I don't remember exactly how the ACSLS menu was laid out, but this is how we got there:


#ssh acslsserver
#su - acsss
#acsss_config


In that menu there was a secure firewall setting. Disabling it, restarting ACSLS, and rebooting the media server resolved the issue.

 

Add comment

Security code
Refresh