[Resolved] /scratch and /data partially offline
Oct. 11, 2018—Update, 10/16/2018: /scratch and /data are now 100% available again and we have successfully upgraded the cache in all of our GPFS storage appliances. Please open a helpdesk ticket with us if you have any questions or notice anything unusual. Update, 10/15/2018, 5pm: We are still working to get parts of /scratch and /data back...
[Resolved] Cluster inaccessible
Oct. 1, 2018—Update, 2pm: The cluster is accessible again. Thank you for bearing with us and we sorted through these issues. We were able to successfully upgrade the cache on three of our six GPFS storage appliances. The third one is where we encountered issues. A hardware problem prevented us from booting the device cleanly until an...
[Resolved] Network maintenance ongoing; some login issues and/or stale file handles possible
Sep. 27, 2018—Update, 10/1/2018: This has been resolved. We have been experiencing some network issues for the last few days that have impacted logins and caused occasional stale file handle errors. Later today, we will be performing some maintenance that we hope will resolve this issue or at least further isolate the root cause. This work (which...
[Resolved] Sporadic stale file handles & potential error messages when logging in to CentOS 7 environment
Sep. 18, 2018—Update 9/21/2018: It appears this issue has been resolved. Please open a Helpdesk ticket if you experience any problems. This morning we began sporadically seeing stale file handle messages in our CentOS 7 environment, which may have also impacted logins to login7.accre.vanderbilt.edu. We initially thought this due to a bad hard drive on one of...
CentOS 7 Upgrade: login.accre.vanderbilt.edu now points to CentOS 7, login6.accre.vanderbilt.edu available temporarily if needed
Aug. 13, 2018—Update, 12/20/2018: GPUs are now available for CentOS 7. See: GPU Nodes Available for Testing in CentOS 7 Environment [Resolved] Full cluster downtime on Wednesday, Dec 19 starting at 6am; make sure to log out and halt any running processes before downtime starts Update, 9/28/2018: We have now pointed login.accre.vanderbilt.edu to the CentOS 7 environment....
[Resolved] CentOS 7 login not working
Aug. 8, 2018—Update, 8/9/18: This appears to be resolved. We are looking into issues with logging in to the ACCRE CentOS 7 gateway this afternoon and we will update this when we have more information. In the meantime you may continue to use login.accre.vanderbilt.edu to access the cluster. Thanks! Updated 4:52pm to clarify that it is CentOS 7...
[Resolved] Tape recovery on files impacted by May 29 disk failure is now complete
Jul. 23, 2018—Update, 7/23/2018: After nearly two months, the tape recovery on /data and /scratch is virtually complete and we will go ahead and mark this as resolved. This has to do with the logical disk failure on May 29 that caused 5% of files on /data and /scratch to become unavailable. It is unrelated to the...
[Resolved] Brief cluster maintenance for Monday, July 23rd has been cancelled
Jul. 19, 2018—Update, 7/22/2018: We are canceling the maintenance described below. Since this email was sent, we learned that a drive in one of our storage appliances was in a bad state (not bad enough to trigger our monitoring but bad enough to impact performance). On Saturday morning we proactively failed over to a spare drive and...
[Resolved] Some compute nodes offline for rebooting; /data and /scratch maintenance complete
Jul. 9, 2018—Update, 7/15/2018: The reboot is nearly complete; we will go ahead and mark this as resolved. Update, 7/15/2018: The maintenance is complete and /scratch and /data are available on all gateways and most of the rest of the cluster. However, there are a moderate number of compute nodes that will need to be rebooted to get the...
[Resolved] /data and /scratch sluggishness
Jul. 2, 2018—Update, 10/1/2018 2pm: The cluster is back up again. Update, 10/1/2018: The cluster is down due to today’s maintenance. See: Cluster Inaccessible Update, 9/27/2018: We will be performing cache upgrades of our GPFS storage appliances on Monday, Oct 1 beginning at 4:30AM. We expect the maintenance to last between 3 – 4.5 hours. During this...