May/June 2021 storage issues resolved; /scratch and Slurm are back to normal use
Jun. 1, 2021—We have verified that /scratch is performing at the same level it was prior to the down time. You may resume using it as you normally would as well as submit jobs to Slurm without using the special AtRiskUnstableEnvironment reservation. Prior to the downtime /scratch was operating with only 1 of the 2 controllers in...
Some programs on “maxwell” and “pascal” GPU nodes may need to be upgraded following May downtime
May. 4, 2021—We are going to upgrade the underlying operating system on the GPU nodes in the next downtime two weeks from now. As we tested we found that the OpenMPI associated with GCC 5.4 (OpenMPI/1.10.3) does not work with the newly installed driver, so if the system gets upgraded then the 1.10.3 version of the MPI...
Cluster access restored following downtime on May 18-20
May. 3, 2021—The May 2021 scheduled downtime has concluded. Updates on the subsequent storage issues have been moved to a separate post. Update, 5/21/2021 5pm: While we wait for the results of the vendor’s analysis of the system events, we are restoring job submissions to the cluster so they can run over the weekend. Jobs will be...
Info on /scratch outages from 4/14 and 4/23
Apr. 14, 2021—Update, 4/23 10pm: The /scratch storage subsystem is recovered. You may resume any affected jobs and report any issues via our helpdesk. Update, 4/23 7pm: The /scratch storage subsystem was unable to gracefully handle the controller failure and the remaining controller began losing connections to some of the LUNs. We are doing a clean shutdown...
Ask ACCRE: How do I map my home directory on ACCRE as a network drive on my computer?
Apr. 12, 2021—By mapping ACCRE as a networked drive, we can move files from our computer to ACCRE, and from ACCRE to our computer, as if it was a drive on our computer. Here are instructions for mapping ACCRE as a networked drive on Windows and macOS. Mapping ACCRE to a computer running Windows We will be...
Improvements to the ACCRE onboarding process, including training classes offered on demand
Mar. 18, 2021—Starting today, our training classes for new users for ACCRE will be offered in a new online, on-demand format hosted on RedCap. This applies to all three classes – Intro to Unix, Intro to the Cluster and Intro to Slurm – and will replace the three day series of classes we have held in the...
/data and /home interruption, MATLAB jobs interrupted
Feb. 23, 2021—Update, 2/24: In a separate issue we have updated the license file for MATLAB and a couple of jobs have probably died in the process. Please check the output of your MATLAB jobs and try again if this was the case. The cluster experienced a momentarily loss in connectivity to /home and /data at approximately 4:07pm...
Networked storage refusing to connect
Feb. 8, 2021—Further updates will be added to the downtime announcement. Update, 2/9 1pm: In order to avoid having two general downtimes in quick succession, ACCRE will begin both the GPFS fix as well as the planned maintenance items at noon tomorrow, Feb 10th. Please plan accordingly and open a helpdesk ticket if you need any assistance...
ACCRE networked storage connectivity
Dec. 11, 2020—Update, 12/12: We were able to identify the source of the problem that was preventing the export services from operating normally. An update was applied just afternoon today and that has cleared the errors that were causing the NFS service to crash. The workstations of one of the groups that use the service got into...
February 2021 maintenance: All ACCRE systems are back online
Dec. 11, 2020—Update, 2/12: We have finished all the work in scope for the planned 3-day downtime and restored the network storage service. All our production and ACCREx systems are back online. New hardware for /scratch has been installed and full network redundancy between the rooms in the data center has been reestablished. We did not shutdown...