The gateway which our uplink is connected to was offline during the following periods this morning:
06:41a - 06:43a
06:50a - 06:56a
07:06a - 07:43a
The OCF was not externally accessible during this time. Since the gateway is operated by the campus IST department, we cannot verify the cause of the outage.
We suspect this was related to a planned power outage at the IST data center, although the campus network which we are connected to was not planned to be affected.
It was discovered that users could not send mail through webmail or the webserver for the past month.
If you sent an email from OCF webmail between March 25 and April 31, please confirm that it was actually received.
You will not have received emails from your website in the past 4 weeks.
These issues have now been fixed. We apologize for any inconvenience and for the delay in discovering them.
The mail server was taken down from 04:39a to 05:55a today for maintenance and cleanup. SSH and SFTP logins may have failed during that time. All services should now be operational.
The login server (tsunami aka ssh.OCF), webmail, and wiki are not accessible since 1:01pm today.
These services are provided by virtual machines. The physical hypervisor in which the virtual machines are run has issues accessing its data store, and we are investigating SCSI failure with the hard drives and controller.
No user data is stored on these machines, so your data is not affected. We apologize for the inconvenience.
If you need SFTP/SSH access, you may temporarily connect to one of the desktops, pileup.OCF.Berkeley.EDU, instead of connecting to ssh.OCF.Berkeley.EDU which is currently unavailable.
UPDATE 6:45p: The login server (SFTP/SSH) is back up.
UPDATE 6:52p: Webmail and wiki are back up.
The campus router which provides our Internet connection went down at 5:40am. All services are down.
UPDATE 5:45am: Router is online. Services should be up again.
As the previous post notes, our primary mail server (war.OCF aka mail.OCF) was down on Friday and parts of Thursday and Saturday.
At around 3:30a Thursday the server began to slow down on processing mail, and by 3:15p Friday the server had crashed. We could not turn the server back on, suspecting file corruption on the SPARC hypervisor the Solaris LDOM runs on.
We began rebuilding the mail server with Debian 6 on physical x86 hardware Friday night, and after 9am Saturday services were coming up. IMAP, POP, SMTP, and SMTPAUTH should all be up as of 12:30p Saturday.
We're using a new SSL certificate on the new mail server.
Since we're still running parts of mail on a mix of other machines running Solaris and Debian 5, this is a temporary solution. Maintenance work should be expected soon. We apologize for the inconvenience.
As always, you can email any suggestions or comments to staff@OCF.Berkeley.EDU.
UPDATE Sat 5:15p: Our incoming mail server (sandstorm.OCF) has a backlog of emails to process. There may be delays in receiving emails.
UPDATE Sat 11:45p: special local (slocal) mail delivery mail processing is no longer supported, please update your .forward files accordingly.
UPDATE Tues 11:55p: We've had difficulties with the incoming mail server, so mail may have been temporarily (or possibly in some rare cases, permanently rejected) when sent to an OCF address. We've replaced the Solaris machine with the newly-rebuilt mail server. We apologize for the inconvenience, and hope that there will be no bugs.
The mail server is experiencing some issues and is currently down. We apologize for any inconvenience this may have caused.
The web server (death/www.OCF) had crashed again. We apologize for any inconvenience this may have caused.
UPDATE 12:03am: Fixed, web server is functioning again.
Our webserver ran into troubles with virtualization and crashed at 10:15pm last night. We are working to bring it back up and apologize for the inconvenience. User data should not been affected and is still accessible over SSH.
Update 11:07am: Should be back up.
flood.OCF has been taken offline as the machine is being rebuilt. Users should see no effect as all services running on flood have been transitioned to another server.