User information hosted in LDAP was unavailable from 4p to 9p PDT. Web hosting, email, and SSH/SFTP services were intermittently interrupted. Mail service was also taken down from 10p to 11p PDT for maintenance. We apologize for the inconvenience.
Although no user data was affected, some email messages sent to OCF accounts during the service interruption may have been rejected and bounced to the sender.
The login server (tsunami) was restarted at 10:14 PM for unanticipated maintenance. Zabbix, an uptime monitoring software, was not communicating with tsunami properly. The iptables firewall were flushed, causing network connectivity to be disrupted briefly. We apologize for the inconvenience.
There was intermittent down time between 11a-12:30p PDT due to software updates.
Secure shell login and access to the wiki were unavailable Monday morning and part of the afternoon. These services are hosted on virtual machines whose hypervisor (once again) had problems accessing its SCSI drives. The machine was reset, and is running normally again.
Additionally, the switch connecting the alternate login machine (pileup) to the OCF had been switched off. This has also been remedied.
As was the case last time, no user data is stored on these machines, so your data is not affected. We apologize for the inconvenience.
The gateway which our uplink is connected to was offline during the following periods this morning:
06:41a - 06:43a
06:50a - 06:56a
07:06a - 07:43a
The OCF was not externally accessible during this time. Since the gateway is operated by the campus IST department, we cannot verify the cause of the outage.
We suspect this was related to a planned power outage at the IST data center, although the campus network which we are connected to was not planned to be affected.
It was discovered that users could not send mail through webmail or the webserver for the past month.
If you sent an email from OCF webmail between March 25 and April 31, please confirm that it was actually received.
You will not have received emails from your website in the past 4 weeks.
These issues have now been fixed. We apologize for any inconvenience and for the delay in discovering them.
The mail server was taken down from 04:39a to 05:55a today for maintenance and cleanup. SSH and SFTP logins may have failed during that time. All services should now be operational.
The login server (tsunami aka ssh.OCF), webmail, and wiki are not accessible since 1:01pm today.
These services are provided by virtual machines. The physical hypervisor in which the virtual machines are run has issues accessing its data store, and we are investigating SCSI failure with the hard drives and controller.
No user data is stored on these machines, so your data is not affected. We apologize for the inconvenience.
If you need SFTP/SSH access, you may temporarily connect to one of the desktops, pileup.OCF.Berkeley.EDU, instead of connecting to ssh.OCF.Berkeley.EDU which is currently unavailable.
UPDATE 6:45p: The login server (SFTP/SSH) is back up.
UPDATE 6:52p: Webmail and wiki are back up.
The campus router which provides our Internet connection went down at 5:40am. All services are down.
UPDATE 5:45am: Router is online. Services should be up again.
As the previous post notes, our primary mail server (war.OCF aka mail.OCF) was down on Friday and parts of Thursday and Saturday.
At around 3:30a Thursday the server began to slow down on processing mail, and by 3:15p Friday the server had crashed. We could not turn the server back on, suspecting file corruption on the SPARC hypervisor the Solaris LDOM runs on.
We began rebuilding the mail server with Debian 6 on physical x86 hardware Friday night, and after 9am Saturday services were coming up. IMAP, POP, SMTP, and SMTPAUTH should all be up as of 12:30p Saturday.
We're using a new SSL certificate on the new mail server.
Since we're still running parts of mail on a mix of other machines running Solaris and Debian 5, this is a temporary solution. Maintenance work should be expected soon. We apologize for the inconvenience.
As always, you can email any suggestions or comments to staff@OCF.Berkeley.EDU.
UPDATE Sat 5:15p: Our incoming mail server (sandstorm.OCF) has a backlog of emails to process. There may be delays in receiving emails.
UPDATE Sat 11:45p: special local (slocal) mail delivery mail processing is no longer supported, please update your .forward files accordingly.
UPDATE Tues 11:55p: We've had difficulties with the incoming mail server, so mail may have been temporarily (or possibly in some rare cases, permanently rejected) when sent to an OCF address. We've replaced the Solaris machine with the newly-rebuilt mail server. We apologize for the inconvenience, and hope that there will be no bugs.