Skip to content

2012

  • 1 min read

Mail outage

Mail was unavailable between 9:30pm Monday and 7pm Tuesday as a result of electrical work for the UPS, which broke NFS, see post Moving to Hearst Gym. We apologize for the inconvenience.
  • 1 min read

Backup server compromised

We discovered that our backup server (pollution.ocf) had been recently misconfigured and was subsequently compromised. pollution maintains copies of account data (home directories, web directories, MySQL databases, and mail inboxes) and administrative credentials.

We immediately investigated, changed the affected credentials, and notified campus. Please assume that the attackers had access to a copy of your account data, and take appropriate measures, especially if you store sensitive or restricted data. Although we determined that account credentials (e.g., password hashes), which are not stored on pollution, were not compromised, feel free to reset your password as an additional security measure.

We sincerely apologize for this inconvenience. If you need any assistance, please let us know.
  • 1 min read

Mail is down

OCF email service abruptly stopped due to an issue in our authentication system. Please check back for updates.

Problem was resolved. (2:27 pm 8/4)
  • 1 min read

File server and user directory migration

We will be migrating the NFS disk array (file server) and LDAP server (user directory) to new hardware after 8pm tonight. This is necessary to maintain uptime and acceptable performance as much as possible during and after our move to Hearst Gym in August.

We will attempt to keep files read-only where possible so that services including web hosting will not be as severely affected.

We lack the extra hard drives on hand (we will be using the same hard drives in the new hardware, which means they must be backed up, formatted, and restored), so partial downtime of all OCF services will be unavoidable.

Edit 07/29 02:00am:  LDAP migration completed successfully without service interruption. NFS migration (during which file access will be read-only) postponed to later today.

Edit 07/29 06:00pm:  NFS migration started. Mail service is offline. Web hosting and SSH are read-only.

Edit 07/30 01:10am:  All services except mail restored.

Edit 07/30 01:25am:  All services restored. NFS disk array may require minor downtime in the near future. Good night.
  • 1 min read

Systemwide downtime

Unscheduled maintenance on virtual machine hypervisors made many OCF services unavailable between 3am and 6:30am. A somewhat-related issue affected web hosting yesterday.
  • 1 min read

Mail maintenance

Mail services (IMAP/POP/SMTP/webmail) are going down as we transition the mail server to a virtual machine. By doing so, we hope to keep mail available on a temporary server during our move out of Eshleman in August, since virtual machines (guests) can be migrated on-the-fly between physical hypervisors (hosts).

Update 02:04p: All services back up. Mail spool (inbox files) will be migrated later.
  • 1 min read

Scheduled webserver downtime again

We are (again) scheduling downtime on our primary server (hal) between 9:00pm and 10:30pm July 5st, to add two new processors. Notably, this will affect web hosting, MySQL, IRC, and wiki.
Update 9:15pm: Services going down.
Mid-downtime action shot!
(via Kenny Do)
Update 10:24pm: Services are back up.
  • 1 min read

Scheduled webserver downtime

We are scheduling downtime on our primary server (hal) to physically inspect the machine between 6:30pm and 7:30pm July 1st. Notably, this will affect web hosting, MySQL, IRC, and wiki. We will again schedule downtime in one or two weeks to add two additional processors to hal.

Update 6:47pm:  hal is being taken offline.
Update 7:25pm:  hal and all services except IRC are back.
Update 7:32pm:  All services operating normally.
  • 1 min read

Web server replaced

Web serving or: How We Learned to Stop Worrying and Love Debian

The OCF web server, which hosts http://www.ocf.berkeley.edu and hundreds of virtual hosts for group accounts, was migrated over to our newest (and fastest) server running Debian GNU/Linux on June 9.

The Good

Websites, especially with dynamic content, should be loaded (much) faster. Features might be added later down the road.

The Bad

This is possibly our most visible non-lab-related (intentional) change in the past few years, and when that happens, stuff tends to break. We made the change over summer to allow us the most time to fix things with minimal disruption, and before we are moved out of the basement of Eshleman Hall (sometime during or after August).

The Ugly

It might take us a while to fix things. Contact us if something doesn't work that worked before.

Known unresolved issues

Troubleshooting

  • Are your CGI scripts returning an "Internal Server Error"? Some users have hard coded the path to old Perl or Python interpreters. We recommend changing the first line of these scripts to "#!/usr/bin/env perl" or "#!/usr/bin/env python", respectively.
  • Does your CGI script or binary depend on a particular architecture? If so, it may need to be compiled again for amd64 (Debian squeeze).