Skip to content

News from the staff team

  • 1 min read

Expected downtime

Hi all,
Yesterday's reboot left behind some nasty problems in our mail volume, we are going to take some downtime to fix it. 7pm-11pm, I expect system-wide downtime, I apologize for whatever inconvenience this has caused. In hindsight we should have handled this better, but we learn from our mistakes.

Sorry for the trouble.

Update 4:56pm
We may start working a bit early at around 5:30-6:00 so don't be surprised if /var/mail disappears during that time.

Update 8:40pm
Stuff should be working again for the most part, thanks for your patience

  • 1 min read

Disk array reboot

Hi all,
We rebooted our disk array, resulting about 30 mins of downtime system wide. Just notifying all in case there are any stray stale nfs handles hanging around, or if any scripts broke. Don't panic if you couldn't log into our machines during this time, it was nothing serious.

  • 1 min read

Webmail Restored

We've restored webmail functionality. Please let us know if there are any problems or hiccups.

  • 1 min read

Update

Hi all,
Some of you may have noticed your cgi scripts have broken, this is probably due to the fact that our new webserver has a different directory structure, and your scripts rely on a hard coded directory path. Don't worry this is only temporary. I used different nfs mount points to make mounting user data easier while we were swapping ip's around. It should be resolved when the server is on its proper hardware.

  • 1 min read

Anticipate downtime this weekend

Hi all,
We will try to rebuild one of our main servers this weekend, anticipate some downtime and/or loss of functionality.

  • 1 min read

Servicing our disk array

The OCF will likely be down the entire weekend as we update, error check, and test our disk array. Please check back later for more updates

  • 1 min read

Mysql Upgrade

Hi all,
We just upgraded our mysql server, let me know if anything doesn't work. Some of our client programs may be old, so if it says stuff like "Authentication protocol not supported" don't worry, its just that our clients are using old programs.

  • 1 min read

Further Updates

Thank you for all your patience, its really appreciated. Our new disk array should be back up, and we will slowly restore services after the July 4th break. I don't want to rush anything since its just going to make it harder to fix if something goes wrong.

I want to make sure everyone knows we have multiple goals when addressing the current problems. While we try to maximize our uptime, we are also concurrently rebuilding our current system. At the beginning of summer, we did send a downtime announcement that spanned the whole summer. At some point we may decide the current system is not worth salvaging, and focus our efforts on the rebuild.

Furthermore, we will shortly attempt to rebuild our core servers, which will take down our entire system. We will try our best to build ad hoc login servers, but anticipate more downtime in July.