Monday, May 23, 2011

FamilySearch Indexing Outage Continues

Some time around noon on Sunday, FamilySearch Indexing stopped working. As of 1:00 pm Mountain Time, the outage has not been fixed.

A message at https://indexing.familysearch.org states:

Indexing is down for maintenance

I contacted FamilySearch for comment and was told that a public statement will be made shortly. They do not yet know how long it will take to fix the issue. They are characterizing this as a “major outage.”

While the website states that Indexing is down for maintenance, it is unlikely that FamilySearch intentionally brought the site down on a Sunday. Judging from the number of single day indexing records, Sunday is their biggest day.

Indexers receive much less helpful notices. One common notice is this one, which you see when you try to run the Indexing program:

Launcer failed notice from FamilySearch Indexing program.

If you got it running, there was no notice in the message area of the Indexing program:

image

The only notice many indexers will see is a lack of projects in the batch download window.

With FamilySearch Indexing down, no projects show up in the batch download window

I received a notice that was unhelpful. The notice said I must update my Indexing program before I could download a batch. I dutifully uninstalled FamilySearch Indexing. When I tried to re-install, I found the web notice that Indexing was down for maintenance. Growl. Now I’ll have to needlessly reinstall when Indexing comes back up.

FamilySearch advises users not to try to re-install. They admit is won’t fix anything and can slow the system down.

I wonder how many older indexers FamilySearch loses because of its failure to provide better messaging.

Facebook

One place to check for messages is FamilySearch Indexing on Facebook. Go to www.facebook.com/familysearchindexing and click on Wall. (Wall is on the left side of the window, underneath the image.) A little after 5 yesterday, FamilySearch posted this message:

Most of you have already noticed, the indexing system has been struggling today. The engineers just shut everything down and will bring it back up shortly. If you receive an error message or a notice to update your program, please ignore it, wait a few minutes, and try again. Thank you for your patience!

At 6:06 pm:

The system is still down. Latest word from the engineers: They have many individuals involved, trying to get the system running again.

We hope they can get it running soon.

At 9:14 am, message from Brandon Bauman:

No ETA yet. We're doing our best. Thanks for your continued patience.

Forums

It’s nice that Facebook users were informed, but I wonder how many of FamilySearch’s indexers are on Facebook. You would think that FamilySearch’s own support forum would be the place to go for messages about the outage. Apparently not.

Go to forums.familysearch.org and click on FamilySearch Indexing. I didn’t find any messages from FamilySearch about the outage, but there were messages from volunteers, commiserating or dumping.

User “oxxguy” wrote,

Wouldn't it be nice if there were an indexing system health page maintained on a separate server (on facebook, say, or on an LDS server) so that we could find out if the system is down instead of just guessing "is it me?"

Unregistered Guest wrote,

It seems to me that the IT department doesn't have much concern for volunteers. Maybe they're too busy to even notice that they are putting their users out. I know IT departments tend to give notice. I don't see anything so far about advanced notice about may 22, 2011 being a maintenance day.

Maybe the server just crapped out and they have to replace something and they don't have failover, which is quite annoying.…

I wish that they'd put some technologically savvy people on the phone support because, I'm a techie and I want to know exactly what's going on.

User “indaloman” wrote,

I am sick of everytime they update their software, I cannot get into the program. They could not care less about volunteers, if something goes wrong and we lose hours of work, well thats just tough!

I think I have had enough, and I will go back to FreeGEN, FreeCEN etc, at least they give their volunteers support.

Private message to FamilySearch: Please re-read my article on the Cluetrain Manifesto.

Bottom line: Talk openly to your volunteers. Over-communicate. Or watch the train leave the station without you.

1 comment:

  1. In the Forums is a statement from one person who said a tech said in a ~private skype chat~ that the IT folks wanted indexers not to try to use the system at all while it was being worked on.

    There was no communication about this at all from IT in the Forums. Evidently no emails went out to indexers either.

    I used to work for an outfit that had big problems with the "3-Ps": Policy, Planning, Procedure. If you don't communicate policy, you don't have one. If you can't plan how to implement it, you don't have one. If you don't have or don't communicate a procedure for how to operate within a Policy, you don't have one.

    One of this outfit's major products was a massive procedural document written under a required US Government mandate. It was all in the passive voice as if forms and reports wrote themselves, and no individual or title or role-definition was specified that would receive the described contingency reports or take responsibility for any of it. It was issued to managerial staff as an unsigned, undated document: the outfit director did not want to take responsibility for it either.

    I think that taking responsibility is the key issue here, as elsewhere (such as in the research Wiki superstructure).

    ReplyDelete