Monday, August 31, 2009

Take Me to the Pilot

The FamilySearch Digital Pipeline One regular reader of my column writes,

Dear Ancestry Insider,

   I'm a regular reader of your column.

   A question that has been in my mind for a long time is - when will we see digitized images of the LDS [microfilm] records? I don't mean the excellent pilot site with its almost 250 sources. I mean the unindexed stuff that we now have to order and go to an LDS [family history] center to see. I know they're supposedly in the process of being digitized and that new records from the field are supposed to be in the digital format.

   Is there any time frame for when these records will start coming online? I assume they'll do them bit-by-bit, not wait the 10 years or so until they're all done.

   I'd love to start seeing these. Readability is better on a computer, and I'm not tied to the miniscule hours offered by the library (usually in the AM and I'm a late night person.)

   I rarely see updates on this and no one who seems to cover the LDS [microfilm] records talks about this. One of the very few areas in which the LDS does not do a good job is in letting us know when records are filmed (I was told "keep checking online. That's great for Tiny Town,, but doesn't work for large cities, too many entries to be able to tell what is new) and/or how the digital program is going.

  Thanks for reading this.


Dear Bonz,

It goes without saying that New FamilySearch (NFS) is job one for FamilySearch. Fortunately, gathering and protecting copies of the world’s records has gone on unabated even while spare resources have been applied to the NFS rollout.

During this time, more has been going on behind the “excellent pilot site” than you have realized. Behind this unassuming web face are hundreds of FamilySearch personnel—not to mention indexers—going about the mammoth task of building and operating a high-capacity digital pipeline—a digital record factory, really—that can provide the very services you’re wondering about, and more:

  • Publish un-indexed, scanned images from FamilySearch’s microfilm collection, the “LDS records” that you mention.
  • Publish indexes with links to images on partner websites. This is part of the Records Access Program, which can greatly accelerate the publication of data once the FamilySearch digital pipeline is ready to operate at full capacity.
  • Quickly publish un-indexed images from the digital cameras that are replacing the microfilm cameras in the field.
  • Publish legacy vital record indexes previously published on CDs or on the legacy FamilySearch web site. Indexes were created using UDE, the predecessor to FamilySearch Indexing. Sometimes the legacy images used with UDE are also published.
  • Publish legacy vital record indexes never before published. Indexes were created using UDE.
  • Same, but with the legacy images used with UDE.
  • Publish legacy extracted records previously published as part of the IGI, sometimes with previously unpublished, legacy images.

and of course the scenario one usually thinks about,

  • Publish scanned images with indexes from FamilySearch Indexing.
FamilySearch uses Swivel Chair manual processes during pipeline construction
FamilySearch uses Swivel Chair manual
processes to bridge pipeline gaps
during pipeline construction.
Image Credit: chair clipart

Much is being accomplished behind the scenes in what could be called a Pipeline Pilot. Collections representing each of the scenarios above are being pushed through to Record Search, enabling developers and operators to learn by doing manually, to build replacement parts, and then to operate this marvel of engineering. The FamilySearch Operations team is fabulous, manually bridging gaps in the pilot pipeline. (See “Swivel Chairs” in this article.)

I believe the following are examples of the scenarios from above:

FamilySearch is still not informing anyone when new microfilm is added to their collection (if that is still happening). I was working on a project to do that prior to my job change last year, but haven’t pursued it since then.

Fortunately, they are very good at issuing press releases for new digital collections. I rarely pass these press releases on to you, however, so you’ll need to subscribe to another newsletter to see them. Watch for my recommendation in upcoming days.


-- The Ancestry Insider


  1. It seems like a simple thing to add a statement (and link) to the Family History Library catalog indicating that images (and indexes if applicable)of the data are available. What am I missing? -kk

  2. Great insight. I am a regular reader of your blog. Thanks for the great "inside" info.

  3. Insider,

    I am going to repost the following question and comments from an earlier post. I said:

    "I have a question that I don't think has been addressed, although perhaps I missed it here or in other posts. This is about microfilm ending its lifecycle. While I can see the reasons for that, and indeed twice this month two state archives have told me that they no longer offer duplication services, what is the plan for replacing microfilm and on what time scale?

    The way I understand FS indexing which I participate in, the indexes will always be free but the images generally hosted by a commercial fee-based company, which is fine (to me at least because I don't expect the LDS church to be an image farm for genealogy past its other enterprises). So when will I not be able to order the court minutes or deed books on microfilm for Any County, AnyState? And what will be the alternative? Will Ancestry, Footnote et al. really be up to hosting such county level records? Or will the FHL image those microfilms and distribute same on DVD or peer-to-peer download at a FHC?

    Also at the end of your post in the part on collaboration, you list 1999 as the year when film goes to public libraries. Does this mean that film can be ordered via interlibrary loan from the FHL to any public library, or only those public libraries that incorporate a FHC somehow? I specifically asked one of those state archives which does lend to public libraries in other states, whether that included FHCs. I was told that the FHL does not loan to them and they don't loan to the FHL. If you want the name of the specific library I will email it to you privately."

    You have answered that partly in the response to Bonz. But I am pretty sure that Bonz in asking about unidexed digital images from microfilm, primarily meant meat and potatoes county level microfilm, as opposed to state level census and vital statistics.

    So the question is where in the pipeline is that (county level records), especially given that microfilm is ending its lifecycle? If I recall correctly, the FHL can scan a microfilm roll in 2 minutes by imaging as one large image. But then the question is who is going to further process and host such images?

    My understanding is also that no new Family History Centers are being built in new LDS churches. So are they too ending their lifecycle in anticipation of a day when everything currently on micro at the FHL is available online?



  4. "But I am pretty sure that Bonz in asking about unidexed digital images from microfilm, primarily meant meat and potatoes county level microfilm, as opposed to state level census and vital statistics."

    Partly. I wanted to know both - when newly digitized in the field as well as stuff in the vaults on microfilm will start showing up. I am definitely waiting for the microfilm stuff to arrive online and while an index is great, if that slows the process, I'd gladly do without the indexing.

    Mostly what I see now is census stuff and indexed records that were sponsored by some society or group.

    What I got from the informative article today is that a) it's coming and that a bit of it is here now and b) there are other priorities competing for attention. What I don't know and would like to know is when will the day come when I'll start seeing more than a small sampling of records.

    For intense research (like Hungarian parish records for example,) it would be so much better to see them digitized on my larger screen.

    They're coming, I know, but when? I'm just so eager to see this happen. It would be awesome to do this from home. No stopping because the library is closing mid-afternoon, just when I'm ready to really research. And I'll be able to do stuff late at night when my work is done and I can have a nice block of uninterrupted time. Not many libraries are open at midnight. ;)

  5. Dear kk,

    It is my understanding that Cataloging is holding off wholesale links to Record Search because it is still in pilot phase and links are expected to change.

    Another small nuance is that digital collections sometimes encompass multiple FHLC listings. I've been told that the 1900 census is an example. Before publication on Record Search, the catalog contained 50 state-level entries, but no national-level catalog entry. Rumor has it that one has since been added and has a link to the pilot site.

    -- The Insider

  6. Dear MikeF,

    Patience, dear friend. I wrote up answers to your questions days ago. The article is currently scheduled for publication on Wednesday.

    -- The Insider

  7. Dear Bonnie,

    I happened to see Hungarian records discussed in a pipeline meeting at 1:00pm today. I can't remember if they were parish records. I don't remember if they were designated "ready-for-publication." And I can't remember if there is space allocated to publish them. It's just as well. If I remembered any of those things, I couldn't tell you. Stay tuned!!!

    -- The Insider

  8. *grins*

    I most certainly will stay tuned. It's a real pity about that memory. ;)


Note: Only a member of this blog may post a comment.