Sunday, November 1, 2009

Genealogical Maturity Model Proposal

Genealogical Maturity Levels, layered in chocolate
The
Genealogical Maturity Model
is made of levels
like a chocolate cake.
© 2008, Jimmy McDonald.
Used with permission.

This is the first article in a a series discussing my proposed Genealogical Maturity Model (GMM).

Cross-pollination from one field of knowledge to another is a common source of innovation. Today, I’d like to introduce a concept to the genealogy world that I learned from the software world: maturity models. Carnegie Mellon University studied software companies’ ability to successfully complete projects. Experts noticed that companies typically matured their software development capabilities in a specific order. They constructed a five level framework to model these improvements:1

1. Initial (chaotic, ad hoc) The starting point
2. Repeatable The process is used repeatedly, with roughly repeatable outcomes
3. Defined The process is defined
4. Managed Process metrics are managed
5. Optimized Management includes process optimization and improvement

 

I’ve applied the concept of maturity models to genealogists to describe the growing maturity, capabilities, and knowledge through which I’ve observed genealogists progress. The Genealogical Maturity Model is a work-in-progress. I invite you to leave comments with your feedback, corrections, and additions.

The Genealogical Maturity Model

The Genealogy Maturity Model has five levels, GML 1 through GML 5.

1. Entry Newly interested in genealogy
2. Emerging Emerging knowledge of how to do genealogy
3. Practicing Usually produces verifiable genealogical conclusions
4. Proficient Produces verifiably correct genealogies
5. Stellar Produces well-regarded genealogies

 

Sources and Citations

One can describe the genealogical maturity level for many different categories. In the tables, below, some descriptions are additive. That means a genealogist not only performs as described for a given level, but also practices all positive skills mentioned in lower levels. The citation descriptions below are an example.

# GML Sources Citations
1. Entry

Typically relies on compiled genealogies.

Captures URLs for online sources and citations for published sources.

2. Emerging

Mostly relies on compiled genealogies and online sources.

Increasingly captures necessary information for manuscript sources.

3. Practicing

Uses a limited number of record types and repositories. Mostly relies on online and microfilmed sources.

Typically produces complete source citations.

4. Proficient

Uses a wide variety of record types. Often contacts record custodians to obtain copies of high-quality sources.

Gives complete and accurate source citations including provenance and quality assessment.

5. Stellar

Insightfully pursues research at multiple, targeted repositories, making use of a plethora of records and record types.

Overcomes limitations of genealogical software to create well organized, industry standard reference notes and source lists.

 

Information and Evidence

As genealogists learn, we handle information and evidence in more effective ways.

# GML Information Evidence
1. Entry

Has not learned to abstract information from sources (still thinking in terms of primary and secondary sources), making analysis more difficult.

Limited understanding of what constitutes evidence and the role it plays.

2. Emerging

Learning to separate the concepts of sources and information. Beginning to see the need for healthy skepticism.

Growing dependence on evidence. Learning to evaluate the quality of sources, information, and evidence.

3. Practicing Collects information from the consulted sources and analyzes source characteristics separately from information characteristics.

Correlates the evidence arising from the collected information.

4. Proficient

Captures all information from the consulted sources.

Carefully accounts for all conflicting evidence.

5. Stellar

Recognizes usable information where others don't.

Able to find evidence in spite of burned counties, illegitimacy, and other dead ends that stump other researchers.

 

Conclusions and Conclusion Trees

As genealogists become more experienced, we make better conclusions and as our conclusions get better, our conclusion trees grow in quality.

# GML Conclusions Conclusion Trees
1. Entry

In the absence of evidence, uses logic and instinct to eyeball conclusions from available information.

Often incorrectly merges or combines individuals in trees.

2. Emerging

May discount conflicting evidence in preference to the most prevalent evidence.

Growing hesitancy to merge or combine individuals without evidence.

3. Practicing

Accurately assigns correspondence between records and people. Resolves conflicting evidence.

Never merges compiled genealogies into own tree. Contributes or changes community trees only with evidence.

4. Proficient

Creates soundly reasoned, coherently documented conclusions. Can utilize indirect evidence.

Manages evidence separately from conclusion tree. Not interested in trusting high quality conclusions to a community tree.

5. Stellar

Develops new research methodologies. Creates clear and convincing written conclusions. Teaches and inspires others.

Has put so much work into their conclusion tree, they are highly likely to publish it.

 

I invite the feedback of the entire genealogical community (including you grammarians, editors, and perfectionists).

  • In each category, see if you think I’ve described the natural maturity progression.
  • Do the single word GML names and the single sentence definitions reflect the descriptions across all the categories?
  • Also, I’m hoping a majority of genealogists would score the same GML in each category (sources, citations, etc.). Score yourselves and let me know if you think I need to make adjustments to any categories. Of course, since we are all different, we should expect that some people will have different scores in different categories.
  • Do you have another category you’d like to see added? Feel free to contribute draft descriptions for each level in the category. I have a couple of my own to introduce next time.

The Genealogical Maturity Model can be a useful tool for evaluating ourselves, setting goals, and improving our capabilities. Together we can fine tune this tool and we can optimize its value.

Stay tuned…


      1. Wikipedia contributors, "Capability Maturity Model," Wikipedia, The Free Encyclopedia (http://en.wikipedia.org/w/index.php?title=Capability_Maturity_Model&oldid=322826971 : accessed October 30, 2009).

17 comments:

  1. AI, interesting post. Think it could be improved by adding definitions of:
    --Information
    --Sources
    --Evidence

    I particularly hate the term 'information' because it means quite different things to different people. Some think that if they take 'information' from IGI it is inherently true. Persons in the more Mature levels recognize that IGI is badly corrupted by Family Group Sheets submitted by persons who did not bother with evidentiary research, and by extractors under LDS auspices who made many transcription errors (and in some databases, omissions).

    Some think that any assertion at all, including what is in Trees, is 'information.' The phrase in message boards and mailing lists, "I have . . . . " nearly always means that someone has lifted some material from trees or web sites, and that the person wants us to think that they did some research that led to the 6-generation "information" posted after the phrase.

    The people devoted to cobbling together pieces from others' internet trees do not distinguish between assertion and fact. It is evident that many do not even read what they have collected for basic sense; the persons siring children after death, marrying their mothers and siring themselves, and living in nonexistent places for 245 years are now epidemic throughout Treedom. These 'problems' do not even embrace those tracing genealogy to non-documentable personages in Central Asia, or to fictional persons living in Troy at an impossible time.

    Perhaps, in light of the above, the first category should be divided into two.

    The scheme could possibly be applied to orgaizations in the genealogical realm, as expressed in their genealogical efforts.

    I would put Ancestry.com at level one. Where would NewFamilySearch go?

    ReplyDelete
  2. I think you have provided an important step in the development of genealogy's theoretical basis, a prime requirement for gaining acceptance in academia. I am convinced that genealogy has become its own social science, with a basis in Jacobus and Greenwood, seminal work in the theory of evidence by Elizabeth Shown Mills, and now the theoretical models you have presented here. I think this is an exciting time to be a genealogist, certainly for those of us (like me), with an academic bent.

    Food for thought indeed, and I am going to be referring to your post, and to whatever I find that will grow out of it, for a long time to come. I was quite easily able to place myself in each of your models.

    Addressing Geolover's comments about the term "information," I think it is a perfectly good term when used in the sense of Mills's model. It is up to the rest of us to insure that newcomers to the field, and dabblers in it, are apprised of the proper meaning and use of the term. Information can be accurate or inaccurate. What is in IGI (and a variety of other locations) is information, albeit often corrupted or just plain wrong, as Geolover points out. The important aspect to remember is that it is what we do with the information -- how we evaluate, analyze, and employ it -- that counts.

    I do agree that it will be helpful in your models to add definitions of terms. Emphasis on the important distinction between sources (as original or derivative) and information (as primary or secondary) is crucial to the understanding of the models, as well as understanding why the two items are not evidence but rather that evidence is adduced from them.

    Excellent work.

    ReplyDelete
  3. As you present this as a rough idea, it seems more pertinent to comment on the overall idea than to debate details such as how well your proposed level definitions match observed reality.
    I think Geolover quite rightly hinted at the difference between data and information, but I'd like to draw your attention to something more fundamental.

    You want to apply the maturity level concept to individuals as well as their work products.
    The Capability Maturity Model was developed to rate maturity of software development processes in organisations. Software developers partake in a process, but do not have an individual CMM level.
    Aside: the underlying ideas, applied to individuals, have been captured in the Personal Software Process (PSP).

    Software developers have (a bewildering variety of) certifications and experience. Likewise, genealogist have certifications and experience.
    You may want to debate the value of current certifications, and desire to improve these to better match natural development of researchers, perhaps by introducing several levels of certification.
    However, fact is that genealogical organisations are already offering professional certification.

    Defining a Genealogy Maturity Model for organisations may be an interesting idea, but actually doing will probably take years of research.
    Therefore, the first question you need to ask is whether it is worth doing at all; what would we gain once we have it? For CMM the practical answer is that it enables the government to rate suppliers.
    What is the answer for a Genealogical Maturity Model? That it helps users to pick genealogy hosting companies?

    ReplyDelete
  4. I do agree with Tamara Jones that the basic question as to what will be achieved needs to be answered first.

    I have a basic apprehension of developments in categorisation of systems, reporting, sourcing and people. For accreditation purposes it may be necessary, but for other purposes I am less certain. Let us take sourcing as an example, as long as it is clear and reproducible then that should be sufficient.

    Standardisation, or making square pegs fit into a round holes, have led to the ridiculous position where England is described as a "state" - it is emphatically not, it is a country as are Scotland and Wales, the latter also being a principality. Northern Ireland is a province of the UK and Ireland is a republic. The Isle of Man and the Channel Isles are not, and never have been, part of the UK. GB is an alliance of 3 countries. Now try and fit that into the four field standard!

    If the proposal is for America only then perhaps it should say so, but even then it is likely to creep into international arenas where it is probably not applicable.

    ReplyDelete
  5. Insider,

    Very nice blog entry. I think your descriptions are very good. If you want an expert opinion perhaps you could email Elizabeth Mills or Dr. Thomas Jones and ask them if they would comment. There is a table missing which I will address later.

    Perhaps though in the table on sources and citations you need to add in derivative (printed) sources as in printed abstracts of various original records, in addition to mentioning compiled genealogies. An over-reliance on such derivative sources is what distinguishes a barely competent person from a competent one many times. The more competent or "mature" genealogists will use such printed sources as finding aids to access the originals, but will trust them as far as negative searches unless lack of progress dictates page by page searches of original records in the hope they were misindexed or poorly transcribed (and an ability to evaluate the relative worth of such printed sources though their introductions is an important skill).

    As to the missing table in your scheme, I mean re research methodology. You mention creating new methodologies in the stellar rating of the conclusions table, but research methodologies deserves its own table in my opinion. Not keeping good research calendars/logs including negative searches are a sure sign of lack of "maturity". Or failure to check the pagination of a county's census when one can't find the search target. As to the top level of using methodology, you could search for Marsha H. Rising's article which I think is titled, "The Core of Competence of a Genealogist". I recall it being in an APG quarterly which thus should be available through PERSI. If you need I can try to dig out my copy.

    And of course re methodology, each geographic area and each time period will have its own strategies that are needed. Of course these are often the subject of presentations at conferences.

    Knowledge of sources + methodology + ability to logically correlate and analyze information and evidence + ability to write good reports and summaries of same = maturity/competence.

    One last observation which is outside the scope of your post. Since you brought up the concept of genealogical maturity in the context of a NFS system requiring evidence/proof/citations and arbitration of same, it bears saying again that the vast majority of people engaged in genealogical "research" will perpetually be stuck in the first two levels you give. Most of these people can't or won't be educated. And two decades down the road such a newbie of today will tout his/her "twenty years of experience", when in reality all they have is one year of experience twenty times over.

    MikeF

    ReplyDelete
  6. I agree with MikeF. Many of my genealogical friends are perpetual entry level, or perhaps dilettantes or dabblers. They make very little effort to improve their skill and knowledge and/or the quality of their data base.
    JohnC

    ReplyDelete
  7. As someone who has been doing this stuff and writing about it (and teaching it) for 30+ years I don't think you can draw a meaningful parallel between any sort of semi-academic research in the social sciences (which is what genealogy is) with corporate software development schemes.

    Nor, if you really feel a need for this sort of categorization, do you need anything like five levels. That sort of pseudo-precision doesn't exist in this field. The assignment to categories is entirely subjective, for one thing.

    I see three sorts of "family history people":

    (1) Novices. They don't get it. (Name collectors never get it.)

    (2) Experienced genealogists. They get it. Where most active "real" genealogists are. Not an automatic progression. You have to want to get it.

    (3) Mentors. Very experienced, and also possessing the talent or ability to help others move from the first level to the second, through writing and teaching and one-on-one guidance. A very small number of people.

    ReplyDelete
  8. Dear Michael,

    I'll meet your 30 years of experience and raise you 10 plus 30 years experience in software development. Seriously, there is no attempt at drawing parallels between genealogical proficiency and software development process management.

    The choice of five levels is arbitrary and I am open to suggestions on expansion or contraction. Please be aware that I no not regard name collectors as genealogists. I plan on a future article showing GML 0 for name collectors. In contrasting your levels against mine, you'll want to take that into account and compare your three against my six.

    I like your thoughts about "don't get it" versus "get it." I'd like to hear more. Are the two separated by an "aha moment," or is it a gradual realization? What separates the two groups? Is the division unbreachable?

    Thanks again for your thoughts,
    -- The Ancestry Insider

    ReplyDelete
  9. Dear Tamura and Ron,

    The purpose of the GMM is to help individuals improve. I created it with the desire to inform, teach, and motivate greater genealogical proficiency. I created it with the hope that it would assist genealogical product managers design better products, leading users to greater genealogical success.

    -- The Insider

    ReplyDelete
  10. I can see the bumper sticker now: "NAME COLLECTORS ARE GML 0"

    ReplyDelete
  11. Coming from a background of history academia, the thing that drives me crazy about genealogists is their willingness to rely on derivative sources. In good scholarly historical research, a derivative source is never used unless it adds analysis or refers to a source that no longer exists. A derivative should be just research that is still waiting to be done. I think that is alot of what you are describing here. Maybe the word "derivative" should be used more. And to help "entry" and "emerging" genealogists, a list of definitions for "information" "evidence" "sources" and "derivative" would be a good appendage.

    Maybe applying the Genealogical Proof Standard here will help test the comprehensiveness of this model. It looks like you address #1 in Sources, #2 in Information and Evidence, #3 in Conclusions, #4 in Citations, and #5 in Conclusion Trees. If I could add anything, I think there needs to be another table covering what the output of this effort is. Something about what the genealogist produces. You somewhat cover that in Conclusion Trees but it could go beyond that. Mike's suggestion of a methodology table is a good one too.

    This is a wonderful and much needed analysis for every genealogist. In answer to Tamura Jones as to the purpose, I hope it will cause some self-reflection and push genealogists to reach for the next level. And, like Karen says, it gives genealogy more theoretical basis to push it toward becoming more academic. I agree with the above that genealogists for the most part don't want to progress beyond the first two levels. Genealogy is a funny entity that way. I think for most people it is a hobby or for some perhaps a religious obligation. You don't get as much of that in other social sciences. In this field it seems the balance of educated, accredited, scholarly practitioners is far outweighed by the number of people who want to dabble in it. That is sad to me. Genealogy has alot to offer to the world in the sense of family, identity, self awareness and emotional healing. It is too bad that so many genealogists never get to where they are doing it well.

    Elizabeth Shown Mills is talking about analysis when she says, “Bias, ego, ideology, patronage, prejudice, pride or shame cannot shape our decisions as we appraise our evidence. To do so is to warp reality and deny ourselves the understanding of the past that is, after all, the reason for our labor.”(Evidence Explained. Pg 15) I think that applies to all aspects of good research though. Why bother if what you are producing isn't as accurate as you can make it?

    Off to read "The Core Competence of a Genealogist"....

    ReplyDelete
  12. Ancestry Insider,

    I work for a Fortune 500 company at CMM level 3. It took us many years to get there.
    Your company is just starting out with CMM? You seem overwhelmed and confused by it.

    I agree with Michael that you cannot compare genealogy proficiency with software development levels.
    Your attempt at drawing a parallel between those two things makes no sense.
    Tamura rightly asks what use defining individuals levels is if there already are certification programs. Aren't these certifications good enough?
    Ron asked what use your 'genealogy maturity levels' are for genealogy organizations, and your reply again confuses it with individual proficiency...
    He is right that you are trying to put square pegs in round holes.

    ReplyDelete
  13. This comment has been removed by the author.

    ReplyDelete
  14. I disagree with those who say you can not compare the software development process to genealogy. Every competency matures: whether it is an individual who acquires more knowledge and skills; a process that is reviewed and revised over time; or a "field of endeavor" such as family history or genealogy.

    I found your GMM model useful to me to evaluate where I am at in the overall maturing process and where I may need to improve.

    One caveat: my experience with CMM is that many organizations developed a huge outpouring of paperwork to prove what level they should be ranked in while obscuring the fact that they changed nothing. I would not like to see that happen to genealogy.

    ReplyDelete
  15. Insider,

    I just wanted to post a link to Randy Seaver's blog discussing your post and model, as well as his commentators including Ms. Mills.

    MikeF

    http://www.geneamusings.com/2009/11/square-pegs-into-round-holes.html#links

    ReplyDelete
  16. Personally, I think the GMM is a great idea-- especially if it can be written such that it aligns with other genealogy standards.

    I also see its use as three-fold:

    1. It helps evolving (hopefully maturing) "genealogists" like me assess where I am and see where I need to improve.

    2. It helps organizations, companies and other professionals who offer products and services to consumers understand different target audiences. And hopefully (one can dream) improve their own offerings along a similar scale.

    3. It may help people at different levels identify those products and services that match where they are on the scale. For example, it would demonstrate that, on its face, Ancestry.com is mostly targeted at the less mature audience ("share and copy trees here"). In fact, I'm moved to break down the offerings of Ancestry.com along this model so that the user who comes in through Ancestry's front door realizes there are many more resources buried within for more mature research.

    (Which reminds me, AI, I want to take up in a separate topic, your point made in a far earlier post that Ancestry.com's future lay in trees. Ugh... I hope its trees are its demise... but different topic.)

    I'm with ESM who posted on another blog that this is an incredibly important discussion to be having.

    For myself, I am increasingly frustrated with the exponentially growing BAD genealogy out there-- due in great part to such services as Ancestry.com, which does very little to encourage *good* research and does a whole lot to perpetuate bad or no research.

    From an emerging practitioner,

    Jillaine Smith

    ReplyDelete
  17. I agree that basing a GMM on the CMM is a flawed premise. The CMM, as described, is a somewhat objective measure by which a sophisticated purchaser (e.g., US govt.) evaluates potential suppliers (e.g. software cos.) In genealogical terms, this construct might apply to an institute / school / conference evaluating potential lecturers, a major genealogy provider (Ancestry, Footnote, etc.) assessing potential partners or subcontractors, or -- in rare cases -- a sophisticated genealogist hiring a fellow genealogist.

    As described here, the GMM is, in contrast, to be used for individual self-assessment. If this is indeed its intended focus, a more relevant comparison, IMO, is with educational self-assessment. Years ago (when I kept current in the field), self-assessment was all the rage in schools at all levels, but I gather it has fallen out of favor pedagogically. (Certainly for young elementary students; possibly not for graduate and post-graduate studies). I did some limited searching to find the current state of self-assessment's perceived value in reliability, predicability, and other factors.

    I found that at least some studies have found self-assessment valuable when 1) students are taught how to self-assess, 2) students' self-assessment is weighted with teacher assessment, and 3) there are hard rules against which to judge (e.g., it's more effective in math than in language classes). None of these criteria would seem to apply to genealogy generally (though it might to genealogy institutes / certificate programs, where there are teachers to handle items 1 and 2).

    Without the somewhat more objective leavening of teacher input and evaluation against a known "correct" answer, I question how the GMM would be any different than the current self-assessment researchers are prone to when they call themselves "newbie," "experienced," or "expert." Because of this, I also doubt any reputable educational institute would regard the GMM (as proposed) as evidence of the academic maturity of the discipline, as some would wish.

    A more formal literature search on self-assessment studies (when they work and when they don't) would seem to me to be a necessary first step to developing any genealogical self-assessment guidelines.

    I'm not a teacher, but I played one on TV... no, that's not right, ...but I worked with lots of Ph.D.s in education theory. ;-)

    ReplyDelete