EditBOT updating the profiles [no longer using command]

Question

There were Ideas to implement some way to instruct EditBOT to update dead ancestry links on the profiles. Here I am preparing the solution.

There will be a template https://www.wikitree.com/wiki/Template:Update_Profiles Instructions are on the template.

It will be possible to place it on these type of pages and EditBOT will perform the following actions.

• Category: Check all profiles in the category.
• WikiTree member profile: Checks all profiles managed by the user.
• Any other profile: Checks all relatives.

To see how the template will be displayed, see:

• https://www.wikitree.com/wiki/Category:Pokopali%C5%A1%C4%8De_%C5%BDale%2C_Ljubljana%2C_Slovenija
• https://www.wikitree.com/wiki/Trtnik-2
• https://www.wikitree.com/wiki/Trtnik-4

---

Update: the {{Update Profiles}} command should not be used on any profiles. We will be disabling it. -- Chris, 16 Jan 2019

Comments

Aleš, thanks for the post. Do I understand correctly that the bot will only edit those profiles where it finds the category? Ie., if we want editbot to fix the links, we need to manually add the category to those profiles we want it to do its magic on?

---

Jillaine,

I could be wrong but it looks like it means that you can add the template to a category page and it will update all of the profiles in that category or you can add it to individual profiles or you can add it to your own profile and it will check and edit all profiles managed by you.

---

Yes, as  Deb says, you can use it in each of these places.

---

Thank you, Ales! What you're doing with EditBot is terrific.

I should mention, using templates to give instructions to EditBot is not an officially approved usage of templates <https://www.wikitree.com/wiki/Help:Templates>. We have never formally discussed and agreed upon this as a community. (Instructions to EditBot on category pages could be considered Category Templates. These aren't as tightly controlled because only advanced members edit category pages. They are less broadly collaborative and aren't the heart of what WikiTree does.)

If these commands are to become widely used I think it should be discussed. We may also need to further develop the guidelines on bots in general. <https://www.wikitree.com/wiki/Help:Bots>

To emphasize, Ales is doing great stuff with EditBot and I fully support it, but I would caution against widespread community usage until more community rules are agreed upon.

---

Let me give you some examples.

{{Update Profiles|ASAP}} on Category page will check all profiles in this category on next BOT run.

{{Update Profiles|ASAP|1}} on Category page will check all profiles in this category and all its 1st level subcategories.

{{Update Profiles|ASAP|1}} on Category page will check all profiles in this category and all its 1st level subcategories.

{{Update Profiles|ASAP|2}} on Category page will check all profiles in this category and all its 1st level and 2nd level subcategories.

So one way to update a profile would be by placing some category on it. It can be any category. I guess I will not run it on 1000000 profiles in Category:Unsourced_Profiles.

For Projects, it can be placed on Project's  WikiTree-xx account and all project managed profiles will be checked.

Any WikiTreeMember can place {{Update Profiles|ASAP}} on their own profile and all profiles managed by a user will be checked.

In case you come to some random profiles with wrong links, you can put {{Update Profiles|ASAP|9}} on it and 9 generations in all directions will be checked.

That is what I can do.

---

Chris / Aleš,

Just a thought, but would it be possible to add a Bot: namespace for this type of work?

---

What is the definition of "dead" in this case? (I'm on a ship in Grand Caynen and my connection is too wonky to do research or add sources).

---

All, I'm just an average member of WikiTree and I have had very little experience with such bots.  I hate to show my ignorance, but how does ASAP identify any category?  For that matter how does ASAP 1, ASAP 2, etc. identify anything?  When I look at ASAP all I see is AS SOON AS POSSIBLE.  Does that relate to what you are doing?

---

I believe it's the older ones with trees.ancestry.com in the link.

---

Judy, in this case ASAP is just a parameter of the template and it means exactly what you have noted (in the grand scheme of its usage).

---

I'm thrilled.  I hate dead links and manually checking is painfully slow.

---

Now I implemented also update of profiles by relatives. I am a bit surprised about the number. It goes over 4000 in 9 generations.

Now thinking about it, it can go to a lot of profiles by following the mariage connections.To prevent this, I think it would be best to follow only blood connections and add their spouces without following further.

---

Here are the results of the first run. Number of changes was limited to 50 per tree. If all updates were performed, template was removed from the profile.

http://www.softdata.si/wt/EditBot/ChangeLog%2020181225%20004125.htm

I think that I will switch to only blood relatives if number of generations will be more than 5.

---

A decision has been made: the {{Update Profiles}} command should not be used on any profiles. We will be disabling it.

The reason for this isn't directly related to the style questions regarding using templates as commands (explained in my original answer below). It's because EditBot has "sysop" powers and operates on all profiles. It doesn't follow the rules developed by the community and explained on Help:Bots. Therefore, its operations need to be carefully managed by the team.

Answer 1

I changed the appearance of the template to a Sticker. I am very pleased with its look. It goes nice with SAT, CAT stickers, or can be placed anywhere in the profile.

Now that I have a bit more time, I will perform initial run. I will only make first 5 operations to see what will be done and how users will respond.

Comments

Looks great on my profile, Aleš. I look forward to seeing how it does with that import that I hurried with a while back.

---

First 5 changes were done for all Users with Managed profiles command. Here are the results.

http://www.softdata.si/wt/EditBot/ChangeLog%2020181217%20204621.htm

Unlisted profiles were skipped.

Let me know what you think of the changes. If there are no problems, I can run 50 or 100 changes tomorrow to check the actions.

---

Mine look OK. Not as many changes as I expected.

---

You had change on every 10th profile. so you will have over 100 updates.

Is there something you expected to change on this 40 profiles, that were examined?

---

The majority of changes on these were from one citation that was included in my first GEDCOM. Only one on list of updates had any Ancestry links. None of the profiles that are currently in my personal Bio Cleanup Needed category were touched. I can put the EditBot Template on the category but I'm not sure all the profiles that need to be touched are in there.

---

I will run the full update now, since there were no problems with the test.

---

Bigger run for 50 updates went through.

http://www.softdata.si/wt/EditBot/ChangeLog%2020181218%20213128.htm

---

I added 2 new operations to EditBOT on save. 

• It corrects all HTTP links to WikiTree to secure protocol HTTPS. Those links will work a bit faster.
• I remove the text: ''This biography is a rough draft. It was auto-generated by a GEDCOM import and needs to be edited.'' this was added on GEDCOM import for a few years back in 2011. 

---

Thanks for that little cleanup to help clean the profiles!

Answer 2

Aleš, the notice it leaves in a category: EditBOT maintenance: Profiles will be updated on next run. Check will be performed only on profiles in this category and 1st level subcategories.

Is there any way to move that under the category members? It's prominently displayed at the top.

Comments

It can't be moved under the categories.I could make it more like a sticker. I am even considering no output or just image with a hint and link. I also don't like the display, but that is the first draft.

First we have to define the functionality, then the display.

Answer 3

Will EditBot remove the template after it performs the corrections? It seems to me that this is a one-shot function. Once the links have been corrected, the template is no longer needed. I haven't seen this mentioned anywhere.

Comments

In case of ASAP parameter, after the updates are completed, the template will be removed.

Template with Month period, will remain and Editbot would check each month if any new update is needed.

Answer 4

Thanks Ales looks like a very handy one, especially that it only has to be added to one profile or category and will check and update them all :)

Comments

There are some reservations on privacy.

I think that the template on category and on any profile by generations will have to obey the privacy and will be performed only on open profiles.

The version on WikiTree users profile will perform the change on all managed profiles. User's profile is always private and only the user can add the template to it. And that user has edit access to all profiles, he is the manager of.

Answer 5

As I noted in a comment, I have some reservations related to this. In particular, I'm not confident that we should endorse using the command on profiles other than your own account profile.

Giving instructions to bots is not an officially approved usage of templates. https://www.wikitree.com/wiki/Help:Templates

On category pages they could be considered Category Templates. We aren't as strict about templates on categories because only advanced members need to edit those pages. They aren't at the center of WikiTree collaboration. Similarly, the usage of templates on members' own account profiles has never been tightly controlled. If any profile can said to be "owned" and fully controlled by one member, it's their own account profile.

Profiles of non-living people are not supposed to be owned and controlled by any individual. https://www.wikitree.com/wiki/Help:Ownership_and_Control

The fact that we collaborate on them is point #1 of the Honor Code.

If we encourage members to add and remove these commands from profiles it implies that they have the right to control EditBot's behavior. EditBot needs to be controlled, but that control should be exercised by the community and team through our rules on bots and our normal process for developing and changing those rules.
https://www.wikitree.com/wiki/Help:Bots
https://www.wikitree.com/wiki/Help:Developing_New_Rules

Maybe I am overstating this point. I think that requesting that EditBot spend time on certain profiles is probably OK. The number of edits that EditBot can make is limited and some profiles may need them more than others.

There are just things about putting the instructions inside profiles that makes me uncomfortable. As I said, there they feel more like commands than requests.

Another concern I have is the general one about templates, the reason more of them have not been approved. We have generally approved the usage of templates as feature boxes. We have encouraged new members to recognize {{these}} as tags that insert a specially-formatted box on profile: a Research Note Box, a Sticker, or a Project Box. If members need to understand that these tags could be doing all sorts of different things, we are raising the bar for participation on WikiTree.

Every additional usage of templates marginally increases the complexity of collaboration here. We always need to guard against the natural tendency of systems to become more complex over time.

Comments

So, as an alternative to the template, would it be acceptable to just have EditBot make limited runs of updates to old Ancestry links in batches of say 1,000 at a time to any Open profiles? [Or pick a number, whatever doesn't overburden the system.]  Could be a part of regular maintenance - daily, weekly, whatever. Over time, the old links would be gradually fixed, and we wouldn't need to add this layer of template complexity to WikiTree.

---

Ancestry tree links are used on 297635.  That is the same as the number of all edits of EditBOT since its beginning of operations. So I wouldn't do an overall update.

Answer 6

I agree with Chris, EditBot actions across the board could get distracting.  I don't have access to Ancestry, so I never know if a link if still good or not.  I tend to leave them on the profile pages, so anyone who does have access can use them if they so choose.  If they find the link is dead, they can take appropriate action on a case-by-case basis.  And sometimes links are only temporarily out of commission, so how can EditBot tell the difference?

Comments

Case-by-case editing of these links by a human is painful. Consider a profile like https://www.wikitree.com/wiki/Paine-427 , which is one of many that are essentially shrines to the perpetuation of Ancestry dead links. To manually convert a URL like http://trees.ancestry.com/rd?f=sse&db=worldmarr_ga&h=920480&ti=0&indiv=try&gss=pt to the template, it is necessary to extract the database name (in this instance worldmarr_ga) and record number (in this instance 920480) from the gibberish and copy them into the appropriate template (in this instance I think it's {{Ancestry Record}}), so the proper template would be {{Ancestry Record|worldmarr_ga|920480}}. If I'm sufficiently desperate to see the source, I can do that, but when the profile has about eight dead Ancestry URLs and I'm fretting about thousands of similar profiles included in a project, the job is much too large to contemplate doing manually.

---

I hear you, indeed it can be painful Ellen.  But although technology is great, it also has the drawback that it doesn't think or look further than what it can see.  Which is why I am very leery of such a program being run.

---

That is the reason I introduced the template and Ellen can execute it on her managed profiles and saves a lot of hours of manual work and you can do it manually.

Answer 7

Can the updates be restricted to the trees.ancestry.com domain?

I am very interested in having EditBot update URLs in the domain trees.ancestry.com because the domain is dead and all of those URLs have been dead for a couple of years now. These URLs are in many WikiTree profiles, and I consider them an embarrassment. Updating is badly needed to correct numerous dead links and enable WikiTree members with Ancestry accounts to check the linked contents.

I understand, however, that the Ancestry external link templates would update all Ancestry URLs, whether they need it or not. I am not at all  enthusiastic about having EditBot perform these functions on valid working links to Ancestry because the edit would replace the current link text with a bit of generic text. For example, on profiles where I added a link to an Ancestry record using a format like the one below, the Ancestry link templates would obliterate the source description information that I used in the link:

United States Census, 1870. [http://interactive.ancestry.com/7163/4278555_00480?pid=9903198 Enumeration sheet for Millersburg, Dauphin, Pennsylvania]. 12 AUG 1870. Roll: M593_1335; Page: 678A; Image: 480; Family History Library Film: 552834.

In that example, I don't want to see "Enumeration sheet for Millersburg, Dauphin, Pennsylvania" get replaced with something generic like "Ancestry Image."

Comments

Shorter version of question:
If I use this template, can I expect that the Ancestry edits will be limited to the trees.ancestry.com domain, or will the Bot also convert working links to the template format?

---

At the moment the bot replaces 

Ancestry Record Links

http://trees.ancestry.com/rd?f=sse&

with

http://search.ancestry.com/cgi-bin/sse.dll?

Ancestry Tree Links 

    http[s]?://(www\.ancestry\.com/pt/PersonMatch|trees\.ancestry\.com/pt/AMTCitationRedir)\.aspx\?tid=(?<tid>\d{2,})&pid=(?<pid>[-]?\d{6,})

with

https://www.ancestry.com/family-tree/person/tree/${tid}/person/${pid}/facts

Ancestry Tree (no person)

http[s]?://(www\.ancestry\.com/pt/PersonMatch|trees\.ancestry\.com/pt/AMTCitationRedir)\.aspx\?tid=(?<tid>\d{2,})&pid=(?<pid>\d{0,5})

with

https://www.ancestry.com/family-tree/tree/${tid}/family

I have to update this to use new templates. For now only this is updated. those are all dead links corrected to working one (or not if they were deleted) 

For now I don't update working links, but I think I should and will at some point, since they could eventually stop working.

Answer 8

I could add another parameter to define which operations editbot  should perform.

Those parameters would be 

• CONT
• Ancestry or maybe AncestryDead
• FAG or FAGOld 

I also Remove empty lines at the beginning and the end, but those are removed on save even if I don't remove them.

This way I could develop new cleanup modules, that users would perform on managed profiles or not.