G2G: Getting 2nd 100,000 in wikitree+ via csv [closed]

Question

I am sure I did this last month, but now cannot get it to work. I have a wikitree+ query that I wish to output. It will take 2 csv files. The original query that I am using is: https://plus.wikitree.com/default.htm?report=srch1&Query=TemplateFull%3DUS_Black_Heritage+not+birthlocation%3Dmissinglocation+or+TemplateFull%3DAfrican-American_Sticker+not+birthlocation%3Dmissinglocation+or+TemplateFull%3DUSBH_Platinum+not+birthlocation%3Dmissinglocation&MaxProfiles=500000&Format=&SortOrder=BiLoc&PageSize=-1

I change this to "Separated values" and I get the first 100000 without problem. This is the url: https://plus.wikitree.com/default.htm?report=srch1&Query=TemplateFull%3DUS_Black_Heritage+not+birthlocation%3Dmissinglocation+or+TemplateFull%3DAfrican-American_Sticker+not+birthlocation%3Dmissinglocation+or+TemplateFull%3DUSBH_Platinum+not+birthlocation%3Dmissinglocation&MaxProfiles=500000&Format=CSV&SortOrder=BiLoc&PageSize=-1

I thought I needed to change &PageSize=-1 to &pagesize=100000&pageIdx=1 in order to get the 2nd 100000 but it is not working.

Thanks for your help.

Answer 1

Both links are just the prefill of the search page and pageidx can't be entered. PageIdx is used on the result page where a link for next page contains the correct pageidx parameter, but that is not possible in CSV file. You have to construct the link manually.

The actuall result is retrieved using

https://plus.wikitree.com/function/WTWebProfileSearch/Profiles.csv?

followed by the query parameter

Query=TemplateFull%3DUS_Black_Heritage%20not%20birthlocation%3Dmissinglocation%20or%20TemplateFull%3DAfrican-American_Sticker%20not%20birthlocation%3Dmissinglocation%20or%20TemplateFull%3DUSBH_Platinum%20not%20birthlocation%3Dmissinglocation

and the control ones

&format=CSV&MaxProfiles=500000&pagesize=100000&pageIdx=0

or

&format=CSV&MaxProfiles=500000&pagesize=100000&pageIdx=1

I wouldn't use &SortOrder=BiLoc parameter since you are able to sort the results as you need and default order is much faster to process for wt+ server.

Comments

Thank you very much. I was able to get it to work last month, but my notes were far too cryptic. Your explanation is very helpful.