Documents similarity fails when run on the non-deduplicated graph

[marekhorst]

Originally reported on redmine: https://issue.openaire.research-infrastructures.eu/issues/7333#note-12

Here is the failure example:

http://iis-cdh5-test-m3.ocean.icm.edu.pl:19888/jobhistory/tasks/job_1614993033499_83511/m

all failed attempts had the following entry in the log:

Timed out after 600 secs

what indicates timeout issue.

The script failed at coansys-integrated-similarity_docsim-recalculate-sim-about-1-filter docsim subworfklow during the sim1_postprocess_s1_e1_filter_input phase (sim1-postprocess-s1-e1-filter-sims.pig script).

We might need to increase the timeout by setting mapreduce.task.timeout to a higher value than the default 10 minutes. The problem is docsim sources are maintained in coansys project so we need to hotpatch the sources (what we already do to cope with slightly different docsim issue.

We should run several experiments to make sure we need to increase the timeout just in this single place and to find the proper value (we could start with 30 mins).

[marekhorst]

After several test rounds it turned out two changes are required to be applied on sim1_postprocess_s1_e1_filter_input phase:

1. increasing mapper task timeout

• from default 10 minutes to at least 30 minutes (several attempts exceeded 10 minutes margin)
• needs to be applied directly on sim1-postprocess-s1-e1-filter-sims.pig script therefore requires pull request created for docsim origins maintained at https://github.com/CeON/CoAnSys
• temporarily we could hotfix IIS workflow in the same way we are dealing with Replace buggy pig rank function with custom solution #927

2. increasing xmx memory allocated to mapper

• this can be done on an oozie workflow.xml level by bumping -Xmx10g currently defined in IIS primary uber workflow up to -Xmx12g which proved to be working correctly

[marekhorst]

Pull request related to the sim1-postprocess-s1-e1-filter-sims.pig script modification was already issued: CeON/CoAnSys#435.

Until this gets merged there was one another pull request created addressing IIS deployment script modification:
https://git.icm.edu.pl/openaire/iis-deployment/-/merge_requests/8
which should cover docsim PIG script hot-patching.