Problem 1746 - With MPI and MT enabled it is possible that events share the same RNG seeds
Summary: With MPI and MT enabled it is possible that events share the same RNG seeds
Status: CLOSED FIXED
Alias: None
Product: Examples/Extended
Classification: Unclassified
Component: paralle/MPI (show other problems)
Version: 10.1
Hardware: All All
: P5 minor
Assignee: Koichi Murakami
URL:
Depends on:
Blocks:
 
Reported: 2015-05-18 18:47 CEST by Andrea Dotti
Modified: 2017-06-20 09:47 CEST (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this problem.
Description Andrea Dotti 2015-05-18 18:47:34 CEST
The current strategy to prepare seeds for MPI has a logic flaw when used in conjunction with MT. Even if technically we do not observe issues it is theoretically possible to have all ranks producing exactly the same events (e.g. reuse of seeds).

The problem is the following:
MPI prepares the seeds for the ranks in a buffer so that:
We start from a master Seed, than the ranks seeds are prepared:
Rank#0 <-> Seed 0
Rank#1 <-> Seed 1
Rank#2 <-> Seed 2
and so on

However with MT each Rank is a MT run that has the master seeded from the MPI seed. From this master a per-event seed is generated in a similar way and there is no guarantee that the seeds are re-used. This situation could theoretically happen:
Rank#0 Event#0 <-> Seed 0
Rank#0 Event#1 <-> Seed 1
Rank#0 Event#2 <-> Seed 2
...
Rank#1 Event#0 <-> Seed 1 (The same as Rank#0,Event#1!!!
Rank#1 Event#1 <-> Seed 2 (The same as Rank#0,Event#2!!!

Practically this does not happen only because with MT we generate couples of seeds for the RNG engines.

The correct strategy would be to modify the strategy in MPI for which the per-rank seeds are staggered taking into account the number of events each rank will simulate. For example, if each rank will simulate 100 events:
Rank#0 <-> generate Seed 0,1,....,99 <-> Seed master thread with Seed 0
Rank#1 <-> Seed 100,101,...,199 <-> Seed master thread with Seed 100
...

I propose to make this change for 10.2
Andrea
Comment 1 Koichi Murakami 2015-12-11 06:56:11 CET
I will implement an additional random number generator for that purpose.
Comment 2 Andrea Dotti 2016-12-14 23:56:07 CET
Hi Koichi,
it is my understanding that the new MIXMAX engine should solve automatically this. Let's keep this open a bit, I'll confirm this and then we can close.