Ytukyg

I know this topic is fairly discussed but it's still not completely clear to me what the most standard way to produce a permutation of a Dask DataFrame is.

To produce a random index in a non distributed fashion the first thing someone would likely try would be

df['random_index'] = np.random.permutation(len(df))

But in the context of Dask len(df) will trigger a computation. It's not clear to me whether invoking such a computation to realise the length makes sense. An alternative I see instead is to do something like

ds = ds.map(

    lambda (col_1, col_2): (

         <random_string>, col_1, col_2

    )

)

this will create lazily a new pseudorandom column that can be used as in index. Do you see anything wrong with that? I guess to create the random strings someone should use a good hashing algorithm to make sure the keys are evenly distributed. I was thinking of something like that

import hashlib

from random import randint  



hashlib.sha1(bytes(randint(1, 1e16))).hexdigest()

that is both fairly fast and will produce evenly distributed keys. Let me know if I am falling into any obvious pitfall(?)

edit

actually there is no reason to use a string instead of the plain integer. you just need to make sure you produce much bigger indexes than the size of your dataset