Ytukyg

I have data from a full factorial experiment. For example, for each of N samples, I have J types of measurement and K measurement loci. I receive this data in long format, for example,

import numpy as np

import pandas as pd

import itertools

from numpy.random import normal as rnorm



# [[N], [J], [K]]

levels = [[1,2,3,4], ['start', 'stop'], ['gene1', 'gene2', 'gene3']]



# fully crossed

exp_design = list(itertools.product(*levels))



df = pd.DataFrame(exp_design, columns=["sample", "mode", "gene"])



# some fake data

df['x'] = rnorm(size=len(exp_design))

which results in 24 observations (x) with a column for each of the three factors.

> df.head()

    sample  mode    gene    x

0   1       start   gene1   -1.229370

1   1       start   gene2   1.129773

2   1       start   gene3   -1.155202

3   1       stop    gene1   -0.757551

4   1       stop    gene2   -0.166129

I want to convert these observations to the corresponding (N,J,K)-shaped tensor (numpy array). I was thinking pivoting to wide format with a MultiIndex, then extracting values would generate the correct tensor, but it simply comes off as a column vector:

> df.pivot_table(values='x', index=['sample', 'mode', 'gene']).values

array([[-1.22936989],

       [ 1.12977346],

       [-1.15520216],

       ...,

       [-0.1031641 ],

       [ 1.1296491 ],

       [ 1.31113584]])

Is there a quick way to get tensor formatted data from a long format pandas.DataFrame?

Try with

df.agg('nunique')



Out[69]: 

sample     4

mode       2

gene       3

x         24

dtype: int64

s=df.agg('nunique')

df.x.values.reshape(s['sample'],s['mode'],s['gene'])

Out[71]: 

array([[[-2.78133759e-01, -1.42234420e+00,  5.42439121e-01],

        [ 2.15359867e+00,  6.55837886e-01, -1.01293568e+00]],

       [[ 7.92306679e-01, -1.62539763e-01, -6.13120335e-01],

        [-2.91567999e-01, -4.01257702e-01,  7.96422763e-01]],

       [[ 1.05088264e-01, -7.23400925e-02,  2.78515041e-01],

        [ 2.63088568e-01,  1.47477886e+00, -2.10735619e+00]],

       [[-1.71756374e+00,  6.12224005e-04, -3.11562798e-02],

        [ 5.26028807e-01, -1.18502045e+00,  1.88633760e+00]]])