Ytukyg

I plan to read the feature data from multi csv files.

Each csv file has 150 columns, and the batch size is 256.

The time cost for just read 1000 iteration needs roughly 12s.

I feel the time cost for that should not be that much, any one here can have some suggestion?

def _parse_csv_row(*vals):

    features = tf.convert_to_tensor(vals[0:f_size * 5])

    class_label = tf.cast(vals[f_size * 5] + tf.convert_to_tensor(1.0, tf.float64), tf.int64)

    return features, class_label





def get_batch_data(name):

    root_path="g:\market\2018-11-12\feature_{}\".format(name)

    file_queue = list(map(lambda x: "{}{}".format(root_path, x), fnmatch.filter(os.listdir("g:\market\2018-11-12\feature_{}\".format(name)), "*.sz_result.csv")))

    record_defaults = [tf.float32] * f_size * 5 + [tf.float64]

    selected_cols = reduce(lambda x, y: x + y, [list(range(1 + x * 29, 1 + x * 29 + 9)) for x in range(0, 5)]) + [146]

    dataset = tf.contrib.data.CsvDataset(

        file_queue,

        record_defaults,

        buffer_size=1024 * 1024 * 10,

        header=True,

        na_value='0.0',

        select_cols=selected_cols)

    dataset = dataset.apply(tf.contrib.data.map_and_batch(

        map_func=_parse_csv_row, batch_size=train_config.BATCH_SIZE))

    dataset = dataset.prefetch(256 * 1024)

    dataset = dataset.repeat()



    dataset = dataset.shuffle(buffer_size=32)

    iterator = dataset.make_one_shot_iterator()

    feature_batch, label_batch = iterator.get_next()

    return feature_batch, label_batch



config = tf.ConfigProto()

config.gpu_options.allow_growth = True

with tf.Session(config=config) as sess:

    sess.run(tf.global_variables_initializer())

    a, b = get_batch_data("train")

    start_time = time.time()

    for x in range(1000):

        v = sess.run([a,b])

    print(time.time() - start_time)

The line dataset = dataset.prefetch(256 * 1024). It's written after map_and_batch operation. That means that your are prefetching 256 * 1024 batches. So, when your program tries to load 1st record, it actually first loads 256 * 1024 * 256 records. Probably your intention was to prefetch only 1024 batches. In real life it's enough to prefetch only one record.

I would put ds.prefetch(1) line as the last operation on the dataset. See Summary of Best Practices.

Also you allocate pretty big buffer for reading CSV file (buffer_size parameter). If your intention is to cache the whole csv file, you can use ds.cache() operation. Without parameters it caches contents in the memory. Put it before the ds.repeat() operation.