Ytukyg

I am currently using Keras (Tensorflow backend) to write a Reinforcement Learning Agent using the Asynchronous Advantage Actor-Critic (A3C) Algorithm using image input data. To test the Network structure and the training process, I want the agent to learn to control the OpenAI-Gym environment Pendulum-v0 (because of the continuous action space).

The network architecture looks as follows:

Input (pendulum scene) -> CNN -> LSTM Layers -> Dense Layers -> Policy and Value Output

Since a single image of the current scene does not deliver any temporal information (in this case about the pendulums velocity which is mandatory to be able to solve this task), I added the recurrent LSTM layers on top of the feature extraction part (the CNN).
I also don't want to use multiple images from the previous time-steps as inputs (e.g. by arranging the last 4 scene images as tiles) due to efficiency reasons, the temporal dependencies should be handled by the LSTM layers.

Since my previous experience was limited to simple feed-forward neural networks,
I prepared myself by training a CNN to classify the Cifar10-Dataset. For the recurrent part, I wrote an LSTM network to generate text based on a WhatsApp chat history. None of these faced me with major problems, but there seems to be some things about combining these two in a reinforcemen learning agent that I dont get:

The LSTM layers need a sequence input! In case of text generation, this has not been a problem since I delivered the last 100 characters as a sequence input, received the next character as output and added this to the input while dropping the first entry to generate the next character.

But in case of a reinforcement learning agent with policy gradients, I don't have a sequence of (e.g.) 100 states as images before predicting the first action to take. What is the usual method to choose when handling temporal dependencies (not necessarily in deep reinforcement learning) without having a dynamic input?
I guess there must be a way to connect the feature extraction and the recurrent part, such that I can feed the network a single input picture in every timestep and nevertheless handle temporal dependencies by using recurrent layers after the feature extraction part.

Are there any good examples, tutorials or code snippets on how to design such a network? I read about Keras "TimeDistributed"-Wrapper but at a first sight it seems to work contrary to what I need.

Defining the sequence length of the LSTM Network to be 1 and reshaping the output of the CNN to be (BatchSize, 1, n_out) also seems like a "wrong" way to tackle this problem.

Best regards