Main Content

rlDQNAgentOptions

Options for DQN agent

Since R2019a

expand all in page

Description

Use anrlDQNAgentOptionsobject to specify options for deep Q-network (DQN) agents. To create a DQN agent, userlDQNAgent.

For more information, seeDeep Q-Network (DQN) Agents.

For more information on the different types of reinforcement learning agents, seeReinforcement Learning Agents.

Creation

Syntax

opt = rlDQNAgentOptions
选择= rlDQNAgentOptions(名称、值)

Description

opt= rlDQNAgentOptionscreates an options object for use as an argument when creating a DQN agent using all default settings. You can modify the object properties using dot notation.

example

opt= rlDQNAgentOptions(Name,Value)sets optionpropertiesusing name-value pairs. For example,rlDQNAgentOptions('DiscountFactor',0.95)creates an option set with a discount factor of0.95. You can specify multiple name-value pairs. Enclose each property name in quotes.

Properties

expand all

Flag for using double DQN for value function target updates, specified as a logical value. For most application setUseDoubleDQNto"on". For more information, seeDeep Q-Network (DQN) Agents.

Options for epsilon-greedy exploration, specified as anEpsilonGreedyExplorationobject with the following properties.

Property Description Default Value
Epsilon Probability threshold to either randomly select an action or select the action that maximizes the state-action value function. A larger value ofEpsilonmeans that the agent randomly explores the action space at a higher rate. 1
EpsilonMin Minimum value ofEpsilon 0.01
EpsilonDecay Decay rate 0.0050

At the end of each training time step, ifEpsilonis greater thanEpsilonMin, then it is updated using the following formula.

Epsilon = Epsilon*(1-EpsilonDecay)

Note thatEpsilonis conserved between the end of an episode and the start of the next one. Therefore, it keeps on uniformly decreasing over multiple episodes until it reachesEpsilonMin.

If your agent converges on local optima too quickly, you can promote agent exploration by increasingEpsilon.

To specify exploration options, use dot notation after creating therlDQNAgentOptionsobjectopt. For example, set the epsilon value to0.9.

opt.EpsilonGreedyExploration.Epsilon = 0.9;

Critic optimizer options, specified as anrlOptimizerOptionsobject. It allows you to specify training parameters of the critic approximator such as learning rate, gradient threshold, as well as the optimizer algorithm and its parameters. For more information, seerlOptimizerOptionsandrlOptimizer.

Batch data regularizer options, specified as anrlConservativeQLearningOptionsobject. These options are used to train the agent offline, from existing data. If you leave this option empty, a defaultrlConservativeQLearningOptionsoptions object is used (if needed).

For more information, seerlConservativeQLearningOptions.

Example:opt.BatchDataRegularizerOptions = rlConservativeQLearningOptions(MinQValueWeight=5)

Smoothing factor for target critic updates, specified as a positive scalar less than or equal to 1. For more information, seeTarget Update Methods.

Number of steps between target critic updates, specified as a positive integer. For more information, seeTarget Update Methods.

Option for clearing the experience buffer before training, specified as a logical value.

Maximum batch-training trajectory length when using a recurrent neural network for the critic, specified as a positive integer. This value must be greater than1when using a recurrent neural network for the critic and1otherwise.

大小的随机mini-batch经验,指定为a positive integer. During each training episode, the agent randomly samples experiences from the experience buffer when computing gradients for updating the critic properties. Large mini-batches reduce the variance when computing gradients but increase the computational effort.

When using a recurrent neural network for the critic,MiniBatchSize是经验的数量在batc轨迹h, where each trajectory has length equal toSequenceLength.

Number of future rewards used to estimate the value of the policy, specified as a positive integer. For more information, see chapter 7 of [1].

N-step Q learning is not supported when using a recurrent neural network for the critic. In this case,NumStepsToLookAheadmust be1.

Experience buffer size, specified as a positive integer. During training, the agent computes updates using a mini-batch of experiences randomly sampled from the buffer.

Sample time of agent, specified as a positive scalar or as-1. Setting this parameter to-1allows for event-based simulations.

Within a Simulink®environment, theRL Agentblock in which the agent is specified to execute everySampleTimeseconds of simulation time. IfSampleTimeis-1, the block inherits the sample time from its parent subsystem.

Within a MATLAB®environment, the agent is executed every time the environment advances. In this case,SampleTimeis the time interval between consecutive elements in the output experience returned bysimortrain. IfSampleTimeis-1, the time interval between consecutive elements in the returned output experience reflects the timing of the event that triggers the agent execution.

Discount factor applied to future rewards during training, specified as a positive scalar less than or equal to 1.

Object Functions

rlDQNAgent Deep Q-network (DQN) reinforcement learning agent

Examples

collapse all

Open Live Script

Create anrlDQNAgentOptionsobject that specifies the agent mini-batch size.

opt = rlDQNAgentOptions(MiniBatchSize=48)
opt = rlDQNAgentOptions with properties: UseDoubleDQN: 1 EpsilonGreedyExploration: [1x1 rl.option.EpsilonGreedyExploration] CriticOptimizerOptions: [1x1 rl.option.rlOptimizerOptions] BatchDataRegularizerOptions: [] TargetSmoothFactor: 1.0000e-03 TargetUpdateFrequency: 1 ResetExperienceBufferBeforeTraining: 0 SequenceLength: 1 MiniBatchSize: 48 NumStepsToLookAhead: 1 ExperienceBufferLength: 10000 SampleTime: 1 DiscountFactor: 0.9900 InfoToSave: [1x1 struct]

You can modify options using dot notation. For example, set the agent sample time to0.5.

opt.SampleTime = 0.5;

References

[1] Sutton, Richard S., and Andrew G. Barto.Reinforcement Learning: An Introduction. Second edition. Adaptive Computation and Machine Learning. Cambridge, Mass: The MIT Press, 2018.

Version History

Introduced in R2019a

expand all

See Also

Objects

Topics

Baidu
map