Robot Learning · Control · Simulation

Reinforcement Learning for Robotics

Risk-Sensitive Learning · Demonstration-Guided Policies · Physics-Based Simulation

Clinton Enwerem · University of Maryland, College Park

I develop risk-sensitive reinforcement learning algorithms and build simulation pipelines for training, integrating, and evaluating robotic policies. My work includes published distributional RL, a reproduced demonstration-guided dexterous learning baseline, and residual RL integrated with bimanual assembly control.

Learning Baseline Reproduction · MuJoCo / Adroit

From Demonstrations to Learned Dexterous Relocation

I ported the authors’ Demonstration-Augmented Policy Gradient (DAPG) [1] implementation to a maintained Adroit simulation environment, trained a relocation policy from 25 demonstrations, and evaluated the initial and trained policies on held-out reset seeds. Behavior cloning initialized the policy; subsequent DAPG training produced successful object relocation.

Behavior Cloning versus Trained DAPG. The blue object must be relocated to the green target. Playback: 0.5×.

95 / 100DAPG · Mean-Action Successes

98 / 100DAPG · Stochastic Successes

0 / 100BC · Each Evaluation Mode

Training Configuration

Demonstrations	25 author-provided demonstrations; 9,942 samples
Training	5 BC epochs, then 100 DAPG iterations; 4 million control steps
Algorithm	Authors’ BC/DAPG implementation and example configuration
Runtime	Gymnasium Robotics 1.2.0; MuJoCo 2.3.7
Final Evaluation	Reset seeds 40000–40099; 200 steps per episode; sustained successful-step criterion

Paper [1] · Authors’ Implementation

Prior-Data RL Adaptation · Drake Simulation

Integrating Residual RL with Bimanual Assembly Control

I adapted Reinforcement Learning with Prior Data (RLPD) [2] to an A7 bimanual arm–hand system, combining recorded native-controller transitions with online residual-policy interaction. A bounded joint-reference residual acts through the existing feedback controller while the robot stabilizes a housing, acquires and transfers a part, inserts it, and withdraws its hands.

Bimanual Assembly with Residual RL. The learned policy adjusts joint references while the feedback controller executes the assembly sequence. Playback: 1×.

12 / 12Initial Policy · Completed Assemblies

12 / 12Updated Policy · Completed Assemblies

The residual policy adjusts seven arm joint references and learns from prior demonstrations and online interaction. Training used 6,300 prior transitions and 6,300 online decisions, with 100 actor updates and 2,000 critic updates.

Native Prior + Online TransitionsRLPD LearnerBounded Residual + Native Feedback ControlAssembly Simulation

Assembly Results by Friction

Friction	Native	Zero Residual	Initial Policy	Updated Policy
0.70	1/1	1/1	4/4	4/4
0.65	1/1	1/1	4/4	4/4
0.60	1/1	1/1	4/4	4/4

Paper [2] · Authors’ Learner

Published Research · IEEE CDC 2025

Risk-Sensitive Distributional Reinforcement Learning

For reach–avoid navigation, I developed quantile-based action-value iteration with a Conditional Value-at-Risk (CVaR) penalty estimated from the distribution of constraint-violation costs [3]. I integrated the risk-sensitive loss with quantile value estimation and evaluated the learned navigation policies in MuJoCo / Safety-Gymnasium. The paper includes a contraction proof for the action-value update.

Goal Attainment and Violation Costs

100 episodes across five evaluation seeds.

Total goals reached and mean violation cost over 100 episodes.
Algorithm	Goals Reached	Violation Cost (Mean ± SD)
QR-AVI	239	0.05 ± 0.01
Expected-Value QR-AVI	329	0.06 ± 0.01
CVaR QR-AVI, β = 0.90	448	0.05 ± 0.01
CVaR QR-AVI, β = 0.95	448	0.05 ± 0.01

Paper [3]

Algorithm and Implementation

The learner represents action-value distributions with quantile regression. Kernel density estimation provides a distribution of constraint-violation costs, from which CVaR measures adverse-tail risk. A risk penalty augments the quantile loss before the action-value network update.

Experience BufferQuantile Value Estimation + Cost DistributionCVaR-Regularized UpdateNavigation Evaluation

Reward Design · Isaac Lab / PhysX

G1 PPO Reward Evaluation

I trained policies in 1,024 parallel Isaac Lab environments to investigate geometric pregrasp rewards.

Pregrasp Reward Comparison

For a fixed-base Unitree G1 equipped with a RealHand L6, the Grasp Distance Field reward measures proximity to candidate arm–hand configurations. PPO training compares this reward with a palm-to-object distance reward.

Robot StateArm–hand configuration and object state

Geometric RewardDistance to candidate pregrasp configurations

PPO TrainingPolicy updates from state, action, and reward samples

Evaluation Results

Seed-1 checkpoints: 64 episodes per policy.
Reward Configuration	Pregrasp Arrivals	Lifts
GDF (Full)	64 / 64	0 / 64
Palm Distance (Gated)	0 / 64	0 / 64

An arrival occurs when the pregrasp field value falls below 0.25 during an episode. The field measures a smooth minimum of weighted joint-space distances.

Pregrasp field value over simulation time for the two recorded policy rollouts, with the arrival threshold at 0.25. — **Pregrasp Reward Comparison.** Distance to the candidate pregrasp set during policy execution.

Grasp Distance Fields

References

Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine.
Robotics: Science and Systems (RSS), 2018.
arXiv:1709.10087 · Authors’ Implementation
Efficient Online Reinforcement Learning with Offline Data.
Philip J. Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine.
International Conference on Machine Learning (ICML), 2023.
arXiv:2302.02948 · Authors’ Learner
Safety-Aware Reinforcement Learning for Control via Risk-Sensitive Action-Value Iteration and Quantile Regression.
Clinton Enwerem, Aniruddh G. Puranic, John S. Baras, and Calin Belta.
2025 IEEE 64th Conference on Decision and Control, pp. 4890–4895.
DOI: 10.1109/CDC57313.2025.11312575 · arXiv:2506.06954

← Research Overview