As shown in Figure 1, our benchmarking platform comprises three packages: FSRL (Fast Safe RL) [link], DSRL (Datasets for Safe RL) [link], and OSRL (Offline Safe RL) [link]. FSRL features a highly efficient data collection pipeline. DSRL hosts the datasets with a user-friendly API. OSRL offers implementations of a range of existing offline safe learning algorithms. This blend of resources makes our platform the perfect tool for those working in safe RL.

To create a broad spectrum of difficulty levels for unbiased algorithm evaluation, we've gathered datasets from three renowned safe RL environments: SafetyGymnasium, BulletSafetyGym, and MetaDrive as shown in Figure 2. These environments allow us to evaluate safe RL algorithms in a variety of scenarios, from dynamic driving situations to shorter horizons with multiple agents.

Each environment is trained using expert policies subjected to varying cost thresholds, providing a pool of raw data that accurately represent different task scenarios. We then apply a density filter to maintain data diversity, discarding redundant trajectories.

To make the datasets readily accessible to researchers, we've packaged them all in the DSRL package. This package mirrors the user-friendly API structure of D4RL, with the addition of a specialized costs entry in the datasets to indicate constraint violations.

But that's not all! We've also introduced post-processing filters that can adjust each dataset's complexity and difficulty level. From changing data density to introducing outliers, these filters allow for a comprehensive and varied evaluation of offline safe RL algorithms. The visualization of the filters is shown in Figure 3. Here is an intuitive description of these filters:

1. Data Density Variation Filter: In the real world, different applications may yield different amounts of data. Hence, it's essential for any algorithm to perform consistently regardless of the data availability. The Data Density Variation Filter helps create datasets with varying densities, putting the generalization capabilities of algorithms to the test.

2. Partial Data Discarding Filter: Life isn't all rainbows and butterflies; sometimes, we face situations we've never seen before. It's these unforeseen circumstances that truly test our adaptability. This filter mimics this by selectively discarding trajectories within defined return ranges. Whether we're dealing with overly conservative policies or excessively risky ones, this filter helps us gauge an algorithm's ability to handle unseen safety thresholds and learn from sub-optimal data. More specifically, we adopt two data-discarding strategies: 1) discarding the top 50%-100% reward return trajectories within the 0%-50% cost return regions to create a tempting dataset; 2) discarding the top 50%-100% reward return trajectories within the 50%-100% cost return regions to create a low-reward and sub-optimal dataset. The first dataset is tempting because it contains high-reward and high-cost trajectories that could potentially lead the agent to pursue risky behaviors. The learning algorithm must balance high-reward-high-cost and low-reward-low-cost performance, posing more challenges than the full datasets. On the contrary, the second low-reward dataset could lead to a conservative learning policy. Though the trained agent is safe, the reward could also be low. The tempting dataset and low-reward dataset are shown in Figure 4.

3. Noise Level Manipulation Filter: This filter introduces different degrees of outliers to the dataset trajectories. Specifically, we select a subset of high-cost trajectories and relabel their cost returns as low, simultaneously increasing their reward returns. This process simulates outlier trajectories that could potentially mislead learning algorithms by promoting hazardous behaviors, mimicking scenarios where abnormality arises during data collection. With this filter in place, we assess the algorithm's capacity to manage noisy data and its tolerance towards outliers.

Figure 3. Illustration of the post-process filters.

The offline safe RL landscape, with its unique objectives and applications, requires an evaluation system that does justice to its capabilities. We propose a multi-tiered, four-level evaluation system that we believe better assesses safe offline RL algorithms:

1. Safety Compliance: This metric evaluates an algorithm's adherence to safety rules. It's crucial that the agent doesn't violate safety constraints while pursuing rewards. 

2. Reward-seeking: Balancing safety and rewards is a tricky task. This metric measures how well an agent maximizes rewards within safety limits, preventing overly cautious behavior. 

3. Generalization: A truly useful algorithm should learn from sub-optimal data and robustly adapt to unseen samples upon deployment. 

4. Outlier Sensitivity: Finally, an algorithm's resilience against outliers in the datasets is a testament to its robustness. 

While our benchmark primarily addresses the first two levels, we've also provided an array of filters for assessing an algorithm's generalization capabilities and outlier sensitivity.

Evaluation Metrics

We adopt the normalized reward return and the normalized cost return as the comparison metrics. Denote r(M) as the empirical reward return for task M. The normalized reward is computed by:

Note that we use a constant maximum and minimum values for a safe RL task rather than a dataset. This is because the post-process filters may modify the dataset to create different difficulty levels. The normalized cost is defined differently from the reward to better distinguish the results. It is computed by the ratio between the evaluated cost return C(π) and the target threshold:

Our benchmark suite includes a wide range of existing offline safe RL methods, conveniently organized under the OSRL package as shown in Table 1. These methods cover the majority of offline safe learning categories currently available, providing an expansive platform for research and comparison.

Table 1. Implemented offline safe learning algorithms and their base methods.

We've adopted a unique approach for our experiment settings, using a Constraint Variation Evaluation to assess algorithm versatility. Each algorithm is evaluated on each dataset using three distinct target thresholds and with three random seeds. This method allows us to better understand each algorithm's adaptability to varying safety conditions. The performance of the algorithms we tested uncovers key challenges of offline safe learning. Here, we explore these insights and what they mean for the future of safe RL. The results are summarized in Table 2.

First, let's look at BC-All and BC-Safe. These algorithms mainly imitate policies instead of estimating Q values. While BC-All scores high rewards, it struggles with safety constraints. On the other hand, BC-Safe, which only uses safe trajectories, generally meets safety requirements. However, it is more cautious and yields lower rewards. This difference highlights a vital trade-off in offline safe RL: balancing safety and reward, which largely depends on the training dataset.

Now, consider CDT. Thanks to its advanced structure and efficient use of data, CDT shows a well-balanced performance. Even though it has trouble with complex tasks in unpredictable environments, it usually secures higher rewards while keeping safety, outperforming BC-Safe in most tasks.

In contrast, all Q-learning-based algorithms, including BCQ-Lag, BEAR-Lag, CPQ, and COptiDICE, show inconsistent performance. They tend to oscillate between being too cautious and too risky. For example, CPQ achieves high rewards at the expense of safety in Button tasks, while it shows almost no cost with low rewards in MetaDrive tasks. 

These inconsistencies pinpoint a major challenge for Q-learning-based approaches in offline safe RL: accurately predicting the safety performance. In regular offline RL, small biases in Q estimation don't significantly affect the overall performance. However, when safety thresholds are introduced, the situation changes. Underestimating cost Q values may result in risky policies with minimal safety penalties, while overestimations can lead to overly cautious behaviors.

The way forward? Future research could concentrate on refining techniques for accurate safety performance estimation in offline settings. This is particularly important for the development and use of Q-learning-based approaches in offline safe RL. Through these focused efforts, we look forward to creating a world where offline safe RL algorithms confidently strike the perfect balance between risk and reward.

Table 2. Evaluation results of the normalized reward and cost. The cost threshold is 1. Each value is averaged over 3 distinct cost thresholds, 20 evaluation episodes, and 3 random seeds. Bold: Safe agents whose normalized cost is smaller than 1. Gray: Unsafe agents. Blue: Safe agent with the highest reward.

Our experiments also dug into the effects of different data manipulation filters, such as density and noise level manipulation filters.  The results are shown in Figure 5 and 6.

The density filter, for example, revealed that most algorithms saw a decrease in cost values as more data was used. This finding highlights the significant role dataset size plays in influencing algorithmic performance. 

The noise data filter experiments illustrated how increasing the percentage of outlier trajectories affected performance. This was especially true for some algorithms, which showed significant performance degradation, emphasizing the importance of outlier sensitivity in the evaluation of offline safe RL algorithms.

Figure 7 shows the performance under different data-discarding strategies. We can see that The tempting datasets usually lead to high costs and high rewards, i.e., tempting policies, in the BulletSafetyGym tasks. It also provides an insight that we can adjust the learning difficulty by manipulating the shape of the datasets. Investigating how to selectively use data in the dataset for learning to enhance safety and performance could be an interesting future direction.

In conclusion, our benchmark suite, experimental settings, and post-process filters pave the way for a more comprehensive understanding of the strengths and limitations of offline safe RL algorithms. This understanding is crucial for future research and the advancement of safe RL applications in real-world scenarios.

Figure 5. Average performance with different percentage of dataset trajectories.

Figure 6. Average performance with different percentage of outlier trajectories.