[RLlib] Cleanup examples folder #13. Fix main examples docs page for RLlib. #45382

sven1977 · 2024-05-16T09:53:55Z

Cleanup examples folder #13. Fix main examples docs page for RLlib.

Why are these changes needed?

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: sven1977 <svenmika1977@gmail.com>

…rithm_config_dissolve_resources_method

Signed-off-by: sven1977 <svenmika1977@gmail.com>

…nup_examples_folder_11_fractional_gpus

…p_examples_folder_11_fractional_gpus

Signed-off-by: sven1977 <svenmika1977@gmail.com>

…p_examples_folder_11_fractional_gpus # Conflicts: # rllib/utils/test_utils.py

Signed-off-by: sven1977 <svenmika1977@gmail.com>

angelinalg

Just some style nits and a few typos.

doc/source/rllib/rllib-advanced-api.rst

angelinalg · 2024-05-20T20:29:48Z

rllib/utils/error.py

-    `num_gpus_per_worker` to 0 (they may be set to 1 by default for your
-    particular RL algorithm)."""
+    machine does not have any GPUs, you should set the config keys
+    `num_gpus_per_learner` and `num_gpus_per_env_runner` to 0 (they may be set to


Suggested change

`num_gpus_per_learner` and `num_gpus_per_env_runner` to 0 (they may be set to

`num_gpus_per_learner` and `num_gpus_per_env_runner` to 0. They may be set to

angelinalg · 2024-05-20T20:29:58Z

rllib/utils/error.py

-    particular RL algorithm)."""
+    machine does not have any GPUs, you should set the config keys
+    `num_gpus_per_learner` and `num_gpus_per_env_runner` to 0 (they may be set to
+    1 by default for your particular RL algorithm)."""


Suggested change

1 by default for your particular RL algorithm)."""

1 by default for your particular RL algorithm."""

angelinalg · 2024-05-20T20:30:14Z

rllib/utils/error.py

-    particular RL algorithm)."""
+    machine does not have any GPUs, you should set the config keys
+    `num_gpus_per_learner` and `num_gpus_per_env_runner` to 0 (they may be set to
+    1 by default for your particular RL algorithm)."""

 ERR_MSG_INVALID_ENV_DESCRIPTOR = """The env string you provided ('{}') is:
 a) Not a supported/installed environment.


Suggested change

a) Not a supported/installed environment.

a) Not a supported or installed environment.

angelinalg · 2024-05-20T20:30:34Z

rllib/utils/test_utils.py

@@ -1346,6 +1347,11 @@ def run_rllib_example_script_experiment(
        tune_callbacks: A list of Tune callbacks to configure with the tune.Tuner.
            In case `args.wandb_key` is provided, will append a WandB logger to this


Suggested change

In case `args.wandb_key` is provided, will append a WandB logger to this

In case `args.wandb_key` is provided, appends a WandB logger to this

angelinalg · 2024-05-20T20:30:51Z

rllib/utils/test_utils.py

+        keep_config: Set this to True, if you don't want this utility to change the
+            given `base_config` in any way and leave it as-is. This is helpful
+            for example script that want to demonstrate how to set those settings
+            that are usually taken care of automatically in this function (e.g.


Suggested change

that are usually taken care of automatically in this function (e.g.

that are usually taken care of automatically in this function (e.g.,

…nup_examples_folder_13_folder_readme Signed-off-by: sven1977 <svenmika1977@gmail.com> # Conflicts: # doc/source/rllib/rllib-advanced-api.rst # doc/source/rllib/rllib-learner.rst # rllib/BUILD # rllib/algorithms/algorithm.py # rllib/algorithms/algorithm_config.py # rllib/algorithms/dreamerv3/tests/test_dreamerv3.py # rllib/core/learner/learner.py # rllib/core/learner/scaling_config.py # rllib/examples/checkpoints/restore_1_of_n_agents_from_checkpoint.py # rllib/examples/gpus/fractional_gpus_per_learner.py # rllib/tuned_examples/dreamerv3/atari_100k.py # rllib/tuned_examples/dreamerv3/atari_200M.py # rllib/tuned_examples/dreamerv3/dm_control_suite_vision.py # rllib/utils/test_utils.py

Signed-off-by: sven1977 <svenmika1977@gmail.com>

…nup_examples_folder_13_folder_readme

Signed-off-by: sven1977 <svenmika1977@gmail.com>

…nup_examples_folder_13_folder_readme

Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Sven Mika <sven@anyscale.io>

Signed-off-by: sven1977 <svenmika1977@gmail.com>

…r_readme' into cleanup_examples_folder_13_folder_readme

…nup_examples_folder_13_folder_readme

Signed-off-by: sven1977 <svenmika1977@gmail.com>

…nup_examples_folder_13_folder_readme Signed-off-by: sven1977 <svenmika1977@gmail.com> # Conflicts: # rllib/examples/inference/policy_inference_after_training.py

Signed-off-by: sven1977 <svenmika1977@gmail.com>

simonsays1980

LGTM.

simonsays1980 · 2024-06-11T16:43:43Z

doc/source/rllib/key-concepts.rst

@@ -114,7 +114,7 @@ The following figure shows *synchronous sampling*, the simplest of `these patter

 RLlib uses `Ray actors <actors.html>`__ to scale training from a single core to many thousands of cores in a cluster.
 You can `configure the parallelism <rllib-training.html#specifying-resources>`__ used for training by changing the ``num_env_runners`` parameter.
-Check out our `scaling guide <rllib-training.html#scaling-guide>`__ for more details here.
+See this `scaling guide <rllib-training.html#scaling-guide>`__ for more details here.


The scaling guide also needs to be overhauled.

simonsays1980 · 2024-06-11T16:56:30Z

rllib/algorithms/algorithm_config.py

                CUDA devices. For example if `os.environ["CUDA_VISIBLE_DEVICES"] = "1"`
-                then a `local_gpu_idx` of 0 will use the GPU with ID=1 on the node.
+                and `local_gpu_idx=0`, RLlib uses the GPU with ID=1 on the node.


This feels counterintuitive. The GPU index 0 does not equal the environment variable 1 and we have two or more GPUs for a single learner. A user would expect a single GPU for a single learner when multiple GPUs are available on a node to be indicated with an ID or index. Do I misunderstand sth here?

simonsays1980 · 2024-06-11T16:57:23Z

rllib/algorithms/algorithm_config.py

-                `num_learners` x `train_batch_size_per_learner` and can
-                be accessed via the property `AlgorithmConfig.total_train_batch_size`.
+                `num_learners` x `train_batch_size_per_learner` and you can
+                access it with the property `AlgorithmConfig.total_train_batch_size`.


We should refer hereto in the scaling guide ~ if not done yet.

sven1977 added 13 commits May 16, 2024 04:33

wip

cb937c6

Signed-off-by: sven1977 <svenmika1977@gmail.com>

wip

c931bed

Signed-off-by: sven1977 <svenmika1977@gmail.com>

wip

02d3d04

Signed-off-by: sven1977 <svenmika1977@gmail.com>

wip

3ada50a

Signed-off-by: sven1977 <svenmika1977@gmail.com>

Merge branch 'master' of https://github.com/ray-project/ray into algo…

bccdde6

…rithm_config_dissolve_resources_method

wip

496c5ee

Signed-off-by: sven1977 <svenmika1977@gmail.com>

Merge branch 'master' of https://github.com/ray-project/ray into clea…

edd0a91

…nup_examples_folder_11_fractional_gpus

Merge branch 'algorithm_config_dissolve_resources_method' into cleanu…

4cdebb3

…p_examples_folder_11_fractional_gpus

wip

68aa7ba

Signed-off-by: sven1977 <svenmika1977@gmail.com>

wip

794b960

Signed-off-by: sven1977 <svenmika1977@gmail.com>

Merge branch 'algorithm_config_dissolve_resources_method' into cleanu…

3a6d05e

…p_examples_folder_11_fractional_gpus # Conflicts: # rllib/utils/test_utils.py

wip

03be7f5

Signed-off-by: sven1977 <svenmika1977@gmail.com>

wip

e6a8a2e

Signed-off-by: sven1977 <svenmika1977@gmail.com>

sven1977 requested review from avnishn, ArturNiederfahrenhorst, maxpumperla, kouroshHakha, simonsays1980 and a team as code owners May 16, 2024 09:53

sven1977 assigned simonsays1980 and angelinalg May 16, 2024

sven1977 added rllib RLlib related issues rllib-docs-or-examples Issues related to RLlib documentation or rllib/examples rllib-newstack rllib-oldstack-cleanup Issues related to cleaning up classes, utilities on the old API stack labels May 16, 2024

wip

d65082f

Signed-off-by: sven1977 <svenmika1977@gmail.com>

angelinalg approved these changes May 20, 2024

View reviewed changes

sven1977 added 3 commits June 3, 2024 21:23

wip

5d1fa21

Signed-off-by: sven1977 <svenmika1977@gmail.com>

Merge branch 'master' of https://github.com/ray-project/ray into clea…

5322241

…nup_examples_folder_13_folder_readme

sven1977 and others added 12 commits June 10, 2024 12:21

wip

d0995ae

Signed-off-by: sven1977 <svenmika1977@gmail.com>

wip

0916ce7

Signed-off-by: sven1977 <svenmika1977@gmail.com>

Merge branch 'master' of https://github.com/ray-project/ray into clea…

f36f7dc

…nup_examples_folder_13_folder_readme

Apply suggestions from code review

23fcf8d

Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Sven Mika <sven@anyscale.io>

Apply suggestions from code review

8e2afcc

Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Sven Mika <sven@anyscale.io>

Apply suggestions from code review

8966d52

Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Sven Mika <sven@anyscale.io>

fix

cbc7f5b

Signed-off-by: sven1977 <svenmika1977@gmail.com>

Merge remote-tracking branch 'origin/cleanup_examples_folder_13_folde…

9d509c3

…r_readme' into cleanup_examples_folder_13_folder_readme

Merge branch 'master' of https://github.com/ray-project/ray into clea…

11b207e

…nup_examples_folder_13_folder_readme

fix

ed21e41

Signed-off-by: sven1977 <svenmika1977@gmail.com>

fix

3ea64bf

Signed-off-by: sven1977 <svenmika1977@gmail.com>

fix

f192bc3

Signed-off-by: sven1977 <svenmika1977@gmail.com>

sven1977 changed the title ~~[RLlib] Cleanup examples folder 13: Add READMEs to folder and all sub-folders.~~ [RLlib] Cleanup examples folder #13. Fix main examples docs page for RLlib. Jun 11, 2024

sven1977 added 3 commits June 11, 2024 14:15

fix

2dbe142

Signed-off-by: sven1977 <svenmika1977@gmail.com>

Merge branch 'master' of https://github.com/ray-project/ray into clea…

73820dc

…nup_examples_folder_13_folder_readme Signed-off-by: sven1977 <svenmika1977@gmail.com> # Conflicts: # rllib/examples/inference/policy_inference_after_training.py

fix

5172242

Signed-off-by: sven1977 <svenmika1977@gmail.com>

sven1977 enabled auto-merge (squash) June 11, 2024 13:52

github-actions bot added the go Trigger full test run on premerge label Jun 11, 2024

simonsays1980 approved these changes Jun 11, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RLlib] Cleanup examples folder #13. Fix main examples docs page for RLlib. #45382

[RLlib] Cleanup examples folder #13. Fix main examples docs page for RLlib. #45382

sven1977 commented May 16, 2024 •

edited

angelinalg left a comment

angelinalg May 20, 2024

angelinalg May 20, 2024

angelinalg May 20, 2024

angelinalg May 20, 2024

angelinalg May 20, 2024

simonsays1980 left a comment

simonsays1980 Jun 11, 2024

simonsays1980 Jun 11, 2024

simonsays1980 Jun 11, 2024

	`num_gpus_per_learner` and `num_gpus_per_env_runner` to 0 (they may be set to
	`num_gpus_per_learner` and `num_gpus_per_env_runner` to 0. They may be set to

	1 by default for your particular RL algorithm)."""
	1 by default for your particular RL algorithm."""

	a) Not a supported/installed environment.
	a) Not a supported or installed environment.

		@@ -1346,6 +1347,11 @@ def run_rllib_example_script_experiment(
		tune_callbacks: A list of Tune callbacks to configure with the tune.Tuner.
		In case `args.wandb_key` is provided, will append a WandB logger to this

	In case `args.wandb_key` is provided, will append a WandB logger to this
	In case `args.wandb_key` is provided, appends a WandB logger to this

	that are usually taken care of automatically in this function (e.g.
	that are usually taken care of automatically in this function (e.g.,

[RLlib] Cleanup examples folder #13. Fix main examples docs page for RLlib. #45382

Are you sure you want to change the base?

[RLlib] Cleanup examples folder #13. Fix main examples docs page for RLlib. #45382

Conversation

sven1977 commented May 16, 2024 • edited

Why are these changes needed?

Related issue number

Checks

angelinalg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simonsays1980 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sven1977 commented May 16, 2024 •

edited