This PR reworks and largely fixes long-standing issue or limitations with the Navigation RVO Avoidance.
Started as an implementation of this proposal and escalated ... ups!

While build on top of two "official" RVO libraries for 2D and 3D there are a lot of modifications and custom features included to make avoidance more useful in Godot by covering more gameplay situations.

What is new?

• Adds RVO-2D avoidance and reworks existing RVO-3D avoidance.
• Adds static avoidance obstacles (2D avoidance).
• Adds avoidance mask and layers, and avoidance priority.
• Adds "layered" 2D avoidance for 3D with agent height.
• Adds avoidance debug visuals for static obstacles and avoidance radius.
• Adds more graceful "recovery" if avoidance simulation is broken
• Adds automatic avoidance agent constrain to navigation mesh for NavigationRegion2D (experimental)
• Adds script templates for common navigationagent types that extend Node/CharacterBody/Rigidbody 2D/3D (removed)
• Tons of smaller or larger fixes and changes.

Fixed Issues

• Fixes NavigationAgent2D does not apply radius settings around NavigationPolygonInstance in 3.5 beta 4 #60546 NavigationAgent2D does not apply radius settings around NavigationPolygonInstance in 3.5 beta 4 (for original issue if using avoidance and constrains, does nothing for the topic derailed discussion about 2D navmesh baking)
• Fixes NavigationAgent doesn't seem to take radius into account until some agent gets freed #62469 NavigationAgent doesn't seem to take radius into account until some agent gets freed (fixed by simulation recovery and general RVO rework)
• Fixes Using NavigationObstacle with StaticBody produces a warning #66932 Using NavigationObstacle with StaticBody produces a warning (obstacles are now self-sufficient and no-longer depend on a parent)
• Fixes Strange behaviour of NavigationAgent3D around NavigationObstacle3D #67712 Strange behaviour of NavigationAgent3D around NavigationObstacle3D (demo situation now works fair enough with 2D avoidance and some adjusted parameters)
• Fixes Collision avoidance doesn't work occasionally #72202 Collision avoidance doesn't work occasionally (confirmed by OP)

While there were many smaller issues one of the major causes of bugs in the old Godot avoidance was that Godot was updating many internal RVO values every single frame, something explicitly discouraged by the RVO library devs and even part of their manual. Many other engines solve this by locking or queuing avoidance user input for a time period which is rather awkward to use and leads to a sluggish on-rails avoidance movement. As a compromise Godot still allows most changes every frame but now by default protects some internal RVO values to keep some consistency.

Implemented proposals

Closes godotengine/godot-proposals#4522 Add RVO NavigationMesh Obstacle Planes for NavigationAgent2D/3D Avoidance (through static obstacles)
Closes godotengine/godot-proposals#1966 Improve Navigation with multiple agents (through static obstacles and avoidance layers / mask / priority)
Closes godotengine/godot-proposals#3812 Add the ability to select an obstacle or an agent as a target (through avoidance layers / mask / priority)

New RVO-2D avoidance

The RVO-2D library is optimizes for avoidance on a 2D plane and comes with static obstacle support defined by vertices.
2D avoidance does not mean it is limited to 2D games, it only means it is limited to movement in 2D space along the xy-axis, or in case of 3D games, the xz-axis. 2D avoidance is the preferred avoidance mode for any game type with normal ground movement on mostly flat surfaces for both performance and quality reasons.

Reworked RVO-3D avoidance

The existing RVO lib that Godot uses now restored to the original purpose of avoidance in 3D space along the xyz-axis. 3D avoidance supports (but also expects) movement in all xyz-axis for e.g. 3D space or underwater games. The old 3D implementation attempted to create a 2D avoidance by disabling the y-axis with the "ignore_y" property which caused all kinds of velocity bugs. This was replaced with a "use_3d_avoidance" property that switches the agent / obstacle between the the real 2D and 3D avoidance (on the next physics frame).

New static avoidance obstacles

Available for the 2D avoidance. Compared to the old obstacles that used only a radius the new obstacles can be defined by vertices to create a hard do-not-cross obstacle edge. The vertices winding order decides if the agents are pushed out or in by the obstacle. Static obstacles can be used to reliably constrain agents even in cramped situations, something that the old "soft" obstacles (that still exist) never could. The static avoidance obstacles of the default RVO2-2D library are not intended to be moved or changed after simulation start. Obviously this limitation did not work for Godot so they can be warped / rebuild when changes are made but this has a cost and should not be done every frame for complex obstacles.

If you need a moving and predictable obstacle use the obstacle radius and set the obstacle velocity.
If you want a reliable way to constrain agents with a static polygon shape use the obstacle vertices.

New avoidance mask and layers, and avoidance priority

Similar to physics, navigation or visuals, the avoidance has now a layers bitmask system. An agent with a matching mask bit will avoid all agents or obstacles with a matching layer bit. On top there is now a avoidance priority value on a range from 0.0 - 1.0. Agents with higher priority will act as bullies and ignore agents with lower priority even when their bitmasks match. This leads to the low priority agents making room for the high priority agents whenever possible.

New "layered" 2D avoidance for 3D with agent height

2D avoidance considers the new agent height property and y-axis position in 3D and is also height "layered", e.g. think of a 3D building with different floors. Previously agents would avoid each other through floors in 3D due to their often required large avoidance radius spheres. Now if an agent or obstacle position + height is below or above an agent's position + height the agent will ignore the other avoidance object automatically even if their avoidance would otherwise overlap. This also makes it possible to stack multiple 2D avoidance in 3D on different elevation levels without interfering.

New avoidance debug visuals

All the previously invisible properties like agent or obstacle radius now have visuals in the Editor or when a game is run with "Visible Avoidance" from the Editor Debug menu enabled. The colors of the visuals can be changed in ProjectSettings and individual debugs can be enabled / disabled (may require a Editor / Scene restart).

New graceful "recovery" if avoidance simulation is broken

Previously if users created unsolvable situations avoidance agents would get stuck and jitter in place cause Godot was resetting the simulation every frame. This was less noticeable in isolated test demos with only a few agents but in the wilds it was a regular issue in cramped situations with many agents when users forced conflicting positions or velocities updates every frame on the agents. The jitter and stuck agents made it very, very obvious for common users that something broke in the game. This made it very hard to ship a game that uses avoidance as such situations can sometimes happen due to bad luck and timing. While this broken situations can still be created now users no longer replace by default the internal velocities. This gives the simulation the ability to recover more graceful by moving all agents to new valid simulation positions while still causing no collision. In order to accomplish this the avoidance "cheats" a little on the velocities and makes agents move slightly faster until they reached their new and valid simulation position. While not perfect this makes a broken avoidance simulation far less noticeable to the untrained eye, some would even think it is part of an intended "catch-up" feature. There is no agent limit, it works with 2 agents that fight for the same position or 20000.

If you teleport agents to a new position use the agent_set_velocity_forced() functions on the same frame to reset the internal velocities. If you don't do that the agent will have lightning speed on the next frame. If your project requires the old, bug-ridden avoidance you can use agent_set_velocity_forced() function every frame to simulate the old avoidance behavior.

New automatic avoidance agent constrain to navmesh for NavigationRegion2D (experimental)

Automatically creates avoidance obstacles around the NavigationPolygon used by a NavigationRegion2D to constrain avoidance using agents, including one level of polygon holes. This is an experimental feature with known limitations as it shares the same issues why 2D has currently no navmesh baking which is a lack of good, automatic margin generation for the polygons. Since the navmesh edges and the created avoidance obstacles currently overlap agents can get often stuck when pushed on top the edge e.g. by physics or bad velocities.

Known Issues / Performance

NavigationAgent pathfollowing with avoidance
Most visible avoidance related bugs are now the result of the NavigationAgent movement logic that requires a far larger rework. E.g. current NavigationAgent movement logic continuously resets paths when to far away from the next path point. This can lead to agents getting stuck when surrounded by avoidance constrains. It can also get agents stuck in an endless loop when the target position is occupied by another avoidance agent or obstacle that the pathfinding is not aware of. Also current NavigationAgent movement logic can create situations with conflicting velocity interests that can get agents also stuck on edges when navigation mesh edges and avoidance obstacle edges overlap.

Static obstacles update performance
The avoidance world is currently rebuilding all static obstacles when a single static obstacle is changed cause each obstacle holds ref to some of it's neighbors which can be costly at runtime. The original RVO-2D library did not support editing static obstacles at all at runtime so this is an evolving custom feature with a lot of room to optimize in a follow-up.

Conversions between Godot and RVO
There is currently a lot of back and forth conversions and copy between Godot and the RVO libraries, e.g. stuffing everything into std::vectors or converting Godot Vector2/3 to RVO vectors back and forth. If the core avoidance functions from the libraries would be extracted and integrated directly in Godot to work with native Godot types performance could possibly be improved considerably in a follow-up.

TO DO

• EditorPlugins are wip and a last second addition. They are functional to draw obstacle vertices but other parts like proper handle or vertex edit are unfinished. There is for now some intentional leftover code from copy&paste from the CollisionPolygon3D plugin.
• Correctly credit the third-party libs and create working "patches" (no idea really how).
• Probably forgot to mention half the new / changed stuff.
• Book a long holiday after this ....

Test Project

This project includes some of the more salvageable pieces and 3D demo scenes.
RVOTestProject4.zip