Out of memory errors and other similar errors are due to the server basically running out of memory also known as RAM. The cause is a bit more difficult to determine.
Note: Every Minecraft package has a set amount of RAM.
The use of plugins will use up this RAM, and each user in the server will take up a chunk of that RAM as well. Thus, the more plugins the server has, the fewer players can be online at the same time. In this case, lower the max player slots of the server.
Using multi-world plugins is notorious for causing out of memory crashes because each world added uses up quite a lot of memory, even if no players are on that world.
Other causes of out of memory errors can be due to out-of-date plugins or incompatible plugins with the version the server uses. It is always best to ensure that everything on the server is up-to-date to reduce the chances of this occurring.
Note: A 100% stable server is nearly impossible to have due to the nature of Minecraft/CraftBukkit servers and plugins.
It can also be caused by things in the world such as mobs, lava, water, redstone, pistons, entities, etc., as these can cause memory leaks and have to be removed manually.