Bug Closed · Resolved

Memory leak/crash?

#4399 opened by Unknown 2 years ago

Upvote 0
Unknownreported this bug

Never happened before, after updating to MEG 4 and running online for a few days the server crashed.

Server: purpur-1.20.1.jar:git-Purpur-2050

[17:38:16] [Craft Async Scheduler Management Thread/ERROR]: Caught previously unhandled exception :
[17:38:16] [Craft Async Scheduler Management Thread/ERROR]: Craft Async Scheduler Management Thread
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
        at java.lang.Thread.start0(Native Method) ~[?:?]
        at java.lang.Thread.start(Thread.java:1560) ~[?:?]
        at java.lang.System$2.start(System.java:2528) ~[?:?]
        at jdk.internal.vm.SharedThreadContainer.start(SharedThreadContainer.java:160) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:953) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1021) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1158) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1589) ~[?:?]
[17:38:16] [Server thread/ERROR]: Encountered an unexpected exception
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
        at java.lang.Thread.start0(Native Method) ~[?:?]
        at java.lang.Thread.start(Thread.java:1560) ~[?:?]
        at java.lang.System$2.start(System.java:2528) ~[?:?]
        at jdk.internal.vm.SharedThreadContainer.start(SharedThreadContainer.java:160) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:953) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364) ~[?:?]
        at java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:759) ~[?:?]
        at org.bukkit.craftbukkit.v1_20_R1.scheduler.CraftAsyncScheduler.mainThreadHeartbeat(CraftAsyncScheduler.java:7$
        at org.bukkit.craftbukkit.v1_20_R1.scheduler.CraftScheduler.mainThreadHeartbeat(CraftScheduler.java:462) ~[purp$
        at net.minecraft.server.MinecraftServer.tickChildren(MinecraftServer.java:1506) ~[purpur-1.20.1.jar:git-Purpur-$
        at net.minecraft.server.dedicated.DedicatedServer.tickChildren(DedicatedServer.java:486) ~[purpur-1.20.1.jar:gi$
        at net.minecraft.server.MinecraftServer.tickServer(MinecraftServer.java:1420) ~[purpur-1.20.1.jar:git-Purpur-20$
        at net.minecraft.server.MinecraftServer.runServer(MinecraftServer.java:1191) ~[purpur-1.20.1.jar:git-Purpur-205$
        at net.minecraft.server.MinecraftServer.lambda$spin$0(MinecraftServer.java:322) ~[purpur-1.20.1.jar:git-Purpur-$
        at java.lang.Thread.run(Thread.java:1589) ~[?:?]
[17:38:17] [Server thread/WARN]: Failed retrieving info for group processor
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
        at java.lang.Thread.start0(Native Method) ~[?:?]
        at java.lang.Thread.start(Thread.java:1560) ~[?:?]
        at java.lang.System$2.start(System.java:2528) ~[?:?]
        at jdk.internal.vm.SharedThreadContainer.start(SharedThreadContainer.java:160) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:953) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1375) ~[?:?]
        at java.lang.ProcessHandleImpl.completion(ProcessHandleImpl.java:140) ~[?:?]
        at java.lang.ProcessImpl.initStreams(ProcessImpl.java:368) ~[?:?]
        at java.lang.ProcessImpl.lambda$new$0(ProcessImpl.java:331) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
        at java.lang.ProcessImpl.<init>(ProcessImpl.java:330) ~[?:?]
        at java.lang.ProcessImpl.start(ProcessImpl.java:249) ~[?:?]
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1110) ~[?:?]
        at java.lang.Thread.start(Thread.java:1560) ~[?:?]
        at java.lang.System$2.start(System.java:2528) ~[?:?]
        at jdk.internal.vm.SharedThreadContainer.start(SharedThreadContainer.java:160) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:953) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1375) ~[?:?]
        at java.lang.ProcessHandleImpl.completion(ProcessHandleImpl.java:140) ~[?:?]
        at java.lang.ProcessImpl.initStreams(ProcessImpl.java:368) ~[?:?]
        at java.lang.ProcessImpl.lambda$new$0(ProcessImpl.java:331) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
        at java.lang.ProcessImpl.<init>(ProcessImpl.java:330) ~[?:?]
        at java.lang.ProcessImpl.start(ProcessImpl.java:249) ~[?:?]
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1110) ~[?:?]
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1073) ~[?:?]
        at java.lang.Runtime.exec(Runtime.java:615) ~[?:?]
        at java.lang.Runtime.exec(Runtime.java:517) ~[?:?]
        at oshi.util.ExecutingCommand.runNative(ExecutingCommand.java:118) ~[oshi-core-6.2.2.jar:6.2.2]
        at oshi.util.ExecutingCommand.runNative(ExecutingCommand.java:95) ~[oshi-core-6.2.2.jar:6.2.2]
        at oshi.util.ExecutingCommand.runNative(ExecutingCommand.java:78) ~[oshi-core-6.2.2.jar:6.2.2]
        at oshi.driver.linux.Lshw.queryCpuCapacity(Lshw.java:91) ~[oshi-core-6.2.2.jar:6.2.2]
        at oshi.hardware.platform.linux.LinuxCentralProcessor.queryProcessorId(LinuxCentralProcessor.java:148) ~[oshi-c$
        at oshi.util.Memoizer$1.get(Memoizer.java:87) ~[oshi-core-6.2.2.jar:6.2.2]
        at oshi.hardware.common.AbstractCentralProcessor.getProcessorIdentifier(AbstractCentralProcessor.java:123) ~[os$
        at net.minecraft.SystemReport.putProcessor(SystemReport.java:149) ~[purpur-1.20.1.jar:git-Purpur-2050]
        at net.minecraft.SystemReport.lambda$putHardware$4(SystemReport.java:72) ~[purpur-1.20.1.jar:git-Purpur-2050]
        at net.minecraft.SystemReport.ignoreErrors(SystemReport.java:85) ~[purpur-1.20.1.jar:git-Purpur-2050]
        at java.lang.Runtime.exec(Runtime.java:615) ~[?:?]
        at java.lang.Runtime.exec(Runtime.java:517) ~[?:?]
        at oshi.util.ExecutingCommand.runNative(ExecutingCommand.java:118) ~[oshi-core-6.2.2.jar:6.2.2]
        at oshi.util.ExecutingCommand.runNative(ExecutingCommand.java:95) ~[oshi-core-6.2.2.jar:6.2.2]
        at oshi.util.ExecutingCommand.runNative(ExecutingCommand.java:78) ~[oshi-core-6.2.2.jar:6.2.2]
        at oshi.driver.linux.Lshw.queryCpuCapacity(Lshw.java:91) ~[oshi-core-6.2.2.jar:6.2.2]
        at oshi.hardware.platform.linux.LinuxCentralProcessor.queryProcessorId(LinuxCentralProcessor.java:148) ~[oshi-c$
        at oshi.util.Memoizer$1.get(Memoizer.java:87) ~[oshi-core-6.2.2.jar:6.2.2]
        at oshi.hardware.common.AbstractCentralProcessor.getProcessorIdentifier(AbstractCentralProcessor.java:123) ~[os$
        at net.minecraft.SystemReport.putProcessor(SystemReport.java:149) ~[purpur-1.20.1.jar:git-Purpur-2050]
        at net.minecraft.SystemReport.lambda$putHardware$4(SystemReport.java:72) ~[purpur-1.20.1.jar:git-Purpur-2050]
        at net.minecraft.SystemReport.ignoreErrors(SystemReport.java:85) ~[purpur-1.20.1.jar:git-Purpur-2050]
        at net.minecraft.SystemReport.lambda$putHardware$4(SystemReport.java:72) ~[purpur-1.20.1.jar:git-Purpur-2050]
        at net.minecraft.SystemReport.lambda$new$2(SystemReport.java:47) ~[purpur-1.20.1.jar:git-Purpur-2050]
        at net.minecraft.SystemReport.ignoreErrors(SystemReport.java:85) ~[purpur-1.20.1.jar:git-Purpur-2050]
        at net.minecraft.SystemReport.lambda$new$2(SystemReport.java:47) ~[purpur-1.20.1.jar:git-Purpur-2050]
        at net.minecraft.CrashReport.<init>(CrashReport.java:32) ~[purpur-1.20.1.jar:git-Purpur-2050]
        at net.minecraft.server.MinecraftServer.constructOrExtractCrashReport(MinecraftServer.java:1277) ~[purpur-1.20.$
        at net.minecraft.server.MinecraftServer.runServer(MinecraftServer.java:1221) ~[purpur-1.20.1.jar:git-Purpur-205$
        at net.minecraft.server.MinecraftServer.lambda$spin$0(MinecraftServer.java:322) ~[purpur-1.20.1.jar:git-Purpur-$
        at java.lang.Thread.run(Thread.java:1589) ~[?:?]

The crash report is ~20 MB, but the main information out of it is:

   Threads: { BLOCKED Craft Scheduler Thread - 81317 - ModelEngine: [ModelEngine-4.0.2.jar//com.ticxo.modelengine.api.utils.ticker.PseudoThread.tick(PseudoThread.java:47), ModelEngine-4.0.2.jar//com.ticxo.modelengine.api.utils.ticker.PseudoThread$1.run(PseudoThread.java:31), org.bukkit.craftbukkit.v1_20_R1.scheduler.CraftTask.run(CraftTask.java:101), org.bukkit.craftbukkit.v1_20_R1.scheduler.CraftAsyncTask.run(CraftAsyncTask.java:57), com.destroystokyo.paper.ServerSchedulerReportingWrapper.run(ServerSchedulerReportingWrapper.java:22), java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144), java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642), java.base@19.0.1/java.lang.Thread.run(Thread.java:1589)], BLOCKED Craft Scheduler Thread - 82549 - ModelEngine: [ModelEngine-4.0.2.jar//com.ticxo.modelengine.api.utils.ticker.PseudoThread.tick(PseudoThread.java:47), ModelEngine-4.0.2.jar//com.ticxo.modelengine.api.utils.ticker.PseudoThread$1.run(PseudoThread.java:31), 


.........


BLOCKED Craft Scheduler Thread - 74104 - ModelEngine: [ModelEngine-4.0.2.jar//com.ticxo.modelengine.api.utils.ticker.PseudoThread.tick(PseudoThread.java:47), ModelEngine-4.0.2.jar//com.ticxo.modelengine.api.utils.ticker.PseudoThread$1.run(PseudoThread.java:31), org.bukkit.craftbukkit.v1_20_R1.scheduler.CraftTask.run(CraftTask.java:101), org.bukkit.craftbukkit.v1_20_R1.scheduler.CraftAsyncTask.run(CraftAsyncTask.java:57), com.destroystokyo.paper.ServerSchedulerReportingWrapper.run(ServerSchedulerReportingWrapper.java:22), java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144), java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642), java.base@19.0.1/java.lang.Thread.run(Thread.java:1589)], BLOCKED Craft Scheduler Thread - 76795 - ModelEngine: [ModelEngine-4.0.2.jar//com.ticxo.modelengine.api.utils.ticker.PseudoThread.tick(PseudoThread.java:47), ModelEngine-4.0.2.jar//com.ticxo.modelengine.api.utils.ticker.PseudoThread$1.run(PseudoThread.java:31), org.bukkit.craftbukkit.v1_20_R1.scheduler.CraftTask.run(CraftTask.java:101), org.bukkit.craftbukkit.v1_20_R1.scheduler.CraftAsyncTask.run(CraftAsyncTask.java:57), com.destroystokyo.paper.ServerSchedulerReportingWrapper.run(ServerSchedulerReportingWrapper.java:22), java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144), java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642), java.base@19.0.1/java.lang.Thread.run(Thread.java:1589)],}
Unknowncommented

I think it is related to #16

Unknowncommented

@Ticxo happened again on MEG 4.0.3 & Purpur 1.20.1

> [3116.710s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 2048k, guardsize: 0k, detached.                  [3116.711s][warning][os,thread] Failed to start the native thread for java.lang.Thread "Craft Scheduler Thread - 23117"
[3116.711s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 2048k, guardsize: 0k, detached.
[3116.712s][warning][os,thread] Failed to start the native thread for java.lang.Thread "Craft Async Scheduler Management Thread"
[15:26:45 ERROR]: Caught previously unhandled exception :
> [3116.740s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 2048k, guardsize: 0k, detached.
[3116.741s][warning][os,thread] Failed to start the native thread for java.lang.Thread "Craft Async Scheduler Management Thread"
[15:26:45 ERROR]: Craft Async Scheduler Management Thread
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
        at java.lang.Thread.start0(Native Method) ~[?:?]
        at java.lang.Thread.start(Thread.java:1560) ~[?:?]
        at java.lang.System$2.start(System.java:2528) ~[?:?]
        at jdk.internal.vm.SharedThreadContainer.start(SharedThreadContainer.java:160) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:953) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1021) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1158) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1589) ~[?:?]
[15:26:45 ERROR]: Encountered an unexpected exception
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
        at java.lang.Thread.start0(Native Method) ~[?:?]
        at java.lang.Thread.start(Thread.java:1560) ~[?:?]
        at java.lang.System$2.start(System.java:2528) ~[?:?]
        at jdk.internal.vm.SharedThreadContainer.start(SharedThreadContainer.java:160) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:953) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364) ~[?:?]
        at java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:759) ~[?:?]
        at org.bukkit.craftbukkit.v1_20_R1.scheduler.CraftAsyncScheduler.mainThreadHeartbeat(CraftAsyncScheduler.java:73) ~[purpur-1.20.1.jar:git-Purpur-2050]
        at org.bukkit.craftbukkit.v1_20_R1.scheduler.CraftScheduler.mainThreadHeartbeat(CraftScheduler.java:462) ~[purpur-1.20.1.jar:git-Purpur-2050]
        at net.minecraft.server.MinecraftServer.tickChildren(MinecraftServer.java:1506) ~[purpur-1.20.1.jar:git-Purpur-2050]
        at net.minecraft.server.dedicated.DedicatedServer.tickChildren(DedicatedServer.java:486) ~[purpur-1.20.1.jar:git-Purpur-2050]
        at net.minecraft.server.MinecraftServer.tickServer(MinecraftServer.java:1420) ~[purpur-1.20.1.jar:git-Purpur-2050]
        at net.minecraft.server.MinecraftServer.runServer(MinecraftServer.java:1191) ~[purpur-1.20.1.jar:git-Purpur-2050]
        at net.minecraft.server.MinecraftServer.lambda$spin$0(MinecraftServer.java:322) ~[purpur-1.20.1.jar:git-Purpur-2050]
        at java.lang.Thread.run(Thread.java:1589) ~[?:?]

   Threads: { BLOCKED Craft Scheduler Thread - 9711 - ModelEngine: [ModelEngine-4.0.3.jar//com.ticxo.modelengine.api.utils.ticker.PseudoThread.tick(PseudoThread.java:47), ModelEngine
Unknowncommented

Worth mentioning that since updating to MEG 4.0.3 (no other changes in plugins/purpur version) it started happening more often. Before updating it happened once, after updating it happens couple times a day even with very low player count.

Unknowncommented

I have a theory. The repeating Bukkit scheduler seems to call the run method without waiting for the last tick to finish, which might drive all my synchronized code into a deadlock. Can you send your latest crash log and also DM me on discord? I will send you a jar and see if that fixes anything.

Sign in to join the discussion.