nd4j memory leak in 1.0.4b and 1.0.6b

[barrybecker4]

There seems to be a separate serious leak when using 1.0.0-beta4 or 1.0.0-beta6 version of nd4j.
It does not happen with 1.0.0-b3.

I lowered memory settings to
"-Xms128M",
"-Xmx256M",
"-Dorg.bytedeco.javacpp.maxbytes=256M",
"-Dorg.bytedeco.javacpp.maxphysicalbytes=1G",
and turned off all experience collecting an training, and it still ran out of memory by episode 9.
Probably the problem is with the building of the nodes.
Maybe go back to 1.0.3 and test.

[barrybecker4]

Even with 1.0.0-b3, I still see a leak. I was running a 300 episode case with batch = 5. Memory settings at -Xms2G, -Xmx8G. It ran out of memory on episode 266 with this error:

[info] 04:53:32.525 [main] ERROR o.d.util.CrashReportingUtil - >>> Out of Memory Exception Detected. Memory crash dump written to: E:\projects\dl4j\ScalphaGoZero\dl4j-memory-crash-dump-1583668387662_1.txt
[info] 04:53:32.525 [main] WARN  o.d.util.CrashReportingUtil - Memory crash dump reporting can be disabled with CrashUtil.crashDumpsEnabled(false) or using system property -Dorg.deeplearning4j.crash.reporting.enabled=false
[info] 04:53:32.525 [main] WARN  o.d.util.CrashReportingUtil - Memory crash dump reporting output location can be set with CrashUtil.crashDumpOutputDirectory(File) or using system property -Dorg.deeplearning4j.crash.reporting.directory=<path>
[error] Exception in thread "main" java.lang.OutOfMemoryError: Cannot allocate new LongPointer(9): totalBytes = 20882K, physicalBytes = 3074M
[error]         at org.bytedeco.javacpp.LongPointer.<init>(LongPointer.java:76)
[error]         at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.exec(NativeOpExecutioner.java:1815)
[error]         at org.nd4j.linalg.convolution.Convolution.im2col(Convolution.java:211)
[error]         at org.deeplearning4j.nn.layers.convolution.ConvolutionLayer.preOutput(ConvolutionLayer.java:367)
[error]         at org.deeplearning4j.nn.layers.convolution.ConvolutionLayer.activate(ConvolutionLayer.java:411)
[error]         at org.deeplearning4j.nn.graph.vertex.impl.LayerVertex.doForward(LayerVertex.java:110)
[error]         at org.deeplearning4j.nn.graph.ComputationGraph.outputOfLayersDetached(ComputationGraph.java:2316)
[error]         at org.deeplearning4j.nn.graph.ComputationGraph.output(ComputationGraph.java:1727)
[error]         at org.deeplearning4j.nn.graph.ComputationGraph.output(ComputationGraph.java:1686)
[error]         at org.deeplearning4j.nn.graph.ComputationGraph.output(ComputationGraph.java:1616)
[error]         at org.deeplearning4j.scalphagozero.agents.ZeroTreeNodeCreator.createNode(ZeroTreeNodeCreator.scala:26)
[error]         at org.deeplearning4j.scalphagozero.agents.MonteCarloPlayer.valueFromMCPlayout(MonteCarloPlayer.scala:22)
[error]         at 
:
[error]         at org.deeplearning4j.scalphagozero.agents.MonteCarloPlayer.valueFromMCPlayout(MonteCarloPlayer.scala:23)
[error]         at org.deeplearning4j.scalphagozero.agents.ZeroAgent.$anonfun$selectMove$1(ZeroAgent.scala:80)
[error]         at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:156)
[error]         at org.deeplearning4j.scalphagozero.agents.ZeroAgent.selectMove(ZeroAgent.scala:63)
[error]         at org.deeplearning4j.scalphagozero.simulation.ZeroSimulator$.doSimulation(ZeroSimulator.scala:65)
[error]         at org.deeplearning4j.scalphagozero.simulation.ZeroSimulator$.simulateLearningGame(ZeroSimulator.scala:48)
[error]         at org.deeplearning4j.scalphagozero.app.Trainer.$anonfun$runSimulationsAndTrain$1(Trainer.scala:25)
[error]         at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:156)
[error]         at org.deeplearning4j.scalphagozero.app.Trainer.runSimulationsAndTrain(Trainer.scala:22)
[error]         at org.deeplearning4j.scalphagozero.app.Demo.run(Demo.scala:50)
[error]         at org.deeplearning4j.scalphagozero.app.ScalphaGoZero$.delayedEndpoint$org$deeplearning4j$scalphagozero$app$ScalphaGoZero$1(ScalphaGoZero.scala:14)
[error]         at org.deeplearning4j.scalphagozero.app.ScalphaGoZero$delayedInit$body.apply(ScalphaGoZero.scala:10)
[error]         at scala.Function0.apply$mcV$sp(Function0.scala:34)
[error]         at scala.Function0.apply$mcV$sp$(Function0.scala:34)
[error]         at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
[error]         at scala.App.$anonfun$main$1$adapted(App.scala:76)
[error]         at scala.collection.immutable.List.foreach(List.scala:389)
[error]         at scala.App.main(App.scala:76)
[error]         at scala.App.main$(App.scala:74)
[error]         at org.deeplearning4j.scalphagozero.app.ScalphaGoZero$.main(ScalphaGoZero.scala:10)
[error]         at org.deeplearning4j.scalphagozero.app.ScalphaGoZero.main(ScalphaGoZero.scala)
[error] Caused by: java.lang.OutOfMemoryError: Physical memory usage is too high: physicalBytes (3074M) > maxPhysicalBytes (3072M)
[error]         at org.bytedeco.javacpp.Pointer.deallocator(Pointer.java:584)
[error]         at org.bytedeco.javacpp.Pointer.init(Pointer.java:124)
[error]         at org.bytedeco.javacpp.LongPointer.allocateArray(Native Method)
[error]         at org.bytedeco.javacpp.LongPointer.<init>(LongPointer.java:68)
[error]         ... 53 more
[error] Nonzero exit code returned from runner: 1