Growing memory usage in go-zenon

[0x3639]

@aliencoder are you aware of the memory usage issue on go-zenon? We were discussing w/ Kaine this week. The znnd process consumes more and more memory until it fails. The process then restarts.

Kaine asked which version we are building znnd with. The znn-controller uses 1.18. He said we need to use 1.19.4.

I tried that and it did not fix the issue. You can see the memory usage of znnd over the last 48 hours. It climbs steadily. I have about 8 different nodes that ALL exhibit this same behavior. All the community members have this issue too. Many run a cron job to restart the znnd process every day.

The image might be difficult to see. The bottom line is the znnd process usage. It rises to 85% and then restarts. I have monit monitor the process to automatically restart it. It restarts about every 48 hours.

I think we need to look into this issue.

[aliencoder]

@0x3639

@aliencoder are you aware of the memory usage issue on go-zenon?

No, I’m not aware. I’ll try to finish first with the PR stuff and after that explore the go-zenon codebase in more detail. I think first we need a cicd pipeline to be able to build and test automatically znnd

[CryptoFish]

I was experiencing the same behaviour with my old server, but now has been running 4,5 weeks straight after I replaced it with the new one.

I’m still using your initial docker image. Node commit: 7e9c440f7e5e368efbade77ca900efa1219af0d4.

[aliencoder]

Can we move discussions around go-zenon/znnd to a separate thread @CryptoFish @0x3639? Maybe @mehowbrainz can help us migrate the replies.

[0x3639]

done

[0x3639]

what happens if you select 7 days for the duration?

[CryptoFish]

[0x3639]

ya, same behavior as mine. Looks like Monit is restarting znnd at 85% memory usage.

[aliencoder]

Let’s implement pprof tool and see exactly where the bottleneck is.

[CryptoFish]

Oke, I thought it would result in a reboot of the pc. Good to know.

[aliencoder]

Started the implementation. I’ll keep you updated with the progress.

[aliencoder]

You can find the implementation here.

Download or build znnd from source and start it (default address is http://localhost:6060):

./znnd --pprof

Download and install the pprof tool:

go install github.com/google/pprof@latest

Here is a step-by-step tutorial.