Growing memory usage in go-zenon

Archived
1

@aliencoder are you aware of the memory usage issue on go-zenon? We were discussing w/ Kaine this week. The znnd process consumes more and more memory until it fails. The process then restarts.

Kaine asked which version we are building znnd with. The znn-controller uses 1.18. He said we need to use 1.19.4.

I tried that and it did not fix the issue. You can see the memory usage of znnd over the last 48 hours. It climbs steadily. I have about 8 different nodes that ALL exhibit this same behavior. All the community members have this issue too. Many run a cron job to restart the znnd process every day.

The image might be difficult to see. The bottom line is the znnd process usage. It rises to 85% and then restarts. I have monit monitor the process to automatically restart it. It restarts about every 48 hours.

I think we need to look into this issue.

2023-03-31 17.41.28
2023-03-31 17.41.281280×376 62.9 KB

2

No, I’m not aware. I’ll try to finish first with the PR stuff and after that explore the go-zenon codebase in more detail. I think first we need a cicd pipeline to be able to build and test automatically znnd :grin:

1 Like
3

I was experiencing the same behaviour with my old server, but now has been running 4,5 weeks straight after I replaced it with the new one.

I’m still using your initial docker image. Node commit: 7e9c440f7e5e368efbade77ca900efa1219af0d4.

Monitor Services - Dashboards - Dashboards - Grafana - Brave 1-4-2023 10_42_37 (2)
Monitor Services - Dashboards - Dashboards - Grafana - Brave 1-4-2023 10_42_37 (2)1909×691 98.5 KB

4

Can we move discussions around go-zenon/znnd to a separate thread @CryptoFish @0x3639? Maybe @mehowbrainz can help us migrate the replies.

1 Like
5

done

6

what happens if you select 7 days for the duration?

Screenshot 2023-04-01 at 6.39.21 PM

7

Growing memory usage in go-zenon - Development _ ╰ Bugs - Forum - Brave 2-4-2023 09_52_23
Growing memory usage in go-zenon - Development _ ╰ Bugs - Forum - Brave 2-4-2023 09_52_231920×850 120 KB

Zenon Node - Dashboards - Dashboards - Grafana - Brave 2-4-2023 09_47_52
Zenon Node - Dashboards - Dashboards - Grafana - Brave 2-4-2023 09_47_521920×705 122 KB

8

ya, same behavior as mine. Looks like Monit is restarting znnd at 85% memory usage.

9

Let’s implement pprof tool and see exactly where the bottleneck is.

10

Oke, I thought it would result in a reboot of the pc. Good to know.

1 Like
11

Started the implementation. I’ll keep you updated with the progress.

1 Like
12

You can find the implementation here.

Download or build znnd from source and start it (default address is http://localhost:6060):

./znnd --pprof

Download and install the pprof tool:

go install github.com/google/pprof@latest

Here is a step-by-step tutorial.

1 Like
13

For reference.

http://127.0.0.1:8848/debug/pprof/  # Main Interface

http://127.0.0.1:8848/debug/pprof/allocs     # Sampling of all past memory allocations
http://127.0.0.1:8848/debug/pprof/block      # Stack tracing leading to synchronous blocking 
http://127.0.0.1:8848/debug/pprof/cmdline    # The full call path to the command line of the current program
http://127.0.0.1:8848/debug/pprof/goroutine  # Stack trace for all current Goroutines
http://127.0.0.1:8848/debug/pprof/heap       # Sampling of memory allocation for active objects
http://127.0.0.1:8848/debug/pprof/mutex      # Stack trace for contending mutex lock holders
http://127.0.0.1:8848/debug/pprof/profile    # CPU Configuration File
http://127.0.0.1:8848/debug/pprof/threadcreate # Creating a stack trace for a new OS thread
http://127.0.0.1:8848/debug/pprof/trace      # Tracking of current program execution
1 Like