# Growing memory usage in go-zenon **URL:** http://forum.zenon.org/t/growing-memory-usage-in-go-zenon/1343 **Category:** Archived **Created:** [March 31, 2023, 10:45pm UTC](http://forum.zenon.org/t/growing-memory-usage-in-go-zenon/1343 "2023-03-31T22:45:25Z") **Posts on this page:** 16 **Page:** 1
### Author: ![0x3639](http://forum.zenon.org/user_avatar/forum.zenon.org/0x3639/32/1651_2.png) [@0x3639](http://forum.zenon.org/u/0x3639) #### Post date: [March 31, 2023, 10:45pm UTC](http://forum.zenon.org/t/growing-memory-usage-in-go-zenon/1343/1 "2023-03-31T22:45:25Z")
@aliencoder are you aware of the memory usage issue on go-zenon? We were discussing w/ Kaine this week. The znnd process consumes more and more memory until it fails. The process then restarts. Kaine asked which version we are building znnd with. The znn-controller uses 1.18. He said we need to use 1.19.4. I tried that and it did not fix the issue. You can see the memory usage of znnd over the last 48 hours. It climbs steadily. I have about 8 different nodes that ALL exhibit this same behavior. All the community members have this issue too. Many run a cron job to restart the znnd process every day. The image might be difficult to see. The bottom line is the znnd process usage. It rises to 85% and then restarts. I have monit monitor the process to automatically restart it. It restarts about every 48 hours. I think we need to look into this issue. ![2023-03-31 17.41.28](https://forum.zenon.org/uploads/default/original/2X/3/3c3369ce8deb9a6c1a5a71e5f3f7cb0f7a3c7dc2.jpeg) ---
### Author: ![aliencoder](http://forum.zenon.org/user_avatar/forum.zenon.org/aliencoder/32/1467_2.png) [@aliencoder](http://forum.zenon.org/u/aliencoder) #### Post date: [April 1, 2023, 7:23am UTC](http://forum.zenon.org/t/growing-memory-usage-in-go-zenon/1343/2 "2023-04-01T07:23:12Z")
> [@0x3639](#): > > @aliencoder are you aware of the memory usage issue on go-zenon? No, I’m not aware. I’ll try to finish first with the PR stuff and after that explore the `go-zenon` codebase in more detail. I think first we need a `cicd` pipeline to be able to build and test automatically `znnd` 😁 ---
### Author: ![CryptoFish](http://forum.zenon.org/user_avatar/forum.zenon.org/cryptofish/32/171_2.png) [@CryptoFish](http://forum.zenon.org/u/CryptoFish) #### Post date: [April 1, 2023, 8:50am UTC](http://forum.zenon.org/t/growing-memory-usage-in-go-zenon/1343/3 "2023-04-01T08:50:17Z")
I was experiencing the same behaviour with my old server, but now has been running 4,5 weeks straight after I replaced it with the new one. I’m still using your initial docker image. Node commit: 7e9c440f7e5e368efbade77ca900efa1219af0d4. ![Monitor Services - Dashboards - Dashboards - Grafana - Brave 1-4-2023 10_42_37 (2)](https://forum.zenon.org/uploads/default/original/2X/0/040d7c63c1a9bf7c6ba6711f717bd4d872835db7.png) ---
### Author: ![aliencoder](http://forum.zenon.org/user_avatar/forum.zenon.org/aliencoder/32/1467_2.png) [@aliencoder](http://forum.zenon.org/u/aliencoder) #### Post date: [April 1, 2023, 1:55pm UTC](http://forum.zenon.org/t/growing-memory-usage-in-go-zenon/1343/4 "2023-04-01T13:55:39Z")
Can we move discussions around `go-zenon`/`znnd` to a separate thread @CryptoFish @0x3639? Maybe @mehowbrainz can help us migrate the replies. ---
### Author: ![0x3639](http://forum.zenon.org/user_avatar/forum.zenon.org/0x3639/32/1651_2.png) [@0x3639](http://forum.zenon.org/u/0x3639) #### Post date: [April 1, 2023, 11:38pm UTC](http://forum.zenon.org/t/growing-memory-usage-in-go-zenon/1343/5 "2023-04-01T23:38:34Z")
done ---
### Author: ![0x3639](http://forum.zenon.org/user_avatar/forum.zenon.org/0x3639/32/1651_2.png) [@0x3639](http://forum.zenon.org/u/0x3639) #### Post date: [April 1, 2023, 11:39pm UTC](http://forum.zenon.org/t/growing-memory-usage-in-go-zenon/1343/6 "2023-04-01T23:39:50Z")
what happens if you select 7 days for the duration? ![Screenshot 2023-04-01 at 6.39.21 PM](https://forum.zenon.org/uploads/default/original/2X/6/6c8565b250ef4e4391c02959d8fc2368e763f728.png) ---
### Author: ![CryptoFish](http://forum.zenon.org/user_avatar/forum.zenon.org/cryptofish/32/171_2.png) [@CryptoFish](http://forum.zenon.org/u/CryptoFish) #### Post date: [April 2, 2023, 7:53am UTC](http://forum.zenon.org/t/growing-memory-usage-in-go-zenon/1343/7 "2023-04-02T07:53:27Z")
![Growing memory usage in go-zenon - Development _ ╰ Bugs - Forum - Brave 2-4-2023 09_52_23](https://forum.zenon.org/uploads/default/original/2X/3/386300726ad7f6b354d6df48b42826c726b8ff59.png) ![Zenon Node - Dashboards - Dashboards - Grafana - Brave 2-4-2023 09_47_52](https://forum.zenon.org/uploads/default/original/2X/1/116ed0a09589978a0c9a52125294bfa778b02b42.png) ---
### Author: ![0x3639](http://forum.zenon.org/user_avatar/forum.zenon.org/0x3639/32/1651_2.png) [@0x3639](http://forum.zenon.org/u/0x3639) #### Post date: [April 2, 2023, 9:03am UTC](http://forum.zenon.org/t/growing-memory-usage-in-go-zenon/1343/8 "2023-04-02T09:03:49Z")
ya, same behavior as mine. Looks like Monit is restarting znnd at 85% memory usage. ---
### Author: ![aliencoder](http://forum.zenon.org/user_avatar/forum.zenon.org/aliencoder/32/1467_2.png) [@aliencoder](http://forum.zenon.org/u/aliencoder) #### Post date: [April 2, 2023, 9:34am UTC](http://forum.zenon.org/t/growing-memory-usage-in-go-zenon/1343/9 "2023-04-02T09:34:23Z")
Let’s implement [pprof](https://github.com/google/pprof) tool and see exactly where the bottleneck is. ---
### Author: ![CryptoFish](http://forum.zenon.org/user_avatar/forum.zenon.org/cryptofish/32/171_2.png) [@CryptoFish](http://forum.zenon.org/u/CryptoFish) #### Post date: [April 2, 2023, 12:03pm UTC](http://forum.zenon.org/t/growing-memory-usage-in-go-zenon/1343/10 "2023-04-02T12:03:37Z")
Oke, I thought it would result in a reboot of the pc. Good to know. ---
### Author: ![aliencoder](http://forum.zenon.org/user_avatar/forum.zenon.org/aliencoder/32/1467_2.png) [@aliencoder](http://forum.zenon.org/u/aliencoder) #### Post date: [April 6, 2023, 8:33am UTC](http://forum.zenon.org/t/growing-memory-usage-in-go-zenon/1343/11 "2023-04-06T08:33:49Z")
Started the implementation. I’ll keep you updated with the progress. ---
### Author: ![aliencoder](http://forum.zenon.org/user_avatar/forum.zenon.org/aliencoder/32/1467_2.png) [@aliencoder](http://forum.zenon.org/u/aliencoder) #### Post date: [April 6, 2023, 2:26pm UTC](http://forum.zenon.org/t/growing-memory-usage-in-go-zenon/1343/12 "2023-04-06T14:26:37Z")
You can find the implementation [here](https://github.com/alienc0der/go-zenon/tree/pprof). Download or build `znnd` from source and start it (default address is `http://localhost:6060`): ```auto ./znnd --pprof ``` Download and install the `pprof` [tool](https://github.com/google/pprof): ```auto go install github.com/google/pprof@latest ``` Here is a step-by-step [tutorial](https://www.sobyte.net/post/2022-06/go-pprof/). ---
### Author: ![0x3639](http://forum.zenon.org/user_avatar/forum.zenon.org/0x3639/32/1651_2.png) [@0x3639](http://forum.zenon.org/u/0x3639) #### Post date: [June 16, 2023, 4:47pm UTC](http://forum.zenon.org/t/growing-memory-usage-in-go-zenon/1343/13 "2023-06-16T16:47:55Z")
For reference. ```auto http://127.0.0.1:8848/debug/pprof/ # Main Interface http://127.0.0.1:8848/debug/pprof/allocs # Sampling of all past memory allocations http://127.0.0.1:8848/debug/pprof/block # Stack tracing leading to synchronous blocking http://127.0.0.1:8848/debug/pprof/cmdline # The full call path to the command line of the current program http://127.0.0.1:8848/debug/pprof/goroutine # Stack trace for all current Goroutines http://127.0.0.1:8848/debug/pprof/heap # Sampling of memory allocation for active objects http://127.0.0.1:8848/debug/pprof/mutex # Stack trace for contending mutex lock holders http://127.0.0.1:8848/debug/pprof/profile # CPU Configuration File http://127.0.0.1:8848/debug/pprof/threadcreate # Creating a stack trace for a new OS thread http://127.0.0.1:8848/debug/pprof/trace # Tracking of current program execution ``` ---
### Author: ![mehowbrainz](http://forum.zenon.org/user_avatar/forum.zenon.org/mehowbrainz/32/1532_2.png) [@mehowbrainz](http://forum.zenon.org/u/mehowbrainz) #### Post date: [September 12, 2023, 7:00pm UTC](http://forum.zenon.org/t/growing-memory-usage-in-go-zenon/1343/14 "2023-09-12T19:00:30Z")
---
### Author: ![mehowbrainz](http://forum.zenon.org/user_avatar/forum.zenon.org/mehowbrainz/32/1532_2.png) [@mehowbrainz](http://forum.zenon.org/u/mehowbrainz) #### Post date: [September 12, 2023, 7:00pm UTC](http://forum.zenon.org/t/growing-memory-usage-in-go-zenon/1343/15 "2023-09-12T19:00:43Z")
---
### Author: ![mehowbrainz](http://forum.zenon.org/user_avatar/forum.zenon.org/mehowbrainz/32/1532_2.png) [@mehowbrainz](http://forum.zenon.org/u/mehowbrainz) #### Post date: [September 12, 2023, 7:01pm UTC](http://forum.zenon.org/t/growing-memory-usage-in-go-zenon/1343/16 "2023-09-12T19:01:19Z")