pg3x exceeded disk quota

[Xathros]

Hello everyone,

pg3 is down on my eISY.  After some research, I have determined that pg3 is not starting because it has exceeded its disk quota.  Looks like the database has ballooned and I've reached the 10G limit. Pretty sure this is my fault for having the log level set to debug on my Kasa node server.  I didn't realize that the log data was stored in the database.  My bad.

How to clean up the log data, compact the DB and get back online?

Any assistance is greatly appreciated.

-Xathros

 

[Geddy]

15 minutes ago, Xathros said:

How to clean up the log data, compact the DB and get back online?

Perhaps the easiest way would be to enter a support ticket to get help getting to where the logs are and delete bloated logs and hopefully get you running again.

https://www.universal-devices.com/my-tickets

Additionally they would confirm that it was your log issue that was causing problems. 

 

[DennisC]

Since its pg3, maybe @bpwwercan help?

[bpwwer]

No log data is stored in any database.  However, it does keep a fair amount of log info and debug logs can get quite large.

Most of the log data should already be compressed and I believe cleared once it's older than 2 weeks.

Also, even if the Kasa log exceeded some limit, it shouldn't be effecting PG3x's ability to start and run.

I'd need more information on what PG3x is doing, how you're determining that it's log file related, etc.

[Xathros]

Thanks all.

Michel remoted in and helped me out today.

 

41 minutes ago, bpwwer said:

No log data is stored in any database.  However, it does keep a fair amount of log info and debug logs can get quite large.

Most of the log data should already be compressed and I believe cleared once it's older than 2 weeks.

Also, even if the Kasa log exceeded some limit, it shouldn't be effecting PG3x's ability to start and run.

I'd need more information on what PG3x is doing, how you're determining that it's log file related, etc.

It was mostly my Kasa node server.  I have 7 polys running and had all of the logs set to debug.  Kasa with 50 nodes is *VERY* chatty and filled the 10G allotment for the polyglot filesystem.  Between Kasa and the other 6 polys, it ran out of space with less than 2 weeks of logs!  With no free space, pg3 would not start.  It appears to me that in the current version, logs are not being compressed daily if at all.

We deleted all the logs and I have set my log levels to error instead of debug. 

I think there will be some changes coming to prevent this sort of thing from happening again.

-Xathros

 

[Geddy]

1 hour ago, Xathros said:

I think there will be some changes coming to prevent this sort of thing from happening again.

Yeah, it's a note to not leave logs in "debug" mode. Why put them all in that stage and leave them that way? That's usually only meant to debug a specific error. Leaving them in "info" or "error" mode, whichever is default, should be fine for day-to-day operations. The few that I run are in "info" mode as I don't worry about errors until I start having them. Then figure error or debug is what I would use to try to learn what's happening. 

Glad Michel got you sorted out. 

 

[bpwwer]

I wrong about the logging.  The node server logging for PG3 node servers was copied from PG2 and has been configured to rotate daily and, I believe, keep 30 days of old logs.  It's been like this for about 3 years (I.E. before PG3 existed).

It doesn't compress the logs.

As @Geddysays, there's no good reason to leave the level set to debug as some node servers can be very chatty when set to that.

Having a 10G quota on the PG3 directory seems like a new(ish) thing too. 

[Xathros]

2 hours ago, Geddy said:

Yeah, it's a note to not leave logs in "debug" mode. Why put them all in that stage and leave them that way? That's usually only meant to debug a specific error. Leaving them in "info" or "error" mode, whichever is default, should be fine for day-to-day operations. The few that I run are in "info" mode as I don't worry about errors until I start having them. Then figure error or debug is what I would use to try to learn what's happening. 

Glad Michel got you sorted out. 

 

I totally agree.  During setup of a new poly, I like to see everything to get a feel for what's going on under the hood.  I know the logs would be larger but, it's just text and compresses well and it's on a *nix platform so they'll rotate and it will be fine - or so I thought.  Everything is set to error now and I've learned once again to never assume anything

As usual, Michel and UDI's support is second to none.

 

-Xathros

 

[Xathros]

1 hour ago, bpwwer said:

I wrong about the logging.  The node server logging for PG3 node servers was copied from PG2 and has been configured to rotate daily and, I believe, keep 30 days of old logs.  It's been like this for about 3 years (I.E. before PG3 existed).

It doesn't compress the logs.

As @Geddysays, there's no good reason to leave the level set to debug as some node servers can be very chatty when set to that.

Having a 10G quota on the PG3 directory seems like a new(ish) thing too. 

After this, I will endeavor to only set debug level for debugging and not for normal daily operations.  10Gb of uncompressed text is still an awful lot of text.  It should be more than enough.  I do think 1 to 2 weeks of retained logs should be sufficient.  Just curious though, why not compress on rotation? 

 

-Xathros

[bpwwer]

20 minutes ago, Xathros said:

After this, I will endeavor to only set debug level for debugging and not for normal daily operations.  10Gb of uncompressed text is still an awful lot of text.  It should be more than enough.  I do think 1 to 2 weeks of retained logs should be sufficient.  Just curious though, why not compress on rotation? 

 

-Xathros

Logging for node servers is done using a Python logging library.  I don't believe it had support for compressing the old logs back when first used.

I've changed it now to only keep 14 days, but this type of change will only really effect new node server installations, not existing one.  I'm looking into compression to see if that's been added to the library.

[Xathros]

58 minutes ago, bpwwer said:

Logging for node servers is done using a Python logging library.  I don't believe it had support for compressing the old logs back when first used.

I've changed it now to only keep 14 days, but this type of change will only really effect new node server installations, not existing one.  I'm looking into compression to see if that's been added to the library.

I love how things get done around here!  Issue discovered and a change in the pipeline less than 6 hrs later.  Nowhere else does this happen!
 

What about updates/new versions to/of already installed node servers? Or re-installs of existing node servers?  Just curious.  I suspect I'll/We'll be fine with 30 days of logs as long as they aren't set to debug for normal ops.

-Xathros