WebFaction
Community site: login faq

I see from https://docs.webfaction.com/software/general.html#logs that our raw Apache logs are rotated once a day and deleted after 7 days.

Like the user at https://community.webfaction.com/questions/13730/apache-logs-never-delete-them , I have an application where I need to retain all access and Apache logs forever.

Unlike that user, I am using the shared Apache instance so I cannot control httpd.conf (and even if I could, I would prefer to leave logs at ~/logs initially so that awstats and webalizer can get at them).

So it seems the best option is for me to copy logs out of ~/logs to a different directory before the original logs get deleted.

My question is: before I spend time cooking up a script / cron job and debugging whatever edge cases pop up, has anyone also solved this problem and can share a working, well-debugged script?

Thanks.

asked 21 May '16, 05:14

cpirazzi
199113
accept rate: 16%


We rotate your logs in ~/logs every day and keep a seven day history.

The actual time that we rotate them is usually at 3am server time, but that may vary.

So, if you want to keep a history of your logs, then I recommend you that copy the logs from the prior day every 24 hours, at some time well after 3am server time.

A cron job like this, run daily, should suffice:

0 12 * * * cd $HOME/logs/frontend; for log in *.1; do cp $log ~/mylogs/$log-$(date +\%Y\%m\%d); done

Hope that helps!

permanent link

answered 22 May '16, 03:56

seanf
12.2k31836
accept rate: 37%

Also, if for some reason the logs are unusually large, one could use a hardlink ('ln') instead of a copy ('cp').

(22 May '16, 04:06) ryans ♦♦

Cool thanks Sean; very simple!

I tried the ln before but I get "operation not permitted," perhaps because the files are owned by root? Anyway my storage should be enough.

(22 May '16, 04:55) cpirazzi

You'd want to test to ensure that the automated lograte process doesn't truncate files before rotating them (so that the hardlink count drops from 2 to 1 and the content persists).

As long as you're creating the hardlink in a location that you have write access to, there shouldn't be an "operation not permitted" error as far as I am aware - I tested this under a test account to confirm. Making a hardlink uses the same permissions as a copy.

In any case, the storage would not be affected; after a rotation, there should be only one remaining link to the underlying inode, so it counts fully toward your disk usage. What's saved is the overhead of making the copy in the first place (which is only significant for very large logs, but in some cases this will be preferable). Sean's answer is the safer and simpler solution, but I wanted to point out this alternative as well.

(22 May '16, 05:26) ryans ♦♦

Here's what I see (current directory in this example is my home directory, where I have write permission):

/home/cpirazzi% ln logs/frontend/error_yawningfields_com.log bar ln: failed to create hard link ‘bar’ => ‘logs/frontend/error_yawningfields_com.log’: Operation not permitted

(22 May '16, 09:43) cpirazzi

Oh, it looks like you're right - now I see the problem. Hardlinking to root-owned files is now restricted in the newer kernel used on our CentOS 7 (web500+) servers; this behavior has changed very recently, and I was not aware of it until now. Thanks for pointing this out!

It looks like the cp solution by Sean will be the canonical solution going forward.

(22 May '16, 09:56) ryans ♦♦
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×224
×28
×25
×4
×3

question asked: 21 May '16, 05:14

question was seen: 1,215 times

last updated: 22 May '16, 09:56

WEBFACTION
REACH US
SUPPORT
AFFILIATE PROGRAM
LEGAL
© COPYRIGHT 2003-2019 SWARMA LIMITED - WEBFACTION IS A SERVICE OF SWARMA LIMITED
REGISTERED IN ENGLAND AND WALES 5729350 - VAT REGISTRATION NUMBER 877397162
5TH FLOOR, THE OLD VINYL FACTORY, HAYES, UB3 1HA, UNITED KINGDOM