Thread Info
  • State Verified Answer
  • Locked Locked
  • Replies 4 replies
  • Subscribers 3 subscribers
  • Views 3919 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Can't get remote log archiving via SCP to work

Harro Verton

I'm running out of diskspace for my logs, so I decided to activate remote log archiving. In my environment, only SCP is an option. Our archive server also does NFS, but that option is not presented.

Version of our UTM's:

<M> loginuser@firewall-1:/home/login > cat /etc/version
 9.407003

Testing the configuration works fine:

<M> loginuser@firewall-1:/home/login > sudo logarchiver.plx -t -d 15
DEBUG: **************************************************
DEBUG: logarchiver.pl ($Revision: 1.33.2.1.2.1 $)
DEBUG: Options:
DEBUG:  - debug mode: 15
DEBUG:  - remote archive testing run
DEBUG: **************************************************
DEBUG: * doing remote storage test...
DEBUG: - writing test file: /opt/tmpfs/remote-log-test.txt
DEBUG: - storing test file...
 INFO: - doing remote storage via scp
DEBUG: found data: (key:REMOTE_HOST) 172.17.2.5
DEBUG: found data: (key:REMOTE_USER) firewall
DEBUG: found data: (key:REMOTE_PATH) /data/firewall
DEBUG: checking private key: yes
DEBUG: checking ssh config : yes
DEBUG: host: 172.17.2.5 user: firewall path: /data/firewall file: /opt/tmpfs/remote-log-test.txt
DEBUG: checking log file archive: yes
       remote-log-test.txt   100%  183     0.2KB/s   00:00
DEBUG: successfully stored log backup
 INFO: - deleting successfully transferred file(s)
DEBUG:     /opt/tmpfs/remote-log-test.txt

but running it gives

<M> loginuser@firewall-1:/home/login > sudo logarchiver.plx
root's password:
 INFO: -> no log files to process!

and I keep getting emails that my log space is over 90% in use, and old logs are being removed (which I don't want, but have to)

Anyone have any idea that is wrong here, and how to fix it?



This thread was automatically locked due to age.
Parents
  • 0

    In addition, running

    /usr/sbin/logrotate -v /etc/logrotate.conf
    /usr/local/bin/logarchiver.plx
     
    does archive and copy fine, but only copies the archives for 21/11, 22/11 and 23/11, not the older files already archived on the firewall itself. Now I have to look in two places for logs?
  • 0 in reply to Harro Verton

    Bump...

    I'm slowly developing sweat drops on my forehead. I now get emails saying the log volume has reached 95%. I have configured the UTM to start deleting logs at this percentage, but I don't want to lose logs, I need to keep them for compliance reasons.

    Help?!

  • 0 in reply to Harro Verton

    Hmmm....

    Think I've panicked too soon. It does now archive the logs to the remote server, but only on rotating the current logs. It doesn't not archive/move the logs that are already archived on the log volume itself, and as long as that doesn't happen, my log volume will stay 90+ % filled.

    So let's rephrase the question: how to move the local archive to the remote log server in the same format the UTM archives it itself?

  • 0 in reply to Harro Verton

    For those with the same question:

    WORK=`mktemp -d -p "/tmp"`
    if [[ ! "$WORK" || ! -d "$WORK" ]]; then
     echo "Could not create temp dir"
     exit 1
    fi

    LOGS="/var/log"
    cd $LOGS

    DATES=`find . -name "*201?-*.gz" | rev | cut -d "/" -f1 | rev | cut -d "." -f1 | rev | cut -d "-" -f 1,2,3 | rev | sort | uniq`

    for DATE in $DATES; do
     find . -name "*$DATE*.gz" -exec mv {} $WORK \;
     cd $WORK
     DEST=/tmp/logfiles-$DATE.tgz
     tar -czvf $DEST *
     scp -i /etc/logremote/logremote_rsa $DEST firewall@logserver.hosting.exite.eu:/data/firewall/logs
     rm -f *
     rm -f $DEST
     cd $LOGS
    done

    function cleanup {
     rm -rf "$WORK"
    }

    trap cleanup EXIT

    After which you can delete the logfiles archived on the firewall.

    update 19-03-2018: the file that stores the ssh key has changed name, it is now called logremote_rsa, instead of id_rsa!

Reply
  • 0 in reply to Harro Verton

    For those with the same question:

    WORK=`mktemp -d -p "/tmp"`
    if [[ ! "$WORK" || ! -d "$WORK" ]]; then
     echo "Could not create temp dir"
     exit 1
    fi

    LOGS="/var/log"
    cd $LOGS

    DATES=`find . -name "*201?-*.gz" | rev | cut -d "/" -f1 | rev | cut -d "." -f1 | rev | cut -d "-" -f 1,2,3 | rev | sort | uniq`

    for DATE in $DATES; do
     find . -name "*$DATE*.gz" -exec mv {} $WORK \;
     cd $WORK
     DEST=/tmp/logfiles-$DATE.tgz
     tar -czvf $DEST *
     scp -i /etc/logremote/logremote_rsa $DEST firewall@logserver.hosting.exite.eu:/data/firewall/logs
     rm -f *
     rm -f $DEST
     cd $LOGS
    done

    function cleanup {
     rm -rf "$WORK"
    }

    trap cleanup EXIT

    After which you can delete the logfiles archived on the firewall.

    update 19-03-2018: the file that stores the ssh key has changed name, it is now called logremote_rsa, instead of id_rsa!

Children
No Data