ColdFusion Muse

Fixing Locked Jrun Log Files Without Restarting Coldfusion

Mark Kruger May 3, 2006 11:27 AM Coldfusion Troubleshooting Comments (18)

In a follow up to my previous post on Jrun Processor Pegging Issues and Solutions I was intrigued by an idea given to me by Steven Erat. The problem is that the "-err" and "-out" logs do not roll over like the event logs. This is a problem that is slated to be fixed in the next version. To "fix" this problem Steven thought it might be possible to move or delete the file using CFFILE without restarting CF. He reasoned that while windows could not defeat the lock, perhaps CF could defeat it because it owned the lock to begin with. Here's what I found.

Deleting, Moving and Renaming the File will all Fail

If you try to delete the file:

<Cffile action="DELETE"
      file="*path*\logs\coldfusion-out.log"
   />

   ...OR...
   <Cffile action="DELETE"
      file="*path*\logs\*instance*-out.log"
   />
It errors out with a message that says "ColdFusion could not delete the file '*path*\logs\coldfusion-out.log' for an unknown reason.". If you try to either move or rename the file as in these examples

Move
<Cffile action="MOVE"
   source="*path*\logs\coldfusion-out.log"
   destination="*path*\oldlogs\"
/>

...OR...
<Cffile action="MOVE"
   source="*path*\logs\*instance*-out.log"
   destination="*path*\oldlogs\"
/>

Rename
<Cffile action="RENAME"
   source="*path*\logs\coldfusion-out.log"
   destination="*path\logs\oldOut.log"
/>

...OR...
<Cffile action="RENAME"
   source="*path*\logs\*instance*-out.log"
   destination="*path\logs\oldOut.log"
/>
The code errors out with a message that the "The value of the attribute source, which is currently '*path*\logs\coldfusion-out.log', is invalid." That's the bad news. The good news is that it actually is possible to solve this problem without restarting the coldfusion instance. How? Use the "WRITE" action.

Overwriting the File

When I tried the write attribute like this:

<Cffile
   action="WRITE"
   file="*path*\logs\coldfusion-out.log"
   output="## Created by Bob, May 3, 2006"
   addnewline="yes"
/>
It worked perfectly. When I opened the file it now contains my message, "#Created by Bob, May 3, 2006", as the header. I restarted the server just to make sure that this file was not corrupted by my effort. When I reexamined the file it was collecting log data normally.

The Catch

The thing to note here is that the data is now gone. To quote Monty Python's Dead Parrot sketch, "It is deceased, it is no more, it has passed away, crossed over, pushing up daisies, gone toward the light, kicked the bucket, given up the ghost, bought the farm..." So before you try the code above, copy the file using windows so you have a copy of the data if you need it.

Why copy it using windows? Why not just do both operations using CFFILE? Ah... well, if you read my post from yesterday (5/2/2006) you will note that the problem we are most often trying to solve is that the "out" and "err" log files grow quite large (hundreds of megabytes perhaps even gigabytes) and do not "roll over" like the other "event" style log files. This creates a problem and I suspect it is the culprit in at least one case of JRUN pegging at 100%. Have you ever tried to copy a very very large file using Coldfusion? Quite simply, Coldfusion is not the best tool for this sort of thing. I suspect that Coldfusion reads the entire file into memory before writing it out to disk. If it's a very large file, it may even exhaust your server resources. Instead, use an alternative method for backing up the file, then overwrite it using the code above. This will give you the best of both worlds.

  • Share:

18 Comments

  • Raymond Camden's Gravatar
    Posted By
    Raymond Camden | 5/3/06 10:12 AM
    Thanks for the tip. My server has CF on C, and C is a bit small, so I run into this problem quite often.
  • Jason Troy's Gravatar
    Posted By
    Jason Troy | 5/3/06 10:16 AM
    > Instead, use an alternative method for backing up the file, then overwrite it using the code above.
    I like the idea Mark! Thank you!
    I'm going to do it and suffer the consequences of the "yeah buts" later I know it ... as long as we're overwriting the file in ColdFusion, couldn't we grab a copy of the data and output it to a incrimented log file based on a schedule you choose based on needs? For example, archive the file daily / weekly / annually - whatever works for your situation?
  • mkruger's Gravatar
    Posted By
    mkruger | 5/3/06 10:23 AM
    Jason,

    Yes - that would be possible. A scheduled task to run periodically and simply grab the contents and offload them - then overwrite the existing file contents. The big issue there would be file size. The task would need to run often enough so that it doesn't have to manage very large files. Still, that could be a nice automated solution. I like it - thanks for the input.
  • Steven Erat's Gravatar
    Posted By
    Steven Erat | 5/3/06 11:20 AM
    >> "I suspect that Coldfusion reads the entire file into memory before writing it out to disk."

    Ok, another test using the same set I commented about in your related post.

    I created a 200MB coldfusion-out.log file (not 2GB this time just so the motherboard wouldn't explode), then wrote a script to copy the log to the temp directory.

    The script ran for about 30 seconds, and the memory for ColdFusion did not substantially increase. I piped top to a file to capture the system and process memory usage (using pstree -p | grep cfmx7 to find the parent process of the ColdFusion server, the second pid in that tree). Here's the file:
    http://www.talkingtree.com/blog/downloads.cfm?item...

    So file copies (on Linux) do not impact ColdFusion server memory usage.
  • Big Kev's Gravatar
    Posted By
    Big Kev | 5/3/06 1:00 PM
    Could you not build a Custom Gateway that montiors the log files and if a file goes over xmb then back it up and nam eit using say server-err_YYMMDD-HHMMSS.log

    Just an idea?
  • Mkruger's Gravatar
    Posted By
    Mkruger | 5/3/06 1:06 PM
    Kev,

    that's a great idea - a bit overkill perhaps, but still - I like it.

    -mk
  • Russ Michaels's Gravatar
    Posted By
    Russ Michaels | 5/4/06 7:14 AM
    A windows scheduled task which calls a btach file, which makes a copy of the file would be the solution. Better than doing it manually.
    If you set the scheduled task to run with the same permissions as CF, it should also be able to write the new blank file as well.
  • mkruger's Gravatar
    Posted By
    mkruger | 5/4/06 7:34 AM
    Russ, ... Clever - I hadn't thought of that.
  • Big Kev's Gravatar
    Posted By
    Big Kev | 5/4/06 7:43 AM
    I had thought of doing that but couldn't find a way to zero length the files while still locked?
  • Matthew Plunkett's Gravatar
    Posted By
    Matthew Plunkett | 5/4/06 9:43 AM
    Don't use ColdFusion for this at all. Write a program in CMD, perl, whatever your system programming language of choice is, etc, and then wrap it as a service with srvany.exe. Cleaning up old files is a problem that isn't specific to ColdFusion.

    Even better would be to have this app run on some internal server with domain privileges, and have it clean up multiple servers.
  • Matthew Plunkett's Gravatar
    Posted By
    Matthew Plunkett | 5/4/06 9:55 AM
    If you are running Windows 2000, it is not a good idea to allow the Windows Scheduler to run. It is a pretty large security vulnerability, easy to run escalation of privilege attacks. Not so much of a problem in 2003 though.
  • Patrick McDonough's Gravatar
    Posted By
    Patrick McDonough | 11/2/06 1:27 PM
    I tried this and an interesting thing happened. The code worked properly and replaced the coldfusion-out file however after a few moments the file swelled back to 383 MB. Any ideas on why that may happen?
  • Patty's Gravatar
    Posted By
    Patty | 11/3/06 3:28 PM
    I had a similar experience as Patrick McDonough.
    I overwrote the file 'coldfusion-out.log' (which changed the filesize from 1.1Gb to 1Kb) but it came back the next time coldfusion wrote to the file (right back up to 1.1Gb). It's almost as if coldfusion remembers where it was saving to the last time it wrote to the file and writes to that same location again - instantly restoring the original filesize.
  • mkruger's Gravatar
    Posted By
    mkruger | 11/3/06 3:47 PM
    Dang... I'll look into it. but your comment (Patty) makes some sense... if CF was holding onto an EOF marker and appending to it - that would make sense. Can you (Patty or Patrick) tell me what version of CF you were using?
  • Patty's Gravatar
    Posted By
    Patty | 11/3/06 4:26 PM
    My CF Version:
    Server Details
    Server Product ColdFusion MX
    Version 7,0,0,91690
    Edition Enterprise (DevNet)
  • Redtopia's Gravatar
    Posted By
    Redtopia | 12/9/06 1:10 PM
    I have the same problem... my log file regains its original size after I make the cffile call.
  • Patty's Gravatar
    Posted By
    Patty | 3/5/07 1:42 PM
    Run the Latest Hotfix for Coldfusion to fix the problem:
    http://www.adobe.com/cfusion/knowledgebase/index.c...
  • Redtopia's Gravatar
    Posted By
    Redtopia | 4/2/07 6:24 PM
    That fix is for MX 7... my server is still running 6.1, which I have not found a patch for.