OL Learn

Workflow spool files cleanup

I have a process that creates a large number of files in the process spool folder. It appears that when it finishes the process and returns to the Folder Capture step it doesn’t clear this out.

With the way this job runs I’ve noticed that it can cause the Workflow Service to stall and not be able to access the file system.

Has anyone else seen this happen before? When does the spool folder clear out - after each pass or only if there are no more files in the Folder Capture plugin?

Is it possible to manually clear the spool folder to stop this error without causing any other issues?

Once a process has gone through all its tasks, it always goes back one last time to the initial input task in order to determine if any new files have come in since it processed the last batch. If there are no such additional files, the process truly stops and cleans up after itself. But if a new batch of files are found, then they are processed immediately without first cleaning up the spool file folder… and then the same operation repeats itself, with the process returning to the initial input task to check for more files.

On an extremely busy system, this means the spool folder can rack up quite a large number of files before it gets cleaned up.

You can’t clean the folder manually because you might be deleting a file that is currently being processed. However, if you know for sure that no processing is occurring and yet find that you still have a large number of files in the spool folder, you can definitely delete them.

Thanks. That makes a lot of sense as to why we are having issues.

I have a process that splits a number of XML files into over 100,000 records. When the XSLT split runs it creates each file in the spool folder. At some point the watch process stops being able to process jobs. It still tries to run but has OS file access errors.

I might have to take this process out of watch or call a script and handle it all externally. Not optimal.

You may just need to re-work things a bit now that you know how it works.

For example, step back and think why you’re splitting that XML file in the first place. If you’re running it through a Connect template, perhaps it would make more sense to run the unsplit XML file through and split the output through an Output Preset.