Upland OL User community

Schedule a Process

Is there a way to run a Process only in a given schedule ?
I have tried to set up a schedule to run only between 00.00 - 03.00 - se picture
I have put 10 mio csv files in the input folder.
The proces starts at 00.00 and processes the files, but it does not stop at 03.00 - it just contiunes when there is still files in the input folder.
The input folder is the start folder in the proces.
This proces puts the CPU to 100% and takes around 5 days if it runs 24/7 - so I can only run durring nighttime - the server needs to be used for something else in day time.

If the server runs 24/7 with the current proces - PP Workflow crashes with disk error - even though there is plenty of disk. PP Version 2023.1

That’s not how Workflow works. The Scheduler tells the process when it should run the initial process input, but that input task will not stop until it has completed its job, even if that means it needs to keep running outside of the allotted time period in the scheduler.

I think I have a pretty good idea of how to achieve what you’re asking for, but it will require some testing.

Essentially, you have to use a Create File task as your input. Contrary to the Folder Capture task, this task only runs once (whereas the Folder Capture task runs until there are no more files to capture). It will run again according to the polling interval set in the process properties, and only in the time period specified.

The Create File task dynamically generates a PowerShell script. That script uses the values in local variables %{Folder} and %{FileMask} to get the list of the first 100 files in the target folder (using PowerShell to do this is much more efficient that using the folder listing task, or the DIR command, because they both list all the files, whereas PowerShell is able to list only the first n files.

The script file that was generated is then renamed with a PS1 extension, because that’s the only extension PowerShell accepts for batch commands. The script is then executed. It generates a list of 100 file names (or less), and that list is stored in Workflow’s current working folder under the name LogFiles.txt.

The next task loads LogFiles.Txt as the new data file, and then a Splitter task is used to split the file on each line. That splitter task (and all the tasks under it) will run as many times as there are lines in the LogFiles.txt file.

The next task is the actual start of your processing: it’s a Folder Capture task that captures the file name from the split data file.

Admittedly, it’s all a bit convoluted, but it should achieve what you’re looking for.

The worst case scenario would be that the process is triggered exactly at the end of the scheduled period, in which case it would not process more than 100 additional files before stopping and waiting for then next polling period to begin.

Here’s what the process looks like:

Here is a link to a sample Workflow configuration. Note that is has only been surface-tested, you will have to tweak it somewhat, but I think it should get you started in the proper direction.

Folder Capture process running only during scheduled time.OL-workflow (24.3 KB)

Final note: this was a fun challenge! I think I may write a blog article to describe this technique in more detail. Thanks for the inspiration!

Thanks again Phil
Used you exampel a while ago and modified, with the number of files it should pickup.
The only strange thing is that I get a 3 special chars infront of the first line in the filelist, so I used a script to replace them. It is the same chars every time - logfile = logfile.replace(“”,"");
Don’t know where the come from.


Use the Add/Remove Text plugin to remove them. This is because your file is a UTF-8 BOM