We have a monthly process where we produce tens of thousands of invoices using csv files, using PlanetPress to generate the letters has been immensely helpful however in order to keep a track of the letters being produced and also to cut down on processing time per batch our print room operative currently manually sorts and splits our initial data.
In other words, we’ll have an initial data set of 40-80 thousand lines of data, and our operative will:
- Split the data by invoice type (the type is indicated in the data)
- Sort each invoice type by account reference (the reference is indicated in the data)
- Split the data again into separate csv’s of no more than 1000 records each.
I understand that PlanetPress can sort the data in this way between the datamapper and the print output, however there are two problems with just trying to process the original 40-80 thousand line csv in one go and letting PlanetPress sort it ‘behind the scenes’:
- Firstly, it takes PlanetPress hours to process the data in this way, and while this would be okay if we’re able to process it overnight, PlanetPress often crashes for us if we give it too much data to process at once so we can end up coming in in the morning with no work ready to print.
- Secondly, if a job of 1000 gets mangled in the printer (rare, but it happens), although we could make an educated assumption of what 1000 they would be if we tried to mimic PlanetPress’ data sorting, there’s no way for us to guarantee which 1000 records were in that job and which therefore need to be re-processed.
I was wondering therefore if there was a way to use PlanetPress to split the csv before it is processed, so that we can identify each batch of 1000 invoices separately without our operative having to manually sort and split the original data. I vaguely understand that if I set up a process in Workflow for this I need to mark each field as a local variable and turn that into metadata, and at the end of the process I need to create a file, I’ve tried playing with this idea but I’ve not at all been able to get it to do what I want.
If I can get this working it would be ideal if I could get PlanetPress to name each split csv file with a batch number, and also include the batch number in an additional column within the split data to allow us to add it into the print job name for easy reconciliation.
Would someone be able to point me in the right direction to get this idea to work? Assuming that it’s something PlanetPress is able to do?