Upland OL User community

Data Mapper never completing V2023

#1

Hi

We have any issue which is been investigated by support where by when you send a large ish file through a data mapper via workflow the mapper never seems to finish. This is causing issues on several jobs for us which used to work fine on V2021.

Has anyone else come across this issue also as I’m surprised it’s not been raised by anyone else. What is confusing however if that on our development server the jobs work fine, however this was upgraded from 2021 where by the server we are having problems with was a fresh install

James

#2

As far as I know, we haven’t received reports to that effect. What type of data is it (CSV, XML, PDF, etc.) and what constitutes “large” in terms of MBs?

#3

@Phil I’ve got the issue happening on a XML file which is 100mb, a CSV which is 13mb and a PDF that is 200mb. So doesn’t seem to be limited to a certain data type or anything like so was just surprised no-one else was having any issues.

Apparently it’s when it’s saving the metadata back to the workflow, however we’ve changed the mapper to only output ID’s and then put a restore items straight after the mapper as we need the metadata for a script after the mapper, but it’s made no difference

#4

That’s going to require some investigation in the Connect logs. You might want to open a ticket with our Support team, they are used to this kind of stuff.
But you can already take a peek at the DataMapper log (C:\Users\[USER]\Connect\logs\Datamapperengine) after running one of those jobs and see if you can find anything in there. Normally, after having run a DM operation, the log contains something like:

[2023-11-10 07:43:10,271] INFO  [IPC Input] com.objectiflune.datamining.engine.product.internal.DataminingEngineCommandProcessor.a(DataminingEngineCommandProcessor.java) Estimated number of records: 4 (Dm_1734)
[2023-11-10 07:43:10,311] INFO  [IPC Input] com.objectiflune.datamining.engine.product.internal.DataminingEngineCommandProcessor.a(DataminingEngineCommandProcessor.java) PROGRESS: 25 (Cc_118)
[2023-11-10 07:43:10,333] INFO  [IPC Input] com.objectiflune.datamining.engine.product.internal.DataminingEngineCommandProcessor.a(DataminingEngineCommandProcessor.java) PROGRESS: 50 (Cc_118)
[2023-11-10 07:43:10,347] INFO  [IPC Input] com.objectiflune.datamining.engine.product.internal.DataminingEngineCommandProcessor.a(DataminingEngineCommandProcessor.java) PROGRESS: 75 (Cc_118)
[2023-11-10 07:43:10,399] INFO  [IPC Input] com.objectiflune.datamining.engine.product.internal.DataminingEngineCommandProcessor.a(DataminingEngineCommandProcessor.java) DataMapper completed for datasetID: 3,278,351 (Dm_1733)
[2023-11-10 07:43:10,403] INFO  [IPC Input] com.objectiflune.datamining.engine.product.internal.DataminingEngineCommandProcessor.a(DataminingEngineCommandProcessor.java) PROGRESS: 100 (Cc_118)
[2023-11-10 07:43:10,407] INFO  [IPC Input] com.objectiflune.datamining.engine.product.internal.DataminingEngineCommandProcessor.run(i.java) DataMapper command completed (Dm_1708)

If you find that all jobs appear to complete properly on the DataMapper side, then the issue is likely to occur on the Workflow side when retrieving the results.