Plus Feature Request: The export file path is always static. I'd like to have the possibility to create a dynamic file path with properties like File path = C:\Server\PublicFolder\Ek_Export\{CUSTOMER}

Today I use this function in a workflow, but the problem is that the export for every workflow is sent to the same folder.

Cheers

Daniel

Hi Steve,

Except for the standard limitations, there are no specific ones tied to EK.

Kind regards,

Mihovil

Hi Mihovil

We need a solution here. I get responses from people that they can't use our created "Export-Workflow" because it only can export into one fixed folder AND it needs way too much time.

We are talking about 11 Minutes for one document. After the process was finished it was 14 hours for 65 Files / 43 MB.

Export via client a thing of a second / copy & paste.

I think there is a problem. Could you please check?

Cheers

Daniel

Hi Daniel,

Yes, I can understand that performance issues can make use of this module not as effective. I've tried to replicate Export functionality and I've found that there aren't performance issues even with very large files (over 30 MB). Of course, my configuration and environment cannot be the same as yours (or any other user experiencing these issues), but it proves that Data Transfer can work fast even with large files.

So, I've tested two scenarios:

1. User has 3 large DOCX files (Class: Document) and wants to export all three into a folder on his C: drive

 - All three files are saved to local folder under a second

2. User has 3 large DOCX files (Class: Document) and wants to export specific file when that file enters Export state in Workflow

- Specific file is saved to a local folder under a second as well

Note: for testing purposes you can download large DOCX from https://www.learningcontainer.com/sample-docx-file-for-testing/

You can try to recreate those scenarios on freshly set up vault using default Document class, with latest officially released EK (5.0.0). If you still have performance issues, I would suspect that there is a problem outside EK. Or, if you want to test it on existing environment, you can use said rules on Document class provided it has the same structure as default.

If the performance issues persist I would suggest restarting the server and the vault or trying to run Data Transfer on objects that are not being affected by other EK rules.

Regards,

Mihovil

Hi Mihovil,

I'm not sure if we've understood each other correctly. The problem isn't that we're using large files, but the number of documents processed in the export. We are conducting three exports simultaneously, and this process has already been ongoing since last Friday.

Please check the JSON configuration file again. I would also like to suggest that we look at this together in a Teams session. This way, the technical situation will surely become clearer for you, and we can make progress more efficiently and satisfy our mutual client. This function is essential for us, and we urgently need a solution to this problem.

Best regards and thanks in advance
Steve

Hi Steve,

I've revisited your original post and I have some comments.

Firstly, all three rules have the same trigger, which is not uncommon. The only thing to keep in mind is that when the object of class "CL.GF.Export", and the workflow "WF.Export" enter the state "state.Export.StarteExport", all three rules will be set to execute in background in one batch, in the order in which they have been defined in the module, so:

#1: OT Geschäftsfall: CL Export - Revision / Buha-Akt
#2: OT Geschäftsfall: CL Export - Revision / Bankbelege
#3: OT Geschäftsfall: CL Export - Revision / Beilagen

A potential issue can occur when there are consecutive triggers occurring; in that case, multiple batches of said rules will be run, and each rule will overwrite a previously exported file.

This increases the total time of export until the last rule finishes its course.Please check if there are multiple triggers occurring in your case. If so, I would suggest that you set the parameter "Overwrite" to "No" in Export settings. This will ensure that multiple instances of running rules skip saving the file if it already exists. If you need this option, do not change it.

Secondly, "OT Geschäftsfall: CL Export - Revision / Buha-Akt" has very loose search conditions for export. For example, it doesn't have Class condition, which means that there can be many files that will be exported regardless of class. And if so, it will delay the execution of the other two rules.
Please check if that condition is missing in Export settings - Objects to Export - Search - Property conditions.

You mentioned that export is running for days now, could you screenshot the "Data Transfer" and "Asynchronous operations" sections in EK Dashboard and attach them?
Also, how many files are you wanting to export (approximately) for each rule? 
Could you browse for logs on your server and attach them for further analysis?

For testing purposes in a non-production environment, I would suggest configuring these rules as "Run once", removing Trigger Source completely and running them through the Dashboard individually. That way, the rule will be triggered only once.

Kind regards,

Mihovil

Hello Mihovil,

Thank you for your response.

Indeed, using the same trigger for three exports is not unusual and in this case, it is necessary and required. As soon as we can use a Property-Setter with this module, we would cascade these exports and use different triggers for each.

Overwriting the files is not possible, as each export uses its own directory. Therefore, this could not be the cause of the problem.

The search conditions for the export "OT Business Case: CL Export - Revision / Buha-Act" are correct and also necessary. When I search manually with these conditions in M-Files Desktop, I immediately get the desired results. Therefore, I exclude search performance as an issue on my part.

The export was completed yesterday. However, I have just started it again (run once), and the log shows that only every few minutes a file is written to the file system. See an extract from the log Line 1&2 at 10:40 AM, then Line 8 at 10:46 AM. What is the reason for this? The export started at around 10:40 AM. By 11:27 AM, only 9 files were exported.

As I have already offered, I suggest we look at the problem in a joint Teams session. I feel that this way we could find a solution more quickly.

Attached is an extract from the log and the requested screenshots.

Thank you in advance.

Best regards
Steve

Hi Steve,

We are checking this issue with our development team and we will post our findings as soon as possible.

Kind regards,

Mihovil

Hi Steve,

We are still working on that, for now it seems that the problem lies in multiple rules triggered at once for that same object.

Could you please provide us with some additional information,
- Is performance good if you trigger these rules one by one?
- What is the approximate number of documents that these rules export?

Kind regards,
Tadej

Hey Tadej,

Posted by: @daniel-mislovic

↑

We are talking about 11 Minutes for one document. After the process was finished it was 14 hours for 65 Files / 43 MB.

For the second Question, see above. First one will Steve answer later on.

Cheers

Daniel

We changed to Orchestrator, this helped a lot:

Performance is acceptable.

Hi Daniel,

Apologies for my late response. I'm glad to hear that, I believe the orchestrator, in the end, is the best option for your case.

Regarding the dynamic value for the Folder path, it was added to our plan and it will be available in the next release of the Extension Kit.

Kind regards,
Tadej

Hi Together

After the M-Files and EK update. The performance is really bad. It takes houres again to export approx 100 Files instead of a few minutes.

See here:

And here:

Attached the configuration from the "Orchestration" module and from the "Data Transfer" module.

It is a critical task in our project. Could you help fast?

Hi Steve,

Looking at the configurations, there seems to be nothing wrong with them. Especially if you used them earlier without issues, they haven't been altered? If so, I would suggest that you look at the logs at the time export process begins, specifically at the objects affected by the export. If another rule (or rules) are altering the objects in question, that could hinder the performance. If those rules are present, I would suggest that you try to limit their trigger to specific property change, or move the trigger outside of Export state.

Furthermore, there could be an issue with the server's resources after prolonged usage time, if possible try to observe the CPU and memory consumption during these jobs. Vault restart and server restart could help with performance issues.

Kind regards,

Mihovil

Hello Mihovil

We checked the server ressouces. CPU, RAM or Disk acitivities are not close to the edge (less than 15 %).

What I see is that Task: 2058596 was started at 2024-10-22 07:04:10.7294.

And executed at 2024-10-22 07:28:09.1703.

Between this time only a few objects were processed. So close to no activities on the server. But a lot of acitivities in the Log. I am not able to evaluate the most of them.

For example:

2024-10-22 07:13:45.9252 [TRACE] MFiles.VAF.AppTasks.TaskManager [Task: ] [] Process loop cycle starting. Queues:Unitfly.ExtensionKit.VAF.VaultApplication.Rules,Unitfly.ExtensionKit.VAF.VaultApplication.Scheduled,Unitfly.ExtensionKit.VAF.VaultApplication.RunOnce,Unitfly.ExtensionKit.VAF.VaultApplication.Recalculate,Unitfly.ExtensionKit.VAF.VaultApplication.DailyPropertyRecalculation,Unitfly.ExtensionKit.VAF.VaultApplication.EmailNotifications.Digest,Unitfly.ExtensionKit.VAF.VaultApplication.EmailNotifications.MailboxListener,Unitfly.ExtensionKit.VAF.VaultApplication.NamedValueStorageCleanup,Unitfly.ExtensionKit.VAF.core.broadcast,Unitfly.ExtensionKit.VAF.extensions.scheduler,Unitfly-ExtensionKit-VAF-VaultApplication-BackgroundOperations

2024-10-22 07:13:45.9252 [TRACE] MFiles.VAF.AppTasks.TaskManager [Task: ] [] Tasks polled. Queues:Unitfly.ExtensionKit.VAF.VaultApplication.Rules,Unitfly.ExtensionKit.VAF.VaultApplication.Scheduled,Unitfly.ExtensionKit.VAF.VaultApplication.RunOnce,Unitfly.ExtensionKit.VAF.VaultApplication.Recalculate,Unitfly.ExtensionKit.VAF.VaultApplication.DailyPropertyRecalculation,Unitfly.ExtensionKit.VAF.VaultApplication.EmailNotifications.Digest,Unitfly.ExtensionKit.VAF.VaultApplication.EmailNotifications.MailboxListener

2024-10-22 07:13:45.9252 [TRACE] MFiles.VAF.AppTasks.TaskManager [Task: ] [] Process loop cycle completed. Queues:Unitfly.ExtensionKit.VAF.VaultApplication.Rules,Unitfly.ExtensionKit.VAF.VaultApplication.Scheduled,Unitfly.ExtensionKit.VAF.VaultApplication.RunOnce,Unitfly.ExtensionKit.VAF.VaultApplication.Recalculate,Unitfly.ExtensionKit.VAF.VaultApplication.DailyPropertyRecalculation,Unitfly.ExtensionKit.VAF.VaultApplication.EmailNotifications.Digest,Unitfly.ExtensionKit.VAF.VaultApplication.EmailNotifications.MailboxListener,Unitfly.ExtensionKit.VAF.VaultApplication.NamedValueStorageCleanup,Unitfly.ExtensionKit.VAF.core.broadcast,Unitfly.ExtensionKit.VAF.extensions.scheduler,Unitfly-ExtensionKit-VAF-VaultApplication-BackgroundOperations

2024-10-22 07:13:46.3002 [INFO] MFiles.VAF.Extensions.TaskManagerEx [Task: ] [] Cancelling future executions on queue Unitfly.ExtensionKit.VAF.VaultApplication.Scheduled of task Scheduled.

2024-10-22 07:13:46.3314 [INFO] MFiles.VAF.AppTasks.TaskManager [Task: ] [] Waiting tasks cancelled. Queues:Unitfly.ExtensionKit.VAF.VaultApplication.Scheduled TaskType:Scheduled Tasks:2058637 Transactional:False

Between start and end of this process less than 50 entries in the log for object 104-104. Fact is that it runs approx 24 minutes and I have no clue why. And I see no other rules than the orchestrated and in the beginning the title by module "extended auto properties" were executed.

I attached the Log. Can you help us to trace down the problem by teams instead of by ticked?

Kind regards
Steve

Please check there is a time of 6-7 minuntes between the exported files. There is a recognizable pattern for us. What could that be?

Kind regards

Hi Steve,

Can you please first set the file log in the Logging module like this? You can simply modify your current configuration - the important settings are the Categories and Layout.

{
"Enabled": true,
"MinimumLogLevel": "Trace",
"Name": "File",
"FolderName": "C:\\Unitfly\\ExtensionKit\\Tests\\Logs\\",
"Categories": {
"TASK_MANAGER": "Off",
"SCRIPT_HANDLERS": "Off",
"CONFIGURATION_UPGRADING": "Off",
"LIFE_CYCLE": "Off"
},
"Advanced": {
"Layout": "${longdate} [${level:uppercase=true:padding=3}] ${logger} [Task: ${mdlc:TaskID}] [${mdlc:ModuleName}] ${mdlc:Vault} ${mdc:Rules} ${message}${onexception:${newline}${exception:format=ToString:innerformat=ToString:separator=\\r\\n}}"
},
"LevelOverrides": [],
"FileRotation": {
"ArchiveAboveSize": 50000000,
"ArchiveType": "Day"
},
}

This way, the logs will be cleaner and contain all the details about rule execution.
After you set this, try executing the rule again, and then please send me the log file so that I can inspect it and test everything locally first. After that, we can arrange a call if needed.

Regards