my workflow does not work when processing large amounts of text.

[SummerChill]

I designed a workflow myself, mainly used to process the information output by the mysql client in the clipboard, separated by the passed in separator. When the text is relatively small, the work is very fast, but once it exceeds a certain line count, there is no response.

My workflow design is as follows, mainly using "Run Script" components, which use pyhon code.

Does the workflow in Alfred have restrictions on processing large amounts of text?  what should I do?

 

Other:

masOS Catalina 10.15.7

alfred:4.2.1

 

 

 

 

 

 

 

 

 

[deanishe]

51 minutes ago, SummerChill said:

Does the workflow in Alfred have restrictions on processing large amounts of text?  what should I do?

 

Alfred doesn't impose any restrictions. It's impossible to say for sure because you haven't provided the workflow and problematic data for us to run, but if it's a large amount of data, it's possible that an IO buffer is full. Try reading the output from pbpaste line by line instead of trying to load it all into memory at once.

[SummerChill]

Hi @deanishe:

Here is my workflow file and my test data:

Link: https://pan.baidu.com/s/1nRav43NhuTULNH2WxC5saQ   password: vvmi

In this file, there are 642 rows of data and 3 columns.These data are processed normally using workflow, but if one more row is added, the workflow will have no respose.

The usage of the workflow is to copy the file in the text, and then enter "rowhandle + delimiter specified by yourself" in alfred.

 

==============

I finally find out the reason..

because the pipe size is 64KB limit, In my python code, to get clipboard content function part ,  process.wait() is used.

Thank you very much for your help, Best Wishes!

 

 

 

 

 

 

 

 

 

Edited December 1, 2020 by SummerChill

[deanishe]

1 hour ago, SummerChill said:

because the pipe size is 64KB limit, In my python code, to get clipboard content function part ,  process.wait() is used.

 

Good guess by me there

 

Would you consider posting your code for incrementally reading from a subprocess for people who run into the same problem in the future?

[SummerChill]

13 hours ago, deanishe said:

Would you consider posting your code for incrementally reading from a subprocess for people who run into the same problem in the future?

I just learned python, and I used the method provided below to get the contents of the clipboard in python.

https://stackoverflow.com/questions/43860227/python-getting-and-setting-clipboard-data-with-subprocesses

import subprocess
def getClipboardData():
    p = subprocess.Popen(['pbpaste'], stdout=subprocess.PIPE)
    retcode = p.wait()
    data = p.stdout.read()
    return data

The problem is this code "retcode = p.wait()"  

I checked the official documentation of python, It says that "This will deadlock when using stdout=PIPE and/orstderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data.  Use communicate() to avoid that."

I don't know the use of this code,  I finally deleted this line of code and there was no problem.

I hope this problem I encountered can help others.

 

 

 

 

 

[deanishe]

2 hours ago, SummerChill said:

I don't know the use of this code,  I finally deleted this line of code and there was no problem.

 

Yeah. Your code will work because p.stdout.read() won’t exit till all the pbpaste data has been read, but it doesn’t actually wait for the program to exit (which it should and you should normally check that it didn’t throw an error).

 

The code using communicate would be:

 

import subprocess

def get_clipboard_data():
    p = subprocess.Popen(['pbpaste'], stdout=subprocess.PIPE)
    data, _ = p.communicate()
    if p.returncode:  # pbpaste exited with non-zero status
        raise RuntimeError('pbpaste exited with: %d' % p.returncode)

    return data

 

Edited December 2, 2020 by deanishe

[SummerChill]

@deanishe I understand, thank you for explaining the principle of the code to me.😀👍