Jump to content

Is there a limit to environment variable quantity?


Mingwei

Recommended Posts

I'm trying to create a script filter that dynamically generates workflow environment variables on each rerun. The number of variables can get quite large, typically around 300+. I've not been able to call the variables after passing them, but I'm quite sure my script filter JSON is formatted correctly (see below for an example). Is the problem the number of environment variables I'm trying to set?

 

Related to this: Am I right to assume that previously set environment variables are not automatically unset when I pass a new set of variables that do not include them on a subsequent rerun?

 

Example JSON:

{
    "rerun": 2,
    "variables": {
        "71114": 369018953,
        "66776": 7091547666,
        "528": 1188619262847,
        "654": 9873755229,
        "71680": 69060795,
        "70589": 289630428,
        "70712": 26009830573,
        "589": 978183848817,
        "625": 261194160090,
        "2390": 12221103447,
        "71065": 624306639048,
        "67866": 1229316369,
        "70427": 9739510787,
        "752": 252220960370,
        "71585": 147073393,
        "64861": 18919027172,
        "175": 336645919469,
        "67750": 439291658,
        "2391": 3801114097862,
        "71115": 35413996,
        "688": 39562066662,
        "67610": 561323856,
        "70713": 5900752114,
        "56812": 16642660773,
        "59248": 791372000125,
        "754": 102640709518,
        "629": 907869918301,
        "69220": 22712486493,
        "70758": 9072044678,
        "67609": 257898324,
        "727": 297295601577,
        "67611": 4603671846,
        "64695": 10199393455,
        "64700": 545706904869,
        "25863": 40562974624,
        "728": 9128981472,
        "71067": 28075681,
        "2394": 120283103934,
        "70547": 12186120352,
        "729": 63155527498,
        "71163": 33315621,
        "71281": 132950770,
        "25696": 22579543291,
        "67634": 756496911,
        "67159": 35604950316,
        "1274": 769329740,
        "66261": 14347537664,
        "63774": 1617932835,
        "70358": 165023076,
        "71164": 13229162,
        "67847": 6072538920,
        "71070": 48162710,
        "71120": 1914524805,
        "541": 184061231008,
        "64675": 2282215683691,
        "2398": 2769403599,
        "70595": 36009674597,
        "542": 207137122531,
        "50660": 11896548786,
        "71589": 1690818387,
        "67848": 112795628765,
        "71071": 239428198,
        "71237": 59626195,
        "2386": 838073975103,
        "67708": 6421800433,
        "2399": 73789359479,
        "71591": 480641121,
        "544": 137084210155,
        "71121": 185427787,
        "640": 62987972683,
        "68179": 25875047786,
        "70907": 59574734369,
        "610": 48842444746,
        "1309": 1061183502,
        "2374": 50959080591,
        "575": 1668668864963,
        "64252": 62679400156,
        "2405": 176057209062,
        "641": 12325618751,
        "6699": 63826169024,
        "71451": 8527346476,
        "546": 935556953066,
        "66715": 10089254807,
        "71072": 13750250,
        "56791": 74720968651,
        "70316": 6929901282,
        "2406": 34243853750,
        "67615": 149195016,
        "71592": 2192282125,
        "57433": 1030840303702,
        "578": 2953770018714,
        "67588": 650618255,
        "67923": 659736294,
        "7254": 403858472,
        "64677": 425511208300,
        "2389": 450048076198,
        "579": 2044050672972,
        "2407": 8079028895,
        "549": 15542068733,
        "71100": 566914210552,
        "63615": 254528580195,
        "59327": 61344550069,
        "65247": 1472468698,
        "67638": 603198686,
        "70647": 128110357,
        "67662": 307527307,
        "67616": 2862886462,
        "71593": 118739910,
        "67852": 343303235,
        "2378": 1382801477292,
        "69226": 150935026,
        "70598": 92359668,
        "4053": 198430500,
        "12775": 14241542463,
        "12021": 67316061669,
        "67685": 753747513,
        "66505": 5475011188,
        "67969": 27296507869,
        "745": 22203257798,
        "68349": 652152798,
        "71074": 129132667,
        "66953": 5141514814,
        "70342": 7380274566,
        "67617": 643576734,
        "71594": 160901100,
        "67947": 35562773756,
        "57435": 450255868,
        "71690": 57181002,
        "717": 223271404436,
        "64679": 31100962480,
        "70722": 1277240126,
        "71097": 22322585589,
        "71102": 115375607,
        "71220": 493987411,
        "2520": 13832749697,
        "59375": 49924713364,
        "66670": 22680053688,
        "56794": 30070275434,
        "64681": 31241928139,
        "67876": 56835538817,
        "70555": 318967337,
        "71053": 4427853611,
        "942": 172311100438,
        "71691": 103594999,
        "70723": 740746034,
        "67904": 2322232363,
        "2522": 59046062592,
        "56795": 10531107289,
        "12519": 3292170848025,
        "71623": 134388698,
        "2548": 1083687076,
        "65703": 24735747490,
        "71054": 648608231,
        "70628": 11969658073,
        "68353": 1206769001,
        "976": 13021255813,
        "560": 23958309655,
        "2671": 10835283022,
        "70724": 471055731,
        "71104": 26813722,
        "67951": 34219569512,
        "71099": 4113173217,
        "64799": 13349943687,
        "977": 114131122672,
        "561": 77949149767,
        "71077": 32713393,
        "592": 179761929082,
        "59261": 2133549421,
        "978": 566097917453,
        "532": 214254747560,
        "71055": 83164588,
        "70629": 90286787,
        "979": 25269174898,
        "563": 507513574471,
        "27461": 6297770146666,
        "1239": 402998372611,
        "67622": 4061394924,
        "564": 141107040468,
        "71647": 1860042410,
        "68332": 22486559545,
        "57440": 110731840109,
        "67689": 4461036686,
        "2513": 10727910360,
        "68399": 598754046,
        "56797": 5018793480175,
        "56802": 5734847990,
        "71625": 144410122,
        "1769": 28202523634,
        "15574": 9883062488,
        "630": 187675748352,
        "71078": 1054732296,
        "66627": 8133864114,
        "70558": 78290879,
        "67031": 34239921497,
        "68377": 77729452,
        "71603": 130996739,
        "660": 229897277936,
        "662": 73256705970,
        "67645": 2044431538,
        "632": 233805492309,
        "71080": 83755331,
        "25924": 68577684607,
        "70324": 1366048484,
        "567": 352262973683,
        "71576": 14309917084,
        "12713": 3152884701,
        "57439": 8381979949,
        "70726": 499825491,
        "71648": 272559198,
        "71224": 3061246636,
        "3458": 1276086185188,
        "694": 7823815633,
        "538": 731824456436,
        "2528": 4143702353336,
        "66580": 27989508341,
        "66628": 19311763137,
        "730": 51429669296,
        "604": 27340695259,
        "695": 116833096002,
        "59263": 1538454473,
        "67998": 25408810946,
        "71650": 102437227,
        "68501": 697833331,
        "71604": 121105492,
        "605": 93039320385,
        "539": 745823049035,
        "2516": 27843511136,
        "636": 93843970776,
        "67670": 13753502157,
        "70985": 171640800,
        "12714": 20042237167,
        "697": 23255572020,
        "70727": 527869860,
        "71649": 65015879,
        "67200": 9899328801,
        "637": 812864986587,
        "67624": 1050923313,
        "67597": 3737231881,
        "67602": 58622464562,
        "15576": 407223717,
        "67033": 407547170095,
        "71605": 123027526,
        "609": 26768313813,
        "66843": 6450138665,
        "67671": 1479549479,
        "71272": 3575028719,
        "71060": 61646917,
        "70728": 37986537,
        "70304": 6049005613,
        "67603": 14329354609,
        "15577": 4333511684,
        "70400": 5698458297,
        "59265": 1127879779,
        "71606": 699880578,
        "71059": 3658356876,
        "67648": 1329105859,
        "71083": 172683881,
        "67626": 1208566426,
        "67956": 14322885874,
        "25594": 103358736970,
        "67650": 377107965,
        "2508": 190516072884,
        "70541": 22560196613,
        "3395": 21517105909,
        "67599": 4592604708,
        "71607": 11859024291,
        "25880": 4770156809,
        "71301": 41804323956,
        "2509": 19051744558,
        "71084": 27626525,
        "68147": 41350042330,
        "4726": 8508175253,
        "26002": 133312380803,
        "1332": 1025289560,
        "67957": 32476690680,
        "71062": 21024491,
        "70518": 135032201,
        "67651": 84109249,
        "57445": 879929403,
        "70542": 3710428168,
        "25879": 978573829190,
        "580": 14929076147,
        "67605": 313814810,
        "66160": 28226804188,
        "71608": 314287592,
        "64691": 4644953039,
        "582": 2350107662660,
        "70989": 22832365,
        "2455": 895469909,
        "456": 758874992392,
        "70849": 454783610,
        "67864": 121632363294,
        "57446": 111295263660,
        "553": 114455820583,
        "939": 39946568444,
        "523": 374059355049,
        "66585": 9168789486,
        "7334": 40670443948,
        "70403": 7841795359,
        "67630": 24344729633,
        "71609": 100989871,
        "585": 147045994983,
        "70588": 3997793068,
        "70711": 93424997,
        "71633": 396749651,
        "621": 375226396400,
        "64858": 1258026862,
        "586": 424583970545,
        "67251": 6459717519,
        "556": 69946306748,
        "64764": 48049491043,
        "622": 2564520656895,
        "70544": 113695255,
        "67607": 1919992603,
        "623": 284508299880
    },
    "items": [...]
}

 

Link to comment
9 hours ago, Mingwei said:

I'm trying to create a script filter that dynamically generates workflow environment variables on each rerun.


Why? What are you trying to do? That solution is pretty atypical, so let’s avoid an XY Problem.

 

9 hours ago, Mingwei said:

I've not been able to call the variables after passing them


Call how? Without knowing what action you’re taking it’s not possible to say if it is correct. What does the debugger say?

Link to comment
1 hour ago, vitor said:

Why? What are you trying to do?

I'm trying to generate a refreshing list of system processes organised by cpu usage percentage. `ps` would be an obvious way to do this, but I think the cpu% figure it outputs represents cpu time used over the entire runtime of the process. In contrast, I want to get usage time over a short sample period (the rerun interval) so that sudden spikes in cpu usage are reflected, much like in Activity Monitor. To do this I have to calculate the amount of cpu time used by each process between script filter reruns, divided by the rerun interval. At the moment I'm trying to do so by saving the amount of cpu time used by each process in a variable on each script filter run. Since this means one variable for each process, it easily adds up to hundreds of variables. 

 

As for why... no particular reason haha. I'm doing this for fun, and to learn more about how system stats are calculated and how APIs like libproc or kinfo_proc work. Likely won't end up using the resulting workflow much, though I can imagine using it in place of activity monitor to kill an unresponsive process if it's faster (so far it is). It's ok if it's not possible, but if there's an alternative way to do this / a mistake I've not found that would be great. 

 

2 hours ago, vitor said:

Call how? Without knowing what action you’re taking it’s not possible to say if it is correct.

I'm not using the variables in a subsequent workflow action, just in subsequent reruns of the script filter. The key is each process's PID, I just retrieve the environment variables in a loop. The swift code retrieving the variables is something like `guard let cpuTime: String = ProcessInfo.processInfo.environment[pid] else { return }` but I'm quite sure it's not the problem, have used it without issue elsewhere. 

 

2 hours ago, vitor said:

What does the debugger say?

I tried echoing the variables in a Run Script action to see what would happen. I ran `echo ${(P)pid}`, where pid is a variable created by the actioned script filter item containing the process's pid (which is in turn the name of a variable containing the process's cpu usage time). The output in the debugger log seems to vary depending on how many items there are.

 

When there's less than 300 items (roughly, not entirely sure where the boundary is), I get this: 

[13:51:10.955] Search Processes[Script Filter] Processing complete
[13:51:11.021] Search Processes[Script Filter] Passing output '/path/to/some/process' to Run Script
[13:51:11.028] Search Processes[Run Script] Processing complete
[13:51:11.031] Search Processes[Run Script] Passing output '
' to Post Notification

 

When there's more, I get this: 

13:50:36.827] Search Processes[Script Filter] Processing complete
[13:50:36.912] Search Processes[Script Filter] Passing output '/path/to/some/process' to Run Script
[13:50:36.914] Search Processes[Run Script] Processing complete
[13:50:36.916] Search Processes[Run Script] Passing output '/path/to/some/process
' to Post Notification

 

Not really sure how to interpret either output, but the fact that it just passes the item arg along when the number of variables exceeds a certain number is what made me think the problem might be too many variables.

Link to comment

Just tried something else, and it seems Alfred might not be able to set variables with numeric names? I created a test workflow that sets a variable with a numeric name (e.g. 12345) using the Arg and Vars utility and tried to output it via Run Script. I received an empty output, just like in the example above: 

 

[15:00:39.790] numeric varname test[Keyword] Processing complete
[15:00:39.790] numeric varname test[Keyword] Passing output '' to Arg and Vars
[15:00:39.794] numeric varname test[Arg and Vars] Processing complete
[15:00:39.797] numeric varname test[Arg and Vars] Passing output '' to Run Script
[15:00:39.868] numeric varname test[Run Script] Processing complete
[15:00:39.886] numeric varname test[Run Script] Passing output '' to Large Type

 

But when I set the variable in the Run Script action itself, as in `12345=hello` or `12345=37283` it works as expected. 

 

I then tried making the names of my dynamically generated variables in the original workflow non-numeric, by using process names instead of PIDs. After doing this, I had no issues retrieving the environment variables and calculating cpu usage. 

 

This isn't really a solution though, because process names don't identify their processes uniquely and this throws off the calculation whenever there are multiple processes with the same name. I'm also still not able to retrieve environment variables when their number exceeds around 300+, so I think it's likely the problem I was facing was caused by a combination of too many variables & Alfred not being able to set variables with numeric names. 

 

Really sorry for the long posts - I realise this is a really niche problem that won't affect most use cases, so I don't want to take up too much time. Just want to document this here, and if I'm not missing any obvious alternate causes / there are no obvious workarounds I'm happy to put this workflow aside. 

 

Link to comment

UPDATE: Finally figured out what the actual problems are, and it's not what I described above (oops!) Documenting it here in case anyone else experiences similar issues: 

  1. Alfred isn't able to set variables with names that start with a digit, but variable names can contain digits. This might be a system limitation? Added an underscore in front of each numeric variable name, now they can be set without issue. 
  2. There's no hard limit on environment variable quantities (or at least I didn't hit it). I observed the workflow working when there were less than ~300 items and failing when there were more because I happened to be testing quantities by toggling between showing all processes (which requires running the script filter from root) and showing only user processes. Of course, Alfred is only able to set environment variables in the user scope, which of course were not in my environment when running as root. 

In retrospect, 2. at least should have been obvious. But I guess I needed to think out loud (and very publicly) on this forum to figure it out haha 😅

 

 

Link to comment
9 hours ago, Mingwei said:

I'm trying to generate a refreshing list of system processes organised by cpu usage percentage.

 

You should go ahead since you’re doing it for learning, but take a look at Process Control.

 

2 hours ago, Mingwei said:

Alfred isn't able to set variables with names that start with a digit, but variable names can contain digits. This might be a system limitation?

 

Yep, it’s not an Alfred thing. Programming languages in general disallow that as well.

 

2 hours ago, Mingwei said:

But I guess I needed to think out loud (and very publicly) on this forum to figure it out

 

Rubberducking!

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...