List Filter is not respecting multiline fields in CSVs [Fixed 3.6.1 b907 pre-release]

[vitor]

CSVs allow for fields to be multiline

A1, A2, A3
B1, B2, "B
3"
C1, C2, C3

B3 is a multiline field. As long as it’s encapsulated in quotes the newline is included. However, if dropped into a List Filter, this is what happens:

 

As soon as the List Filter sees the newline, it assumes a new record. Multiline support is particularly relevant to the Arg field of List Filters, since those can have multiple lines if inserted manually.

 

Not something I’d personally need, but noticed when advising on another post.

Edited November 23, 2017 by vitor

[deanishe]

23 minutes ago, vitor said:

CSVs allow for fields to be multiline

 

I think Alfred/@Andrew should define a specific CSV dialect for List Filters, i.e. which delimiter(s) is/are acceptable and how fields need to be quoted.

 

By default, both Excel and Numbers use comma as the delimiter, " as the quotation mark and "" as an escaped quotation mark.

 

[vitor]

3 hours ago, deanishe said:

By default, both Excel and Numbers use comma as the delimiter

 

From what I’ve found in the wild (a couple of times, no more), Excel’s default (at least on Windows) seems to depend on a system regional setting, and not on Excel itself. The times when I bumped into this were when I was trying to help someone (someone needed some transformations on tabular data, that I could quickly do with a few commands) and it didn’t work immediately, because the delimiter was ;. Maybe that’s the default for Portugal.

 

3 hours ago, deanishe said:

I think Alfred/@Andrew should define a specific CSV dialect for List Filters, i.e. which delimiter(s) is/are acceptable and how fields need to be quoted.

 

No strong opinion there, as it’s not a feature I use. But I do think multiline support will be useful for the people that do.

Edited November 23, 2017 by vitor

[deanishe]

3 minutes ago, vitor said:

the delimiter was ;.

 

That's actually what I was expecting to see when I saved as CSV from Excel.

 

4 minutes ago, vitor said:

No strong opinion there, as it’s not a feature I use. But I do think multiline support will be useful for the people that do.

 

Definitely. I mean, not handling newlines within quoted fields is basically a bug.

 

I also think it makes sense to document what Alfred expects somewhere. The above-described dialect is the "standard" one, AFAIK, and how both Excel and Numbers save CSV files. Other apps, like LibreOffice, and libraries let you specify different delimiters and quote characters, however.

 

[Andrew]

I'm currently updating Alfred's CSV importer to adhere to RFC 4180 which is pretty well summarised on wikipedia:

 

https://en.wikipedia.org/wiki/Comma-separated_values

 

What's interesting is that in @vitor's original post here, he added spaces around the fields which are supposed to be significant (and likely will be in Alfred's updated CSV importer). In that wiki page, it says that some importers trim fields, and where spaces are significant, the fields should be quoted. I feel this may suit Alfred's purpose better, but then this deviates from the RFC 4180 spec.

 

Any thoughts?

 

Cheers,

Andrew

[vitor]

It feels more natural to me that the spaces are trimmed. However, I also think there might be users on both sides and saying “here’s the spec” makes more sense than “here’s the spec but with these changes”.

 

So I think it’s best to follow the existing spec.

[Andrew]

@vitor Alfred will still be trimming certain stuff after importing regardless of the CSV, such as the title and subtitle (because that makes sense, and Alfred generally trims where sensible), but the arg would not be trimmed as that's more significant.

[vitor]

Just now, Andrew said:

but the arg would not be trimmed as that's more significant

 

I think that’s fine, since the argument can still be trimmed in the Workflow without changing the CSV or resorting to coding (via the Transform Utility).

[Andrew]

This is now fixed in the 3.6.1 pre-release.

 

I've newly written Alfred's CSV parser to adhere to RFC 4180, and will create a document page to cover how Alfred deals with the ambiguities surrounding quoted lines with significant leading spaces. Essentially, for a line to be quoted (allowing for newlines and commas), the " needs to be either at the very beginning of a line, or directly after a comma. If there are spaces after a comma then a ", this field is not treated as quoted, and any " marks are treated as literal.

 

I spent some time looking at how some of the online CSV parsers (differently) dealt with this specific ambiguity, and felt that this way best covered all bases

 

As mentioned earlier, Alfred's CSV parser never discards significant spaces, but the List Filter subsequently trims the title and subtitle fields.

 

Cheers,

Andrew