I was surprised that I could not find a straightforward answer elsewhere. When speaking of "delimiters", I want to confirm the following.

Consider the two strings:

str1 = "Hello My Name Is Sam"

str2 = "Hello0My0Name0Is0Sam"

In str1, the symbol behaving as a delimiter (w.r.t. a sequence of "common characters" pulled from the English alphabet) is represented by the space character .

In str2, the symbol behaving as a delimiter (w.r.t. a sequence of "common characters" pulled from the English alphabet) is represented by the 0 character 0.

The question, specifically, is about how one should define delimiter. There are many classic examples that I have found when people refer to delimiters: spaces, commas, semicolons, etc...and then there are "bracketed delimiters" that come in pairs (e.g. (), [], etc). However, the definition seems like it should be much more general.

Consider this final string just to prove my point:

str3=ac1b238b3ca217ba1c234

In this string, I will bring to your attention the fact that 8, 7, and 4 are functioning as delimiters to alphanumeric set restricted to a,b,c,1,2,3.

If all of this is correct, then can a delimiter be thought of much more generally using this two step process?

  1. define a set of symbol that are "non-delimiters"
  2. every other symbol not defined in 1. is thus a delimiter.

Any insight is greatly appreciated. Thank you!

4

1 Answer

Yes, you can define it that way, but it's more common to do the opposite. First we have a set of symbols S, and then a set of delimiters D such that D ⊂ S. You could define it as D ⊆ S too, but if D = S, it's pretty useless for obvious reasons. So it's reasonable to demand D to be a proper subset, although it's not strictly necessary.

Often, you want the symbols in D to be available both as delimiters and non-delimiters. This can be achieved by defining another subset of escape characters E such that E ⊂ S.

In practice, both D and E usually have the size of 1, meaning there's only one delimiter and one escape character. The string is usually parsed from left to right, and when an escape character is found, the next symbol, irregardless if it belongs to D, E or none, gets treated as a normal symbol.

2