What I hear in this is that at the core we’re trying to protect information and share it on the right channels without signaling too loud, where adversaries could make benefit.
There are two parts to my response, one is methods I have found useful in answering this question in my own experiences. The second half I share an opinion about getting data in to systems.
Data has a flow - and when it stops then we have an issue.
There are some considerations in that video the important ones I am sharing below.
Expectation of Flow: Users might have an expectation of how their information should be shared or kept private.
Legitimacy of Flow: The legitimate flow of information should be based on actual, practical considerations and societal norms.
Meaning Information is contained in or revealed by,
Meaning each document or digital creation has information that could, if put together convey something else.
Some examples of this:
A document with lists of addresses contains the information.
An article mentioning the list reveals the information.
The lists were revealed by the article that was written.
What I try to think of:
What are the sources, and what is the length that this data or information will still be useful?
I try to focus on the longest time that a piece of information will be useful, I have found that README with instructions about the data retention period, or defining the data retention period in the files creation (there are programmatic and system considerations to apply ).
So new derivative documents I write, then can mention or exclude the source, when it makes sense based on the audience and trust levels inside that space.
- What do I want to protect?
- Who do I want to protect it from?
- How bad are the consequences?
- How likely is it that I will need to protect it?
- How much trouble am I willing to go through to try to prevent potential consequences?
I think that in terms of information that should get ingested in to systems, that the lists should be opinionated and in TSV format, TSV is different than CSV by disallowing TABs and newlines in the data.
The escape syntax enables CSV to fully represent common written text, great for humans and spreadsheets. But this comes at a cost, reading it requires programs to parse the escape syntax.
Unix tools like cut, sort, awk, and diff do not process CSV escapes
TSV data is simple. Records can be read using the typical readline this is found in MOST programming languages. Fields in each record can be found using split routines.
awk -F "\t", sort -t $'\t' NO SPECIAL PARSER needed.
And it’s faster, we are not spending CPU time parsing the escape syntax.
TSV also allows increased speed for record counts, deduplication, file splitting and shuffling and new line search routines like memchr