Querying grouping & sorting across several text files with the FoldingText Command Line Interface

So I was thinking last night that this set of tools could be a neat way to create a Taskpaper weekly forecast based on a folder of FoldingText project files. The Taskpaper file would be more portable to iOS for reminders (e.g., through the Listacular app) or for remembering non-computer based tasks. (Personally, I like to plan weekly on the Mac, but then work through the tasks for the week using iOS for reminders and quick reviews of what’s planned for each day).

Where I got stuck was imagining how tasks marked “done” on the Taskpaper report might be “synced” back to the original FoldingText files. Would this be possible?

Here’s my imagined workflow in more detail:

  • Create one FoldingText file per project–mix of tasks (in .todo sections), paragraphs of text, outlines, etc–located in one specific folder (e.g., Projects).
  • Run a perspective report using this script that would return something like below, formatted in Taskpaper format, with colons for headings and with tasks indented under the heading:
    • Overdue Tasks:
    • Started (and not complete) Previous to this Week:
    • Due Monday:
    • Starting Monday:
    • Due Tuesday:
    • Starting Tuesday:
    • etc. for the rest of the week
  • Sync that Taskpaper file with iOS
  • Check tasks as complete on iOS Taskpaper file
  • On Friday (or whenever), run the “reverse” script, which scans Taskpaper file for tasks tagged “done” and adds that tag to corresponding lines/tasks in each particular FoldingText project file.

I suppose, even if it’s not possible to automate this process, it would be easy enough to follow ft2doc:// links on each line back to the task and use a keyboard shortcut to mark it “done” in FT.

Thanks - I’ll give all that some thought, and sketch out first versions of the queries

I’ve continued to play with this script and am finding it very useful. I’ve run into two more general features that I think would be useful. I regret to say that I am still out of my depth in trying to read/interpret the ViewMenu.json file. I was able to add a report that swapped due dates for start dates, but other than that, I haven’t been able to tinker much and get any results.

At any rate: would it be possible to:

  1. return child lines along with tagged lines? E.g., my files often look like this:

    - task name with tags @tag note for the task (this line should be indented to be a child of above line but forum software not letting me do that)

but in the reports, the note line (the child) doesn’t appear.

  1. display other tags on lines in the reports?

For example, in a report that displays lines by due date, I’d like to also see the other tags that were on the line

e.g.,

original:

- task name @tag1 @tag2 @due(2014-07-22)

would become:

`# Due 7-22-2014

  • task name @tag1 @tag2`

whereas now the additional tags don’t appear.

Thanks!

Hi – things have been a bit busy, have done a bit of maintenance on this, and hope to get back to it on Friday. In the meanwhile:

Q2 (tags)

If $item is mapped to a line, {$item} will just be translated to the ‘text’ part (no syntax prefixes or terminal tags), but
{$item@line} will be translated to the whole raw line, prefixes, tags and all.
If it’s the value of a particular @mytag that you want, you can write:

return: “- {$item} {$item@mytag}”

but it sounds as if I should add something like fn:tags($item) to return the tags of the line without its syntactic prefixes.

Q1 (descendants)

Do you want the full set of descendants of the displayed line, or just its immediate children ?

(If you want to send me the rough JSON of what you want, or the existing template that you want to work from, I should be able to sketch an approach for you )

PS I think that in the version you have, you should be able to write:

return: "- {$item} {$item@tags}"

for:

# Due 7-22-2014
- task name @tag1 @tag2  @due(2014-07-22)

but I’ll need to add an fn:tagsubset of some kind so that we can select which ones to show inline.

I think in practice this would be the same for the way I use it, but for edge cases, I think full descendants would be best.

This is great–you’ve taken my meaning exactly. Thanks for responding to my requests. No worry on the timeline–if you’re busy put if off!

"Started (with ftdoc:// links)": {
		"title": "## Started This WEEK",
		"for": "$item in //@start < {now}",
		"let": "$day = fn:daypart($item@start)",
		"groupby": "$day",
		"orderby": "$day",
		"return": [
			"### {$day}",
			{
				"for": "$i in $item",
				"orderby": "$i@start",
				"return": "- fn:timepart({$i@start}) {$i} {$i@tags} [Link]({$i@url})"
			},
			""
		]
	},

Based on your recent response, this is pretty close to exactly what I’d want (adapted from your “due with ftdoc links” report).

The only other changes that might be nice would be:

  • if “Link” could be replaced with the file name from which the line came (which in my use, is the project name).
  • if tagged lines brought their descendants (Q1 descendants, from above)

Here’s a new query I wrote that returns “@next” tagged items and groups them by their parent item. In other words, if you have lines tagged “@next” in .todo blocks, it returns something like:

## Original project header.todo
   - task name :: [Link](ftdoc://link to original item) @next @read

I use this to search across all my project files for next actions. Would be best if the project header also contained a reference to the filename as well (which, for me, is the name of the overarching project), but I can’t figure out how to get that to display yet.

"Next Actions with Link": {
		"title": "# Next Actions",
		"for": "$item in //@next",
		"let": "$project = $item@parent",
		"orderby": "$project",
		"groupby": "$project",
		"return": [
			"## {$project}.todo",
			{
				"for": "$i in $item",
				"return": "- {$i} :: [Link]({$i@url}) {$i@parent} {$i@tags}"
			},
			""
		]
	},

Looking for a bit of help here. In my “next actions” query above, $i@parent correctly returns the parent line of any line tagged @next. However, why doesn’t $i@child do the same thing? Instead of returning the child line, it returns some gibberish numbers. Here’s the whole query:

"Next Actions with Link": {
		"title": "# Next Actions",
		"for": "$item in //@next",
		"let": "$project = $item@parent", 
		"orderby": "$project",
		"groupby": "$project",
		"return": [
			"## {$project}.todo",
			{
				"for": "$i in $item",
				"return": "- {$i} :: [Link]({$i@url}) {$i@tags} \n {i@child}",
			},
			""
		]
	},

Excellent - I’m hoping to get back to all that on Friday, and I’ll take a look at including filenames then.

Made some improvements to my started query. Now it groups by date, then groups by “project” (the parent header of the returned tasks) and it filters out @done tasks:

"Started (with ftdoc:// links)": {
		"title": "# Starting in the Next 7 Days",
		"for": "$item in //@start <= {today + 7d} except //@done",
		"let": "$day = fn:daypart($item@start)",
		"groupby": "$day",
		"orderby": "$day",
		"return": [
			"## {$day}",
			{
				"for": "$i in $item",
				"let": "$project = $item@parent",
				"groupby": "$project",
				"return": [
					"### {$project}.todo",
					{
						"for": "$j in $i",
						"return": "- {$j} :: [Link]({$j@url}) {$j@tags}",
					},
					""
				]
			},
			""
		]

	},

@parent is a uniquely defined property of a node/line, and you can look at its derivation in the getAttrib() function in the plugin.

@child is not a uniquely defined property of a node (any given node might have various different children), so it’s not one of the properties that getAttrib() can return.

But could you tell me more about what exactly you mean by the the/a child line ? (To Foldingtext its any one of the next generation descendants … but it sounds like you may have something more narrowly defined in mind)

Just trying to bring along the task’s “note”. E.g., I have my .todo sections set up similar to Taskpaper format:

# Header.todo
  - task @next @read
    my note about the task
  - my second task @read

In other words, if a line has a “note” (a descendant? I’m not sure of the nomenclature here), I’d like it to be brought along into the report.

OK, so just the first child line ? or might the note have line breaks, and consist of a set of peer lines ?

yeah, some notes do have line breaks and would consist of peer lines. But in my usage, no “notes” would have children of their own. But, if necessary, I could survive with just the first child line (this covers 90% of my current files and I could easily change practices here to accommodate).

I think we should be able to give you the set of immediate children of type body - off the top of my head, it will probably be by entering one more return loop, and defining that set of children.

(But possibly cleaner to add a @note property to getAttrib(), I’ll take a look – thanks for the work and feedback – it’s very helpful)

Running this quickly through Jesse’s parser, it looks as though notes following a todo bullet are parsed as peers of the list item rather than children.

# Header1.todo
- task @next @read
	my note about the task
- my second task @read


# Header2
- task @next @read
		my note about the task
- my second task @read


┌ Tree
┠┰∅ [0, root]
┇┠ ∅ [1, empty]
┇┠┰# Header1.todo [2, heading]
┇┇┠ - task @next @read [3, unordered]
┇┇┠ ▸my note about the task [4, body]
┇┇┠ - my second task @read [5, unordered]
┇┠ ∅ [6, empty]
┇┠ ∅ [7, empty]
┇┠┰# Header2 [8, heading]
┇┇┠┰- task @next @read [9, unordered]
┇┇┇┠ ▸▸my note about the task [10, body]
┇┇┠ - my second task @read [11, unordered]
┇┇┠ ∅ [12, empty]

I’ll experiment with adding a @note property which harvests immediately following nodes which are body or empty

(editor.tree().toString() is a very helpful function in the FoldingText scripting API incidentally – it lets us look directly at the structure of the underlying tree):

property pstrJS : "
	function(editor, options) {return editor.tree().toString()}
"

on run
	set varResult to missing value
	tell application "FoldingText"
		tell (make new document with properties {text contents:"
# Header1.todo
- task @next @read
	my note about the task
- my second task @read


# Header2.todo
- task @next @read
		my note about the task
- my second task @read
"})
			
			set varResult to (evaluate script pstrJS)
		end tell
	end tell
	return varResult
end run

Cool, thanks! Looking forward to the results.

I’m still just tinkering with cut-and-paste with this (no formal javascript knowledge yet), so digging into Jesse’s tools are a bit out of reach yet. That’s why I appreciate your examples for this plugin so much–they’re rich and easy enough to learn from.

I’ll try to put out a fuller set.

In the meanwhile, Jesse has explained to me that notes are recognised as children of todo items when they have an extra tab indent (I misinterpreted what the tree diagram was showing in the case with an extra tab).

If you see nodePath searching as something which might become more central to your workflow, that means that you might well get some benefits from indenting your notes one tab more deeply under the list items.

(For the purposes of a @note attribute for the getAttrib() function for reports, we can make it recognise either layout, but might be worth bearing in mind that bigger queries (multiple files) are likely to run a little faster if note-children are made immediately recognisable by a further tab of indentation)

PS I forgot, of course, the stronger and more immediate argument for an extra tab to mark notes clearly as children of list items, rather than simply successors/peers

If you add one more tab, to mark them as children, ⌘/ collapsing of the notes under an item will then work properly.

Hmm. That’s interesting. I wonder why a note needs to be “double” indented to be recognized as a child? I thought I remembered at one point, notes would “fold up” to the todo item as long as they were indented under it–it wasn’t necessary to tab twice.

At any rate, it’s not a problem to double indent–just need to get in the habit.