I’m sorry I haven’t been more active in these forums lately. Here’s a review and status update of what I’m working on now.
The road to FoldingText 2.0
My basic idea for FoldingText has always seemed pretty simple: take the plain text outlining idea from TaskPaper and extend to more then just todo lists.
My goal was to build an extensible plain text outliner. A place to “think in text”. A place that removed all UI complexities allowing you to work directly with your ideas in text.
I’ve been working towards this goal since 2009. Full time work started on early 2011. FoldingText 1.0 came out in Oct 2012 but had a buggy editor. FoldingText 2.0 solved that in Jun 2014.
FoldingText 2.0
After the 2.0 release I started to add support for due dates. The constrain that everything is in plain text and visible to the end user made this more difficult then I’d imagined.
In that context I began to realize how complex all of FoldingText really was. I’d got rid of traditional UI complexity, only to replace it with Markdown text formatting complexity.
FoldingText 2.0 is a nice Markdown editor with some unique features. But it’s not really what I set out to build. The Markdown aspect adds lots of complexity that obscures my original goals.
UI Complexity
Unless you know Markdown FoldingText doesn’t make much sense. Even if you do know Markdown it’s not designed to visually represent hierarchies.
The underlying outline model is hard to see and manipulate. Indent a heading and only a single extra “#” is added. Indent a list item and the text moves to the right. There’s no consistency to the visualization of the outline model.
My kids are school aged and will soon need a place to “think in text”. But they are not ready and have no need to learn the intricacies of Markdown. Same with most people I know.
API Complexity
Extensibility has always been one of my goals.
But the plugin API is much more complex then it needs to be because everything is always changed indirectly… change text first and then the parser runs to update the underlying model.
Markdown formatting rules make manipulating the outline hierarchy unpredictable. You must check the type of everything ahead of time to make the move work. Lots of things don’t make sense, such as adding a header to a list item.
FoldingText 2.0 lessons learned
In retrospect using Markdown to encode the outline model wasn’t a good idea. It added lots of complexity and obscured the bigger goal of an extensible plain text outliner.
I now think that the deeper problem is that its just not possible do what I’ve been trying to do in plain text without lots of compromises.
Every new feature requires new formatting rules and they quickly make the app too complex.
Plain text is wonderfully portable in the sense that you can open and edit it in any text editor. But as soon as you start adding lots of syntax rules it becomes less and less useful except in the editor where it’s designed to be used.
And the format gets so complex that it’s impossible to parse and process outside the context of the FoldingText app. So while its portable in the sense that you can open it in any text editor, the actual data is pretty much locked into FoldingText.
What I’m working on now
Because of the problems listed above I started working on a new approach in Aug 2014.
The basic idea is to expose FoldingText’s underlying outline model directly in the UI. It looks and works more like a traditional outliner then like a text editor.
Its command based instead of plain text formatting based. For example to make text bold you select it and choose the bold command instead of surrounding it with **'s. To make a heading you choose the “Make heading” command and all selected paragraphs are made into headings.
The outline structure visible and understandable. Expanding and collapsing items and moving items around is all animated so even if you aren’t used to outlines you and understand what’s happening and how the outlining command work.
The plugin API is now direct. Moving an item from one place to another is simple, there are not formatting rules that might mess things up along the way.
The data format is a subset of HTML. It’s a nested list with simple spanning formatting elements such as bold, italic, and links. Being HTML it’s also extensible in a predictable ways. Plugins can easily store their own attributes along with a list item (such as due dates) and choose how to display (or not) them in the UI that’s presented to the user.
Being HTML you can open the document in any web browser and see the basic structure. It’s also simple to parse and process in any language with an HTML parser. For practical editing, and using plugin features, you’ll still need to use FoldingText as the editor.
It’s fairly strait forward to map plain text formats such as TaskPaper or Markdown to this model, so importers exporters will be possible though they are not something I’m working on yet.
I’m in the process of switching to use the open source Atom.io project to host the outliner. This provides tons of useful infrastructure: Workspace for managing multiple documents in tabs and panes; File browser and load/save infrastructure; Cross platform (OS X, Windows, Linux); Plugin management an lots of generic plugin APIs so I only need to worry about providing the API for outline manipulation.
Wrap Up
I’m definitely still here and typing away.
I think FoldingText 2.0 works pretty well as a Markdown editor with folding, but it’s more limited and complex then what I was envisioning. I’ll continued to keep it updated and working, but won’t be adding big new features.
I’m spending most of my time (since last Aug) working on this next iteration that I hope will get FoldingText closer to the original goal.
I don’t have any demo for you to try yet, but I’ll post a screencast demo in the next few days to give you a better idea of what I’ve build so far and what still needs to be done.
I’m sure you’ve got lots of ideas, thoughts, and questions in your mind about this new version. But I don’t want to start that conversation in depth yet. I think it’s better for me to first get a demo out that you can try, then we can go from there. The basic outliner is ready to demo, but I still need to figure out how I’ll embed and integrate it into Atom.io. I expect to have something to show in about a month.
I look forward to your demo and any beta versions that you would like to share. Even though version 2 doesn’t meet your own goals it is an indispensable part of my current workflow when developing short tech articles and workshops. Keep up the good work!
I’ve had a number of private messages concerned with losing markdown from FoldingText… They like Markdown and FoldingText is their dedicated Markdown editor. They don’t want to loose that.
I addressed this very briefly above:
Let me expand upon that here.
In the above I’ve listed a number of changes, but internal application model stays the same. Currently (FoldingText 2.0) when you load a markdown file it gets parsed into a node-tree data structure. Each paragraph is a node in the tree and each node has things like (textContent, nodeType, tags, etc). This next version has the same internal data structure.
Because of that it’s possible and reasonable to build a import/export for the new version that will read in a Markdown file. And then on save export it back to markdown from this internal node-tree representation. The export process might have some issues since the internal representation no longer tracks your exact markdown formatting and there are a number of different ways you can format markdown. So for instance when exporting a list it won’t know if you want to indent using spaces, how many, etc. But in general the round trip is possible.
So to recap on this issue:
The default format will be HTML subset. This is easier to parse and work with outside FoldingText, has a place for metadata (like persistent unique ids for every item), etc.
I’m not working on them yet, but it should be possible to create importers exports for Markdown/TaskPaper formats. They wouldn’t be perfect some specific formatting might get lost, but the meaning of everything should stay the same.
I haven’t yet done any of this yet. I’m sure it’s all possible, but I don’t know what issues will come up. Right now none of this is a focus, I’m just working on the underlying outline model and editor, no importers or exporters.
Markdown can still be as central as we want with this approach (I’m finding that conversion scripts takes about 3 milliseconds on my system with a typical FT MD text).
The main difference is that the one-way MD → HTML conversion which we are continually doing in plain text workflows will become two way, with full round-tripping and preservation of tagging and formats.
(Jesse’s HTML format can preserve all the tagging and line-type data from Markdown, support much clearer and more solid outlining, and enable us to write back into fully tagged and formatted Markdown whenever we want to )
One could easily envisage a Preferences option to save by default as Markdown, and the unordered list format which Jesse is planning to support has some rather major advantages. It will enable very powerful perspectives menus, making direct use of OS X’s built-in ‘xQuery’ support, and giving an integrated overview of dates and data across several different files, with live links back to each file and line, even after particular lines have been moved around by edits.
( Plain Markdown lines have no id, so a moved line can’t be found by a link from outside )
It’s easy to forget that even in the ‘purest’ plain text workflows we are continually doing a lossy swapping into HTML as soon as we want to print, display etc. What Jesse is offering now is a small move into more fertile middle ground between plain text+HTML, making the HTML part editable (and richly queryable) as well as displayable, and enabling a round-trip return to plain text without loss of information.
On reflection I think this clearly opens up the best of both worlds.
In traditional plain text workflows, HTML has always been a decorative dead-end.
Editable HTML outlines, with full and fast round trip conversion of tag information and line types ⇄ Markdown outlines, gives us direct access not just to MD and plain text tools, but to XML querying and formatting tools as well.
Glad to hear you’re still going strong on the FoldingText project. I very much like the sound of the new direction you’re taking. While the Markdown-based model of the current version is conceptually interesting (and very useful), I do see how there were some very difficult problems inherent in that model. In any case, you’ve made a great effort of it!
Making the new document format valid HTML is a very smart idea. I’ve increasingly felt that HTML may be the format to centralize on for post-Microsoft-Word documents. RTF is messy; Markdown feels uncomfortable to many people as a “final” document format. HTML is pervasive: as you mention, it can be opened in a web browser, and posted directly to a web site. It’s very surprising to me that there aren’t very many simple HTML-based rich-text editors (aside from those CMS editing interfaces).
While I use the current version of FoldingText daily, I’ve been a bit hesitant to recommend it to less tech-savvy users. A rich-text editing interface would, I think, greatly expand the potential audience for the app.
Looking forward to hearing (and seeing) more about this new version!
Jesse, your new approach sounds both logical and very exciting. I look forward to using the next version of Foldingtext.
For those who are addicted to keeping all (or most) files as plain text (not necessarily Markdown), do you have any plans to keep Taskpaper (which I use every day and still love) alive?
Yes… often when I say “FoldingText” I really mean the bigger idea (and the underlying text editing engine) that I’ve been working on since TaskPaper. I’ve always intended to use the FoldingText engine in the next version of TaskPaper, so all changes mentioned here also apply to TaskPaper.
And for TaskPaper in particular I do think some sort of easy plain text round tripping is important since lots of other apps rely on TaskPaper plain text.
If I understand this correctly, the plan is to move away not only from Markdown, but also to move away from storing things in and parsing plain text?
I don’t care about Markdown specifically that much, but personally, I’m not interested in a program for note taking and basic writing where I have to export files to make them readable in any text editor.
Of course, as you say, even if it’s plain text, certain features may be »locked« to the program, but I think the chances are higher that those features will be picked up by others if it’s just plain text.
For example, Editorial on iOS has supported highlighting plain text lists saved as *.taskpaper via tags for a while, whereas a mobile FoldingText app seems to be way off in the future.
Programs come and go, and it’s why I will never again use a note taking app that uses a complicated file format, proprietary or not. Eventually, they all get abandoned and die.
Hey Daniel— like you, my workflow benefits greatly from syncing plain text files between devices via Dropbox, and if I thought that would be compromised in any way, I’d be a little worried. But Jesse’s post above gives me hope that the experience of using plain text with the new update will (eventually?) be pretty much seamless:
I wouldn’t worry too much on that score – there’s actually a continuous spectrum of text file formats, without any real and clear breaks, and while Markdown is less marked than HTML, it’s (fortunately) not really unmarked ‘plain’ text and it would be much less useful if it were : - )
Jesse’s small shift along the text file spectrum towards slightly more marked text files preserves human legibility and wide usability, and is much more fit for the purposes of text outlining, text tagging and text linking.
Unlike OPML it’s directly displayable by any browser, and I have already drafted a couple of scripts to experiment with converting MD ⇆ Jesse’s very simple HTML outline format . Easy to write and fast to run – so fast that it would be possible to use the new version in a mode which transparently read from and wrote to MD – particularly if you didn’t want the advantages like permanent link targets and solid outlining, and can put up with the fiddle and ambiguity of alternating between hash levels and unordered list levels, and changing your mind about which is which, and what is a child or parent of what :- )
I find it disappointing that FoldingText is taking this change of direction. The application as it stands fulfills a specific requirement of my daily routine as an excellent Markdown editor with programmatic extensibility.
My workflow syncs the FoldingText files I write to OwnCloud, where if need be, I can edit them online with a markdown plugin. Changing the native format so that the rich content/metadata is outside of what is visible negates this tool’s utility for me. Importing/Exporting may be possible, but it seems like specific filters would need to be written for import and export of any add-on tools that would write the metadata that make the tool useful to me. This seems tedious from a programming perspective, and fragile from a user perspective.
Supposing I have no power to persuade you to revert this course - since you’ve already been working on it for months without any mention of the intent to do so - what is the possibility of continued support of FoldingText as it stands as a separate, useful application? At the very least, finally release 2.0 out of dev and call it done forever?
Don’t forget that you will be able to use the new editor in a Markdown authoring mode if you choose to, and that the metadata will be visible, editable, and used by stylesheets.
Still MD in and MD out if you prefer, or whenever that works best.
The in-editor representation as an HTML unordered list does bring some advantages (full accessibility to OS X’s built in XPath and XQuery, persistent line ids etc) but you don’t have to use it as your default serialisation if you don’t need those.
In addition, the Atom platform brings advantages and depths of its own. Tree View and Search in Project, for example, are like having nvAlt quick search and archive built into the editor, which also has its own MD preview, and in which you can work with text files in any variety of markups.
[quote=“complexpoint, post:16, topic:676, full:true”]
Don’t forget that you will be able to use the new editor in a Markdown authoring mode if you choose to, and that the metadata will be visible, editable, and used by stylesheets.
Still MD in and MD out if you prefer, or whenever that works best.[/quote]
Supposing that a plugin exists to implement the current .todo functionality, would it not need to supply a means to convert the metadata stored in this HTML subset into the @done(date) and other markdown decorations? And also convert it back from that markdown text to the internal format? Would this also be the case for any other plugin - that they would not only have to implement the functionality within the constraints of the new internal structure, but also provide methods to translate bi-directionally between that structure and markdown? I can’t personally conceive of a serializer succeeding in doing this generically, but maybe it’s simpler than I think.
If it’s the case that plugins must include their own metadata serializers for markdown, then it seems unlikely that all plugins will be able to write markdown, meaning that I will be limited to a subset of functionality that supports conversion to markdown, leaving me wondering whether this is the markdown editor I wanted or another custom-format editor focusing on functionality I don’t really need. Ironically, the feature touted most often as FoldingText’s focus around here, folding/outlining, is the feature I find least personal use for.
My bottom line is merely this – I need an editor that:
Saves and loads markdown-format (plaintext) files to local disk
Styles markdown content as I write it, rather than requiring me to use its styling tools
Allows me to associate interactive behavior to the rendered text when certain text tags are added via plugins that universally are capable of storing their metadata in markdown
If this new editor does that, then great. But I suspect it does not, and am curious about the possibility of locking in the current 2.0 featureset as-is so that I can at least maintain that functionality until I find a working replacement, whether it’s the next FoldingText or otherwise.
No – there’s no problem with meta data moving back and forth between HTML5 data- attributes and FT @tags and @key(value) pairs – it’s an automatic one to one mapping.
Take a look the quick scripts which I wrote for FT ⇄ BML (HTML UL) copy/paste and saving
(Jesse is already beginning to build this kind of functionality into the new editor itself).
FT line types and tag/values both automatically become <li>data- attributes, enabling round-tripping by default.
All three of your requirements there - you can opt to load and save in MD, mark a line as header | list | ordered | quote | code etc and see it automatically styled, have automatic behaviour based on key value pairs, entered, if you prefer, in @key(value) style, though the editor will offer faster | simpler routes.
I’m a bit confused. I was looking forward to the demo but I haven’t seen anything appear since the original status update. I did find some references to ‘Birch’ scattered over both here and Jesse’s github account. I haven’t been able to locate the software itself via Atom. How can I take a gander at it?
At the moment I’ve posted a very early version of Birch in a private category here. I’ve limited that catagory just to (level 2) Discourse users… not sure how Discourse calculates it, but means you’ve been here a long time and made lots of posts. Birch isn’t ready for real world use, and trying out the demo will likely use a lot more of your time then it saves. With that said if you have time and are used to using Terminal app and things like that send me a message and I’ll give you access to that category.
