Coding XML Formats in Cocoa

I’ve long been intending to add some technical, coding-related content to this blog. I’ve always admired the blogs of other developers who share some of the coding problems they have faced and solutions they have reached, and in developing Scrivener there are a number of issues that I’ve come across and found solutions for that I’m sure could be useful to other developers. So, this is the first more techie, coding-orientated post to make it to the blog. Those with no interest in Cocoa development, look away now.

I have recently been rewriting Scrivener’s file format. As you may or may not know, .scriv files are packages - essentially folders that on OS X just look like, and are treated the same as, files. Were you to move a .scriv file to a different platform, it would appear as a regular folder. On OS X, you can ctrl-click on a .scriv file in the Finder and select “Show Package Contents”. File packages are great for programs such as Scrivener. Regular files usually have to be loaded into the program’s memory in their entirety when loaded; with a file package, the program can just look inside it and open whatever it needs as and when it needs it. Given that Scrivener can import movie, sound, PDF and image files, you can imagine how much memory might get eaten up if it had to load everything into memory right from the get-go. Instead, with its package format, it can just load the binder structure file and then load up each document as you select it in the binder, and flush from memory any large files that aren’t currently being used.

For Scrivener 2.0, .scriv files will still be file packages, but I’ve been doing some work on the format of the files inside the .scriv file. Currently all text files are saved internally as RTFD files (RTFD stands for “rich text format directory”). RTF files are a standard rich text format that can be opened on all major platforms (they are essentially plain text files with formatting mark-up), and were designed by Microsoft; RTF files support pretty much everything that Word documents do. RTFD is Apple’s extension of this format. An RTFD file is a file package in itself (as with .scriv files, you can ctrl-click on them in the Finder and select “Show Package Contents” - you will find a TXT.rtf file inside there, for instance, which holds the actual text). Apple designed the format so that such files could also hold QuickTime files and any other file type; but the trouble is that RTFD files can only be opened on Macs. Most of the other files inside .scriv packages use the Apple .plist format - I may have given some of them the .xml extension (.plist files are technically XML files), but internally they are just Apple .plists. (.plist stands for “property list”).

This format for Scrivener files was generally a great and solid 1.x file format. It works, and it doesn’t take too much code to maintain on my part - the Cocoa frameworks make it very easy to write to RTFD and PLIST formats. However, for 2.0 I wanted to make Scrivener’s format less platform-specific. The current format has two main flaws:

1) Its use of .plist and .rtfd files means it’s a format that can only be read on the Mac. Although I personally have no plans to switch to or code for other platforms, this would be a significant hurdle for anyone we wanted to work with to port Scrivener to, say, Windows.

2) The .plist format is not human-readable - at least, not when used with the sort of data that Scrivener has to write out. This makes it difficult for anyone on any platform, including the Mac, to write utilities that might work with the Scrivener format.

For these reasons, I am in the process of making the following changes to the .scriv package format:

1) The .scriv package will no longer contain all files in the root folder. Instead, subdirectories will be used to make it easier to navigate. That way, should it be ported to a different platform that doesn’t support packages, it will be easy for the user to find the file required to open the project. And in general it’s just neater, of course.

2) I will no longer use the RTFD format and will switch to using RTF instead. The only reason I didn’t use RTF to begin with was that Apple’s standard RTF reader and writer - the one provided in the Cocoa frameworks - ignores images. That is, it fails to load or save images in the text. Over the years this is something I’ve fixed myself, though, so I use a modified version of the RTF reader/writer to save and load RTF files that retain images with no problems. This not only means that all the text files stored inside a .scriv file are now platform-independent, but also that they are using a file format that has been around for over twenty years. It’s also a format that can be opened in a plain-text reader.

3) Instead of using .plist files, I am creating my own XML file formats where applicable.

(There are other changes too - for instance I am now using a checksum file that can tell which files have been changed since the last session; this means that should Scrivener crash, you will no longer be faced with the time-consuming “Synchronising…” panel, as Scrivener will be able to update only the search indexes for the files that have changed rather than going through every single file in the project.)

Needless to say, writing my own XML file formats is the most time-intensive part of this process. Fortunately, Cocoa has some excellent and easy-to-use classes for generating and reading XML - the NSXML… classes. For instance, suppose I wanted to create the following XML:

<MyXMLFormat Foo="Yes">

<Text>This is some XML.</text>

</MyXMLFormat>

It’s as easy as this:

NSXMLElement *myXMLElement = NSXMLElement alloc] initWithName:@"MyXMLFormat"];

[myXMLElement addAttribute:[NSXMLElement attributeWithName:@"Foo" stringValue:@"Yes"]];

NSXMLElement *textElement = [[NSXMLElement alloc] initWithName:@"Text" stringValue:@"This is some XML."];

[myXMLElement addChild:textElement];

[textElement release];

// Do something with myXMLElement, such as write it to disk using NSXMLDocument.

[myXMLElement release];

Still, I came across a couple of problems in writing my own XML, and that’s what I want to share with you here - the problems and the solutions.

Writing Less Code

The first aspect of working with an XML format in Cocoa I’m going to talk about isn’t actually a problem at all. It’s just that I’m lazy - I get bored of writing out the same things over and over again. The NSXML classes and methods are fantastic - they contain everything you need to read and write good XML - but they are also rather generic, designed to cover all uses. But for many common uses, this can mean writing more code. For instance, suppose you need to read the example XML above and you know that the MyXMLFormat element only has one <Text> sub-element; or, if it has more than one, your app can only handle one anyway so that either way all you really need to do is get the first child element named “Text”. Using the NSXML classes as is, this is easy enough but it takes a few lines of code:

NSString *textString = nil; // Initialise the variable with nil.

NSArray *childElements = [myXMLElement elementsForName:@"Text"]; // Gets all of the child elements named "Text".

if ([childElements count] > 0) // Did we find "Text" sub-elements?

{

textString = [[elements objectAtindex:0] stringValue];

}

if (textString != nil)

{

// do something.

}

Pedants will note that I can easily reduce the string-reading part of the code above to two lines of code if I sacrifice a micron of readability:

NSArray *childElements = [myXMLElement elementsForName:@"Text"];

NSString *textString = ([childElements count] > 0 ? [[elements objectAtIndex:0] stringValue] : nil);

Still, as I say, I’m lazy. Why write two lines of code when you can reduce it to one? So I wrote my own NSXMLElement category to reduce the lines of code in my XML read/write classes. Here are the two methods that do the job:

@implementation NSXMLElement (KBAdditions)

- (NSXMLElement *)firstChildElementWithName:(NSString *)name

{

NSArray *elements = [self elementsForName:name];

if ([elements count] == 0)

return nil;

return (NSXMLElement *)[elements objectAtIndex:0];

}

- (NSString *)stringValueOfFirstChildElementWithName:(NSString *)name

{

NSXMLElement *element = [self firstChildElementWithName:name];

return (element != nil ? [element stringValue] : nil);

}

@end

(Note that I split it into two methods as sometimes you will not want only the string value but will want to get the whole element - for instance to examine its attributes. -firstChildElementWithName: does that, and -stringValueOfFirstChildElementWithName is just a convenience method that calls the former method and returns only its string value.)

Now, to get the string value of the sub-element <Text> from <MyXMLFomat>, I only have to write one line of code:

NSString *textString = [myXMLElement stringValueOfFirstChildElementWithName:@"Text"];

Nice.

This reduced how much code I had to write for reading my custom XML files, but I was still finding there was one sequence of code I was repeating quite a lot in the code for writing to XML. Again, consider the example from above:

NSXMLElement *myXMLElement = NSXMLElement alloc] initWithName:@"MyXMLFormat"];

[myXMLElement addAttribute:[NSXMLElement attributeWithName:@"Foo" stringValue:@"Yes"]];

NSXMLElement *textElement = [[NSXMLElement alloc] initWithName:@"Text" stringValue:@"This is some XML."];

[myXMLElement addChild:textElement];

[textElement release];

// Do something with myXMLElement, such as write it to disk using NSXMLDocument.

[myXMLElement release];

In this case the “Text” element is really basic - it contains only a string value and no attributes. Still, it takes three lines of code to add it:

NSXMLElement *textElement = NSXMLElement alloc] initWithName:@"Text" stringValue:@"This is some XML."];

[myXMLElement addChild:textElement];

[textElement release];

Again, pedants will note that I can reduce this to one line if I really want:

[myXMLElement addChild:[NSXMLElement alloc] initWithName:@"Text" stringValue:@"This is some XML."] autorelease]];

That’s not so bad. But what if I want to add an attribute to the element? In that case I have to go back to my original code so that I have a reference to the element to work with:

NSXMLElement *textElement = NSXMLElement alloc] initWithName:@"Text" stringValue:@"This is some XML."];

[myXMLElement addChild:textElement];

[textElement release];

[textElement addAttribute:[NSXMLElement attributeWithName:@"SomeElement" stringValue:@"SomeValue"]];

So, I added another method to my NSXMLElement category:

- (NSXMLElement *)addChildElementWithName:(NSString *)name stringValue:(NSString *)stringValue

{

NSXMLElement *childElement = [[NSXMLElement alloc] initWithName:name stringValue:stringValue];

[self addChild:childElement];

return [childElement autorelease];

}

This adds the child element for me, creating it using the name and string value passed-in, and then returns a reference to the child element that was added. So now, to add the "Text" element, I can just do this:

[myXMLElement addChildElementWithName:@"SomeElement" stringValue:@"SomeValue"];

And if I need to add an attribute, that’s easy too:

NSXMLElement *textElement = [myXMLElement addChildElementWithName:@"SomeElement" stringValue:@"SomeValue"];

[textElement addAttribute:[NSXMLElement attributeWithName:@"SomeElement" stringValue:@"SomeValue"]];

My code is tidier and easily readable; I’m happy.

The other XML class I decided to write a category for was NSXMLDocument. To load an XML document from file, I need to do this:

NSXMLDocument *xmlDoc = [[NSXMLDocument alloc] initWithContentsOfURL:url options:NSXMLNodePreserveWhitespace error:&error];

I need the “preserve whitespace” option because otherwise Cocoa’s XML loader will obliterate any return and tab characters, and obviously I want to preserve them. (Although I use RTF for saving the text of files, there may be plain text representations of the text in some files - for instance, in the search.indexes file - and the user may have entered a tab or return in some other locations that get saved in the XML file.) However, sometimes I found that this could return nil unless I also passed in the NSXMLDocumentTidyXML option to tidy up any malformed XML (obviously I’m working hard to ensure that Scrivener can’t generate any malformed XML, but there are instances in the program where it may need to try to read some, and equally obviously I need to ensure that it doesn’t fail to read projects in the event that it has created some dodgy XML somewhere). But you wouldn’t normally want the NSXMLDocumentTidyXML option passed in, because it wipes leading tab characters among other things; you only want to use it (or I did in this case) if there is a problem reading the XML document without this option. In other words, I want to initialise an NSXMLDocument with the NSXMLNodePreserveWhitespace option, but if that fails I want to try again using NSXMLDocumentTidyXML. That’s easy enough:

NSXMLDocument *xmlDoc = [[NSXMLDocument alloc] initWithContentsOfURL:url options: options:NSXMLNodePreserveWhitespace error:&error];

// Did it fail? If so, try passing in the tidy option in case it failed because of malformed XML.

if (xmlDoc == nil)

[[NSXMLDocument alloc] initWithContentsOfURL:url options: options:NSXMLNodePreserveWhitespace|NSXMLDocumentTidyXML error:&error];

if (xmlDoc == nil) // If it still didn’t work, log an error.

[[NSAlert alertWithError:error] runModal];

That does the job nicely, but rather than have three lines of code every time I need to do this, I can reduce it to one by separating the above out into an NSXMLDocument category:

- (id)initWithContentsOfURLPreservingWhitespace:(NSURL *)url error:(NSError **)error

{

if (self = [self initWithContentsOfURL:url options:NSXMLNodePreserveWhitespace error:error])

return self;

self = [self initWithContentsOfURL:url options:NSXMLNodePreserveWhitespace|NSXMLDocumentTidyXML error:error];

return self;

}

@end

So, again, I have less code to maintan in all of my XML reading objects:

NSXMLDocument *xmlDoc = [[NSXMLDocument alloc] initWithContentsOfURLPreservingWhiteSpace:url error:&error];

if (xmlDoc == nil) // If it still didn’t work, log an error.

[[NSAlert alertWithError:error] runModal];

That’s the convenience methods built to aid and abet my laziness out of the way. But I had a bigger problem.

Dealing with invalid XML characters

Scrivener users paste text in from Word and other sources, and some of that text is going to end up in my XML files - maybe because the user copied some Word text into a document title, or maybe into the text itself, which will have a plain-text representation saved in the search.indexes XML file. The trouble is, it turns out that not all Word characters are valid XML characters. Not all characters are valid XML characters, period. I discovered this when trying to load a project that had been converted to use XML and having it fail with an XML reading error message about “invalid pcdata char value 12” or some such. After some poking around, I discovered that this was caused by a page break character - Cocoa’s NSFormFeedCharacter - occurring in the text. The text is stored as RTF on disk - that part is fine - but also has a plain text representation saved in the search.indexes XML file. And that was the problem; the unicode character used for NSFormFeedCharacter (0x000c) made the XML reader choke when trying to load the search indexes table from the XML file. So I took the path of least resistance and wrote a shoddy method that removed all form feed characters from the plain-text version of any text that got saved in the search indexes XML file, making a mental note to look into it further later as I knew that couldn’t be the only character that would cause problems. Sure enough, a while later another file refused to load - and this time it was some other random Word character that had been pasted in that was causing the problems. The character appeared as some odd symbol in Scrivener and isn’t supported by the text system anyway, but when saved in the XML file it would cause the XML reader to choke and fail to open the file.

That was when I started doing some more digging around and came across this:

http://cse-mjmcl.cse.bris.ac.uk/blog/2007/02/14/1171465494443.html

Wich in turn led me to the XML specs:

http://www.w3.org/TR/2000/REC-xml-20001006#NT-Char

It turns out that there are whole ranges of unicode characters that make for invalid XML, and I had wrongly assumed that the NSXML classes would take care of all of this for me when they won’t. As specified in the specs above, valid unicode ranges for XML files are:

Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF] /* any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. */

The invalid characters aren’t the sort of thing you are likely to be using in your document titles or text anyway, but that doesn’t mean they can’t appear in Scrivener documents when pasted in from other sources (such as Word). And it turns out that the NSXML classes will allow you to write any string value, but will fail when you try to read a string value that contains invalid XML characters.

In other words, the Cocoa classes leave it up to the individual developer to ensure that whatever he is writing to XML contains only valid XML characters. So the trick now was to write an NSString category method that would clean up any strings to be written to XML by removing any characters that fell within the invalid XML ranges. I hacked something together based on the web page mentioned above, essentially going through a string character by character and checking it didn’t fall within the invalid ranges, but it threw up some odd warnings and, although it seemed to work, I knew it wasn’t very efficient. Thinking that I might need some low-level C magic to build an NSString out of unichars or some such (can you tell that I’m a high-level Cocoa guy?), I asked for advice on the Cocoa developer lists (lists.apple.com), which are an amazing resource. There, the always helpful Jens Alfke (thank you, Jens!) pointed out I could specify ranges of unicode characters using NSMutableCharacterSet, which was something I had completely missed and which led to the solution, namely:

• Build a character set containing all of the unicode characters that are valid XML as specified in the XML specs.

• Invert this character set so that we have another character set containing all of the invalid characters, and keep this character set around in memory so that it only ever needs creating once (because we don’t want to go through the process of creating it and adding all the character ranges every time it is required, which could be a lot).

• Check to see if the passed-in string contains any of the invalid characters contained in the character set.

• If not, the string can be used as-is.

• If it does contain invalid XML characters, though, go through it looking for all invalid characters and remove them.

The resulting NSString category method is below:

@implementation NSString (XMLMethods)

- (NSString *)validXMLString

{

// Not all UTF8 characters are valid XML.

// See:

// http://www.w3.org/TR/2000/REC-xml-20001006#NT-Char

// (Also see: http://cse-mjmcl.cse.bris.ac.uk/blog/2007/02/14/1171465494443.html )

//

// The ranges of unicode characters allowed, as specified above, are:

// Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF] /* any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. */

//

// To ensure the string is valid for XML encoding, we therefore need to remove any characters that

// do not fall within the above ranges.

// First create a character set containing all invalid XML characters.

// Create this once and leave it in memory so that we can reuse it rather

// than recreate it every time we need it.

static NSCharacterSet *invalidXMLCharacterSet = nil;

if (invalidXMLCharacterSet == nil)

{

// First, create a character set containing all valid UTF8 characters.

NSMutableCharacterSet *xmlCharacterSet = [[NSMutableCharacterSet alloc] init];

[xmlCharacterSet addCharactersInRange:NSMakeRange(0x9, 1)];

[xmlCharacterSet addCharactersInRange:NSMakeRange(0xA, 1)];

[xmlCharacterSet addCharactersInRange:NSMakeRange(0xD, 1)];

[xmlCharacterSet addCharactersInRange:NSMakeRange(0x20, 0xD7FF - 0x20)];

[xmlCharacterSet addCharactersInRange:NSMakeRange(0xE000, 0xFFFD - 0xE000)];

[xmlCharacterSet addCharactersInRange:NSMakeRange(0x10000, 0x10FFFF - 0x10000)];

// Then create and retain an inverted set, which will thus contain all invalid XML characters.

invalidXMLCharacterSet = [[xmlCharacterSet invertedSet] retain];

[xmlCharacterSet release];

}

// Are there any invalid characters in this string?

NSRange range = [self rangeOfCharacterFromSet:invalidXMLCharacterSet];

// If not, just return self unaltered.

if (range.length == 0)

return self;

// Otherwise go through and remove any illegal XML characters from a copy of the string.

NSMutableString *cleanedString = [self mutableCopy];

while (range.length > 0)

{

[cleanedString deleteCharactersInRange:range];

range = [cleanedString rangeOfCharacterFromSet:invalidXMLCharacterSet

options:0

range:NSMakeRange(range.location,[cleanedString length]-range.location)];

}

return [cleanedString autorelease];

}

@end

So now I can ensure that I’m writing valid XML by passing any strings written to XML through my -validXMLString method:

NSXMLElement *fooElement = [[NSXMLElement alloc] initWithName:@"Foo" stringValue:[[binderDoc title] validXMLString]];

Phew. Now I can get back to finishing Scrivener’s new internal XML generators confident that they won’t end up creating any projects that refuse to open because of bad XML.

P.S. I apologise for the really poor formatting of the code on this page - no matter what I try, I can’t get Blogger to format it nicely. I’ve tried http://formatmysourcecode.blogspot.com and other methods, but no matter what it just looks rubbish. If you know of a good way of formatting code so that it doesn’t look terrible on Blogger, please let me know (I’d love to make it automatic in Scrivener’s “Copy as HTML” code too).

Why I Hate Sky But Quite Like David

Grr, Sky! Grr, Cornwall! (The sky I am referring to is the corporation owned by devil-incarnate Rupert Murdoch*, by the way, and not the most excellent canopy, the brave overhanging firmament, the majestical roof fretted with… Not the actual sky, I mean.)

Despite the year being 2010 (2010! Where’s my hovercar? Where’s my robot butler?), thanks to the Cornish infrastructure, my broadband connection is capped at 1.5MB per second. Mr Internet tells me it’s capable of 2.5MB (woo!), but is for some reason limited to 1.5. (Perhaps this snail-like connectivity is because the Cornish generally distrust anything fast - SUCH AS DRIVING ABOVE 15MPH, for instance. Seriously, a 17 year-old in a Mini told me to slow down the other day because I was doing 30 in a residential area. This English county - just provoking the locals there - is as strange as it is beautiful. But then after thirteeen years of living in London, I still haven’t got used to people being friendly and generally chilled out, and I still retain some of that city-dweller’s tendency of thinking that everyone else is just in my way.)

Why is my internet connection in any way interesting, you ask? It’s not, but it is frustrating when you try to download the iPad (iPhone) SDK, which is 2Gb in size (and downloading is the only way you can get it, as far as I can see). And it’s even more frustrating when Sky Broadband suddenly reduce their 80Gb monthly usage to 10Gb without telling you and then keep cutting off your internet connection when the download gets to 1.9Gb. Thanks, Sky. The conspiracy theorist in me can’t help but think there is something intentional in this - after three attempts at downloading I was already at half of my monthly usage allowance, and it’s only the start of the month - and if I go over again the prices go up. Of course.

Maybe I wouldn’t mind if Sky had ever provided a half-decent service, but have you actually tried using a Sky+ box? It’s as though they kidnapped Sir Clive Sinclair from the past, fresh from inventing the ZX81, stopped off in the early nineties to pick up parts, and then locked him in a room to knock the thing together without bothering to tell him about any technological advance since Teletext. I swear it’s John Titor all over again. I don’t know why I’m surprised by this - after all, the Sky box was designed by Amstrad. Using Amstrad technology as the basis for the country’s most popular (um, only) satellite television box is a little like giving the troops on the front hum-vees built on the chassis of Robin Reliants. Oh look, it’s raining - the signal’s down. (Pity I live in Cornwall, then.) Oh look - it’s just turned itself off again. Oh look - it’s frozen on Peter Andre’s face for the past hour. But living in Cornwall you either use Sky or put up with four channels of drivel rather than a hundred (note to Richard Branson: hurry up and install cable down here you b*****d!). And I need those extra channels if only to pretend I’m not loving Glee or really looking forward to the new series of Supernatural, and so that I can bitch in an informed way about how rubbish Caprica is and how Ron Moore should be made to wear a hair-shirt and personally apologise to everyone on the planet for his crimes against Good Endings That Don’t Make Me Want To Thrust Pencils Into My Eyes. Oh wait, some readers don’t like me mentioning that, best move on…

But anyway. Thanks to David, who since joining Literature & Latte has consistently won Employee of the Month every single month,** for saving the day by downloading a copy of the SDK and getting it to me on DVD the very next day. (Even the British postal system is better than you, BSkyB!)

Which is really just a roundabout way of saying that I now at least have the iPhone SDK (the one capable of iPad development), so I can take a look and make a more informed judgement on future possibilities for Scrivener when I get some time. This doesn’t change anything in the short-term, of course - Scrivener for the Mac remains my number one priority, Scrivener 2.0 will take up all my development time for the next few months, and I have no intention of doing anything that will have a long-term negative impact on the Scrivener I use and love myself. But it can’t hurt to take a peek, can it?

* I’m pretty sure Omen 3 was a Murdoch biopic sent back through time from the future.

** Can you guess how many employees we have?

Stardust and Inciting Incidents

Just to warn those who read this blog purely for Scrivener-related news: this post is only tangential to Scrivener, as it’s about my own fumbling attempts at writing (for which I use Scrivener, obviously), so you can safely tune out if you don’t like reading self-indulgent prattle.

One of the reasons I created Scrivener in the first place was that, over the years, in the process of attempting to write a novel I had written a lot of scenes, snippets and ideas but had no idea how they all fitted together into anything like a coherent whole. Rather naively, I believed that if I kept plugging away at writing down those ideas and scenes, one day something would magically “click”, and in my Eureka moment I would suddenly just know how everything I had written so far fitted together into a tightly interwoven whole that would win me the Booker Prize for sure.

As I say - naive; as though by working on the parts the whole would somehow assemble itself without much conscious effort on my part. Nowadays I understand that this would be the same as attempting to come up with a writing program by building a corkboard, an outliner, a word processor, and so on, and then hoping that how they all fitted together would just come to me while I was pondering the curious blue hair of the old lady sitting in front of me on the bus. Suffice to say, that’s not how I designed Scrivener. Sure, there were some small things I knew I wanted in Scrivener beforehand, but it took a lot of design to figure out how it all fitted together. (The defunct Scrivener Gold, an early beta of Scrivener, didn’t fit together. It had four separate interfaces for doing different things. In fact, I wrote a whole series of blog posts about my quest to work out a decent integrated interface. And Scrivener 2.0 integrates some of the features that don’t work so well together in 1.0. In other words, that design has taken years and is still ongoing.)

When, after a decade of halted attempts at finding what I was really writing about, that hoped-for Eureka moment still hadn’t come, I started thinking that maybe I would be able to figure out how everything fitted together if only I could get a good overview of everything. I wanted an easy way of assigning a synopsis to each of my ideas and scenes; I wanted to be able to move these synopses around as a way of moving the underlying text around; I wanted to be able to see the whole, the parts, or just an overview, and to work with each. I’ve covered the conception of Scrivener in depth before, but what I’m saying here is that I thought that by getting an overview of the whole I would start seeing how everything fitted together. Well, Scrivener has been around several years now, and although I have certainly got much further along in my projects since having Scrivener, I still end up stumbling at this point: where is it going? How does this fit together?

You’re probably ahead of me: just because you wrote it and used the same character names doesn’t mean it’s part of the same story. I wanted to pour everything into one story - the only problem being that I had no story.

Part of the problem is that over the years I’ve grown attached to some pieces of writing I’ve done that really don’t belong anywhere. Now, I’ve been very lucky in that developing Scrivener has brought me into contact with professional, published authors who have been very generous with their time and advice. (Yes, my dialogue with some users goes something like this: Author: “Hey, thanks for Scrivener. Just started using it for my new book and it’s really useful.” Me: “Great! Wait, you’re a published author?” Author: “Er, yes…” Me: “Tell me everything you know about the masonic secrets of writing a novel! Help!”) I used to be terrified of published authors, as in terms of what they have achieved they represent everything I hope for, so I’ve always put the very notion of a published author up on a pedestal, but it turns out that they’re human beings who use e-mail and are often really, really nice. Who’d have thunk it? And pretty much all of the authors whose time I have stolen have told me this in one form or another: KILL YOUR DARLINGS. It’s old advice, they say, and it’s really difficult to follow, but it’s good, sage advice. It’s taken me a long time to accept it, but they are of course right. I may be attached to all those passages of purple prose I wrote in my twenties, but really, if I’m just trying to find a place to slot them in so I can use them somewhere, I’m not really thinking about what this particular story needs, am I? And I’m certainly not writing. If anything, I’m avoiding it by trying to reuse old stuff. For instance, there’s one passage I wrote years and years ago about a girl being born in the hour when the clocks go back, when BST becomes GMT again. I was really proud of that passage. It had all these phrases that at the time I considered dead clever, such as “edited from the spool of time by the scissoring of clock hands” and others you’d probably laugh at. Even after the birth of my first child six years ago I clung to it, despite now knowing that its description - and my younger self’s understanding - of birth was woefully inaccurate. From that passage you would have thought that birth was something like a scene from Alien. Oh, and for years I was desperate to find a use for this sentence: “The precipitation precipitated his capitulation.” Oscar Wilde eat your heart out. Hell, I might still use that one, actually. Anyway, my point is simply this: I was going at everything backwards; “arse-about-tit” in the proverbial phrase. I was trying to construct a story out of parts that had been written randomly with no story in mind. Imagine a painter folding up the canvas into tiny squares, painting something that appealed to him or her on each segment, and then unfolding it and trying to figure out what the hell sort of picture could be painted around the edges to make sense of a ten-foot pillbox, a bonsai tree, Medusa in a tutu and a purple duck. Same thing.

Like many would-be-writers-but-probably-won’t-bes, one of my fondest pastimes is procrastination. Indeed, some have pointed out that I have taken procrastination to whole new levels - I decided that before I could write The Novel, I had to spend several years writing The Software in which to write it. (For the record, this blog post is itself procrastination.) And one of my favourite forms of procrastination is reading books about writing. (As any wannabe-writer knows, this is priceless - you can trick yourself into believing that you are furthering your writing efforts even though you really know that you are doing exactly the opposite because you are reading a book on writing instead of, y’know, actually writing.) For the longest time I rejected all ideas about structure, narratology, acts, and so on; and I was - and still am, if to a slightly lesser extent - sceptical of any approach to writing that tries to give you a formula. Still, in my desperation I’ve recently been a little more open to taking on board ideas about approaching structure, and I have found some things particularly useful. For instance, I found Jeffrey Alan Schechter’s assertion that in most stories the protagonist goes through four stages of character development (orphan, wanderer, warrior, martyr) particularly insightful. It sounds like one of those rigid paint-by-numbers formulas until you realise that all he is really saying is that a protagonist changes over the course of a story - he or she starts off in need of something and ends up having to make sacrifices to get it after all the easy options have been exhausted. The four archetypes are just a way of hammering into your head that you need to think about how your character is changing over the course of the story, and you can see them operating in the character arc of almost any movie or story. (Jeffrey, incidentally, is the guy behind Contour from Mariner Software, and there is a lot of other great stuff in his Contour theory.) But I’ve also been more open to the things you get told just must be in a story, whereas before my fingers were placed firmly in my ears as I sang, “La, la, formula-la-la-la-la, not listening.” And the one thing that everyone agrees on is this:

There must be an inciting incident.

Well, duh. That’s just a posh way of saying, “Something has to happen,” right? Except, I think I only recently realised how little importance I had attached to perhaps the most crucial part of working out my story. In his superb On Writing, Stephen King says that most of his books start with a “What if?” situation and then he writes the book to find out what happens as a natural outcome of that initial premise. And despite having read that, and despite all those authors I’ve hounded having told me that they usually start from something similar - whether they plan every little detail before they write or just make it up as they go along - I seem to have stumbled through years of failed writing attempts thinking, “I’ll worry about that later; if I just start with these characters and this location, something’s bound to happen…” Unsurprisingly, the things that tended to happen were all a bit jumbled and didn’t go anywhere. Then, I recently came across this site:

http://thedarksalon.blogspot.com

It’s the blog of American suspense and horror writer Alexandra Sokoloff. There’s absolutely loads on there - she used to be a Hollywood screenwriter, so a lot of it is about using screenwriting methods in writing fiction. One thing I found really useful, though, was her suggestion - which she repeats again and again - that if you’re stuck with a story, watch several of your favourite films and read several of your favourite books in your genre, breaking them down and making lists of very specific things. For instance, you might break down the structure of each one - she suggests using the eight-sequence structure such as the one described by Chris Soth and some other Hollywood screenwriters. Or you might just be looking at how the romance subplot is handled, or how plants and payoffs work - whatever it is you’re stuck on. Her main point, I think, is that if you’re going to use any sort of formula, it should be one that you work out on your own, based on your own favourite stories - because presumably they are what made you want to write stories in the first place. Now, these suggestions appealed no end to the procrastinator in me - not only do I get to read books (and websites) about writing and tell myself I’m working, but now I get to watch movies too (notebook in hand, of course). Brilliant! So I tried her suggestion.

At this point, you may be reading this and thinking, “Wow, you really are the most useless wannabe novelist I have ever come across. Just get on with it already!” I wholeheartedly agree. Although I would ask you, why are you reading this and not writing yourself, huh? Huh? So if you’re not thinking that and are instead one of the pitiful creatures like me who will try anything to help you get past three chapters before giving up in despair, you may find this useful. (Did I mention that you get to watch movies and pretend you’re working? Genius.) I got a lot out of it. For a start, I found that identifying the structure in the films and books I liked best made me less afraid of structure as a concept. A couple of authors have told me that although they would hate to outline their whole novel, they use sort of “stepping stones” - that is, they have something like ten or twenty sentences or questions written down that guide them along as they write the story; rough markers that give them something to head for. Actually looking for these markers (set-pieces, I suppose) in stories I liked, and thinking about how the rest of the story built towards them, was something I found really helpful. Everyone knows that if you want to write you have to be a reader first, but normally when I read I’m not thinking about how it all fits together - I’m just getting sucked into the story. Stepping back and looking at certain specific elements as I read or watched (rather from memory) was an eye-opener. Which brings me back to…

The inciting incident.

Everything I watched and read that I loved had a really well-crafted inciting incident. I say “crafted” because I doubt the writers were as lazy as I hoped to be and had the inciting incident fall wholesale out of the sky into their laps. Oh, I bet some of the inciting incidents came to them out of nowhere, but then they surely must have crafted them to do a lot more work. The one that struck me as absolutely brilliant in its elegance, though, was the one in Stardust (film or book). I’m not normally a fan of fantasy or anything even vaguely romantic, but hey, the film has Claire Danes and Robert de Niro in it… Actually, I love the film. Don’t get me wrong, Neil Gaiman’s book is great, but Stardust the film gives me the same great warm-inside feeling as The Princess Bride. Hmm, I probably shouldn’t be admitting to any of this in public. Whether you like the film or not, though, its inciting incident is perfect in that one single event kicks off the main plot and every subplot, putting into place all of the conflicts that will drive the narrative through to its conclusion. Namely:

The dying King of Stormhold, having several sons remaining alive, which is against tradition (they are supposed to kill each other), throws the “power of Stormhold” - a gem on a necklace - out of the window and declares that the first male heir to retrieve it will take his place as king. This being a fairy tale, the stone soars into space and hits a star, which plummets down to earth. Three witches witness the star’s fall, as does Tristan, the protagonist, who in trying to woo the uppity Victoria declares that he will retrieve it for her in return for a kiss. (If you haven’t seen the film, this all sounds rather sickly, I know - trust me, it’s a fun story. I mean, it includes Robert de Niro doing the can-can in a dress.) The star hits the ground over in the faery kingdom of Stormhold - and it’s a beautiful girl; turns out stars in this world can take the form of people. So, with that one action - the throwing of the stone - everything is set up: Tristan is going to travel into the faery kingdom to fetch the star, and we know he is going to find a girl, not a lump of stone - and we already hope that he ditches Victoria for the star; the King’s sons are now out on a quest to find the stone, which the star possesses, and we have seen that one of them in particular is murderous; and we soon learn that the three witches want to cut out the star’s heart because eating it will restore their youth.

You may not like the story itself, but I hope you’ll at least agree that this is a great inciting incident in the way it sets everything in motion. It made me realise how little I’d thought about the inciting incident in my own novel; in fact, it made me realise that I had several small, not-particularly-exciting inciting incidents that were only vaguely connected and that this was one of the big reasons why the various subplots I had been working on didn’t seem to gel very well. So, in light of this, I’ve gone back to the drawing board and am now looking at my various plot threads and working backwards to see how they could all have been fired off by the same inciting incident rather than different ones. I printed off the synopses of a lot of my ideas (yes, Scrivener 2.0 supports the printing of synopsis index cards) and am going through them with a fresh eye. It’s made me realise that I have too many What Ifs for one story, and by thinking more carefully about the inciting incident I’m actually starting to see which plot threads and characters just don’t belong in this particular story, no matter how much I like them. It’s hard, but I’m determined to kill my darlings.

And it’s still a bit arse-about-tit, I admit, but at least it feels like progress.

P.S. While I’m at it, I’ll just put in a plug for a couple of Scrivener-using authors who regularly touch on various aspects of writing on their own blogs:

http://www.davidhewson.com - David’s book The Blue Demon is out tomorrow and is the first book he wrote from start to finish in Scrivener (even though his books adorn shelves all over the world, it still took this one two years to get from final draft to publication, apparently).

http://www.neil-cross.com/wordcount - Neil’s past couple of books have been written in Scrivener and he is now documenting the process of writing his next on his “wordcount” blog.

(I’m sure both would be appalled to read the above and realise that, despite the great advice they have handed out to others and to me personally over the past couple of years, I am still a fumbling simpleton when it comes to my own writing.)

The iPad and Mac Development

So. The iPad. (You may have heard of it. It’s a neat little gadget Apple released last week without much fanfare.) There are commentators out there declaring it the world’s most expensive Etch-a-Sketch (unfair; it has no stylus), and others praising it as being as “magical” and “revolutionary” as Steve Jobs and Jonathan Ive would have us believe. My own opinion on the device is somewhat schizophrenic (in the colloquial sense, obviously). I’m split between my thoughts as a user and my thoughts as a Mac developer.

Most of what follows is my thoughts on, and reaction to, the iPad from my perspective as a Mac developer. That is, as a developer rather than as an end-user, and specifically as a developer who has dedicated the past several years to developing an application for the Mac platform. My views on the iPad as a Mac developer are different from my views as an end-user. I think it’s important to establish this up front, in case what follows comes across as me bashing the iPad before it’s even in stores. Far from it. As an end-user, my reaction can be summed up as: “Pretty cool for a lot of people!” As someone who works on a writing and organisational program for the Mac, my reaction is, in short: “Oh bum.”

So, just for the record: I like the look of the iPad. I think it could turn out to be a fantastic device for consuming media and information (which is what it is for - more on that below). The main problem with computers as they currently exist is that they make many otherwise intelligent people feel like idiots. No one has to write to customer support just to work out how to use their television. Part of our job as developers is to make our programs as easy to use as possible - I’d go so far as to argue that this is in fact the hardest part of our job. (“What do you mean you don’t understand why the outliner button is greyed out? It makes sense to me, and I designed it!”) And it’s sometimes tough to admit that if more than one user is getting frustrated by an element of your program, then it’s probably a design flaw. (Note: 2.0 has a more fluid and integrated approach to the outliner, corkboard and Edit Scrivenings buttons - no more arbitrary greying out. Also note: If anything confuses you about Scrivener, I will still never admit it’s a design flaw.) With the iPad, Apple have thought about what most users actually do with their computers and tried to make the resulting device as easy to use as possible. That has to be applauded, not least by those of us who spend time trying to help family members troubleshoot computer problems. In that regard, I think this guy nails it:

http://speirs.org/blog/2010/1/29/future-shock.html

But if you’re reading this, you probably don’t care much about whether or not the iPad will result in me receiving fewer phone calls from family members with computer issues; what you most likely want to know is what impact the iPad has on Scrivener, if any.

The first user-request for an iPad version of Scrivener came before Steve Jobs had even finished speaking - I don’t think he’d even got to the part about iBooks yet. (Just for the record, we developers have no inside information - we are just as much the “little guys” to Apple as everyone else.) Since then there has been a steady trickle of requests for a version of Scrivener that runs on the iPad. This is as understandable as it is inevitable. I have long objected to an iPhone version of Scrivener because Scrivener’s interface just would not scale down to the iPhone screen real-estate, and a scaled-down version of Scrivener that had to drop all of the features for which I created it in the first place always struck me as pointless. Moreover, I don’t see the iPhone as a serious writing tool anyway; it’s something for taking notes on when you’re out and about, sure, but for that you can use the brilliant WriteRoom and then import your notes into Scrivener using the simpletext.ws importer. (Yes, I know I’m a bit late in getting 1.54 out, which updates writeroom.ws to simpletext.ws, and many apologies for that - 2.0 has been taking up all my time, but 1.54 hasn’t been forgotten and will be out soon.) I’m also not averse to working with other iPhone developers to offer compatibility with other note-taking or outlining apps (quite the contrary - feel free to contact me if you are such a developer). But the iPad is a different beast isn’t it? After all, there is a keyboard dock for the iPad and Apple have even ported iWork, so how can my objections to an iPhone version hold up when it comes to the iPad?

So yes, you may quite reasonably be wondering: Are there any plans for an iPad version of Scrivener? And if not, why not?

The short answer to the first question is: Not at the moment. But I’ll attempt to give a full answer to both.

Before I continue, I’ll say again: what follows is my opinion as a Mac developer. My opinion as a gadget-geek and user is somewhat different, and will follow at the end. Sorry, but I have to reiterate this simply because I know from bitter experience that there is a minority of folk who vociferously take to task anyone whose words can be interpreted as constituting even the slightest attack on Apple (my partner once wrote a piece for a national UK newspaper - which was unfortunately poorly sub-edited - that dared to suggest Apple were a little too tight-lipped when it came to certain ongoing product defect and support issues at the time, and the ugly comments left by a handful of dolts on the otherwise brilliant MacRumors still make me ashamed to be of the same species as the commenters to this day) - though of course I’m sure this doesn’t apply to you, good reader. Right, enough prefatory apologies.

The iPad as Netbook Killer

Mac users (note: Mac users - Windows and Linux users have plenty of options) have been craving an Apple netbook for years now. The nearest thing they were given was the MacBook Air, which was a disappointment to many - myself included - partly because of its utter lack of connectivity unless you used wireless and paid-for adapters. Mainly, though, the problem was that it wasn’t cheap - certainly not cheap enough to become a college kid’s knockabout machine - and it wasn’t much smaller than a MacBook - just thinner. So thin, you could fit it in an envelope - it just had to be a big envelope.

(Another quick aside here: I’m not one of the users who have been craving an Apple netbook. I love my MacBook. It is my favourite machine ever, and given how much I loved my old iBook that is quite an achievement. But then I don’t keep it in my rucksack much - it gets plugged into my monitor and external keyboard during the day for development and writing, then taken downstairs and plonked on my lap in the evening so I can search IMDB to find out who the heck that actor on TV is, you know, the guy who was in thingy, with whatsherface. So as a user I am more than catered for; I’m happy. But as a developer, I have to take into account what other users are after, and how that affects Scrivener.)

Even after the Air, then, Mac users have been hoping for some sort of netbook. Something light, affordable, and with a 10” screen and keyboard. (A bunch of other users have been hoping for a tablet, but mainly for art and graphic design work, which is something the ModBook at least provides, even if Apple aren’t making money from it - it’s an entirely different sort of tablet to the iPad.) There is an elephant in the room here - a very small and possibly illegal elephant. I’m hesitant to mention it in case anyone misinterprets my doing so as condoning behaviour that violates Apple’s EULA (I’m not), but to omit it would be ignoring an important indicator that there is demand for a Mac netbook. I’m talking, of course, about the significant number of people who, fed up with lugging their MacBooks around and waiting for Apple to provide them with something smaller, have resorted to using a so-called “hackintosh” - that is, hacking OS X onto an existing Intel netboook such as a Dell mini 10v. These are otherwise loyal Apple users (although the idea of “brand loyalty” is rather sickly, isn’t it? May as well be loyal to a genus of flowering begonia), people who haven’t used anything but Macs for many years, who don’t want to use anything other than Mac OS X, but who just want something smaller on which to run their applications when out and about - writers on research field trips, students moving between lectures all day.

Did the iPad deliver what these users wanted?

Well, yes and no.

To quote Steve Jobs from his keynote speech:

“Everybody uses a laptop and/or a smart phone. And the question has arisen lately, is there room for a third category of device in the middle? Something that’s between a laptop and a smart phone? And of course we’ve pondered this question for years as well. The bar’s pretty high. In order to really create a new category of devices, those devices are going to have to be far better at doing some key tasks... Better than the laptop, better than the smartphone. What kind of tasks? Things like browsing the web... Doing e-mail. Enjoying and sharing photographs. Video... Enjoying your music collection. Playing games. Reading e-books. If there’s going to be a third category of device, it has to be better at these tasks than a laptop or a smart phone, otherwise it has no reason for being. Now, some people have thought that that’s a netbook. The problem is, netbooks aren’t better at anything. [Cheers and laughter of recognition from the audience, who have cleary thought exactly the same thing for many years now.] ... They’re just cheaper than laptops.”

(Digression: Honestly, if Steve Jobs had a sense of humour, after revealing the name of his latest product he would wait until the inevitable whooping and cheering had died down and reveal a steaming pile of faecal matter, or a dead kitten or something. I swear the crowds at these things more and more think they are at a U2 concert. But then I’m English, and to us a stroke of the chin accompanied by a muted, “Hmm, interesting,” could be considered over-effusive.)

The moment Steve Jobs opened his keynote with the above words (not the digression about the faecal matter and dead kitten, obviously; the bit about netbooks), my heart sank. It was as clear a statement as we’re going to get that Apple has no interest in entering the netbook market; at least, not with an actual netbook. So is Mr Jobs right? Is a netbook really no better than a laptop or smart phone at anything? Within the parameters he set above, I’d have to agree with him; the iPad is indeed probably better than a netbook - at the tasks he listed. But you’ll note that the list of tasks he gave excludes any kind of content creation. (Yes, later in his speech he showed off iWork for the iPad, but that was given to the team to “see what they could do”, and content creation was not on his list of essential tasks at which this new device should excel. I would say that is important, and telling.)

So, for the sake of argument, and ignoring for a moment the lists given on many other sites about multitasking and video conferencing and suchlike, what do netbooks do better than laptops from the point of view of the average Scrivener customer?*

1) Most importantly of all, they are smaller, lighter, more portable. (Great! - So is the iPad.)

2) They still allow you to type fast on a small keyboard. (Okay - so does the iPad, although you’ll need to carry around the keyboard and assemble it - fine for writing in coffee houses, but not so good if you need to write on your lap for any length of time, I’d wager. Of course I could be proved wrong on this - maybe the dock is particularly sturdy and good at balancing on knees.)

3) Despite their lightness and portability, you can still run your preferred OS, and therefore most of your preferred programs on them. (Ah…)

So, again, did all of those Mac users who wanted a netbook device - a more portable Mac - get what they wanted? Well, it depends. Yes (in a way) and no.

Yes, Apple delivered a small, light, affordable and portable device. It takes some extra assembly to add a decent keyboard, but you can type on it, and accordingly to initial reports it responds well to fast touch-typing.

But also no - because Mac users cannot run Mac OS X on Apple’s answer to the netbook. And thus they cannot run their favourite applications unless those applications are rewritten for the iPhone OS, which is what the iPad uses.

In other words, Mac users hoping for a Mac netbook are, frankly, out of luck. If you were after a more mobile device that runs Mac OS X, it still doesn’t exist - and although I hope I’m proved wrong, Mr Jobs seemed strongly to imply that it never will, because the iPad is their rebuttal to the very existence of the netbook; Steve Jobs doesn’t like netbooks, so you’re not getting one. But then, does it really matter that the iPad doesn’t run OS X? (Obviously not if you’re not a Mac user anyway, but what if you are?) After all, it does almost everything else, doesn’t it? Well, again, it depends. I would imagine that to the vast majority of people, no, it doesn’t matter one jot. Apple has provided a device that may well bring them even more customers - the iPhone wasn’t something limited to the Mac community, and likewise, the iPad has a global appeal. No Windows user is likely to object to it because it’s a Mac and they don’t like Macs - because it isn’t a Mac. In a sense, in terms of computers, the iPad is platform agnostic - no one worries that their phone OS doesn’t run the same OS as the one on their computer, and by not pidgeon-holing this device as a Mac, Apple are not limiting their latest computing device to users of Mac computers. Quite the opposite - with the iPad they are after the same general user-base they achieved with the iPod and iPhone - it doesn’t matter what OS you are accustomed to on your main computer, because this device is intended as something different. This is something new that is dedicated to making the access of all that content out there easy and convenient; it fits between a smart phone and a computer.

However, no matter how you look at it, it’s not a Mac netbook, and thus to those hoping to have the full Mac experience on a mobile device - to those who use their computers predominantly to create content and who therefore wish to run a full range of Mac productivity applications without pared-down feature-sets - that surely does matter. For those users, the iPad is not a viable alternative to a netbook. Such users are almost certainly in the minority, I absolutely understand that, and perhaps the point of the iPad is that computers were built for content generation and finally here is something for everyone else. So, the number of computer users who spend more time generating content (outside of the day job, that is) than consuming it must be a minority to begin with. And of those, most will be happy to use a desktop or laptop machine for content generation, and an iPad for general browsing and the odd bit of note-taking. It’s only the minority of the minority who will now still long for a more mobile laptop on which they can run the whole host of Mac apps. I still wish there was something coming for them, though - not least because a good number of Scrivener users seem to fall into that very minority.

And for me, as someone who develops for the Mac platform, it also matters. How much does it matter? I don’t know yet. Why do I keep asking myself rhetorical questions? Maybe I’m lonely. But I digress (again). Nobody can possibly know how much it matters because it’s not even in the stores yet. Half the pundits are declaring that the iPad is going to be a runaway success and every home will have one while the other half are saying it’s a damp squib that fails to be either computer or smart phone, neither fish nor fowl, and so is surely utterly pointless. It’s too early to know if either side will be right or something inbetween will be true. Whatever the outcome, even at this early stage it would be irresponsible not to consider the implications of the iPad for my own Mac application, especially with so many users already clamouring for Scrivener-for-the-iPad.

Mac Development and the iPad

Okay, I’ll admit it: when I saw that what everybody had expected all along had actually been realised - that the iPad was essentially a large iPod Touch - I was gutted; gutted because of the implications for Scrivener, which currently runs only on the Mac. Perhaps the biggest disappointment and frustration for me as a developer is that, in a way, I feel that Apple’s decision to use the iPhone OS for a tablet that is ostensibly intended as an entry into the netbook niche of the market (rather than saying straight up that it is something completely different to a netbook) is a bit of a two-fingers-up to us indie Mac developers; developers who have been quietly contributing great applications (at least, I like to think Scrivener is a great application…) to the Mac platform, and who have even persuaded users of other platforms to switch (I’ve lost count of the number of users who have e-mailed me to say that they bought a Mac just to use Scrivener - perhaps the biggest compliment of all). Getting people to switch is irrelevant to Apple in the case of the iPad, though, because it’s not a Mac, so hasn’t set out to rival Windows at all; its stroke of genius is that it sets out to replace a hardware solution (the netbook) and by adopting the iPhone OS instead of Mac OS X it raises itself above platform partisanship; it transcends the Mac/Windows divide.

I should step back a little here and explain what I mean by “indie development”, because this isn’t something that everyone realises about us. Often when we think of a software company, we tend to think of offices and cubicles and a handful of sweaty developers being hounded by a line manager, a PR team, some techies giving phone support and advertising folks on the other side of the office writing copy about how they are producing the “funnest” thing ever or suchlike. Or, at least, that’s what I used to think of when I thought of software companies. But that image is only true of the larger software houses (although I’m sure none of their coders are sweaty and many may even shower daily). Many software companies aren’t like that at all - we aren’t. If you’ve read our About page, you’ll know that we are just two guys. I wrote Scrivener because it was the tool I wanted for my own writing - in fact, I taught myself to program just so I could create Scrivener. I didn’t study programming at university (I studied history and medieval literature, since you asked). I’m only a “professional” programmer in the sense that it is now my full time job (since I gave up teaching because Scrivener had taken off). David came on board to help out with all the non-coding stuff (for a large pay cut, so he could work with his friend - which would be me). He’s not a programmer either (he studied physics, since you’re so nosey). There are no Literature & Latte offices. My office is a spare room in my rented home, with a view of the garden and the occasional dog-walker in the fields earmarked for housing estate development opposite. David’s office is his garage. (No snow days for him.)

The great thing about the Mac is that this sort of one-guy-and-maybe-a-friend software development is still feasible. Years ago a lot of software was written by only a couple of guys - the old ZX Spectrum games, for instance; even the first versions of Final Draft. But it’s getting more and more difficult for a single person to code and maintain a complex application for a large enough user-base. We could argue all day about Apple’s faults, but one of the great things about Apple is that they have really fostered a strong independent development community. There are some wonderful independently-developed apps on Windows, don’t get me wrong, but I’d argue that it takes Windows developers a lot more time and money to polish them up to a high spec, and to do so they have to pay for a lot of third-party frameworks and development tools. On the Mac, all your development tools are free, so you can try developing software even if you don’t know whether or not you’ll ever make a penny back, and you don’t have to think about start-up costs. At all. And the development tools are fantastic, a real pleasure to use. Even better, because Apple have control over all their hardware, you know that as long as you follow their guidelines and test your app on all the versions of OS X you want to support, it should then work the same on the computer of every user out there. Windows programmers don’t have it nearly so good.

So why is the iPad being based on the iPhone OS such a potential problem for Mac indie developers, in my own humble opinion?

Because suddenly our Mac apps don’t run on all the machines a user might expect them to run on. Because those of us with only one programmer are now faced with the demand for supporting two viable computing platforms. We could debate the semantics of whether the iPad is indeed a computer or not, but initial murmurs would indicate that there are a good number of users out there who want to use it that way, if only as a secondary platform to their main workstation or laptop.

Wait, surely the iPhone and iPad are teeming with applications from independent developers, so how can this be a bad thing for indie development? Well, yes. That’s true. I’m not trying to say that Apple aren’t still fostering a strong independent development community, because they clearly are - it’s just on a different platform, no longer the Mac. And it’s a fantastic thing for all those iPhone developers out there; I envy them the excitement they must be feeling right now about the iPad, along with all the software houses who have the resources to start thinking about an iPad version of their flagship programs straight away, such as the excellent Omni Group. But as an independent Mac developer, here’s the thing: The iPhone and iPod Touch are undoubtedly small mobile platforms - in the smart phone category - and even if users ask from time to time for a scaled-down version of their favourite app to run on them, no one really expects to do any serious work on them - that has until now been left to Mac OS X, which was - was - Apple’s computing operating system. But building the iPad on the iPhone OS and then declaring that it is intended as something to replace netbooks, and by providing a keyboard accessory and porting iWork, their own main productivity suite, to it, Apple have changed the game. The resulting implication is this: either you build an iPhone/iPad version of your application, or you miss out on all the users that wanted a netbook and so bought the iPad - because the iPad is Apple’s answer to the netbook.

That’s a pretty big blow, so I’m going to reiterate it: Mac users have been clamouring for an Apple netbook for years now. Users have been wanting something with a 10” screen that they can throw in their bag, something smaller, lighter and easier to haul around than a 13” MacBook. And I think it’s fair to say that everyone expected that, were Apple to introduce such a device, more users would come over to the platform from Windows. I get requests for a Windows version of Scrivener nearly every day, and along with price the other main reason many of these Windows users cite for not wanting to buy a Mac is size - they want a small, cheap netbook for college, something they can throw in their bag and carry between classes without shoulder-ache, for instance. They don’t want to carry a MacBook - it’s portable, but not something you want weighing on your shoulder all day. They want something smaller.

Well, this week Apple told us all, “You didn’t know it, but all along you didn’t want a netbook at all. This is what you wanted.” And then they introduced just such a cheap, portable device - but it won’t be bringing users over to OS X, at least not directly. So all those of us who develop for the Mac won’t see anything of those Windows users who are tempted by the iPad unless we switch our allegiances to the iPhone OS. And even worse, there are plenty of Mac users who have been putting up with a MacBook (which they consider too clunky) who are now talking about ditching their MacBooks in favour of an iPad. We could in fact lose users. On the other hand, as apparently happened with the iPhone, the iPad may draw users to the Mac platform indirectly - Windows users may buy an iPad, fall in love with it and discover that the nearest full computing experience is Mac OS X. Only time will tell.

So, there’s my dilemma. As a user, I think the iPad looks like a wonderful gadget, and I think it is going to be welcomed by the vast number of people who get frustrated by the often apparently arcane nature of computers - the people it’s for. I’m looking forward to trying one out myself, especially once they get iBooks running in the UK (I hope that’s the plan). As a developer, it makes me want to scream.

Projecting forward into the murky and unfathomable future, another concern must be whether or not this augurs anything for Apple’s long-term commitment to Mac OS X. Jonathan Ive has clearly stated that he sees the iPad as the future, the right direction for Apple. I’m not saying it’s a bad direction, but I do wonder where that will leave those of us still tied to traditional platforms such as OS X in five years’ time. Again, it’s too early to say; only Apple really knows what the future is for these platforms. But I think I’m justified in being a little worried about the implications of Apple’s netbook solution not running OS X for those of us who have dedicated ourselves to providing software for that platform. Will there ever be a Mac OS 11? And if there is, will it be a version of the iPhone OS? Or are we heading the same way the games industry has headed, with a handful of people using full computers to create content that is then delivered to closed devices (as with games programmers who write for the Playstation or Xbox, for instance)?

Before I’m accused of overreacting, let me just say that I don’t for one minute think that Apple intended this as a big-two fingers-up at its indie Mac development community. Of course not. Apple are just doing what they do - working on great technology with the end-user in mind, and they are right to do that, because that is ultimately their job. I’m merely expressing how this step feels as an independent Mac developer without the resources to support multiple platforms - because at the end of the day, the iPad (and iPhone) is, to all extents and purposes, another platform.

So, onto the reality of getting Scrivener onto the iPad.

Will Scrivener Come to the iPad?

Well, were the iPad running OS X, or even a stripped-down version of OS X, a port would have been a no-brainer. (There is a common misunderstanding that the iPhone OS is just a stripped-down version of OS X; it’s not. They both use some of the same core libraries and are built around Objective-C and an AppKit - Cocoa. But the classes available for use in each are very different. For instance, Scrivener’s text input is a highly customised version of Apple’s NSTextView. It uses a modified version of its rich text system. The iPhone has no such thing - it has a UITextView, which doesn’t even support rich text. Scrivener’s binder is a customised version of NSOutlineView. But the iPhone doesn’t support outline views at all to the best of my knowledge. The only way to navigate trees on the iPhone is by drilling down into tables. These may seem superficial UI differences, but the differences go much deeper. As I’m not overly familiar with the iPhone frameworks (yet) I’m sure someone else could explain the differences better, but suffice to say it’s not a case of just changing a few lines of code and rebuilding.) But given that the iPad is not running OS X, right now I can’t see a way for us to produce an iPad version of Scrivener in the near future. Not because I’m not interested in such a thing - although I am reserving judgement until I hold one of these things in my hands. Practicalities, however - the dreaded Real World - get in the way. So, below are some of the reasons why we are not planning an iPad version just yet.

1) Resources. We’re not Apple, we’re not Omni. As I say, there’s just me and David. Scrivener is a niche product - it has a whole bunch of users who are kind enough to rave about Scrivener, but not so many that we’re driving around in sports cars. So, we can’t afford to hire someone else to code Scrivener-for-the-iPad. But if I try to code it myself, then what happens to Scrivener-for-the-Mac? Realistically, I can’t do both - or at least, I can’t do both well. Scrivener is a big program, very complicated internally, with hundreds of thousands of lines of code. And there are still months of development to do on Scrivener 2.0. For me to develop a version of Scrivener for the iPad would mean abandoning Scrivener for the Mac (at least temporarily, but even after an initial release my time would then have to be divided), then learning to code for the iPhone OS, and then designing and building a mobile version of Scrivener. That’s not great for Mac users of Scrivener, even the ones who want an iPad version. (Oh, and did I mention I created Scrivener so I could use it myself? I really would like to get that novel written one of these days, too.) Perhaps the eventual ideal solution - should the iPad turn out to be a viable writing environment, which still remains to be seen - would be to team up with an iPhone/iPad developer who uses, understands and loves Scrivener and wants to give us huge swathes of the profits for his or her work (ha). But the problem there is that I’m a perfectionist control freak and like to keep control over everything that goes into Scrivener, so that’s not as straightforward as it seems. (In other words, I’m hell to work with; just ask David.)

2) Design. Is the sort of writing for which Scrivener is designed really the sort of thing for which the iPad will be used? In a way, the iPad - like the iPhone - is a notebook, a Moleskine, but Scrivener is the typewriter and corkboard, the book of clippings, the ring-binder - the stuff that requires more space. You might pull out your notebook in a café or on the train, you might even shuffle some index cards, but you wouldn’t spread your index cards out, label them, reshuffle your whole manuscript, pull out your clippings folder and lay out images and reference documents while you write in such locations - all the tasks that Scrivener takes up screen-space doing, the things it was built to do. You take your notebook with you when you’re working on a book, but not your whole ring-binder. Writing a long text is still for the most part something you do at a desk. Laptops enable you to take that desk with you; the iPad is very deliberately not the entire desk. It is stripped down, something to read, to play games on, for playing music and videos, and in a pinch you can use it to take down some notes or throw a document together if you really need to. Trying to force Scrivener onto something like the iPad seems to me to be missing the point of both Scrivener and the iPad itself. I am ready to be proved utterly wrong on this, but these are my impressions based on the way the iPad was presented to us in the keynote (see the section below on The iPad From a User’s Perspective), and until it hits stores that’s all I have to go on.

3) Inclination. I created Scrivener because it was the program I wanted to use for my writing - although Scrivener is now a business which pays my and David’s wages, hippy-at-heart that I am, money still isn’t my chief motivation for continuing the development of Scrivener. I do it because I know it could still be better, there are things I want to refine and get really right. But I love my MacBook; it’s the ideal writing machine for me. I can’t see myself ever using an iPad for serious writing - no, I haven’t seen one yet, but I write this with my MacBook on my lap with my fingers comfortable on a decent-sized keyboard and that works for me. So, I can get excited about the iPad for many things - for iBooks (maybe), for casual browsing, for watching videos - but not for use as a writing machine. Many Scrivener users may disagree and see the iPad an ideal writing platform, but if I haven’t the motivation - if I’m just doing it because I think I might make a quick buck - then I’m not qualified to do it right. (I know what you’re going to say: maybe the iPad isn’t good for all your writing, but what about notes and ideas while you are out and about? Well, again, in that case you could use something like WriteRoom and bring your notes into Scrivener later - there would be no advantage of having a version of Scrivener just for that.)

4) Scaling Scrivener down. This would be a massive feat in itself. It may sound simple - just do less! But it doesn’t work like that. And would a scaled-down version of Scrivener even be Scrivener? Would it do any more than other writing or notebook apps? Moreover, users are bound to disagree over what would be essential in a pared-down Scrivener. One user would argue it should just be a corkboard linked to notes; another would suggest it should be an outlining app, and yet another a table that allows navigation through text documents. Could the iPad even handle a .scriv file package? And what about users who have .scriv files of hundreds of megabytes - even gigabytes - in size? I can’t answer that yet. But I don’t relish the idea of cutting any of the features I built Scrivener to have, let alone starting from scratch. So the jury’s still out. In fact, that’s worth clarifying, as the difference between operating systems and the problems involved in porting software programs between them isn’t something that’s necessarily immediately obvious to end-users. Getting Scrivener to run on the iPad is not just a matter of taking the existing code base and cutting here, modifying there. I’m sure some of the code could be reused, but for the most part we are talking about having to write a completely new version of Scrivener from the ground up for a new platform.

5) The Touch interface. Although Scrivener is predominantly a writing application, a lot of the structural work is done with the mouse, or by tabbing around views and using keyboard shortcuts and the arrow keys to achieve the same effect. On the iPad, like the iPhone, this sort of manipulation is done using your fingers. While iPhone apps such as CarbonFin Outliner do a wonderful job at providing a full outliner in small environment, even on the larger screen of the iPad something such as Scrivener’s binder would be severely limited. Because you would need to use your fingers to manipulate the rows, moving items around would either be frustratingly fiddly or the rows would need to be so large that you wouldn’t be able to see many of them on screen at any one time. And given that one of Scrivener’s purposes is to allow you to step back to get an overview of your project, I’d say that’s not an insignificant problem. Think of all the features that would need rethinking: the binder, the way keywords, label and status are assigned in the different views, references, dragging between different folders that are far apart in the structure, and so on. In other words, not only would Scrivener need rewriting from the ground up, but as the Mac version is built to take full advantage not only of screen real estate but also of the keyboard and mouse, its whole interface - the very way the user interacts with the documents and meta-data on screen - would need completely redesigning for the touch interface.

6) I’m admittedly not a big fan of the App Store, because it’s a closed shop. Either Apple accept your app and put it on the store, or you can’t sell or distribute it. And Apple’s criteria for accepting and rejecting applications for the iPhone are notoriously nebulous and apparently whimsical. I certainly don’t relish the prospect of spending a year developing an iPad version of Scrivener only to have it rejected (perhaps because it is seen as competing with Pages or suchlike). Without transparency to the acceptance/rejection process, there’s no way of knowing in advance what problems you may run into. I have already read that Apple used a number of private APIs to create Pages for the iPad, for instance - but other developers have their apps rejected for using private APIs. Even legitimate workarounds can be rejected, though - one developer I know had an update to his app rejected from the App Store because it used HTML to render editable rich text, and Apple apparently wouldn’t allow this - meaning that only Apple are currently able to create apps capable of editable rich text on the iPhone. So while the App Store setup has worked very nicely for plenty of developers, it’s just not the sort of commercial environment I’m crazy about entering. I much prefer everything being open to all. I’ve nothing against Apple having their own store and limiting the products available there to only the ones they approve, but I don’t like the idea of that being the only way a user can access your application.

So, if you’re a user hoping for an iPad version of Scrivener in the near future - I’m sorry, but for all of the above reasons there are no plans in place yet. It’s not because we have anything againt the iPad - quite the opposite. Like many, I’m rather excited about the iPad. But even if there were an enormous audience for an iPad version of Scrivener and we were thus denying ourselves a fortune in not developing it (as a couple of enthusiastic users have insisted - perhaps not quite realising just how much of a niche application something like Scrivener is), this is sadly beside the point - because it’s not a matter of refusal but of being realistic about the resources we have available right now, and how to use those limited resources to best serve the user-base that is Scrivener’s foundation. Still, that’s not to say we aren’t interested in what happens with the iPad - we are, and we remain open-minded. How interested? Well, I have at least signed up for the iPhone OS developer program and ordered a book on iPhone development, if only out of curiosity and to see what is possible - to keep our options open for the future. Were it possible to develop something that didn’t take years (people often seem to understimate just how much work and code is involved in creating and maintaining a program of Scrivener’s scope) and required little maintenance (that’s never possible), so that it didn’t detract from the full Mac version of Scrivener, then who knows what could happen?

The iPad From a User’s Perspective

As I’ve said above, as an end-user, I like the look of the iPad. I could never get excited about the iPhone - I don’t use mobile phones much, and for the little gaming I do these days I much prefer my PS3 or XBox. I bought an iPod Touch and what I really like about it is being able to browse the web - for reading, not posting - in bed or wherever, without having to prop up my laptop. For me, this is where the iPad is going to shine. I can use it for reading online papers in as comfortable a position as I read real papers, and I can watch movies more easily while travelling; but then I usually read print papers and don’t travel much, so in reality I probably wouldn’t have as much use for an iPad at all (I used these same justifications when buying my iPod Touch and I don’t use it as anything more than a regular iPod. On the other hand, I could keep an iPad on my coffee table especially for those IMDB searches.) But I do like the look of it, just because it looks like the sort of thing I imagined to be futuristic as a kid. Give me a hovercar and a robot butler to go with it, and I’m happy.

Perhaps the most salient point I have heard made about the iPad is that it wasn’t really designed for us content producers anyway. To recap on Steve Jobs’ list of key tasks for the iPad:

• Browsing

• Email

• Photos

• Video

• Music

• Games

• eBooks

With the exception of writing e-mails, everything on this list involves the passive use of a computer. It’s not about getting things done. It’s a reference device. It’s the Hitch-hiker’s Guide to the Galaxy made real, and Ford Prefect didn’t write directly into his copy of the Hitch-hiker’s Guide - that’s not what it’s for. Because of the Scrivener support forums, I spend as much time posting to the internet as reading from it - but that’s not average use. Most users of the web spend more time reading than writing - and even heavy Twitter and Facebook users will have no problem firing off short messages with the iPad’s virtual keyboard. Likewise using the iPad to send e-mails to family and friends will be just as natural as sending texts. I spend my day on abortive attempts at writing a novel, writing code, and writing support e-mails. The iPad isn’t aimed at that sort of computer use. Why should it be? Almost every other computer in existence has been designed for producing content rather than specifically for devouring it, and yet there are vast amounts of people out there who just browse the net, write the occasional short e-mail, listen to music and so on. Sure, they may spend all day writing documents in Word and hammering out e-mails, but that’s work. At home the iPad may be ideal for them. And if they really need to write a letter or something a little longer they can always use Pages and the keyboard dock. Someone pointed out to me that this sort of computer use just hasn’t really been catered for until now, and it strikes me they’re right; you don’t need a four-track recorder to listen to a record or a typewriter to read a book, after all, but until now it has been computer-to-do-everything or nothing at all (or a smart phone). In that regard, the iPad could be a killer, and if enough users see it in that way then it could well be as revolutionary as Apple hope - it’s finally something really affordable and usable for the computer user who doesn’t really like computers.

(I’m still not convinced about iBooks. The idea of reading for long periods from a backlit screen really isn’t appealing. iBooks itself looks beautiful - but then it would, because so does Delicious Monster - but the beauty of the reading environment has little to do with reading itself; for reading, all you need is something that isn’t going to strain your eyes and a good font. But maybe it’s aimed at the casual reader who just wants to read a couple of pages on the train to work, who knows?)

So. The iPad.

My final thought on the iPad as a netbook replacement is simply this: does it really have to be either/or? Is the iPad really a replacement for the netbook, a better solution for everything a netbook can do, or is it really something different to a netbook, something better at some of the common tasks we use all computers for? Couldn’t Apple have created the iPad for consumers of content and still created a smaller MacBook running OS X for producers of content? Why not give Mac users, their most loyal customer base, what they have been hoping for as well as bringing a fantastic browsing and reading experience to the wider public with the iPad? It seems a particular shame given all the streamlining work that went into Snow Leopard - 10.6 seemed a surefire contender as a netbook OS. Sadly, the answer may be simply that the iPod and iPhone have given Apple a taste of the big time, and have made Mac users less important to the company as a target demographic. After all, why target 5% of potential users when you can target 100%? The result is that the iPad is a great device for casual computer users regardless of the platform they are used to, but Mac users are still left in the cold without a really portable Mac and independent Mac developers have to face the possibility of losing existing MacBook-using customers as well as potential switchers. Or perhaps ultimately all of this speculation will turn out to be no more than a storm in a teacup; the iPad could become wildly popular as a means of reading and browsing while writers and other creators continue to use their Macs for the act of creation itself. And who knows, maybe all that work on streamlining Snow Leopard wasn’t for nothing - we can’t rule out its appearance on a netbook entirely. Remember what Steve Jobs said about the Kindle and e-reading?

Incidentally, if you were wondering what that strange sound was throughout your reading of this rambling post, it’s nothing to worry about; it was just the sound of my knee jerking uncontrollably.

(On the plus side, this post was written in Scrivener and copied to Blogger using Scrivener 2.0’s new “Copy Special > Copy as HTML” feature.)

*NOTE: There is no such thing as an “average” Scrivener customer. They are all extraordinary. Did I mention we have a paid update coming out this year?