Making epub happen

Sigil 0.5.2 Released

2012-02-10T02:43:00.002+01:00

0.5.2 is now available for download! This is only a bug fix release but it fixes the "add existing file" feature that broke with 0.5.1.

There is no Linux x64 build at this time because my build machine died while building this package. I'll be looking into options to replace this machine but Linux x64 users will have to make due with the x86 Linux release for the time being. I've been having problems with my build machine (since the 0.5.0 release) but this time it looks like it's might be beyond reviving.

Sigil 0.5.1 Released

2012-02-05T22:11:00.000+01:00

This is primarily a maintenance release with a large number of fixes. Find and replace (F&R), GUI tweaks and spell check were the major focus. Highlights include: F&R now supports wrapping and the ability to search within selected files. The GUI now (hopefully) remembers cursor position when switching views. en_GB dictionary was added, all dictionaires were updated and hyphenation dictionaries are now included by default. For a full list of all changes for this release please see the Changelog.

This release wouldn't have come off as smoothly, quickly or with as many fixes and new features without the help of meme from MobileRead. Thank you for the time and effort you put into making this release possible.

Starting with this release I want to start making smaller releases with less time between them. I plan to target approximately 4 weeks between releases give or take two weeks. I believe this will make development more fluid and allow for greater user input.

My short term plans for upcoming releases are (not all of these will be in the next release):

Make book view editing more robust.
Firebug style element inspection.
Make code view editing more robust.
Plugin interface.
Move auto cleaning and restructuring into plugins.

My long term plans are EPUB 3. This is not a short term goal but I plan / want to make this a reality by the end of this year.

How Find Searches in Sigil 0.5.0

2012-01-29T22:31:00.002+01:00

There have been some confusion about how find works in, the now released, 0.5.0. The confusion stems from the the 0.4.90x betas. One method was used in the early betas and it was changed later on. This all stems from the regular expression engine being changed from QRegExp to PCRE. The issue at hand is how and when the cursor is taken info account when running a find. In this regard 0.5.0 works no different than 0.4.2.

When doing a count the cursor is ignored. The entire document is taken into account from start to end.

When doing a find next the find starts from the cursor location. Everything before the cursor is ignored and not taken into account. This can, in some cases when using a regular expression, lead to the number of matches being different from the total returned by count. Again, this can only happen when using a regular expression. The reason is a regular expression can have matches that match the expression within a single match. For example: Expression:

<div>.+</div>

Text:

<div>blah <div> blah </div> blah </div>

The expression will match the text from beginning to end. If you put the cursor to the right of the first < then the math will start from the second div and go to the end. This is because regular expressions can match a variable amount of text. Unlike a fixed expression like "abc" which will always match "abc".

Finding backwards will match from the start of the document up until the cursor position. This is done by finding all matches from the start to the cursor then using the last match. Again, in the case of regular expressions, a backward find can match different text than a forward find.

Find forward and backward find from the cursor so its position in the document taken into account. In the majority of instances find backward, forward and count will all match the same exact text. However, it is possible, due to their nature, to construct a regular expression that can match differently segments of text within a segment of text depending on where the cursor is located.

The above also applies to replace as a find is run to find the text to replace.

Consolidation of Sigil Help Forums

2012-01-28T01:18:00.002+01:00

For some time now Sigil has had two different help forums. One at MobileRead and the other as a Google Group. This has caused quite a bit of confusion because people don't know the best place to go for help.

I've decided to close (it's already done so don't ask me to reconsider) the Google Group. This was an easy decision because 1) MobileRead's sub forum gets more traffic, 2) I use MobileRead and I don't use the Google Gorup, 3) Most posts on the Google Group were unanswered making it a poor place to go for help.

Sigil 0.5.0 Released

2012-01-21T18:17:00.001+01:00

I'm happy to announce the release of Sigil version 0.5.0.

0.5.0 comes with a number of bug fixes and some major new features:

Inline spell check in code view
Support for PCRE in search and replace
Translations into 15 languages

Please see the changelog for a full list of changes in this release.

One smaller change is I've decided to drop OpenCandy. Surprisingly I've only encountered one complaint about OpenCandy and it was directed at bundling offers for other software inside of an installer not at OpenCandy in particular. I want to make it clear that this change is not due to user request or opposition to OpenCandy but my own decision.

While I respect what OpenCandy the company is doing and I don't see anything wrong with the offerings they provide I don't think their system is right for Sigil. The big thing I don't like is OpenCandy's installer components are distributed as closed source binary modules. While my understanding is this can be used without running afoul of the GPL I fully believe it goes against the spirit of the GPL and open source in general.

Thank you to everyone who provided feedback and helped during the beta process.

0.4.902 (0.5 beta) Avaliable

2011-12-13T03:55:00.002+01:00

The first beta for 0.5 (0.4.902) is now available.

There are a few new features I'm most interested in getting feedback on. Inline spell check, translations, and the new PCRE engine. Of course crashes and major issues will be looked into and hopefully fixed before the final release.

Sigil and Data Loss Bugs

2011-11-09T01:21:00.001+01:00

The majority of the data loss issues have been mitigated at this point. With a work flow of open, save as after major changes and saving after minor ones, catastrophic data loss can be worked around to the point that Sigil can and is being used on a day to day basis.

That said, there are issues with data loss in Sigil and they are a priority. I'm currently finishing up the 0.5 release (I do not have a set release date at this point) which is mainly a feature release and only addresses some of the the data loss issue. For example you can still have everything in an entire XHTML document removed by putting a malformed XML header in the document.

The issue has three components that require major work to fix. I hope to have it all completed for the 0.6 release but it's going to be some time it's ready.

The issues are:

1) Sigil currently uses Tidy to clean all XHTML to ensure it conforms (as much as it can) to the XHTML spec. I have seen Tidy remove tags it thinks are empty when they influence how the document is rendered. I want to keep Tidy as part of Sigil but I believe it should only be run when the user asks for it and any changes it makes the user should be able to revert.

2) An intermediate data store is used that requires valid XML is used. This store shuffles data between the book and code view. Due to this store requiring valid XML (valid XHTML conforms) there is the potential for data loss if it has to auto correct the XHTML. If you are in code view and have malformed structural issues with the XHTML and move out of it there is a warning dialog. This only appears when you are working on one file at a time. If you are replacing across multiple files auto correction is used and this can lead to data loss. This data store needs to be replaced with one that does not require valid XML.

3) Putting malformed content into the book view will cause the book view to try to correct it. Again auto correction can lead to data loss. This is mitigated by the malformed error dialog but many users just disable it and find that sections of their document are missing after looking at it in book view. Also, the book view is a WYSIWYG tool so it does make structural changes to the document and these may or may not be what the user expects. As with Tidy changes made by the book view need to be able to be reverted. I am thinking about ways to make the fact that the book view more obvious that it makes changes to the document. This way the user is aware that they need to use undo (doesn't currently work for book view changes) to revert the changes if they don't like them. I'm thinking about using a preview mode by default that doesn't make any changes and an edit mode to make this distinction obvious.

The above issues can be fixed but they are not quick or easy changes. I plan on making them for the 0.6 release as part of the changes necessary to support EPUB 3. However, there is the possibility that they will slip to 0.7 due to how large they are. Unfortunately, all I can say right now is I'm aware of the issue, I know what the cause is, and I have an idea of how to correct it but it's not going to happen tomorrow.

Sigil Now Supports Translations

2011-10-09T00:37:00.001+02:00

One of the the new features that has been implemented for 0.5 (release date yet to be determined) is support for Translations. For Sigil's first supported language Grzegorz Wolszczak has provided a Polish translation. Currently translations are loaded based upon the current system locale. There no support for choosing the language via preferences. This may come at a later time but for now I believe that using the system locale will handle the majority of user needs.

I've put together a wiki page with instructions for creating translations. This first revision is a bit basic but as people have questions I plan to update it to make it more robust.

Sigil Keyboard Shotcuts

2011-10-01T20:04:00.004+02:00

Thanks to Grzegorz Wolszczak Sigil now (will be part of the 0.5 release) allows users to change keyboard shortcuts for many actions. Grzegorz has been helping out a lot and helped to introduce a preferences dialog and provided user configurable keyboard shortcuts.

Sigil's Future Direction (Post 0.4.x)

2011-09-04T15:15:00.001+02:00

Introduction

With 0.4 my focus has been on getting the existing features in a stable state. I foresee 0.4 being around for quite some time as development shifts to accommodate new features. I wanted to be sure a relatively bug free version is available for people to use. If data loss is a constant then there wouldn't be any point in using Sigil. Now that 0.4 is done it's time to start working on what's next.

Just what is next? For the time being I've marked a number of issues on the issue tracker as Milestone-0.5. My plan is to have 0.5 just implement the most commonly requested and most interesting features. 0.5 has no vision and is just a stop gap while I familiarize myself with Sigi's code base. 0.5 is my short term plan. It's not grand but it's functional and sufficient.

Recently I posted the conclusion of my Sigil user study. The findings are Sigil is most used and most useful to power users and small professional ebook creating houses. Also, the overlap between the two is significant. Thus I want to target these two group and make Sigil even more useful for them. Keep this in mind because these two groups are who is going to shape my views of where I want to take Sigil.

Please realize that not everything I'm going to talk about is set in stone. A lot of it probably will never happend. Also, this is part plans, part what I want to do, and part rant about what Sigil does that I don't like. This is what my ideal Sigil would look like and it is what I'm going to work toward. However, nothing is set in stone.

Plugins

If you've ever used calibre or Firefox you will know that plugins are amazing. They allow for easy and quick changes and additions to be made without having to change the main application. Both calibre and Firefox have large third party plugin communities. I would like to bring this to Sigil and I want a framework where all book manipulation is available over a plugin interface.

My feelings with Sigil are plugins should make small self contained changes. Similar to calibre's heuristic processing. For instance, italicize common cases, up / down shift headings, and normalize CSS. To make plugins really useful I want to have a system where multiple plugins can be chained together and run in sequence. This would be super basic internal script functionality.

For plugins themselves I'm undecided about how they should be implemented. I don't mean API wise because that isn't even a thought at this point. I'm talking about what languages they should be able to be written in. C++ as a shared library will of course be supported because Sigil is written in C++. However, I want to Sigil to be able to load plugins written in scripting languages.

My first thought is Python because I'm very familiar with it and love to work with it. I'm also thinking about Lua and QtScript (Javascript without DOM). I don't support frameworks for every one of these languages due to the amount of maintenance required. So I want to support only one scripting language. Python is big and slow. Lua is small but doesn't have the advanced text manipulation libraries Python offers. QtScript is Javascript with is an abomination of a language. Added size of Sigil's install, execution speed, ease of supporting, knowledge by contributors and text manipulation support are all major considerations.

Editor

Currently Sigil does not respect the structure of existing files. When you open an EPUB in Sigil it restructures the file layout. It even goes as far as to rewrite each XHTML file by running it through Tidy. With 0.4.0 cleaning with Tidy can be disabled but pretty printing is still used and alters the XHTML. I absolutely hate this! If I want my XHTML or file structure changed I'll do it myself.

I want to change Sigil to not be as automatic. Restructuring and cleaning of the XHTML should be moved to plugins and run when the user requests it. This way a user can open Sigil, change the metadata, save, and the only thing that changes is the OPF with the metadata changes. Not every single piece of the EPUB.

I also hate WYSIWYG editing because it inherently must make drastic changes to the underlying code. I don't think it's a good idea to remove it though. I would prefer to have the book view default to a preview mode that is read only. There wouldn't be any changes made to the code by using book view. Read only is the default but the user should be able to have an edit toggle that will set the book view to edit mode which will work like it already does. This way a user can make changes that may not be valid or work, check them, see there is an error (say a missing tag) without losing any work. They can see the issue fix it and still be able to use WYSIWYG editing when they want.

Data Store

Right now XML (XHTML included) data is stored as a Xerces DOMDocument. This is then loaded into the book or code view depending on which one is focused. The use of a DOMDocument often leads to data loss. Putting malformed XML into a DOMDocument can have unintended consequences. Especially when then loading that into a QWebView and getting back a string.

I want to replace the DOMDocument with a plain string as the data store. This will prevent a lot of data loss, especially combined with the book view defaulting to read only. Further, this combined with not making automatic changes to the code will make the well-formed error warning unnecessary.

Not auto processing with Tidy and checking for errors automatically will allow Sigil to produce invalid EPUBs. I really don't care that this can happen. The tools (FlightCrew) will still be there to check that the file conforms to the spec. It's up to the author to ensure they're publishing valid EPUBs. An EPUB that is being actively edited doesn't have to be valid at all times. I'd rather put the onus on the person using Sigil to ensure their EPUB is correct before publishing versus having Sigil force validity at every moment.

Undo

Undo is terrible right now. Some actions cannot be undone, some can. The book view's undo is completely separate from the code view. You can't undo a replacement when doing it across all HTML files on files that aren't open in a tab. I want to see a unified single undo that allows for setting back out of any change.

Further along this line I would like some graphical display where you can look at the changes that have been made to make it easy to find exactly how far back to undo. Something like Apple's Time Machine but for the state of the book.

Conclusion

Here is where I want to take Sigil: less hand holding, less automatic changes and more advanced text manipulation though a plugin interface. The big question is, should I skip putting out a 0.5.0 release with just the Milestone-0.5.0 marked changes and get started on the above now?

Sigil and Linux Distribution Packages

2011-09-04T01:18:00.000+02:00

The official Linux packages for Sigil are generic packages. They're bundled in an InstallJammer installer and contain a number of libraries that Sigil depends on. This is not ideal but it's not possible to provide Linux packages for every distro.

I've created a wiki page which I'm putting together a list of Linux distributions that have their own Sigil packages. These are the best packages for users to install because they're smaller and tailored.

If your distro isn't listed and it has Sigil packages let me know and I'll add it to the list. If your distro doesn't package Sigil let them know you would like to see them package it. I'm always willing to lend a hand to get Sigil in more Linux distros.

Sigil 0.4.2 Released

2011-09-02T14:48:00.001+02:00

Sigil 0.4.1 is complete and available. This is mainly a maintenance release and fixes a number of bugs. Specifically a few bugs related to data loss. There was one major user visible change. The well-formed error dialog can be toggled not to show. This will cause errors to be auto fixed. Use this with care because the auto fix Sigil makes might not be what you want. As always see the changelog for a complete list of changes.

Sigil User Study

2011-09-01T01:45:00.002+02:00

Introduction

Since taking over as the maintainer of Sigil I have spent some time reaching out to specific people in the ebook community to ask them about Sigil. Specifically if they use Sigil? Why or why not? What do they see as Sigil's shortcomings? How do they use Sigil in their work flow? Why doesn't Sigil work in their work flow. Basically, their thoughts and opinions on Sigil.

I asked specific people privately because I didn't want to be inundated with responses. The people can be broken down into three different groups: self publishers, power users, and professionals. After talking to professionals I've come to realize that they can be broken down into small and large. The size relating to the size of the company and production volume. I spoke with about 8 people total and I tried to keep it even between the various groups.

I wanted to find out who is using Sigil, who isn't using Sigil and why so I can determine where I want to take Sigil in the future. The only ebook editing I do is cleaning up a few books here and there. Learning how people use Sigil will help me to determine the best direction to take the project.

Self Publishers

Self publishers are authors. These are people who write their book and then want to sell it as an ebook themselves. Typically these people are using Word for writing. they export their work as HTML, then import into an ebook editor for final adjustments and savings as an ebook file. The two biggest things self publishers are looking for are easy and high quality .doc or .docx import and one click send to store functionality.

Self publishers are also interested in WYSIWYG editing and don't want to know about the internals of ebooks. They are primarily writers who see ebooks one of many distribution methods. They don't care about the intricacies of EPUB for instance, they just want their work to look good and be readable by their audience.

The typical tools I hear being used by self publishers are calibre for format shifting. Atlantis Word Processor and Jutoh for formatting and base ebook creation. Atlantis and Jutoh both provide very easy to use WYSIWYG interaction and you can use these without ever seeing a line of code.

Power Users

These are people who prepare works in their spare time as a hobby. They are not motivated by money and do not sell the works they publish. Typically the works power users deal with are public domain such as Shakespeare. This group also encompasses people who do not distribute works covered by copyright but spend their time cleaning and reformatting their favorite books strictly for their own enjoyment and personal use.

Power users are comfortable using either WYSIWYG and code editors. The biggest feature requested and talked about by power users is robust regular expression support for search and replace. Many of the books power users work with have terrible and often non-existant formatting. These works typically started life as either a scanned copy of a print book or a PDF file. Both of which typically leave broken paragraphs and misspellings thought the document. Which leads to spell check being the next most common request from this group. They are trying to take a jumble of half sentences and put them back together into a visually appealing layout.

The tools used by power users are Sigil, calibre, Word or Open Office macros, and many custom scripts. Also an advanced text editor like BBEdit and Notepad++ are must have tools.

Professionals

Professionals format ebooks for one purpose, money. This is what they do for a living. An author comes to them and pays to have the company turn their work into an ebook. For a modest fee an author can have a beautiful ebook produced without any headaches or hassle. Many authors prefer paying someone to do this portion of publishing for them just like they will pay an editor to edit, a print house to print, cover artist to design a cover and so forth. Authors write and typically want to concentrate solely on writing. Many self publishers format their own ebooks out of necessity because of the cost of hiring a professional.

With both small and large professionals I'm specifically talking about ebook publishing and digitization services. I'm not talking about huge publishers like Macmillan that do everything. However, the larger publishers I talked to makes me believe their process is the same as the huge publishers. The big difference between small and large professionals are the tools they use.

Small

Small professionals tend to use either Sigil or Adobe's InDesign for a good portion of their work. Both fill a very similar role in ebook creation. The big draw of InDesign over Sigil is InDesign supports print book layout creation. It's an all in one tool. This type of professional tends to use off the shelf tools that are readily available. Sigil and InDesign are not the only exclusive tools they use but one or the other tends to be a heavily used tool in their tool box.

Large

Large professionals tend to use custom tools. They staff people who's sole job is to develop and maintain ebook creation and formatting tools. They can afford to have custom tools that integration directly into their process. They don't use off the shelf or vanilla tools. This group is all about custom everything. This allows them to quickly adapt to changes.

Professional Tools

Sigil or InDesign and custom tools are all I know. Many professionals are vague about their process and tools. Some even declined to talk to me at all. They use tools in some way that works for them but their methods and implementation are proprietary.

What Does This Mean For Sigil?

Out of all of these groups I have little desire to target self publishers. There are existing tools that do a great job of meeting this groups needs. Sigil has a WYSIWYG editor and it can certainly be improved but I don't want to tie Sigil to a particular store or stores like Amazon or B&N. Also, I want to keep Sigil as an EPUB editor and not a generic ebook editor. I believe that Sigil's strength lies in being able to manipulate the internals of the EPUB format itself. I want to target this aspect more.

Power users are the major group I want to target. Out of all of the people I spoke with power users use Sigil the most and get the most out of it. Advanced editing of an EPUB's structure and code is where I want to take Sigil. That along with advanced text manipulation. Think expansion of calibre's heuristic processing.

Small professionals are major users of Sigil and I do not want to discount them. I believe that their use of Sigil overlaps with power users enough that targeting power users will also target small publishers. I do not want to alienate small professionals and will continue to take their needs seriously. From what I've learned about small professionals tools that make code manipulation easier will be a benefit and hopefully reduce their need for other formatting tools.

The last group, large professionals, do not use Sigil. I don't believe that changing Sigil to accommodate this group will get them to use Sigil. They use their own custom tools and Sigil doesn't fit into their work flow and I don't see it ever doing so. Thus I don't see it being worth while to work toward making Sigil "the tool" for this group.

Sigil 0.4.1 Released

2011-08-26T13:05:00.001+02:00

Sigil 0.4.1 is complete and available. This is mainly a maintenance release and fixes a number of bugs. There are a few new features mostly around the code view. As always see the changelog for a complete list of changes.

SCM Move to Git Completed Sucessfully

2011-08-20T15:08:00.001+02:00

Last night I moved Sigil's SCM from Mercurial (hg) to Git (git). The change was completed successfully and without any issues. 0.4.0 had locations for things like the user manual pointing to the git locations. These are now live and working again. These links are now broken in any release before 0.4.0. This is unfortunate but unavoidable.

Sigil 0.4.0 and FlightCrew 0.7.2 Released

2011-08-15T13:03:00.002+02:00

The long awaited Sigil 0.4.0 release is now out. Along with Sigil is a new release of FlightCrew. FlightCrew version 0.7.2 is the latest version bundled with Sigil.

If you're an OS X user then one thing need to be aware of is these two releases bump the minimum OS X version to 10.6 (Snow Leopard). The Linux builds (x86 and x64) are built on Ubuntu 11.04.

A lot of work went into 0.4.0. New features and bug fixes galore. See the ChangeLog for full details. I have to thank Strahinja Marković (the original creator) for leaving the 0.4.0 in a nearly finished state. Also, Charles King for being a bug fixing monster and helping make this release great.

One major change that is going to take place next week is, I am switching the source code management (SCM) system from Mercurial (hg) to Git. This will break the update checker in previous versions. Hence waiting a week before making the change. I have already converted FlightCrew (needed to test) so FlightCrew's updater in previous version won't inform you of an update. This is unfortunate but unavoidable.

Due to the change to Git the links for the manual in Sigil are not going to work for a week. This is because I've updated the locations in Sigil to point to the new locations that won't be available until the switch to Git.

Donation change

2011-08-07T04:54:00.002+02:00

I've updated the donation links they now point to my PayPal account. Going forward donations will be directed toward me instead of Strahinja. This is something we had talked about and planned. I just want to warn people who have donated before because you will see a different email address than least time.

Thinking of Changing Sigil's SCM

2011-08-03T02:54:00.006+02:00

Right now Sigil is using the Mercurial SCM (source control management). I'm thinking of switching to Git. Since taking over Sigil I have had one person contributing and he is okay with the change.

I'm not looking to change simply for the sake of changing. Before Sigil I have never used Mercurial. I am not well versed with it and I have spend 50% of my time fighting with Mercurial. Charles (the person contributing bug fixes) has had the same experience.

My personal preference and favorit SCM is Bazaar. If I could I would switch to it in an instant. However, Google Code does not support Bazaar. Google Code is a good platform and I like it a lot. I think that Google Code does everything except for SCM right. If it supported Bazaar it would be perfect. That said I have no plans to move away from Google Code to Launchpad.

Since I can't use Bazaar I'm left with Git. Git works well enough and I'm more familiar with it than Mercurial. One reason I'm thinking of switching is, Git is very popular. Using an SCM someone isn't familiar with will prevent them from submitting patches. I'm hoping that by moving to Git it will make Charles and my lives easier. I'm also hoping it will encourage more people to hack on Sigil.

Windows Acquired

2011-08-03T01:15:00.001+02:00

Thanks to a Sigil user (Bryan) I now have in hand a copy of Windows 7 Ultimate. Thanks Bryan for sending me a copy of Windows! The plan is to get RC 2 builds out this weekend.

All 0.4 blocker bugs squashed!

2011-07-28T03:20:00.002+02:00

Tonight I was able to fix bug 813 (see 837 for details). This is the last blocker bug for the 0.4 release. I merged a few patches there were sitting around. I've also fixed and issue with changes to metadata not being relayed to the GUI as the file having been changed. With these bugs fixed 0.4 is in a state that I feel is ready for release.

I have little experience with building releasable, binary packages for OS X and Windows. So It will be a few days (maybe weeks) before they're out. There is a strong possibility I need to buy a copy of Windows and that will take a few days to get. Also, I need to actually setup a Windows build environment. I'm going to try building using an old netbook I have (it's the only computer I can use that has a copy Windows for building packages.

I don't want to do this release piece by piece so I'm going to wait to release all packages (OS X and source) together. Also, I plan to put out an RC 2 before 0.4 final. The RC 2 will be exactly the same as 0.4 final as far as the code is concerned. RC 2 is to make sure I package Sigil properly. It won't do any good to put out 0.4 that can only be run if you have all of the development libraries Sigil uses installed independently of Sigil.

Taking Over Sigil

2011-07-25T03:34:00.000+02:00

The other day it was announced I am now the maintainer of the Sigil. Back in June Strahinja announced that he was looking for someone to take over the project. I highly respect the work Strahinja has done with Sigil and this is a project I don't want to see die. I myself use Sigil one occasion and when dealing with e-books I often find myself recommending its use.

After seeing the announcement by Strahinja that it was time to move on I contacted him about taking over in his stead. He agreed and has now given me control over the project. From this point on I will be handling releases, bug wrangling and everything else that goes into managing an open source project. I don't plan to remove Strahinja's access to the code repository. If he asks I will but Sigil started as his baby and if he wants to start working on it again I fully trust him and I would have no problem with this.

The good news is Sigil is not going to die tor stagnate. I am fully committed to continuing the project and bettering the application. That said things are different now than when Strahinja was here. Strahinja was pretty much a one man show when it came to fixing bugs and implementing new features in Sigil. He had a lot of time he was able to spend working on it. I unfortunately do not have nearly that amount of free time to work on Sigil. I have a day job unrelated to programming, publishing, or books in general. I can only work on Sigil in my spare time (mainly after work and weekends). I also work on other projects too which will be sharing my free time with Sigil development. I cannot match the pace of development Sigil users have come to expect so unless others step up to help me with coding development will slow considerably.

My plans for Sigil are as follows. Short term I want to get 0.4 released. Currently it's sitting at RC 1 and there are only a few small bugs I would like to fix for it. I have been spending the past few days becoming familiar with Sigil's build system. Once 0.4 is out I will need to spend some time getting to know the ins and outs of the code itself. From there I will move onto working toward the 0.5 release. Right now I'm going to commit to a hyperlink editor and spell check support for 0.5. I will need to look over the existing bugs and see what else would be a good fit for 0.5. At some point when the EPUB 3 specification is finished I will work on bringing Sigil up to speed with it. That will take place in what ever release number Sigil is at at that time.

I'm not going to give any release target dates for either 0.4 or 0.5. Each Sigil release will take the form of when it's ready it will ship. One big difference between me and Strahinja is undoubtedly how we handle version numbers. I only use this system, major.minor.revision. With major numbers 0 is for feature incomplete and an unstable API. Going from 0 to 1 simply means that I feel the application is mature, stable and has a set API. Going from 1 to any other number means it's a massive change in some way (features, functionality, UI, API...). Minor numbers are for new features. Revisions changes mean there are no new features only bug fixes. Up until 1.0 is released the majority of releases you will see will be minor release numbers. Such as 0.4 followed by 0.5 instead of 0.3.0 followed by 0.3.1 and so forth.

Aside from Sigil development there is also the Sigil development blog. I plan to use it to communicate Sigil announcements. I do want to point out that I do keep a personal blog which I also use. I will be posting all Sigil announcements on my blog as well as on the Sigil blog. However, I plan to keep posts on the Sigil blog to only Sigil content. If you want to keep up with everything I'm working on including Sigil then read my blog. If you want to keep up with only Sigil then read the Sigil blog.

As always feel free to contact me with any questions or concerns. I am always available to help if I can. MobileRead, email or blog comments are all ways to get in touch with me. However, I do ask that support type questions be directed to the appropriate section of MobileRead as I'm not an expert on all aspects of e-books. There are a lot of smart people there who can help too and often you'll get an answer faster than asking me directly.

All good things…

2011-07-23T17:51:00.001+02:00

As of today, the official maintainer for both Sigil and FlightCrew is John Schember (“user_none” on MobileRead). He’s a very bright and competent guy. He’s got what it takes, trust me. As a community, please give him the same consideration and respect you gave me. It will take him many weeks (if not a few months) to get fully up to speed with the codebase, so patience and a warm welcome from the community will make sure things go smoothly.

To those that are still on the fence on whether to contribute to Sigil’s development or not, get off that fence and help out John. :)

New maintainer needed

2011-06-16T18:57:00.001+02:00

Ok, so this is a tough post to write. A lot of you are going to take it the wrong way, I know that. It’s human nature. But try to resist that urge.

Here goes: I plan to transfer project ownership and development to a new maintainer by mid-September at the latest.

Don’t panic!

Let’s take this step by step. First, the reasons.

The mighty “Reasons”

As some of you may know already, I started Sigil while I was a CS undergrad. I made it both my Bachelor’s and my Master’s thesis so that I could justify the time investment and continue working on it.

All of that is now over. I graduated. I got a job. Life got complicated in numerous other ways. That’s three reasons right there. But there are others. For instance, I’ve been looking at this codebase for over two years now, and frankly I’ve grown weary of it. I want to work on other things in my spare time, contribute to other open-source projects. Running such a popular project comes with it’s own set of obligations and responsibilities, and I’d like to not have them anymore. I’m worn out and tired. Some of this has been covered in the “serious conversation” post.

Bottom line, come mid-September, I don’t believe I’ll be able to continue contributing to Sigil at the level both you and I are used to. It will be time for someone else to take over and lead this project to future pastures.

How’s all this going to work exactly?

I’m not just going to hand this over to the first person who raises his hand. Fuck that. If you want to be the new maintainer, you’re going to have to earn it. What do I mean by that? Well you’re going to have to convince me that you can manage this beast. There’s a distinct reason why I brought this all up three months in advance. Currently, I have all the time in the world^[1]. This means I can and will dedicate a lot of time to Sigil. I plan to get a lot of stuff done in this time frame. I’ll also be more than willing and able to answer any questions a new Sigil developer might have.

If you want to become a new maintainer, here’s the process:

Send me an email to sigil@gmx.com (or my personal email address, or a PM on MobileRead). Introduce yourself, your background etc. Basically say “hi”.
Come up with some feature you’d like to introduce to Sigil or a bug you’d like to fix. Start small. Explain to me how you’re going to implement the change, code-wise. If you don’t have a clue what you want to work on, just say so and I’ll get you something small and easy to whet your appetite.
Create a new clone of the codebase. Sigil uses Mercurial. Don’t know Mercurial? Here’s a great tutorial by Joel Spolsky.
Implement the change and push it to your clone. Please do your best to follow the code style. I know it’s not written down (I’ll fix that eventually), but it shouldn’t be too hard to figure it out by just looking at the current code. Don’t worry too much about it, I’ll point out any issues if they arise. A few pointers:
- Always keep readability in mind, it’s the single most important factor of any codebase.
- Use long, descriptive names for functions and variables. Again, look at the current code.
- Keep functions short and simple. Split long functions into multiple smaller ones.
- Comment the why, not the how. If the how is really complicated, then do comment it, but try to make the code simpler first.
Tell me when you’re done so I can take a look at it.
I’ll merge your changes after any points of contention are resolved.
Don’t forget you can ask as many questions as you want. I’ll be glad to help.

Continue making good changes to the codebase and we’ll get along swimmingly. Do great and supervision will gradually be reduced. Do awesome and become the new maintainer, eventually.

I’d love it if several people came forward wishing to contribute, even if they didn’t want the maintainer’s role at the end. The more the merrier. In fact, I’d love it if one of the major commercial users of Sigil hires a group of people to work on the project. I’d gladly transfer control to them as a group, as long as promises are made that Sigil will remain FOSS.

I’m a user of Sigil. Should I panic?

Don’t panic. Popular open-source projects don’t just die out when the original creators move on. That just doesn’t happen. In fact, new maintainers taking over is a natural progression of the FOSS model.

I’m not going to disappear either. I’ll be around and active for the next three months at least, and I’ll probably contribute small fixes/features on an irregular basis after that too.

[Related discussion can also be found in this MobileRead thread.]

Footnotes

[1] Actually nowhere near it, but let’s pretend.

Thesis done, job secured… next?

2011-06-09T14:52:00.001+02:00

Two important things happened. The first is that I have finished my Master’s Thesis. My mentor signed off on it and it’s currently being printed and bound. That’s a huge weight off my shoulders. Everything is on track so I should be graduating in July.

The second important event is that after months of job offers, interview prep, flying around the planet so the interviews could be conducted etc., I have finally accepted an offer. From Google, no less. I’ll be moving to Mountain View in the San Francisco Bay Area some time in September or October of this year.

I’d like to thank all the companies I’ve interviewed with/received offers from for their time and consideration.

What does this all mean for Sigil? Well it doesn’t impact my timeline much. I have pretty much the entire summer to screw around, and a lot of that time will go to Sigil dev work. After that… well that’s material for a different blog post you should see in a few days.

Why is Sigil 0.4.0 taking so long?

2011-05-05T18:53:00.001+02:00

You, the users of Sigil, deserve an answer to this question.

First of all, it has nothing to do with Sigil itself. The amount of work needed to fix the remaining release-blocking bugs is not that large. It’s just that I don’t have enough time to dedicate to Sigil right now.

Let’s back up a bit. Remember that in February I talked about how my university studies are pretty much over? A part of that meant that I needed to start looking for a job. There were (and there still are) a lot of options and offers on the table, but I wanted to explore opportunities of working for one of the big international tech corps.

So I wrote a resume and sent it to several companies I thought looked interesting. All of them contacted me with a desire to start the interview process.

This has required two things:

Time to prepare for all the technical interviews.
Time to fly from one part of the planet to another so the interview could be conducted, and then back again.

This doesn’t leave a lot of room Sigil work. Sorry. All schedules for 0.4.0 are in the water since as soon as I return to Zagreb from one set of interviews, I have to prepare to leave for a different set with only a few days between them to rest up and handle local affairs. Don’t forget, this all goes on top of my usual non-Sigil related workload.

I have at least another week of this. Ugh. After it’s all done and all the offers (or rejections) are in, I’ll examine all of my options and then pick a company.

Sigil 0.4.0RC1

2011-04-22T15:06:00.001+02:00

We’re in Release Candidate territory now. You can get the new release from the downloads area. Changelog follows:

fixed an issue with splitting on SGF chapter markers creating the new HTML files in the wrong order (issue #828)
fixed a rare crash/memory corruption issue with automatic OPF updates
made the CSS resource path updates faster and more robust
updated FlightCrew so that CSS resource use is now far more robustly detected (issue #822)
fixed a rare hang when opening the Meta Editor on Win XP machines
a more accurate error message is now displayed for problems with file loading/saving (issue #772)
fixed an issue with incorrect font filepath updates in the CSS (issue #736)
fixed an issue with paths in the OPF not being URL-encoded (issue #823)

Looking at the outstanding bug reports, 0.4.0 is pretty stable now. There’s only one bug that I really want to fix before making the FINAL, and that’s issue 813. You can’t see the issue on the tracker since it’s private, but issue 837 is a duplicate of that issue and it’s public. Basically, there’s something wonky going on with FC validation on Macs when the FC engine is called from Sigil. I still haven’t been able to fix this, mostly because my access to Mac hardware is limited (and I can’t really debug a Mac issue on a Windows box, can I?). I expect to kill this before FINAL.

Anyway, use the new RC and report any bugs you find.

Sigil 0.4.0β3

2011-03-24T10:00:00.001+01:00

The third beta of 0.4.0 is now in the downloads area. Changelog follows:

added a workaround for loading broken epub files that use the incorrect mimetype for the NCX (issue #815)
fixed a rare issue with loading epub files with OPF's that had the XML version set to 1.1, but also had other attributes in the XML declaration (issue #812)
fixed an issue with entries appearing at random in the TOC after a split on SGF markers is followed up with "Generate TOC from headings" request (issue #804)
fixed an issue with files not appearing in the Book Browser after a split on SGF markers (issue #816)
fixed a regression that caused the auto-cover-setting heuristics to unset covers when they are already set (issue #806)
opening an HTML file now automatically builds a TOC from the headings
fixed a regression that caused the Add Semantics menu to stop working (issue #807)

Again, keep the bug reports coming. :)

Sigil 0.4.0ß2

2011-03-21T14:29:00.001+01:00

New beta is now in the downloads area. Changelog follows:

fixed an issue with the ID of the manifest item of a resource not being updated when the resource was renamed
fixed an issue with chapter splits being created in the wrong reading order (issue #797)
fixed an issue with some HTML files disappearing after a save/load cycle if chapter splitting was performed before the save
fixed an issue with renaming a file and then splitting it causing duplicate ID's in the OPF manifest (issue #800)
fixed a validation issue with the Meta Editor not adding the correct namespace prefix to some Dublin Core metadata element attributes

Keep the bug reports coming. The more you report, the more I can fix and the happier we'll all be.

Sigil 0.4.0β1

2011-03-20T20:46:00.001+01:00

So the first beta of 0.4.0 is now out. You can get it from the downloads area. The changelog follows:

fixed an issue with CSS @import rules in the '@import "something.css"' format not being recognized and thus not updated on import
removed the "CustomID" basic metadata entry from the Meta Editor; those wishing to use custom ID's can now add them directly to the OPF
Sigil now preserves custom unique identifiers in the OPF (issue #552)
removed support for the Sigil-proprietary SGF format
the user can now edit the OPF file by hand (issue #281)
the user can now edit the NCX file by hand (issue #282)
the OPF file is now preserved on import (issue #586)
the NCX file is now preserved on import (issue #283)
the Table Of Contents editor has been replaced with a new Table of Contents sidebar; clicking on an item in this sidebar takes the user to the target location, enabling TOC navigation (issue #100)
a dialog now informs the user if his XHTML, NCX or OPF documents are not well-formed XML (with error location and description), thus allowing him to fix the potential problems by hand instead of leaving them to Tidy to fix (issue #519)
fixed a rare issue with no tab opened by default when loading epubs
made the sigil.sh startup script more robust (courtesy of Craig Sanders) (issue #737)

The big features everyone has been waiting for are now in. The OPF and NCX are now preserved on import, and you can edit them by hand. You can still let the GUI do everything, but you now have the power to make changes directly.

The TOC editor is gone and has been replaced by the TOC sidebar. A button at the bottom of the TOC sidebar allows the generation of the NCX form the headings in your epub. You can ignore that button if you want and just build the NCX by hand. Typing in the NCX file updates the TOC sidebar live, using a background thread to keep things fast and responsive. Clicking on an item in the TOC sidebar takes you directly to the target location. Yes, you can now navigate the epub with the TOC.

A new dialog is now presented to the user when you screw up the markup in your XHTML (and NCX and OPF) files so badly that they become ill-formed XML. Previously Tidy was used to correct such errors, but now this dialog informs you of the problem and lets you fix it by hand if you wish, with the “Fix Manually” option. By selecting the “Fix Automatically” option in the dialog, everything goes through Tidy and it will then fix it for you, just like it did so previously. Basically the dialog is there to let power users skip Tidy’s error correction and fix the problem by hand. Naturally, the dialog tells you the line/column location of the error and a brief description.

With this dialog and the previous Tidy cleaning on/off button (which controlled the dreaded “clean” option), the negative aspects of Tidy are now something you can completely avoid.

This whole NCX/OPF preservation and editing by hand deal required a very thorough rewriting of a lot of Sigil’s internals. This has certainly brought many bugs so bear that in mind.

The whole point of the beta process is to discover and report bugs in unstable versions of the software. So report any bugs you find! And don’t be surprised when you find them, because you will certainly encounter some of them in this beta. If you’re uncomfortable with that, then don’t use the beta. Wait for 0.4.0 FINAL. This is pre-release software, that’s why it’s called a beta.

There were some features that I initially planned for the first beta but that eventually didn’t make the cut. I spent days on one of them to finally realize that it simply isn’t going to work (through no fault of mine, but it’s a long story for a new blog post). The other major feature that didn’t make it in was simply pushed back to a future beta. I’d like feedback and bug reports on the massive rewrite needed to implement OPF/NCX preservation/editing sooner rather than later, so I thought it wise to push the first beta out today.

Sigil 0.3.4b for Windows

2011-03-08T13:12:00.001+01:00

There was an interesting problem with the recent OpenCandy integration: Microsoft Security Essentials started flagging the Sigil Windows installers as containing adware. From discussions with the OC people, this appeared to be because the MS people didn’t like that OC was storing a certain key in the registry and then transmitting it to the OC servers. The key acted exactly like a browser cookie, and pretty much every website you’ve ever visited stores one on your computer. Other AV vendors and security researchers agreed with the OC folks that this is just fine and doesn’t constitute “adware” behavior, but the MS folks disagreed.

So after two weeks of back-and-forth, the OC folks have decided to remove this specific behavior from their SDK. For their part, MS has agreed that the new OC plugin isn’t adware even by their definition, so installers using it will not throw warnings.

I have just uploaded Sigil 0.3.4b for Windows to the downloads area, and it contains this new version of the plugin. MSE has no complaints about the new installers.

Sigil itself remains unchanged in this version, so if you have 0.3.4 installed, there is no reason to update.

The last nail in the coffin of SGF

2011-02-27T16:45:00.001+01:00

The next version of Sigil (0.4.0) is dropping support for opening documents in the old SGF format that was proprietary to Sigil versions 0.1.0 through 0.1.9. Sigil 0.2.0 dropped support for writing files in that format (that was a year ago), and now the second part of that transition is happening. That previous link describes what SGF is if you’re unfamiliar with it. Bottom line, if you’ve never heard of it, you never will and shouldn’t care about this change at all.

For those that still have some SGF files left, just open them with any version of Sigil <=0.3.4 and save them as epub. That’s all there is to it. All those versions of Sigil will be available for download pretty much forever, so there’s no rush.

An analysis of EPUB3 (and, uh, a bit more)

2011-02-20T16:42:00.001+01:00

[I swear when I’m frustrated. That makes this post obscene even by Chris Rock’s standards. Proceed with caution. Also, this was (and is) supposed to be about EPUB3, but as I kept writing it, it kept growing. Fuck it, I’ll just post it as it is.]

The IDPF published the current draft of the new EPUB3 spec a few days ago. Time to see if was worth the wait. Note that this will be a long post.

I’ve read all the sub-specs of EPUB3, and my general feeling about them is one of… “meh”. That’s the best way I can describe it. “meh” leaning towards “not good”. I jotted down notes in my notebook while I was reading them^[1], and what follows is a digested summary of my views and sentiments.

Assume I agree with everything in the specs I don’t explicitly disagree with here^[2]. Also, while I’ll take this opportunity to mostly rant about the bad parts of EPUB3, keep in mind that there are quite a few good parts as well.

All hail the mighty iPad

I’ll start this section by saying how I have absolutely nothing against JavaScript on the web. That would be stupid. I mean, without JS all of this web app business would not exist, and while arguments against “cloudification” of applications have weight, no matter where you stand on the debate, you can’t say that Gmail isn’t useful.

But we’re talking about books here. In EPUB2, JavaScript execution was under RFC2119 “SHOULD NOT”; for all intents and purposes, this means forbidden. It’s not a “MUST NOT”, but still.

In EPUB3, JS support is now optional. This means you can start using JS in your epub books, yippee! You can now go all Web 2.0 on you e-books. I’ll talk about why this is bad in a moment, but first I’ll like to give credit where credit is due and note that the spec text explicitly mentions that content creators should avoid using JS if at all possible. Here’s a quote:

Scripting consequently should be used only when essential to the User experience, as it greatly increases the likelihood that content will not be portable across all Reading Systems and creates barriers to accessibility and content reusability.

Sadly, no one will listen to this. But at least the IDPF has this warning, even though it won't do shit. Now that JS support has moved from "should not" to "optional", people will go out of their way to redefine "essential to the user experience" so that it includes JS. This will break horribly. We'll get epub books created solely for iBooks and all other Reading Systems can go to hell. Progressive enhancement? We will never see it. The people who create epub books are not web developers, they work in publishing. They have no idea what writing code for the web looks like (or writing code at all for that matter), so we'll see hacks upon hacks that work on iBooks pretty much by accident and on nothing else. I've always said that the day that the epub specs start mandating JS support is the day those same specs jump the shark. We're not quite there yet, but the gates of Hell are now slightly ajar.

This is what will happen:

EPUB3 brings "optional" JS support.
Publishers start adding crappy JS to their books hoping it will make them "stand out", "embrace the future", "fuck goats" or whatever.
We now have thousands of books with JS scripts in them that are absolutely useless but whose execution is nevertheless required, otherwise reading the book is impossible. You know, things like special navigation menus, buttons to expand example source code, footnotes in "tooltip" style windows or similar "brilliant design ideas" that stop working when you don't run the book's JS.
EPUB4 now demands JS support. I mean really, you can't expect publishers to go over all those crappy epubs and rework them with progressive enhancement in mind, do you? No, no, no. They'll just lobby the IDPF to make JS support mandatory, and they'll succeed.
Welcome to the web circa 2000! Ah, what a fun place that was.

But I don't blame the IDPF for moving JS support from "should not" to "optional". Actually, that's a lie. Of course I blame them. But I understand they had little choice when a lot of the people who make up the EPUB Working Group are the same people who have abused the term "HTML5" to the point where it doesn't mean anything anymore. Quite literally, nothing. It’s become a whizz-bang-pow marketing term hyped into oblivion by a fruit company. This “HTML5 is love, sex and the future of human civilization” nonsense has even pushed the WHATWG into renaming the spec to just “HTML” (even though they won’t admit the reason publicly). That’s right, the term “HTML5” now officially stands for nothing. Here’s a funny link about this “HTML5 means everything”. Bruce Lawson is specifically calling out the CSS3 == HTML5 == JavaScript idiocy, but you get the point.

Interactivity in books? My God, how ever did books survive the last five thousand years without JavaScript, <video>, <audio> and <canvas>? It boggles the mind.

Publishers: "HTML5 BOOKS? MOAR! LOOK MA IM IN TEH FUTUAR! We're totally not going to go extinct now!"

JavaScript, <video>, <audio>, <canvas> in books == "This book needs more cowbell."

I know I’m being a cynic, but I can’t help myself. The iPad came along, was declared “the savior of the publishing industry” and now everyone seems to be losing their mind.

Again, “HTML5”? Great for the web. Actually, fucking awesome for the web. For e-books? I don’t remember the last time I thought “this book really needs some video”. In fact, 99.99% of all epubs would be far better off with only the most basic HTML and maybe a few lines of CSS.

I know it’s not the IDPF’s fault this is all going to be so shamefully abused, but I still think it should have all stayed at the “SHOULD NOT” level. You want interactivity in e-books? That’s not an e-book, that’s an app. Go make one. You are not going to be able to write an interactive book and expect it to run on all the Reading Systems the same (or at all). That will not happen. Save yourself a lot of trouble and just make an app.

For those with the brilliant ideas of tooltip windows, custom navigation menus and the like which books would be far better without, just don’t do it. No, it does not look “sharp”. Or “hip”. Not even “trendy”. What it is, is stupid. You’re making a book, not a website. Please bear that in mind.

I’m sure that there are valid use cases for all of these technologies in books. A smart person using them appropriately can truly make something wondrous. Sadly, most people think they’re the smart ones. They’re usually not.

I remember how all this got started. Back in the old days, when EPUB was just an idea, this was the train of thought: “How are we going to represent electronic books? Raw, custom XML like in DocBook? Huh… maybe it would be better to use web tech like HTML. It is widely understood and there are ready-made components that will make it easier to build Reading Systems.”

So web tech was used because it lowered the barrier to entry. Instead of using DocBook’s <para>, why not use HTML’s <p>? We get free styling with CSS too.

But this changed. Now it’s not “we’re using web tech to make e-books”, it’s “we’re using e-books to package web tech”. It’s not about making books anymore, it’s about using web tech offline. You think I’m exaggerating? Do you know what term was used to “succinctly describe EPUB” during development of EPUB3? Here it comes: “website in a box”. I’m not kidding. It was used in the IDPF meetings and was even in the November 12, 2010 draft of the EPUB Overview document. Here’s a direct quote:

An EPUB publication can be thought of as a "website in a box".

No. No it shouldn’t be. Never ever.

No required glyph coverage

Honestly, the worst problem with EPUB2 was that there was no required Unicode glyph coverage. Let me explain what that means. On the surface, EPUB is all Unicode. Everything has to be either in UTF-8 or in UTF-16. “That’s great! This means I can use any letter I want!” Not quite. While you can specify any letter you want, Reading Systems aren’t required to display that letter. It would certainly be unreasonable to mandate that all RS’s have glyphs for the entire Unicode range, but there is no minimum coverage specified either, and that would be a good thing. I was hoping this would be fixed in EPUB3, but no such luck.

With the way things are, RS’s can just support ASCII and be done with it. Some support more than that, some support only that. Yes, you can get around this problem by embedding fonts with the required glyphs in your epubs, but most people don’t know they have to. See this FAQ entry for the most popular question about Sigil. I couldn’t even begin to describe the number of people who say infinitely moronic things like “Sigil doesn’t support Unicode” because the book they saved displays as a bunch of question marks in ADE and in all the hardware readers that use Adobe’s RMSDK.

It’s not just Sigil’s problem, it’s everyone’s. People have made epubs that were tested only on the iPad, and since iBooks has fonts with wider glyph coverage than ADE, some characters in those books end up as question marks over there.

There should be some minimum coverage specified. One might ask "but where do we draw the line at mandated coverage? Should CJK support be mandatory? Where is the line?" You're right, those are tough questions. That's why we have a Working Group, to answer them. Too hard to draw a line somewhere? Ok, how about adding one of those shiny RFC2119 "SHOULD" statements asking for greater coverage? It wouldn't do shit, but hey, it just might.

The problem is that nobody at the IDPF seems to give a crap about this problem. That's what we get when the vast majority of Working Group members live in ASCII land, I know, but these guys are making an international standard. Show some breadth of understanding. This lack of mandated coverage is a far bigger problem of EPUB2 than "well damn I can't put video in my books". Trust me.

Living in fairyland

There are plenty of things in these new specs that are wonderful or interesting on the surface, but will never see the light of day. Things like "container-constrained" for JavaScript (great idea!) or the “epub:trigger” element (silly idea, people will just use JS). But they will never be supported on the various Reading Systems, and if they will be, then no one will use them. People who make RS's are by-and-large hacks (exceptions do exist though) who slap some custom controls onto WebKit and call it a day. They won't modify WebKit to support epub-specific elements. That's "too hard". Am I the only one who remembers EPUB2's custom "switch" and "case" elements? Or inline XML islands? Or the whole of DTBook? The only RS that supported those was ADE (hats off to Adobe for trying, I really mean that). Everyone else just pretended it didn't exist. And not even Adobe implemented support for “oeb-page-foot” and “oeb-page-head”, and those were damn useful (on paper at least).

History has shown that wherever the EPUB specs went beyond what popular browser engines implemented, the specs were actively ignored. It's just "too hard". It's not, of course. It "merely" requires two things: competent developers and people who give a shit. Both of these are very, very rare. Combined? Good luck. Oh yes, it requires one more thing: actual fucking work. Not just taking an existing browser engine, making it display XHTML in pages and calling that an RS. No, actual software development is required, and the most difficult kind of all: working with a huge, foreign code base. That's too much for most.

As an example, I have to work with HTML Tidy since it's an internal component of Sigil. I can't tell you how happy it makes me to know I'll have to implement an HTML5 parser for it because of EPUB3. I'm truly ecstatic about this prospect. I fucking love the very idea of it. But I'll do it because it has to be done. And for the love of God bear in mind the difference in the quality of code between something like WebKit and Tidy.

Tidy could easily be the world's most horrible code base. It's 40k lines of straight-up C, written in the most god-awful way. 800 line functions; cryptic, single-letter variable names; hacks upon hacks that step on other hacks; source comments that are either out of date, worthless^[3] or usually just plain wrong. Just... absolute, worthless junk abandoned by the original devs (and those that followed them) many years ago. Nobody works on Tidy anymore, at least not with the official project.

And yet I work with it because I know I have to. WebKit source is worked on and maintained by hundreds of people and it's extremely well written. RS developers, get off your damn asses!

To tell you the truth, I've been thinking about implementing an open-source RS for both the desktop and memory- and power-constrained devices "just to show 'em how it's done". I have some sweet, sweet ideas for it. But I can bloody barely find the time to work on Sigil and FlightCrew. A third project? I can't put the gun in my mouth fast enough.

And don’t even get me started on the “quality” of the Reading Systems out there. I remember the day when we used to complain about ADE. Today, ADE is pretty much the best RS available. Do note that I said “the best available”, not “great”. Today, I’d give it a C. Everyone else gets an F on a good day, and a kick in the balls otherwise.

The worst is certainly iBooks, as any epub creator will tell you. Ask one about their opinion of iBooks, and you can rest assured that the response will be filled to the brim with “fuck”s, “shit”s, “cunt”s, “motherfucker”s and “asshole”s. Apple loves to boast about support for open standards and how they’re important. As long as we’re talking about killing Flash. The EPUB specs can go fuck themselves. It’s not that they’re lazy, incompetent or don’t feel like investing the resources to improve their support for EPUB. It’s not about “missing” functionality. They intentionally went out and broke things in Mobile WebKit to further their agenda. If they ever tried something similar in Safari, there would be a pitchfork-mob in Cupertino.

But the number of people who make EPUB books (or work in publishing in general) compared to the number of people who develop for the web is… somewhat small. Not to mention that we as an industry are too busy sucking Apple’s dick to notice what’s going on. They can safely brick in our mouth. Oh no, Apple demands 30% off all subscriptions, in-app purchases and a lowest price? That really is the last straw. We’re now going to start sucking that dick very, very slowly in protest. That will teach ‘em!

You’d think I have something against what Apple is doing. Not at all. If someone lets you exploit them, by all means, go right ahead. Apple is screwing us only so much as we as an industry let them screw us. And now that people are starting to come around, we’re all like “OMG we have fifteen inches of Apple’s cock up our ass! What the hell happened?!”. We let it happen, that’s what. Inch by inch they kept shoving it, and we let it slide (yes, pun intended). Now we’d like them to back up a bit. Well guess what, when you have fifteen inches of cock up your ass, it’s hard to negotiate. The cock is the one setting the terms.

Weren’t we talking about EPUB3?

You’re right, I forgot. Here are some bullet points since I’m tired:

Greater emphasis on accessibility in the specs. Good that someone realized that the e-book movement is a godsend to people with poor vision. There was some support for this before, true, but now it's more front-and-center.
"xml-stylesheet" support is required? Interesting and unexpected. I doubt any RS's will actually support it though.
"This schema is normative. In case of conflicts between the specification prose and this schema, the schema shall be considered definitive." Hell yeah! At least now we know which is considered definitive. Trust me, you'd encounter this problem if you ever tried to implement a validator. But NVDL, RELAX NG and Schematron? May as well say "you have to use Jing". Some of us don't want to. How about providing an XML Schema schema? It's standard practice. Great, now I'll have to write my own... again...
Supplementary resources with <link> in <metadata>? Fancy.
DCMES metadata elements are being replaced by DCTERMS properties. This really is a good idea, the new system should be much more flexible. The transition period will be ugly, yes, but it's necessary. Good call on both the replacement and the transition.
"Although the EPUB Navigation Document is required in EPUB Publications, it is optional to include it in the spine." Yes! This will eliminate the need for those ugly "inline TOCs" people like to build where they would basically end up with two different files describing the TOC. Now the NCX is basically an XHTML document that can be styled, and if you really want it in the reading order, go right ahead and include it in the <spine>. Very nice.
“page-spread-left" and "page-spread-right" on <spine> <itemref>s. Nice, but how many books use two-page spreads?
Embedded MathML support is great. Nobody will care about the restrictions the IDPF has placed on its use. When RS's support MathML, it will be because the browser engine they use internally supports it, and that engine couldn't care less about the IDPF's MathML "restricted subset".
page-list nav gives support for cross-referencing an epub with the page numbers of a printed edition. This is important and as such will be used by publishers and (should be) supported by RS's… eventually.
landmarks nav replaces the OPF <guide>. This is also very good.
Media Overlays: feel free to ignore the existence of the entirety of the Media Overlays sub-spec. I really mean that, you don't even have to read it. Just pretend it's not there since nobody will ever implement it. To add insult to injury, support for it is officially optional, so nobody even needs to implement it. It's dead on arrival, much like DTBook was as a valid EPUB2 OPS syntax.
Don't get me wrong, it would be great if RS's supported this. But they won't. Nobody ever made crazy money by catering to the visually impaired.

Canonical Fragment Identifiers

This deserves a separate section.

Canonical Fragment Identifiers are ridiculous, at least the scheme presented. They're complex to the point of absurdity. Even-numbered indices so as "not to be sensitive to XML parser handling of whitespace, entity references, and CDATA sections."? This is ridiculously over-engineered. It has support for not only pointing at elements and their textual content (worthy goal), but at pixels in raster images, logical units in vector images, temporal locations in audio an video and if I understand the exclamation mark rules, even support for crossing documents.

Look, you can't cram all that into a single scheme. You just can't. The WG should have just stopped at trying to point at textual content. The CFI scheme as written is silly and reminds me of the crap in SVG 1.2. Most people don't know that SVG requires support for things like raw sockets and file uploading. There's this desire in specification working groups to support every single use case imaginable and then some. Common sense goes out the door, and nobody is either willing or able to just say "NO!" to some of the requests. This is exactly the kind of thing people think of when they say “design by committee”.

This CFI scheme is absurd. The way it's designed, no one will support it. I know I certainly won't.

Suma summarum

That’s it. I’m all out. EPUB3 is nice, but most of it will be either a) misused or b) ignored. Neither is really the IDPF’s fault, but some of it is. The parts that people will support are nice and shiny, like page-list nav instead of the NCX or DCTERMS instead of DCMES.

What do I know. Ignore me.

Footnotes

[1] I ended up writing more than five pages, but most of the notes relate to low-level understanding of the changes from EPUB2, possible contradictions and implementation problems and the like. You know, the things I’ll need to pay attention to whilst working on Sigil and FlightCrew, but usually not things content creators need to care about.

[2] That’s probably not accurate, but let’s pretend it is. There are some things that are just not worth complaining about.

[3] "Thanks to X.Y. for reporting a problem with this function!" What kind of developer would actually write that above a function?

Official Twitter account for Sigil

2011-02-17T15:21:00.001+01:00

For those who missed the MobileRead thread, here’s a copy of what I said there:

I've decide to create an official Twitter account for Sigil. Why? Because on many occasions I've felt the desire to provide more up-to-date info on Sigil's development, but didn't think it was worth writing a whole blog post about.

Now it will be possible to occasionally say something like "I'm working on X, and it looks like this: [PICTURE]", or "Life got complicated, dev work stopping for a couple of weeks", or "A new release should be coming in the next few days" etc. That last example is a good one; such pre-announcements would enable people (and especially larger organizations) that are using Sigil to prepare for a new release.

The updates should be fairly low-volume (also, erratic; I'll post when I feel I have something worthwhile to share with others).

FlightCrew-related info will also end up there.

The dev blog will still be around as it's not being replaced.

There’s already a screenshot of a few new 0.4.0 features in action. Check it out.

And final exams are over… for good

2011-02-01T16:56:00.001+01:00

About a week ago I had my final final exam (that sounds weird). This is it now. For the entire spring semester I have only my Master’s thesis to write, and it’s all finally over. While my friends will spend the next several months writing code for their thesis project, I’m basically free of this chore. Why? Well I’ve been working on Sigil for the last two years, and while it will never technically be finished, it is as far as the university is concerned. My internal thesis goals for it have been reached many moons ago (back in the 0.2.0 rewrite). That’s not to say I plan to stop working on it, but that I have no deadline pressures. Yippee.

I’ll be working on it (hopefully) pretty much the same way I’ve been working on it until now, and that’s “when I find the time and the inclination”. Seeing how I don’t have any immediate obligations (I can write the ~50 pages for my thesis in a week or so, and the deadline is in June), the “time” part has gone up immensely and the “inclination” part is still holding strong. So hopefully I’ll be getting some serious work done.

Actually there’s a mix of things I want to do in this fairly large block of free time. Here’s a list in approximate order in which I plan on doing them:

REST! I spent the last week basically just sleeping and breathing. The obligations, pressures and sleep-deprivation of the last several months have taken their toll. This R&R goal is fairly short-term. On a related note, it’s been a while since I spent a whole 24 hour period with my girlfriend. It’s strange how I almost forgot that those make life worth living.
Read some fucking novels. I make a goddamn e-book editor, it’s about time I get the chance to read a few of the e-books I’ve made for myself. I used to devour novel after novel, and while I still read voraciously, the books I read don’t exactly qualify as literature. If I see just one more textbook on machine learning, cryptography, project management, numerical analysis or discreet math in general, I’ll claw my eyes out.
Read some technical books. I enjoy learning new things and deepening my understanding of the things I think I already know; besides, in this industry one needs to be constantly sharpening their skills. For instance, I’m currently going through Jon Skeet’s C# in Depth, Second Edition^[1]. After that, it’s Chris Smith’s Programming F#. I know a bit of F# (and I really mean a bit), but it’s time to get serious. It’s a truly wonderful language, and I’ve been enamored with the functional paradigm since my brush with Haskell two years ago, but F# strikes me as more practical than Haskell. Also, a book on advanced Python wouldn’t kill me, but I’ve yet to find an appropriate one^[2].
Start looking for a job, I guess. I hate this last part…

Anyway, this change of pace will do me good. A lot of it, I expect.

Footnotes

[1] I’ve literally been waiting for the second edition to come out for years. It came out in November, and I’ve been waiting for some free time to read it ever since. Now, I’ve written tens of thousands of lines of C# code and I’d like to think I’m more than just proficient with the language, but Skeet can teach anyone something new. That includes me.

[2] I really mean “advanced” Python. I know Python fairly well; I’ve read Dive Into Python 2 many years ago and I’ve used the language on many projects since then, but I’m looking for a book from a true expert in the field that goes into the nitty-gritty. Something like Skeet’s C# in Depth, but for Python. The official Python docs only go so far. They’re also a bit… let’s say conceptually scattered.

Sigil 0.3.4

2011-01-12T14:34:00.001+01:00

There was a regression in Sigil 0.3.3. This release is meant to address that. Changelog follow:

fixed a regression ("Not a folder" error) with opening certain epub files on Mac and Linux systems (issue #731)

This was caused by a bug in the ZipArchive library which I updated from 4.0.1 to 4.1.0 for Sigil 0.3.3. The bug has now been fixed.

The bug only affected Mac and Linux users, so Windows users don’t really have a reason to update.

Sigil 0.3.3

2011-01-08T12:10:00.001+01:00

The new release is in the downloads area. Changelog follows:

added a small "Donate" button to the toolbar and a related entry in the Help menu
added a .desktop file for the make install target (courtesy of Richard Gibert)
this time *really* worked around a Tidy bug that added blank lines to the start of <pre> and <style> elements (issue #655)
updated ZipArchive from 4.0.1 to 4.1.0
fixed a regression crash bug with loading extremely rare HTML content documents that have an internal DTD subset
added a workaround for a crash bug caused by invalid epubs that use obfuscated fonts but with incorrect UUID URN key syntax (issue #709)
integrated the FlightCrew epub validation library; a new toolbar icon triggers epub validation and displays the results (issue #28)
fixed a rare input truncation problem when the input file contains a unicode nbsp and also specifies standalone="yes" in the XML declaration (issue #677)

So there it is, FlightCrew is finally a part of Sigil. There’s a green checkmark in the toolbar now; use it to trigger validation. You can also double-click in the list of validation results and Sigil will take you to the location of the error, if available and possible. Why “if possible”? Well Sigil doesn’t display the OPF and NCX files to the user (yet), so if you get a validation error in one of those, it can’t take you there. There are other limitations to the “take me to the error” feature, but it should work fairly well for XHTML files.

Also, this release brings OpenCandy to the Windows installers. Read about the reasoning for that in this blog post. A new installer builder is used for Windows installers (Inno instead of InstallJammer) which should fix some of the outstanding bugs with the IJ ones. Please uninstall the old version of Sigil first! Nothing really bad should happen if you don’t, but just to be on the safe side, please do.

Several people have also suggested and asked for a donate button in the UI, so one has been added. It’s a small red heart. You’ll be able to turn it off as soon as I make an options screen. :D

I’ve also made some changes to the Mac build system, so if I screwed something up, please yell as loudly as possible… or just create an issue on the tracker about it.

A more serious conversation

2011-01-02T13:53:00.001+01:00

[This (very long) post contains a certain amount of frustrated swearing. You have been warned.]

Open source applications usually face one of three fates:

If they’re not popular, they will be abandoned and forgotten.
If they’re merely useful on occasion, they stagnate.
If they’re popular and used often, they will get bloated beyond both recognition and usefulness.

I’m going to talk about the third one.

A developer has an idea for an application. There’s a certain task he needs help with, and nothing out there really meets his needs. So he decides to sit down and design an application that will perform the required functions and help him achieve his goals.

He decides that he will open source the code, for whatever reason. Maybe he’s an idealist. Maybe he’s a moron. It doesn’t matter, he just does it.

The application proves popular, and people start coming to him with requests for new features and bug reports for existing ones. Proud of what he has created and more than happy to help others, he starts spending more and more time working on his app.

He has a vision of what the application is supposed to look like; he sees the big picture and where all the features—both current and future ones—fit in. Some people disagree with this picture, but that’s their problem; he likes it. He thinks it’s the bee’s knees, the cat’s whiskers and a bottle of Johnny Blue all making sweet, sweet love together. Basically, he thinks his view of what it’s supposed to look like is fuckin’ A.

He also realizes that any piece of software can’t just keep piling up obscure features; there’s a certain threshold of diminishing returns where the new “features” start getting in the way of using the old ones even for the old-timers, and the new users have no fucking clue what they’re looking at. It all just becomes an ungodly mess of buttons, windows and menus that no one sane can use to get any sensible work done even after reading through a 500 page manual.

For a software application to stay away from bloat, the features added need to be useful for most of the users; not for just a vocal minority, and certainly not for just a few. Every new feature you add makes it slightly harder to use the old ones; it adds to the noise and eventually makes it difficult for the user to find what he wants and impossible to discover it through exploration.

All of this entails saying “no” to the requests that could otherwise be considered sensible, but that just don’t “jive” with the idea of what he’s trying to build. For instance, a request to “play FLAC encoded audio files” would make sense if he were building a music player, but makes somewhat less sense for an accounting application. Again, a request for a “counter that would track the number of noobs owned”; great for Unreal Tournament, less great for an FTP client.

But these requests aren’t the problem. The requests that are the problem are the ones that are useful to the person requesting them and maybe five other people.

I’ll elaborate shortly. Let’s imagine that our imaginary developer’s application has something to do with editing text. Now let’s imagine that this application is called… Ligis. Yeah, that’s what we’re going to call it. Sounds kinda catchy.

So someone comes along and says: “I work with romance novels every day. My publisher demands that all the text backgrounds be bright pink. Now, I could select the text in every chapter and apply the color to each of these, but my life would be much easier if the main toolbar had a ‘Make all backgrounds pink’ button. Could you add one?”.

Now, what our developer wants to say is: “What are you, fucking retarded? Of course I’m not going to add that. Stop wasting my time. GTFO”. But being impolite to strangers is not something he’s willing to do (especially not over the Internet; there's too much unkindness here as it is), so he merely responds with: “No, that would not be a useful feature”. He understands that the person who’s asking for this would really find it useful, it’s just that no one else would.

The biggest problem are the people who come in with obscure feature requests like these but that are also willing to implement them themselves. This being open source and all, they’re willing to write code to get them in the mainline releases. It’s hard to say no to these folks, but a crappy feature is a crappy feature no matter who writes it. So he still says no, but hey if you’re willing to write code, the issue tracker is full of accepted feature requests you could implement. But the other dev doesn’t feel like it; he just wants his one issue in and doesn’t have the time for something else. And our original developer can respect that, honestly. Working on something for free purely for the benefit of others instead of spending the time to, say, watch a movie or read a book requires an unfortunately rare amount of altruism.

And probably stupidity and/or naiveté.

Ligis becomes even more popular. The more popular it gets, the more feature requests and bug reports that are actually valuable start coming in.

For our intrepid developer, this reaches a point where there just isn’t enough time in the day to both work on Ligis and to fulfill his other responsibilities in life. Or he just becomes too stressed out to work on it. Or he starts hating it all. Or he just flakes out. Or whatever. Again, it doesn’t matter; for some reason, he decides to stop working on it and hands off the reins to a new maintainer.

The new maintainer does what most maintainers do: he fixes the major bugs as they’re reported and maybe adds a minor feature or two. He pushes a new release once every six months, tops. He doesn’t add major features since he doesn’t have the time it would take to implement them. Hell, that’s the reason why the original developer left.

But now random people start showing up willing to work on Ligis. The maintainer gladly accepts their help. The new devs start adding the features they want, not necessarily the features that most of the users would find useful. I mean c’mon, this is open source, if the lusers want something done they should do it themselves. The new devs certainly don’t feel any kind of obligation towards them. And why should they? The new devs make drive-by changes they need and usually disappear afterwards. Even newer devs that also need some obscure feature replace them.

This goes on and on, until Ligis becomes freakin’ huge. Now it doesn’t merely have the kitchen sink; it has the sink, the toilet, the washing machine, the hair dryer, the bathtub and that little bidet shit the French use to wash their ass in. In twenty colors each. Yes, gold and papaya too.

The new devs were just throwing shit up against the wall until there was no more wall left, just a mountain of shit. Nobody had any vision of what the whole app was supposed to look like, no sense of scope. Scope? Fuck you and the scope you rode in on. I need my backgrounds pink and I need them pink yesterday.

A user of an old version of Ligis tries out one of the newer ones and feels the need to talk to the devs.

User: “Hi guys! I like the bidets in the new version. They really bring out the splash screen. I also enjoy the new mango-cyan color scheme. I have just one question though: I’ve noticed that there’s a mountain of shit where this wall used to be. There was a door in this wall, and I used to go through it every day on my way to work. Really useful door, that. Do you know what happened to it?”

Devs: “Oh the door is still there. We just didn’t have anywhere to put all this shit so we just piled it on top of the door. We have a couple of shovels in the back if you still want to get to it. This is our usual advice to the ever-popular how-do-I-get-to-the-door question. Have you read the Manual?”

User: “Uh… can’t you just remove the shit so that people wouldn’t have to shovel their way through?”

Devs: “Like we said, the door is still there. Take this shovel and the Manual, volumes three through sixteen; they will explain how to dig out a path. Watch for the landmines, the shark and the koala. The shark is friendly but the koala bites.”

User: “But why do I have to dig this long and winding path? Why can’t I for instance remove this pile of shit right here for good?”

Devs: “That pile makes all the backgrounds pink. You can’t remove that, it’s our killer feature.”

User: “And this pile right here?”

Devs: “That’s a Lisp interpreter. Don’t touch that.”

User: “*sigh*, forget the door. Could you just fix this one obvious bug? Every time I hit the spacebar the whole computer reboots. It makes it difficult to type.”

Devs: “Yeah, we’ve been meaning to get to that one. Steve has been working on it since ‘95.”

Steve: “Still haven’t been able to crack it though. I can follow the call from the GUI layer to the new COBOL backend and through the kernel modules, but I get lost in the Space Shuttle diagrams.”

User: “Screw you guys, I’m going back to the old version.”

This is what happens when the original developers go away. True, sometimes it happens while the original developers are still very much active, but that’s a different issue entirely.

The point, please?

Why am I talking about all this?

I’m talking about it because this scenario¹ literally keeps me up at night. I’m able to justify the time I put into Sigil while I’m still in grad school, but that’s slowly coming to an end. I graduate soon. College takes a lot of your energy, but it’s in ups and downs. There are weeks where you spend twelve hours a day doing school-related work, and then there are “quiet” weeks. It’s during those that I worked on Sigil. Even then, I could only justify the time investment by making Sigil both my Bachelor’s and my Master’s thesis.

Since that’s all ending, it would be great to find some funding for future development. To that end, I’d like to add an OpenCandy powered advertisement to the Sigil installers for Windows². It’s one of those “would you like to also install software X?” recommendations you sometimes see in installers. It would be turned OFF by default, meaning that if you just click Next, Next, Next, Finish, you will only ever install Sigil and nothing else. You can read an interesting CNET article about OpenCandy here.

Some key points:

As mentioned, any such “do you want to also install X” question would be set to NO by default (or deselected entirely).
The installers wouldn’t grow in size since the recommended software wouldn’t be included in the installer, only a piece of logic that would query the OpenCandy servers for a recommendation and download a different installer if the user chose to install that extra software.
No ads are displayed after the installer finishes. No ads would be in Sigil.
No malware, no adware and no spyware would be allowed to advertise. Developers get full control over the applications that can advertise in their installers, and can block anything they disagree with. An example of the type of companies/apps that currently advertise in the OpenCandy network are Kaspersky, Bing, Winamp, StumbleUpon etc. Respectable companies.
This OpenCandy service seems popular in the OSS world as a funding source. WinSCP uses it, Miro uses it, Audacity uses it.
OpenCandy would also advertise other OSS software inside the installer. That seems nice. Sigil is becoming more and more prominent, and I’d like to use that to promote other OSS apps as well.
Sigil is still free and open source. No proprietary code will enter the repository or Sigil.
Sigil functionality is not changed in any way, regardless of whether you install the extra software or not.
Mac and Linux users are not affected in any way.

Basically it’s just a single ad in the Windows installer. Nothing else.

But since the Sigil user community is something absolutely awesome, I’m asking for opinions. If enough of you come forward with valid concerns and arguments against this deal, I’m saying no. To tell you the truth, I don’t expect much from this deal in terms of revenue; peanuts per month, if that. But it would give me something to point at and say: “See? I can justify the time investment”. And hey, maybe it amounts to something one day.

You can post your opinions here in the comments or in this MobileRead thread. I’m not looking for a vote, I’m looking for a discussion. Bring your eloquence.

Footnotes

[1] OK, something less insane but you get the point.

[2] They actually approached me about it. Good timing.

Sigil 0.3.2

2010-11-24T13:25:00.001+01:00

You can get the release in the downloads area. Changelog follows:

added a new toolbar button for turning Tidy cleaning on/off; this option is also available from the Tools menu (issue #553)
added support for TrueType Collection fonts with extension TTC
InDesign (as of CS5) refuses to list the fonts it embeds in the OPF manifest of the epub files it exports, even though the epub standard demands it. This causes Sigil to not pick up these fonts when opening such epub files. A workaround has been added that will detect such problematic epubs and then load the font files.
worked around a Tidy bug that added blank lines to the start of <pre> and <style> elements (issue #655)
fixed a rare crash issue when loading epub files

All of these fixes have been in the repo for a while now; I wanted to get a few more things in before pushing a new release, but I’m so pressed for time right now that I know I won’t be able to do it any time soon.

The most notable changes are a new toolbar button for turning off Tidy cleaning. A bare-bones version of Tidy will still run to make sure your source is valid XHTML, but there is no element rewriting or CSS extraction etc. When you turn it off, it’s off for loading, view switches… everywhere.

The second notable change is a workaround for InDesign’s crappy epubs with embedded fonts. Since ID doesn’t list them in the manifest, previous versions of Sigil wouldn’t pick them up during import (that’s what the standards say should happen). 0.3.2 will notice when such ID epubs are loaded and pick up the fonts.

Same thing goes for TrueType Collection fonts; ID puts them in the epub even though the standard only allows TTF’s and OTF’s… Sigil will pick up TTC’s too now if your epub has them. You shouldn’t use them, but Sigil shouldn’t silently drop them if you do. The choice should be up to the user.

With these and previous font de-obfuscation changes in place, support for InDesign epubs with embedded fonts should be fairly complete.

FlightCrew 0.7.1

2010-11-11T21:42:00.001+01:00

The release is here. Changelog follows:

added an automatic update checker to the GUI app
the GUI now displays a "No problems found" message when the epub passes all checks (issue #9)
fixed an issue with missing XHTML files causing the GUI to show a dialog about an std::exception and the CLI to report that the epub itself was not present (issue #8)
fixed an issue that was causing empty error messages for incorrect uses of XML encodings (issue #5)
fixed an issue with anchor links to the current file (links with fragments only) incorrectly throwing errors in the reachability analysis (issue #3)

Besides some nice bug fixes, you also get two nice features: a message is now displayed when no problems have been found (lots of people asked for that one… and I agree, it should have been there from day one) and an update checker for the GUI app. The update checker is the same module used in Sigil, so it works the same way. It will notify you when you start the app that a new version exists if it detects one on the server. The check is only performed if the last one was more than six hours ago.

This of course means that FlightCrew-gui will now access the Internet when you start it.

Also note that the installers now install everything in a “FlightCrew” folder by default, and not in “FlightCrew-gui”. This means that FC 0.7.1 will not overwrite an installation of FC 0.7.0. You should uninstall the old one before installing the new one, otherwise you’ll end up with two different versions of FC on your computer. It won’t prevent either from working correctly, it’s just annoying.

Mac users will see that there is now a “FlightCrew-cli” application along with “FlightCrew-gui.app” in the DMG. It should have been there in the first release, but I forgot to add it. Such is life.

0.3.1

2010-11-07T23:14:00.001+01:00

You know the drill. You can get the new release here, and the changelog follows:

added a new "Font Obfuscation" context menu for font files in the Book Browser; the user can now select (or de-select) the use of Adobe's or the IDPF's font obfuscation methods; this also resolves the problem where Sigil refused to open epub files that use such obfuscated fonts with the message that the epub has DRM
fixed a validation issue caused by Sigil using "application/x-font-opentype" as the OPF mimetype instead of "application/vnd.ms-opentype"
fixed a crash when opening the TOC editor for some epub files (issue #654)

There are two reasons why you are seeing a new release so soon: the first one is an unfortunate bug that causes a crash when opening the TOC editor for certain epub files. While rare, it can still happen and frankly, it’s not rare enough. This needed to be fixed pronto.

The second reason is the recent introduction of the “this epub has DRM and cannot be opened” message for epubs that have an “encryption.xml” file in the META-INF folder. The message is valid since Sigil can’t open encrypted epub files (nothing can without the key, that’s the sad point of DRM), but unfortunately Adobe InDesign obfuscates embedded font files. This, naturally, creates an entry in the “encryption.xml” and now Sigil refuses to open your epub. And the only thing you did was tell InDesign to embed a font.

As far as I know, it’s not possible to tell InDesign not to obfuscate embedded fonts. It does this by default. The sad thing is that most people don’t know this is happening; they just embed the fonts and since they appear to work fine when the epub is opened with ADE, they call it a day. Personally, I have a problem with this: if Adobe wants to placate the font foundries with some patently absurd font mangling scheme, fine. But give the user a choice of turning this crap off. Or at the very least tell them it’s happening. If I were Adobe, I’d spend less time thinking up such nonsense and more time improving their epub export option since it’s currently… somewhat unusable^[1].

Don’t get me wrong, I know perfectly well why they want fonts obfuscated: licensing issues. Most font licenses don’t allow embedding a font in a way that the font itself can be easily extracted from the distributed file. Font obfuscation is thought to solve this.

It doesn’t.

De-obfuscating a font mangled in such a way is laughably trivial; a twelve-year-old could write a program in 10 minutes that de-obfuscates the font files. The same thing goes for the IDPF’s method. It doesn’t prevent anyone from doing anything. While I find the whole thing ridiculous, I understand some want to give themselves at least an illusion of legal justification. All I’m saying is that Adobe shouldn’t be forcing people who know better into using something so silly.

Little do the users of InDesign know that this scheme makes their epub files invalid and that such fonts will only work in Reading Systems that support this. It works in ADE since Adobe naturally made sure that their RS supports their font obfuscation method. While I personally think that font obfuscation is utterly pointless, lots of Sigil’s users have epubs coming from InDesign… So Sigil has to cave.

You will notice that there is now a new right-click context menu entry for font files in the Book Browser. Under the “Font Obfuscation” sub-menu, you will see two actions: “Use Adobe’s Method” and “Use IDPF’s Method”. These select which font obfuscation method to use on the font files.

By default, the state in which the font file was found when the epub was opened is preserved when saving; so if you open an epub file with InDesign-embedded fonts, you will see a check-mark next to “Use Adobe’s Method” in the “Font Obfuscation” sub-menu. This means that the obfuscation is in place and will be preserved after a save. You can also click on “Use Adobe’s Method” to remove the check-mark, thus un-obfuscating the font file (I highly recommend this). Same thing goes for “Use IDPF’s Method”; clicking on a checked method unchecks it and removes the obfuscation.

Since Sigil now supports font obfuscation (and de-obfuscation), you will not see that “cannot open because of DRM” message for epubs that use such fonts.

Don’t forget that InDesign still^[2] flat-out refuses to to list the embedded fonts in the OPF file, even though the specs say it has to. That means the fonts won’t be picked up when the epub is opened in Sigil. I’ll probably add yet another InDesign workaround down the line to handle this.

Footnotes

[1] That’s the most restraint I’ve shown in a year.

[2] As of CS5. I just checked.

0.3.0 FINAL

2010-11-04T15:33:00.002+01:00

So here it is, the final version of 0.3.0. The changelog from 0.3.0 FINAL follows. Don’t forget that if you’re coming from 0.2.4, changes from RC1 and RC2 also apply.

root rights no longer needed to install on Linux
fixed an issue with some child headings being attached to incorrect parent headings in the TOC if the parent was marked as "not in TOC" (issue #476)
fixed an issue with some UTF-8 characters outside the BMP (usually CJK characters) not being saved properly (issue #180)
fixed an issue with certain path types not being correctly updated as a result of the fix for issue #501 (issue #561)
the Book Browser now prevents adding files that already exist in the epub
previously, when adding external XHTML files through the Book Browser, any files (like CSS stylesheets or images) that were linked from that file were included in the epub under a different name if their original name was "taken"; this caused duplicates so this behavior has changed: files whose names are "taken" are now skipped over (issue #482)
fixed a rare issue that caused incorrect path updates for anchor links pointing to file names that were suffixes of other chapter file names, and the anchor had a fragment ID (issue #598)
fixed an issue with the image paths in background-image CSS rules not being updated when the path changes (issue #594)
Sigil now informs the user that DRMed files cannot be opened, instead of just crashing (issue #624)
this time *really* fixed the "acknowledgments" error that was reported as fixed in RC1
fixed a crash on load (with an error dialog) issue on Linux systems occurring when multiple users on the same machine tried to use Sigil (issue #642)
fixed a randomly occurring crash, usually triggered on Macs during loading (issue #643)

And now I’m off to give FlightCrew some much-needed love.

Sigil 0.3.0RC2

2010-10-08T20:27:00.001+02:00

Well that was fast.

A file-corruption-on-save issue was detected in RC1 after it was published. It was taken down until the problem was resolved, and now it has been.

RC2 is now available for download.

Sigil 0.3.0RC1

2010-10-08T17:19:00.001+02:00

I’ve just released Sigil 0.3.0RC1. Please note that this release include some major changes under the hood, and as such may not be very stable. From the test I’ve made, it works great. Any problems I’ve found I’ve quickly fixed. But since this release brings a new version of Qt and completely replaces the internal XML DOM provider from QDom to Xerces, there are bound to be some regressions I’ve missed. Bear that in mind.

Here’s the changelog:

fixed a validation issue caused by using the American spelling for "acknowledgments" where the OPF spec uses the British "acknowledgements" (issue #611)
Sigil now uses "application/x-font-ttf" as the media type in the OPF for TrueType fonts (issue #609)
on Mac OS X, the universal build now includes an x64 version of Sigil, and builds now use Cocoa instead of Carbon; support for Mac OS X 10.4 is dropped along with support for PowerPC Macs
fixed a problem with opening files from the Ubuntu "Open With" menu (issue #524)
made Tidy handle common user errors like "&co." in the HTML source instead of "&co."
fixed a rare Tidy bug with disappearing spaces when the only whitespace in an element was the newline following a start tag (issue #387)
changed the internal DOM engine from Qt's QDom to Xerces; this should also bring numerous bug fixes and performance improvements plus a small (~10%) decrease in memory consumption (issue #367)
updated Qt to 4.7: this should bring a 400%+ performance increase in Book View rendering along with countless smaller performance improvements and bug fixes across the board
switched to the MSVC 10 compiler for Windows releases; should bring ~5% general performance improvement
fixed several crash/error problems relating to opening, saving and modifying epub files which have onerous file permissions set for internal content files (issue #574)
added a workaround for broken epubs created by other epub-producing software which caused a crash on certain searches with the Find dialog (issue #548)
fixed a problem with Book Browser's "Merge with previous" action if a file was previously deleted from the Book Browser (issue #565)
fixed a problem with chapter splits being placed in the wrong reading order if a file was previously deleted from the Book Browser (issue #497)

Performance improvements all around, plus some major fixes. Mac and Linux users would sometimes get a “permission denied” error when opening an epub with Sigil; this was caused by badly constructed epub files. A workaround has been implemented so you should not see this anymore.

Mac users also get an x64 build in the universal binary, and this should bring another 10% performance improvement to people who have a Mac that can support this architecture. Support for Tiger is dropped along with support for PowerPC Macs.

Linux users get some custom love too: you should now be able to use Ubuntu’s “Open With” menu with Sigil.

If you see any regressions, please report them ASAP. Sigil 0.2.4 stays the “official” versions until any major problems with 0.3.0 are resolved.

Introducing FlightCrew, the epub validator

2010-10-02T22:26:00.001+02:00

I’ve been talking about this for a while under the name of “that epub validating library”, and now it has a name. That name is FlightCrew.

It’s a C++, cross-platform, native code epub validator (it’s also open source). The project is composed of three parts:

FlightCrew, the validation library;
FlightCrew-cli, the command-line front-end to the FlightCrew library;
FlightCrew-gui, the GUI front-end to the FlightCrew library.

There are installers and DMG’s for download that package FlightCrew-gui, which provides a nice GUI interface to the underlying library. Errors have a reddish background (ok, it’s pink), while warnings have a yellow one. Here’s a screenshot:

I’ve kept the interface to a minimum on purpose. There’s something to be said about simplicity. As Antoine de Saint-Exupéry said: “Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.”

You can also drag files from your desktop or file browser and drop them on the FlightCrew-gui window. This will instantly run validation checks on it. This drag-and-drop interface works on all platforms.

FlightCrew-cli is included in all packages of FlightCrew-gui, and does the same thing as that application only from a command-line interface. It works the way epubcheck works—feed it a file, it spews out warnings and errors if necessary.

The current version number for all this is 0.7.0, which I’m using as a sort-of indication of it’s completness. I started working on this back in July, but since there was a rather lovely summer between now and then, it has only had about a month and a half’s work put into it. It’s still roughly 20 KLOC with a complete test suite, so it’s no slouch.

Why FlightCrew is better than epubcheck

First off, “better” is a dirty word. Each tool has its pros and cons. Epubcheck’s (EC) advantage is that it checks for a few things FlightCrew (FC) doesn’t (yet). But the reverse is also true: FC checks for a lot of things EC doesn’t. Off the top of my head, FC performs an extensive reachability analysis and will warn you if you have some resources listed in the manifest that are not used anywhere. It will also report an error if you have an OPS¹ document that the user can reach—through the <guide> or <tours> element, the NCX or just normal links in the text—but that is not listed in the <spine>. This is one crucial mistake that can now be caught. Reachability analysis also catches files that are used but not present in the manifest.

There are many other things that FC will check for that EC will not, and most of those you care about deeply and just don’t know it. The things EC checks for that FC doesn’t? Two “big ones”: OPF-listed fallbacks and DTBook syntax verification. If you haven’t heard of either, then you’ve never used them, never will and probably shouldn’t. These are very rarely used features of epub that I have personally never seen used in practice. But they’re big parts of the epub specifications so FC should check for them (and will, fairly soon) for the sake of completeness. There are a few other odds and ends that EC looks for but FC doesn’t.

But here’s where FC blows EC out of the water…

Error reporting done right

Let’s pretend I don’t know most of the epub specs by heart and that I’m a newcomer to epub. I made my first epub book, and I’ve heard that I should validate it. I’ve downloaded both FC and EC and now I’m going to use both. I’m going to use EC first because I’ve heard it’s “what the pros use”. Note that the a pair of EC/FC examples refers to the exact same problem with the file, and the messages usually come with line numbers (unless otherwise specified) which have been omitted. Commentary has been added for the sake of ridicule.

EC: length of first filename in archive must be 8, but was 19 [no line number, ed.]

Um… what? WTF is that supposed to mean? What filename? And why must it be exactly 8? What the hell are you talking about?

FC: Bytes 30-60 of your epub file are invalid. This means that one or more of the following rules are not satisfied:
1. There needs to be a "mimetype" file in the root folder.
2. Its content needs to be *exactly* "application/epub+zip".
3. It needs to be the first file in the epub zip archive.
4. It needs to be uncompressed. [no line number, ed.]

Ah… not only does this point out the problem (correctly!), it also tells me how to fix it. Nice.

EC: required attributes missing

Huh? I understand that you’re trying to tell me that some required attributes are missing (one or more? you haven’t said), but how about telling me which ones you frigging bastard. Am I supposed to read through the entire XHTML specification, hunting down which attributes this element should declare or even know that I’m supposed to do exactly that?

FC: missing required attribute 'alt'

Thanks! That was awesome. Saved me a ton o’ hassle.

EC: unfinished element

Exceedingly useful, that. Mind telling me how it’s unfinished?

FC: The <title> element is missing.

Now that’s more like it. I’d kiss you if I could.

EC: unfinished element

Didn’t you just say this? And on the exact same element? Why am I getting this again, I thought I fixed this…

FC: The <identifier> element is missing.

You just keep getting better and better!

EC: unfinished element

Fuck you, epubcheck. Fuck you.

FC: The <language> element is missing.

Want to pet my cat? ‘Cause you’re awesome, and I only let awesome things pet my cat.

Back to reality. I hope you got the point, cause if you haven’t, I can pull out tons of other examples.

Now I know that most of the error messages from EC are actually coming from an internal component of it called Jing and that has crappy error messages, but as a user I don’t care. Adobe should use something better instead, or fix Jing. And lots of the crappy messages come from EC core; that little “length of filename” gem was all theirs.

Have I also mentioned that EC development is pretty much dead? From it’s public source repository, it has had a whopping one source code commit in the last ten months, and that commit was four days ago.

In short, use FC first, then EC to get some of the checks FC doesn’t (yet) perform. After FC becomes a strict superset of all EC functionality (roughly a couple of months), drag epubcheck down to the cellar and shoot it in the back of the head.

Or just stop using it, if you prefer.

Footnotes

[1] A funny way of saying HTML document. It’s more than that of course, but for now mentally replace “OPS” with “HTML” .

Mac OS X 10.4 support is going the way of the dodo

2010-10-01T19:42:00.001+02:00

Just a quick note to people who may still be using Sigil on Tiger: 0.2.4 was the last version to feature support for that operating system. Why? Many reasons. Let’s start with a few:

Sigil 0.3.0 will use Qt 4.7, and Nokia is slowly discontinuing support for Carbon with this release. It’s still there, but it’s second-tier.
So far, Sigil was offered in 32bit versions for PPC and Intel architectures on Mac (as a universal binary). I want to provide 64bit versions, and for that I need to use the Cocoa version of Qt, which doesn’t exist for Tiger.
From site statistics, about 4.5% of all Mac users of Sigil use Tiger. 73% use Snow Leopard. These people would like explicit 64 bit support, and could certainly put it to good use. As previously reported, Sigil runs about 10% faster on such architectures.

There are a few other reasons.

In short, this is the right way to go. I know some people will be seriously pissed off at this, but I have to think about the majority of the users. Sigil 0.2.4 will be indefinitely available for download on the project site (as with all versions of Sigil), so you can always just continue using that.

So Qt 4.7 is out… where’s Sigil?

2010-09-21T16:09:00.002+02:00

As the title says, Qt 4.7 is out. The trunk version of Sigil has been running on Qt 4.7 RC1 for weeks now, and any problems that I’ve noticed I’ve already fixed. But it’s not all roses…

I’ve also transitioned to Visual Studio 2010 and the MSVC 10.0 C++ compiler… and it apparently miscompiles Qt 4.7 when compiling as x64. Here’s the bug report on Nokia’s tracker. To be clear, this is not a Nokia bug, but a Microsoft bug. MS apparently has a hotfix already, and it’s about to be released any day now. They’re testing it to make sure it doesn't break most of the world’s software. :)

What does that mean for Sigil? Well I was hoping that the hotfix would be released before Qt, but that didn’t happen. So the release of Sigil 0.3.0RC1 is delayed for the time being. There were a few more bugs I wanted to fix anyway.

I’m going to give MS about a week more to release the hotfix; if it’s not out by then, Sigil goes back to MSVC 9.0 (the same version used for all Sigil releases so far) until it is.

Typing like crazy at 25wpm

2010-09-05T18:22:00.002+02:00

So what have I been doing for the last few weeks? Two things: working furiously on the epub checking library and trying to transition from the QWERTY keyboard layout to Dvorak (touch typing, of course).

You have no idea how hard the latter is.

I tried to switch a couple of times this year, but failed both times. The problem is that it takes you about two to three weeks of regular, daily practice (we’re talking a couple of hours a day just typing) to get to something like 50wpm. During that time, you’re crawling like a snail. It’s absolutely insufferable when you’re used to your thoughts just flowing out of your fingers at 60wpm. And if you have something with a deadline, something you just have to get done, forget it. You’re either not going to do it, or you’re going to switch back to QWERTY. And that will kill you.

You absolutely have to go cold turkey on your old layout to be able to switch to a new one. Trying to “ease yourself into it” doesn’t work; believe me, I’ve tried twice :). You basically have to rewire your brain to the new layout, and reminding it of the old one repeatedly undoes everything you learn.

So this is a whole new world of frustration for me. But it’s worth it in the end, it’s much more comfortable to type this way. My hands used to hurt after a long day; they don’t anymore. Others have told me that after you really get used to it (a couple of months), you type about 30% faster than you did, and without pain.

But enough about that. Epub!

The epub checking library^[1] is progressing quite nicely:

OPS checking is done (including SVG Full and OPS <switch>), except for the DTBook syntax^[2].
OPF checking is about 90% done with fallbacks remaining plus a few other checks.
OCF checking is done for all six META-INF XML files as it pertains to schemas.
NCX checking is done at the schema level; link checking remains.
The CLI and GUI clients have been designed, but no code has yet been written.

Note: “done” means done to the point I’m aiming for in the first release, and which includes most of the things you care about.

All in all, it’s going well. My last “class-based” semester is starting tomorrow, so naturally things are going to slow down now. Otherwise I’d be done in a week. With classes, it’s more likely to be three weeks, barring unexpected complications.

Also, Qt 4.7 is at the RC stage, and I’m aiming to integrate Xerces as a QDom substitute for the next release of Sigil to go along with the new Qt version. So that will eat at least a week too.

Footnotes

[1] It has a name now, but for a few fairly ridiculous reasons I’m not willing to reveal it until the first release.

[2] Because nobody uses it! AFAIK it’s also scheduled for deprecation in epub 2.1. Validation for this syntax will eventually make it in for the sake of completeness, but not for the first release.

0.2.4

2010-08-12T20:44:00.001+02:00

I just pushed a new release. Here’s the changelog (highlights bolded):

fixed a problem with updating image paths for images with the same filename but coming from different parent directories (issue #501)
added a new "Merge With Previous" context menu action for XHTML files in the Book Browser (issue #265)
changed Tidy to handle the common typing mistake of ending entities with a ':' instead of a ';' (issue #535)
fixed a bug where double-clicking a file in the Book Browser for a file that was already opened in a tab switched that tab back to Book View; the tab now retains whatever View it was in previously
newly opened tabs now default to the View of the current tab (issue #468)
re-engineered the locations where Sigil stores its work files; the system-provided temp folder is now used; this should alleviate some permissions issues on certain machines, especially Macs (issue #404)
Sigil now prevents the renaming of files in the Book Browser to file-system invalid names (issue #493)
changed the keyboard shortcut that opens the Replace dialog from Cmd+H to Cmd+Shift+F on Macs only; Cmd+H is used by Mac OS X for window hiding (issue #477)
fixed an issue with Sigil using XHTML 1.0 for OPS doctypes, instead of XHTML 1.1 (issue #503)
several files can now be marked as having the Text semantic type (issue #522)
fixed an issue with Direction: All in book wide searching skipping last XHTML file (issue #520)
fixed an issue with the declared XML encoding not being picked up if it was wrapped in single quotes instead of the more standard double quotes
fixed an issue where the user could avoid the warning dialog for book-wide searching in Book View if he switched to this mode in Code View, and then switched back

This is mostly a bugfix release, with the notable exception of the “Merge With Previous” feature. Lots of people asked for that, so there it is.

I’ve also changed the location where Sigil stores its work files since it was causing an error when starting on certain Mac machines. I’ve also changed the “open Replace dialog” shortcut from Cmd+H to Cmd+Shift+F. You can now use Cmd+H to hide Sigil windows. :)

Vacation time over, get back to work!

2010-07-25T16:24:00.002+02:00

So I just returned from vacation.

I didn’t write a single line of code for the never-ending doxygen comment conversion, but I did end up working quite a bit on the epub validating library. It’s progressing quite nicely, it’s already a couple of KLOC. I’m also doing it TDD-style which I’m finding to be a rather pleasant workflow, once you get used to it. Seeing a test fail and catch a regression I wouldn’t notice until after it was too late brings a smile to my face. On a related note, gtest and gmock are awesome.

Oh, remember when I said that writing XML validation checkers by hand instead of using schemas would be painful? Guess what, it is. It really, really is. :)

One of the other benefits of working on this library is that I’m solving all sorts of problems with Xerces integration. The library uses Xerces for everything XML, and working with it has given lots of insights that will be applicable to Sigil when I start replacing QDom with it.

I’ll be mostly working on this library over the coming weeks, so Sigil will see only bug fixes going forward. Hey, the library is the major future feature for Sigil since it will be integrated into it, so transitively I’ll still be working on Sigil all this time.

Qt 4.7 is just around the corner too^[1]. When it does arrive, you can expect a version of Sigil integrating it very quickly, provided there are no problems migrating to the newer Qt^[2]. The QtWebKit improvements alone make me giddy like a little schoolgirl. 4x faster rendering? Hell yeah!

Footnotes

[1] Actually I expected it to be released while I was on vacation. Or at least QtWebKit 2.0. Nokia devs said that was coming in May, and it’s the very end of July now… grumble…

[2] I seriously expect no problems. Nokia has always been adamant about backwards compatibility, and from past experience I can say they usually do a good job on this.

Vacation and validation

2010-07-08T14:33:00.001+02:00

The semester is finally over so I’ll be heading on vacation tomorrow. I’ll be gone for about two weeks, and in that time no issues will be examined, no code will be written etc. Well, not really. In general, I won’t have internet access but I may pop in from time to time to check the Sigil forum on MobileRead. No promises though.

While I’ll be spending most of my time lying on the beach, soaking up the sun with a comfortable book^[1] by my side, I’ll also be doing some light development work. By “light” I mean finishing the doxygen comments conversion for Sigil (which has been taking forever) and some code for the new epub validation library I’m writing. Not a lot of code, just a bit.

What’s that last part about? Well what’s the most requested feature for Sigil on the tracker? It’s integrated epub validation. I’ve planned for months to start working on this as a separate project in August, and I’ve just started doing some architectural design for the library^[2]. The dependencies are in place, the build system works across platforms and I have a pretty good idea of how things are going to mesh together.

The project will in fact be composed of the following:

the main epub validating library, written in portable C++;
a CLI application that uses that library (think epubcheck);
a GUI application with pretty buttons and drag-and-drop support.

I know a ridiculous number of people who don’t validate their epub books just because they are too scared of the command line to use epubcheck. While I think that’s absurd and childish, I can’t deny that having a simple GUI would be easier for the average user. And if the GUI supported dragging an epub file from the desktop and dropping it on the app window to initiate the validation, well that would be very useful. It could certainly speed things up.

The goals of the project are as follows:

check for everything epubcheck checks for, and much more;
be easily embeddable into native code applications (no frigging Java);
be easy to use and easy to understand: “unfinished element” means diddly-squat to people who don’t already know what the element is supposed to contain^[3];
have developers that are active and responsive (and an active development process in general);
optionally warn about valid epub constructs that cause problems for certain high-profile Reading Systems.

That last part is interesting. We all know ADE has many quirks, and so does the iPad and other RS’s. Wouldn’t be lovely if you could instruct your epub validator to check for at least some of those problems as well?

I think that would be fairly useful.

This library will then be embedded into Sigil, and a simple toolbar button press will validate your epub and report any problems.

Before anyone gets ahead of themselves, bear in mind I won’t hit all those goals for the first release. It will be incremental and it will get better in time. But the scope of this project is thankfully much smaller than Sigil’s, so that’s a big relief.

Footnotes

[1] Currently it’s The Stand by Stephen King. That book has like a billion pages. It’s probably not helping matters that I’m reading the extended version.

[2] A software library, not a building. There was a misunderstanding about this a couple of days ago, and needless to say I laughed my ass off.

[3] This will entail writing some of the internal validators by hand and not resorting to schemas. Painful, yes, but necessary. Not just because of usability, but correctness too. For instance, a schema can’t check that the files listed in the manifest are actually in the epub. It won’t tell you if you’ve included the same file multiple times with different ID’s. There are many examples, most of which are checks that are way more important than an ID starting with a number.

I won’t do this for everything (I still intend to use an XML Schema for XHTML validation), but I will do it for the OPF. Most validation problems are there anyway. The OPF is the heart of an epub.

0.2.3

2010-06-23T00:38:00.002+02:00

Bumpy ride lately. Changelog follows:

fixed an issue with the new data from one view sometimes not being saved in the final epub when switching to the other view
worked around a Qt focus issue causing current tab data to sometimes not be saved; this was uncovered by fixing the hang-on-save issue, which was caused by the same underlying problem (issue #466)

The work in 0.2.1 has caused some unfortunate synchronization issues.

Problem #1: You work in Code View, make some changes and switch to Book View. You see the changes transfer to BV, then save the epub. Opening the epub, the changes are not there.

This was my fault and should now be fixed.

Problem #2: You work in Book View (or Code View), make changes and save. Sometimes the changes do save, sometimes they don’t. Same thing goes for the TOC editor sometimes not seeing your changes, or the Find/Replace dialog messing things up and losing data.

This is caused by a “lost” focus event. I’ve observed this only once on Windows 7, but it’s reportedly much more frequent on Win XP machines. Linux and Mac machines seem to be immune (although similar issues caused by problem #1 can masquerade as this problem). This seems to be a Qt issue, and I’ve now worked around it. The same underlying “missing focus event” issue was causing the previous hang problems.

There was a third problem reported: you make changes in CV, then switch directly to BV. Any changes are now gone. I’ve only had one user report this, all other reports appear to be caused by problems #1 and #2. I’m still unable to reproduce this, no matter what I do or what machine I use. If you have this issue with the new 0.2.3, please report it ASAP. Use the issue tracker.

I’ve been stealing away hours from my other obligations to work on Sigil since these are major problems, but man, doing this has already started to haunt me…

0.2.2

2010-06-20T23:02:00.001+02:00

This is a very minor bugfix release. Changelog follows:

simplified the resource locking mechanism; should eliminate the hang-on-save issue
fixed a problem with Book View chapter splitting sometimes not being registered on save, causing duplicate content (issue #450)

I’m trying to leave a rather stable version of Sigil before going on my coding break, so this release is important. Users have reported a higher chance of encountering the hang-on-save problem, and now I've completely changed a fundamental aspect of the resource locking architecture and also minimized all the critical sections so this should really be fixed now. If it’s not… then it’s caused by something completely unrelated. Which is possible, seeing as how I can’t reproduce the issue and thus get an exact fix on the culprit. The bug is not deterministic, which clearly points to the threading-enabled code.

The second major bug was caused by a line a code I forgot to add after some recent refactorings.

If you see the hang-on-save issue with this new release (or any other kind of hang bug, or any other kind of bug at all :) ), please report it ASAP with as much detail as possible.

0.2.1

2010-06-19T01:24:00.002+02:00

I’ve just pushed 0.2.1, which is mostly a bugfix-only release. Changelog follows:

XHTML files that specify two different encodings are now fixed by removing the incorrect one
Sigil now checks the XML encoding attribute for an encoding before the HTML metatag and charset; should now be more compatible with Calibre created epub books
created/used 16px versions of all icons; icons in menus are not blurry anymore (issue #121)
the Find&Replace dialog now uses the currently selected text (if any) as the default search term (issue #370)
fixed issues with unnecessary reloads of the code view (issue #412, issue #398)
fixed an issue with the HTML file filter in the open file dialog not correctly filtering files (issue #416)
fixed an issue with files without extensions not being saved in the final epub (issue #400)
fixed an issue with XPGT resources sometimes being saved blank (issue #433)
fixed a regression that made it impossible to add removed headings back into the TOC (issue #439)
fixed a problem with some file-wide replacements reverting
fixed a problem with the opened tabs not being updated until the user gave them keyboard focus when a file-wide replace was performed (issue 408)
fixed a problem with Book View not reflecting changes done in Code View when the Code View was used for editing, and then the tab closed
use of custom synchronization primitives should resolve most infrequent hang bugs
fixed a problem with the search not progressing in Book View find&replace when using recursive replacements
fixed an issue with inserting images that have apostrophes in the filename (issue #391)
TOC text now has leading and trailing whitespace trimmed, and inner whitespace condensed (issue #422)
an empty ALT attribute is now added to "img" elements that don't have them (issue #406)
added the build time to the About dialog, showing date and time in UTC

Remember when I said that 0.2.0 was surprisingly free of major bugs? I spoke too soon. :)

The major issue was that replacements performed with the Find&Replace dialog could sometimes revert back. Horrible, I know. This should now be fixed.

The next major issue actually goes way back to the start of 0.2.0 betas, I just couldn’t track it down: Sigil could sometimes completely hang. This usually happened during saves, and should also be fixed now. I’ve written some custom synchronization primitives^[1] that should alleviate these problems. This also involved rewriting the way tabs release their locks etc. Lot’s of nice things. Should work now.

And the last major problem was a regression that made it impossible to add removed TOC items back in.

Footnotes

[1] Nothing too fancy, I needed a ReadWriteLock that was shallow and non-recursive. Qt provides one that is non-recursive by default, but not shallow. By “shallow”, I mean a lock that allows only one level of locking by the same thread, thus silently letting multiple lock calls succeed, even though only the first one did. Same thing goes for unlock: the first call unlocks, the others just skip. The lock and unlock calls are appropriately called LockIfNeeded() and UnlockIfNeeded().

Admittedly, a shallow lock is rarely needed, and if you’re using one, you damn well better know what you’re doing.

A brighter future

2010-06-14T17:21:00.001+02:00

Well it’s been quite a while since the last post. I’ve been busy with university work, and while I have one more paper to submit in a few days and finals in about ten, I have a bit of free time now.

I’ll be working through the bug reports I received since 0.2.0 final went live. I must say I’m pleasantly surprised that nothing major was reported. I’d like to thank all the people who have written thorough bug reports. I haven’t had time to go through them all and respond appropriately, but I’m getting there.

Sigil 0.2.1 will probably come sometime in late July/early August. While I have a few days free this week, I’ll be studying the following three weeks and immediately after that I’m going on a two-week vacation.

There’s an ever-so-slight chance that I’ll push a very minor bugfix release in about a week and postpone all the work I wanted to do for 0.2.1 into 0.2.2.

I’ve also started work on replacing all the uses of QDom with Xerces-C++ in a separate repo, but that’s going to take quite a bit of time.

What I’ve really spent a lot of time with over the last several days is QtWebKit. There was a bug on the WebKit Bugzilla for the general QtWebKit performance problems… When the bug was reported, my testcase on one machine took 55 seconds to load and render. For Qt 4.7beta1, Nokia got that down to 21 seconds. With the trunk version of Qt 4.7 from a couple of days ago, it’s down to 13.3 seconds. Bottom line, they’ve done some good work speeding it up, and some more work still remains. It’s not native-WebKit or Firefox fast, where the testcase renders instantaneously, but it’s improving.

I downloaded the Qt trunk source a few days ago and extensively profiled QtWebKit rendering. 95% of the time is spent shaping glyphs in the call tree of QFontMetrics::width(), and man does a lot of code get run just to get the width of a text string. The calls drop into Harfbuzz, and all sorts of functions end up eating a bit of wall time here and there, and this all adds up. There’s no single point that does something very stupid that could then be optimized away, at least not in this module. There’s even an internal “simplified” codepath for getting the width of a text string, but I’m not sure why the Nokia guys are not using it in QtWebKit. I’m probably missing a piece of the puzzle. Either way, there was no low-hanging fruit to remove.

The most fruitful optimization work will probably come from higher up the call chain. For instance, there are superfluous layout calls for page rendering, which (if I’m reading the bug comments right) make the whole thing calculate the render tree three times. Ouch. But they’re working on it.

The point to take home is that when Qt 4.7 lands, Sigil should see a very nice performance boost for loading books and switching to the Book View. Probably for general WYSIWYG actions in the Book View too. Hurrah for that.

0.2.0 FINAL

2010-05-11T23:00:00.002+02:00

And here it finally is. After months and months and months of finger-breaking^[1] work, it’s finally done. Changelog is minimal going from RC4:

added new entries to the help menu for the online manual and the FAQ

Now cue tens of thousands of 0.1.9 users screaming “WHY DO I ONLY SEE JUST THE FIRST PAGE??!!”. See the FAQ.

Footnotes

[1] Like back-breaking, only for programmers.

Manually, but for the web

2010-05-09T16:32:00.002+02:00

When I decided I wanted to write the manual for Sigil, the first thing that came to mind was putting it online as some sort of website. A web version can be always up to date, you can put a link to it in the Help menu of your application and you don’t have to blow up the download size by including it in the installers.

But that was wishful thinking; Sigil is hosted at Google Code, and it doesn’t offer a way to host custom HTML pages, and I’m not buying a domain and webhosting just for a damn manual. So sadly, that couldn’t work.

The second idea was writing the manual in pure LaTeX, since if I’m going to write an “offline” version of the manual, I’m going to typographically do the best job I can. Epub can be used for technical manuals, but some of the nice typesetting tricks I hand in mind couldn’t be done in it.

Then it occurred to me that having a PDF-only version of the manual for an epub editor was… not the best political decision. I wanted to find a way to eat my cake and have it too: to offer both a nice PDF and a nice epub. I was willing to sacrifice some of the things I could do with custom LaTeX if the resulting PDF was “good enough”.

reStructuredText and Sphinx came into the picture. As I mentioned in a previous post, the PDF version of the manual sucks. Links don’t go where they’re supposed to, the layout is shoddy and in general, it just looks bad (to me at least). Sphinx was primarily designed for creating HTML documentation, and the LaTeX generator suffers because of it. I decided to upload what I had now, gather some feedback on the content and then rewrite the thing in custom LaTeX.

Then came the expected politics…

While I’m convinced that I’m right^[1], I wanted to talk to a few friends who lead or are members of other open source projects (some hosted on GC too) what their thoughts were on the subject. I got universal agreement that a LaTeX PDF was the way to go, but one of them pointed out that GC indeed does provide a way of hosting sites, just… unofficially. The Google guys apparently have no problems with people hosting their site inside their source code repositories. This could of course only work if your site serves only static content, which a manual would certainly be. They’ve even improved the SVN backend for direct web content serving, so they certainly approve of the practice.

Sigil uses Mercurial, and Google provides every mercurial-based project with many repositories. So I used Sphinx to generate the HTML, committed it all to a new “web” repo and voilà! An online manual.

Sphinx produces wonderful HTML, and I’m willing to sacrifice some typesetting beauty for online accessibility. It’s also much easier to customize the HTML output by injecting new CSS. Tweaking the LaTeX output was a nightmare.

There won’t be an offline epub version since there’s no point anymore and I want people to see the code samples and images on a large screen. But if someone wants one, they can easily download the manual sources and build one with Sphinx.

I think everyone should be satisfied now.

Footnotes

[1] Big surprise there.

0.2.0RC4

2010-05-08T15:57:00.001+02:00

Another minor update:

fixed a regression that broke FindNext opening new tabs when searching across HTML files (issue #384)
fixed an issue with autocompletion in Find dialog ignoring the case of search terms (issue #385)

I’ll be doing these until the explosions stop.

Manually

2010-05-08T14:42:00.001+02:00

I’ve just uploaded the manual for Sigil. You can get it from the downloads section.

The manual was written in reStructuredText (RST) and then processed with Sphinx. Honestly, I’m not too happy with the layout of final file compiled from the Sphinx-generated LaTeX. Some of the hyperlinks don’t lead to where they’re supposed to (bug in Sphinx I guess), and typographically in general… well I don’t like it.

Initially, I wanted to write the manual in straight hand-coded LaTeX. I decided against that because that would mean the manual would only be available in PDF form.

Now, I’m the kind of person who always tries to use the right tool for the job. I love epub and absolutely abhor the PDF “ebook” novels. PDF for novels just doesn’t work, and if you’re producing them… don’t. Make epubs. But for technical manuals with complex layouts, code listing, figures, diagrams, sidebars and the like, PDF is the right choice IMO. Epub is just not the right call for these kinds of books.

Sure, you could make epub versions of technical manuals, but you’d have to sacrifice some of your layout.

Then there’s politics. An epub editor with a manual in PDF form only? That just looks bad. That’s kind of like Microsoft employees using Linux.

I was pondering this until someone told me that the next version of Sphinx will have an epub generator. You would give it RST and it would spit out a nice epub file from the same source it would generate LaTeX markup. That sounded like an amazing deal, so I went with that.

I loved RST until I started using it. I don’t love it anymore. It has a weird syntax and to get anything above the barebones, you have to do all these strange contortions etc. Add to this that the resulting LaTeX markup produces a PDF that I dislike. I tried to tweak it here and there, and I’ve improved the output considerably over the last few days, but still… Having an epub file to silence the naysayers is good, but not at the expense of quality.

I’m pretty sure I’ll be rewriting the manual in pure LaTeX when I get the time (a few weeks from now, probably). Until then, enjoy the current version.

And if someone feels that the new PDF-only manual shows Sigil in a bad light, I say screw ‘em.

0.2.0 RC3

2010-05-07T00:51:00.001+02:00

Ok, RC3 is now up on the site. Download it here.

Changelog:

fixed a regression that messed up the tab order of the controls in the Find dialog (issue #380)
fixed an issue with cross-file FindNext causing a hang when the search term is not in the book and "Direction: All" is used (issue #378)
the Text folder in the Book Browser is now expanded by default after loading

This is also the first release that was completely built and uploaded by an automatic process composed of several Python scripts. Since it’s brand new, it could have introduced some problems. The whole point of this automated build system is to eliminate those nasty situations where I would upload the wrong file, forget to include some critical libraries into the installers or something else.

Once the system has been set up to produce correct builds, it should stay that way.

If anyone’s wondering when 0.2.0 final will be released, my rule of thumb is three days without any showstopper bugs reported. In RC2, the showstopper was the FindNext hang that has now been fixed.

0.2.0RC2

2010-05-05T01:46:00.001+02:00

The second release candidate is now available. Changelog follows:

fixed an issue with ReplaceAll across files not using correct replacement lengths
fixed an issue with code in Code View not being pretty-printed
fixed an issue with the ReplaceAll across files not informing the opened tabs of the change

Small but important. I consider all of these to be showstoppers. Hopefully these are all there is.

If anyone finds any other bugs, please report them on the tracker. If this release stays showstopper-free, it will be promoted to a full release later this week.

0.2.0RC1

2010-05-04T00:22:00.010+02:00

The first Release Candidate is here and you can get it from the downloads section. Between now and final 0.2.0, I’m fixing only showstopper bugs and maybe a few of the trivial ones. If all goes according to plan^[1], there won’t be any need for further changes before the final release and thus you’ll see it a few days from now. This is not called a Release Candidate for nothing.

So hammer this version hard, especially the Find & Replace functionality.

Here’s the changelog. It’s by far the largest batch of changes to Sigil for one release (disregarding the first 0.2.0 beta, of course). Also, some very important bugs were finally fixed plus lots of new features, so this really is a big release. Some highlights have been… highlighted:

changes in the Book Browser now update the modified state of the main window (issue #331)
the Book Browser can now be opened/closed from the View menu (issue #335)
all the toolbars now have UI-facing names
by injecting a custom XML reader into QDom, the following issues were fixed:
- Book View search sometimes skipping over instances (issue #253)
- Book View ReplaceAll causing Sigil to hang on rare occasions (issue #293)
- spaces disappearing from some HTML constructs (issue #352)
implemented component-wide search&replace for Code View searches (issue #372)
the Find&Replace dialog now remembers up to 20 previously used search and replace strings (issue #369)
fixed an issue with positive regex lookaheads in normal Replace (not ReplaceAll) (issue #261)
fixed a rare off-by-one error in Book View searching when the caret was at the start of the matched string; this made the search skip that instance of the match (issue #280)
fixed an issue with the Find Dialog not correctly scrolling to the found text in Book View (issue #195)
fixed an issue with Tidy not fixing free ampersands into "&", even when configured to do so (issue #365)
fixed an issue with the current tab unnecessarily reloading after book saves (issue #354)
fixed issues with filename basenames being read only until the first dot; was causing problems with OPF manifest ID generation (issue #351)
hitting the keyboard shortcut for the Find&Replace window while the window is open now switches focus to that window (issue #362)
fixed an issue with the applied headings not "sticking" and not showing up in the TOC editor (issue #300)
the special iPad- and Calibre-friendly cover meta tag information is now preserved after loading
added a new "Cover Image" entry for image resource in the "Add Semantics" Book Browser menu
if an image is not set as a cover image manually, Sigil now uses heuristics on save to determine if the epub has a cover image
if an epub has an image set as a cover image, Sigil will now write a special meta tag that identifies this image in the OPF; this tag is then used by the iPad (and Calibre) for the book cover, for instance
all OPF <guide> element information when loading epubs is now preserved
added a new "Add Semantics" menu for XHTML documents; it can be used to mark XHTMLs as "Dedication", "Colophon", "Glossary" etc. for the <guide> element of the OPF
the status bar now shows a message after chapter split operations
fixed an issue with filenames with characters that should not appear in valid XML IDs having those characters added anyway (issue #344)
fixed an issue with files with uppercase extensions not having a mimetype set in the OPF (issue #349)
fixed an issue with Sigil rewriting headings when the TOC was opened and no heading was edited (issue #327)
fixed an issue where adding an existing HTML file through the Book Browser would clear the current metadata in the book (issue #329)
added a check that prevents Sigil from loading the same resource multiple times in invalid epubs (issue #339)
fixed a bug that made the direct XHTML references in the NCX file less likely (issue #333)
fixed an issue with Sigil crashing when trying to save a loaded epub that had some badly formed metadata elements (issue #325)

It’s pretty damn huge.

Book-wide search

For now, you have to perform your book-wide searches in Code View. I know, I know. It’s not everything you wanted. But technical restrictions are causing problems^[2]. I’ll try to write a post about it tomorrow explaining the issue, I’m too tired to go into it now. Just learn to live with only Code View searching until I work around this^[3]. It should cover 95% of all your needs.

I’m making it up with the new remember-20-last-used-search-strings feature. The input fields also provide automatic text completion for previous searches. All of this was scheduled for 0.2.1, but hey.

Footnotes

[1] Do I even need to chuckle at that?

[2] Cookies to the first person that points out the framework giving me grief.

[3] Something like 0.2.2, maybe sooner.

The new “Add Semantics” menu

2010-04-26T21:18:00.006+02:00

After the iPad came out, everyone (myself included) wanted to know how well it would handle epub. As it turns out, not great. No embedded font support? Bad Apple, bad!

Anyway, the iBooks application for reading epub books on the iPad comes with this large shelf view where all the books display their covers. Shiiiiny. But as a content producer, you have to explicitly tell the iPad which image in the epub archive is the cover.

There was an interesting discussion about this on MobileRead. The idea to take away is that you need a special meta tag in the metadata section of your OPF file. Something like this:

<meta name="cover" content="coverID" />

The “coverID” needs to be the ID of the cover image in the manifest section.

Until now, you had to save your epub with Sigil, extract the epub, edit the OPF, add this meta tag and then zip everything up properly (mimetype file first with no compression etc.). Oh and you could never open that epub again with Sigil since it would remove that special meta tag.

A horrible PITA, wasn’t it?

Well that’s fixed now. The next version of Sigil will have a right-click context menu for images with which you’ll be able to mark an image as a cover. Done. Sigil does everything else required for you.

Along with this “Add Semantics” menu come new entries for the HTML resources. For those, you can now add the <guide> element semantic information. So if you mark one HTML file as, say, a “Title Page”, Sigil will add this information to the <guide> element in the OPF.

Naturally, this feature would be useless if Sigil wouldn’t preserve all of this information after opening an epub that already has it. So it does that now too: all information from the <guide> element is now preserved, including custom values for the “title” attribute (even though you can’t see that value in Sigil, it’s stored). The special meta tag identifying the cover image is read and that information is preserved, too.

Being smart about it

Sigil always does its best to help you out. Well, it at least tries :).

I’ve added heuristics to Sigil that will mark the appropriate image as the cover if you don’t do it yourself. If the first HTML file in the reading order is “very small” and has only one image in it^[1], that image will be selected as the cover.

So if you follow best practices, Sigil helps you out. Still, mark it by hand if you can. You will always know better than the machine.

Things to note…

While the OPF spec technically does allow you to, for instance, specify several HTML files as the title page in the <guide> element (for God knows what reason…), Sigil stops you from doing this. It allows you to set only one instance of one <reference> type per book. So if one file is set as the title page, setting another file as the title will unmark the last one.

The exception are loaded epub files. If your loaded file specifies several HTML files as, for instance, the preface, then all of those are still marked as such in Sigil after they’re loaded. While I personally think you should never use more than one reference type instance per book^[2], if you did this to one of your books before opening it in Sigil, that information will remain. Sigil won’t step on your toes here.

Finally, here are two images showing off the new menu. The file loaded is Three Men in a Boat. If you don’t want to wait for the next release^[3], you can build from repo source and start using this right now.

Footnotes

[1] Sigil looks for a normal <img> tag or an SVG <image> one.

[2] It’s a terrible idea, and it would probably wreak havoc on unsuspecting Reading Systems. The spec actually doesn’t explicitly allow it, it just doesn’t talk about this possibility at all. I’m still not convinced that the spec writers didn’t just forget to forbid this behavior.

So Sigil won’t let you do this.

[3] Which will be RC1. No more betas! :)

QtWebKit 2.0 AKA pigs flying

2010-04-20T16:36:00.002+02:00

So by now, you should know that QtWebKit is slow. Damn slow. If you don’t, refresh your memory.

If you're performing an editing operation in Sigil, and the UI blocks for 10+ seconds, it’s QtWebKit. If you’re loading a large HTML file into the Book View and everything grinds to a halt for 30+ seconds, yeah, that’s still QtWebKit. I’m not saying all the code I write is perfect (far from it), but QtWebKit has a special place in my heart of hatred.

Back in that loading performance post I linked, I talked about how it takes 75 seconds to load Three Men in a Boat in Sigil 0.1.9. Some of the Linux users may have been confused by that, since on comparable hardware, it would load much faster on a Linux machine.

And it’s true. Sigil performs about an order of magnitude faster on Linux than on Windows. Why is that?

Well it’s because while Qt is thoroughly tested on Windows, it’s certainly much less tested than on Linux. Qt developers are Linux users, and a lot of them are KDE developers as well. So the machines they use don’t run Windows. It’s a lot like the Sigil UI being fine-tuned for Windows, since I use it and develop on it almost exclusively.

So when there’s a performance regression in Qt on Windows, there’s a fair chance someone at Nokia will miss it. And they did.

The test case I attached to the bug report loads in 55 seconds on Windows and 6 seconds on Linux. Bear in mind the same test case renders instantaneously in Firefox, and probably also in Safari (although I haven’t tested it) which uses vanilla WebKit. Horrible, I know. They’re finally coming around to fixing that, since the bug is now included as one of the “release critical” bugs for QtWebKit 2.0 (which is now officially a separately released project from Qt). That means they’ll kill it before the next release, which should be sometime in May (Qt 4.7 a couple of months after that).

Here’s to pigs flying.

…and another thing

Remember the font variants issue preventing official font embedding support in Sigil? The QtWebKit bug causing the underlying problem has been tracked for the last eight fifteen months on their various trackers, and I’ve been told just a few days ago that it’s “not considered critical to the release” of QtWebKit 2.0… which probably means it won’t be fixed for at least another six months.

I ♥ Nokia.

β3

2010-03-24T01:42:00.002+01:00

The third beta of Sigil 0.2.0 has just been released. Here are the release notes:

added two new WYSIWYG actions that work for both Views: "Insert SGF Chapter Marker" which inserts the old SGF horizontal rule chapter breaking marker and "Split On SGF Chapter Markers" which splits the current chapter according to the placement of these markers (issue #262)
chapter splitting now works in Code View
fixed an issue with Sigil adding "xmlns='http://www.w3.org/1999/xhtml'" to every element when performing a chapter break operation (issue #313)
fixed a rare issue with false spaces being inserted into words during import (issue #139)
added a confirmation dialog for removing items in the Book Browser (issue #306)
fixed an issue with the line number area overlapping the text in the Code View
made Sigil remove the CSS cruft WebKit was adding to the "body" element
fixed an issue with spaces in filenames causing bad anchor element path updates
fixed an issue with spaces in filenames not being URL encoded in "href" and "src" attributes in the OPF and NCX files
fixed an issue with spaces in filenames causing invalid IDs (issue #301)
fixed a regression causing Sigil to crash when importing HTML files that reference resources that don't exist on disk
Tidy now converts all uppercase attributes to lowercase; mixed-case attributes are left as is
fixed an issue with Tidy choking on uppercase attribute names

The beta process is taking substantially longer than I expected. This is mostly because I have less time to work on Sigil than I expected to have, but it’s also because of a huge number of suggestions and bug reports contributed by users since the process began. I can’t thank you all enough, Sigil is better for it.

Some of the feature requests have surprised me, but in a good way. The long awaited SGF chapter markers and splitting are now finally in, so I hope you all put it to good use (and report bugs if you find any).

Epub-wide search&replace is still MIA since I’ve yet to figure out a way to do it that doesn’t suck. I’m not satisfied with the approaches I’ve tried. But I’ll figure it out, don’t worry.

The next three weeks will see little development activity for Sigil since I’m heading into another round of university exams.

β2

2010-03-11T00:17:00.001+01:00

The second beta of Sigil 0.2.0 has just been released. Here are the release notes:

fixed an issue with exported HTML/CSS/etc. files inside EPUBs having superfluous newlines
fixed an issue with the TOC editor adding empty "class" attributes to headings (issue #297)
added a new "Window" menu item with new "Next Tab", "Previous Tab" and "Close Tab" actions (issue #273)
fixed an issue with the font used in the line number area in the Code View being incorrect when the Code View is first opened; the problem affected mostly Mac machines (issue #290)
Sigil now handles corrupt epub files with an OPF referencing non-existent files (issue #289)
the Book Browser now doesn't scroll back to the top when an item is deleted or added (issue #263)
the Book Browser now allows a file's extension to change between HTM, HTML, XHTML and XML (issue #264)
OPF and NCX files don't rely anymore on UTF-8 XML default parsing, but specify their UTF-8 encoding directly in the declaration
fixed an issue with changes in the TOC editor not being reflected in the book (issue #277)
fixed an issue with the TOC editor not recognizing the "title" attribute on headings (issue #271)
fixed an issue with the user seeing the old, unclean source in the Code View (issue #286)
fixed an issue with the user being prompted to save when quitting even when no changes have been performed on the new/loaded file (issue #276)
fixed an issue with Book/Code View keyboard shortcuts firing in the wrong view (issue #266)
tentatively fixed an issue with Sigil locking up when chapter breaking (issue #267)
fixed an issue with Tidy adding a superfluous “lang” attribute that is also not allowed in XHTML 1.1
making sure that ID attributes used in the manifest section of the OPF are always valid
fixing export of epubs with XML files for OPS documents

Quite a few things made it to the second beta. As you can see, a truly great deal of bugs were fixed. Some of the things I wanted to include in this release didn’t make it in (like breaking chapters on SGF chapter breaks, issue #262). Sorry guys, but there were some really critical bugs in the previous beta (like the TOC editor not working… at all), and I wanted to push fixes for those ASAP.

My university duties and responsibilities are picking up again, so Sigil development will slow down a bit. Not too much I hope…

β1

2010-03-02T18:50:00.004+01:00

Let’s get this out of the way: the first beta of Sigil 0.2.0 is now out. It’s what all of you want to know, right? There you go. There is also a release thread in the Sigil subforum on MobileRead, and all discussion on the beta should happen there.

You can only get it from the “Downloads” section, not from the main page. The 0.1.9 version is still the “official” one, and the new version notification will not be triggered. Bear in mind that this is the very first public release of the code I’ve been working on for the past 3+ months, so it’s going to be buggy. I’m hoping most of the major bugs will be reported quickly and the second beta will have them remedied.

So really try to bash it. Do whatever you normally do in Sigil and report any (unintentional) differences on the tracker. Or anything else you feel should be reported. Just don’t forget to state that you’re using the β1 release in your issue. This would also be a good opportunity to read the Reporting Issues wiki page if you haven’t already.

Why a beta and not a Release Candidate as I’ve previously announced? Well because not all the features that I want in the 0.2.0 release are done. The biggest omission is the cross-file search: in the current beta, you can only search (and replace) in the currently open tab, but not across all the files in the book. The whole search mechanism is about to be overhauled, since Qt 4.6 finally brought a native API for interacting with elements in a QWebPage.

There are numerous other minor annoyances that need to be dealt with before I proclaim a “real” 0.2.0. Things like being able to select multiple items in the Book Browser and delete them; currently you have to delete them one by one, which I find annoying.

Bottom line, it needs a bit of polish. I tried to stay away from that since you always try to make things work first, and then gradually improve them later.

So, betas first, and then after a few of those you’ll see at least one Release Candidate. When I’m satisfied with the way it looks and behaves, I’ll mark the next release after that as “official”.

With that settled, let’s move on.

SGF is now dead

The SGF format was created to provide Sigil with a native file format that could be changed and modified as needed. Initially, I wanted to make epub Sigil’s native format, but that didn’t seem like a good idea at the time. Back then, I wanted Sigil to eventually be able to save many different e-book formats, so I needed to make sure Sigil could store anything the epub format couldn’t but that was potentially in use by the other formats. So SGF was born. Whenever someone asked about it, the short answer was “SGF is to Sigil what PSD is to Photoshop”.

It was a product of thinking for the future… but the future has changed. Sigil is now mainly focused on producing epub books. With that, the major reason for SGF’s existence was gone.

SGF had a few advantages over epub in Sigil 0.1.x. For one thing, it stored the text you saw in the Code View raw, with no preprocessing on save. Saving as epub would split the One Huge Flow™ into different XHTML files according to your chapter breaks, and a great deal of other “book normalization” transformations were applied as well. So SGF was definitely the more “native” format.

In Sigil 0.2.0, the separate files are now kept separate, and any advantage the SGF format had is gone. With no advantage and no real future need for it, SGF format export has been removed: epub is now Sigil’s native file format.

Before you start screaming “but I have hundreds of SGF files!”, 0.2.0 can still open SGF. From there, you’re just a click away from saving it as epub. I’m not leaving you people out in the rain. :)

Show & Tell

I prepared this little screen cast of 0.2.0 in action. Things to watch out for:

The loading speed;
The “paragraph merging” speed;
The Table of Contents open/close speed;
The CSS tab;
The XPGT tab;
The image tab, which supports SVG.

And here is the video…

Loading performance and 0.2.0

2010-02-13T00:20:00.003+01:00

I’m the first person to acknowledge that loading epub and HTML files in Sigil 0.1.x is slow. Very slow. Abysmally.

Why is that? Well there are two components in the 0.1.x loading process:

Extracting the epub file, reading the OPF, running Tidy on the source, updating resource references etc. Let’s call this the “file load”.
After the file load creates one large HTML flow, this is then sent to the Book View (integrated QtWebkit) for rendering. Let’s call this the “QtWebkit load”.

You can tell when the file load has finished and the QtWebkit load is starting: the moment you see “File Loaded” in the status bar is the moment that all Sigil code stops executing and now it’s up to QtWebkit to render the page.^[1]

QtWebkit load

QtWebkit load takes longer, by far. Using the Three Men in a Boat epub file^[2] from the MobileRead ebook uploads forum as a reference, the whole loading procedure takes 75 seconds in Sigil 0.1.9.^[3] Way, way too long.

Of these 75 seconds, only 14.5 are spent in the file load. The rest is all QtWebkit, so not something I can directly influence. The QtWebkit load used to be more than twice as fast in Sigil 0.1.5, but the subsequent versions of Sigil include Qt 4.6 (instead of 4.5), and in that version QtWebkit is much slower. Nokia developers admit they introduced some major performance regressions and are currently working on fixing that.

I have no intention on waiting for that to happen. It was bad before, but now it’s horrible. I considered it far too slow in Qt 4.5, and now?… So how about I come up with a way to work around this problem?

Currently, Sigil takes in all of the XHTML files in an epub, puts them all together and displays them as one. So you get one large “flow” where you can do all the editing. I chose this model because it’s the one used in the popular^[4] Book Designer (which doesn’t support epubs).

This is where Sigil 0.2.0 comes in.

Sigil no longer does that. All the original XHTML files are preserved and are edited one by one. Since there is no one huge flow, QtWebkit rendering performance goes up tremendously (since there is less to render).

Now, when Sigil loads your epub file, the first XHTML file by reading order is loaded in the initial tab. Since this first XHTML file is usually a cover page, it takes less than half a second to render. So now instead of 60 seconds for the QtWebkit load, you get 0.2 seconds (for the TMB file).

Great, ha? :)

File load

But then again, there are those 14.5 seconds for the file load. It would be great if I could get that down.

Most of that time is spent doing two things: running Tidy on the large concatenated HTML document, and updating the resource reference paths. The updating process takes much longer.

The resource updating process is necessary since Sigil 0.1.x renames your resource files. Since images, CSS files, HTML files, fonts etc. now have different names, all the HTML tags and style rules referencing them have to be updated. This takes a long time.

In Sigil 0.2.0, there are now multiple XHTML files, and they all have to be updated the same way. The original resource filenames are now preserved, but the file structure changes so we still need to update the paths. All the XHTML’s have different content, but the file path updates are universal. This means we can now parallelize this:

We create a thread pool equal to the number of logical CPU’s^[5] on the system;
We split the updating process into “tasks”, where each task represents the required update operations to be performed on each XHTML file;
We let the threads munch the tasks as they become ready to process them.

So if you have a dual core system like I do, two different threads execute two tasks at the same time. As they finish the task they have been working on, they arbitrarily pick a new one and work on that. So the more logical CPU’s you have, the more threads you can run, the more tasks your computer can work on at the same time, the faster the file loading will be.

I then plugged in the old updating subsystem into this multi-threaded architecture and ran it on my dual core. The file load on TMB dropped to 11 seconds. Not quite the ideal linear behavior, but that’s to be expected since there’s overhead in talking to the threads, managing the task pool etc. And the OS eats your cores too, so your threads can’t stay active all the time. Also, not everything in the file load can be parallelized; lots of things have to stay sequential.

The other major problem is that TMB has a huge number of images, meaning that many HTML “<img>” elements have to be updated. With a more conservative epub file, the numbers would be even better.

But a 25% improvement on a measly dual core isn’t half bad. It would certainly be faster on a quad. But I can take that even more down, I know I can.

So I spent about six hours in front of a code profiler and Visual Studio, tracing the bottlenecks and optimizing the “hot” paths. The major bottleneck was—as expected—the large and cumbersome resource updating subsystem. After rewriting it in what must have been ten different ways (each version slightly faster than the previous), I came up with the final design.

For the sake of reference, my profiler says that the old version takes on average 470 milliseconds to run through one XHTML file in TMB. After six hours messing with it, the final version takes 15 milliseconds. That’s 31 times faster.

File load for TMB? It’s now 3.3 seconds. Including the 0.3 for the rendering of the cover page, it’s 3.6.

So from 75 seconds in Sigil 0.1.9 down to 3.6 seconds in the development version of 0.2.0, I think I’ve done a pretty good job improving the loading speed.

For epubs with a “normal” number of images and computers with more logical CPU’s, it’s even faster.

epub name	Time – 0.1.9 (s)^[3]	Time – dev0.2.0 (s)^[3]
Three Men in a Boat	75	3.6
Sylvie and Bruno	82.5	6
Savage Stories of Conan	90.2	4.5
David Copperfield	98	3.2

These are all x86 times. For x64, knock off 10%.

Footnotes

[1] “Render” means that the colors for the pixels on the screen have to be calculated, i.e. the screen has to be “painted”.

[2] Written by Jerome K. Jerome and painstakingly hand-crafted by MobileRead user zelda_pinwheel. It’s a great book, you should read it. It’s also an amazing epub file, I use it as my main reference during Sigil development.

[3] x86 Windows version of Sigil, on Windows 7 x64. Computer is a Core 2 Duo 6400 with 4GB RAM.

[4] But horrible.

[5] Logical CPU’s are the number of actual cores on your system and any “virtual” cores from HyperThreading.

63 bits plus one

2010-02-08T01:01:00.003+01:00

Some of you may have noticed that while you can get precompiled binaries of Sigil for Linux in x86 and x64^[1] flavors, you only have an x86 version for Windows. Why? Well Microsoft made very a good job ensuring that 32 bit applications still run on 64 bit Windows. So there was no need for x64 Sigil. On Linux, it’s slightly different and nowhere near that easy.^[2]

So what are the main benefits (from an application’s point of view) to the newer instruction set/architecture?

The application has access to a larger address space (both physical and virtual),
Registers are 64 bits, not 32; this allows better/faster 64 bit math,
Double the number of general-purpose registers,
Double the number of XMM registers,
SSE instructions can be safely used knowing that all x64 CPU’s have to support them.

And several other things. The drawback is that your compiled code is now larger: pointers are all 64 bits, but the caches on the CPU’s are the same size. This is a big issue.

In the end, whether you’ll see direct performance improvements from moving to x64 depends entirely on the application. Some will, but some won’t. The Visual Studio devs have chosen not to make the transition just yet. I know other developers who have stated that their apps also behave worse on x64.

So you have to profile it to know for sure.

I used to think Sigil wouldn’t benefit from x64, performance wise. But then there was that nagging feeling telling me I should test it and see. The main reason why I didn’t want to do this is because I’d have to setup an entire new build system, with an entirely new Qt etc., etc. I already have four: Win x86, Lin x86, Lin x64 and Mac Universal. And building Qt (AGAIN) takes about five hours. You basically have to sit in front of the console, watching green text fly by because it likes to flake out in the middle and then you have to start it again. Not my ideal way to spend an afternoon.

But today I came down with a fever so I was too weak to do anything useful anyway. I may as well sit dozing in front of a screen. So I did.

Many hours later, after I got everything compiled and working, I set out to test Sigil 0.1.8 in x86 versus the same in x64. The test would measure the time it takes to load an epub book, from start to finish (this is easily the longest running operation in Sigil 0.1.8). I chose three epub files, did five runs for each on both versions, recorded the times and voila!

The x64 version was consistently 10% faster.

So my assumption was wrong. In light of these results, the next public release of Sigil (which should be 0.2.0) will in all likelihood include an x64 version for Windows.

In other news, I’ll be getting the 2010 version of Visual Studio when it ships in two months, for several reasons. But it will also provide tangible performance improvements for Sigil, since MSVC10 C++ compiler optimizations have improved. That’s another 10%.

And then there’s Sigil 0.2.0 and multithreaded, multi-flow loading. Now if only Nokia could get QtWebKit in a respectable shape…

Footnotes

[1] Some say “64 bit”, some “x64” or “x86-64” or “AMD64” or “Intel 64” and they all argue which is the correct one… I don’t care. I’m calling it “x64”, since that’s what people around me seem to be using. You know what I mean: the 64 bit extension to the x86 instruction set that AMD came up with and then Intel licensed.

[2] Also, Linux users tend to yell a lot more when their needs and/or desires are not met. Believe me, you don’t want to know.

Back in town

2010-01-31T02:45:00.003+01:00

Finals… finally… over!

I have a couple of papers due in a few days, so I’m still working, but after that I’ll have about three weeks of vacation time. The first few days I’ll spend soaking my brain in warm water and sleeping, but after that, I plan to spend most of my time working on Sigil.

In other news, I’ve decided to shift Sigil’s long-term development focus.

Way back when I started thinking about creating an eBook editor (more than a year ago), I primarily wanted to make an epub editor. The format badly needed one, since no such editor existed at the time. But I also wanted to replace Book Designer for most of the formats it exports. If you’re not familiar with BD, believe me, you don’t want to be. Let’s just leave it at “it’s horrible”. BD development is also completely dead (and has been for many years), and the application isn’t open source so no one can pick up where the original developers left off. The beauty of closed-source apps…

So the plan was this: make an eBook editor with an emphasis on epub, and then slowly add export support for all the other formats. A general-purpose eBook editor.

Back then I recognized several major formats I wanted to support, those that had more than a niche market: LRF (AKA BBeB), MOBI and LIT^[1]. But the scene has changed a lot since then…

LRF

First off, LRF is dead. It was proprietary to Sony’s reading devices, and Sony just switched their store over to epub. All of their Readers that support LRF support epub, too^[2]. LRF’s are not being sold anymore. Some people are still clinging to the format for personal use though; mostly because all epub books are displayed left-justified on PRS-505’s. Newer Sony Readers don’t have this limitation, but there it is. It still doesn’t change the fact that the format is dead. I’m sorry, that’s just the way it is.

I can’t say I’m sad it’s gone. There’s basically zero information on the format itself: what the OSS community knows about it, it knows mostly through reverse engineering. That’s not a happy place to be.

Anyway, it’s not on Sigil’s roadmap anymore. Even Kovid has stopped fixing bugs in Calibre’s LRF output plug-in.

LIT

This is the format for Microsoft Reader. It’s pretty much dead too. Even the website I linked got this shiny new look just a couple of months ago, until then it looked like something out of the ‘90s. I guess Microsoft thinks that eBooks are cool now and wants back in. They are, but the gravy train has left. LIT is only popular on PocketPC and Windows Mobile, and now epub is chewing that away. LIT is basically OEBPS 1.0, a somewhat direct precursor to epub.

MS abandoned LIT a long time ago, and now it’s too late to play catch up.

MOBI

I’ve never actually used MOBI. I hear it’s very popular with cell phones and similar devices. But ever since Amazon bought Mobipocket SA, the format seems to be on the way out. Amazon uses a custom version (AZW) for the Kindle’s, and you can’t read Kindle books on anything but a Kindle or an Amazon-sanctioned reading application. That’s not much of an open format. They also seem to be going with something called “Topaz”^[3].

The main problem for MOBI is that Mobipocket SA doesn’t allow device makers to support both MOBI DRM and some other eBook DRM on their electronic readers. And the manufacturers are seeing the writing on the wall and switching to Adobe’s Reader Mobile SDK which supports epub and PDF DRM. More money in that.

So MOBI is not in great shape, long-term. But it’s still popular enough that I’m keeping it in mind.

Conclusion

Seeing as how the other three major formats are either dead or dying—mostly thanks to epub—and the new player in town is solidifying his position quite nicely, it may be prudent to focus just on that. Sigil will from now on focus on bringing the best epub editing experience. It also means I won’t be spreading myself too thin.

Do understand that I’m only talking about MOBI, LIT and LRF exporting. Sigil will certainly one day be able to import files of those types (and others). That hasn’t changed one bit. Also, HTML and RTF export are not in jeopardy: those are very general-purpose and quite useful, so exporters for those will be written.

The current importers and exporters will be rewritten to use a plug-in architecture, so anyone who still wishes to develop export plug-ins for the above three formats will be able to do so with ease^[4]. All I’m saying is that I’m personally not going to develop them. Not any time soon, at least. They’re off the roadmap.

Sidenote

Google Analytics reports a 500% traffic spike on Sigil’s project page starting last Wednesday and peaking on Thursday. Seems a lot of people suddenly felt the urge to search for “epub editor”… I wonder what could have caused that…

I personally like the device and seeing as how the iBookstore will be selling epub books, it can only mean good things for the format. I plan on getting one of these when they become available. Hopefully this will also push Adobe to improve Adobe Digital Editions, which has more than a few rendering quirks. But honestly the biggest complaint I have against it is its utterly abysmal Unicode coverage with the default fonts. I’m pretty sure the iPad won’t have these problems: the videos clearly show user-selectable fonts, and the ones we can see preloaded^[5] are known to have good coverage.

Hurrah for competition.

Footnotes

[1] I’m not counting PDF since that’s not an eBook format, or RTF and HTML which are more general-purpose and have to be supported.

[2] The PRS-500 users can send their Readers for a free firmware upgrade for epub support, the PRS-505’s already has this upgrade and the others read it natively.

[3] Which I hear is actually Amazon’s proprietary implementation of epub. Go figure.

[4] And drop them in Sigil without violating the GPL if they choose not to provide the source code. I’m trying to lend a hand to the publishers that currently use Sigil.

[5] Like Times New Roman.

C'est la vie

2010-01-08T19:06:00.001+01:00

I haven’t been able to work on Sigil the last week, and I won’t be able to work on it for the next three weeks, either. University-related obligations are currently squeezing the life out of me and I can barely find time to sleep, let alone work on Sigil.

But the good news is that after this three week stint I’ll have loads of free time. More than enough to whip 0.2.0 into shape. I’ll just have to persevere until then.

Dublin Core in HTML and new releases

2009-12-30T21:04:00.001+01:00

OK, so Sigil 0.1.7 was released a couple of days ago. It’s a bugfix release for the 0.1.x branch while work on 0.2.0 is underway… and then 0.1.8 was released about an hour ago.

It seems that lately I can’t make a release without breaking something. This time it was (ironically) a fix for issue #139 that was the problem: CSS was being jumbled up into one line. Everything still worked, but if you are importing an epub book with hand-coded CSS, you want it to stay human-readable. This fix has been removed and will be looked at after the redesign.

Normally I’d release a “b” version, but this was major enough to warrant a whole new release treatment with a notification update pushed to everyone using 0.1.7.

But onto lighter topics… what did 0.1.7 bring? Mostly fixes, as I’ve said. View switching used to be less than reliable and could occasionally cause Sigil to crash. This was caused by differences in what WebKit thinks is a node, and what Qt’s QDom implementation thinks is one. Apparently they can’t agree whether continuous whitespace is just one text node or several, even after you tell both of them to normalize the tree. Or what is a child node, and whose child is it anyway. Plus a few other disagreements.

The net effect was that Sigil would crash if the tree-descending instructions it created from WebKit’s DOM couldn’t be executed upon the QDom DOM. This should be remedied now. If Sigil can’t quite figure out where it needs to scroll the View, it will scroll to the place before the ambiguity begins. In other words, as close as it can get. Still, this is only needed in rare cases.

The other important fixes are the line ending issue and the encoding detection improvements I talked about in previous posts. For a complete list of what was fixed, refer to the changelog.

But there was also one new feature: Sigil now imports HTML metadata if that metadata conforms to the Dublin Core standard. It’s been requested, and Kevin Hendricks kindly provided the code that implements this.

An interesting discussion on what is currently supported and what can still be added has started on MobileRead. Anyone interested in this feature should check out that thread to see examples of how you can use this new functionality, and if you happen to have any ideas for its improvement, we’d all love to hear them.

The Redesign: New GUI

2009-12-23T18:31:00.007+01:00

So how is oft-touted redesign progressing? Well it’s coming along nicely. Here’s what the new GUI looks like as of 5 minutes ago:

It should mostly stay that way.

So you have a fancy new “Book Browser” docked to the left by default; you can drag it to the right, or pull it out into its own window if you want. It lists all the files that make up your book (notice how the files are not renamed anymore), in neat little folders:

Text
Styles
Images
Fonts
Misc

You will be able to right-click in the Book Browser to add/remove files, rename them (F2 also works), open file information and the like. The file icons are queried from the OS, so whatever is the default for your system is what you’ll see in Sigil too.

The files are sorted alphabetically in all folders except the Text folder, where they are sorted according to the (imported) reading order. It’s also the only folder where you can rearrange the files, as the order determines the aforementioned reading order on export.

CSS files are now also preserved, and not loaded as style tags. Double-clicking one opens it in a special CSS editor with CSS syntax highlighting. Sigil 0.2.0 should also ship with an image viewing (but not editing) tab capability. Somewhere down the line when font embedding makes an appearance, there will be a font viewing tab as well. The “quick brown fox jumps over the lazy dog” kind. Double-click a font file in Windows and you’ll know what I mean.

The tabbed interface is also finally here. Very firefoxy. Yes, you can reorder the tabs. I hate it when applications don’t let me do that (like Foxit Reader), so Sigil is trying to be a bit smarter.

Before anyone goes all “ZOMG I CAN HAS NOW? MOAR!”, I’m merely 10% of the way through. This is just the UI, and its functionality. That’s the easy part. The very, very hard part was designing a new architecture for the back-end. But now comes the long and arduous task of integrating the old components into this new system. By “integrating” I mean rewriting. Not the Meta Editor, but the TOC editor definitely. All the importers too, plus the exporters. And a million other things. For instance, CSS and images are not currently loaded correctly in Book View.

After all of that is implemented, then of course I have to test everything.

When I feel I’ve gotten it to a point where others can try it, there will be a series of Release Candidates with the goal of shaking out any major bugs (and hopefully most of the minor). If I wasn’t so terribly bogged down with university work, I could do all this in a few weeks. But since I am, it will probably be February before you see something you can install.

Yeah, it sucks.

Line endings and why you shouldn’t trust Nokia (Part 2)

2009-12-14T15:33:00.006+01:00

So you’re using the Qt framework. You want to read an HTML file that probably has its encoding specified, but then again, maybe it doesn’t.

You’ll start off with something like this:

QString fullpath = QFileInfo( "test.xhtml" ).absoluteFilePath();
QFile file( fullpath );
file.open( QFile::ReadOnly | QFile::Text );

QByteArray data = file.readAll();
QString html_source = QTextCodec::codecForHtml( data )->toUnicode( data );

// This is here only for debug output
qDebug() << html_source;

Sounds reasonable, doesn’t it? And it looks reasonable too. But as I’ve said before, the QTextCodec::codecForHtml() function is horribly lacking. You’ll have to either write your own or at least augment this in some way.

But let’s say you have. Let’s say you called Magic, Inc.® and bought a replacement function that always returns the correct codec for a file. It doesn’t matter for this example, since for what I’m about to show you codecForHtml() is perfectly alright.

You now have this HTML file, encoded as UTF-16BE with Windows-style line endings (CRLF):

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
    <title>Test Page</title>
</head>
<body>
    <p>This is some text.</p>
    <p>This is some more text.</p>
</body>
</html>

You try to load this file with the above code. What does qDebug() print out?

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"

Yeah. That’s it.

Actually there’s quite a bit more data in html_source, but it’s all junky, garbled text after that line so it doesn’t even print out. So your file doesn’t load. You check the return of codeForHtml(), and by George, it actually got it right (it saw the BOM). It says the codec is UTF-16BE, which is correct. So what happened then?

Every experienced developer’s initial response is “OK so I screwed something up”. No one would ever initially doubt a mature, stable and popular framework like Qt. Blaming your framework is like blaming your compiler: you grow out of it when you realize it’s always you. But this time…

QFile has this great “feature” when you use QFile::Text as an OpenMode flag: if your file contains “CRLF” substrings (that is, Windows-style line breaks), it “automagically” converts them to “LF”, AKA Unix-style line breaks. Sadly, this doesn’t work so hot with when the file is UTF-16 encoded, since it screws up the byte stream. And now your file is unreadable. Dare I mention it all works perfectly with Unix newlines? Tends to show Qt’s Unix roots and focus…

Sure, if you use QTextStreamer, things somehow manage to work themselves out. But the point is QFile shouldn’t be doing this. The other problem caused by this line ending mangling nonsense is that files with Mac-style terminators (CR) get read as one large line of text. All of those “CR” characters go puff, and they’re gone! They’re not converted, they just disappear.

So Sigil yesterday switched to binary file reading and manual conversion of line endings. Importing old Mac files is now supported, as well as HTML files with UTF-16 encoding (which I assumed worked before… so much for assumptions).

Encodings and why you shouldn't trust Nokia (Part 1)

2009-12-14T02:31:00.008+01:00

So you have an HTML file, and you want to import it into Sigil. That's great. I hope your file has an encoding specified. You need to have an encoding specified. An encoding tells Sigil (and anything else that wants to read your document) how your characters are represented in the file.

Why? Well, computers work with numbers, not characters. You can only ever store numbers on a computer. So to store a character, we assign it a number, and those numbers are then stored in the files. But how do we know which number represents which character? We don't. We have to agree on a mapping of numbers to characters and vice versa. This mapping is called a character encoding.

So when your file doesn't have an encoding specified, Sigil doesn't know how to interpret the numbers in the file and translate them into the correct characters. And then your text is garbled.

So how do you specify an encoding in an HTML file? It's really simple:

<meta http-equiv="Content-type" content="text/html;charset=UTF-8" />

Just put that line of code into the <head> element. That's the XHTML way. Be sure that you're using the correct encoding! Using the wrong one is worse than using none at all; if you don't specify an encoding, a smart application may be able to guess correctly (we'll get to that in a moment).

You can also use the XML way, if your file is XHTML:

<?xml version="1.0" encoding="UTF-8"?>

Sometimes though, even specifying the correct encoding doesn't seem to be enough. As the linked issue illustrates, Sigil inherited a bug from Nokia's "QTextCodec::codecForHtml()" function, which in all its brilliant wisdom chooses to only look for the meta tag in the first 512 characters (if I recall correctly). If the encoding is further down than that… well it pretends it's not. So I worked around that.

And then we have the HTML files with no encoding whatsoever. There's a lot of them. So what should an application do then?

Guess, basically.

We can try to make it an educated guess, but let's not kid ourselves, it's still a guess.

Since v0.1.0, Sigil used to fallback on UTF-8 when it couldn't detect the encoding of your HTML file. No guessing, it just said: "what the hell, let's read this like it's UTF-8". It was a fairly good assumption. UTF-8 is the most widely used Unicode character encoding, and it's backward compatible with ASCII (which in layman's terms means "plain text"). A lot of people use it today as their default. It's as good a fallback as any.

But then three months ago, issue #133 made its way to the tracker. Suddenly, the UTF-8 fallback didn't seem so hot. Made me rethink my original decision. So I changed the code to use the default encoding of the computer running Sigil, AKA the "locale-aware" fallback. This seemed to work better, and landed in v0.1.4.

But still… every time anyone tried to load an UTF-8 encoded document (with no encoding), it would get garbled like crazy since no locale-specific encoding matches it.

After this bit me today (for the last time!), I decided it's time to revisit. After some searching around on the web, I found this little nugget of Perl code for checking whether a string was UTF-8 encoded. I also found a C translation of that code somewhere (can't find the link now), which I modified for use in Sigil. Here it is:

// This function goes through the entire byte array 
// and tries to see whether this is a valid UTF-8 sequence.
// If it's valid, this is probably a UTF-8 string.
bool Utility::IsValidUtf8( const QByteArray &string )
{
    // This is an implementation of the Perl code written here:
    //   http://www.w3.org/International/questions/qa-forms-utf-8
    //
    // Basically, UTF-8 has a very specific byte-pattern. This function
    // checks if the sent byte-sequence conforms to this pattern.
    // If it does, chances are *very* high that this is UTF-8.
    //
    // This function is written to be fast, not pretty.    

    if ( string.isNull() )

        return false;

    int index = 0;
    const unsigned char *bytes = NULL;

    while ( index < string.size() )
    {
        QByteArray dword = string.mid( index, 4 );

        if ( dword.size() < 4 )

            dword = dword.leftJustified( 4, '\0' );

        bytes = (const unsigned char *) dword.constData();

        // ASCII
        if (   bytes[0] == 0x09 ||
               bytes[0] == 0x0A ||
               bytes[0] == 0x0D ||
               ( 0x20 <= bytes[0] && bytes[0] <= 0x7E )                    
           ) 
        {
            index += 1;
        }

        // non-overlong 2-byte
        else if (  ( 0xC2 <= bytes[0] && bytes[0] <= 0xDF ) &&
                   ( 0x80 <= bytes[1] && bytes[1] <= 0xBF )            
                ) 
        {
            index += 2;
        }
           
        else if (  (     bytes[0] == 0xE0                         &&         // excluding overlongs 
                         ( 0xA0 <= bytes[1] && bytes[1] <= 0xBF ) &&
                         ( 0x80 <= bytes[2] && bytes[2] <= 0xBF )        ) || 
                  
                   (     (   ( 0xE1 <= bytes[0] && bytes[0] <= 0xEC ) ||     // straight 3-byte
                             bytes[0] == 0xEE                         ||
                             bytes[0] == 0xEF                     ) &&
                    
                         ( 0x80 <= bytes[1] && bytes[1] <= 0xBF )   &&
                         ( 0x80 <= bytes[2] && bytes[2] <= 0xBF )        ) ||

                   (     bytes[0] == 0xED                         &&         // excluding surrogates
                         ( 0x80 <= bytes[1] && bytes[1] <= 0x9F ) &&
                         ( 0x80 <= bytes[2] && bytes[2] <= 0xBF )        )
                 ) 
        {
            index += 3;
        }
 
          
        else if (    (   bytes[0] == 0xF0                         &&         // planes 1-3
                         ( 0x90 <= bytes[1] && bytes[1] <= 0xBF ) &&
                         ( 0x80 <= bytes[2] && bytes[2] <= 0xBF ) &&
                         ( 0x80 <= bytes[3] && bytes[3] <= 0xBF )      ) ||

                     (   ( 0xF1 <= bytes[0] && bytes[0] <= 0xF3 ) &&         // planes 4-15
                         ( 0x80 <= bytes[1] && bytes[1] <= 0xBF ) &&
                         ( 0x80 <= bytes[2] && bytes[2] <= 0xBF ) &&
                         ( 0x80 <= bytes[3] && bytes[3] <= 0xBF )      ) ||
                
                     (   bytes[0] == 0xF4                         &&         // plane 16
                         ( 0x80 <= bytes[1] && bytes[1] <= 0x8F ) &&
                         ( 0x80 <= bytes[2] && bytes[2] <= 0xBF ) &&
                         ( 0x80 <= bytes[3] && bytes[3] <= 0xBF )      )
                ) 
        {
            index += 4;
        }

        else
        {
            return false;
        }
    }

    return true;
}

This code is currently used as a last resort when the encoding isn't found. If this function says there's a high probability of the text being UTF-8, then it's read as such. Instant success with the test files I had on hand.

I also added an explicit check for the "encoding" XML attribute which the "QTextCodec::codecForHtml()" function apparently doesn't check for at all. Don't ask me why, ask Nokia.

This along with the UTF-16 auto-detection and line-ending fix (more on this tomorrow) makes Sigil now very robust to files with unspecified encodings. For the user, things should "just work" more often.