Towards a better report format to replace legacy (paper based) ones (Word, pdf)

[actuary21c]

Dear AO

The question I'm asking here is: what format will/should most reports take in 2-5 years' time (as opposed to today's typical Word/pdf/Excel ones)?

The area I work in is principally web apps (mainly financial but other areas too).

It has taken time, but the web has now become a rich environment for displaying information to users, thanks to some powerful table and charting controls, which enable the user to run queries, change the display (via zooming in or out, also changing parameters, decide which series to include in the charts). In some areas (pivot charts) the web is not quite as good as say Microsoft Excel, but I think that will improve over the next year or so.

After the users/clients (and think of these ranging from Joe Public to senior executives/decision makers in corporations, regulators etc) have played around with the parameters/data and strategies on the web, the question then turns to one essentially required by quality control/documentation/compliance:

How do we responsibly record the key inputs/output and decisions taken?
So like the desktop counterparts that they are increasingly replacing, web apps need to produce Reports which the user/client can point to and say to compliance (and/or their own clients, eg if you are an actuary reporting to a Board of Directors, or the Trustees of a European pension plan):

Here is the key data used, here is the key output, and here are the key options/strategies/decisions to be discussed or taken.

Traditionally, such Reports have been produced in files which ultimately were intended to be printed, hence in Word or pdf files or similar. And so our web apps do this, because that is what our clients/users want.

But we ourselves (and I think the world generally) are increasingly moving away from paper. 5 to 10 years ago, there was little alternative to paper, because people on the move needed something that they could refer to, and share with others.

But now with tablets, smartphones and small netbooks becoming widely available, the case for paper (and reports designed to be printed on A4) is far from compelling. Here are some key disadvantages of the legacy reports designed to be printed on A4/letter:

1. The width is tiny when compared to what can be displayed on the tablets and other computer devices that most clients use during most of their daily work OR leisure time. (With the exception of phones, but if people are looking at detailed financial matters, they will probably accept that to do this efficiently, they should use their pc or tablet rather than a phone).
   
2. For people with vision problems (including many people over 50, which is the age at which the eye muscles start to weaken, and bi-focal lenses are often needed), there is little or no zoom facility. (Yes there is in a Word document on a screen, as opposed to a printed copy, but the content's width will already have been shrunk down as part of formatting it for paper printing). So in particular, tables and charts look small and compressed compared to how they look either in the web app, or when exported to Excel.
   
3. Many data records have too many fields to fit on a printed page. Once you get past about 7 or 8 fields, it is much better to display such data in electronic format, such as used by Excel or tables (as on web pages and in database packages like Access, SQLServer, MySql etc). So imo Excel files play an important part in reporting. (But I'm asking below if we can improve on this)
   
4. It is much harder for the user to interact with a Word or pdf file than a more friendly electronic format (eg an Excel file). Of course for compliance purposes, a record needs to be kept of the original data used, so a master copy needs to be kept (eg as ReadOnly in the cloud) of the original Excel or other electronic files in the report, but there is no reason why the user shouldn't be allowed to perform further analysis on the data, or even on modified copies of the data.
   
5. Paper files need to be scanned to be backed up. So for compliance purposes, electronic is better than paper. Of course, Word files can be backed up, but this goes back to the main point: why produce reports in Word if there is much less need to print the report out?

So, because of client demand, our reports currently typically consist of a zip file containing one or more Word files (for printing purposes) plus one or more Excel appendix files (to contain the data inputs used, and the output data).

One possible improvement is to stop using Word, and instead put that content in an Excel worksheet, which is still designed to be printed on A4 or letter paper, and this is useful for producing pdf or paper copies of Company Accounts (eg to get the P&L and Balance Sheet on the same page, with Notes on a second page). But using Excel to produce A4 is still messy and still a workaround based on the legacy issue of paper.

I can't help feeling that a better way is possible and will emerge over the next few years, as the use of reports printed out on paper continues to decline.

What do other AOers do with regard to reporting/documentation? What are your thoughts please? There's scope for someone here to develop/invent a new and better format.

[campbell]

We've been moving away from paper for decades.

Our printer still gets used a hell of a lot.

(Of course, I work in a place whose business is to publish, but still)

[JMO]

Originally Posted by campbell

We've been moving away from paper for decades.

Our printer still gets used a hell of a lot.

(Of course, I work in a place whose business is to publish, but still)

^ This.

I remember when the Computer Science* Section Council was asked for a recommendation on a standardized file format to use for SOA publications. At the time the obvious choice was .pdf. I don't know if anyone has given more than a passing thought to revisiting the question, but maybe it should happen.

*Later renamed the Technology section. I was no longer on the council when the renaming happened.

[JMO]

I always wondered why the printing default for excel was portrait layout. On the screen you saw landscape. Sure, you could change the default, but for quick stuff it would have been more convenient the other way - and let people who want portrait change from the default.
/

[actuary21c]

Originally Posted by JMO

^ This.

I remember when the Computer Science* Section Council was asked for a recommendation on a standardized file format to use for SOA publications. At the time the obvious choice was .pdf. I don't know if anyone has given more than a passing thought to revisiting the question, but maybe it should happen.

*Later renamed the Technology section. I was no longer on the council when the renaming happened.

Thanks mpc and JMO. I agree that people have stubbornly stuck to paper for decades - as I said, the game changer here imo is the increasing use of tablets, smart phones and net books.

Just to clarify given that you refer above to SOA publications: I'm not talking about news and adverts of the sort that appear in magazines (eg "The Actuary") but business reports, eg:

-valuation reports for a pension plan or insurance company
-financial wealth projections for an individual or couple
-investment performance reports
-tax computations
-sales reports

Academic papers (as in technical journals for actuaries, mathematicians, scientists etc) could also benefit from a better format.

Pdfs are not the solution imo because they are still limited by being designed for printing (A4 landscape is still not as wide as a web format, which will have horizontal and vertical scrollbars). pdfs are also not very interactive (sure you can add comments, but copying and pasting data out of them can be very messy, similarly exporting data).

Maybe html format is an improvement over pdf because the report could then be designed to look as similar to the webapp's output as possible. It won't be as rich as the latter, because it won't have (given that it is designed to standalone) as much interactivity (from javascript calling server code), but it could still have some interactivity via javascript. (Update: one disadvantage of html files - and XPS too I think - over Word, pdf and Excel ones is that it is difficult to add your own notes/comments to them. Update: I suppose you could open an html file with a text editor and add comments next to the relevant html code - maybe there is - or will be - a way in html5 to make those comments easily visible/accessible when viewing the document?).

Incidentally it does bug me that as developers we spend a lot of time getting the web (or desktop) app's user interface to display the output in as user friendly a format as possible, and then when it comes to producing a report, the same sort of work has to be done all over again. It would be an improvement for everyone if the reporting process could be aligned much more closely to the actual app's output process.

[actuary21c]

i was saying that a downside of html files is that it is not easy for users to add comments/notes to the document and view them.

I've just searched and found this which is interesting:

http://annotatorjs.org/

I've not had time to see how/where the comments are stored - it may be that this only works for live websites, not standalone reports stored in html format.

PS:

One could add an html title attribute to text within an html page, but, this is messy without a javascript library to do the user interface:

Eg if the following is saved as MyFile.html and viewed in a browser, then the comment "Here is my comment!" appears when the mouse is moved over the "with a comment" text (which is highlighted in yellow to indicate to the user that a comment exists):

<html>
<head>
<style>
.commented {
background-color: rgba(255,255,10,0.3);
}
</style>
</head>
<body>
<div>
<span>Here is some text <span class="commented" title = "This is my comment!">with a comment</span> in the middle of it.</span>
</div>
</body>
</html>

A javascript library would be needed to make it easy for the user to add/edit/remove comments without having to use a text editor. In this example, the comments are stored by adding them to the document.

[Ron Weasley]

It sounds like the OP is trying to combine too many things. Today, dashboards provide a type of interactive reporting. This is much different than the need to document, say, an actuarial statement of opinion.

Current pushes toward reproducible research, apps, etc. will change what the world looks like in five to ten years, but nothing goes completely to zero. Even engineering schools still have libraries with paper books, even if the books are retrieved by robots. At least they don't still use scrolls (well, unless you want to count the drawings and banners that get printed on large format plotters).

Heck, you can still buy buggy whips on Amazon: http://www.amazon.com/Weaver-Leather...rds=buggy+whip

[Chuck]

I'm no expert, but perhaps the solution lies with not so much whether to use Excel, Word, Acrobat, or a browser, but rather to have all software use some kind of universal data storage format like XML (which I think already happens to some degree, eg HTML is a subset of XML)?

Then you can do all the things you want to do with your data (modify, use, present, etc) by choosing which app uses the data. Somehow maintain rules for all the apps so that data can be read and modified seamlessly from app to app.

[actuary21c]

Originally Posted by Chuck

I'm no expert, but perhaps the solution lies with not so much whether to use Excel, Word, Acrobat, or a browser, but rather to have all software use some kind of universal data storage format like XML (which I think already happens to some degree, eg HTML is a subset of XML)?

Then you can do all the things you want to do with your data (modify, use, present, etc) by choosing which app uses the data. Somehow maintain rules for all the apps so that data can be read and modified seamlessly from app to app.

Interesting thought. NB XML is a very verbose/inefficient format however, with tags repeated for each record. Excel and Word files are already (at least if you use the OpenXML standard format, see https://support.office.com/en-ca/art...1-31ef14555b18) stored as zip folders containing xml files.

[actuary21c]

I've just looked at a very complex/rich calculation output page from one of our webapps and saved it to my desktop (via Ctrl+S). This produces an html file and a folder with 26 files in it (images and javascript files).

I can open the file from my desktop and it half works. The half that doesn't is that I get some javascript errors (presumably because the file is not on the server). I can tell the browser to stop showing me error dialogs from that page, and after that (the half that works) I can see all the tables and charts and other content that I had in the web app, and I've also got some of the interactivity (in this case panel bars that open and close to show and hide content).

It certainly is richer (and looks far better) than a Word report page.

When I have time, I might explore the above approach further. In the meantime, the point of posting this on the AO (as opposed to just internally) is to try and use "the wisdom of the crowd", ie bounce these ideas off you in the hope that some of you may find them interesting, and may share your ideas too.