Aggregated indicators & history

[mdutoo]

We'd need to provide a list of indicators aggregated across the SOA model, such as :

• number of deployed services (where endpoint.available == true)
• number of services without documentation (with document folder empty and no code-extracted documentation)
• number of services without consumer
• number of services not used in a business process' System

What's the best way to do it (simplest & most scalable) ? Some ideas :

• do it in NXQL (limits ?) or visiting the whole repository. This can be helped by storing (partial) aggregates at all levels of tree.
• do it in BIRT reporting. Same idea and limits as before, just a pretty configuration & display UI if I get it : http://community.nuxeo.com/5.3/books/nuxeo-book/html/nuxeo-reporting-birt.html
• do it in SQL over underlaying nuxeo database (limits ?)
• on changes, save it (JDBC) in a database dedicated to doing this kind of aggregate / reporting queries (in SQL)

Also, how to keep track of the history of those indicators, in order to display them as graphs ? Merely a dedicated database with the hitoricized values of those indicators ? (up to Talend ETL-filling it and going to OLAP...)

UPDATE following tdelprat's answer :

• TODO OW study where is NXQL is possible
• TODO OW support intermediary aggregation with 2. (extend it like Count & Size or not ; see in QuotaComputerProcessor) how it first computes new count values and only then updates them,
• TODO OW what about deployment / sla events (mashup info from SOA platform / monitoring) ?
• TODO OW support live vs daily / monthly entry in audit logs (available by EJBQL in AuditReader, "category" qualifier maybe ex. "dailySoaStats" / "monthlySoaStats", extendedInfo) or elsewhere (see Open Wide projects statistics, sample WebAccessLog)

[mdutoo]

tdelprat answer :

The BIRT option seems pretty overkill to me, at least for just counters.
So, I think there are mainly 2 approaches :

1 - do it in NXQL
There is no count support in NXQL, but you can simply use CoreSession.queryAndFetch that can return just the UUIDs and not all the documents.
As long as you don't have millions of items, this will work without problem.

2 - pre-compute
An other approach is to pre-compute this number and store it in a facet on the documents.
We have an addons that does similar things :

• maintain a count of the children on each folderish document
• maintain a count of blob size on all documents

These are 2 default contributions to the nuxeo-quota service :
https://github.com/nuxeo/nuxeo-quota

About keeping indicator history : you may want to use the Audit log for that...