8. Release History

8.1. Development

  • Switch from alertify.js (deprecated) to SweetAlert2 library.

  • Add pre-ping to reuse of pooled connections towards a PostgreSQL souce. This ensures that pooled timed out connections are not reused for subsequent queries.

  • Improved configuration.

8.2. Release 1.3.4 (2020-03-17)

  • Renaming output columns can be performed by either clicking enter or clicking outside the edit box.

  • Add contextmenu on source tables for adding to output columns, in addition to the drag-and-drop capability.

  • Added column type information to sample window.

8.3. Release 1.3.3 (2020-03-10)

  • Store project configuration items as JSON.

  • Add timing info to exports.

  • Combine SFTP and Azure Blob recipes.

  • Fix for “is within date” and “is within date in dataset” when the date datatype is used instead of timestamp.

  • Improve PostgreSQL source handling of failed connections.

8.4. Release 1.3.2 (2019-11-26)

  • Remove query_string from permission request search to avoid HTTP requests being too large.

  • Make workspace_uuid and user_uuid children of event_payload.

8.5. Release 1.3.1 (2019-11-21)

  • Add workspace_uuid and user_uuid to auditlogging where available.

  • Improved session protection, it is now optional (default on), and provides more comprehensive logging.

  • Also logout at Azure AD when logging out of QueryBuilder.

  • Use concurrency defaults more suited to running in Kubernetes.

8.6. Release 1.3.0 (2019-11-15)

  • Switch from syslog to FluentD for audit logging.

  • Use given_name + family_name as a fallback when name is not available from the Azure AD.

  • Improve database connection management in cronrunner.

  • Determine join sequence now strictly along the conditions that were established in the join algorithm. Previously partitioned queries could result in a join sequence that did not resolve the partition soon enough for the source database.

8.7. Release 1.2.8 (2019-11-08)

  • Improve the join algorithm to ensure different ordering of projection columns result in the same optimal/minimal query.

  • Document audit logging.

  • Switch from Azure AD B2B to B2C.

  • Add concurrency configuration statements that control the number of worker processes started, and the size of the database connection pools.

8.8. Release 1.2.7 (2019-10-16)

  • Fix auditlog for errors in Graph API calls.

  • Fix Azure user registration to be separate from AD user registration. For AD users current active email address can be retrieved when needed, for Azure users this is not possible.

8.9. Release 1.2.6 (2019-10-07)

  • Improve job administration. Running multiple jobs in rapid succession could lead to multiple workers picking up the same job.

  • Improve error handling for the PostgreSQL backend, to accurately log the root cause of a query problem.

8.10. Release 1.2.5 (2019-10-03)

  • Improve DAS user creation under high load.

  • Use unique cursors per thread for the PostgreSQL backend. This improves asynchronous job execution.

  • Fix database migration for cron table.

8.11. Release 1.2.4 (2019-10-01)

  • Improve incremental resultset retrieval for SQL Server.

  • Prevent hierarchical CTEs for SQL Server.

  • Support for export via Azure blob storage.

  • Support for logins via Azure AD.

  • Improve drag and drop of columns to filters; a dragged item will now also be added to the projection, to satisfy the rule that a filter needs to be part of the projections.

8.12. Release 1.2.3 (2019-09-25)

  • Allow export scheduling of queries.

  • Fix cloning queries to correctly register parent dependencies, if appropriate.

  • Prevent redirection to http://app. In some cases the user was redirected to this internal URL, e.g. in multi-proxy environments or with IE11.

  • Show that query was asked approval for, instead of showing the same message as a query for which approval was never asked.

  • Do not allow requesting approval for an already approved query.

  • Remove removed projection columns from filter dropdown.

  • Improve incremental resultset retrieval for PostgreSQL.

  • Detect jobs marked running at the start of the job runner. These jobs do not have an active process actually moving data along, and need to be marked failed for the user to be able to detect problems.

8.13. Release 1.2.2 (2019-06-06)

  • Fix quick removal of columns that include a column that another query depends on. These columns cannot be removed, but all other around them can. Remove as much as possible and report back to the user what columns need further attention.

  • Add a heuristic that removes the first level of quotes around input strings to improve matching with how data will most likely be stored in the database.

  • Mark result retrieval done on validation error. This bug manifested itself e.g. when an empty filter rule was added, output columns were deleted and then all filter conditions were deleted.

  • Increase contrast of row counts.

  • Retain current page in pagination when reloading queries on the project page.

8.14. Release 1.2.1 (2019-05-13)

  • Allow quick removal of all output columns using a delete all button.

  • Allow customisation of status texts.

8.15. Release 1.2.0 (2019-04-29)

  • Multiple queries can now be selected in the project screen. This allows the user to request approval or export for multiple queries in one go.

  • Query approval requests now include details about the query: the query project; the number of rows in the query result; the populations and their sizes in the query result; the source tables accessed by the query.

  • Query approvals are now scheduled as asynchronous jobs, to aid in responsiveness for the ui, while allowing full analysis options for query detail determination.

  • Highlight query authorisation and export status conforming to AA accessibility standards.

  • Show the size of underlying source tables in the query ui.

  • Allow quick removal of all query filters using a delete all button.

  • Allow ENTER press in IE11 to run a filter.

  • Using new DAS API call for multiple contexts in allowed operations.

  • Enable searchable dropdown boxes for queries and attributes.

  • Added option to disable filtering on PII attributes.

  • Disable caching of dropdowns in IE11.

  • Add insert list modal for in/not in queries.

  • Allow customisation of approval and export button and modal texts.

  • Run programs within container as non-root user. Note that this requires setting the user in stack.yml: ‘user: mgrid’. Also note that the port number used inside the container has changed from 443 to 8443.

  • Allow running the container in read-only mode. The /etc/mgrid and /tmp directories need to be writable. In stack.yml: ‘read_only: true’ and under volumes ‘- /etc/mgrid’ and ‘- /tmp’.

8.16. Release 1.1.2 (2019-01-21)

  • Fix ‘results could not be loaded’ error.

  • More robust handling of deid batch sizes.

  • Renaming a query updates its name in DAS.

  • Autosize dropdown boxes for dataset items.

  • Fixed ability to override default CA certificates.

8.17. Release 1.1.1 (2019-01-14)

  • Fix filter list not displaying the full set of options.

8.18. Release 1.1.0 (2019-01-11)

  • Fix IE11 “is within date in dataset” and “is within date”.

  • MGRID branding update. Added CONTACT_EMAIL setting.

8.19. Release 1.0.9 (2018-12-31)

  • More robust handling of output column deletions.

  • Disallow two queries to share the same name.

  • Document model database upgrade procedure.

  • Improved database connection handling.

8.20. Release 1.0.8 (2018-12-11)

  • Fix IE11 caching of queries, sample and population.

  • Disallow simultaneous session logins for the same user.

  • Make session cookies harder to copy.

  • Disallow requesting approval on an empty query.

  • Improved test coverage of interaction between QueryBuilder and DAS.

  • Fix for data not refreshing correctly in the project page. If the user stayed on the project page when a request was approved, the export button was clickable, but did not perform the action.

8.21. Release 1.0.7 (2018-11-22)

  • Added ability to set the application log level.

  • Disable action on request approval and export buttons when they are grayed out.

  • Fix dragging and dropping into output column in IE11.

  • Fix new query being ready for export.

  • Improved job error handling. If a job fails, the user is informed in the UI and the job is not automatically retried.

  • An SFTP key error or mkdir error now closes the SFTP transport.

8.22. Release 1.0.6 (2018-11-14)

  • The SFTP session was not closed when an export finished, resulting in Paramiko logging about unhandled channel request keepalive@openssh.com.

  • When a user logs in and has no access to projects, a warning is shown instead of redirecting to the login page.

  • When generating the query for a join using multiple implicit keys now sort the implicit key names to ensure deterministic generation.

  • Gray out request approval button when user is already authorised and gray out export button when user is not allowed to export.

8.23. Release 1.0.5 (2018-11-08)

  • Allow query definitions that return 0 rows because of empty source tables.

  • Validate project creation and updating parameters more strictly to avoid incomprehensible errors on wrong content-type of client. For creation ‘name’ and ‘authgroup’ are mandatory, for updating at least one field must be present.

  • Improved ‘is within dates’ operator that also allows qualifiers for before/after boundaries. The filter description needs to be changed for old filters using between_dates. An database upgrade will remove these filters from existing query definitions.

  • If an empty database is found at startup, the database is initialized and stamped with the current source revision.

  • Screen height was fixed for tables, projections and filters, rendering the ui unusable if one of the three started getting large. Removed height restriction.

  • Allow list retrieval for all queries, even if a query has structural problems like broken filter specification. This allows users to still edit such queries.

  • Relax CSP style-src to allow jquery modifications to HTML elements.

  • Better client side validation; textually point out what filter attribute is wrong, do not attempt to load results if client side filter validation fails.

  • Improved error reporting for management API. Getting, putting and deleting an unknown project now results in 404 status and descriptive message in JSON.

8.24. Release 1.0.4 (2018-10-22)

  • Always retrieve displayName for AD users at AD login. This displayName will be stored as the users’ fullname, and be used to display ‘Signed in as fullname’ in all ui screens. If no displayName can be retrieved, we fall back to the part of the upn leading up to the first at sign.

  • Add management API documentation.

  • Fix incorrect handling of multiple schemas. This bug resulted in Querybuilder thinking that a table should also be present in schemas other than the one it is actually in.

  • Fix permission denied error on flask_session directory. The default setting now uses a subdirectory of /tmp.

8.25. Release 1.0.3 (2018-09-26)

  • Added two factor authentication using DUO

8.26. Release 1.0.2 (2018-09-11)

  • Ensure that a particular population is only counted once, even if multiple accessors (same population identifier, from different source tables) are available.

8.27. Release 1.0.1 (2018-08-09)

  • Increased font size to work better on small screens

  • Renamed query screen panes to ‘Output Columns, Filter Conditions, Preview Results’

  • Renamed ‘Overview’ screen to ‘Home’.

  • Allow drag-and-drop of tables into Output Columns for quick table selection

  • New dataset operator ‘between days around date’. Allows dates for current query to be compared to within a range on a different event query. E.g. allows first to select for a specific event in one query, and then to select items around that date in a second query.

  • Improved client side validation of filter values. This allows users to explore inputs and makes for a more intuitive ui.

  • Added default deidentification. World definition allows identification of sensitive attributes. During export these are deidentified, during preview they are shown in ui as ‘deidentified’.

  • Added default export destination. Export destination is set on a per provisioned project basis and allows users to simply export by pressing export with no further selection.

  • Added population counts for current query. Within the world definition the administrator can mark certain fields as identifiers for a population. E.g. patient_id identifies patient, or hadm_id identifies hospital_admission. If available these populations are tallied and displayed on the ‘Results preview’.

  • Authorisation via DAS now operates on the SQL query as posed by the user. This allows a data steward insight in the query that the users wishes to execute for export.

8.28. Release 1.0.0 (2018-06-09)

  • First release