How Do I Refresh Data from an SIS

Overview

The data that is viewable in Cedar Labs' Connect and CedarView products is stored in the Cedar Labs' ODS and available via a rich set of query APIs.  However, that data is sourced from Student Information Systems across the implementation.  The data in the CedarLabs' ODS is updated whenever an SIS sends an event for a data change (Update, Add, Delete) or when the CedarLabs' system issues a request for new data.  These requests are made automatically on a rotating schedule by the CedarLabs' system, but they can be made manually by a user of CedarView or Connect tools.

Anatomy of a Request

In SIF2 a system that wants to receive data from an SIS makes an asynchronous request for that data.  This is a 5 step process that looks like the following:

1. Requester (CedarExpert) issues a data request to HostedZone
2. HostedZone routes the request to the SIS agent's queue
3. The SIS agent picks the request off of their queue
4. The SIS responds to HostedZone with the data for the request
5. HostedZone puts the request on the Requester's queue
6. The Requester (CedarExpert), retrieves the SIS agent's response from the queue and stores the data in the CedarLabs' database.

The CedarLab's infrastructure combines all these steps into a single "Refresh From Source" call.

Refresh From Source

This call can be made from 3 places in the Cedar Labs' applications, CedarView, Global Search (Connect), and Create Requests (Connect).  Functionally these all do the same thing, but the filter applied to the records that are requested is different.

CedarView

In CedarView the filter field is free form.  This filter is passed directly onto the SIS with a refresh call.  This allows the user to request data from the SIS that might not yet be in the Cedar Labs' ODS.  For example if an enrollment for a student wasn't showing up, the user could issue a request for all StudentSchoolEnrollments for a student using the query StudentPersonalRefId=12345 and the SIS would send back all enrollments for that student.

The caveat here is that some SISs don't handle complex queries, even though data is returned from CedarLabs' ODS when the query is executed, this does not guarantee that the SIS will be able to handle the query.

Connect-Global Search

A refresh can be issued from a global search result in Connect.  This requests the data for a single SIF record (by RefId) or the constituent parts of a derived record by all the RefIds of all the SIF Records making up the derived record.

The caveat here is that if there is a record that is not showing in Connect the user would not be able to make the request.  For example, if an enrollment is not present for a student it could not be requested.

Connect-Create Request

The Create Request page allows the user to request all data for a given set of record types for many districts.  For example, all the StudentPersonals and StudentSchoolEnrollments could be requested across many districts using this page.  This page lets the user quickly do "bulk" requests, but not target specific data.

Hard Refresh

Sometimes data is deleted in an SIS without the SIS sending a Delete event.  When this occurs, the CedarLabs' system will eventually figure out that the record has been deleted as a part of the scheduled "Upkeep" process, but this is intentionally a conservative process and could take several days. 

One of the options for a Refresh operation is to do a "hard" refresh.  This first deletes the data in the CedarLabs' ODS and then requests the data from the SIS.  This can be a useful tool in cases where the SIS hasn't sent the correct events, but should be used sparingly as it leaves the data in the CedarLabs' ODS in a potentially invalid state until the SIS responds to the request.